2024-04-22T15:31:45.360 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-22T15:31:45.361 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668390 --owner scheduled_adking@teuthology --description orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --verbose --name adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi -- /home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668390/orig.config.yaml 2024-04-22T15:31:45.417 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T15:31:45.507 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668390 branch: wip-adk-testing-2024-04-22-0809 description: orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: adking@redhat.com first_in_suite: false job_id: '7668390' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/v18.2.1 ' name: adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-adk-testing-2024-04-22-0809 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-adk-testing-2024-04-22-0809 sha1: 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa owner: scheduled_adking@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 523 sha1: 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa sleep_before_teardown: 0 subset: 1/16 suite: orch:cephadm suite_branch: wip-adk-testing-2024-04-22-0809 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa targets: smithi016.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGuIltwlxu+b+CR4WWLji5QXGaO6uWoNfEcpueKIHSMXXn0Vc+eLuPH72aHufXGCm1xtWKUHVxpCVHLmZ0+DMvo= smithi027.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF73ngVw+aTmO1YSGJa7hm+0juPGZDkEac+b0a5GAXQhiqliVIgThAOnjisQug3tc9ZpZGN+Bdm/f9H/wUB63Bg= tasks: - install: exclude_packages: - ceph-volume tag: v18.2.1 - print: '**** done install task...' - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.io/ceph/ceph:v18.2.1 roleless: true - print: '**** done end installing v18.2.1 cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - ceph-fuse: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-22_14:50:52 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: adking verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-04-22T15:31:45.507 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/qa; will attempt to use it 2024-04-22T15:31:45.508 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/qa/tasks 2024-04-22T15:31:45.508 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-22T15:31:45.509 INFO:teuthology.task.internal:Checking packages... 2024-04-22T15:31:45.533 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5883d494d3973b30b3f03e4a3f0e1e06e3c131aa' 2024-04-22T15:31:45.533 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T15:31:45.533 INFO:teuthology.packaging:ref: None 2024-04-22T15:31:45.533 INFO:teuthology.packaging:tag: None 2024-04-22T15:31:45.533 INFO:teuthology.packaging:branch: wip-adk-testing-2024-04-22-0809 2024-04-22T15:31:45.534 INFO:teuthology.packaging:sha1: 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:31:45.534 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-adk-testing-2024-04-22-0809 2024-04-22T15:31:45.757 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3223.g5883d494 2024-04-22T15:31:45.759 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-22T15:31:45.768 INFO:teuthology.task.internal:no buildpackages task found 2024-04-22T15:31:45.768 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-22T15:31:45.799 INFO:teuthology.task.internal:Saving configuration 2024-04-22T15:31:45.820 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-22T15:31:45.826 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T15:31:45.852 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668390', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 15:25:29.844895', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGuIltwlxu+b+CR4WWLji5QXGaO6uWoNfEcpueKIHSMXXn0Vc+eLuPH72aHufXGCm1xtWKUHVxpCVHLmZ0+DMvo='} 2024-04-22T15:31:45.870 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668390', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 15:25:29.843310', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF73ngVw+aTmO1YSGJa7hm+0juPGZDkEac+b0a5GAXQhiqliVIgThAOnjisQug3tc9ZpZGN+Bdm/f9H/wUB63Bg='} 2024-04-22T15:31:45.870 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-22T15:31:45.876 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-04-22T15:31:45.876 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-04-22T15:31:45.876 INFO:teuthology.run_tasks:Running task console_log... 2024-04-22T15:31:45.967 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f297e2db160>, signals=[15]) 2024-04-22T15:31:45.968 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-22T15:31:45.973 INFO:teuthology.task.internal:Opening connections... 2024-04-22T15:31:45.974 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2024-04-22T15:31:45.976 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T15:31:46.054 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi027.front.sepia.ceph.com 2024-04-22T15:31:46.055 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T15:31:46.134 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-22T15:31:46.141 DEBUG:teuthology.orchestra.run.smithi016:> uname -m 2024-04-22T15:31:46.160 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2024-04-22T15:31:46.160 DEBUG:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2024-04-22T15:31:46.219 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="CentOS Stream" 2024-04-22T15:31:46.219 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="9" 2024-04-22T15:31:46.219 INFO:teuthology.orchestra.run.smithi016.stdout:ID="centos" 2024-04-22T15:31:46.219 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="rhel fedora" 2024-04-22T15:31:46.219 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="9" 2024-04-22T15:31:46.219 INFO:teuthology.orchestra.run.smithi016.stdout:PLATFORM_ID="platform:el9" 2024-04-22T15:31:46.219 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T15:31:46.219 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31" 2024-04-22T15:31:46.219 INFO:teuthology.orchestra.run.smithi016.stdout:LOGO="fedora-logo-icon" 2024-04-22T15:31:46.219 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T15:31:46.219 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://centos.org/" 2024-04-22T15:31:46.220 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T15:31:46.220 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T15:31:46.220 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T15:31:46.220 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2024-04-22T15:31:46.242 DEBUG:teuthology.orchestra.run.smithi027:> uname -m 2024-04-22T15:31:46.260 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64 2024-04-22T15:31:46.261 DEBUG:teuthology.orchestra.run.smithi027:> cat /etc/os-release 2024-04-22T15:31:46.319 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="CentOS Stream" 2024-04-22T15:31:46.319 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="9" 2024-04-22T15:31:46.319 INFO:teuthology.orchestra.run.smithi027.stdout:ID="centos" 2024-04-22T15:31:46.319 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="rhel fedora" 2024-04-22T15:31:46.319 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="9" 2024-04-22T15:31:46.319 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el9" 2024-04-22T15:31:46.319 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T15:31:46.319 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31" 2024-04-22T15:31:46.320 INFO:teuthology.orchestra.run.smithi027.stdout:LOGO="fedora-logo-icon" 2024-04-22T15:31:46.320 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T15:31:46.320 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://centos.org/" 2024-04-22T15:31:46.320 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T15:31:46.320 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T15:31:46.320 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T15:31:46.320 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2024-04-22T15:31:46.342 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-22T15:31:46.351 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-22T15:31:46.374 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-22T15:31:46.375 DEBUG:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2024-04-22T15:31:46.378 DEBUG:teuthology.orchestra.run.smithi027:> test '!' -e /home/ubuntu/cephtest 2024-04-22T15:31:46.396 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-22T15:31:46.402 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-22T15:31:46.402 DEBUG:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2024-04-22T15:31:46.436 DEBUG:teuthology.orchestra.run.smithi027:> test -z $(ls -A /var/lib/ceph) 2024-04-22T15:31:46.463 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-22T15:31:46.520 INFO:teuthology.run_tasks:Running task kernel... 2024-04-22T15:31:46.535 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-22T15:31:46.535 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-22T15:31:46.535 DEBUG:teuthology.orchestra.run.smithi016:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T15:31:46.536 DEBUG:teuthology.orchestra.run.smithi027:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T15:31:46.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T15:31:46.553 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-04-22T15:31:46.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T15:31:46.554 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-04-22T15:31:46.611 INFO:teuthology.orchestra.run.smithi016.stdout:5.14.0-437.el9.x86_64 2024-04-22T15:31:46.611 INFO:teuthology.task.kernel:Running kernel on smithi016: 5.14.0-437.el9.x86_64 2024-04-22T15:31:46.611 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum install -y kernel 2024-04-22T15:31:46.612 INFO:teuthology.orchestra.run.smithi027.stdout:5.14.0-437.el9.x86_64 2024-04-22T15:31:46.613 INFO:teuthology.task.kernel:Running kernel on smithi027: 5.14.0-437.el9.x86_64 2024-04-22T15:31:46.613 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum install -y kernel 2024-04-22T15:31:48.795 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-22T15:31:50.161 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.0 MB 00:01 2024-04-22T15:31:51.605 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-04-22T15:31:51.979 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-04-22T15:31:55.789 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-04-22T15:31:57.371 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.2 MB 00:01 2024-04-22T15:31:59.155 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 16 kB 00:00 2024-04-22T15:32:03.340 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 5.2 MB/s | 21 MB 00:04 2024-04-22T15:32:08.715 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-22T15:32:10.654 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T15:32:10.655 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T15:32:10.751 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:32:10.752 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-04-22T15:32:10.753 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:32:10.846 DEBUG:teuthology.orchestra.run.smithi016:> echo no | sudo yum reinstall kernel || true 2024-04-22T15:32:10.947 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 1.2 kB/s | 16 kB 00:13 2024-04-22T15:32:11.368 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 03:32:08 PM UTC. 2024-04-22T15:32:11.505 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:32:11.512 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:32:11.512 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-04-22T15:32:11.512 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:32:11.512 INFO:teuthology.orchestra.run.smithi016.stdout:Reinstalling: 2024-04-22T15:32:11.512 INFO:teuthology.orchestra.run.smithi016.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T15:32:11.512 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:32:11.512 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-22T15:32:11.512 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:32:11.512 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:32:11.512 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 255 k 2024-04-22T15:32:11.513 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 0 2024-04-22T15:32:11.513 INFO:teuthology.orchestra.run.smithi016.stderr:Operation aborted. 2024-04-22T15:32:11.569 INFO:teuthology.orchestra.run.smithi016.stdout:Is this ok [y/N]: 2024-04-22T15:32:11.569 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum reinstall -y kernel || true 2024-04-22T15:32:12.084 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:04 ago on Mon 22 Apr 2024 03:32:08 PM UTC. 2024-04-22T15:32:12.205 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:32:12.206 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:32:12.206 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-04-22T15:32:12.207 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:32:12.207 INFO:teuthology.orchestra.run.smithi016.stdout:Reinstalling: 2024-04-22T15:32:12.207 INFO:teuthology.orchestra.run.smithi016.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T15:32:12.207 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:32:12.207 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-22T15:32:12.207 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:32:12.207 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:32:12.207 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 255 k 2024-04-22T15:32:12.207 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 0 2024-04-22T15:32:12.207 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-22T15:32:12.742 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-437.el9.x86_64.rpm 774 kB/s | 255 kB 00:00 2024-04-22T15:32:12.742 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:32:12.742 INFO:teuthology.orchestra.run.smithi016.stdout:Total 476 kB/s | 255 kB 00:00 2024-04-22T15:32:12.760 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-22T15:32:12.931 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-22T15:32:12.932 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-22T15:32:12.936 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-22T15:32:12.937 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-22T15:32:13.077 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-22T15:32:13.143 INFO:teuthology.orchestra.run.smithi016.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T15:32:13.143 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T15:32:13.792 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T15:32:13.792 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T15:32:14.502 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T15:32:14.502 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:32:14.502 INFO:teuthology.orchestra.run.smithi016.stdout:Reinstalled: 2024-04-22T15:32:14.502 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T15:32:14.502 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:32:14.502 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:32:14.672 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T15:32:14.720 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T15:32:14.720 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T15:32:14.720 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T15:32:14.721 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-22T15:32:14.721 DEBUG:teuthology.orchestra.run.smithi016:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T15:32:14.756 INFO:teuthology.orchestra.run.smithi016.stdout:ttyS1 2024-04-22T15:32:14.780 DEBUG:teuthology.parallel:result is None 2024-04-22T15:32:30.291 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 1.1 MB/s | 21 MB 00:19 2024-04-22T15:32:35.638 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 10 kB/s | 1.7 kB 00:00 2024-04-22T15:32:37.569 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T15:32:37.570 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T15:32:37.660 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:32:37.662 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-04-22T15:32:37.662 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:32:37.790 DEBUG:teuthology.orchestra.run.smithi027:> echo no | sudo yum reinstall kernel || true 2024-04-22T15:32:38.308 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 03:32:35 PM UTC. 2024-04-22T15:32:38.441 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:32:38.446 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:32:38.446 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-04-22T15:32:38.446 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:32:38.446 INFO:teuthology.orchestra.run.smithi027.stdout:Reinstalling: 2024-04-22T15:32:38.447 INFO:teuthology.orchestra.run.smithi027.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T15:32:38.447 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:32:38.447 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-22T15:32:38.447 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:32:38.447 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:32:38.447 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 255 k 2024-04-22T15:32:38.447 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 0 2024-04-22T15:32:38.447 INFO:teuthology.orchestra.run.smithi027.stderr:Operation aborted. 2024-04-22T15:32:38.511 INFO:teuthology.orchestra.run.smithi027.stdout:Is this ok [y/N]: 2024-04-22T15:32:38.512 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum reinstall -y kernel || true 2024-04-22T15:32:39.026 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:04 ago on Mon 22 Apr 2024 03:32:35 PM UTC. 2024-04-22T15:32:39.149 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:32:39.149 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:32:39.149 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-04-22T15:32:39.149 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:32:39.149 INFO:teuthology.orchestra.run.smithi027.stdout:Reinstalling: 2024-04-22T15:32:39.149 INFO:teuthology.orchestra.run.smithi027.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T15:32:39.149 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:32:39.149 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-22T15:32:39.149 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:32:39.149 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:32:39.149 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 255 k 2024-04-22T15:32:39.150 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 0 2024-04-22T15:32:39.150 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-22T15:32:39.690 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-5.14.0-437.el9.x86_64.rpm 598 kB/s | 255 kB 00:00 2024-04-22T15:32:39.690 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:32:39.690 INFO:teuthology.orchestra.run.smithi027.stdout:Total 470 kB/s | 255 kB 00:00 2024-04-22T15:32:39.710 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-22T15:32:39.920 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-22T15:32:39.920 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-22T15:32:39.925 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-22T15:32:39.925 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-22T15:32:40.047 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-22T15:32:40.116 INFO:teuthology.orchestra.run.smithi027.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T15:32:40.117 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T15:32:40.614 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T15:32:40.615 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T15:32:41.311 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T15:32:41.311 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:32:41.311 INFO:teuthology.orchestra.run.smithi027.stdout:Reinstalled: 2024-04-22T15:32:41.311 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T15:32:41.311 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:32:41.311 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:32:41.471 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T15:32:41.519 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T15:32:41.519 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T15:32:41.519 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T15:32:41.519 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-22T15:32:41.520 DEBUG:teuthology.orchestra.run.smithi027:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T15:32:41.549 INFO:teuthology.orchestra.run.smithi027.stdout:ttyS1 2024-04-22T15:32:41.577 DEBUG:teuthology.parallel:result is None 2024-04-22T15:32:41.578 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-22T15:32:41.590 INFO:teuthology.task.internal:Creating test directory... 2024-04-22T15:32:41.590 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T15:32:41.593 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T15:32:41.632 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-22T15:32:41.639 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-22T15:32:41.667 INFO:teuthology.task.internal:Creating archive directory... 2024-04-22T15:32:41.668 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T15:32:41.671 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T15:32:41.701 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-22T15:32:41.708 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-22T15:32:41.708 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T15:32:41.739 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T15:32:41.779 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T15:32:41.780 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T15:32:41.790 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T15:32:41.791 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T15:32:41.792 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-22T15:32:41.800 INFO:teuthology.task.internal:Configuring sudo... 2024-04-22T15:32:41.800 DEBUG:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T15:32:41.835 DEBUG:teuthology.orchestra.run.smithi027:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T15:32:41.861 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-22T15:32:41.871 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-22T15:32:41.871 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T15:32:41.904 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T15:32:41.919 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T15:32:42.008 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T15:32:42.054 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T15:32:42.137 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T15:32:42.180 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-22T15:32:42.181 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T15:32:42.247 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T15:32:42.289 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T15:32:42.342 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T15:32:42.426 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T15:32:42.466 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-22T15:32:42.466 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T15:32:42.530 DEBUG:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2024-04-22T15:32:42.533 DEBUG:teuthology.orchestra.run.smithi027:> sudo service rsyslog restart 2024-04-22T15:32:42.566 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T15:32:42.605 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T15:32:43.121 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-22T15:32:43.129 INFO:teuthology.task.internal:Starting timer... 2024-04-22T15:32:43.129 INFO:teuthology.run_tasks:Running task pcp... 2024-04-22T15:32:43.155 INFO:teuthology.run_tasks:Running task selinux... 2024-04-22T15:32:43.187 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-22T15:32:43.188 DEBUG:teuthology.orchestra.run.smithi016:> sudo service auditd rotate 2024-04-22T15:32:43.240 INFO:teuthology.orchestra.run.smithi016.stdout:Rotating logs: 2024-04-22T15:32:43.243 DEBUG:teuthology.orchestra.run.smithi027:> sudo service auditd rotate 2024-04-22T15:32:43.298 INFO:teuthology.orchestra.run.smithi027.stdout:Rotating logs: 2024-04-22T15:32:43.300 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-22T15:32:43.300 DEBUG:teuthology.orchestra.run.smithi016:> /usr/sbin/getenforce 2024-04-22T15:32:43.322 INFO:teuthology.orchestra.run.smithi016.stdout:Permissive 2024-04-22T15:32:43.322 DEBUG:teuthology.orchestra.run.smithi027:> /usr/sbin/getenforce 2024-04-22T15:32:43.352 INFO:teuthology.orchestra.run.smithi027.stdout:Permissive 2024-04-22T15:32:43.353 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi016.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi027.front.sepia.ceph.com': 'permissive'} 2024-04-22T15:32:43.353 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T15:32:43.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T15:32:43.390 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T15:32:43.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T15:32:43.419 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-22T15:32:43.419 DEBUG:teuthology.orchestra.run.smithi016:> sudo /usr/sbin/setenforce permissive 2024-04-22T15:32:43.458 DEBUG:teuthology.orchestra.run.smithi027:> sudo /usr/sbin/setenforce permissive 2024-04-22T15:32:43.484 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-22T15:32:43.497 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-22T15:32:43.503 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-22T15:32:43.629 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-22T15:32:43.644 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-22T15:32:43.646 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi016.front.sepia.ceph.com,smithi027.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-22T15:36:20.912 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi016.front.sepia.ceph.com'), Remote(name='ubuntu@smithi027.front.sepia.ceph.com')] 2024-04-22T15:36:20.914 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-22T15:36:20.915 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T15:36:20.995 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-04-22T15:36:21.078 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-22T15:36:21.078 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-04-22T15:36:21.079 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T15:36:21.155 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-04-22T15:36:21.235 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-04-22T15:36:21.236 INFO:teuthology.run_tasks:Running task clock... 2024-04-22T15:36:21.247 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-22T15:36:21.247 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T15:36:21.247 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T15:36:21.250 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T15:36:21.250 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T15:36:21.279 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T15:36:21.293 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T15:36:21.307 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T15:36:21.317 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: ntpd: command not found 2024-04-22T15:36:21.321 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T15:36:21.328 INFO:teuthology.orchestra.run.smithi016.stdout:506 Cannot talk to daemon 2024-04-22T15:36:21.342 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T15:36:21.345 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: ntpd: command not found 2024-04-22T15:36:21.356 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T15:36:21.356 INFO:teuthology.orchestra.run.smithi027.stdout:506 Cannot talk to daemon 2024-04-22T15:36:21.370 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T15:36:21.384 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T15:36:21.417 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-04-22T15:36:21.420 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T15:36:21.420 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-04-22T15:36:21.420 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T15:36:21.420 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T15:36:21.420 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T15:36:21.420 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T15:36:21.441 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-04-22T15:36:21.444 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T15:36:21.444 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-04-22T15:36:21.444 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T15:36:21.444 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T15:36:21.444 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T15:36:21.444 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T15:36:21.445 INFO:teuthology.run_tasks:Running task install... 2024-04-22T15:36:21.453 DEBUG:teuthology.task.install:project ceph 2024-04-22T15:36:21.454 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5883d494d3973b30b3f03e4a3f0e1e06e3c131aa'}} 2024-04-22T15:36:21.454 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v18.2.1', 'flavor': 'default', 'sha1': '5883d494d3973b30b3f03e4a3f0e1e06e3c131aa'} 2024-04-22T15:36:21.454 INFO:teuthology.task.install:Using flavor: default 2024-04-22T15:36:21.459 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-22T15:36:21.460 INFO:teuthology.task.install:extra packages: [] 2024-04-22T15:36:21.460 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5883d494d3973b30b3f03e4a3f0e1e06e3c131aa', 'tag': 'v18.2.1', 'wait_for_package': False} 2024-04-22T15:36:21.460 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-22T15:36:21.460 INFO:teuthology.packaging:ref: None 2024-04-22T15:36:21.460 INFO:teuthology.packaging:tag: v18.2.1 2024-04-22T15:36:21.460 INFO:teuthology.packaging:branch: None 2024-04-22T15:36:21.461 INFO:teuthology.packaging:sha1: 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:36:21.867 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph-ci v18.2.1^{} -> None 2024-04-22T15:36:21.867 INFO:teuthology.packaging:Tag 'v18.2.1' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-04-22T15:36:25.187 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph v18.2.1^{} -> 7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-04-22T15:36:25.187 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-04-22T15:36:25.190 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5883d494d3973b30b3f03e4a3f0e1e06e3c131aa', 'tag': 'v18.2.1', 'wait_for_package': False} 2024-04-22T15:36:25.190 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-22T15:36:25.191 INFO:teuthology.packaging:ref: None 2024-04-22T15:36:25.191 INFO:teuthology.packaging:tag: v18.2.1 2024-04-22T15:36:25.191 INFO:teuthology.packaging:branch: None 2024-04-22T15:36:25.191 INFO:teuthology.packaging:sha1: 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:36:25.191 INFO:teuthology.packaging:Tag 'v18.2.1' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-04-22T15:36:25.191 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-04-22T15:36:25.335 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/ 2024-04-22T15:36:25.335 INFO:teuthology.task.install.rpm:Package version is 18.2.1-0 2024-04-22T15:36:25.366 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T15:36:25.366 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-22T15:36:25.366 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T15:36:25.396 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-22T15:36:25.397 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-22T15:36:25.397 INFO:teuthology.packaging:ref: None 2024-04-22T15:36:25.397 INFO:teuthology.packaging:tag: v18.2.1 2024-04-22T15:36:25.397 INFO:teuthology.packaging:branch: None 2024-04-22T15:36:25.397 INFO:teuthology.packaging:sha1: 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:36:25.397 DEBUG:teuthology.orchestra.run.smithi016:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T15:36:25.413 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/ 2024-04-22T15:36:25.413 INFO:teuthology.task.install.rpm:Package version is 18.2.1-0 2024-04-22T15:36:25.439 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T15:36:25.440 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-22T15:36:25.440 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T15:36:25.469 DEBUG:teuthology.orchestra.run.smithi016:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T15:36:25.471 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-22T15:36:25.471 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-22T15:36:25.471 INFO:teuthology.packaging:ref: None 2024-04-22T15:36:25.471 INFO:teuthology.packaging:tag: v18.2.1 2024-04-22T15:36:25.471 INFO:teuthology.packaging:branch: None 2024-04-22T15:36:25.471 INFO:teuthology.packaging:sha1: 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:36:25.471 DEBUG:teuthology.orchestra.run.smithi027:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T15:36:25.541 DEBUG:teuthology.orchestra.run.smithi027:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T15:36:25.546 DEBUG:teuthology.orchestra.run.smithi016:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T15:36:25.611 INFO:teuthology.orchestra.run.smithi016.stdout:check_obsoletes = 1 2024-04-22T15:36:25.612 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum clean all 2024-04-22T15:36:25.620 DEBUG:teuthology.orchestra.run.smithi027:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T15:36:25.686 INFO:teuthology.orchestra.run.smithi027.stdout:check_obsoletes = 1 2024-04-22T15:36:25.688 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum clean all 2024-04-22T15:36:25.851 INFO:teuthology.orchestra.run.smithi016.stdout:48 files removed 2024-04-22T15:36:25.879 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-radosgw 2024-04-22T15:36:25.926 INFO:teuthology.orchestra.run.smithi027.stdout:48 files removed 2024-04-22T15:36:25.959 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-radosgw 2024-04-22T15:36:26.351 INFO:teuthology.orchestra.run.smithi016.stdout:ceph packages for x86_64 384 kB/s | 76 kB 00:00 2024-04-22T15:36:26.427 INFO:teuthology.orchestra.run.smithi027.stdout:ceph packages for x86_64 390 kB/s | 76 kB 00:00 2024-04-22T15:36:26.585 INFO:teuthology.orchestra.run.smithi016.stdout:ceph noarch packages 50 kB/s | 9.4 kB 00:00 2024-04-22T15:36:26.611 INFO:teuthology.orchestra.run.smithi027.stdout:ceph noarch packages 68 kB/s | 9.4 kB 00:00 2024-04-22T15:36:26.794 INFO:teuthology.orchestra.run.smithi016.stdout:ceph source packages 12 kB/s | 2.2 kB 00:00 2024-04-22T15:36:26.811 INFO:teuthology.orchestra.run.smithi027.stdout:ceph source packages 13 kB/s | 2.2 kB 00:00 2024-04-22T15:36:27.369 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-22T15:36:30.420 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.0 MB 00:03 2024-04-22T15:36:30.495 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 9.4 MB/s | 19 MB 00:02 2024-04-22T15:36:35.340 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 5.5 MB/s | 6.2 MB 00:01 2024-04-22T15:36:36.872 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 16 kB 00:00 2024-04-22T15:36:38.852 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-22T15:36:39.670 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 2.3 MB/s | 19 MB 00:08 2024-04-22T15:36:44.074 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-22T15:36:46.277 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:36:46.278 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:36:46.278 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-04-22T15:36:46.278 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:36:46.279 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-04-22T15:36:46.279 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw x86_64 2:18.2.1-0.el9 ceph 7.7 M 2024-04-22T15:36:46.279 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrading: 2024-04-22T15:36:46.279 INFO:teuthology.orchestra.run.smithi016.stdout: librados2 x86_64 2:18.2.1-0.el9 ceph 3.3 M 2024-04-22T15:36:46.279 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1 x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-04-22T15:36:46.279 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-04-22T15:36:46.279 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T15:36:46.279 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base x86_64 2:18.2.1-0.el9 ceph 5.2 M 2024-04-22T15:36:46.279 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common x86_64 2:18.2.1-0.el9 ceph 18 M 2024-04-22T15:36:46.279 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux x86_64 2:18.2.1-0.el9 ceph 24 k 2024-04-22T15:36:46.279 INFO:teuthology.orchestra.run.smithi016.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-04-22T15:36:46.279 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T15:36:46.280 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T15:36:46.280 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T15:36:46.280 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2 x86_64 2:18.2.1-0.el9 ceph 658 k 2024-04-22T15:36:46.280 INFO:teuthology.orchestra.run.smithi016.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T15:36:46.280 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T15:36:46.280 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T15:36:46.280 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1 x86_64 2:18.2.1-0.el9 ceph 474 k 2024-04-22T15:36:46.280 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T15:36:46.280 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2 x86_64 2:18.2.1-0.el9 ceph 4.5 M 2024-04-22T15:36:46.280 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T15:36:46.280 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T15:36:46.280 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T15:36:46.280 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse x86_64 2:18.2.1-0.el9 ceph 45 k 2024-04-22T15:36:46.281 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common x86_64 2:18.2.1-0.el9 ceph 124 k 2024-04-22T15:36:46.281 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs x86_64 2:18.2.1-0.el9 ceph 161 k 2024-04-22T15:36:46.281 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T15:36:46.281 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados x86_64 2:18.2.1-0.el9 ceph 321 k 2024-04-22T15:36:46.281 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd x86_64 2:18.2.1-0.el9 ceph 297 k 2024-04-22T15:36:46.281 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw x86_64 2:18.2.1-0.el9 ceph 99 k 2024-04-22T15:36:46.281 INFO:teuthology.orchestra.run.smithi016.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T15:36:46.281 INFO:teuthology.orchestra.run.smithi016.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T15:36:46.281 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:36:46.281 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-22T15:36:46.281 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:36:46.281 INFO:teuthology.orchestra.run.smithi016.stdout:Install 28 Packages 2024-04-22T15:36:46.281 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrade 2 Packages 2024-04-22T15:36:46.282 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:36:46.282 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 53 M 2024-04-22T15:36:46.282 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-22T15:36:46.892 INFO:teuthology.orchestra.run.smithi016.stdout:(1/30): ceph-base-18.2.1-0.el9.x86_64.rpm 19 MB/s | 5.2 MB 00:00 2024-04-22T15:36:47.192 INFO:teuthology.orchestra.run.smithi016.stdout:(2/30): ceph-radosgw-18.2.1-0.el9.x86_64.rpm 13 MB/s | 7.7 MB 00:00 2024-04-22T15:36:47.342 INFO:teuthology.orchestra.run.smithi016.stdout:(3/30): ceph-selinux-18.2.1-0.el9.x86_64.rpm 53 kB/s | 24 kB 00:00 2024-04-22T15:36:47.375 INFO:teuthology.orchestra.run.smithi016.stdout:(4/30): libcephfs2-18.2.1-0.el9.x86_64.rpm 3.5 MB/s | 658 kB 00:00 2024-04-22T15:36:47.401 INFO:teuthology.orchestra.run.smithi016.stdout:(5/30): libradosstriper1-18.2.1-0.el9.x86_64.rp 7.9 MB/s | 474 kB 00:00 2024-04-22T15:36:47.543 INFO:teuthology.orchestra.run.smithi016.stdout:(6/30): librgw2-18.2.1-0.el9.x86_64.rpm 27 MB/s | 4.5 MB 00:00 2024-04-22T15:36:47.568 INFO:teuthology.orchestra.run.smithi016.stdout:(7/30): python3-ceph-argparse-18.2.1-0.el9.x86_ 269 kB/s | 45 kB 00:00 2024-04-22T15:36:47.593 INFO:teuthology.orchestra.run.smithi016.stdout:(8/30): python3-ceph-common-18.2.1-0.el9.x86_64 2.4 MB/s | 124 kB 00:00 2024-04-22T15:36:47.635 INFO:teuthology.orchestra.run.smithi016.stdout:(9/30): python3-cephfs-18.2.1-0.el9.x86_64.rpm 2.4 MB/s | 161 kB 00:00 2024-04-22T15:36:47.669 INFO:teuthology.orchestra.run.smithi016.stdout:(10/30): python3-rados-18.2.1-0.el9.x86_64.rpm 4.2 MB/s | 321 kB 00:00 2024-04-22T15:36:47.719 INFO:teuthology.orchestra.run.smithi016.stdout:(11/30): python3-rbd-18.2.1-0.el9.x86_64.rpm 3.5 MB/s | 297 kB 00:00 2024-04-22T15:36:47.744 INFO:teuthology.orchestra.run.smithi016.stdout:(12/30): python3-rgw-18.2.1-0.el9.x86_64.rpm 1.3 MB/s | 99 kB 00:00 2024-04-22T15:36:47.928 INFO:teuthology.orchestra.run.smithi016.stdout:(13/30): ceph-common-18.2.1-0.el9.x86_64.rpm 14 MB/s | 18 MB 00:01 2024-04-22T15:36:48.270 INFO:teuthology.orchestra.run.smithi016.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 132 kB/s | 45 kB 00:00 2024-04-22T15:36:48.353 INFO:teuthology.orchestra.run.smithi016.stdout:(15/30): boost-program-options-1.75.0-8.el9.x86 169 kB/s | 107 kB 00:00 2024-04-22T15:36:48.379 INFO:teuthology.orchestra.run.smithi016.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 253 kB/s | 160 kB 00:00 2024-04-22T15:36:48.479 INFO:teuthology.orchestra.run.smithi016.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 422 kB/s | 42 kB 00:00 2024-04-22T15:36:48.621 INFO:teuthology.orchestra.run.smithi016.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.1 MB/s | 292 kB 00:00 2024-04-22T15:36:48.705 INFO:teuthology.orchestra.run.smithi016.stdout:(19/30): librdkafka-1.6.1-102.el9.x86_64.rpm 1.5 MB/s | 662 kB 00:00 2024-04-22T15:36:48.847 INFO:teuthology.orchestra.run.smithi016.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 301 kB/s | 111 kB 00:00 2024-04-22T15:36:48.930 INFO:teuthology.orchestra.run.smithi016.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 312 kB/s | 26 kB 00:00 2024-04-22T15:36:49.022 INFO:teuthology.orchestra.run.smithi016.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 530 kB/s | 49 kB 00:00 2024-04-22T15:36:49.114 INFO:teuthology.orchestra.run.smithi016.stdout:(23/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 628 kB/s | 309 kB 00:00 2024-04-22T15:36:49.139 INFO:teuthology.orchestra.run.smithi016.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 577 kB/s | 67 kB 00:00 2024-04-22T15:36:49.281 INFO:teuthology.orchestra.run.smithi016.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-04-22T15:36:49.323 INFO:teuthology.orchestra.run.smithi016.stdout:(26/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.9 MB/s | 838 kB 00:00 2024-04-22T15:36:49.398 INFO:teuthology.orchestra.run.smithi016.stdout:(27/30): libarrow-9.0.0-10.el9.x86_64.rpm 6.6 MB/s | 4.4 MB 00:00 2024-04-22T15:36:49.524 INFO:teuthology.orchestra.run.smithi016.stdout:(28/30): librbd1-18.2.1-0.el9.x86_64.rpm 24 MB/s | 3.0 MB 00:00 2024-04-22T15:36:49.565 INFO:teuthology.orchestra.run.smithi016.stdout:(29/30): thrift-0.15.0-2.el9.x86_64.rpm 5.6 MB/s | 1.6 MB 00:00 2024-04-22T15:36:49.624 INFO:teuthology.orchestra.run.smithi016.stdout:(30/30): librados2-18.2.1-0.el9.x86_64.rpm 11 MB/s | 3.3 MB 00:00 2024-04-22T15:36:49.629 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:36:49.629 INFO:teuthology.orchestra.run.smithi016.stdout:Total 16 MB/s | 53 MB 00:03 2024-04-22T15:36:49.706 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-22T15:36:49.747 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-22T15:36:49.747 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-22T15:36:50.192 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-22T15:36:50.193 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-22T15:36:51.421 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 795 kB/s | 6.2 MB 00:07 2024-04-22T15:36:51.627 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-22T15:36:51.718 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-04-22T15:36:51.754 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-04-22T15:36:51.805 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-04-22T15:36:52.191 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-04-22T15:36:52.237 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librados2-2:18.2.1-0.el9.x86_64 5/32 2024-04-22T15:36:52.330 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:18.2.1-0.el9.x86_64 5/32 2024-04-22T15:36:52.412 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rados-2:18.2.1-0.el9.x86_64 6/32 2024-04-22T15:36:52.447 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-04-22T15:36:52.493 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-04-22T15:36:52.522 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-04-22T15:36:52.566 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 9/32 2024-04-22T15:36:52.642 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-04-22T15:36:52.666 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-04-22T15:36:52.732 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-04-22T15:36:52.842 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-04-22T15:36:52.854 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 71 kB/s | 16 kB 00:00 2024-04-22T15:36:52.873 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-04-22T15:36:52.913 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-04-22T15:36:52.947 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-04-22T15:36:52.974 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-04-22T15:36:53.010 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-04-22T15:36:53.042 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-04-22T15:36:53.356 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-04-22T15:36:53.406 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-04-22T15:36:53.496 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-04-22T15:36:53.527 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rbd-2:18.2.1-0.el9.x86_64 21/32 2024-04-22T15:36:53.563 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-04-22T15:36:53.984 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-04-22T15:36:54.016 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-04-22T15:36:54.088 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-04-22T15:36:54.166 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rgw-2:18.2.1-0.el9.x86_64 25/32 2024-04-22T15:36:54.189 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-04-22T15:36:54.193 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-common-2:18.2.1-0.el9.x86_64 26/32 2024-04-22T15:36:56.306 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-04-22T15:36:56.342 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-04-22T15:36:56.908 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-04-22T15:36:56.953 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-04-22T15:36:57.056 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-04-22T15:36:57.056 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T15:36:57.056 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T15:36:57.056 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:36:57.091 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-04-22T15:36:59.440 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-22T15:37:01.702 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:37:01.704 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:37:01.704 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-04-22T15:37:01.704 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:37:01.704 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-22T15:37:01.704 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw x86_64 2:18.2.1-0.el9 ceph 7.7 M 2024-04-22T15:37:01.704 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrading: 2024-04-22T15:37:01.704 INFO:teuthology.orchestra.run.smithi027.stdout: librados2 x86_64 2:18.2.1-0.el9 ceph 3.3 M 2024-04-22T15:37:01.704 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1 x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-04-22T15:37:01.704 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-04-22T15:37:01.704 INFO:teuthology.orchestra.run.smithi027.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T15:37:01.704 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base x86_64 2:18.2.1-0.el9 ceph 5.2 M 2024-04-22T15:37:01.705 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common x86_64 2:18.2.1-0.el9 ceph 18 M 2024-04-22T15:37:01.705 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux x86_64 2:18.2.1-0.el9 ceph 24 k 2024-04-22T15:37:01.705 INFO:teuthology.orchestra.run.smithi027.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-04-22T15:37:01.705 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T15:37:01.705 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T15:37:01.705 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T15:37:01.705 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2 x86_64 2:18.2.1-0.el9 ceph 658 k 2024-04-22T15:37:01.705 INFO:teuthology.orchestra.run.smithi027.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T15:37:01.705 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T15:37:01.705 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T15:37:01.705 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1 x86_64 2:18.2.1-0.el9 ceph 474 k 2024-04-22T15:37:01.705 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T15:37:01.706 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2 x86_64 2:18.2.1-0.el9 ceph 4.5 M 2024-04-22T15:37:01.706 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T15:37:01.706 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T15:37:01.706 INFO:teuthology.orchestra.run.smithi027.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T15:37:01.706 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse x86_64 2:18.2.1-0.el9 ceph 45 k 2024-04-22T15:37:01.706 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common x86_64 2:18.2.1-0.el9 ceph 124 k 2024-04-22T15:37:01.706 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs x86_64 2:18.2.1-0.el9 ceph 161 k 2024-04-22T15:37:01.706 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T15:37:01.706 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados x86_64 2:18.2.1-0.el9 ceph 321 k 2024-04-22T15:37:01.706 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd x86_64 2:18.2.1-0.el9 ceph 297 k 2024-04-22T15:37:01.706 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw x86_64 2:18.2.1-0.el9 ceph 99 k 2024-04-22T15:37:01.706 INFO:teuthology.orchestra.run.smithi027.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T15:37:01.707 INFO:teuthology.orchestra.run.smithi027.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T15:37:01.707 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:01.707 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-22T15:37:01.707 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:37:01.707 INFO:teuthology.orchestra.run.smithi027.stdout:Install 28 Packages 2024-04-22T15:37:01.707 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrade 2 Packages 2024-04-22T15:37:01.707 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:01.707 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 53 M 2024-04-22T15:37:01.708 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-22T15:37:02.433 INFO:teuthology.orchestra.run.smithi027.stdout:(1/30): ceph-base-18.2.1-0.el9.x86_64.rpm 23 MB/s | 5.2 MB 00:00 2024-04-22T15:37:02.458 INFO:teuthology.orchestra.run.smithi027.stdout:(2/30): ceph-selinux-18.2.1-0.el9.x86_64.rpm 968 kB/s | 24 kB 00:00 2024-04-22T15:37:02.491 INFO:teuthology.orchestra.run.smithi027.stdout:(3/30): libcephfs2-18.2.1-0.el9.x86_64.rpm 19 MB/s | 658 kB 00:00 2024-04-22T15:37:02.525 INFO:teuthology.orchestra.run.smithi027.stdout:(4/30): libradosstriper1-18.2.1-0.el9.x86_64.rp 14 MB/s | 474 kB 00:00 2024-04-22T15:37:02.625 INFO:teuthology.orchestra.run.smithi027.stdout:(5/30): ceph-radosgw-18.2.1-0.el9.x86_64.rpm 19 MB/s | 7.7 MB 00:00 2024-04-22T15:37:02.650 INFO:teuthology.orchestra.run.smithi027.stdout:(6/30): python3-ceph-argparse-18.2.1-0.el9.x86_ 1.8 MB/s | 45 kB 00:00 2024-04-22T15:37:02.676 INFO:teuthology.orchestra.run.smithi027.stdout:(7/30): python3-ceph-common-18.2.1-0.el9.x86_64 4.8 MB/s | 124 kB 00:00 2024-04-22T15:37:02.701 INFO:teuthology.orchestra.run.smithi027.stdout:(8/30): python3-cephfs-18.2.1-0.el9.x86_64.rpm 6.3 MB/s | 161 kB 00:00 2024-04-22T15:37:02.769 INFO:teuthology.orchestra.run.smithi027.stdout:(9/30): python3-rados-18.2.1-0.el9.x86_64.rpm 4.7 MB/s | 321 kB 00:00 2024-04-22T15:37:02.793 INFO:teuthology.orchestra.run.smithi027.stdout:(10/30): python3-rbd-18.2.1-0.el9.x86_64.rpm 12 MB/s | 297 kB 00:00 2024-04-22T15:37:02.810 INFO:teuthology.orchestra.run.smithi027.stdout:(11/30): python3-rgw-18.2.1-0.el9.x86_64.rpm 5.8 MB/s | 99 kB 00:00 2024-04-22T15:37:02.886 INFO:teuthology.orchestra.run.smithi027.stdout:(12/30): librgw2-18.2.1-0.el9.x86_64.rpm 12 MB/s | 4.5 MB 00:00 2024-04-22T15:37:03.086 INFO:teuthology.orchestra.run.smithi027.stdout:(13/30): ceph-common-18.2.1-0.el9.x86_64.rpm 21 MB/s | 18 MB 00:00 2024-04-22T15:37:03.320 INFO:teuthology.orchestra.run.smithi027.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 210 kB/s | 107 kB 00:00 2024-04-22T15:37:03.428 INFO:teuthology.orchestra.run.smithi027.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 132 kB/s | 45 kB 00:00 2024-04-22T15:37:03.520 INFO:teuthology.orchestra.run.smithi027.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 253 kB/s | 160 kB 00:00 2024-04-22T15:37:03.637 INFO:teuthology.orchestra.run.smithi027.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 363 kB/s | 42 kB 00:00 2024-04-22T15:37:03.871 INFO:teuthology.orchestra.run.smithi027.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 661 kB/s | 292 kB 00:00 2024-04-22T15:37:03.896 INFO:teuthology.orchestra.run.smithi027.stdout:(19/30): librdkafka-1.6.1-102.el9.x86_64.rpm 1.1 MB/s | 662 kB 00:00 2024-04-22T15:37:04.046 INFO:teuthology.orchestra.run.smithi027.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 271 kB/s | 111 kB 00:00 2024-04-22T15:37:04.222 INFO:teuthology.orchestra.run.smithi027.stdout:(21/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 880 kB/s | 309 kB 00:00 2024-04-22T15:37:04.247 INFO:teuthology.orchestra.run.smithi027.stdout:(22/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 129 kB/s | 26 kB 00:00 2024-04-22T15:37:04.406 INFO:teuthology.orchestra.run.smithi027.stdout:(23/30): libarrow-9.0.0-10.el9.x86_64.rpm 8.7 MB/s | 4.4 MB 00:00 2024-04-22T15:37:04.431 INFO:teuthology.orchestra.run.smithi027.stdout:(24/30): liboath-2.6.7-2.el9.x86_64.rpm 232 kB/s | 49 kB 00:00 2024-04-22T15:37:04.456 INFO:teuthology.orchestra.run.smithi027.stdout:(25/30): libunwind-1.6.2-1.el9.x86_64.rpm 326 kB/s | 67 kB 00:00 2024-04-22T15:37:04.490 INFO:teuthology.orchestra.run.smithi027.stdout:(26/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-04-22T15:37:04.532 INFO:teuthology.orchestra.run.smithi027.stdout:(27/30): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-22T15:37:04.574 INFO:teuthology.orchestra.run.smithi027.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-04-22T15:37:04.690 INFO:teuthology.orchestra.run.smithi027.stdout:(29/30): librados2-18.2.1-0.el9.x86_64.rpm 16 MB/s | 3.3 MB 00:00 2024-04-22T15:37:04.757 INFO:teuthology.orchestra.run.smithi027.stdout:(30/30): librbd1-18.2.1-0.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-04-22T15:37:04.761 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:37:04.761 INFO:teuthology.orchestra.run.smithi027.stdout:Total 17 MB/s | 53 MB 00:03 2024-04-22T15:37:04.832 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-22T15:37:04.873 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-22T15:37:04.873 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-22T15:37:05.317 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-22T15:37:05.318 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-22T15:37:06.713 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-22T15:37:06.819 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-04-22T15:37:06.855 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-04-22T15:37:06.905 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-04-22T15:37:07.154 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-04-22T15:37:07.154 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /sys 2024-04-22T15:37:07.155 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /proc 2024-04-22T15:37:07.155 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /mnt 2024-04-22T15:37:07.155 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /var/tmp 2024-04-22T15:37:07.155 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /home 2024-04-22T15:37:07.155 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /root 2024-04-22T15:37:07.155 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /tmp 2024-04-22T15:37:07.155 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:07.300 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-04-22T15:37:07.354 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librados2-2:18.2.1-0.el9.x86_64 5/32 2024-04-22T15:37:07.446 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:18.2.1-0.el9.x86_64 5/32 2024-04-22T15:37:07.537 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rados-2:18.2.1-0.el9.x86_64 6/32 2024-04-22T15:37:07.572 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-04-22T15:37:07.634 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-04-22T15:37:07.663 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-04-22T15:37:07.708 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 9/32 2024-04-22T15:37:07.775 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-04-22T15:37:07.799 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-04-22T15:37:07.848 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-04-22T15:37:07.953 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-04-22T15:37:07.967 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-04-22T15:37:07.985 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-04-22T15:37:07.985 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:37:07.985 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T15:37:07.986 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T15:37:07.986 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T15:37:07.986 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:07.987 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-22T15:37:07.998 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-04-22T15:37:08.060 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-04-22T15:37:08.073 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-22T15:37:08.073 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T15:37:08.090 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-04-22T15:37:08.117 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-04-22T15:37:08.153 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-04-22T15:37:08.186 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-04-22T15:37:08.507 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-04-22T15:37:08.540 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-04-22T15:37:08.614 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-04-22T15:37:08.645 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rbd-2:18.2.1-0.el9.x86_64 21/32 2024-04-22T15:37:08.681 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-04-22T15:37:09.102 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-04-22T15:37:09.134 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-04-22T15:37:09.206 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-04-22T15:37:09.265 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T15:37:09.266 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-base-2:18.2.1-0.el9.x86_64 1/32 2024-04-22T15:37:09.266 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-common-2:18.2.1-0.el9.x86_64 2/32 2024-04-22T15:37:09.266 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-radosgw-2:18.2.1-0.el9.x86_64 3/32 2024-04-22T15:37:09.266 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-selinux-2:18.2.1-0.el9.x86_64 4/32 2024-04-22T15:37:09.266 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs2-2:18.2.1-0.el9.x86_64 5/32 2024-04-22T15:37:09.266 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libradosstriper1-2:18.2.1-0.el9.x86_64 6/32 2024-04-22T15:37:09.266 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librgw2-2:18.2.1-0.el9.x86_64 7/32 2024-04-22T15:37:09.266 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 8/32 2024-04-22T15:37:09.266 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-common-2:18.2.1-0.el9.x86_64 9/32 2024-04-22T15:37:09.266 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-04-22T15:37:09.267 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rados-2:18.2.1-0.el9.x86_64 11/32 2024-04-22T15:37:09.267 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rbd-2:18.2.1-0.el9.x86_64 12/32 2024-04-22T15:37:09.267 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rgw-2:18.2.1-0.el9.x86_64 13/32 2024-04-22T15:37:09.267 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-04-22T15:37:09.267 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-04-22T15:37:09.267 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-04-22T15:37:09.267 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-04-22T15:37:09.267 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-04-22T15:37:09.267 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-04-22T15:37:09.267 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-04-22T15:37:09.267 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-04-22T15:37:09.267 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-04-22T15:37:09.267 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-04-22T15:37:09.268 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-04-22T15:37:09.268 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-04-22T15:37:09.268 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-04-22T15:37:09.268 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-04-22T15:37:09.268 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-04-22T15:37:09.268 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:18.2.1-0.el9.x86_64 29/32 2024-04-22T15:37:09.268 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-04-22T15:37:09.268 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:18.2.1-0.el9.x86_64 31/32 2024-04-22T15:37:09.277 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rgw-2:18.2.1-0.el9.x86_64 25/32 2024-04-22T15:37:09.312 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-common-2:18.2.1-0.el9.x86_64 26/32 2024-04-22T15:37:09.983 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T15:37:09.984 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:09.984 INFO:teuthology.orchestra.run.smithi016.stdout:Upgraded: 2024-04-22T15:37:09.984 INFO:teuthology.orchestra.run.smithi016.stdout: librados2-2:18.2.1-0.el9.x86_64 librbd1-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:09.984 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-04-22T15:37:09.984 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T15:37:09.984 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:09.984 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:09.984 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:09.984 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:09.984 INFO:teuthology.orchestra.run.smithi016.stdout: fmt-8.1.1-5.el9.x86_64 2024-04-22T15:37:09.984 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T15:37:09.984 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T15:37:09.985 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T15:37:09.985 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:09.985 INFO:teuthology.orchestra.run.smithi016.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T15:37:09.985 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T15:37:09.985 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T15:37:09.985 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:09.985 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T15:37:09.985 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:09.985 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T15:37:09.986 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T15:37:09.986 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T15:37:09.986 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:09.986 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:09.986 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:09.986 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T15:37:09.986 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:09.986 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:09.986 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:09.986 INFO:teuthology.orchestra.run.smithi016.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T15:37:09.986 INFO:teuthology.orchestra.run.smithi016.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T15:37:09.986 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:09.987 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:37:10.243 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-test 2024-04-22T15:37:10.779 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:26 ago on Mon 22 Apr 2024 03:36:44 PM UTC. 2024-04-22T15:37:10.902 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:37:10.902 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:37:10.902 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-04-22T15:37:10.903 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:37:10.903 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-04-22T15:37:10.903 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test x86_64 2:18.2.1-0.el9 ceph 40 M 2024-04-22T15:37:10.903 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-04-22T15:37:10.903 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite x86_64 2:18.2.1-0.el9 ceph 165 k 2024-04-22T15:37:10.903 INFO:teuthology.orchestra.run.smithi016.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T15:37:10.903 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T15:37:10.903 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:10.903 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-22T15:37:10.903 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:37:10.903 INFO:teuthology.orchestra.run.smithi016.stdout:Install 4 Packages 2024-04-22T15:37:10.903 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:10.904 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 40 M 2024-04-22T15:37:10.904 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 166 M 2024-04-22T15:37:10.904 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-22T15:37:11.479 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-04-22T15:37:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-04-22T15:37:11.628 INFO:teuthology.orchestra.run.smithi016.stdout:(1/4): libcephsqlite-18.2.1-0.el9.x86_64.rpm 363 kB/s | 165 kB 00:00 2024-04-22T15:37:12.082 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-04-22T15:37:12.121 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-04-22T15:37:12.228 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-04-22T15:37:12.228 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T15:37:12.228 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T15:37:12.228 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:12.264 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-04-22T15:37:12.503 INFO:teuthology.orchestra.run.smithi016.stdout:(2/4): ceph-test-18.2.1-0.el9.x86_64.rpm 30 MB/s | 40 MB 00:01 2024-04-22T15:37:12.551 INFO:teuthology.orchestra.run.smithi016.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 69 kB/s | 64 kB 00:00 2024-04-22T15:37:12.602 INFO:teuthology.orchestra.run.smithi016.stdout:(4/4): socat-1.7.4.1-5.el9.x86_64.rpm 213 kB/s | 305 kB 00:01 2024-04-22T15:37:12.604 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:37:12.604 INFO:teuthology.orchestra.run.smithi016.stdout:Total 24 MB/s | 40 MB 00:01 2024-04-22T15:37:12.631 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-22T15:37:12.655 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-22T15:37:12.655 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-22T15:37:12.986 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-22T15:37:12.987 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-22T15:37:13.468 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-22T15:37:13.537 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-22T15:37:13.586 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-22T15:37:13.612 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephsqlite-2:18.2.1-0.el9.x86_64 3/4 2024-04-22T15:37:17.567 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephsqlite-2:18.2.1-0.el9.x86_64 3/4 2024-04-22T15:37:17.761 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-test-2:18.2.1-0.el9.x86_64 4/4 2024-04-22T15:37:19.773 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-test-2:18.2.1-0.el9.x86_64 4/4 2024-04-22T15:37:19.773 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-test-2:18.2.1-0.el9.x86_64 1/4 2024-04-22T15:37:19.773 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephsqlite-2:18.2.1-0.el9.x86_64 2/4 2024-04-22T15:37:19.774 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-22T15:37:20.057 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-22T15:37:20.057 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:20.057 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-04-22T15:37:20.057 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test-2:18.2.1-0.el9.x86_64 libcephsqlite-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:20.058 INFO:teuthology.orchestra.run.smithi016.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T15:37:20.058 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:20.058 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:37:20.327 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph 2024-04-22T15:37:20.853 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:36 ago on Mon 22 Apr 2024 03:36:44 PM UTC. 2024-04-22T15:37:20.980 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:37:20.982 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:37:20.982 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-04-22T15:37:20.983 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:37:20.983 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-04-22T15:37:20.983 INFO:teuthology.orchestra.run.smithi016.stdout: ceph x86_64 2:18.2.1-0.el9 ceph 6.4 k 2024-04-22T15:37:20.983 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-04-22T15:37:20.983 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds x86_64 2:18.2.1-0.el9 ceph 2.1 M 2024-04-22T15:37:20.983 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr x86_64 2:18.2.1-0.el9 ceph 1.4 M 2024-04-22T15:37:20.983 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core noarch 2:18.2.1-0.el9 ceph-noarch 242 k 2024-04-22T15:37:20.983 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon x86_64 2:18.2.1-0.el9 ceph 4.4 M 2024-04-22T15:37:20.983 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd x86_64 2:18.2.1-0.el9 ceph 18 M 2024-04-22T15:37:20.983 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T15:37:20.983 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T15:37:20.983 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T15:37:20.983 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T15:37:20.984 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T15:37:20.984 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T15:37:20.984 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T15:37:20.984 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-22T15:37:20.984 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T15:37:20.984 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T15:37:20.984 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T15:37:20.984 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T15:37:20.984 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T15:37:20.984 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T15:37:20.984 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T15:37:20.985 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T15:37:20.985 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T15:37:20.985 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T15:37:20.985 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T15:37:20.985 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T15:37:20.985 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T15:37:20.985 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T15:37:20.985 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T15:37:20.985 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T15:37:20.985 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T15:37:20.985 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T15:37:20.985 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T15:37:20.986 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:20.986 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-22T15:37:20.986 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:37:20.986 INFO:teuthology.orchestra.run.smithi016.stdout:Install 33 Packages 2024-04-22T15:37:20.986 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:20.986 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 30 M 2024-04-22T15:37:20.987 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 110 M 2024-04-22T15:37:20.987 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-22T15:37:21.678 INFO:teuthology.orchestra.run.smithi016.stdout:(1/33): ceph-18.2.1-0.el9.x86_64.rpm 50 kB/s | 6.4 kB 00:00 2024-04-22T15:37:21.737 INFO:teuthology.orchestra.run.smithi016.stdout:(2/33): ceph-mds-18.2.1-0.el9.x86_64.rpm 11 MB/s | 2.1 MB 00:00 2024-04-22T15:37:22.012 INFO:teuthology.orchestra.run.smithi016.stdout:(3/33): ceph-mgr-18.2.1-0.el9.x86_64.rpm 3.1 MB/s | 1.4 MB 00:00 2024-04-22T15:37:22.104 INFO:teuthology.orchestra.run.smithi016.stdout:(4/33): ceph-mgr-modules-core-18.2.1-0.el9.noar 2.6 MB/s | 242 kB 00:00 2024-04-22T15:37:22.279 INFO:teuthology.orchestra.run.smithi016.stdout:(5/33): ceph-osd-18.2.1-0.el9.x86_64.rpm 32 MB/s | 18 MB 00:00 2024-04-22T15:37:22.284 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-04-22T15:37:22.284 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /sys 2024-04-22T15:37:22.284 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /proc 2024-04-22T15:37:22.284 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /mnt 2024-04-22T15:37:22.285 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /var/tmp 2024-04-22T15:37:22.285 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /home 2024-04-22T15:37:22.285 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /root 2024-04-22T15:37:22.285 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /tmp 2024-04-22T15:37:22.285 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:22.355 INFO:teuthology.orchestra.run.smithi016.stdout:(6/33): ceph-mon-18.2.1-0.el9.x86_64.rpm 6.5 MB/s | 4.4 MB 00:00 2024-04-22T15:37:22.447 INFO:teuthology.orchestra.run.smithi016.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 740 kB/s | 253 kB 00:00 2024-04-22T15:37:22.497 INFO:teuthology.orchestra.run.smithi016.stdout:(8/33): python3-pycparser-2.20-6.el9.noarch.rpm 2.6 MB/s | 135 kB 00:00 2024-04-22T15:37:22.522 INFO:teuthology.orchestra.run.smithi016.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 637 kB/s | 106 kB 00:00 2024-04-22T15:37:22.548 INFO:teuthology.orchestra.run.smithi016.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-04-22T15:37:22.581 INFO:teuthology.orchestra.run.smithi016.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 4.1 MB/s | 1.2 MB 00:00 2024-04-22T15:37:22.615 INFO:teuthology.orchestra.run.smithi016.stdout:(12/33): python3-urllib3-1.26.5-5.el9.noarch.rp 2.3 MB/s | 215 kB 00:00 2024-04-22T15:37:22.657 INFO:teuthology.orchestra.run.smithi016.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 461 kB/s | 35 kB 00:00 2024-04-22T15:37:22.682 INFO:teuthology.orchestra.run.smithi016.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 764 kB/s | 51 kB 00:00 2024-04-22T15:37:22.707 INFO:teuthology.orchestra.run.smithi016.stdout:(15/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-04-22T15:37:22.733 INFO:teuthology.orchestra.run.smithi016.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 552 kB/s | 42 kB 00:00 2024-04-22T15:37:22.849 INFO:teuthology.orchestra.run.smithi016.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 260 kB/s | 43 kB 00:00 2024-04-22T15:37:22.908 INFO:teuthology.orchestra.run.smithi016.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2024-04-22T15:37:22.933 INFO:teuthology.orchestra.run.smithi016.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 760 kB/s | 172 kB 00:00 2024-04-22T15:37:22.967 INFO:teuthology.orchestra.run.smithi016.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 301 kB/s | 18 kB 00:00 2024-04-22T15:37:22.992 INFO:teuthology.orchestra.run.smithi016.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 396 kB/s | 23 kB 00:00 2024-04-22T15:37:23.017 INFO:teuthology.orchestra.run.smithi016.stdout:(22/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-04-22T15:37:23.043 INFO:teuthology.orchestra.run.smithi016.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-04-22T15:37:23.046 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-04-22T15:37:23.068 INFO:teuthology.orchestra.run.smithi016.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-04-22T15:37:23.071 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-04-22T15:37:23.071 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:37:23.071 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T15:37:23.071 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T15:37:23.072 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T15:37:23.072 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:23.073 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-22T15:37:23.093 INFO:teuthology.orchestra.run.smithi016.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 613 kB/s | 46 kB 00:00 2024-04-22T15:37:23.127 INFO:teuthology.orchestra.run.smithi016.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 942 kB/s | 79 kB 00:00 2024-04-22T15:37:23.152 INFO:teuthology.orchestra.run.smithi016.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2024-04-22T15:37:23.156 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-22T15:37:23.156 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T15:37:23.177 INFO:teuthology.orchestra.run.smithi016.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-04-22T15:37:23.203 INFO:teuthology.orchestra.run.smithi016.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-04-22T15:37:23.228 INFO:teuthology.orchestra.run.smithi016.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 474 kB/s | 36 kB 00:00 2024-04-22T15:37:23.253 INFO:teuthology.orchestra.run.smithi016.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-22T15:37:23.278 INFO:teuthology.orchestra.run.smithi016.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 400 kB/s | 20 kB 00:00 2024-04-22T15:37:23.328 INFO:teuthology.orchestra.run.smithi016.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-04-22T15:37:23.330 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:37:23.330 INFO:teuthology.orchestra.run.smithi016.stdout:Total 13 MB/s | 30 MB 00:02 2024-04-22T15:37:23.451 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-22T15:37:23.493 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-22T15:37:23.493 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-22T15:37:23.795 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-22T15:37:23.797 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-22T15:37:24.372 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T15:37:24.373 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-base-2:18.2.1-0.el9.x86_64 1/32 2024-04-22T15:37:24.373 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-common-2:18.2.1-0.el9.x86_64 2/32 2024-04-22T15:37:24.373 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-radosgw-2:18.2.1-0.el9.x86_64 3/32 2024-04-22T15:37:24.373 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-selinux-2:18.2.1-0.el9.x86_64 4/32 2024-04-22T15:37:24.373 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs2-2:18.2.1-0.el9.x86_64 5/32 2024-04-22T15:37:24.373 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libradosstriper1-2:18.2.1-0.el9.x86_64 6/32 2024-04-22T15:37:24.373 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librgw2-2:18.2.1-0.el9.x86_64 7/32 2024-04-22T15:37:24.373 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 8/32 2024-04-22T15:37:24.373 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-common-2:18.2.1-0.el9.x86_64 9/32 2024-04-22T15:37:24.373 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-04-22T15:37:24.373 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rados-2:18.2.1-0.el9.x86_64 11/32 2024-04-22T15:37:24.373 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rbd-2:18.2.1-0.el9.x86_64 12/32 2024-04-22T15:37:24.373 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rgw-2:18.2.1-0.el9.x86_64 13/32 2024-04-22T15:37:24.374 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-04-22T15:37:24.374 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-04-22T15:37:24.374 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-04-22T15:37:24.374 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-04-22T15:37:24.374 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-04-22T15:37:24.374 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-04-22T15:37:24.374 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-04-22T15:37:24.374 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-04-22T15:37:24.374 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-04-22T15:37:24.374 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-04-22T15:37:24.374 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-04-22T15:37:24.374 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-04-22T15:37:24.374 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-04-22T15:37:24.374 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-04-22T15:37:24.375 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-04-22T15:37:24.375 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:18.2.1-0.el9.x86_64 29/32 2024-04-22T15:37:24.375 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-04-22T15:37:24.375 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:18.2.1-0.el9.x86_64 31/32 2024-04-22T15:37:24.961 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-22T15:37:25.023 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-22T15:37:25.056 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-22T15:37:25.089 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-22T15:37:25.130 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T15:37:25.130 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:25.130 INFO:teuthology.orchestra.run.smithi027.stdout:Upgraded: 2024-04-22T15:37:25.130 INFO:teuthology.orchestra.run.smithi027.stdout: librados2-2:18.2.1-0.el9.x86_64 librbd1-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:25.130 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-22T15:37:25.130 INFO:teuthology.orchestra.run.smithi027.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T15:37:25.130 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:25.130 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:25.130 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:25.130 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:25.131 INFO:teuthology.orchestra.run.smithi027.stdout: fmt-8.1.1-5.el9.x86_64 2024-04-22T15:37:25.131 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T15:37:25.131 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T15:37:25.131 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T15:37:25.131 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:25.131 INFO:teuthology.orchestra.run.smithi027.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T15:37:25.131 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T15:37:25.131 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T15:37:25.131 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:25.131 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T15:37:25.131 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:25.131 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T15:37:25.131 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T15:37:25.131 INFO:teuthology.orchestra.run.smithi027.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T15:37:25.132 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:25.132 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:25.132 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:25.132 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T15:37:25.132 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:25.132 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:25.132 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:25.132 INFO:teuthology.orchestra.run.smithi027.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T15:37:25.132 INFO:teuthology.orchestra.run.smithi027.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T15:37:25.132 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:25.132 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:37:25.185 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-22T15:37:25.261 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-22T15:37:25.309 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-22T15:37:25.343 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-22T15:37:25.373 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-04-22T15:37:25.406 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-test 2024-04-22T15:37:25.407 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-04-22T15:37:25.441 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-04-22T15:37:25.481 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-04-22T15:37:25.517 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-04-22T15:37:25.551 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-04-22T15:37:25.589 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-04-22T15:37:25.666 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-04-22T15:37:25.817 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-04-22T15:37:25.900 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-04-22T15:37:25.943 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:26 ago on Mon 22 Apr 2024 03:36:59 PM UTC. 2024-04-22T15:37:25.952 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-04-22T15:37:25.989 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-04-22T15:37:26.048 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-04-22T15:37:26.066 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:37:26.067 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:37:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-04-22T15:37:26.067 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-22T15:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test x86_64 2:18.2.1-0.el9 ceph 40 M 2024-04-22T15:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-04-22T15:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite x86_64 2:18.2.1-0.el9 ceph 165 k 2024-04-22T15:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T15:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T15:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-22T15:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout:Install 4 Packages 2024-04-22T15:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:26.069 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 40 M 2024-04-22T15:37:26.069 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 166 M 2024-04-22T15:37:26.069 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-22T15:37:26.111 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-04-22T15:37:26.274 INFO:teuthology.orchestra.run.smithi027.stdout:(1/4): libcephsqlite-18.2.1-0.el9.x86_64.rpm 1.8 MB/s | 165 kB 00:00 2024-04-22T15:37:26.308 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-04-22T15:37:26.376 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-04-22T15:37:26.457 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-04-22T15:37:26.556 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-04-22T15:37:26.601 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-04-22T15:37:26.722 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-04-22T15:37:26.870 INFO:teuthology.orchestra.run.smithi027.stdout:(2/4): ceph-test-18.2.1-0.el9.x86_64.rpm 58 MB/s | 40 MB 00:00 2024-04-22T15:37:26.893 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 28/33 2024-04-22T15:37:26.919 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-04-22T15:37:26.945 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-04-22T15:37:26.946 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:37:26.946 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T15:37:26.946 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T15:37:26.946 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T15:37:26.946 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:27.357 INFO:teuthology.orchestra.run.smithi027.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 59 kB/s | 64 kB 00:01 2024-04-22T15:37:27.584 INFO:teuthology.orchestra.run.smithi027.stdout:(4/4): socat-1.7.4.1-5.el9.x86_64.rpm 218 kB/s | 305 kB 00:01 2024-04-22T15:37:27.587 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:37:27.587 INFO:teuthology.orchestra.run.smithi027.stdout:Total 26 MB/s | 40 MB 00:01 2024-04-22T15:37:27.614 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-22T15:37:27.636 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-22T15:37:27.636 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-22T15:37:27.968 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-22T15:37:27.969 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-22T15:37:28.473 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-22T15:37:28.545 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-04-22T15:37:28.565 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-22T15:37:28.574 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-04-22T15:37:28.574 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:37:28.574 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T15:37:28.574 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T15:37:28.574 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T15:37:28.574 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:28.614 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-22T15:37:28.639 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephsqlite-2:18.2.1-0.el9.x86_64 3/4 2024-04-22T15:37:29.038 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-04-22T15:37:29.063 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-04-22T15:37:29.063 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:37:29.063 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T15:37:29.063 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T15:37:29.063 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T15:37:29.063 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:29.280 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-04-22T15:37:29.304 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-04-22T15:37:29.304 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:37:29.305 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T15:37:29.305 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T15:37:29.305 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T15:37:29.305 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:29.370 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-2:18.2.1-0.el9.x86_64 33/33 2024-04-22T15:37:31.138 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-2:18.2.1-0.el9.x86_64 33/33 2024-04-22T15:37:31.138 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-2:18.2.1-0.el9.x86_64 1/33 2024-04-22T15:37:31.138 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mds-2:18.2.1-0.el9.x86_64 2/33 2024-04-22T15:37:31.138 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-2:18.2.1-0.el9.x86_64 3/33 2024-04-22T15:37:31.138 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mon-2:18.2.1-0.el9.x86_64 4/33 2024-04-22T15:37:31.138 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-osd-2:18.2.1-0.el9.x86_64 5/33 2024-04-22T15:37:31.138 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 6/33 2024-04-22T15:37:31.138 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-04-22T15:37:31.139 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-04-22T15:37:31.139 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-04-22T15:37:31.139 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-04-22T15:37:31.139 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-04-22T15:37:31.139 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-04-22T15:37:31.139 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-04-22T15:37:31.139 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-04-22T15:37:31.139 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-04-22T15:37:31.139 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-22T15:37:31.139 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-22T15:37:31.139 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-22T15:37:31.139 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-22T15:37:31.139 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-22T15:37:31.139 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-22T15:37:31.140 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-22T15:37:31.140 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-22T15:37:31.140 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-22T15:37:31.140 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-22T15:37:31.140 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-22T15:37:31.140 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-22T15:37:31.140 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-22T15:37:31.141 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-22T15:37:31.141 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-22T15:37:31.141 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-22T15:37:31.141 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-22T15:37:32.024 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-22T15:37:32.024 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:32.025 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-04-22T15:37:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 2024-04-22T15:37:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T15:37:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T15:37:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T15:37:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T15:37:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T15:37:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T15:37:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T15:37:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T15:37:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T15:37:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T15:37:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T15:37:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T15:37:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T15:37:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T15:37:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T15:37:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T15:37:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T15:37:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T15:37:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T15:37:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T15:37:32.027 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T15:37:32.027 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T15:37:32.027 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T15:37:32.027 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T15:37:32.027 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T15:37:32.027 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T15:37:32.027 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T15:37:32.027 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:32.027 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:37:32.402 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-base 2024-04-22T15:37:32.546 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephsqlite-2:18.2.1-0.el9.x86_64 3/4 2024-04-22T15:37:32.621 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-test-2:18.2.1-0.el9.x86_64 4/4 2024-04-22T15:37:32.944 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:48 ago on Mon 22 Apr 2024 03:36:44 PM UTC. 2024-04-22T15:37:33.025 INFO:teuthology.orchestra.run.smithi016.stdout:Package ceph-base-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T15:37:33.068 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:37:33.069 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-04-22T15:37:33.069 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:37:33.144 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install cephadm 2024-04-22T15:37:33.680 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:49 ago on Mon 22 Apr 2024 03:36:44 PM UTC. 2024-04-22T15:37:33.805 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:37:33.805 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:37:33.806 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-04-22T15:37:33.806 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:37:33.806 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-04-22T15:37:33.806 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm noarch 2:18.2.1-0.el9 ceph-noarch 221 k 2024-04-22T15:37:33.806 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:33.806 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-22T15:37:33.806 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:37:33.806 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-04-22T15:37:33.806 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:33.806 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 221 k 2024-04-22T15:37:33.806 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 213 k 2024-04-22T15:37:33.806 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-22T15:37:33.895 INFO:teuthology.orchestra.run.smithi016.stdout:cephadm-18.2.1-0.el9.noarch.rpm 2.4 MB/s | 221 kB 00:00 2024-04-22T15:37:33.896 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:37:33.896 INFO:teuthology.orchestra.run.smithi016.stdout:Total 2.4 MB/s | 221 kB 00:00 2024-04-22T15:37:33.896 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-22T15:37:33.902 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-22T15:37:33.902 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-22T15:37:33.904 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-22T15:37:33.905 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-22T15:37:33.978 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-22T15:37:34.468 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-22T15:37:34.559 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-22T15:37:34.633 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-test-2:18.2.1-0.el9.x86_64 4/4 2024-04-22T15:37:34.633 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-test-2:18.2.1-0.el9.x86_64 1/4 2024-04-22T15:37:34.633 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephsqlite-2:18.2.1-0.el9.x86_64 2/4 2024-04-22T15:37:34.633 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-22T15:37:34.835 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-22T15:37:34.915 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-22T15:37:34.915 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:34.915 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-22T15:37:34.916 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test-2:18.2.1-0.el9.x86_64 libcephsqlite-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:34.916 INFO:teuthology.orchestra.run.smithi027.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T15:37:34.916 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:34.916 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:37:35.082 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-22T15:37:35.082 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:35.083 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-04-22T15:37:35.083 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm-2:18.2.1-0.el9.noarch 2024-04-22T15:37:35.083 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:35.083 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:37:35.196 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph 2024-04-22T15:37:35.293 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-immutable-object-cache 2024-04-22T15:37:35.720 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:36 ago on Mon 22 Apr 2024 03:36:59 PM UTC. 2024-04-22T15:37:35.824 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:51 ago on Mon 22 Apr 2024 03:36:44 PM UTC. 2024-04-22T15:37:35.847 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:37:35.849 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:37:35.849 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-04-22T15:37:35.849 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:37:35.850 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-22T15:37:35.850 INFO:teuthology.orchestra.run.smithi027.stdout: ceph x86_64 2:18.2.1-0.el9 ceph 6.4 k 2024-04-22T15:37:35.850 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-04-22T15:37:35.850 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds x86_64 2:18.2.1-0.el9 ceph 2.1 M 2024-04-22T15:37:35.850 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr x86_64 2:18.2.1-0.el9 ceph 1.4 M 2024-04-22T15:37:35.850 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core noarch 2:18.2.1-0.el9 ceph-noarch 242 k 2024-04-22T15:37:35.850 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon x86_64 2:18.2.1-0.el9 ceph 4.4 M 2024-04-22T15:37:35.850 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd x86_64 2:18.2.1-0.el9 ceph 18 M 2024-04-22T15:37:35.850 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T15:37:35.850 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T15:37:35.850 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T15:37:35.850 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T15:37:35.851 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T15:37:35.851 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T15:37:35.851 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T15:37:35.851 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-22T15:37:35.851 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T15:37:35.851 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T15:37:35.851 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T15:37:35.851 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T15:37:35.851 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T15:37:35.851 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T15:37:35.851 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T15:37:35.851 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T15:37:35.851 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T15:37:35.852 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T15:37:35.852 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T15:37:35.852 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T15:37:35.852 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T15:37:35.852 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T15:37:35.852 INFO:teuthology.orchestra.run.smithi027.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T15:37:35.852 INFO:teuthology.orchestra.run.smithi027.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T15:37:35.852 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T15:37:35.852 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T15:37:35.852 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T15:37:35.852 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:35.852 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-22T15:37:35.852 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:37:35.852 INFO:teuthology.orchestra.run.smithi027.stdout:Install 33 Packages 2024-04-22T15:37:35.852 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:35.853 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 30 M 2024-04-22T15:37:35.853 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 110 M 2024-04-22T15:37:35.853 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-22T15:37:35.947 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:37:35.948 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:37:35.948 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-04-22T15:37:35.948 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:37:35.948 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-04-22T15:37:35.948 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-0.el9 ceph 142 k 2024-04-22T15:37:35.949 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:35.949 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-22T15:37:35.949 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:37:35.949 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-04-22T15:37:35.949 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:35.949 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 142 k 2024-04-22T15:37:35.949 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 395 k 2024-04-22T15:37:35.949 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-22T15:37:36.060 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-immutable-object-cache-18.2.1-0.el9.x86_64 1.3 MB/s | 142 kB 00:00 2024-04-22T15:37:36.060 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:37:36.061 INFO:teuthology.orchestra.run.smithi016.stdout:Total 1.2 MB/s | 142 kB 00:00 2024-04-22T15:37:36.061 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-22T15:37:36.069 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-22T15:37:36.069 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-22T15:37:36.129 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-22T15:37:36.129 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-22T15:37:36.278 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-22T15:37:36.344 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:37:36.370 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:37:36.371 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:37:36.371 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T15:37:36.371 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:36.455 INFO:teuthology.orchestra.run.smithi027.stdout:(1/33): ceph-18.2.1-0.el9.x86_64.rpm 54 kB/s | 6.4 kB 00:00 2024-04-22T15:37:36.506 INFO:teuthology.orchestra.run.smithi027.stdout:(2/33): ceph-mgr-18.2.1-0.el9.x86_64.rpm 8.6 MB/s | 1.4 MB 00:00 2024-04-22T15:37:36.564 INFO:teuthology.orchestra.run.smithi027.stdout:(3/33): ceph-mds-18.2.1-0.el9.x86_64.rpm 9.3 MB/s | 2.1 MB 00:00 2024-04-22T15:37:36.589 INFO:teuthology.orchestra.run.smithi027.stdout:(4/33): ceph-mgr-modules-core-18.2.1-0.el9.noar 9.5 MB/s | 242 kB 00:00 2024-04-22T15:37:36.681 INFO:teuthology.orchestra.run.smithi027.stdout:(5/33): ceph-mon-18.2.1-0.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-04-22T15:37:36.915 INFO:teuthology.orchestra.run.smithi027.stdout:(6/33): ceph-osd-18.2.1-0.el9.x86_64.rpm 43 MB/s | 18 MB 00:00 2024-04-22T15:37:37.007 INFO:teuthology.orchestra.run.smithi027.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 607 kB/s | 253 kB 00:00 2024-04-22T15:37:37.040 INFO:teuthology.orchestra.run.smithi027.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 851 kB/s | 106 kB 00:00 2024-04-22T15:37:37.066 INFO:teuthology.orchestra.run.smithi027.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-04-22T15:37:37.099 INFO:teuthology.orchestra.run.smithi027.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 2.9 MB/s | 1.2 MB 00:00 2024-04-22T15:37:37.124 INFO:teuthology.orchestra.run.smithi027.stdout:(11/33): python3-requests-2.25.1-8.el9.noarch.r 1.5 MB/s | 125 kB 00:00 2024-04-22T15:37:37.149 INFO:teuthology.orchestra.run.smithi027.stdout:(12/33): python3-urllib3-1.26.5-5.el9.noarch.rp 2.5 MB/s | 215 kB 00:00 2024-04-22T15:37:37.217 INFO:teuthology.orchestra.run.smithi027.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 378 kB/s | 35 kB 00:00 2024-04-22T15:37:37.258 INFO:teuthology.orchestra.run.smithi027.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 469 kB/s | 51 kB 00:00 2024-04-22T15:37:37.284 INFO:teuthology.orchestra.run.smithi027.stdout:(15/33): python3-mako-1.1.4-6.el9.noarch.rpm 935 kB/s | 172 kB 00:00 2024-04-22T15:37:37.309 INFO:teuthology.orchestra.run.smithi027.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 453 kB/s | 42 kB 00:00 2024-04-22T15:37:37.357 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:37:37.357 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:37.357 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-04-22T15:37:37.357 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:37.358 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:37.358 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:37:37.451 INFO:teuthology.orchestra.run.smithi027.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 226 kB/s | 43 kB 00:00 2024-04-22T15:37:37.510 INFO:teuthology.orchestra.run.smithi027.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2024-04-22T15:37:37.535 INFO:teuthology.orchestra.run.smithi027.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 762 kB/s | 172 kB 00:00 2024-04-22T15:37:37.556 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr 2024-04-22T15:37:37.568 INFO:teuthology.orchestra.run.smithi027.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 302 kB/s | 18 kB 00:00 2024-04-22T15:37:37.594 INFO:teuthology.orchestra.run.smithi027.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 395 kB/s | 23 kB 00:00 2024-04-22T15:37:37.619 INFO:teuthology.orchestra.run.smithi027.stdout:(22/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.1 MB/s | 358 kB 00:00 2024-04-22T15:37:37.645 INFO:teuthology.orchestra.run.smithi027.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2024-04-22T15:37:37.670 INFO:teuthology.orchestra.run.smithi027.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-04-22T15:37:37.695 INFO:teuthology.orchestra.run.smithi027.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 613 kB/s | 46 kB 00:00 2024-04-22T15:37:37.729 INFO:teuthology.orchestra.run.smithi027.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 944 kB/s | 79 kB 00:00 2024-04-22T15:37:37.754 INFO:teuthology.orchestra.run.smithi027.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2024-04-22T15:37:37.779 INFO:teuthology.orchestra.run.smithi027.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-04-22T15:37:37.813 INFO:teuthology.orchestra.run.smithi027.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 610 kB/s | 36 kB 00:00 2024-04-22T15:37:37.838 INFO:teuthology.orchestra.run.smithi027.stdout:(30/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 824 kB/s | 90 kB 00:00 2024-04-22T15:37:37.863 INFO:teuthology.orchestra.run.smithi027.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-04-22T15:37:37.880 INFO:teuthology.orchestra.run.smithi027.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 479 kB/s | 20 kB 00:00 2024-04-22T15:37:37.939 INFO:teuthology.orchestra.run.smithi027.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-04-22T15:37:37.940 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:37:37.940 INFO:teuthology.orchestra.run.smithi027.stdout:Total 14 MB/s | 30 MB 00:02 2024-04-22T15:37:38.061 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-22T15:37:38.092 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:54 ago on Mon 22 Apr 2024 03:36:44 PM UTC. 2024-04-22T15:37:38.101 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-22T15:37:38.101 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-22T15:37:38.174 INFO:teuthology.orchestra.run.smithi016.stdout:Package ceph-mgr-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T15:37:38.218 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:37:38.219 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-04-22T15:37:38.219 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:37:38.293 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-dashboard 2024-04-22T15:37:38.402 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-22T15:37:38.404 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-22T15:37:38.843 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:54 ago on Mon 22 Apr 2024 03:36:44 PM UTC. 2024-04-22T15:37:38.967 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:37:38.968 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:37:38.968 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-04-22T15:37:38.968 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:37:38.968 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-04-22T15:37:38.968 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard noarch 2:18.2.1-0.el9 ceph-noarch 1.8 M 2024-04-22T15:37:38.968 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-04-22T15:37:38.968 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards noarch 2:18.2.1-0.el9 ceph-noarch 23 k 2024-04-22T15:37:38.968 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts noarch 2:18.2.1-0.el9 ceph-noarch 15 k 2024-04-22T15:37:38.968 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-22T15:37:38.968 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T15:37:38.968 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T15:37:38.968 INFO:teuthology.orchestra.run.smithi016.stdout:Installing weak dependencies: 2024-04-22T15:37:38.968 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-22T15:37:38.969 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:38.969 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-22T15:37:38.969 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:37:38.969 INFO:teuthology.orchestra.run.smithi016.stdout:Install 7 Packages 2024-04-22T15:37:38.969 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:38.969 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 2.1 M 2024-04-22T15:37:38.969 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 8.5 M 2024-04-22T15:37:38.970 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-22T15:37:39.288 INFO:teuthology.orchestra.run.smithi016.stdout:(1/7): ceph-grafana-dashboards-18.2.1-0.el9.noa 154 kB/s | 23 kB 00:00 2024-04-22T15:37:39.339 INFO:teuthology.orchestra.run.smithi016.stdout:(2/7): ceph-mgr-dashboard-18.2.1-0.el9.noarch.r 8.8 MB/s | 1.8 MB 00:00 2024-04-22T15:37:39.389 INFO:teuthology.orchestra.run.smithi016.stdout:(3/7): ceph-prometheus-alerts-18.2.1-0.el9.noar 58 kB/s | 15 kB 00:00 2024-04-22T15:37:39.456 INFO:teuthology.orchestra.run.smithi016.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 54 kB/s | 9.0 kB 00:00 2024-04-22T15:37:39.481 INFO:teuthology.orchestra.run.smithi016.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 479 kB/s | 31 kB 00:00 2024-04-22T15:37:39.540 INFO:teuthology.orchestra.run.smithi016.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 235 kB/s | 41 kB 00:00 2024-04-22T15:37:39.563 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-22T15:37:39.565 INFO:teuthology.orchestra.run.smithi016.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.7 MB/s | 188 kB 00:00 2024-04-22T15:37:39.568 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:37:39.568 INFO:teuthology.orchestra.run.smithi016.stdout:Total 3.4 MB/s | 2.1 MB 00:00 2024-04-22T15:37:39.568 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-22T15:37:39.577 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-22T15:37:39.577 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-22T15:37:39.616 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-22T15:37:39.632 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-22T15:37:39.632 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-22T15:37:39.649 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-22T15:37:39.682 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-22T15:37:39.778 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-22T15:37:39.863 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-22T15:37:39.926 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-22T15:37:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-22T15:37:39.978 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-22T15:37:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-22T15:37:40.008 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-04-22T15:37:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-22T15:37:40.042 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-04-22T15:37:40.076 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-04-22T15:37:40.083 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-22T15:37:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-22T15:37:40.117 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-04-22T15:37:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 5/7 2024-04-22T15:37:40.160 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-04-22T15:37:40.186 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-04-22T15:37:40.224 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-04-22T15:37:40.302 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-04-22T15:37:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 6/7 2024-04-22T15:37:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-04-22T15:37:40.453 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-04-22T15:37:40.543 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-04-22T15:37:40.595 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-04-22T15:37:40.633 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-04-22T15:37:40.691 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-04-22T15:37:40.755 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-04-22T15:37:40.925 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-04-22T15:37:40.925 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 1/7 2024-04-22T15:37:40.925 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2/7 2024-04-22T15:37:40.925 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 3/7 2024-04-22T15:37:40.926 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-22T15:37:40.926 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-22T15:37:40.926 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-22T15:37:40.952 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-04-22T15:37:41.012 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-04-22T15:37:41.092 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-04-22T15:37:41.182 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-22T15:37:41.182 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:41.182 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-04-22T15:37:41.182 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 2024-04-22T15:37:41.182 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2024-04-22T15:37:41.183 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 2024-04-22T15:37:41.183 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-22T15:37:41.183 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-22T15:37:41.183 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T15:37:41.183 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T15:37:41.183 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:41.183 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:37:41.190 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-04-22T15:37:41.227 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-04-22T15:37:41.342 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-22T15:37:41.348 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-04-22T15:37:41.512 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 28/33 2024-04-22T15:37:41.538 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-04-22T15:37:41.564 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-04-22T15:37:41.564 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:37:41.564 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T15:37:41.564 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T15:37:41.564 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T15:37:41.564 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:41.870 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:57 ago on Mon 22 Apr 2024 03:36:44 PM UTC. 2024-04-22T15:37:41.995 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:37:41.996 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:37:41.996 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-04-22T15:37:41.996 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:37:41.996 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-04-22T15:37:41.996 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-0.el9 ceph-noarch 7.4 M 2024-04-22T15:37:41.997 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-04-22T15:37:41.997 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T15:37:41.997 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T15:37:41.997 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T15:37:41.997 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T15:37:41.997 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T15:37:41.997 INFO:teuthology.orchestra.run.smithi016.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T15:37:41.997 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T15:37:41.997 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T15:37:41.997 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T15:37:41.997 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T15:37:41.997 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T15:37:41.997 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:41.997 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-22T15:37:41.998 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:37:41.998 INFO:teuthology.orchestra.run.smithi016.stdout:Install 12 Packages 2024-04-22T15:37:41.998 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:41.998 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 38 M 2024-04-22T15:37:41.998 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 209 M 2024-04-22T15:37:41.998 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-22T15:37:42.788 INFO:teuthology.orchestra.run.smithi016.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 798 kB/s | 195 kB 00:00 2024-04-22T15:37:42.881 INFO:teuthology.orchestra.run.smithi016.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 356 kB/s | 33 kB 00:00 2024-04-22T15:37:42.914 INFO:teuthology.orchestra.run.smithi016.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-04-22T15:37:43.039 INFO:teuthology.orchestra.run.smithi016.stdout:(4/12): ceph-mgr-diskprediction-local-18.2.1-0. 15 MB/s | 7.4 MB 00:00 2024-04-22T15:37:43.098 INFO:teuthology.orchestra.run.smithi016.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 99 kB/s | 18 kB 00:00 2024-04-22T15:37:43.123 INFO:teuthology.orchestra.run.smithi016.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 422 kB/s | 35 kB 00:00 2024-04-22T15:37:43.172 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-04-22T15:37:43.182 INFO:teuthology.orchestra.run.smithi016.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.9 MB/s | 3.0 MB 00:00 2024-04-22T15:37:43.201 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-04-22T15:37:43.202 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:37:43.202 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T15:37:43.202 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T15:37:43.202 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T15:37:43.202 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:43.223 INFO:teuthology.orchestra.run.smithi016.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 2.4 MB/s | 244 kB 00:00 2024-04-22T15:37:43.324 INFO:teuthology.orchestra.run.smithi016.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 5.1 MB 00:00 2024-04-22T15:37:43.349 INFO:teuthology.orchestra.run.smithi016.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.0 MB/s | 250 kB 00:00 2024-04-22T15:37:43.450 INFO:teuthology.orchestra.run.smithi016.stdout:(11/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 13 MB/s | 4.7 MB 00:00 2024-04-22T15:37:43.658 INFO:teuthology.orchestra.run.smithi016.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 49 MB/s | 16 MB 00:00 2024-04-22T15:37:43.659 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:37:43.660 INFO:teuthology.orchestra.run.smithi016.stdout:Total 23 MB/s | 38 MB 00:01 2024-04-22T15:37:43.673 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-04-22T15:37:43.698 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-04-22T15:37:43.698 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:37:43.698 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T15:37:43.698 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T15:37:43.698 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T15:37:43.698 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:43.915 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-04-22T15:37:43.940 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-04-22T15:37:43.940 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:37:43.940 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T15:37:43.940 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T15:37:43.940 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T15:37:43.940 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:44.005 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-2:18.2.1-0.el9.x86_64 33/33 2024-04-22T15:37:44.020 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-22T15:37:44.047 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-22T15:37:44.047 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-22T15:37:44.479 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-22T15:37:44.479 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-22T15:37:45.260 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-22T15:37:45.329 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-22T15:37:45.373 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-22T15:37:45.477 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-22T15:37:45.525 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-22T15:37:45.710 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-22T15:37:45.740 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-22T15:37:45.785 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-2:18.2.1-0.el9.x86_64 33/33 2024-04-22T15:37:45.785 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-2:18.2.1-0.el9.x86_64 1/33 2024-04-22T15:37:45.785 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mds-2:18.2.1-0.el9.x86_64 2/33 2024-04-22T15:37:45.785 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-2:18.2.1-0.el9.x86_64 3/33 2024-04-22T15:37:45.785 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mon-2:18.2.1-0.el9.x86_64 4/33 2024-04-22T15:37:45.785 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-osd-2:18.2.1-0.el9.x86_64 5/33 2024-04-22T15:37:45.785 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 6/33 2024-04-22T15:37:45.785 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-04-22T15:37:45.786 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-04-22T15:37:45.786 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-04-22T15:37:45.786 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-04-22T15:37:45.786 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-04-22T15:37:45.786 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-04-22T15:37:45.786 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-04-22T15:37:45.786 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-04-22T15:37:45.786 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-04-22T15:37:45.786 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-22T15:37:45.786 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-22T15:37:45.786 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-22T15:37:45.786 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-22T15:37:45.786 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-22T15:37:45.786 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-22T15:37:45.787 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-22T15:37:45.787 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-22T15:37:45.787 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-22T15:37:45.787 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-22T15:37:45.788 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-22T15:37:45.788 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-22T15:37:45.788 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-22T15:37:45.788 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-22T15:37:45.788 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-22T15:37:45.788 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-22T15:37:45.788 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-22T15:37:45.819 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-22T15:37:46.505 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-22T15:37:46.631 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-22T15:37:46.650 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-22T15:37:46.650 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:46.650 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-22T15:37:46.651 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:46.651 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:46.651 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:46.651 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 2024-04-22T15:37:46.651 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:46.651 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:46.651 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T15:37:46.651 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T15:37:46.651 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T15:37:46.651 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T15:37:46.651 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T15:37:46.651 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T15:37:46.651 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T15:37:46.652 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T15:37:46.652 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T15:37:46.652 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T15:37:46.652 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T15:37:46.652 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T15:37:46.652 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T15:37:46.652 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T15:37:46.652 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T15:37:46.652 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T15:37:46.652 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T15:37:46.652 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T15:37:46.652 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T15:37:46.652 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T15:37:46.652 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T15:37:46.653 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T15:37:46.653 INFO:teuthology.orchestra.run.smithi027.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T15:37:46.653 INFO:teuthology.orchestra.run.smithi027.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T15:37:46.653 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T15:37:46.653 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T15:37:46.653 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T15:37:46.653 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:46.653 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:37:47.002 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-base 2024-04-22T15:37:47.544 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:48 ago on Mon 22 Apr 2024 03:36:59 PM UTC. 2024-04-22T15:37:47.626 INFO:teuthology.orchestra.run.smithi027.stdout:Package ceph-base-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T15:37:47.670 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:37:47.671 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-04-22T15:37:47.673 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:37:47.744 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install cephadm 2024-04-22T15:37:47.926 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T15:37:48.286 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:49 ago on Mon 22 Apr 2024 03:36:59 PM UTC. 2024-04-22T15:37:48.411 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:37:48.412 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:37:48.412 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-04-22T15:37:48.412 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:37:48.412 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-22T15:37:48.412 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm noarch 2:18.2.1-0.el9 ceph-noarch 221 k 2024-04-22T15:37:48.412 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:48.412 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-22T15:37:48.412 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:37:48.412 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-04-22T15:37:48.412 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:48.412 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 221 k 2024-04-22T15:37:48.412 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 213 k 2024-04-22T15:37:48.413 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-22T15:37:48.489 INFO:teuthology.orchestra.run.smithi027.stdout:cephadm-18.2.1-0.el9.noarch.rpm 2.9 MB/s | 221 kB 00:00 2024-04-22T15:37:48.489 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:37:48.490 INFO:teuthology.orchestra.run.smithi027.stdout:Total 2.8 MB/s | 221 kB 00:00 2024-04-22T15:37:48.490 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-22T15:37:48.495 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-22T15:37:48.495 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-22T15:37:48.498 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-22T15:37:48.498 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-22T15:37:48.566 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-22T15:37:48.917 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-22T15:37:49.010 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-22T15:37:49.021 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-22T15:37:49.048 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-04-22T15:37:49.323 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-22T15:37:49.581 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-22T15:37:49.581 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:49.581 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-22T15:37:49.581 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm-2:18.2.1-0.el9.noarch 2024-04-22T15:37:49.581 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:49.581 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:37:49.777 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-immutable-object-cache 2024-04-22T15:37:50.305 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:51 ago on Mon 22 Apr 2024 03:36:59 PM UTC. 2024-04-22T15:37:50.429 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:37:50.430 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:37:50.430 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-04-22T15:37:50.430 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:37:50.430 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-22T15:37:50.430 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-0.el9 ceph 142 k 2024-04-22T15:37:50.430 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:50.430 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-22T15:37:50.430 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:37:50.430 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-04-22T15:37:50.430 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:50.431 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 142 k 2024-04-22T15:37:50.431 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 395 k 2024-04-22T15:37:50.431 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-22T15:37:50.538 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-immutable-object-cache-18.2.1-0.el9.x86_64 1.3 MB/s | 142 kB 00:00 2024-04-22T15:37:50.539 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:37:50.539 INFO:teuthology.orchestra.run.smithi027.stdout:Total 1.3 MB/s | 142 kB 00:00 2024-04-22T15:37:50.539 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-22T15:37:50.545 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-22T15:37:50.545 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-22T15:37:50.604 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-22T15:37:50.605 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-22T15:37:50.765 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-22T15:37:50.829 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:37:50.855 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:37:50.855 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:37:50.855 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T15:37:50.855 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:51.569 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-04-22T15:37:51.569 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 1/12 2024-04-22T15:37:51.569 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-22T15:37:51.569 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-22T15:37:51.569 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-22T15:37:51.569 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-22T15:37:51.569 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-22T15:37:51.570 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-22T15:37:51.570 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-22T15:37:51.570 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-22T15:37:51.570 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T15:37:51.570 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-22T15:37:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:37:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:51.716 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-22T15:37:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 2024-04-22T15:37:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:51.716 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:37:51.930 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr 2024-04-22T15:37:51.940 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-22T15:37:51.940 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:51.940 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-04-22T15:37:51.940 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarch 2024-04-22T15:37:51.940 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T15:37:51.940 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T15:37:51.941 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T15:37:51.941 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T15:37:51.941 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T15:37:51.941 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T15:37:51.941 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T15:37:51.941 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T15:37:51.941 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T15:37:51.941 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T15:37:51.941 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T15:37:51.941 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:51.941 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:37:52.274 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-rook 2024-04-22T15:37:52.465 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:53 ago on Mon 22 Apr 2024 03:36:59 PM UTC. 2024-04-22T15:37:52.550 INFO:teuthology.orchestra.run.smithi027.stdout:Package ceph-mgr-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T15:37:52.593 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:37:52.594 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-04-22T15:37:52.594 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:37:52.657 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-dashboard 2024-04-22T15:37:52.806 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:08 ago on Mon 22 Apr 2024 03:36:44 PM UTC. 2024-04-22T15:37:52.932 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:37:52.933 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:37:52.933 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-04-22T15:37:52.933 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:37:52.933 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-04-22T15:37:52.933 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook noarch 2:18.2.1-0.el9 ceph-noarch 50 k 2024-04-22T15:37:52.933 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-04-22T15:37:52.933 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T15:37:52.933 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T15:37:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T15:37:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T15:37:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T15:37:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T15:37:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T15:37:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T15:37:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T15:37:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T15:37:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T15:37:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T15:37:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:52.934 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-22T15:37:52.934 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:37:52.934 INFO:teuthology.orchestra.run.smithi016.stdout:Install 13 Packages 2024-04-22T15:37:52.935 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:52.935 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 2.2 M 2024-04-22T15:37:52.935 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 26 M 2024-04-22T15:37:52.935 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-22T15:37:53.192 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:54 ago on Mon 22 Apr 2024 03:36:59 PM UTC. 2024-04-22T15:37:53.317 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:37:53.318 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:37:53.318 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-04-22T15:37:53.318 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:37:53.318 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-22T15:37:53.318 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard noarch 2:18.2.1-0.el9 ceph-noarch 1.8 M 2024-04-22T15:37:53.318 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-04-22T15:37:53.318 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards noarch 2:18.2.1-0.el9 ceph-noarch 23 k 2024-04-22T15:37:53.318 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts noarch 2:18.2.1-0.el9 ceph-noarch 15 k 2024-04-22T15:37:53.318 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-22T15:37:53.318 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T15:37:53.318 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T15:37:53.318 INFO:teuthology.orchestra.run.smithi027.stdout:Installing weak dependencies: 2024-04-22T15:37:53.319 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-22T15:37:53.319 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:53.319 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-22T15:37:53.319 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:37:53.319 INFO:teuthology.orchestra.run.smithi027.stdout:Install 7 Packages 2024-04-22T15:37:53.319 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:53.319 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 2.1 M 2024-04-22T15:37:53.319 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 8.5 M 2024-04-22T15:37:53.320 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-22T15:37:53.516 INFO:teuthology.orchestra.run.smithi027.stdout:(1/7): ceph-grafana-dashboards-18.2.1-0.el9.noa 215 kB/s | 23 kB 00:00 2024-04-22T15:37:53.557 INFO:teuthology.orchestra.run.smithi027.stdout:(2/7): ceph-prometheus-alerts-18.2.1-0.el9.noar 98 kB/s | 15 kB 00:00 2024-04-22T15:37:53.625 INFO:teuthology.orchestra.run.smithi027.stdout:(3/7): ceph-mgr-dashboard-18.2.1-0.el9.noarch.r 8.1 MB/s | 1.8 MB 00:00 2024-04-22T15:37:53.689 INFO:teuthology.orchestra.run.smithi016.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 329 kB/s | 26 kB 00:00 2024-04-22T15:37:53.706 INFO:teuthology.orchestra.run.smithi016.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 202 kB/s | 19 kB 00:00 2024-04-22T15:37:53.723 INFO:teuthology.orchestra.run.smithi016.stdout:(3/13): ceph-mgr-rook-18.2.1-0.el9.noarch.rpm 448 kB/s | 50 kB 00:00 2024-04-22T15:37:53.783 INFO:teuthology.orchestra.run.smithi027.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 34 kB/s | 9.0 kB 00:00 2024-04-22T15:37:53.807 INFO:teuthology.orchestra.run.smithi016.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 222 kB 00:00 2024-04-22T15:37:53.849 INFO:teuthology.orchestra.run.smithi016.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 159 kB 00:00 2024-04-22T15:37:53.866 INFO:teuthology.orchestra.run.smithi016.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 915 kB/s | 54 kB 00:00 2024-04-22T15:37:53.883 INFO:teuthology.orchestra.run.smithi027.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 126 kB/s | 41 kB 00:00 2024-04-22T15:37:53.891 INFO:teuthology.orchestra.run.smithi016.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.6 MB/s | 279 kB 00:00 2024-04-22T15:37:53.967 INFO:teuthology.orchestra.run.smithi027.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 106 kB/s | 31 kB 00:00 2024-04-22T15:37:54.026 INFO:teuthology.orchestra.run.smithi027.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 779 kB/s | 188 kB 00:00 2024-04-22T15:37:54.029 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:37:54.030 INFO:teuthology.orchestra.run.smithi027.stdout:Total 2.9 MB/s | 2.1 MB 00:00 2024-04-22T15:37:54.030 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-22T15:37:54.038 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-22T15:37:54.039 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-22T15:37:54.091 INFO:teuthology.orchestra.run.smithi016.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 133 kB/s | 32 kB 00:00 2024-04-22T15:37:54.094 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-22T15:37:54.094 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-22T15:37:54.117 INFO:teuthology.orchestra.run.smithi016.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2024-04-22T15:37:54.325 INFO:teuthology.orchestra.run.smithi016.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 284 kB/s | 59 kB 00:00 2024-04-22T15:37:54.350 INFO:teuthology.orchestra.run.smithi016.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 442 kB/s | 203 kB 00:00 2024-04-22T15:37:54.393 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-22T15:37:54.452 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-22T15:37:54.486 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-22T15:37:54.492 INFO:teuthology.orchestra.run.smithi016.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 538 kB/s | 90 kB 00:00 2024-04-22T15:37:54.526 INFO:teuthology.orchestra.run.smithi016.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.5 MB/s | 1.0 MB 00:00 2024-04-22T15:37:54.527 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:37:54.527 INFO:teuthology.orchestra.run.smithi016.stdout:Total 1.4 MB/s | 2.2 MB 00:01 2024-04-22T15:37:54.542 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-22T15:37:54.573 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-22T15:37:54.597 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-22T15:37:54.601 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 5/7 2024-04-22T15:37:54.614 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-22T15:37:54.614 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-22T15:37:54.707 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-22T15:37:54.708 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-22T15:37:54.850 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 6/7 2024-04-22T15:37:54.888 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-04-22T15:37:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-22T15:37:55.352 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-04-22T15:37:55.352 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 1/7 2024-04-22T15:37:55.352 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2/7 2024-04-22T15:37:55.353 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 3/7 2024-04-22T15:37:55.353 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-22T15:37:55.353 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-22T15:37:55.353 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-22T15:37:55.490 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-22T15:37:55.557 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-22T15:37:55.601 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-22T15:37:55.601 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:55.601 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-22T15:37:55.601 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 2024-04-22T15:37:55.601 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2024-04-22T15:37:55.601 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 2024-04-22T15:37:55.602 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-22T15:37:55.602 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-22T15:37:55.602 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T15:37:55.602 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T15:37:55.602 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:55.602 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:37:55.618 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-22T15:37:55.654 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-22T15:37:55.696 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-22T15:37:55.787 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-22T15:37:55.792 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-22T15:37:55.903 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-22T15:37:55.956 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-22T15:37:56.315 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:57 ago on Mon 22 Apr 2024 03:36:59 PM UTC. 2024-04-22T15:37:56.440 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:37:56.441 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:37:56.441 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-04-22T15:37:56.441 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:37:56.441 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-22T15:37:56.442 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-0.el9 ceph-noarch 7.4 M 2024-04-22T15:37:56.442 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-04-22T15:37:56.442 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T15:37:56.442 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T15:37:56.442 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T15:37:56.442 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T15:37:56.442 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T15:37:56.442 INFO:teuthology.orchestra.run.smithi027.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T15:37:56.442 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T15:37:56.442 INFO:teuthology.orchestra.run.smithi027.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T15:37:56.442 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T15:37:56.442 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T15:37:56.442 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T15:37:56.443 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:56.443 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-22T15:37:56.443 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:37:56.443 INFO:teuthology.orchestra.run.smithi027.stdout:Install 12 Packages 2024-04-22T15:37:56.443 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:37:56.443 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 38 M 2024-04-22T15:37:56.443 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 209 M 2024-04-22T15:37:56.444 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-22T15:37:56.711 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-22T15:37:56.769 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-22T15:37:56.803 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-22T15:37:56.851 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-22T15:37:56.881 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-04-22T15:37:56.949 INFO:teuthology.orchestra.run.smithi027.stdout:(1/12): ceph-mgr-diskprediction-local-18.2.1-0. 45 MB/s | 7.4 MB 00:00 2024-04-22T15:37:57.032 INFO:teuthology.orchestra.run.smithi027.stdout:(2/12): libquadmath-11.4.1-3.el9.x86_64.rpm 787 kB/s | 195 kB 00:00 2024-04-22T15:37:57.082 INFO:teuthology.orchestra.run.smithi027.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 245 kB/s | 33 kB 00:00 2024-04-22T15:37:57.158 INFO:teuthology.orchestra.run.smithi027.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 242 kB/s | 18 kB 00:00 2024-04-22T15:37:57.233 INFO:teuthology.orchestra.run.smithi027.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 469 kB/s | 35 kB 00:00 2024-04-22T15:37:57.317 INFO:teuthology.orchestra.run.smithi027.stdout:(6/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-04-22T15:37:57.484 INFO:teuthology.orchestra.run.smithi027.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 1.4 MB/s | 244 kB 00:00 2024-04-22T15:37:57.672 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-04-22T15:37:57.672 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-rook-2:18.2.1-0.el9.noarch 1/13 2024-04-22T15:37:57.672 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-22T15:37:57.672 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-22T15:37:57.672 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-22T15:37:57.672 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-22T15:37:57.672 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-22T15:37:57.672 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-22T15:37:57.672 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-22T15:37:57.673 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-22T15:37:57.673 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-22T15:37:57.673 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-22T15:37:57.673 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-22T15:37:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-22T15:37:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:57.981 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-04-22T15:37:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook-2:18.2.1-0.el9.noarch 2024-04-22T15:37:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T15:37:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T15:37:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T15:37:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T15:37:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T15:37:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T15:37:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T15:37:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T15:37:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T15:37:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T15:37:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T15:37:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T15:37:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:57.982 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:37:58.175 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-cephadm 2024-04-22T15:37:58.551 INFO:teuthology.orchestra.run.smithi027.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.0 MB/s | 3.0 MB 00:01 2024-04-22T15:37:58.710 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:14 ago on Mon 22 Apr 2024 03:36:44 PM UTC. 2024-04-22T15:37:58.734 INFO:teuthology.orchestra.run.smithi027.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.3 MB/s | 250 kB 00:00 2024-04-22T15:37:58.836 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:37:58.837 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:37:58.837 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-04-22T15:37:58.838 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:37:58.838 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-04-22T15:37:58.838 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm noarch 2:18.2.1-0.el9 ceph-noarch 132 k 2024-04-22T15:37:58.838 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-04-22T15:37:58.838 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T15:37:58.838 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T15:37:58.838 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T15:37:58.838 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T15:37:58.838 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T15:37:58.838 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:58.838 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-22T15:37:58.838 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:37:58.839 INFO:teuthology.orchestra.run.smithi016.stdout:Install 6 Packages 2024-04-22T15:37:58.839 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:37:58.839 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 7.0 M 2024-04-22T15:37:58.839 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 33 M 2024-04-22T15:37:58.839 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-22T15:37:59.185 INFO:teuthology.orchestra.run.smithi027.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 2.4 MB/s | 4.7 MB 00:01 2024-04-22T15:37:59.421 INFO:teuthology.orchestra.run.smithi016.stdout:(1/6): ceph-mgr-cephadm-18.2.1-0.el9.noarch.rpm 1.8 MB/s | 132 kB 00:00 2024-04-22T15:37:59.826 INFO:teuthology.orchestra.run.smithi027.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.2 MB/s | 5.1 MB 00:02 2024-04-22T15:38:00.013 INFO:teuthology.orchestra.run.smithi016.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 927 kB/s | 548 kB 00:00 2024-04-22T15:38:00.071 INFO:teuthology.orchestra.run.smithi016.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 344 kB/s | 248 kB 00:00 2024-04-22T15:38:00.364 INFO:teuthology.orchestra.run.smithi016.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 165 kB/s | 58 kB 00:00 2024-04-22T15:38:00.414 INFO:teuthology.orchestra.run.smithi016.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 149 kB/s | 51 kB 00:00 2024-04-22T15:38:00.972 INFO:teuthology.orchestra.run.smithi016.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.7 MB/s | 6.0 MB 00:01 2024-04-22T15:38:00.973 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:38:00.973 INFO:teuthology.orchestra.run.smithi016.stdout:Total 3.3 MB/s | 7.0 MB 00:02 2024-04-22T15:38:01.049 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-22T15:38:01.062 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-22T15:38:01.062 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-22T15:38:01.165 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-22T15:38:01.165 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-22T15:38:01.595 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-22T15:38:01.774 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-22T15:38:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-22T15:38:02.392 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-22T15:38:02.498 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-22T15:38:02.566 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-22T15:38:02.594 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-04-22T15:38:03.303 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-04-22T15:38:03.303 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 1/6 2024-04-22T15:38:03.303 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-22T15:38:03.303 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-22T15:38:03.303 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-22T15:38:03.303 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-22T15:38:03.554 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-22T15:38:03.554 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:03.554 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-04-22T15:38:03.554 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 2024-04-22T15:38:03.554 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T15:38:03.554 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T15:38:03.554 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T15:38:03.554 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T15:38:03.554 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T15:38:03.554 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:03.554 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:38:03.754 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-fuse 2024-04-22T15:38:04.286 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:20 ago on Mon 22 Apr 2024 03:36:44 PM UTC. 2024-04-22T15:38:04.409 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:38:04.410 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:38:04.410 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-04-22T15:38:04.410 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:38:04.410 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-04-22T15:38:04.410 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse x86_64 2:18.2.1-0.el9 ceph 839 k 2024-04-22T15:38:04.410 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-04-22T15:38:04.410 INFO:teuthology.orchestra.run.smithi016.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T15:38:04.410 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:04.411 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-22T15:38:04.411 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:38:04.411 INFO:teuthology.orchestra.run.smithi016.stdout:Install 2 Packages 2024-04-22T15:38:04.411 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:04.411 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 920 k 2024-04-22T15:38:04.411 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 2.7 M 2024-04-22T15:38:04.411 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-22T15:38:04.707 INFO:teuthology.orchestra.run.smithi016.stdout:(1/2): ceph-fuse-18.2.1-0.el9.x86_64.rpm 7.7 MB/s | 839 kB 00:00 2024-04-22T15:38:05.491 INFO:teuthology.orchestra.run.smithi016.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 90 kB/s | 80 kB 00:00 2024-04-22T15:38:05.491 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:38:05.492 INFO:teuthology.orchestra.run.smithi016.stdout:Total 851 kB/s | 920 kB 00:01 2024-04-22T15:38:05.505 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-22T15:38:05.529 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-22T15:38:05.530 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-22T15:38:05.613 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-22T15:38:05.613 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-22T15:38:05.804 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-22T15:38:05.951 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-22T15:38:06.017 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-04-22T15:38:06.334 INFO:teuthology.orchestra.run.smithi027.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 2.1 MB/s | 16 MB 00:07 2024-04-22T15:38:06.335 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:38:06.336 INFO:teuthology.orchestra.run.smithi027.stdout:Total 3.8 MB/s | 38 MB 00:09 2024-04-22T15:38:06.699 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-22T15:38:06.727 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-22T15:38:06.727 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-22T15:38:06.755 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-04-22T15:38:06.756 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-fuse-2:18.2.1-0.el9.x86_64 1/2 2024-04-22T15:38:07.004 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-22T15:38:07.004 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:07.004 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-04-22T15:38:07.004 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse-2:18.2.1-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-22T15:38:07.004 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:07.004 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:38:07.165 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-22T15:38:07.165 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-22T15:38:07.236 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install librados-devel 2024-04-22T15:38:07.768 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:23 ago on Mon 22 Apr 2024 03:36:44 PM UTC. 2024-04-22T15:38:07.893 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:38:07.894 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:38:07.894 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-04-22T15:38:07.894 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:38:07.894 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-04-22T15:38:07.894 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel x86_64 2:18.2.1-0.el9 ceph 127 k 2024-04-22T15:38:07.894 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:07.895 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-22T15:38:07.895 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:38:07.895 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-04-22T15:38:07.895 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:07.895 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 127 k 2024-04-22T15:38:07.895 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 456 k 2024-04-22T15:38:07.895 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-22T15:38:07.949 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-22T15:38:07.979 INFO:teuthology.orchestra.run.smithi016.stdout:librados-devel-18.2.1-0.el9.x86_64.rpm 1.5 MB/s | 127 kB 00:00 2024-04-22T15:38:07.979 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:38:07.980 INFO:teuthology.orchestra.run.smithi016.stdout:Total 1.5 MB/s | 127 kB 00:00 2024-04-22T15:38:07.980 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-22T15:38:07.986 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-22T15:38:07.986 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-22T15:38:07.998 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-22T15:38:08.042 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-22T15:38:08.053 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-22T15:38:08.053 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-22T15:38:08.148 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-22T15:38:08.184 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-22T15:38:08.205 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-22T15:38:08.336 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:08.369 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-22T15:38:08.399 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-22T15:38:08.478 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-22T15:38:08.664 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:08.903 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:08.903 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:08.903 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-04-22T15:38:08.903 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel-2:18.2.1-0.el9.x86_64 2024-04-22T15:38:08.903 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:08.903 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:38:09.131 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install libcephfs2 2024-04-22T15:38:09.182 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-22T15:38:09.303 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-22T15:38:09.672 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:25 ago on Mon 22 Apr 2024 03:36:44 PM UTC. 2024-04-22T15:38:09.755 INFO:teuthology.orchestra.run.smithi016.stdout:Package libcephfs2-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T15:38:09.798 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:38:09.799 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-04-22T15:38:09.799 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:38:09.862 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install libcephfs-devel 2024-04-22T15:38:10.395 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:26 ago on Mon 22 Apr 2024 03:36:44 PM UTC. 2024-04-22T15:38:10.518 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:38:10.519 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:38:10.519 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-04-22T15:38:10.519 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:38:10.519 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-04-22T15:38:10.519 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel x86_64 2:18.2.1-0.el9 ceph 31 k 2024-04-22T15:38:10.520 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:10.520 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-22T15:38:10.520 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:38:10.520 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-04-22T15:38:10.520 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:10.520 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 31 k 2024-04-22T15:38:10.520 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 139 k 2024-04-22T15:38:10.520 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-22T15:38:10.610 INFO:teuthology.orchestra.run.smithi016.stdout:libcephfs-devel-18.2.1-0.el9.x86_64.rpm 345 kB/s | 31 kB 00:00 2024-04-22T15:38:10.611 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:38:10.611 INFO:teuthology.orchestra.run.smithi016.stdout:Total 339 kB/s | 31 kB 00:00 2024-04-22T15:38:10.611 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-22T15:38:10.614 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-22T15:38:10.614 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-22T15:38:10.617 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-22T15:38:10.617 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-22T15:38:10.640 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T15:38:10.698 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-22T15:38:10.874 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:11.214 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:11.436 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:11.436 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:11.436 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-04-22T15:38:11.436 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel-2:18.2.1-0.el9.x86_64 2024-04-22T15:38:11.436 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:11.437 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:38:11.593 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install librados2 2024-04-22T15:38:11.722 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-22T15:38:11.916 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-04-22T15:38:12.128 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:28 ago on Mon 22 Apr 2024 03:36:44 PM UTC. 2024-04-22T15:38:12.211 INFO:teuthology.orchestra.run.smithi016.stdout:Package librados2-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T15:38:12.254 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:38:12.255 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-04-22T15:38:12.255 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:38:12.319 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install librbd1 2024-04-22T15:38:12.848 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:28 ago on Mon 22 Apr 2024 03:36:44 PM UTC. 2024-04-22T15:38:12.930 INFO:teuthology.orchestra.run.smithi016.stdout:Package librbd1-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T15:38:12.973 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:38:12.974 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-04-22T15:38:12.974 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:38:13.046 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-rados 2024-04-22T15:38:13.579 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:29 ago on Mon 22 Apr 2024 03:36:44 PM UTC. 2024-04-22T15:38:13.660 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-rados-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T15:38:13.703 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:38:13.704 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-04-22T15:38:13.704 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:38:13.757 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-rgw 2024-04-22T15:38:14.155 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-04-22T15:38:14.155 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 1/12 2024-04-22T15:38:14.155 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-22T15:38:14.156 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-22T15:38:14.156 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-22T15:38:14.156 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-22T15:38:14.156 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-22T15:38:14.156 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-22T15:38:14.156 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-22T15:38:14.156 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-22T15:38:14.156 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T15:38:14.156 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-22T15:38:14.293 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:30 ago on Mon 22 Apr 2024 03:36:44 PM UTC. 2024-04-22T15:38:14.375 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-rgw-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T15:38:14.418 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:38:14.419 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-04-22T15:38:14.419 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:38:14.480 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-cephfs 2024-04-22T15:38:14.525 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-22T15:38:14.525 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:14.525 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-22T15:38:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarch 2024-04-22T15:38:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T15:38:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T15:38:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T15:38:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T15:38:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T15:38:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T15:38:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T15:38:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T15:38:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T15:38:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T15:38:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T15:38:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:14.527 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:38:14.925 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-rook 2024-04-22T15:38:15.012 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:31 ago on Mon 22 Apr 2024 03:36:44 PM UTC. 2024-04-22T15:38:15.093 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-cephfs-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T15:38:15.136 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:38:15.137 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-04-22T15:38:15.137 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:38:15.193 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-rbd 2024-04-22T15:38:15.461 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:16 ago on Mon 22 Apr 2024 03:36:59 PM UTC. 2024-04-22T15:38:15.587 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:38:15.588 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:38:15.589 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-04-22T15:38:15.589 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:38:15.589 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-22T15:38:15.589 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook noarch 2:18.2.1-0.el9 ceph-noarch 50 k 2024-04-22T15:38:15.589 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-04-22T15:38:15.589 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T15:38:15.589 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T15:38:15.589 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T15:38:15.589 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T15:38:15.589 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T15:38:15.589 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T15:38:15.589 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T15:38:15.589 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T15:38:15.590 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T15:38:15.590 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T15:38:15.590 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T15:38:15.590 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T15:38:15.590 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:15.590 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-22T15:38:15.590 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:38:15.590 INFO:teuthology.orchestra.run.smithi027.stdout:Install 13 Packages 2024-04-22T15:38:15.590 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:15.591 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 2.2 M 2024-04-22T15:38:15.591 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 26 M 2024-04-22T15:38:15.591 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-22T15:38:15.725 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:31 ago on Mon 22 Apr 2024 03:36:44 PM UTC. 2024-04-22T15:38:15.807 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-rbd-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T15:38:15.850 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:38:15.851 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-04-22T15:38:15.851 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:38:15.921 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install rbd-fuse 2024-04-22T15:38:15.943 INFO:teuthology.orchestra.run.smithi027.stdout:(1/13): ceph-mgr-rook-18.2.1-0.el9.noarch.rpm 651 kB/s | 50 kB 00:00 2024-04-22T15:38:16.435 INFO:teuthology.orchestra.run.smithi027.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 34 kB/s | 19 kB 00:00 2024-04-22T15:38:16.460 INFO:teuthology.orchestra.run.smithi027.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 43 kB/s | 26 kB 00:00 2024-04-22T15:38:16.462 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:32 ago on Mon 22 Apr 2024 03:36:44 PM UTC. 2024-04-22T15:38:16.588 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:38:16.589 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:38:16.589 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-04-22T15:38:16.589 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:38:16.589 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-04-22T15:38:16.589 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse x86_64 2:18.2.1-0.el9 ceph 86 k 2024-04-22T15:38:16.589 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:16.589 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-22T15:38:16.589 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:38:16.589 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-04-22T15:38:16.589 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:16.589 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 86 k 2024-04-22T15:38:16.589 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 226 k 2024-04-22T15:38:16.590 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-22T15:38:16.688 INFO:teuthology.orchestra.run.smithi016.stdout:rbd-fuse-18.2.1-0.el9.x86_64.rpm 880 kB/s | 86 kB 00:00 2024-04-22T15:38:16.688 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:38:16.688 INFO:teuthology.orchestra.run.smithi016.stdout:Total 869 kB/s | 86 kB 00:00 2024-04-22T15:38:16.689 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-22T15:38:16.696 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-22T15:38:16.696 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-22T15:38:16.762 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-22T15:38:16.763 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-22T15:38:16.886 INFO:teuthology.orchestra.run.smithi027.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 236 kB/s | 222 kB 00:00 2024-04-22T15:38:16.921 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-22T15:38:17.019 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:17.285 INFO:teuthology.orchestra.run.smithi027.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 187 kB/s | 159 kB 00:00 2024-04-22T15:38:17.359 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:17.427 INFO:teuthology.orchestra.run.smithi027.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 288 kB/s | 279 kB 00:00 2024-04-22T15:38:17.453 INFO:teuthology.orchestra.run.smithi027.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 193 kB/s | 32 kB 00:00 2024-04-22T15:38:17.478 INFO:teuthology.orchestra.run.smithi027.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 300 kB/s | 15 kB 00:00 2024-04-22T15:38:17.537 INFO:teuthology.orchestra.run.smithi027.stdout:(9/13): python3-requests-oauthlib-1.3.0-12.el9. 82 kB/s | 54 kB 00:00 2024-04-22T15:38:17.562 INFO:teuthology.orchestra.run.smithi027.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.8 MB/s | 203 kB 00:00 2024-04-22T15:38:17.612 INFO:teuthology.orchestra.run.smithi027.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 7.6 MB/s | 1.0 MB 00:00 2024-04-22T15:38:17.626 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:17.626 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:17.626 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-04-22T15:38:17.626 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse-2:18.2.1-0.el9.x86_64 2024-04-22T15:38:17.626 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:17.626 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:38:17.638 INFO:teuthology.orchestra.run.smithi027.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 591 kB/s | 59 kB 00:00 2024-04-22T15:38:17.663 INFO:teuthology.orchestra.run.smithi027.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 908 kB/s | 90 kB 00:00 2024-04-22T15:38:17.665 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:38:17.665 INFO:teuthology.orchestra.run.smithi027.stdout:Total 1.1 MB/s | 2.2 MB 00:02 2024-04-22T15:38:17.736 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-22T15:38:17.754 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-22T15:38:17.754 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-22T15:38:17.830 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install rbd-mirror 2024-04-22T15:38:17.849 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-22T15:38:17.849 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-22T15:38:18.378 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:34 ago on Mon 22 Apr 2024 03:36:44 PM UTC. 2024-04-22T15:38:18.444 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-22T15:38:18.505 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:38:18.506 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:38:18.506 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-04-22T15:38:18.506 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:38:18.506 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-04-22T15:38:18.506 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-04-22T15:38:18.507 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:18.507 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-22T15:38:18.507 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:38:18.507 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-04-22T15:38:18.507 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:18.507 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 3.0 M 2024-04-22T15:38:18.507 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 12 M 2024-04-22T15:38:18.507 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-22T15:38:18.649 INFO:teuthology.orchestra.run.smithi016.stdout:rbd-mirror-18.2.1-0.el9.x86_64.rpm 21 MB/s | 3.0 MB 00:00 2024-04-22T15:38:18.650 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:38:18.650 INFO:teuthology.orchestra.run.smithi016.stdout:Total 21 MB/s | 3.0 MB 00:00 2024-04-22T15:38:18.651 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-22T15:38:18.660 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-22T15:38:18.660 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-22T15:38:18.663 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-22T15:38:18.725 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-22T15:38:18.749 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-22T15:38:18.749 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-22T15:38:18.781 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-22T15:38:18.816 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-22T15:38:18.870 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-22T15:38:18.972 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-22T15:38:19.085 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-22T15:38:19.144 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-22T15:38:19.184 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-22T15:38:19.259 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:19.285 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:19.285 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:38:19.285 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T15:38:19.285 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T15:38:19.285 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T15:38:19.286 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:19.913 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-22T15:38:19.965 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-22T15:38:20.007 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-22T15:38:20.064 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-22T15:38:20.093 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-04-22T15:38:20.336 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:20.336 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:20.336 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-04-22T15:38:20.336 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror-2:18.2.1-0.el9.x86_64 2024-04-22T15:38:20.336 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:20.337 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:38:20.570 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install rbd-nbd 2024-04-22T15:38:20.859 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-04-22T15:38:20.859 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-rook-2:18.2.1-0.el9.noarch 1/13 2024-04-22T15:38:20.859 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-22T15:38:20.859 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-22T15:38:20.859 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-22T15:38:20.859 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-22T15:38:20.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-22T15:38:20.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-22T15:38:20.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-22T15:38:20.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-22T15:38:20.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-22T15:38:20.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-22T15:38:20.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-22T15:38:21.113 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:37 ago on Mon 22 Apr 2024 03:36:44 PM UTC. 2024-04-22T15:38:21.156 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-22T15:38:21.156 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:21.156 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-22T15:38:21.156 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook-2:18.2.1-0.el9.noarch 2024-04-22T15:38:21.156 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T15:38:21.156 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T15:38:21.156 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T15:38:21.156 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T15:38:21.157 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T15:38:21.157 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T15:38:21.157 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T15:38:21.157 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T15:38:21.157 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T15:38:21.157 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T15:38:21.157 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T15:38:21.157 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T15:38:21.157 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:21.157 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:38:21.242 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-22T15:38:21.242 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:38:21.242 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-04-22T15:38:21.243 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:38:21.243 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-04-22T15:38:21.243 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd x86_64 2:18.2.1-0.el9 ceph 171 k 2024-04-22T15:38:21.243 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:21.243 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-22T15:38:21.243 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-22T15:38:21.243 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-04-22T15:38:21.243 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:21.243 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 171 k 2024-04-22T15:38:21.243 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 494 k 2024-04-22T15:38:21.243 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-22T15:38:21.320 INFO:teuthology.orchestra.run.smithi016.stdout:rbd-nbd-18.2.1-0.el9.x86_64.rpm 2.2 MB/s | 171 kB 00:00 2024-04-22T15:38:21.320 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:38:21.320 INFO:teuthology.orchestra.run.smithi016.stdout:Total 2.1 MB/s | 171 kB 00:00 2024-04-22T15:38:21.321 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-22T15:38:21.329 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-22T15:38:21.329 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-22T15:38:21.388 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-cephadm 2024-04-22T15:38:21.396 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-22T15:38:21.396 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-22T15:38:21.549 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-22T15:38:21.634 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:21.920 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:22 ago on Mon 22 Apr 2024 03:36:59 PM UTC. 2024-04-22T15:38:22.033 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:22.047 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:38:22.048 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:38:22.048 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-04-22T15:38:22.048 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:38:22.048 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-22T15:38:22.048 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm noarch 2:18.2.1-0.el9 ceph-noarch 132 k 2024-04-22T15:38:22.048 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-04-22T15:38:22.048 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T15:38:22.048 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T15:38:22.048 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T15:38:22.048 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T15:38:22.048 INFO:teuthology.orchestra.run.smithi027.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T15:38:22.049 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:22.049 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-22T15:38:22.049 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:38:22.049 INFO:teuthology.orchestra.run.smithi027.stdout:Install 6 Packages 2024-04-22T15:38:22.049 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:22.049 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 7.0 M 2024-04-22T15:38:22.049 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 33 M 2024-04-22T15:38:22.050 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-22T15:38:22.259 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:22.259 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:22.259 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-04-22T15:38:22.259 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd-2:18.2.1-0.el9.x86_64 2024-04-22T15:38:22.259 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:38:22.259 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-22T15:38:22.482 DEBUG:teuthology.parallel:result is None 2024-04-22T15:38:22.483 INFO:teuthology.orchestra.run.smithi027.stdout:(1/6): ceph-mgr-cephadm-18.2.1-0.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-04-22T15:38:22.691 INFO:teuthology.orchestra.run.smithi027.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 895 kB/s | 248 kB 00:00 2024-04-22T15:38:22.874 INFO:teuthology.orchestra.run.smithi027.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 315 kB/s | 58 kB 00:00 2024-04-22T15:38:22.916 INFO:teuthology.orchestra.run.smithi027.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.2 MB/s | 548 kB 00:00 2024-04-22T15:38:22.941 INFO:teuthology.orchestra.run.smithi027.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 768 kB/s | 51 kB 00:00 2024-04-22T15:38:23.052 INFO:teuthology.orchestra.run.smithi027.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 9.3 MB/s | 6.0 MB 00:00 2024-04-22T15:38:23.054 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:38:23.054 INFO:teuthology.orchestra.run.smithi027.stdout:Total 6.9 MB/s | 7.0 MB 00:01 2024-04-22T15:38:23.126 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-22T15:38:23.138 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-22T15:38:23.138 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-22T15:38:23.239 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-22T15:38:23.239 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-22T15:38:23.556 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-22T15:38:23.727 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-22T15:38:23.789 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-22T15:38:24.343 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-22T15:38:24.434 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-22T15:38:24.503 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-22T15:38:24.533 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-04-22T15:38:25.267 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-04-22T15:38:25.268 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 1/6 2024-04-22T15:38:25.268 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-22T15:38:25.268 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-22T15:38:25.268 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-22T15:38:25.268 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-22T15:38:25.522 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-22T15:38:25.522 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:25.522 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-22T15:38:25.522 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 2024-04-22T15:38:25.522 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T15:38:25.522 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T15:38:25.522 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T15:38:25.522 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T15:38:25.523 INFO:teuthology.orchestra.run.smithi027.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T15:38:25.523 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:25.523 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:38:25.749 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-fuse 2024-04-22T15:38:26.287 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:27 ago on Mon 22 Apr 2024 03:36:59 PM UTC. 2024-04-22T15:38:26.413 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:38:26.413 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:38:26.413 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-04-22T15:38:26.413 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:38:26.414 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-22T15:38:26.414 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse x86_64 2:18.2.1-0.el9 ceph 839 k 2024-04-22T15:38:26.414 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-04-22T15:38:26.414 INFO:teuthology.orchestra.run.smithi027.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T15:38:26.414 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:26.414 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-22T15:38:26.414 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:38:26.414 INFO:teuthology.orchestra.run.smithi027.stdout:Install 2 Packages 2024-04-22T15:38:26.414 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:26.415 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 920 k 2024-04-22T15:38:26.415 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 2.7 M 2024-04-22T15:38:26.415 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-22T15:38:26.762 INFO:teuthology.orchestra.run.smithi027.stdout:(1/2): ceph-fuse-18.2.1-0.el9.x86_64.rpm 11 MB/s | 839 kB 00:00 2024-04-22T15:38:26.829 INFO:teuthology.orchestra.run.smithi027.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 571 kB/s | 80 kB 00:00 2024-04-22T15:38:26.830 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:38:26.830 INFO:teuthology.orchestra.run.smithi027.stdout:Total 2.2 MB/s | 920 kB 00:00 2024-04-22T15:38:26.841 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-22T15:38:26.864 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-22T15:38:26.864 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-22T15:38:26.946 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-22T15:38:26.946 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-22T15:38:27.133 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-22T15:38:27.272 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-22T15:38:27.338 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-04-22T15:38:27.915 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-04-22T15:38:27.915 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-fuse-2:18.2.1-0.el9.x86_64 1/2 2024-04-22T15:38:28.171 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-22T15:38:28.172 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:28.172 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-22T15:38:28.172 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse-2:18.2.1-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-22T15:38:28.172 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:28.172 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:38:28.453 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librados-devel 2024-04-22T15:38:28.990 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:29 ago on Mon 22 Apr 2024 03:36:59 PM UTC. 2024-04-22T15:38:29.117 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:38:29.118 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:38:29.118 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-04-22T15:38:29.118 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:38:29.118 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-22T15:38:29.118 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel x86_64 2:18.2.1-0.el9 ceph 127 k 2024-04-22T15:38:29.118 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:29.118 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-22T15:38:29.118 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:38:29.118 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-04-22T15:38:29.118 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:29.119 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 127 k 2024-04-22T15:38:29.119 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 456 k 2024-04-22T15:38:29.119 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-22T15:38:29.192 INFO:teuthology.orchestra.run.smithi027.stdout:librados-devel-18.2.1-0.el9.x86_64.rpm 1.7 MB/s | 127 kB 00:00 2024-04-22T15:38:29.193 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:38:29.193 INFO:teuthology.orchestra.run.smithi027.stdout:Total 1.7 MB/s | 127 kB 00:00 2024-04-22T15:38:29.193 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-22T15:38:29.199 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-22T15:38:29.199 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-22T15:38:29.265 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-22T15:38:29.265 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-22T15:38:29.416 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-22T15:38:29.524 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:29.877 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:30.134 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:30.135 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:30.135 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-22T15:38:30.135 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel-2:18.2.1-0.el9.x86_64 2024-04-22T15:38:30.135 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:30.135 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:38:30.396 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install libcephfs2 2024-04-22T15:38:30.936 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:31 ago on Mon 22 Apr 2024 03:36:59 PM UTC. 2024-04-22T15:38:31.019 INFO:teuthology.orchestra.run.smithi027.stdout:Package libcephfs2-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T15:38:31.061 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:38:31.063 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-04-22T15:38:31.063 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:38:31.271 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install libcephfs-devel 2024-04-22T15:38:31.804 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:32 ago on Mon 22 Apr 2024 03:36:59 PM UTC. 2024-04-22T15:38:31.927 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:38:31.928 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:38:31.928 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-04-22T15:38:31.929 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:38:31.929 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-22T15:38:31.929 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel x86_64 2:18.2.1-0.el9 ceph 31 k 2024-04-22T15:38:31.929 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:31.929 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-22T15:38:31.929 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:38:31.929 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-04-22T15:38:31.929 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:31.929 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 31 k 2024-04-22T15:38:31.929 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 139 k 2024-04-22T15:38:31.929 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-22T15:38:32.004 INFO:teuthology.orchestra.run.smithi027.stdout:libcephfs-devel-18.2.1-0.el9.x86_64.rpm 413 kB/s | 31 kB 00:00 2024-04-22T15:38:32.005 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:38:32.005 INFO:teuthology.orchestra.run.smithi027.stdout:Total 406 kB/s | 31 kB 00:00 2024-04-22T15:38:32.005 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-22T15:38:32.009 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-22T15:38:32.009 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-22T15:38:32.011 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-22T15:38:32.011 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-22T15:38:32.101 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-22T15:38:32.210 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:32.662 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:32.870 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:32.870 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:32.870 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-22T15:38:32.871 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel-2:18.2.1-0.el9.x86_64 2024-04-22T15:38:32.871 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:32.871 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:38:33.037 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librados2 2024-04-22T15:38:33.574 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:34 ago on Mon 22 Apr 2024 03:36:59 PM UTC. 2024-04-22T15:38:33.657 INFO:teuthology.orchestra.run.smithi027.stdout:Package librados2-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T15:38:33.701 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:38:33.702 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-04-22T15:38:33.702 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:38:33.776 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librbd1 2024-04-22T15:38:34.309 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:35 ago on Mon 22 Apr 2024 03:36:59 PM UTC. 2024-04-22T15:38:34.389 INFO:teuthology.orchestra.run.smithi027.stdout:Package librbd1-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T15:38:34.432 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:38:34.433 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-04-22T15:38:34.433 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:38:34.509 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rados 2024-04-22T15:38:35.045 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:36 ago on Mon 22 Apr 2024 03:36:59 PM UTC. 2024-04-22T15:38:35.126 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rados-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T15:38:35.168 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:38:35.170 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-04-22T15:38:35.170 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:38:35.250 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rgw 2024-04-22T15:38:35.789 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:36 ago on Mon 22 Apr 2024 03:36:59 PM UTC. 2024-04-22T15:38:35.871 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rgw-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T15:38:35.918 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:38:35.919 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-04-22T15:38:35.919 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:38:35.993 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-cephfs 2024-04-22T15:38:36.523 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:37 ago on Mon 22 Apr 2024 03:36:59 PM UTC. 2024-04-22T15:38:36.604 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-cephfs-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T15:38:36.646 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:38:36.648 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-04-22T15:38:36.648 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:38:36.722 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rbd 2024-04-22T15:38:37.260 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:38 ago on Mon 22 Apr 2024 03:36:59 PM UTC. 2024-04-22T15:38:37.342 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rbd-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T15:38:37.385 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:38:37.386 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-04-22T15:38:37.386 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:38:37.464 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-fuse 2024-04-22T15:38:37.997 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:38 ago on Mon 22 Apr 2024 03:36:59 PM UTC. 2024-04-22T15:38:38.122 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:38:38.123 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:38:38.123 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-04-22T15:38:38.123 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:38:38.123 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-22T15:38:38.123 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse x86_64 2:18.2.1-0.el9 ceph 86 k 2024-04-22T15:38:38.123 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:38.123 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-22T15:38:38.123 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:38:38.123 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-04-22T15:38:38.123 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:38.123 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 86 k 2024-04-22T15:38:38.123 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 226 k 2024-04-22T15:38:38.123 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-22T15:38:38.190 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-fuse-18.2.1-0.el9.x86_64.rpm 1.3 MB/s | 86 kB 00:00 2024-04-22T15:38:38.191 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:38:38.191 INFO:teuthology.orchestra.run.smithi027.stdout:Total 1.2 MB/s | 86 kB 00:00 2024-04-22T15:38:38.191 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-22T15:38:38.198 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-22T15:38:38.198 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-22T15:38:38.264 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-22T15:38:38.264 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-22T15:38:38.534 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-22T15:38:38.631 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:39.082 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:39.303 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:39.303 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:39.303 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-22T15:38:39.303 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse-2:18.2.1-0.el9.x86_64 2024-04-22T15:38:39.303 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:39.303 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:38:39.570 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-mirror 2024-04-22T15:38:40.107 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:41 ago on Mon 22 Apr 2024 03:36:59 PM UTC. 2024-04-22T15:38:40.232 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:38:40.233 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:38:40.233 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-04-22T15:38:40.233 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:38:40.233 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-22T15:38:40.233 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-04-22T15:38:40.233 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:40.234 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-22T15:38:40.234 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:38:40.234 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-04-22T15:38:40.234 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:40.234 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 3.0 M 2024-04-22T15:38:40.234 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 12 M 2024-04-22T15:38:40.234 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-22T15:38:40.327 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-mirror-18.2.1-0.el9.x86_64.rpm 32 MB/s | 3.0 MB 00:00 2024-04-22T15:38:40.327 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:38:40.328 INFO:teuthology.orchestra.run.smithi027.stdout:Total 32 MB/s | 3.0 MB 00:00 2024-04-22T15:38:40.328 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-22T15:38:40.337 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-22T15:38:40.337 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-22T15:38:40.421 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-22T15:38:40.421 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-22T15:38:40.863 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-22T15:38:41.028 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:41.055 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:41.055 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T15:38:41.055 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T15:38:41.055 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T15:38:41.055 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T15:38:41.055 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:41.960 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:41.961 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:41.961 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-22T15:38:41.961 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror-2:18.2.1-0.el9.x86_64 2024-04-22T15:38:41.961 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:41.961 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:38:42.239 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-nbd 2024-04-22T15:38:42.778 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:43 ago on Mon 22 Apr 2024 03:36:59 PM UTC. 2024-04-22T15:38:42.904 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-22T15:38:42.905 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:38:42.905 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-04-22T15:38:42.905 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:38:42.905 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-22T15:38:42.905 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd x86_64 2:18.2.1-0.el9 ceph 171 k 2024-04-22T15:38:42.905 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:42.905 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-22T15:38:42.905 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-22T15:38:42.905 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-04-22T15:38:42.905 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:42.905 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 171 k 2024-04-22T15:38:42.905 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 494 k 2024-04-22T15:38:42.905 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-22T15:38:42.972 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-nbd-18.2.1-0.el9.x86_64.rpm 2.5 MB/s | 171 kB 00:00 2024-04-22T15:38:42.972 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:38:42.973 INFO:teuthology.orchestra.run.smithi027.stdout:Total 2.5 MB/s | 171 kB 00:00 2024-04-22T15:38:42.973 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-22T15:38:42.981 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-22T15:38:42.982 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-22T15:38:43.047 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-22T15:38:43.048 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-22T15:38:43.202 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-22T15:38:43.304 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:43.687 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:43.945 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T15:38:43.945 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:43.945 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-22T15:38:43.946 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd-2:18.2.1-0.el9.x86_64 2024-04-22T15:38:43.946 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:38:43.946 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-22T15:38:44.203 DEBUG:teuthology.parallel:result is None 2024-04-22T15:38:44.203 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-22T15:38:44.204 INFO:teuthology.packaging:ref: None 2024-04-22T15:38:44.204 INFO:teuthology.packaging:tag: v18.2.1 2024-04-22T15:38:44.204 INFO:teuthology.packaging:branch: None 2024-04-22T15:38:44.204 INFO:teuthology.packaging:sha1: 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:38:44.204 INFO:teuthology.packaging:Tag 'v18.2.1' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-04-22T15:38:44.204 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-04-22T15:38:44.426 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T15:38:44.451 INFO:teuthology.orchestra.run.smithi016.stdout:18.2.1-0.el9 2024-04-22T15:38:44.452 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-0.el9 2024-04-22T15:38:44.452 INFO:teuthology.task.install:The correct ceph version 18.2.1-0 is installed. 2024-04-22T15:38:44.454 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-22T15:38:44.454 INFO:teuthology.packaging:ref: None 2024-04-22T15:38:44.454 INFO:teuthology.packaging:tag: v18.2.1 2024-04-22T15:38:44.454 INFO:teuthology.packaging:branch: None 2024-04-22T15:38:44.454 INFO:teuthology.packaging:sha1: 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:38:44.454 INFO:teuthology.packaging:Tag 'v18.2.1' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-04-22T15:38:44.454 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-04-22T15:38:44.668 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T15:38:44.690 INFO:teuthology.orchestra.run.smithi027.stdout:18.2.1-0.el9 2024-04-22T15:38:44.690 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-0.el9 2024-04-22T15:38:44.690 INFO:teuthology.task.install:The correct ceph version 18.2.1-0 is installed. 2024-04-22T15:38:44.693 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-22T15:38:44.693 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-22T15:38:44.693 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T15:38:44.723 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-22T15:38:44.723 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T15:38:44.755 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-22T15:38:44.755 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-22T15:38:44.756 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T15:38:44.792 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T15:38:44.862 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-22T15:38:44.862 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T15:38:44.894 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T15:38:44.959 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-22T15:38:44.959 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-22T15:38:44.959 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T15:38:44.988 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T15:38:45.055 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-22T15:38:45.055 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T15:38:45.079 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T15:38:45.143 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-22T15:38:45.143 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-22T15:38:45.143 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T15:38:45.167 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T15:38:45.232 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-22T15:38:45.232 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T15:38:45.257 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T15:38:45.321 INFO:teuthology.run_tasks:Running task print... 2024-04-22T15:38:45.330 INFO:teuthology.task.print:**** done install task... 2024-04-22T15:38:45.331 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-22T15:38:45.482 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.io/ceph/ceph:v18.2.1', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '5883d494d3973b30b3f03e4a3f0e1e06e3c131aa'} 2024-04-22T15:38:45.483 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v18.2.1 2024-04-22T15:38:45.483 INFO:tasks.cephadm:Cluster fsid is 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T15:38:45.483 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-22T15:38:45.483 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-22T15:38:45.484 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi016': '172.21.15.16', 'mon.smithi027': '172.21.15.27'} 2024-04-22T15:38:45.484 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-22T15:38:45.484 DEBUG:teuthology.orchestra.run.smithi016:> sudo hostname $(hostname -s) 2024-04-22T15:38:45.509 DEBUG:teuthology.orchestra.run.smithi027:> sudo hostname $(hostname -s) 2024-04-22T15:38:45.534 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-04-22T15:38:45.534 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:38:45.678 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '5883d494d3973b30b3f03e4a3f0e1e06e3c131aa', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78798/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3223-g5883d494', 'node_name': '172.21.5.35+adami05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3223.g5883d494'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-22-0809/5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-22 13:59:15.546489', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-adk-testing-2024-04-22-0809', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-adk-testing-2024-04-22-0809/5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-22T15:38:45.879 INFO:tasks.util.chacra:got chacra host chacra.ceph.com, ref reef, sha1 9d98999acebbad2663cc4938e94c9ffd83485cbd from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-04-22T15:38:45.881 INFO:tasks.cephadm:Discovered cachra url: https://chacra.ceph.com/binaries/ceph/reef/9d98999acebbad2663cc4938e94c9ffd83485cbd/centos/9/x86_64/flavors/default/cephadm 2024-04-22T15:38:45.881 DEBUG:teuthology.orchestra.run.smithi016:> curl --silent -L https://chacra.ceph.com/binaries/ceph/reef/9d98999acebbad2663cc4938e94c9ffd83485cbd/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T15:38:46.204 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r--. 1 ubuntu ubuntu 212578 Apr 22 15:38 /home/ubuntu/cephtest/cephadm 2024-04-22T15:38:46.204 DEBUG:teuthology.orchestra.run.smithi027:> curl --silent -L https://chacra.ceph.com/binaries/ceph/reef/9d98999acebbad2663cc4938e94c9ffd83485cbd/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T15:38:46.532 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r--. 1 ubuntu ubuntu 212578 Apr 22 15:38 /home/ubuntu/cephtest/cephadm 2024-04-22T15:38:46.532 DEBUG:teuthology.orchestra.run.smithi016:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T15:38:46.549 DEBUG:teuthology.orchestra.run.smithi027:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T15:38:46.579 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v18.2.1 on all hosts... 2024-04-22T15:38:46.579 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull 2024-04-22T15:38:46.591 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull 2024-04-22T15:38:48.054 INFO:teuthology.orchestra.run.smithi016.stderr:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-04-22T15:38:48.177 INFO:teuthology.orchestra.run.smithi027.stderr:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-04-22T15:39:08.082 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-22T15:39:08.083 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-04-22T15:39:08.083 INFO:teuthology.orchestra.run.smithi027.stdout: "image_id": "5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf", 2024-04-22T15:39:08.083 INFO:teuthology.orchestra.run.smithi027.stdout: "repo_digests": [ 2024-04-22T15:39:08.083 INFO:teuthology.orchestra.run.smithi027.stdout: "quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3", 2024-04-22T15:39:08.083 INFO:teuthology.orchestra.run.smithi027.stdout: "quay.io/ceph/ceph@sha256:e8e55db8b4fd270dbec25bc764437a2a3abb707971c4dba5f559fb83018049dc" 2024-04-22T15:39:08.083 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-22T15:39:08.083 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-22T15:39:11.087 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:39:11.088 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-04-22T15:39:11.088 INFO:teuthology.orchestra.run.smithi016.stdout: "image_id": "5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf", 2024-04-22T15:39:11.088 INFO:teuthology.orchestra.run.smithi016.stdout: "repo_digests": [ 2024-04-22T15:39:11.088 INFO:teuthology.orchestra.run.smithi016.stdout: "quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3", 2024-04-22T15:39:11.088 INFO:teuthology.orchestra.run.smithi016.stdout: "quay.io/ceph/ceph@sha256:e8e55db8b4fd270dbec25bc764437a2a3abb707971c4dba5f559fb83018049dc" 2024-04-22T15:39:11.088 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-22T15:39:11.088 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:39:11.107 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph 2024-04-22T15:39:11.159 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph 2024-04-22T15:39:11.191 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 777 /etc/ceph 2024-04-22T15:39:11.216 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 777 /etc/ceph 2024-04-22T15:39:11.259 INFO:tasks.cephadm:Writing seed config... 2024-04-22T15:39:11.260 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-04-22T15:39:11.260 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-04-22T15:39:11.261 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-22T15:39:11.261 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-22T15:39:11.261 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-22T15:39:11.261 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-22T15:39:11.261 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-22T15:39:11.261 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-22T15:39:11.261 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-22T15:39:11.261 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-22T15:39:11.261 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-22T15:39:11.261 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-22T15:39:11.261 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-22T15:39:11.261 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-22T15:39:11.261 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-22T15:39:11.261 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-22T15:39:11.262 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-22T15:39:11.262 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-22T15:39:11.262 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-22T15:39:11.262 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-22T15:39:11.262 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-22T15:39:11.262 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-22T15:39:11.262 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-22T15:39:11.262 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-04-22T15:39:11.262 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-22T15:39:11.262 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-22T15:39:11.262 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-22T15:39:11.262 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-22T15:39:11.262 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-22T15:39:11.262 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-22T15:39:11.262 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-22T15:39:11.263 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-22T15:39:11.263 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-22T15:39:11.263 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-22T15:39:11.263 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-22T15:39:11.263 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-22T15:39:11.263 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-22T15:39:11.263 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-22T15:39:11.263 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-22T15:39:11.263 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-22T15:39:11.263 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-22T15:39:11.263 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-22T15:39:11.264 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-22T15:39:11.264 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-22T15:39:11.280 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 67ab3a80-00be-11ef-bc93-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-22T15:39:11.280 DEBUG:teuthology.orchestra.run.smithi016:mon.smithi016> sudo journalctl -f -n 0 -u ceph-67ab3a80-00be-11ef-bc93-c7b262605968@mon.smithi016.service 2024-04-22T15:39:11.322 INFO:tasks.cephadm:Bootstrapping... 2024-04-22T15:39:11.322 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 -v bootstrap --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.16 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-22T15:39:11.482 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-22T15:39:11.482 INFO:teuthology.orchestra.run.smithi016.stdout:cephadm ['--image', 'quay.io/ceph/ceph:v18.2.1', '-v', 'bootstrap', '--fsid', '67ab3a80-00be-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.16', '--skip-admin-label'] 2024-04-22T15:39:11.508 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-22T15:39:11.508 INFO:teuthology.orchestra.run.smithi016.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-22T15:39:11.508 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying podman|docker is present... 2024-04-22T15:39:11.532 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-22T15:39:11.532 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying lvm2 is present... 2024-04-22T15:39:11.532 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying time synchronization is in place... 2024-04-22T15:39:11.540 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-22T15:39:11.540 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T15:39:11.546 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-22T15:39:11.546 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-04-22T15:39:11.552 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout enabled 2024-04-22T15:39:11.559 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout active 2024-04-22T15:39:11.559 INFO:teuthology.orchestra.run.smithi016.stdout:Unit chronyd.service is enabled and running 2024-04-22T15:39:11.559 INFO:teuthology.orchestra.run.smithi016.stdout:Repeating the final host check... 2024-04-22T15:39:11.583 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-22T15:39:11.583 INFO:teuthology.orchestra.run.smithi016.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-22T15:39:11.583 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl is present 2024-04-22T15:39:11.584 INFO:teuthology.orchestra.run.smithi016.stdout:lvcreate is present 2024-04-22T15:39:11.590 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-22T15:39:11.590 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T15:39:11.595 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-22T15:39:11.596 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-04-22T15:39:11.602 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout enabled 2024-04-22T15:39:11.608 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout active 2024-04-22T15:39:11.609 INFO:teuthology.orchestra.run.smithi016.stdout:Unit chronyd.service is enabled and running 2024-04-22T15:39:11.609 INFO:teuthology.orchestra.run.smithi016.stdout:Host looks OK 2024-04-22T15:39:11.609 INFO:teuthology.orchestra.run.smithi016.stdout:Cluster fsid: 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T15:39:11.609 INFO:teuthology.orchestra.run.smithi016.stdout:Acquiring lock 140261611072672 on /run/cephadm/67ab3a80-00be-11ef-bc93-c7b262605968.lock 2024-04-22T15:39:11.609 INFO:teuthology.orchestra.run.smithi016.stdout:Lock 140261611072672 acquired on /run/cephadm/67ab3a80-00be-11ef-bc93-c7b262605968.lock 2024-04-22T15:39:11.609 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying IP 172.21.15.16 port 3300 ... 2024-04-22T15:39:11.609 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying IP 172.21.15.16 port 6789 ... 2024-04-22T15:39:11.609 INFO:teuthology.orchestra.run.smithi016.stdout:Base mon IP(s) is [172.21.15.16:3300, 172.21.15.16:6789], mon addrv is [v2:172.21.15.16:3300,v1:172.21.15.16:6789] 2024-04-22T15:39:11.613 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.16 metric 100 2024-04-22T15:39:11.613 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.16 metric 100 2024-04-22T15:39:11.615 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-22T15:39:11.615 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-22T15:39:11.615 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-22T15:39:11.618 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-22T15:39:11.618 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-22T15:39:11.618 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-22T15:39:11.618 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-22T15:39:11.618 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1564/64 scope link noprefixroute 2024-04-22T15:39:11.618 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-22T15:39:11.619 INFO:teuthology.orchestra.run.smithi016.stdout:Mon IP `172.21.15.16` is in CIDR network `172.21.0.0/20` 2024-04-22T15:39:11.619 INFO:teuthology.orchestra.run.smithi016.stdout:Mon IP `172.21.15.16` is in CIDR network `172.21.0.0/20` 2024-04-22T15:39:11.619 INFO:teuthology.orchestra.run.smithi016.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-22T15:39:11.620 INFO:teuthology.orchestra.run.smithi016.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-22T15:39:11.620 INFO:teuthology.orchestra.run.smithi016.stdout:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-04-22T15:39:12.542 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf 2024-04-22T15:39:12.543 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Trying to pull quay.io/ceph/ceph:v18.2.1... 2024-04-22T15:39:12.543 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Getting image source signatures 2024-04-22T15:39:12.543 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying blob sha256:a733d3c618b71f19c168ebecd1953429dce2c1631835ca182e9551c36dce5989 2024-04-22T15:39:12.543 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying blob sha256:7feca07754707458c3945cf0062cf4dabc512f6d90fe1a9a1370b362b6011124 2024-04-22T15:39:12.543 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying config sha256:5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf 2024-04-22T15:39:12.543 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-22T15:39:13.447 INFO:teuthology.orchestra.run.smithi016.stdout:ceph: stdout ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) 2024-04-22T15:39:13.447 INFO:teuthology.orchestra.run.smithi016.stdout:Ceph version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) 2024-04-22T15:39:13.448 INFO:teuthology.orchestra.run.smithi016.stdout:Extracting ceph user uid/gid from container image... 2024-04-22T15:39:14.459 INFO:teuthology.orchestra.run.smithi016.stdout:stat: stdout 167 167 2024-04-22T15:39:14.459 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial keys... 2024-04-22T15:39:15.374 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQCihCZmuP6wLhAA9c195kZ3g6w4sKU92cG9Rw== 2024-04-22T15:39:16.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQCjhCZmp1FOLBAAAxm0ljADcXLvGZM3FP497g== 2024-04-22T15:39:17.355 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQCkhCZmF+eIJxAAM3+ZMW0FyoQE5H+YPpr31A== 2024-04-22T15:39:17.356 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial monmap... 2024-04-22T15:39:18.249 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T15:39:18.249 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-04-22T15:39:18.249 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T15:39:18.249 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T15:39:18.249 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool for smithi016 [v2:172.21.15.16:3300,v1:172.21.15.16:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T15:39:18.250 INFO:teuthology.orchestra.run.smithi016.stdout:setting min_mon_release = pacific 2024-04-22T15:39:18.250 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: set fsid to 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T15:39:18.250 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T15:39:18.250 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:39:18.250 INFO:teuthology.orchestra.run.smithi016.stdout:Creating mon... 2024-04-22T15:39:19.209 INFO:teuthology.orchestra.run.smithi016.stdout:create mon.smithi016 on 2024-04-22T15:39:19.418 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-22T15:39:19.559 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-22T15:39:19.765 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-67ab3a80-00be-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-67ab3a80-00be-11ef-bc93-c7b262605968.target. 2024-04-22T15:39:19.765 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-67ab3a80-00be-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-67ab3a80-00be-11ef-bc93-c7b262605968.target. 2024-04-22T15:39:20.054 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-67ab3a80-00be-11ef-bc93-c7b262605968@mon.smithi016 2024-04-22T15:39:20.054 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to reset failed state of unit ceph-67ab3a80-00be-11ef-bc93-c7b262605968@mon.smithi016.service: Unit ceph-67ab3a80-00be-11ef-bc93-c7b262605968@mon.smithi016.service not loaded. 2024-04-22T15:39:20.215 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-67ab3a80-00be-11ef-bc93-c7b262605968.target.wants/ceph-67ab3a80-00be-11ef-bc93-c7b262605968@mon.smithi016.service → /etc/systemd/system/ceph-67ab3a80-00be-11ef-bc93-c7b262605968@.service. 2024-04-22T15:39:20.731 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:20 smithi016 podman[26230]: 2024-04-22 15:39:20.687103298 +0000 UTC m=+0.332841248 container start b7b7f5125c2d67d717192c290a9df1871215cc7f0edb1215a9a0cffb23595dad (image=quay.io/ceph/ceph:v18.2.1, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_CLEAN=True, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-22T15:39:20.774 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T15:39:20.774 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-04-22T15:39:20.781 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T15:39:20.781 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-04-22T15:39:20.781 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-04-22T15:39:20.781 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to enable service . firewalld.service is not available 2024-04-22T15:39:20.782 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mon to start... 2024-04-22T15:39:20.782 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mon... 2024-04-22T15:39:21.001 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:20 smithi016 ceph-mon[26249]: mkfs 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T15:39:21.001 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:20 smithi016 ceph-mon[26249]: mon.smithi016 is new leader, mons smithi016 in quorum (ranks 0) 2024-04-22T15:39:21.001 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:20 smithi016 bash[26230]: b7b7f5125c2d67d717192c290a9df1871215cc7f0edb1215a9a0cffb23595dad 2024-04-22T15:39:21.001 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:20 smithi016 systemd[1]: Started Ceph mon.smithi016 for 67ab3a80-00be-11ef-bc93-c7b262605968. 2024-04-22T15:39:21.817 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout cluster: 2024-04-22T15:39:21.818 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout id: 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T15:39:21.818 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-22T15:39:21.818 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-22T15:39:21.818 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout services: 2024-04-22T15:39:21.818 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi016 (age 0.550304s) 2024-04-22T15:39:21.818 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-22T15:39:21.818 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-22T15:39:21.818 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-22T15:39:21.818 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout data: 2024-04-22T15:39:21.818 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-22T15:39:21.818 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-22T15:39:21.818 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-22T15:39:21.818 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout pgs: 2024-04-22T15:39:21.818 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-22T15:39:21.818 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.195+0000 7f6c731d6700 1 Processor -- start 2024-04-22T15:39:21.819 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.195+0000 7f6c731d6700 1 -- start start 2024-04-22T15:39:21.819 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.195+0000 7f6c731d6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c6c106580 0x7f6c6c1069a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:21.819 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.195+0000 7f6c731d6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c106f70 con 0x7f6c6c106580 2024-04-22T15:39:21.819 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.195+0000 7f6c70f72700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c6c106580 0x7f6c6c1069a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:21.819 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.195+0000 7f6c70f72700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c6c106580 0x7f6c6c1069a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42852/0 (socket says 172.21.15.16:42852) 2024-04-22T15:39:21.819 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.195+0000 7f6c70f72700 1 -- 172.21.15.16:0/2999453628 learned_addr learned my addr 172.21.15.16:0/2999453628 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:21.819 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.196+0000 7f6c70f72700 1 -- 172.21.15.16:0/2999453628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c6c107780 con 0x7f6c6c106580 2024-04-22T15:39:21.819 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.196+0000 7f6c70f72700 1 --2- 172.21.15.16:0/2999453628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c6c106580 0x7f6c6c1069a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6c600098a0 tx=0x7f6c60009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8539de971fde271 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:21.819 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.196+0000 7f6c6b7fe700 1 -- 172.21.15.16:0/2999453628 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c60004180 con 0x7f6c6c106580 2024-04-22T15:39:21.819 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.196+0000 7f6c6b7fe700 1 -- 172.21.15.16:0/2999453628 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c600042e0 con 0x7f6c6c106580 2024-04-22T15:39:21.819 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.196+0000 7f6c6b7fe700 1 -- 172.21.15.16:0/2999453628 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c600036a0 con 0x7f6c6c106580 2024-04-22T15:39:21.819 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.197+0000 7f6c731d6700 1 -- 172.21.15.16:0/2999453628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c6c106580 msgr2=0x7f6c6c1069a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:21.819 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.197+0000 7f6c731d6700 1 --2- 172.21.15.16:0/2999453628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c6c106580 0x7f6c6c1069a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6c600098a0 tx=0x7f6c60009bb0 comp rx=0 tx=0).stop 2024-04-22T15:39:21.819 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.197+0000 7f6c731d6700 1 -- 172.21.15.16:0/2999453628 shutdown_connections 2024-04-22T15:39:21.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.197+0000 7f6c731d6700 1 --2- 172.21.15.16:0/2999453628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c6c106580 0x7f6c6c1069a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:21.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.197+0000 7f6c731d6700 1 -- 172.21.15.16:0/2999453628 >> 172.21.15.16:0/2999453628 conn(0x7f6c6c101b00 msgr2=0x7f6c6c103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:21.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.197+0000 7f6c731d6700 1 -- 172.21.15.16:0/2999453628 shutdown_connections 2024-04-22T15:39:21.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.197+0000 7f6c731d6700 1 -- 172.21.15.16:0/2999453628 wait complete. 2024-04-22T15:39:21.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.197+0000 7f6c731d6700 1 Processor -- start 2024-04-22T15:39:21.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.197+0000 7f6c731d6700 1 -- start start 2024-04-22T15:39:21.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.197+0000 7f6c731d6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c6c106580 0x7f6c6c07bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:21.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.198+0000 7f6c731d6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c07c170 con 0x7f6c6c106580 2024-04-22T15:39:21.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.198+0000 7f6c70f72700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c6c106580 0x7f6c6c07bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:21.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.198+0000 7f6c70f72700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c6c106580 0x7f6c6c07bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42860/0 (socket says 172.21.15.16:42860) 2024-04-22T15:39:21.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.198+0000 7f6c70f72700 1 -- 172.21.15.16:0/4185786568 learned_addr learned my addr 172.21.15.16:0/4185786568 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:21.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.198+0000 7f6c70f72700 1 -- 172.21.15.16:0/4185786568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c60009550 con 0x7f6c6c106580 2024-04-22T15:39:21.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.198+0000 7f6c70f72700 1 --2- 172.21.15.16:0/4185786568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c6c106580 0x7f6c6c07bc30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6c60004000 tx=0x7f6c60004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:21.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.198+0000 7f6c69ffb700 1 -- 172.21.15.16:0/4185786568 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c60003ce0 con 0x7f6c6c106580 2024-04-22T15:39:21.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.198+0000 7f6c69ffb700 1 -- 172.21.15.16:0/4185786568 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c60020070 con 0x7f6c6c106580 2024-04-22T15:39:21.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.198+0000 7f6c69ffb700 1 -- 172.21.15.16:0/4185786568 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c60011630 con 0x7f6c6c106580 2024-04-22T15:39:21.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.198+0000 7f6c731d6700 1 -- 172.21.15.16:0/4185786568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c6c07a280 con 0x7f6c6c106580 2024-04-22T15:39:21.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.199+0000 7f6c731d6700 1 -- 172.21.15.16:0/4185786568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c6c07a720 con 0x7f6c6c106580 2024-04-22T15:39:21.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.199+0000 7f6c69ffb700 1 -- 172.21.15.16:0/4185786568 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6c60022030 con 0x7f6c6c106580 2024-04-22T15:39:21.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.199+0000 7f6c69ffb700 1 -- 172.21.15.16:0/4185786568 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6c60025d40 con 0x7f6c6c106580 2024-04-22T15:39:21.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.199+0000 7f6c731d6700 1 -- 172.21.15.16:0/4185786568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c6c0623c0 con 0x7f6c6c106580 2024-04-22T15:39:21.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.202+0000 7f6c69ffb700 1 -- 172.21.15.16:0/4185786568 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f6c60025480 con 0x7f6c6c106580 2024-04-22T15:39:21.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.275+0000 7f6c731d6700 1 -- 172.21.15.16:0/4185786568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f6c6c192e10 con 0x7f6c6c106580 2024-04-22T15:39:21.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.275+0000 7f6c69ffb700 1 -- 172.21.15.16:0/4185786568 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f6c60011c20 con 0x7f6c6c106580 2024-04-22T15:39:21.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.276+0000 7f6c731d6700 1 -- 172.21.15.16:0/4185786568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c6c106580 msgr2=0x7f6c6c07bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:21.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.276+0000 7f6c731d6700 1 --2- 172.21.15.16:0/4185786568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c6c106580 0x7f6c6c07bc30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6c60004000 tx=0x7f6c60004750 comp rx=0 tx=0).stop 2024-04-22T15:39:21.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.276+0000 7f6c731d6700 1 -- 172.21.15.16:0/4185786568 shutdown_connections 2024-04-22T15:39:21.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.276+0000 7f6c731d6700 1 --2- 172.21.15.16:0/4185786568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c6c106580 0x7f6c6c07bc30 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:21.822 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.276+0000 7f6c731d6700 1 -- 172.21.15.16:0/4185786568 >> 172.21.15.16:0/4185786568 conn(0x7f6c6c101b00 msgr2=0x7f6c6c18df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:21.822 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.276+0000 7f6c731d6700 1 -- 172.21.15.16:0/4185786568 shutdown_connections 2024-04-22T15:39:21.822 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:21.276+0000 7f6c731d6700 1 -- 172.21.15.16:0/4185786568 wait complete. 2024-04-22T15:39:21.822 INFO:teuthology.orchestra.run.smithi016.stdout:mon is available 2024-04-22T15:39:21.822 INFO:teuthology.orchestra.run.smithi016.stdout:Assimilating anything we can from ceph.conf... 2024-04-22T15:39:21.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:21 smithi016 ceph-mon[26249]: mon.smithi016 is new leader, mons smithi016 in quorum (ranks 0) 2024-04-22T15:39:21.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:21 smithi016 ceph-mon[26249]: monmap e1: 1 mons at {smithi016=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-22T15:39:21.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:21 smithi016 ceph-mon[26249]: fsmap 2024-04-22T15:39:21.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:21 smithi016 ceph-mon[26249]: osdmap e1: 0 total, 0 up, 0 in 2024-04-22T15:39:21.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:21 smithi016 ceph-mon[26249]: mgrmap e1: no daemons active 2024-04-22T15:39:21.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:21 smithi016 ceph-mon[26249]: from='client.? 172.21.15.16:0/4185786568' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-22T15:39:22.908 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:22 smithi016 ceph-mon[26249]: from='client.? 172.21.15.16:0/3136073629' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-22T15:39:22.908 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:22 smithi016 ceph-mon[26249]: from='client.? 172.21.15.16:0/3136073629' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-22T15:39:22.917 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-22T15:39:22.917 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [global] 2024-04-22T15:39:22.917 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout fsid = 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T15:39:22.917 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.16:3300,v1:172.21.15.16:6789] 2024-04-22T15:39:22.917 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-22T15:39:22.917 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-22T15:39:22.917 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-22T15:39:22.917 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-22T15:39:22.917 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-22T15:39:22.917 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-22T15:39:22.917 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-22T15:39:22.917 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-22T15:39:22.918 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [osd] 2024-04-22T15:39:22.918 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-22T15:39:22.918 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-22T15:39:22.918 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-22T15:39:22.918 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-22T15:39:22.918 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-22T15:39:22.918 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.246+0000 7fa7b7fbd700 1 Processor -- start 2024-04-22T15:39:22.918 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.246+0000 7fa7b7fbd700 1 -- start start 2024-04-22T15:39:22.918 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.246+0000 7fa7b7fbd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7b0105470 0x7fa7b0105890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:22.918 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.246+0000 7fa7b7fbd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b0105e60 con 0x7fa7b0105470 2024-04-22T15:39:22.918 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.246+0000 7fa7b5d59700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7b0105470 0x7fa7b0105890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:22.918 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.246+0000 7fa7b5d59700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7b0105470 0x7fa7b0105890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44958/0 (socket says 172.21.15.16:44958) 2024-04-22T15:39:22.918 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.246+0000 7fa7b5d59700 1 -- 172.21.15.16:0/2341608668 learned_addr learned my addr 172.21.15.16:0/2341608668 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:22.919 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.246+0000 7fa7b5d59700 1 -- 172.21.15.16:0/2341608668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b0106670 con 0x7fa7b0105470 2024-04-22T15:39:22.919 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.247+0000 7fa7b5d59700 1 --2- 172.21.15.16:0/2341608668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7b0105470 0x7fa7b0105890 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa7a00098a0 tx=0x7fa7a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a3a6206750b0c87 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:22.919 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.247+0000 7fa7b4d57700 1 -- 172.21.15.16:0/2341608668 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7a0004180 con 0x7fa7b0105470 2024-04-22T15:39:22.919 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.247+0000 7fa7b4d57700 1 -- 172.21.15.16:0/2341608668 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a00042e0 con 0x7fa7b0105470 2024-04-22T15:39:22.919 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.247+0000 7fa7b4d57700 1 -- 172.21.15.16:0/2341608668 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7a00036a0 con 0x7fa7b0105470 2024-04-22T15:39:22.919 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.247+0000 7fa7b7fbd700 1 -- 172.21.15.16:0/2341608668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7b0105470 msgr2=0x7fa7b0105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:22.919 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.247+0000 7fa7b7fbd700 1 --2- 172.21.15.16:0/2341608668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7b0105470 0x7fa7b0105890 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa7a00098a0 tx=0x7fa7a0009bb0 comp rx=0 tx=0).stop 2024-04-22T15:39:22.919 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.247+0000 7fa7b7fbd700 1 -- 172.21.15.16:0/2341608668 shutdown_connections 2024-04-22T15:39:22.919 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.247+0000 7fa7b7fbd700 1 --2- 172.21.15.16:0/2341608668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7b0105470 0x7fa7b0105890 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:22.919 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.247+0000 7fa7b7fbd700 1 -- 172.21.15.16:0/2341608668 >> 172.21.15.16:0/2341608668 conn(0x7fa7b01009f0 msgr2=0x7fa7b0102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:22.919 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.247+0000 7fa7b7fbd700 1 -- 172.21.15.16:0/2341608668 shutdown_connections 2024-04-22T15:39:22.919 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.247+0000 7fa7b7fbd700 1 -- 172.21.15.16:0/2341608668 wait complete. 2024-04-22T15:39:22.919 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.248+0000 7fa7b7fbd700 1 Processor -- start 2024-04-22T15:39:22.920 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.248+0000 7fa7b7fbd700 1 -- start start 2024-04-22T15:39:22.920 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.248+0000 7fa7b7fbd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7b0105470 0x7fa7b01999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:22.920 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.248+0000 7fa7b7fbd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b0199f10 con 0x7fa7b0105470 2024-04-22T15:39:22.920 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.248+0000 7fa7b5d59700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7b0105470 0x7fa7b01999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:22.920 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.248+0000 7fa7b5d59700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7b0105470 0x7fa7b01999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44964/0 (socket says 172.21.15.16:44964) 2024-04-22T15:39:22.920 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.248+0000 7fa7b5d59700 1 -- 172.21.15.16:0/3136073629 learned_addr learned my addr 172.21.15.16:0/3136073629 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:22.920 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.249+0000 7fa7b5d59700 1 -- 172.21.15.16:0/3136073629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a0009550 con 0x7fa7b0105470 2024-04-22T15:39:22.920 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.249+0000 7fa7b5d59700 1 --2- 172.21.15.16:0/3136073629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7b0105470 0x7fa7b01999d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0004000 tx=0x7fa7a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:22.920 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.249+0000 7fa7a6ffd700 1 -- 172.21.15.16:0/3136073629 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7a0003ce0 con 0x7fa7b0105470 2024-04-22T15:39:22.920 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.249+0000 7fa7a6ffd700 1 -- 172.21.15.16:0/3136073629 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a0020070 con 0x7fa7b0105470 2024-04-22T15:39:22.920 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.249+0000 7fa7b7fbd700 1 -- 172.21.15.16:0/3136073629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b019a110 con 0x7fa7b0105470 2024-04-22T15:39:22.920 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.249+0000 7fa7a6ffd700 1 -- 172.21.15.16:0/3136073629 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7a0011630 con 0x7fa7b0105470 2024-04-22T15:39:22.921 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.249+0000 7fa7b7fbd700 1 -- 172.21.15.16:0/3136073629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b019a5b0 con 0x7fa7b0105470 2024-04-22T15:39:22.921 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.249+0000 7fa7a6ffd700 1 -- 172.21.15.16:0/3136073629 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa7a0022030 con 0x7fa7b0105470 2024-04-22T15:39:22.921 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.249+0000 7fa7a6ffd700 1 -- 172.21.15.16:0/3136073629 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa7a0025d00 con 0x7fa7b0105470 2024-04-22T15:39:22.921 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.250+0000 7fa7b7fbd700 1 -- 172.21.15.16:0/3136073629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7b0193290 con 0x7fa7b0105470 2024-04-22T15:39:22.921 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.252+0000 7fa7a6ffd700 1 -- 172.21.15.16:0/3136073629 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fa7a0025480 con 0x7fa7b0105470 2024-04-22T15:39:22.921 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.323+0000 7fa7b7fbd700 1 -- 172.21.15.16:0/3136073629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fa7b002cf20 con 0x7fa7b0105470 2024-04-22T15:39:22.921 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.326+0000 7fa7a6ffd700 1 -- 172.21.15.16:0/3136073629 <== mon.0 v2:172.21.15.16:3300/0 7 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa7a0011c20 con 0x7fa7b0105470 2024-04-22T15:39:22.921 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.326+0000 7fa7a6ffd700 1 -- 172.21.15.16:0/3136073629 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fa7a0038590 con 0x7fa7b0105470 2024-04-22T15:39:22.921 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.327+0000 7fa7b7fbd700 1 -- 172.21.15.16:0/3136073629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7b0105470 msgr2=0x7fa7b01999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:22.921 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.327+0000 7fa7b7fbd700 1 --2- 172.21.15.16:0/3136073629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7b0105470 0x7fa7b01999d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0004000 tx=0x7fa7a0004750 comp rx=0 tx=0).stop 2024-04-22T15:39:22.921 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.327+0000 7fa7b7fbd700 1 -- 172.21.15.16:0/3136073629 shutdown_connections 2024-04-22T15:39:22.921 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.327+0000 7fa7b7fbd700 1 --2- 172.21.15.16:0/3136073629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7b0105470 0x7fa7b01999d0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:22.921 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.327+0000 7fa7b7fbd700 1 -- 172.21.15.16:0/3136073629 >> 172.21.15.16:0/3136073629 conn(0x7fa7b01009f0 msgr2=0x7fa7b0074030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:22.922 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.327+0000 7fa7b7fbd700 1 -- 172.21.15.16:0/3136073629 shutdown_connections 2024-04-22T15:39:22.922 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:22.327+0000 7fa7b7fbd700 1 -- 172.21.15.16:0/3136073629 wait complete. 2024-04-22T15:39:22.922 INFO:teuthology.orchestra.run.smithi016.stdout:Generating new minimal ceph.conf... 2024-04-22T15:39:23.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.310+0000 7f6b4cac2700 1 Processor -- start 2024-04-22T15:39:23.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.310+0000 7f6b4cac2700 1 -- start start 2024-04-22T15:39:23.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.310+0000 7f6b4cac2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4807be60 0x7f6b4807a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:23.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.310+0000 7f6b4cac2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4807a890 con 0x7f6b4807be60 2024-04-22T15:39:23.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.310+0000 7f6b4659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4807be60 0x7f6b4807a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:23.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.310+0000 7f6b4659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4807be60 0x7f6b4807a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44966/0 (socket says 172.21.15.16:44966) 2024-04-22T15:39:23.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.310+0000 7f6b4659c700 1 -- 172.21.15.16:0/1250028193 learned_addr learned my addr 172.21.15.16:0/1250028193 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:23.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.310+0000 7f6b4659c700 1 -- 172.21.15.16:0/1250028193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b4807a9d0 con 0x7f6b4807be60 2024-04-22T15:39:23.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.311+0000 7f6b4659c700 1 --2- 172.21.15.16:0/1250028193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4807be60 0x7f6b4807a2c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6b38009e80 tx=0x7f6b3800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cefd69418220563e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:23.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.311+0000 7f6b4559a700 1 -- 172.21.15.16:0/1250028193 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b3800e070 con 0x7f6b4807be60 2024-04-22T15:39:23.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.311+0000 7f6b4559a700 1 -- 172.21.15.16:0/1250028193 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6b38003e90 con 0x7f6b4807be60 2024-04-22T15:39:23.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.311+0000 7f6b4559a700 1 -- 172.21.15.16:0/1250028193 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b38004380 con 0x7f6b4807be60 2024-04-22T15:39:23.960 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.311+0000 7f6b4cac2700 1 -- 172.21.15.16:0/1250028193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4807be60 msgr2=0x7f6b4807a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:23.960 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.311+0000 7f6b4cac2700 1 --2- 172.21.15.16:0/1250028193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4807be60 0x7f6b4807a2c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6b38009e80 tx=0x7f6b3800b330 comp rx=0 tx=0).stop 2024-04-22T15:39:23.960 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.311+0000 7f6b4cac2700 1 -- 172.21.15.16:0/1250028193 shutdown_connections 2024-04-22T15:39:23.960 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.311+0000 7f6b4cac2700 1 --2- 172.21.15.16:0/1250028193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4807be60 0x7f6b4807a2c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:23.960 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.311+0000 7f6b4cac2700 1 -- 172.21.15.16:0/1250028193 >> 172.21.15.16:0/1250028193 conn(0x7f6b48101b40 msgr2=0x7f6b48103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:23.960 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.312+0000 7f6b4cac2700 1 -- 172.21.15.16:0/1250028193 shutdown_connections 2024-04-22T15:39:23.960 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.312+0000 7f6b4cac2700 1 -- 172.21.15.16:0/1250028193 wait complete. 2024-04-22T15:39:23.960 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.312+0000 7f6b4cac2700 1 Processor -- start 2024-04-22T15:39:23.960 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.312+0000 7f6b4cac2700 1 -- start start 2024-04-22T15:39:23.960 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.312+0000 7f6b4cac2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4807be60 0x7f6b4819c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:23.960 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.312+0000 7f6b4cac2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4819c8b0 con 0x7f6b4807be60 2024-04-22T15:39:23.960 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.313+0000 7f6b4659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4807be60 0x7f6b4819c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:23.960 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.313+0000 7f6b4659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4807be60 0x7f6b4819c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44982/0 (socket says 172.21.15.16:44982) 2024-04-22T15:39:23.961 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.313+0000 7f6b4659c700 1 -- 172.21.15.16:0/2161230586 learned_addr learned my addr 172.21.15.16:0/2161230586 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:23.961 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.313+0000 7f6b4659c700 1 -- 172.21.15.16:0/2161230586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b38009b30 con 0x7f6b4807be60 2024-04-22T15:39:23.961 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.313+0000 7f6b4659c700 1 --2- 172.21.15.16:0/2161230586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4807be60 0x7f6b4819c370 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6b3800b7b0 tx=0x7f6b38004920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:23.961 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.313+0000 7f6b3f7fe700 1 -- 172.21.15.16:0/2161230586 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b3800e040 con 0x7f6b4807be60 2024-04-22T15:39:23.961 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.313+0000 7f6b3f7fe700 1 -- 172.21.15.16:0/2161230586 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6b38003c30 con 0x7f6b4807be60 2024-04-22T15:39:23.961 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.313+0000 7f6b4cac2700 1 -- 172.21.15.16:0/2161230586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b4819cab0 con 0x7f6b4807be60 2024-04-22T15:39:23.961 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.313+0000 7f6b3f7fe700 1 -- 172.21.15.16:0/2161230586 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b3801b740 con 0x7f6b4807be60 2024-04-22T15:39:23.961 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.313+0000 7f6b4cac2700 1 -- 172.21.15.16:0/2161230586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b4819cfb0 con 0x7f6b4807be60 2024-04-22T15:39:23.961 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.314+0000 7f6b3f7fe700 1 -- 172.21.15.16:0/2161230586 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6b38019040 con 0x7f6b4807be60 2024-04-22T15:39:23.961 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.314+0000 7f6b3f7fe700 1 -- 172.21.15.16:0/2161230586 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6b3801a6d0 con 0x7f6b4807be60 2024-04-22T15:39:23.962 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.314+0000 7f6b4cac2700 1 -- 172.21.15.16:0/2161230586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b4810c8e0 con 0x7f6b4807be60 2024-04-22T15:39:23.962 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.316+0000 7f6b3f7fe700 1 -- 172.21.15.16:0/2161230586 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f6b38029060 con 0x7f6b4807be60 2024-04-22T15:39:23.962 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.386+0000 7f6b4cac2700 1 -- 172.21.15.16:0/2161230586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f6b4810caa0 con 0x7f6b4807be60 2024-04-22T15:39:23.962 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.386+0000 7f6b3f7fe700 1 -- 172.21.15.16:0/2161230586 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f6b3801f070 con 0x7f6b4807be60 2024-04-22T15:39:23.962 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.387+0000 7f6b4cac2700 1 -- 172.21.15.16:0/2161230586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4807be60 msgr2=0x7f6b4819c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:23.962 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.387+0000 7f6b4cac2700 1 --2- 172.21.15.16:0/2161230586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4807be60 0x7f6b4819c370 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6b3800b7b0 tx=0x7f6b38004920 comp rx=0 tx=0).stop 2024-04-22T15:39:23.962 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.387+0000 7f6b4cac2700 1 -- 172.21.15.16:0/2161230586 shutdown_connections 2024-04-22T15:39:23.963 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.387+0000 7f6b4cac2700 1 --2- 172.21.15.16:0/2161230586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4807be60 0x7f6b4819c370 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:23.963 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.387+0000 7f6b4cac2700 1 -- 172.21.15.16:0/2161230586 >> 172.21.15.16:0/2161230586 conn(0x7f6b48101b40 msgr2=0x7f6b481028a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:23.963 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.387+0000 7f6b4cac2700 1 -- 172.21.15.16:0/2161230586 shutdown_connections 2024-04-22T15:39:23.963 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:23.387+0000 7f6b4cac2700 1 -- 172.21.15.16:0/2161230586 wait complete. 2024-04-22T15:39:23.963 INFO:teuthology.orchestra.run.smithi016.stdout:Restarting the monitor... 2024-04-22T15:39:24.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:23 smithi016 ceph-mon[26249]: from='client.? 172.21.15.16:0/2161230586' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:39:24.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:23 smithi016 systemd[1]: Stopping Ceph mon.smithi016 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T15:39:24.510 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:24 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016[26244]: 2024-04-22T15:39:24.251+0000 7fd718ad5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi016 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T15:39:24.511 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:24 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016[26244]: 2024-04-22T15:39:24.251+0000 7fd718ad5700 -1 mon.smithi016@0(leader) e1 *** Got Signal Terminated *** 2024-04-22T15:39:24.797 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:24 smithi016 podman[26494]: 2024-04-22 15:39:24.508297801 +0000 UTC m=+0.389595429 container died b7b7f5125c2d67d717192c290a9df1871215cc7f0edb1215a9a0cffb23595dad (image=quay.io/ceph/ceph:v18.2.1, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T15:39:24.797 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:24 smithi016 podman[26494]: 2024-04-22 15:39:24.794664442 +0000 UTC m=+0.675962070 container cleanup b7b7f5125c2d67d717192c290a9df1871215cc7f0edb1215a9a0cffb23595dad (image=quay.io/ceph/ceph:v18.2.1, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=HEAD, ceph=True) 2024-04-22T15:39:25.092 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:24 smithi016 bash[26494]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016 2024-04-22T15:39:25.092 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:24 smithi016 podman[26508]: 2024-04-22 15:39:24.978917247 +0000 UTC m=+0.466812684 container remove b7b7f5125c2d67d717192c290a9df1871215cc7f0edb1215a9a0cffb23595dad (image=quay.io/ceph/ceph:v18.2.1, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T15:39:25.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 systemd[1]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968@mon.smithi016.service: Deactivated successfully. 2024-04-22T15:39:25.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 systemd[1]: Stopped Ceph mon.smithi016 for 67ab3a80-00be-11ef-bc93-c7b262605968. 2024-04-22T15:39:25.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 systemd[1]: Starting Ceph mon.smithi016 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T15:39:25.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 podman[26601]: 2024-04-22 15:39:25.232035463 +0000 UTC m=+0.019240611 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph:v18.2.1 2024-04-22T15:39:25.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 podman[26601]: 2024-04-22 15:39:25.329834519 +0000 UTC m=+0.117039697 container create 72af40873cb62540258a2e042b4224fc5ef49c8669e8586975f9575603a75acf (image=quay.io/ceph/ceph:v18.2.1, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-22T15:39:25.564 INFO:teuthology.orchestra.run.smithi016.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-22T15:39:25.688 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 podman[26601]: 2024-04-22 15:39:25.507200477 +0000 UTC m=+0.294405609 container init 72af40873cb62540258a2e042b4224fc5ef49c8669e8586975f9575603a75acf (image=quay.io/ceph/ceph:v18.2.1, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.vendor=CentOS) 2024-04-22T15:39:25.688 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 podman[26601]: 2024-04-22 15:39:25.510916357 +0000 UTC m=+0.298121490 container start 72af40873cb62540258a2e042b4224fc5ef49c8669e8586975f9575603a75acf (image=quay.io/ceph/ceph:v18.2.1, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1) 2024-04-22T15:39:25.688 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T15:39:25.688 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable), process ceph-mon, pid 2 2024-04-22T15:39:25.688 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: pidfile_write: ignore empty --pid-file 2024-04-22T15:39:25.688 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: load: jerasure load: lrc 2024-04-22T15:39:25.689 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: RocksDB version: 7.9.2 2024-04-22T15:39:25.689 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Git sha 0 2024-04-22T15:39:25.689 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Compile date 2023-12-11 22:07:34 2024-04-22T15:39:25.689 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: DB SUMMARY 2024-04-22T15:39:25.689 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: DB Session ID: RWQYRP8IINLJ8BCOL9LI 2024-04-22T15:39:25.689 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: CURRENT file: CURRENT 2024-04-22T15:39:25.689 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T15:39:25.689 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-22T15:39:25.689 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi016/store.db dir, Total Num: 1, files: 000008.sst 2024-04-22T15:39:25.689 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi016/store.db: 000009.log size: 98361 ; 2024-04-22T15:39:25.689 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.error_if_exists: 0 2024-04-22T15:39:25.689 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.create_if_missing: 0 2024-04-22T15:39:25.689 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.paranoid_checks: 1 2024-04-22T15:39:25.689 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-22T15:39:25.690 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T15:39:25.690 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-22T15:39:25.690 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.env: 0x55dec874a720 2024-04-22T15:39:25.690 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.fs: PosixFileSystem 2024-04-22T15:39:25.690 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.info_log: 0x55decada92e0 2024-04-22T15:39:25.690 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T15:39:25.690 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.statistics: (nil) 2024-04-22T15:39:25.690 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.use_fsync: 0 2024-04-22T15:39:25.690 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_log_file_size: 0 2024-04-22T15:39:25.690 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T15:39:25.691 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T15:39:25.691 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T15:39:25.691 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T15:39:25.691 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.allow_fallocate: 1 2024-04-22T15:39:25.691 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T15:39:25.691 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T15:39:25.691 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.use_direct_reads: 0 2024-04-22T15:39:25.691 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T15:39:25.691 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T15:39:25.691 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.db_log_dir: 2024-04-22T15:39:25.691 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.wal_dir: 2024-04-22T15:39:25.691 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T15:39:25.691 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T15:39:25.691 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T15:39:25.692 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T15:39:25.692 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T15:39:25.692 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T15:39:25.692 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T15:39:25.692 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T15:39:25.692 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.write_buffer_manager: 0x55deca01a140 2024-04-22T15:39:25.692 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T15:39:25.692 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T15:39:25.692 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T15:39:25.692 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T15:39:25.692 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T15:39:25.692 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T15:39:25.692 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T15:39:25.692 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T15:39:25.693 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.unordered_write: 0 2024-04-22T15:39:25.693 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T15:39:25.693 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T15:39:25.693 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T15:39:25.693 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T15:39:25.693 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.row_cache: None 2024-04-22T15:39:25.693 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.wal_filter: None 2024-04-22T15:39:25.693 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T15:39:25.693 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T15:39:25.694 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.two_write_queues: 0 2024-04-22T15:39:25.694 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T15:39:25.695 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.wal_compression: 0 2024-04-22T15:39:25.695 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.atomic_flush: 0 2024-04-22T15:39:25.695 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T15:39:25.695 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T15:39:25.695 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T15:39:25.695 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.log_readahead_size: 0 2024-04-22T15:39:25.695 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T15:39:25.695 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T15:39:25.695 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T15:39:25.695 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T15:39:25.695 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T15:39:25.695 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T15:39:25.695 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-22T15:39:25.696 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_background_jobs: 2 2024-04-22T15:39:25.696 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_background_compactions: -1 2024-04-22T15:39:25.696 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_subcompactions: 1 2024-04-22T15:39:25.696 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T15:39:25.696 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T15:39:25.696 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T15:39:25.696 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T15:39:25.696 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T15:39:25.696 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T15:39:25.696 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T15:39:25.696 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T15:39:25.696 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_open_files: -1 2024-04-22T15:39:25.696 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T15:39:25.697 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T15:39:25.697 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T15:39:25.697 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T15:39:25.697 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_background_flushes: -1 2024-04-22T15:39:25.697 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Compression algorithms supported: 2024-04-22T15:39:25.697 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T15:39:25.697 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: kZSTD supported: 0 2024-04-22T15:39:25.697 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: kXpressCompression supported: 0 2024-04-22T15:39:25.697 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T15:39:25.697 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: kZlibCompression supported: 1 2024-04-22T15:39:25.697 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: kSnappyCompression supported: 1 2024-04-22T15:39:25.697 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: kLZ4Compression supported: 1 2024-04-22T15:39:25.697 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: kBZip2Compression supported: 0 2024-04-22T15:39:25.697 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T15:39:25.698 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-22T15:39:25.698 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi016/store.db/MANIFEST-000010 2024-04-22T15:39:25.698 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-22T15:39:25.698 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T15:39:25.698 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.merge_operator: 2024-04-22T15:39:25.698 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.compaction_filter: None 2024-04-22T15:39:25.698 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T15:39:25.698 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T15:39:25.698 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T15:39:25.698 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T15:39:25.698 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55decada93e0) 2024-04-22T15:39:25.698 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: cache_index_and_filter_blocks: 1 2024-04-22T15:39:25.698 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T15:39:25.698 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T15:39:25.699 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: pin_top_level_index_and_filter: 1 2024-04-22T15:39:25.699 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: index_type: 0 2024-04-22T15:39:25.699 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: data_block_index_type: 0 2024-04-22T15:39:25.699 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: index_shortening: 1 2024-04-22T15:39:25.699 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T15:39:25.699 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: checksum: 4 2024-04-22T15:39:25.699 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: no_block_cache: 0 2024-04-22T15:39:25.699 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: block_cache: 0x55deca0bb350 2024-04-22T15:39:25.699 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: block_cache_name: BinnedLRUCache 2024-04-22T15:39:25.699 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: block_cache_options: 2024-04-22T15:39:25.699 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: capacity : 536870912 2024-04-22T15:39:25.699 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: num_shard_bits : 4 2024-04-22T15:39:25.699 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: strict_capacity_limit : 0 2024-04-22T15:39:25.699 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: high_pri_pool_ratio: 0.000 2024-04-22T15:39:25.699 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: block_cache_compressed: (nil) 2024-04-22T15:39:25.700 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: persistent_cache: (nil) 2024-04-22T15:39:25.700 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: block_size: 4096 2024-04-22T15:39:25.700 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: block_size_deviation: 10 2024-04-22T15:39:25.700 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: block_restart_interval: 16 2024-04-22T15:39:25.700 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: index_block_restart_interval: 1 2024-04-22T15:39:25.700 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: metadata_block_size: 4096 2024-04-22T15:39:25.700 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: partition_filters: 0 2024-04-22T15:39:25.700 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: use_delta_encoding: 1 2024-04-22T15:39:25.700 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: filter_policy: bloomfilter 2024-04-22T15:39:25.700 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: whole_key_filtering: 1 2024-04-22T15:39:25.700 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: verify_compression: 0 2024-04-22T15:39:25.700 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: read_amp_bytes_per_bit: 0 2024-04-22T15:39:25.700 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: format_version: 5 2024-04-22T15:39:25.700 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: enable_index_compression: 1 2024-04-22T15:39:25.700 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: block_align: 0 2024-04-22T15:39:25.701 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: max_auto_readahead_size: 262144 2024-04-22T15:39:25.701 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: prepopulate_block_cache: 0 2024-04-22T15:39:25.701 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: initial_auto_readahead_size: 8192 2024-04-22T15:39:25.701 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: num_file_reads_for_auto_readahead: 2 2024-04-22T15:39:25.701 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T15:39:25.701 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T15:39:25.701 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.compression: NoCompression 2024-04-22T15:39:25.701 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T15:39:25.701 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T15:39:25.701 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T15:39:25.701 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.num_levels: 7 2024-04-22T15:39:25.701 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T15:39:25.701 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T15:39:25.701 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T15:39:25.701 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T15:39:25.702 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T15:39:25.702 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T15:39:25.702 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T15:39:25.702 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T15:39:25.702 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T15:39:25.702 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T15:39:25.702 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-22T15:39:25.702 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-22T15:39:25.702 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T15:39:25.702 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T15:39:25.702 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T15:39:25.702 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T15:39:25.702 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T15:39:25.703 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-22T15:39:25.703 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T15:39:25.703 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T15:39:25.703 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-22T15:39:25.703 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T15:39:25.703 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T15:39:25.703 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T15:39:25.703 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T15:39:25.703 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T15:39:25.703 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T15:39:25.703 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T15:39:25.703 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T15:39:25.703 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T15:39:25.703 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T15:39:25.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T15:39:25.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T15:39:25.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T15:39:25.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T15:39:25.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T15:39:25.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T15:39:25.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T15:39:25.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-22T15:39:25.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.arena_block_size: 1048576 2024-04-22T15:39:25.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T15:39:25.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T15:39:25.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T15:39:25.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T15:39:25.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T15:39:25.705 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T15:39:25.705 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T15:39:25.705 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T15:39:25.705 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T15:39:25.705 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T15:39:25.705 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T15:39:25.705 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T15:39:25.705 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T15:39:25.705 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.table_properties_collectors: 2024-04-22T15:39:25.705 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.inplace_update_support: 0 2024-04-22T15:39:25.705 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T15:39:25.705 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T15:39:25.705 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T15:39:25.705 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T15:39:25.706 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.bloom_locality: 0 2024-04-22T15:39:25.706 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.max_successive_merges: 0 2024-04-22T15:39:25.706 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T15:39:25.706 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T15:39:25.706 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T15:39:25.706 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T15:39:25.706 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.ttl: 2592000 2024-04-22T15:39:25.706 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T15:39:25.706 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-22T15:39:25.706 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-22T15:39:25.706 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.enable_blob_files: false 2024-04-22T15:39:25.706 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.min_blob_size: 0 2024-04-22T15:39:25.706 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T15:39:25.706 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T15:39:25.707 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T15:39:25.707 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T15:39:25.707 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-22T15:39:25.707 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-22T15:39:25.707 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.blob_file_starting_level: 0 2024-04-22T15:39:25.707 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-22T15:39:25.707 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi016/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-22T15:39:25.707 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-22T15:39:25.707 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b478f51e-83dc-4f3f-a0a5-72396151bec8 2024-04-22T15:39:25.707 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713800365540579, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-22T15:39:25.707 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-22T15:39:25.707 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713800365541649, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93605, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 91627, "index_size": 260, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13576, "raw_average_key_size": 49, "raw_value_size": 84236, "raw_average_value_size": 308, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713800365, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b478f51e-83dc-4f3f-a0a5-72396151bec8", "db_session_id": "RWQYRP8IINLJ8BCOL9LI", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-22T15:39:25.707 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713800365541735, "job": 1, "event": "recovery_finished"} 2024-04-22T15:39:25.707 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-22T15:39:25.708 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi016/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T15:39:25.708 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55deca158000 2024-04-22T15:39:25.708 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: DB pointer 0x55deca144000 2024-04-22T15:39:25.708 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-22T15:39:25.708 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-22T15:39:25.708 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: ** DB Stats ** 2024-04-22T15:39:25.708 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T15:39:25.708 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T15:39:25.708 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T15:39:25.708 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T15:39:25.708 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T15:39:25.708 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T15:39:25.708 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T15:39:25.708 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: 2024-04-22T15:39:25.709 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: ** Compaction Stats [default] ** 2024-04-22T15:39:25.709 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T15:39:25.709 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-22T15:39:25.709 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: L0 2/0 93.28 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T15:39:25.709 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Sum 2/0 93.28 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T15:39:25.709 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T15:39:25.709 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: 2024-04-22T15:39:25.709 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: ** Compaction Stats [default] ** 2024-04-22T15:39:25.709 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T15:39:25.709 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T15:39:25.709 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 137.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T15:39:25.709 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: 2024-04-22T15:39:25.709 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-22T15:39:25.709 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: 2024-04-22T15:39:25.710 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T15:39:25.710 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T15:39:25.710 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T15:39:25.710 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T15:39:25.710 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T15:39:25.710 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T15:39:25.710 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Cumulative compaction: 0.00 GB write, 28.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T15:39:25.710 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Interval compaction: 0.00 GB write, 28.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T15:39:25.710 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T15:39:25.710 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Block cache BinnedLRUCache@0x55deca0bb350#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-22T15:39:25.710 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-04-22T15:39:25.710 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: 2024-04-22T15:39:25.710 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T15:39:25.710 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: starting mon.smithi016 rank 0 at public addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] at bind addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi016 fsid 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T15:39:25.711 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: mon.smithi016@-1(???) e1 preinit fsid 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T15:39:25.711 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: mon.smithi016@-1(???).mds e1 new map 2024-04-22T15:39:25.711 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: mon.smithi016@-1(???).mds e1 print_map 2024-04-22T15:39:25.711 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: e1 2024-04-22T15:39:25.711 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:39:25.711 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:39:25.711 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: legacy client fscid: -1 2024-04-22T15:39:25.711 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: 2024-04-22T15:39:25.711 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: No filesystems configured 2024-04-22T15:39:25.711 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: mon.smithi016@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T15:39:25.711 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: mon.smithi016@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T15:39:25.711 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: mon.smithi016@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T15:39:25.711 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: mon.smithi016@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T15:39:25.711 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: mon.smithi016@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-22T15:39:25.712 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 bash[26601]: 72af40873cb62540258a2e042b4224fc5ef49c8669e8586975f9575603a75acf 2024-04-22T15:39:25.712 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 systemd[1]: Started Ceph mon.smithi016 for 67ab3a80-00be-11ef-bc93-c7b262605968. 2024-04-22T15:39:25.712 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: mon.smithi016 is new leader, mons smithi016 in quorum (ranks 0) 2024-04-22T15:39:25.712 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: monmap e1: 1 mons at {smithi016=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-22T15:39:25.712 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: fsmap 2024-04-22T15:39:25.712 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: osdmap e1: 0 total, 0 up, 0 in 2024-04-22T15:39:25.712 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:25 smithi016 ceph-mon[26620]: mgrmap e1: no daemons active 2024-04-22T15:39:26.626 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.050+0000 7fc4746db700 1 Processor -- start 2024-04-22T15:39:26.626 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.050+0000 7fc4746db700 1 -- start start 2024-04-22T15:39:26.626 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.050+0000 7fc4746db700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc46c105470 0x7fc46c105890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:26.626 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.050+0000 7fc4746db700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46c105e60 con 0x7fc46c105470 2024-04-22T15:39:26.626 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.051+0000 7fc472477700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc46c105470 0x7fc46c105890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:26.626 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.051+0000 7fc472477700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc46c105470 0x7fc46c105890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44998/0 (socket says 172.21.15.16:44998) 2024-04-22T15:39:26.626 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.051+0000 7fc472477700 1 -- 172.21.15.16:0/2207756417 learned_addr learned my addr 172.21.15.16:0/2207756417 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.051+0000 7fc472477700 1 -- 172.21.15.16:0/2207756417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc46c106670 con 0x7fc46c105470 2024-04-22T15:39:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.051+0000 7fc472477700 1 --2- 172.21.15.16:0/2207756417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc46c105470 0x7fc46c105890 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc4680098a0 tx=0x7fc468009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25f03945e4bfc8cd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.051+0000 7fc471475700 1 -- 172.21.15.16:0/2207756417 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc468004180 con 0x7fc46c105470 2024-04-22T15:39:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.051+0000 7fc471475700 1 -- 172.21.15.16:0/2207756417 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc4680042e0 con 0x7fc46c105470 2024-04-22T15:39:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.051+0000 7fc471475700 1 -- 172.21.15.16:0/2207756417 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc468011a50 con 0x7fc46c105470 2024-04-22T15:39:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.052+0000 7fc4746db700 1 -- 172.21.15.16:0/2207756417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc46c105470 msgr2=0x7fc46c105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.052+0000 7fc4746db700 1 --2- 172.21.15.16:0/2207756417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc46c105470 0x7fc46c105890 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc4680098a0 tx=0x7fc468009bb0 comp rx=0 tx=0).stop 2024-04-22T15:39:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.052+0000 7fc4746db700 1 -- 172.21.15.16:0/2207756417 shutdown_connections 2024-04-22T15:39:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.052+0000 7fc4746db700 1 --2- 172.21.15.16:0/2207756417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc46c105470 0x7fc46c105890 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.052+0000 7fc4746db700 1 -- 172.21.15.16:0/2207756417 >> 172.21.15.16:0/2207756417 conn(0x7fc46c1009f0 msgr2=0x7fc46c102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.052+0000 7fc4746db700 1 -- 172.21.15.16:0/2207756417 shutdown_connections 2024-04-22T15:39:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.052+0000 7fc4746db700 1 -- 172.21.15.16:0/2207756417 wait complete. 2024-04-22T15:39:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.053+0000 7fc4746db700 1 Processor -- start 2024-04-22T15:39:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.053+0000 7fc4746db700 1 -- start start 2024-04-22T15:39:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.053+0000 7fc4746db700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc46c105470 0x7fc46c19a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.053+0000 7fc4746db700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46c19a670 con 0x7fc46c105470 2024-04-22T15:39:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.053+0000 7fc472477700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc46c105470 0x7fc46c19a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.053+0000 7fc472477700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc46c105470 0x7fc46c19a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45004/0 (socket says 172.21.15.16:45004) 2024-04-22T15:39:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.053+0000 7fc472477700 1 -- 172.21.15.16:0/514073048 learned_addr learned my addr 172.21.15.16:0/514073048 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.053+0000 7fc472477700 1 -- 172.21.15.16:0/514073048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc468009550 con 0x7fc46c105470 2024-04-22T15:39:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.053+0000 7fc472477700 1 --2- 172.21.15.16:0/514073048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc46c105470 0x7fc46c19a130 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc468006af0 tx=0x7fc468011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.054+0000 7fc4637fe700 1 -- 172.21.15.16:0/514073048 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc468003b90 con 0x7fc46c105470 2024-04-22T15:39:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.054+0000 7fc4637fe700 1 -- 172.21.15.16:0/514073048 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc46801a8a0 con 0x7fc46c105470 2024-04-22T15:39:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.054+0000 7fc4746db700 1 -- 172.21.15.16:0/514073048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc46c19a870 con 0x7fc46c105470 2024-04-22T15:39:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.054+0000 7fc4637fe700 1 -- 172.21.15.16:0/514073048 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc468019830 con 0x7fc46c105470 2024-04-22T15:39:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.054+0000 7fc4746db700 1 -- 172.21.15.16:0/514073048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc46c19ad10 con 0x7fc46c105470 2024-04-22T15:39:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.054+0000 7fc4637fe700 1 -- 172.21.15.16:0/514073048 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc46802d440 con 0x7fc46c105470 2024-04-22T15:39:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.054+0000 7fc4637fe700 1 -- 172.21.15.16:0/514073048 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc468024780 con 0x7fc46c105470 2024-04-22T15:39:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.055+0000 7fc4746db700 1 -- 172.21.15.16:0/514073048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc46c0623c0 con 0x7fc46c105470 2024-04-22T15:39:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.057+0000 7fc4637fe700 1 -- 172.21.15.16:0/514073048 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fc46801aa10 con 0x7fc46c105470 2024-04-22T15:39:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.126+0000 7fc4746db700 1 -- 172.21.15.16:0/514073048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fc46c109920 con 0x7fc46c105470 2024-04-22T15:39:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.129+0000 7fc4637fe700 1 -- 172.21.15.16:0/514073048 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc46801abb0 con 0x7fc46c105470 2024-04-22T15:39:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.129+0000 7fc4746db700 1 -- 172.21.15.16:0/514073048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc46c105470 msgr2=0x7fc46c19a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.129+0000 7fc4746db700 1 --2- 172.21.15.16:0/514073048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc46c105470 0x7fc46c19a130 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc468006af0 tx=0x7fc468011a90 comp rx=0 tx=0).stop 2024-04-22T15:39:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.129+0000 7fc4746db700 1 -- 172.21.15.16:0/514073048 shutdown_connections 2024-04-22T15:39:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.129+0000 7fc4746db700 1 --2- 172.21.15.16:0/514073048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc46c105470 0x7fc46c19a130 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.129+0000 7fc4746db700 1 -- 172.21.15.16:0/514073048 >> 172.21.15.16:0/514073048 conn(0x7fc46c1009f0 msgr2=0x7fc46c0740f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.130+0000 7fc4746db700 1 -- 172.21.15.16:0/514073048 shutdown_connections 2024-04-22T15:39:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:26.130+0000 7fc4746db700 1 -- 172.21.15.16:0/514073048 wait complete. 2024-04-22T15:39:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-22T15:39:26.644 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-22T15:39:26.644 INFO:teuthology.orchestra.run.smithi016.stdout:Creating mgr... 2024-04-22T15:39:26.644 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-22T15:39:26.645 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-22T15:39:26.645 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying port 0.0.0.0:8443 ... 2024-04-22T15:39:26.912 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-67ab3a80-00be-11ef-bc93-c7b262605968@mgr.smithi016.rnsnnr 2024-04-22T15:39:26.912 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to reset failed state of unit ceph-67ab3a80-00be-11ef-bc93-c7b262605968@mgr.smithi016.rnsnnr.service: Unit ceph-67ab3a80-00be-11ef-bc93-c7b262605968@mgr.smithi016.rnsnnr.service not loaded. 2024-04-22T15:39:27.065 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-67ab3a80-00be-11ef-bc93-c7b262605968.target.wants/ceph-67ab3a80-00be-11ef-bc93-c7b262605968@mgr.smithi016.rnsnnr.service → /etc/systemd/system/ceph-67ab3a80-00be-11ef-bc93-c7b262605968@.service. 2024-04-22T15:39:27.297 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:27 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/514073048' entity='client.admin' 2024-04-22T15:39:27.587 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T15:39:27.589 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-04-22T15:39:27.595 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T15:39:27.595 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-04-22T15:39:27.595 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-04-22T15:39:27.595 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to enable service . firewalld.service is not available 2024-04-22T15:39:27.603 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T15:39:27.603 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-04-22T15:39:27.610 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T15:39:27.610 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-04-22T15:39:27.610 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-04-22T15:39:27.611 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-04-22T15:39:27.611 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr to start... 2024-04-22T15:39:27.611 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr... 2024-04-22T15:39:28.385 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:28 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/567485986' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T15:39:28.652 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-22T15:39:28.652 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-22T15:39:28.652 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "67ab3a80-00be-11ef-bc93-c7b262605968", 2024-04-22T15:39:28.652 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T15:39:28.652 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T15:39:28.653 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T15:39:28.653 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T15:39:28.653 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:39:28.653 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T15:39:28.653 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T15:39:28.653 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-04-22T15:39:28.653 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-22T15:39:28.653 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T15:39:28.653 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "smithi016" 2024-04-22T15:39:28.653 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-22T15:39:28.653 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-22T15:39:28.653 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T15:39:28.653 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:39:28.653 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-22T15:39:28.653 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T15:39:28.654 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:39:28.654 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T15:39:28.654 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:39:28.654 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T15:39:28.654 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T15:39:28.654 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T15:39:28.654 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T15:39:28.654 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T15:39:28.654 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T15:39:28.654 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:39:28.654 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T15:39:28.654 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T15:39:28.654 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T15:39:28.654 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T15:39:28.655 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T15:39:28.655 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T15:39:28.655 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T15:39:28.655 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T15:39:28.655 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T15:39:28.655 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:39:28.655 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T15:39:28.655 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:39:28.655 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T15:39:28.655 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T15:39:28.655 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:39:28.655 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T15:39:28.655 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T15:39:28.655 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T15:39:28.655 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T15:39:28.655 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T15:39:28.656 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T15:39:28.656 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T15:39:28.656 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-22T15:39:28.656 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T15:39:28.656 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:39:28.656 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T15:39:28.656 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:39:28.656 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T15:39:20.727630+0000", 2024-04-22T15:39:28.656 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T15:39:28.656 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:39:28.656 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T15:39:28.656 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-22T15:39:28.656 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.043+0000 7f59bf2c6700 1 Processor -- start 2024-04-22T15:39:28.656 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.043+0000 7f59bf2c6700 1 -- start start 2024-04-22T15:39:28.657 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.043+0000 7f59bf2c6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59b8106560 0x7f59b8106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:28.657 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.043+0000 7f59bf2c6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b8106f50 con 0x7f59b8106560 2024-04-22T15:39:28.657 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.044+0000 7f59bd062700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59b8106560 0x7f59b8106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:28.657 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.044+0000 7f59bd062700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59b8106560 0x7f59b8106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45026/0 (socket says 172.21.15.16:45026) 2024-04-22T15:39:28.657 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.044+0000 7f59bd062700 1 -- 172.21.15.16:0/3111025997 learned_addr learned my addr 172.21.15.16:0/3111025997 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:28.657 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.044+0000 7f59bd062700 1 -- 172.21.15.16:0/3111025997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b8107760 con 0x7f59b8106560 2024-04-22T15:39:28.657 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.044+0000 7f59bd062700 1 --2- 172.21.15.16:0/3111025997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59b8106560 0x7f59b8106980 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f59b40098a0 tx=0x7f59b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea61efbede01ccc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:28.657 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.044+0000 7f59abfff700 1 -- 172.21.15.16:0/3111025997 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59b4004180 con 0x7f59b8106560 2024-04-22T15:39:28.657 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.045+0000 7f59abfff700 1 -- 172.21.15.16:0/3111025997 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f59b40042e0 con 0x7f59b8106560 2024-04-22T15:39:28.657 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.045+0000 7f59abfff700 1 -- 172.21.15.16:0/3111025997 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59b4011a50 con 0x7f59b8106560 2024-04-22T15:39:28.657 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.045+0000 7f59bf2c6700 1 -- 172.21.15.16:0/3111025997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59b8106560 msgr2=0x7f59b8106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:28.657 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.045+0000 7f59bf2c6700 1 --2- 172.21.15.16:0/3111025997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59b8106560 0x7f59b8106980 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f59b40098a0 tx=0x7f59b4009bb0 comp rx=0 tx=0).stop 2024-04-22T15:39:28.657 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.045+0000 7f59bf2c6700 1 -- 172.21.15.16:0/3111025997 shutdown_connections 2024-04-22T15:39:28.657 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.045+0000 7f59bf2c6700 1 --2- 172.21.15.16:0/3111025997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59b8106560 0x7f59b8106980 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:28.657 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.045+0000 7f59bf2c6700 1 -- 172.21.15.16:0/3111025997 >> 172.21.15.16:0/3111025997 conn(0x7f59b8101b40 msgr2=0x7f59b8103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.045+0000 7f59bf2c6700 1 -- 172.21.15.16:0/3111025997 shutdown_connections 2024-04-22T15:39:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.045+0000 7f59bf2c6700 1 -- 172.21.15.16:0/3111025997 wait complete. 2024-04-22T15:39:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.046+0000 7f59bf2c6700 1 Processor -- start 2024-04-22T15:39:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.046+0000 7f59bf2c6700 1 -- start start 2024-04-22T15:39:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.046+0000 7f59bf2c6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59b8106560 0x7f59b819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.046+0000 7f59bf2c6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b819a700 con 0x7f59b8106560 2024-04-22T15:39:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.046+0000 7f59bd062700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59b8106560 0x7f59b819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.046+0000 7f59bd062700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59b8106560 0x7f59b819a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45038/0 (socket says 172.21.15.16:45038) 2024-04-22T15:39:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.046+0000 7f59bd062700 1 -- 172.21.15.16:0/567485986 learned_addr learned my addr 172.21.15.16:0/567485986 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.047+0000 7f59bd062700 1 -- 172.21.15.16:0/567485986 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b4009550 con 0x7f59b8106560 2024-04-22T15:39:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.047+0000 7f59bd062700 1 --2- 172.21.15.16:0/567485986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59b8106560 0x7f59b819a1c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f59b4000c00 tx=0x7f59b4011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.047+0000 7f59aa7fc700 1 -- 172.21.15.16:0/567485986 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59b4003b90 con 0x7f59b8106560 2024-04-22T15:39:28.658 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.047+0000 7f59aa7fc700 1 -- 172.21.15.16:0/567485986 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f59b40248b0 con 0x7f59b8106560 2024-04-22T15:39:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.047+0000 7f59bf2c6700 1 -- 172.21.15.16:0/567485986 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59b819a900 con 0x7f59b8106560 2024-04-22T15:39:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.047+0000 7f59aa7fc700 1 -- 172.21.15.16:0/567485986 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59b401a8e0 con 0x7f59b8106560 2024-04-22T15:39:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.047+0000 7f59bf2c6700 1 -- 172.21.15.16:0/567485986 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59b819ada0 con 0x7f59b8106560 2024-04-22T15:39:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.047+0000 7f59aa7fc700 1 -- 172.21.15.16:0/567485986 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f59b402d3f0 con 0x7f59b8106560 2024-04-22T15:39:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.047+0000 7f59aa7fc700 1 -- 172.21.15.16:0/567485986 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f59b4019460 con 0x7f59b8106560 2024-04-22T15:39:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.048+0000 7f59bf2c6700 1 -- 172.21.15.16:0/567485986 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59b80623c0 con 0x7f59b8106560 2024-04-22T15:39:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.052+0000 7f59aa7fc700 1 -- 172.21.15.16:0/567485986 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f59b4024a20 con 0x7f59b8106560 2024-04-22T15:39:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.125+0000 7f59bf2c6700 1 -- 172.21.15.16:0/567485986 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f59b810aa10 con 0x7f59b8106560 2024-04-22T15:39:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.125+0000 7f59aa7fc700 1 -- 172.21.15.16:0/567485986 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f59b4035090 con 0x7f59b8106560 2024-04-22T15:39:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.126+0000 7f59bf2c6700 1 -- 172.21.15.16:0/567485986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59b8106560 msgr2=0x7f59b819a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.126+0000 7f59bf2c6700 1 --2- 172.21.15.16:0/567485986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59b8106560 0x7f59b819a1c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f59b4000c00 tx=0x7f59b4011ab0 comp rx=0 tx=0).stop 2024-04-22T15:39:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.126+0000 7f59bf2c6700 1 -- 172.21.15.16:0/567485986 shutdown_connections 2024-04-22T15:39:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.126+0000 7f59bf2c6700 1 --2- 172.21.15.16:0/567485986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59b8106560 0x7f59b819a1c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.126+0000 7f59bf2c6700 1 -- 172.21.15.16:0/567485986 >> 172.21.15.16:0/567485986 conn(0x7f59b8101b40 msgr2=0x7f59b8102870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.127+0000 7f59bf2c6700 1 -- 172.21.15.16:0/567485986 shutdown_connections 2024-04-22T15:39:28.659 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:28.127+0000 7f59bf2c6700 1 -- 172.21.15.16:0/567485986 wait complete. 2024-04-22T15:39:28.660 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (1/15)... 2024-04-22T15:39:31.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-22T15:39:31.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-22T15:39:31.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "67ab3a80-00be-11ef-bc93-c7b262605968", 2024-04-22T15:39:31.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T15:39:31.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T15:39:31.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T15:39:31.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T15:39:31.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:39:31.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T15:39:31.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T15:39:31.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-04-22T15:39:31.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-22T15:39:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T15:39:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "smithi016" 2024-04-22T15:39:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-22T15:39:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-04-22T15:39:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T15:39:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:39:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-22T15:39:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T15:39:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:39:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T15:39:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:39:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T15:39:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T15:39:31.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T15:39:31.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T15:39:31.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T15:39:31.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T15:39:31.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:39:31.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T15:39:31.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T15:39:31.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T15:39:31.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T15:39:31.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T15:39:31.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T15:39:31.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T15:39:31.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T15:39:31.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T15:39:31.746 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:39:31.746 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T15:39:31.746 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:39:31.746 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T15:39:31.746 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T15:39:31.746 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:39:31.746 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T15:39:31.746 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T15:39:31.746 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T15:39:31.746 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T15:39:31.746 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T15:39:31.746 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T15:39:31.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T15:39:31.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-22T15:39:31.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T15:39:31.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:39:31.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T15:39:31.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:39:31.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T15:39:20.727630+0000", 2024-04-22T15:39:31.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T15:39:31.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:39:31.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T15:39:31.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-22T15:39:31.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.056+0000 7f428bd9b700 1 Processor -- start 2024-04-22T15:39:31.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.056+0000 7f428bd9b700 1 -- start start 2024-04-22T15:39:31.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.056+0000 7f428bd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f428407be60 0x7f428407a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:31.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.056+0000 7f428bd9b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428407a890 con 0x7f428407be60 2024-04-22T15:39:31.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.057+0000 7f4289b37700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f428407be60 0x7f428407a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:31.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.057+0000 7f4289b37700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f428407be60 0x7f428407a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45054/0 (socket says 172.21.15.16:45054) 2024-04-22T15:39:31.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.057+0000 7f4289b37700 1 -- 172.21.15.16:0/2076745246 learned_addr learned my addr 172.21.15.16:0/2076745246 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:31.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.057+0000 7f4289b37700 1 -- 172.21.15.16:0/2076745246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f428407a9d0 con 0x7f428407be60 2024-04-22T15:39:31.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.057+0000 7f4289b37700 1 --2- 172.21.15.16:0/2076745246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f428407be60 0x7f428407a2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f42740098a0 tx=0x7f4274009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea0648e85d23cee5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:31.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.057+0000 7f4288b35700 1 -- 172.21.15.16:0/2076745246 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4274004180 con 0x7f428407be60 2024-04-22T15:39:31.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.057+0000 7f4288b35700 1 -- 172.21.15.16:0/2076745246 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f42740042e0 con 0x7f428407be60 2024-04-22T15:39:31.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.057+0000 7f4288b35700 1 -- 172.21.15.16:0/2076745246 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4274011a50 con 0x7f428407be60 2024-04-22T15:39:31.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.058+0000 7f428bd9b700 1 -- 172.21.15.16:0/2076745246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f428407be60 msgr2=0x7f428407a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:31.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.058+0000 7f428bd9b700 1 --2- 172.21.15.16:0/2076745246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f428407be60 0x7f428407a2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f42740098a0 tx=0x7f4274009bb0 comp rx=0 tx=0).stop 2024-04-22T15:39:31.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.058+0000 7f428bd9b700 1 -- 172.21.15.16:0/2076745246 shutdown_connections 2024-04-22T15:39:31.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.058+0000 7f428bd9b700 1 --2- 172.21.15.16:0/2076745246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f428407be60 0x7f428407a2c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:31.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.058+0000 7f428bd9b700 1 -- 172.21.15.16:0/2076745246 >> 172.21.15.16:0/2076745246 conn(0x7f4284101b40 msgr2=0x7f4284103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:31.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.058+0000 7f428bd9b700 1 -- 172.21.15.16:0/2076745246 shutdown_connections 2024-04-22T15:39:31.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.058+0000 7f428bd9b700 1 -- 172.21.15.16:0/2076745246 wait complete. 2024-04-22T15:39:31.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.059+0000 7f428bd9b700 1 Processor -- start 2024-04-22T15:39:31.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.059+0000 7f428bd9b700 1 -- start start 2024-04-22T15:39:31.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.059+0000 7f428bd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f428407be60 0x7f428419c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:31.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.059+0000 7f428bd9b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428419c8b0 con 0x7f428407be60 2024-04-22T15:39:31.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.059+0000 7f4289b37700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f428407be60 0x7f428419c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:31.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.059+0000 7f4289b37700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f428407be60 0x7f428419c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45062/0 (socket says 172.21.15.16:45062) 2024-04-22T15:39:31.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.059+0000 7f4289b37700 1 -- 172.21.15.16:0/878696621 learned_addr learned my addr 172.21.15.16:0/878696621 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:31.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.059+0000 7f4289b37700 1 -- 172.21.15.16:0/878696621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4274009550 con 0x7f428407be60 2024-04-22T15:39:31.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.059+0000 7f4289b37700 1 --2- 172.21.15.16:0/878696621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f428407be60 0x7f428419c370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4274000c00 tx=0x7f4274011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:31.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.060+0000 7f427affd700 1 -- 172.21.15.16:0/878696621 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4274003b90 con 0x7f428407be60 2024-04-22T15:39:31.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.060+0000 7f428bd9b700 1 -- 172.21.15.16:0/878696621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f428419cab0 con 0x7f428407be60 2024-04-22T15:39:31.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.060+0000 7f427affd700 1 -- 172.21.15.16:0/878696621 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f42740248b0 con 0x7f428407be60 2024-04-22T15:39:31.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.060+0000 7f427affd700 1 -- 172.21.15.16:0/878696621 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f427401a8e0 con 0x7f428407be60 2024-04-22T15:39:31.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.060+0000 7f428bd9b700 1 -- 172.21.15.16:0/878696621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f428419cf50 con 0x7f428407be60 2024-04-22T15:39:31.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.060+0000 7f427affd700 1 -- 172.21.15.16:0/878696621 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f427402d3f0 con 0x7f428407be60 2024-04-22T15:39:31.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.060+0000 7f427affd700 1 -- 172.21.15.16:0/878696621 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4274019580 con 0x7f428407be60 2024-04-22T15:39:31.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.061+0000 7f428bd9b700 1 -- 172.21.15.16:0/878696621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42840623c0 con 0x7f428407be60 2024-04-22T15:39:31.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.063+0000 7f427affd700 1 -- 172.21.15.16:0/878696621 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f4274024db0 con 0x7f428407be60 2024-04-22T15:39:31.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.136+0000 7f428bd9b700 1 -- 172.21.15.16:0/878696621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f428419fc30 con 0x7f428407be60 2024-04-22T15:39:31.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.136+0000 7f427affd700 1 -- 172.21.15.16:0/878696621 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f4274024a20 con 0x7f428407be60 2024-04-22T15:39:31.751 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.137+0000 7f428bd9b700 1 -- 172.21.15.16:0/878696621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f428407be60 msgr2=0x7f428419c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:31.751 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.137+0000 7f428bd9b700 1 --2- 172.21.15.16:0/878696621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f428407be60 0x7f428419c370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4274000c00 tx=0x7f4274011ab0 comp rx=0 tx=0).stop 2024-04-22T15:39:31.751 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.138+0000 7f428bd9b700 1 -- 172.21.15.16:0/878696621 shutdown_connections 2024-04-22T15:39:31.751 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.138+0000 7f428bd9b700 1 --2- 172.21.15.16:0/878696621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f428407be60 0x7f428419c370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:31.751 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.138+0000 7f428bd9b700 1 -- 172.21.15.16:0/878696621 >> 172.21.15.16:0/878696621 conn(0x7f4284101b40 msgr2=0x7f42841028a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:31.751 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.138+0000 7f428bd9b700 1 -- 172.21.15.16:0/878696621 shutdown_connections 2024-04-22T15:39:31.751 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:31.138+0000 7f428bd9b700 1 -- 172.21.15.16:0/878696621 wait complete. 2024-04-22T15:39:31.751 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (2/15)... 2024-04-22T15:39:31.836 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:31 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/878696621' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T15:39:34.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:34 smithi016 ceph-mon[26620]: Activating manager daemon smithi016.rnsnnr 2024-04-22T15:39:34.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:34 smithi016 ceph-mon[26620]: mgrmap e2: smithi016.rnsnnr(active, starting, since 0.0029007s) 2024-04-22T15:39:34.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:34 smithi016 ceph-mon[26620]: from='mgr.14100 172.21.15.16:0/2002407206' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T15:39:34.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:34 smithi016 ceph-mon[26620]: from='mgr.14100 172.21.15.16:0/2002407206' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T15:39:34.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:34 smithi016 ceph-mon[26620]: from='mgr.14100 172.21.15.16:0/2002407206' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T15:39:34.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:34 smithi016 ceph-mon[26620]: from='mgr.14100 172.21.15.16:0/2002407206' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-04-22T15:39:34.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:34 smithi016 ceph-mon[26620]: from='mgr.14100 172.21.15.16:0/2002407206' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr metadata", "who": "smithi016.rnsnnr", "id": "smithi016.rnsnnr"}]: dispatch 2024-04-22T15:39:34.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:34 smithi016 ceph-mon[26620]: Manager daemon smithi016.rnsnnr is now available 2024-04-22T15:39:34.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:34 smithi016 ceph-mon[26620]: from='mgr.14100 172.21.15.16:0/2002407206' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi016.rnsnnr/mirror_snapshot_schedule"}]: dispatch 2024-04-22T15:39:34.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:34 smithi016 ceph-mon[26620]: from='mgr.14100 172.21.15.16:0/2002407206' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi016.rnsnnr/trash_purge_schedule"}]: dispatch 2024-04-22T15:39:34.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:34 smithi016 ceph-mon[26620]: from='mgr.14100 172.21.15.16:0/2002407206' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:39:34.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:34 smithi016 ceph-mon[26620]: from='mgr.14100 172.21.15.16:0/2002407206' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:39:34.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:34 smithi016 ceph-mon[26620]: from='mgr.14100 172.21.15.16:0/2002407206' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:39:36.552 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:36 smithi016 ceph-mon[26620]: mgrmap e3: smithi016.rnsnnr(active, since 1.00723s) 2024-04-22T15:39:37.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:37 smithi016 ceph-mon[26620]: mgrmap e4: smithi016.rnsnnr(active, since 2s) 2024-04-22T15:39:37.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:37 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/1134916795' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T15:39:42.870 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-22T15:39:42.870 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-22T15:39:42.870 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "67ab3a80-00be-11ef-bc93-c7b262605968", 2024-04-22T15:39:42.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T15:39:42.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T15:39:42.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T15:39:42.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T15:39:42.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:39:42.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T15:39:42.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T15:39:42.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-04-22T15:39:42.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-22T15:39:42.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T15:39:42.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "smithi016" 2024-04-22T15:39:42.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-22T15:39:42.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-04-22T15:39:42.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T15:39:42.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:39:42.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-22T15:39:42.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T15:39:42.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:39:42.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T15:39:42.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:39:42.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T15:39:42.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T15:39:42.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T15:39:42.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T15:39:42.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T15:39:42.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T15:39:42.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:39:42.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T15:39:42.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T15:39:42.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T15:39:42.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T15:39:42.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T15:39:42.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T15:39:42.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T15:39:42.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T15:39:42.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T15:39:42.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:39:42.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T15:39:42.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:39:42.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T15:39:42.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T15:39:42.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:39:42.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T15:39:42.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T15:39:42.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T15:39:42.874 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T15:39:42.874 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T15:39:42.874 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T15:39:42.874 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T15:39:42.874 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-22T15:39:42.874 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T15:39:42.874 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:39:42.874 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T15:39:42.874 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T15:39:42.874 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T15:39:20.727630+0000", 2024-04-22T15:39:42.874 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T15:39:42.874 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-22T15:39:42.874 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T15:39:42.875 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-22T15:39:42.875 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.393+0000 7f6be5e99700 1 Processor -- start 2024-04-22T15:39:42.875 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.393+0000 7f6be5e99700 1 -- start start 2024-04-22T15:39:42.875 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.393+0000 7f6be5e99700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be007be80 0x7f6be007a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:42.875 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.393+0000 7f6be5e99700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be007a8b0 con 0x7f6be007be80 2024-04-22T15:39:42.875 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.393+0000 7f6bdf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be007be80 0x7f6be007a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:42.875 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.393+0000 7f6bdf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be007be80 0x7f6be007a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57546/0 (socket says 172.21.15.16:57546) 2024-04-22T15:39:42.875 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.393+0000 7f6bdf7fe700 1 -- 172.21.15.16:0/3510233288 learned_addr learned my addr 172.21.15.16:0/3510233288 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:42.875 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.393+0000 7f6bdf7fe700 1 -- 172.21.15.16:0/3510233288 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be007a9f0 con 0x7f6be007be80 2024-04-22T15:39:42.875 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.394+0000 7f6bdf7fe700 1 --2- 172.21.15.16:0/3510233288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be007be80 0x7f6be007a2e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40098a0 tx=0x7f6bd4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51f446b5b0ccaba0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:42.875 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.394+0000 7f6bde7fc700 1 -- 172.21.15.16:0/3510233288 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bd4004180 con 0x7f6be007be80 2024-04-22T15:39:42.875 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.394+0000 7f6bde7fc700 1 -- 172.21.15.16:0/3510233288 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6bd40042e0 con 0x7f6be007be80 2024-04-22T15:39:42.875 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.394+0000 7f6bde7fc700 1 -- 172.21.15.16:0/3510233288 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bd4011a50 con 0x7f6be007be80 2024-04-22T15:39:42.875 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.394+0000 7f6be5e99700 1 -- 172.21.15.16:0/3510233288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be007be80 msgr2=0x7f6be007a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:42.876 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.394+0000 7f6be5e99700 1 --2- 172.21.15.16:0/3510233288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be007be80 0x7f6be007a2e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40098a0 tx=0x7f6bd4009bb0 comp rx=0 tx=0).stop 2024-04-22T15:39:42.876 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.394+0000 7f6be5e99700 1 -- 172.21.15.16:0/3510233288 shutdown_connections 2024-04-22T15:39:42.876 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.394+0000 7f6be5e99700 1 --2- 172.21.15.16:0/3510233288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be007be80 0x7f6be007a2e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:42.876 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.394+0000 7f6be5e99700 1 -- 172.21.15.16:0/3510233288 >> 172.21.15.16:0/3510233288 conn(0x7f6be0101b40 msgr2=0x7f6be0103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:42.876 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.395+0000 7f6be5e99700 1 -- 172.21.15.16:0/3510233288 shutdown_connections 2024-04-22T15:39:42.876 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.395+0000 7f6be5e99700 1 -- 172.21.15.16:0/3510233288 wait complete. 2024-04-22T15:39:42.876 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.395+0000 7f6be5e99700 1 Processor -- start 2024-04-22T15:39:42.876 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.395+0000 7f6be5e99700 1 -- start start 2024-04-22T15:39:42.876 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.395+0000 7f6be5e99700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be007be80 0x7f6be019c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:42.876 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.395+0000 7f6be5e99700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be019c960 con 0x7f6be007be80 2024-04-22T15:39:42.876 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.396+0000 7f6bdf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be007be80 0x7f6be019c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:42.876 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.396+0000 7f6bdf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be007be80 0x7f6be019c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57554/0 (socket says 172.21.15.16:57554) 2024-04-22T15:39:42.876 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.396+0000 7f6bdf7fe700 1 -- 172.21.15.16:0/1134916795 learned_addr learned my addr 172.21.15.16:0/1134916795 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:42.876 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.396+0000 7f6bdf7fe700 1 -- 172.21.15.16:0/1134916795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd4009550 con 0x7f6be007be80 2024-04-22T15:39:42.876 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.396+0000 7f6bdf7fe700 1 --2- 172.21.15.16:0/1134916795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be007be80 0x7f6be019c420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4003de0 tx=0x7f6bd4011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:42.877 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.396+0000 7f6bdcff9700 1 -- 172.21.15.16:0/1134916795 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bd4003930 con 0x7f6be007be80 2024-04-22T15:39:42.877 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.396+0000 7f6bdcff9700 1 -- 172.21.15.16:0/1134916795 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6bd4003a90 con 0x7f6be007be80 2024-04-22T15:39:42.877 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.396+0000 7f6be5e99700 1 -- 172.21.15.16:0/1134916795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be019cb60 con 0x7f6be007be80 2024-04-22T15:39:42.877 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.396+0000 7f6bdcff9700 1 -- 172.21.15.16:0/1134916795 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bd40196c0 con 0x7f6be007be80 2024-04-22T15:39:42.877 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.396+0000 7f6be5e99700 1 -- 172.21.15.16:0/1134916795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be019d000 con 0x7f6be007be80 2024-04-22T15:39:42.877 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.397+0000 7f6bdcff9700 1 -- 172.21.15.16:0/1134916795 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45166+0+0 (secure 0 0 0) 0x7f6bd4019980 con 0x7f6be007be80 2024-04-22T15:39:42.877 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.397+0000 7f6be5e99700 1 -- 172.21.15.16:0/1134916795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be010c8c0 con 0x7f6be007be80 2024-04-22T15:39:42.877 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.397+0000 7f6bdcff9700 1 --2- 172.21.15.16:0/1134916795 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7f6bc80381b0 0x7f6bc803a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:42.877 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.397+0000 7f6bdcff9700 1 -- 172.21.15.16:0/1134916795 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6bd404bf80 con 0x7f6be007be80 2024-04-22T15:39:42.877 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.398+0000 7f6bdeffd700 1 --2- 172.21.15.16:0/1134916795 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7f6bc80381b0 0x7f6bc803a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:42.877 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.398+0000 7f6bdeffd700 1 --2- 172.21.15.16:0/1134916795 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7f6bc80381b0 0x7f6bc803a670 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0006fd0 tx=0x7f6bd0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:42.877 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.402+0000 7f6bdcff9700 1 -- 172.21.15.16:0/1134916795 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6bd4010bb0 con 0x7f6be007be80 2024-04-22T15:39:42.877 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.549+0000 7f6bdcff9700 1 -- 172.21.15.16:0/1134916795 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 4) v1 ==== 45272+0+0 (secure 0 0 0) 0x7f6bd401abc0 con 0x7f6be007be80 2024-04-22T15:39:42.878 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.681+0000 7f6be5e99700 1 -- 172.21.15.16:0/1134916795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6be010cae0 con 0x7f6be007be80 2024-04-22T15:39:42.878 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.681+0000 7f6bdcff9700 1 -- 172.21.15.16:0/1134916795 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f6bd4010bb0 con 0x7f6be007be80 2024-04-22T15:39:42.878 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.683+0000 7f6be5e99700 1 -- 172.21.15.16:0/1134916795 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7f6bc80381b0 msgr2=0x7f6bc803a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:42.878 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.683+0000 7f6be5e99700 1 --2- 172.21.15.16:0/1134916795 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7f6bc80381b0 0x7f6bc803a670 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0006fd0 tx=0x7f6bd0006e40 comp rx=0 tx=0).stop 2024-04-22T15:39:42.878 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.683+0000 7f6be5e99700 1 -- 172.21.15.16:0/1134916795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be007be80 msgr2=0x7f6be019c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:42.878 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.683+0000 7f6be5e99700 1 --2- 172.21.15.16:0/1134916795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be007be80 0x7f6be019c420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4003de0 tx=0x7f6bd4011ab0 comp rx=0 tx=0).stop 2024-04-22T15:39:42.878 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.683+0000 7f6be5e99700 1 -- 172.21.15.16:0/1134916795 shutdown_connections 2024-04-22T15:39:42.878 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.683+0000 7f6be5e99700 1 --2- 172.21.15.16:0/1134916795 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7f6bc80381b0 0x7f6bc803a670 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:42.878 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.683+0000 7f6be5e99700 1 --2- 172.21.15.16:0/1134916795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6be007be80 0x7f6be019c420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:42.878 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.683+0000 7f6be5e99700 1 -- 172.21.15.16:0/1134916795 >> 172.21.15.16:0/1134916795 conn(0x7f6be0101b40 msgr2=0x7f6be01031a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:42.878 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.684+0000 7f6be5e99700 1 -- 172.21.15.16:0/1134916795 shutdown_connections 2024-04-22T15:39:42.878 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:36.684+0000 7f6be5e99700 1 -- 172.21.15.16:0/1134916795 wait complete. 2024-04-22T15:39:42.878 INFO:teuthology.orchestra.run.smithi016.stdout:mgr is available 2024-04-22T15:39:45.314 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:45 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/3294459487' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-22T15:39:45.590 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-22T15:39:45.590 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [global] 2024-04-22T15:39:45.590 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout fsid = 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T15:39:45.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-22T15:39:45.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-22T15:39:45.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-22T15:39:45.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-22T15:39:45.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-22T15:39:45.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-22T15:39:45.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-22T15:39:45.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-22T15:39:45.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [osd] 2024-04-22T15:39:45.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-22T15:39:45.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-22T15:39:45.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-22T15:39:45.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-22T15:39:45.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-22T15:39:45.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.801+0000 7f43dbbb6700 1 Processor -- start 2024-04-22T15:39:45.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.801+0000 7f43dbbb6700 1 -- start start 2024-04-22T15:39:45.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.801+0000 7f43dbbb6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43d4106fe0 0x7f43d41093d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:45.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.801+0000 7f43dbbb6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d4074680 con 0x7f43d4106fe0 2024-04-22T15:39:45.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.802+0000 7f43d9952700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43d4106fe0 0x7f43d41093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:45.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.802+0000 7f43d9952700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43d4106fe0 0x7f43d41093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41220/0 (socket says 172.21.15.16:41220) 2024-04-22T15:39:45.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.802+0000 7f43d9952700 1 -- 172.21.15.16:0/1518944812 learned_addr learned my addr 172.21.15.16:0/1518944812 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:45.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.802+0000 7f43d9952700 1 -- 172.21.15.16:0/1518944812 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d4109910 con 0x7f43d4106fe0 2024-04-22T15:39:45.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.802+0000 7f43d9952700 1 --2- 172.21.15.16:0/1518944812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43d4106fe0 0x7f43d41093d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f43d00098a0 tx=0x7f43d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e7114696a22a658 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:45.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.802+0000 7f43d8950700 1 -- 172.21.15.16:0/1518944812 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43d0004180 con 0x7f43d4106fe0 2024-04-22T15:39:45.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.802+0000 7f43d8950700 1 -- 172.21.15.16:0/1518944812 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f43d00042e0 con 0x7f43d4106fe0 2024-04-22T15:39:45.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.802+0000 7f43d8950700 1 -- 172.21.15.16:0/1518944812 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43d0011a50 con 0x7f43d4106fe0 2024-04-22T15:39:45.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.803+0000 7f43dbbb6700 1 -- 172.21.15.16:0/1518944812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43d4106fe0 msgr2=0x7f43d41093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:45.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.803+0000 7f43dbbb6700 1 --2- 172.21.15.16:0/1518944812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43d4106fe0 0x7f43d41093d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f43d00098a0 tx=0x7f43d0009bb0 comp rx=0 tx=0).stop 2024-04-22T15:39:45.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.803+0000 7f43dbbb6700 1 -- 172.21.15.16:0/1518944812 shutdown_connections 2024-04-22T15:39:45.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.803+0000 7f43dbbb6700 1 --2- 172.21.15.16:0/1518944812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43d4106fe0 0x7f43d41093d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:45.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.803+0000 7f43dbbb6700 1 -- 172.21.15.16:0/1518944812 >> 172.21.15.16:0/1518944812 conn(0x7f43d41009f0 msgr2=0x7f43d4102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:45.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.803+0000 7f43dbbb6700 1 -- 172.21.15.16:0/1518944812 shutdown_connections 2024-04-22T15:39:45.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.803+0000 7f43dbbb6700 1 -- 172.21.15.16:0/1518944812 wait complete. 2024-04-22T15:39:45.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.804+0000 7f43dbbb6700 1 Processor -- start 2024-04-22T15:39:45.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.804+0000 7f43dbbb6700 1 -- start start 2024-04-22T15:39:45.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.804+0000 7f43dbbb6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43d4106fe0 0x7f43d4106940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:45.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.804+0000 7f43dbbb6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d4106e80 con 0x7f43d4106fe0 2024-04-22T15:39:45.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.804+0000 7f43d9952700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43d4106fe0 0x7f43d4106940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:45.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.804+0000 7f43d9952700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43d4106fe0 0x7f43d4106940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41230/0 (socket says 172.21.15.16:41230) 2024-04-22T15:39:45.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.804+0000 7f43d9952700 1 -- 172.21.15.16:0/3294459487 learned_addr learned my addr 172.21.15.16:0/3294459487 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:45.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.804+0000 7f43d9952700 1 -- 172.21.15.16:0/3294459487 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d0009550 con 0x7f43d4106fe0 2024-04-22T15:39:45.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.805+0000 7f43d9952700 1 --2- 172.21.15.16:0/3294459487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43d4106fe0 0x7f43d4106940 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f43d0000c00 tx=0x7f43d0011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:45.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.805+0000 7f43c6ffd700 1 -- 172.21.15.16:0/3294459487 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43d0003b90 con 0x7f43d4106fe0 2024-04-22T15:39:45.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.805+0000 7f43c6ffd700 1 -- 172.21.15.16:0/3294459487 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f43d00248b0 con 0x7f43d4106fe0 2024-04-22T15:39:45.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.805+0000 7f43dbbb6700 1 -- 172.21.15.16:0/3294459487 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d4104f90 con 0x7f43d4106fe0 2024-04-22T15:39:45.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.805+0000 7f43dbbb6700 1 -- 172.21.15.16:0/3294459487 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d4105430 con 0x7f43d4106fe0 2024-04-22T15:39:45.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.805+0000 7f43c6ffd700 1 -- 172.21.15.16:0/3294459487 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43d001a8e0 con 0x7f43d4106fe0 2024-04-22T15:39:45.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.806+0000 7f43c6ffd700 1 -- 172.21.15.16:0/3294459487 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) v1 ==== 45272+0+0 (secure 0 0 0) 0x7f43d001ab50 con 0x7f43d4106fe0 2024-04-22T15:39:45.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.806+0000 7f43dbbb6700 1 -- 172.21.15.16:0/3294459487 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d410a690 con 0x7f43d4106fe0 2024-04-22T15:39:45.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.806+0000 7f43c6ffd700 1 --2- 172.21.15.16:0/3294459487 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7f43c0038320 0x7f43c003a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:45.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.806+0000 7f43c6ffd700 1 -- 172.21.15.16:0/3294459487 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f43d004bff0 con 0x7f43d4106fe0 2024-04-22T15:39:45.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.806+0000 7f43d9151700 1 --2- 172.21.15.16:0/3294459487 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7f43c0038320 0x7f43c003a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:45.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.807+0000 7f43d9151700 1 --2- 172.21.15.16:0/3294459487 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7f43c0038320 0x7f43c003a7e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f43c8006fd0 tx=0x7f43c8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:45.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:44.811+0000 7f43c6ffd700 1 -- 172.21.15.16:0/3294459487 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f43d0018b00 con 0x7f43d4106fe0 2024-04-22T15:39:45.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:45.008+0000 7f43dbbb6700 1 -- 172.21.15.16:0/3294459487 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f43d4105a90 con 0x7f43d4106fe0 2024-04-22T15:39:45.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:45.010+0000 7f43c6ffd700 1 -- 172.21.15.16:0/3294459487 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f43d002c570 con 0x7f43d4106fe0 2024-04-22T15:39:45.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:45.012+0000 7f43dbbb6700 1 -- 172.21.15.16:0/3294459487 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7f43c0038320 msgr2=0x7f43c003a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:45.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:45.012+0000 7f43dbbb6700 1 --2- 172.21.15.16:0/3294459487 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7f43c0038320 0x7f43c003a7e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f43c8006fd0 tx=0x7f43c8006e40 comp rx=0 tx=0).stop 2024-04-22T15:39:45.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:45.012+0000 7f43dbbb6700 1 -- 172.21.15.16:0/3294459487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43d4106fe0 msgr2=0x7f43d4106940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:45.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:45.012+0000 7f43dbbb6700 1 --2- 172.21.15.16:0/3294459487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43d4106fe0 0x7f43d4106940 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f43d0000c00 tx=0x7f43d0011ab0 comp rx=0 tx=0).stop 2024-04-22T15:39:45.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:45.012+0000 7f43dbbb6700 1 -- 172.21.15.16:0/3294459487 shutdown_connections 2024-04-22T15:39:45.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:45.012+0000 7f43dbbb6700 1 --2- 172.21.15.16:0/3294459487 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7f43c0038320 0x7f43c003a7e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:45.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:45.012+0000 7f43dbbb6700 1 --2- 172.21.15.16:0/3294459487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43d4106fe0 0x7f43d4106940 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:45.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:45.012+0000 7f43dbbb6700 1 -- 172.21.15.16:0/3294459487 >> 172.21.15.16:0/3294459487 conn(0x7f43d41009f0 msgr2=0x7f43d4101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:45.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:45.012+0000 7f43dbbb6700 1 -- 172.21.15.16:0/3294459487 shutdown_connections 2024-04-22T15:39:45.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:45.013+0000 7f43dbbb6700 1 -- 172.21.15.16:0/3294459487 wait complete. 2024-04-22T15:39:45.596 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling cephadm module... 2024-04-22T15:39:46.557 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:46 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/3858182551' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-22T15:39:47.097 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.043+0000 7fc004622700 1 Processor -- start 2024-04-22T15:39:47.097 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.043+0000 7fc004622700 1 -- start start 2024-04-22T15:39:47.097 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.043+0000 7fc004622700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbffc106580 0x7fbffc1069a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:47.097 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.043+0000 7fc004622700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbffc106f70 con 0x7fbffc106580 2024-04-22T15:39:47.097 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.044+0000 7fc0023be700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbffc106580 0x7fbffc1069a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:47.097 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.044+0000 7fc0023be700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbffc106580 0x7fbffc1069a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41234/0 (socket says 172.21.15.16:41234) 2024-04-22T15:39:47.097 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.044+0000 7fc0023be700 1 -- 172.21.15.16:0/3736803049 learned_addr learned my addr 172.21.15.16:0/3736803049 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:47.097 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.044+0000 7fc0023be700 1 -- 172.21.15.16:0/3736803049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbffc107780 con 0x7fbffc106580 2024-04-22T15:39:47.097 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.044+0000 7fc0023be700 1 --2- 172.21.15.16:0/3736803049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbffc106580 0x7fbffc1069a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbff80098a0 tx=0x7fbff8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=221ba1677f9544b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:47.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.044+0000 7fc0013bc700 1 -- 172.21.15.16:0/3736803049 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbff8004180 con 0x7fbffc106580 2024-04-22T15:39:47.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.044+0000 7fc0013bc700 1 -- 172.21.15.16:0/3736803049 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbff80042e0 con 0x7fbffc106580 2024-04-22T15:39:47.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.045+0000 7fc0013bc700 1 -- 172.21.15.16:0/3736803049 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbff8011a50 con 0x7fbffc106580 2024-04-22T15:39:47.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.045+0000 7fc004622700 1 -- 172.21.15.16:0/3736803049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbffc106580 msgr2=0x7fbffc1069a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:47.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.045+0000 7fc004622700 1 --2- 172.21.15.16:0/3736803049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbffc106580 0x7fbffc1069a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbff80098a0 tx=0x7fbff8009bb0 comp rx=0 tx=0).stop 2024-04-22T15:39:47.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.045+0000 7fc004622700 1 -- 172.21.15.16:0/3736803049 shutdown_connections 2024-04-22T15:39:47.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.045+0000 7fc004622700 1 --2- 172.21.15.16:0/3736803049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbffc106580 0x7fbffc1069a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:47.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.045+0000 7fc004622700 1 -- 172.21.15.16:0/3736803049 >> 172.21.15.16:0/3736803049 conn(0x7fbffc101b00 msgr2=0x7fbffc103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:47.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.045+0000 7fc004622700 1 -- 172.21.15.16:0/3736803049 shutdown_connections 2024-04-22T15:39:47.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.045+0000 7fc004622700 1 -- 172.21.15.16:0/3736803049 wait complete. 2024-04-22T15:39:47.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.046+0000 7fc004622700 1 Processor -- start 2024-04-22T15:39:47.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.046+0000 7fc004622700 1 -- start start 2024-04-22T15:39:47.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.046+0000 7fc004622700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbffc106580 0x7fbffc19c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:47.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.046+0000 7fc004622700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbffc19c980 con 0x7fbffc106580 2024-04-22T15:39:47.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.046+0000 7fc0023be700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbffc106580 0x7fbffc19c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:47.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.046+0000 7fc0023be700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbffc106580 0x7fbffc19c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41242/0 (socket says 172.21.15.16:41242) 2024-04-22T15:39:47.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.046+0000 7fc0023be700 1 -- 172.21.15.16:0/3858182551 learned_addr learned my addr 172.21.15.16:0/3858182551 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:47.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.046+0000 7fc0023be700 1 -- 172.21.15.16:0/3858182551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff8009550 con 0x7fbffc106580 2024-04-22T15:39:47.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.046+0000 7fc0023be700 1 --2- 172.21.15.16:0/3858182551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbffc106580 0x7fbffc19c440 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbff8009520 tx=0x7fbff8011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:47.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.047+0000 7fbff37fe700 1 -- 172.21.15.16:0/3858182551 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbff8003a80 con 0x7fbffc106580 2024-04-22T15:39:47.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.047+0000 7fbff37fe700 1 -- 172.21.15.16:0/3858182551 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbff8003be0 con 0x7fbffc106580 2024-04-22T15:39:47.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.047+0000 7fc004622700 1 -- 172.21.15.16:0/3858182551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbffc19cb80 con 0x7fbffc106580 2024-04-22T15:39:47.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.047+0000 7fbff37fe700 1 -- 172.21.15.16:0/3858182551 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbff80197b0 con 0x7fbffc106580 2024-04-22T15:39:47.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.047+0000 7fc004622700 1 -- 172.21.15.16:0/3858182551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbffc19d020 con 0x7fbffc106580 2024-04-22T15:39:47.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.047+0000 7fbff37fe700 1 -- 172.21.15.16:0/3858182551 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) v1 ==== 45272+0+0 (secure 0 0 0) 0x7fbff801a430 con 0x7fbffc106580 2024-04-22T15:39:47.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.048+0000 7fc004622700 1 -- 172.21.15.16:0/3858182551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbffc0623c0 con 0x7fbffc106580 2024-04-22T15:39:47.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.048+0000 7fbff37fe700 1 --2- 172.21.15.16:0/3858182551 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7fbfe80382d0 0x7fbfe803a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:47.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.048+0000 7fbff37fe700 1 -- 172.21.15.16:0/3858182551 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbff804c490 con 0x7fbffc106580 2024-04-22T15:39:47.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.048+0000 7fc001bbd700 1 --2- 172.21.15.16:0/3858182551 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7fbfe80382d0 0x7fbfe803a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:47.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.048+0000 7fc001bbd700 1 --2- 172.21.15.16:0/3858182551 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7fbfe80382d0 0x7fbfe803a790 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbfec006fd0 tx=0x7fbfec006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:47.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.053+0000 7fbff37fe700 1 -- 172.21.15.16:0/3858182551 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fbff8018530 con 0x7fbffc106580 2024-04-22T15:39:47.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.305+0000 7fc004622700 1 -- 172.21.15.16:0/3858182551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fbffc1a01b0 con 0x7fbffc106580 2024-04-22T15:39:47.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.554+0000 7fbff37fe700 1 -- 172.21.15.16:0/3858182551 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fbff802d440 con 0x7fbffc106580 2024-04-22T15:39:47.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.555+0000 7fbff37fe700 1 -- 172.21.15.16:0/3858182551 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fbff8023c40 con 0x7fbffc106580 2024-04-22T15:39:47.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.558+0000 7fc004622700 1 -- 172.21.15.16:0/3858182551 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7fbfe80382d0 msgr2=0x7fbfe803a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:47.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.558+0000 7fc004622700 1 --2- 172.21.15.16:0/3858182551 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7fbfe80382d0 0x7fbfe803a790 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbfec006fd0 tx=0x7fbfec006e40 comp rx=0 tx=0).stop 2024-04-22T15:39:47.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.558+0000 7fc004622700 1 -- 172.21.15.16:0/3858182551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbffc106580 msgr2=0x7fbffc19c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:47.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.558+0000 7fc004622700 1 --2- 172.21.15.16:0/3858182551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbffc106580 0x7fbffc19c440 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbff8009520 tx=0x7fbff8011ab0 comp rx=0 tx=0).stop 2024-04-22T15:39:47.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.558+0000 7fc004622700 1 -- 172.21.15.16:0/3858182551 shutdown_connections 2024-04-22T15:39:47.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.558+0000 7fc004622700 1 --2- 172.21.15.16:0/3858182551 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7fbfe80382d0 0x7fbfe803a790 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:47.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.558+0000 7fc004622700 1 --2- 172.21.15.16:0/3858182551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbffc106580 0x7fbffc19c440 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:47.101 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.558+0000 7fc004622700 1 -- 172.21.15.16:0/3858182551 >> 172.21.15.16:0/3858182551 conn(0x7fbffc101b00 msgr2=0x7fbffc1028f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:47.101 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.558+0000 7fc004622700 1 -- 172.21.15.16:0/3858182551 shutdown_connections 2024-04-22T15:39:47.101 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:46.558+0000 7fc004622700 1 -- 172.21.15.16:0/3858182551 wait complete. 2024-04-22T15:39:47.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:47 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/3858182551' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-22T15:39:47.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:47 smithi016 ceph-mon[26620]: mgrmap e5: smithi016.rnsnnr(active, since 12s) 2024-04-22T15:39:48.353 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-22T15:39:48.353 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-22T15:39:48.353 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T15:39:48.353 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "active_name": "smithi016.rnsnnr", 2024-04-22T15:39:48.354 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-22T15:39:48.354 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-22T15:39:48.354 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.466+0000 7f9788c70700 1 Processor -- start 2024-04-22T15:39:48.354 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.466+0000 7f9788c70700 1 -- start start 2024-04-22T15:39:48.354 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.466+0000 7f9788c70700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9784106560 0x7f9784106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:48.354 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.466+0000 7f9788c70700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9784106f50 con 0x7f9784106560 2024-04-22T15:39:48.354 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.467+0000 7f978259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9784106560 0x7f9784106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:48.354 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.467+0000 7f978259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9784106560 0x7f9784106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41264/0 (socket says 172.21.15.16:41264) 2024-04-22T15:39:48.354 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.467+0000 7f978259c700 1 -- 172.21.15.16:0/4274557462 learned_addr learned my addr 172.21.15.16:0/4274557462 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:48.354 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.467+0000 7f978259c700 1 -- 172.21.15.16:0/4274557462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9784107760 con 0x7f9784106560 2024-04-22T15:39:48.354 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.467+0000 7f978259c700 1 --2- 172.21.15.16:0/4274557462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9784106560 0x7f9784106980 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f97780098a0 tx=0x7f9778009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8109c3e54c441a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:48.354 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.467+0000 7f978159a700 1 -- 172.21.15.16:0/4274557462 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9778004180 con 0x7f9784106560 2024-04-22T15:39:48.354 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.467+0000 7f978159a700 1 -- 172.21.15.16:0/4274557462 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f97780042e0 con 0x7f9784106560 2024-04-22T15:39:48.355 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.467+0000 7f978159a700 1 -- 172.21.15.16:0/4274557462 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9778011a50 con 0x7f9784106560 2024-04-22T15:39:48.355 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.468+0000 7f9788c70700 1 -- 172.21.15.16:0/4274557462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9784106560 msgr2=0x7f9784106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:48.355 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.468+0000 7f9788c70700 1 --2- 172.21.15.16:0/4274557462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9784106560 0x7f9784106980 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f97780098a0 tx=0x7f9778009bb0 comp rx=0 tx=0).stop 2024-04-22T15:39:48.355 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.468+0000 7f9788c70700 1 -- 172.21.15.16:0/4274557462 shutdown_connections 2024-04-22T15:39:48.355 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.468+0000 7f9788c70700 1 --2- 172.21.15.16:0/4274557462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9784106560 0x7f9784106980 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:48.355 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.468+0000 7f9788c70700 1 -- 172.21.15.16:0/4274557462 >> 172.21.15.16:0/4274557462 conn(0x7f9784101b20 msgr2=0x7f9784103f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:48.355 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.468+0000 7f9788c70700 1 -- 172.21.15.16:0/4274557462 shutdown_connections 2024-04-22T15:39:48.355 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.468+0000 7f9788c70700 1 -- 172.21.15.16:0/4274557462 wait complete. 2024-04-22T15:39:48.355 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.469+0000 7f9788c70700 1 Processor -- start 2024-04-22T15:39:48.355 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.469+0000 7f9788c70700 1 -- start start 2024-04-22T15:39:48.355 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.469+0000 7f9788c70700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9784106560 0x7f978419a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:48.355 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.469+0000 7f9788c70700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978419a6e0 con 0x7f9784106560 2024-04-22T15:39:48.355 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.469+0000 7f978259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9784106560 0x7f978419a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:48.355 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.469+0000 7f978259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9784106560 0x7f978419a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41278/0 (socket says 172.21.15.16:41278) 2024-04-22T15:39:48.355 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.469+0000 7f978259c700 1 -- 172.21.15.16:0/2252447367 learned_addr learned my addr 172.21.15.16:0/2252447367 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:48.356 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.470+0000 7f978259c700 1 -- 172.21.15.16:0/2252447367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9778009550 con 0x7f9784106560 2024-04-22T15:39:48.356 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.470+0000 7f978259c700 1 --2- 172.21.15.16:0/2252447367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9784106560 0x7f978419a1a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9778006af0 tx=0x7f9778011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:48.356 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.470+0000 7f97737fe700 1 -- 172.21.15.16:0/2252447367 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9778003b90 con 0x7f9784106560 2024-04-22T15:39:48.356 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.470+0000 7f97737fe700 1 -- 172.21.15.16:0/2252447367 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f977801a8a0 con 0x7f9784106560 2024-04-22T15:39:48.356 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.470+0000 7f9788c70700 1 -- 172.21.15.16:0/2252447367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978419a8e0 con 0x7f9784106560 2024-04-22T15:39:48.356 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.470+0000 7f97737fe700 1 -- 172.21.15.16:0/2252447367 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9778019880 con 0x7f9784106560 2024-04-22T15:39:48.356 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.470+0000 7f9788c70700 1 -- 172.21.15.16:0/2252447367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978419ad80 con 0x7f9784106560 2024-04-22T15:39:48.356 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.471+0000 7f97737fe700 1 -- 172.21.15.16:0/2252447367 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f9778019b40 con 0x7f9784106560 2024-04-22T15:39:48.356 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.471+0000 7f9788c70700 1 -- 172.21.15.16:0/2252447367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97840623c0 con 0x7f9784106560 2024-04-22T15:39:48.356 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.471+0000 7f97737fe700 1 --2- 172.21.15.16:0/2252447367 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7f976c0382e0 0x7f976c03a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:48.356 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.471+0000 7f97737fe700 1 -- 172.21.15.16:0/2252447367 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f977804c6f0 con 0x7f9784106560 2024-04-22T15:39:48.356 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.471+0000 7f9781d9b700 1 -- 172.21.15.16:0/2252447367 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7f976c0382e0 msgr2=0x7f976c03a7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1748504461 2024-04-22T15:39:48.356 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.471+0000 7f9781d9b700 1 --2- 172.21.15.16:0/2252447367 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7f976c0382e0 0x7f976c03a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:39:48.356 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.476+0000 7f97737fe700 1 -- 172.21.15.16:0/2252447367 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f977802b730 con 0x7f9784106560 2024-04-22T15:39:48.357 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.672+0000 7f9781d9b700 1 -- 172.21.15.16:0/2252447367 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7f976c0382e0 msgr2=0x7f976c03a7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1748504461 2024-04-22T15:39:48.357 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.672+0000 7f9781d9b700 1 --2- 172.21.15.16:0/2252447367 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7f976c0382e0 0x7f976c03a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T15:39:48.357 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.732+0000 7f9788c70700 1 -- 172.21.15.16:0/2252447367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f978410a950 con 0x7f9784106560 2024-04-22T15:39:48.357 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.733+0000 7f97737fe700 1 -- 172.21.15.16:0/2252447367 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f97780172b0 con 0x7f9784106560 2024-04-22T15:39:48.357 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.735+0000 7f9788c70700 1 -- 172.21.15.16:0/2252447367 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7f976c0382e0 msgr2=0x7f976c03a7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T15:39:48.357 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.735+0000 7f9788c70700 1 --2- 172.21.15.16:0/2252447367 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7f976c0382e0 0x7f976c03a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:48.357 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.735+0000 7f9788c70700 1 -- 172.21.15.16:0/2252447367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9784106560 msgr2=0x7f978419a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:48.357 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.735+0000 7f9788c70700 1 --2- 172.21.15.16:0/2252447367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9784106560 0x7f978419a1a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9778006af0 tx=0x7f9778011a90 comp rx=0 tx=0).stop 2024-04-22T15:39:48.357 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.735+0000 7f9788c70700 1 -- 172.21.15.16:0/2252447367 shutdown_connections 2024-04-22T15:39:48.357 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.735+0000 7f9788c70700 1 --2- 172.21.15.16:0/2252447367 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7f976c0382e0 0x7f976c03a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:48.357 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.735+0000 7f9788c70700 1 --2- 172.21.15.16:0/2252447367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9784106560 0x7f978419a1a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:48.357 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.735+0000 7f9788c70700 1 -- 172.21.15.16:0/2252447367 >> 172.21.15.16:0/2252447367 conn(0x7f9784101b20 msgr2=0x7f9784102840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:48.357 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.735+0000 7f9788c70700 1 -- 172.21.15.16:0/2252447367 shutdown_connections 2024-04-22T15:39:48.358 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:47.735+0000 7f9788c70700 1 -- 172.21.15.16:0/2252447367 wait complete. 2024-04-22T15:39:48.358 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for the mgr to restart... 2024-04-22T15:39:48.358 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr epoch 5... 2024-04-22T15:39:48.858 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:48 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/2252447367' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T15:39:53.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:53 smithi016 ceph-mon[26620]: Active manager daemon smithi016.rnsnnr restarted 2024-04-22T15:39:53.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:53 smithi016 ceph-mon[26620]: Activating manager daemon smithi016.rnsnnr 2024-04-22T15:39:53.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:53 smithi016 ceph-mon[26620]: osdmap e2: 0 total, 0 up, 0 in 2024-04-22T15:39:53.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:53 smithi016 ceph-mon[26620]: mgrmap e6: smithi016.rnsnnr(active, starting, since 0.00328948s) 2024-04-22T15:39:53.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:53 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-04-22T15:39:53.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:53 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr metadata", "who": "smithi016.rnsnnr", "id": "smithi016.rnsnnr"}]: dispatch 2024-04-22T15:39:53.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:53 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T15:39:53.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:53 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T15:39:53.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:53 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T15:39:53.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:53 smithi016 ceph-mon[26620]: Manager daemon smithi016.rnsnnr is now available 2024-04-22T15:39:53.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:53 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:39:53.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:53 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:39:53.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:53 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:39:53.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:53 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:39:53.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:53 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:39:53.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:53 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi016.rnsnnr/mirror_snapshot_schedule"}]: dispatch 2024-04-22T15:39:53.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:53 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi016.rnsnnr/trash_purge_schedule"}]: dispatch 2024-04-22T15:39:54.776 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:54 smithi016 ceph-mon[26620]: Found migration_current of "None". Setting to last migration. 2024-04-22T15:39:54.776 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:54 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:39:54.776 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:54 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:39:54.776 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:54 smithi016 ceph-mon[26620]: mgrmap e7: smithi016.rnsnnr(active, since 1.00494s) 2024-04-22T15:39:55.095 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-22T15:39:55.095 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-22T15:39:55.096 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-22T15:39:55.096 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-22T15:39:55.096 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.765+0000 7fe774ccb700 1 Processor -- start 2024-04-22T15:39:55.096 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.765+0000 7fe774ccb700 1 -- start start 2024-04-22T15:39:55.096 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.765+0000 7fe774ccb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe770106560 0x7fe770106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:55.096 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.765+0000 7fe774ccb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe770106f50 con 0x7fe770106560 2024-04-22T15:39:55.096 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.765+0000 7fe76e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe770106560 0x7fe770106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:55.096 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.766+0000 7fe76e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe770106560 0x7fe770106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41290/0 (socket says 172.21.15.16:41290) 2024-04-22T15:39:55.096 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.766+0000 7fe76e59c700 1 -- 172.21.15.16:0/3810045893 learned_addr learned my addr 172.21.15.16:0/3810045893 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:55.096 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.766+0000 7fe76e59c700 1 -- 172.21.15.16:0/3810045893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe770107760 con 0x7fe770106560 2024-04-22T15:39:55.096 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.766+0000 7fe76e59c700 1 --2- 172.21.15.16:0/3810045893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe770106560 0x7fe770106980 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe7640098a0 tx=0x7fe764009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=900f17095d28e48f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:55.096 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.766+0000 7fe76d59a700 1 -- 172.21.15.16:0/3810045893 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe764004180 con 0x7fe770106560 2024-04-22T15:39:55.096 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.766+0000 7fe76d59a700 1 -- 172.21.15.16:0/3810045893 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe7640042e0 con 0x7fe770106560 2024-04-22T15:39:55.096 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.766+0000 7fe76d59a700 1 -- 172.21.15.16:0/3810045893 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe764011a50 con 0x7fe770106560 2024-04-22T15:39:55.097 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.767+0000 7fe774ccb700 1 -- 172.21.15.16:0/3810045893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe770106560 msgr2=0x7fe770106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:55.097 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.767+0000 7fe774ccb700 1 --2- 172.21.15.16:0/3810045893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe770106560 0x7fe770106980 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe7640098a0 tx=0x7fe764009bb0 comp rx=0 tx=0).stop 2024-04-22T15:39:55.097 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.767+0000 7fe774ccb700 1 -- 172.21.15.16:0/3810045893 shutdown_connections 2024-04-22T15:39:55.097 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.767+0000 7fe774ccb700 1 --2- 172.21.15.16:0/3810045893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe770106560 0x7fe770106980 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:55.097 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.767+0000 7fe774ccb700 1 -- 172.21.15.16:0/3810045893 >> 172.21.15.16:0/3810045893 conn(0x7fe770101b40 msgr2=0x7fe770103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:55.097 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.767+0000 7fe774ccb700 1 -- 172.21.15.16:0/3810045893 shutdown_connections 2024-04-22T15:39:55.097 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.767+0000 7fe774ccb700 1 -- 172.21.15.16:0/3810045893 wait complete. 2024-04-22T15:39:55.097 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.768+0000 7fe774ccb700 1 Processor -- start 2024-04-22T15:39:55.097 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.768+0000 7fe774ccb700 1 -- start start 2024-04-22T15:39:55.097 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.768+0000 7fe774ccb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe770106560 0x7fe77019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:55.097 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.768+0000 7fe774ccb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77019a700 con 0x7fe770106560 2024-04-22T15:39:55.097 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.768+0000 7fe76e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe770106560 0x7fe77019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:55.097 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.768+0000 7fe76e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe770106560 0x7fe77019a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41298/0 (socket says 172.21.15.16:41298) 2024-04-22T15:39:55.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.768+0000 7fe76e59c700 1 -- 172.21.15.16:0/1794615003 learned_addr learned my addr 172.21.15.16:0/1794615003 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:55.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.768+0000 7fe76e59c700 1 -- 172.21.15.16:0/1794615003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe764009550 con 0x7fe770106560 2024-04-22T15:39:55.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.769+0000 7fe76e59c700 1 --2- 172.21.15.16:0/1794615003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe770106560 0x7fe77019a1c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe764006af0 tx=0x7fe764011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:55.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.769+0000 7fe75f7fe700 1 -- 172.21.15.16:0/1794615003 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe764003b90 con 0x7fe770106560 2024-04-22T15:39:55.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.769+0000 7fe75f7fe700 1 -- 172.21.15.16:0/1794615003 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe76401a8a0 con 0x7fe770106560 2024-04-22T15:39:55.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.769+0000 7fe774ccb700 1 -- 172.21.15.16:0/1794615003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe77019a900 con 0x7fe770106560 2024-04-22T15:39:55.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.769+0000 7fe75f7fe700 1 -- 172.21.15.16:0/1794615003 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe764019880 con 0x7fe770106560 2024-04-22T15:39:55.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.769+0000 7fe774ccb700 1 -- 172.21.15.16:0/1794615003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe77019ada0 con 0x7fe770106560 2024-04-22T15:39:55.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.770+0000 7fe75f7fe700 1 -- 172.21.15.16:0/1794615003 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fe764019b40 con 0x7fe770106560 2024-04-22T15:39:55.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.770+0000 7fe75f7fe700 1 --2- 172.21.15.16:0/1794615003 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7fe7580382e0 0x7fe75803a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:55.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.770+0000 7fe75f7fe700 1 -- 172.21.15.16:0/1794615003 --> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe75803aeb0 con 0x7fe7580382e0 2024-04-22T15:39:55.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.770+0000 7fe75f7fe700 1 -- 172.21.15.16:0/1794615003 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe76404c6c0 con 0x7fe770106560 2024-04-22T15:39:55.098 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.770+0000 7fe76dd9b700 1 -- 172.21.15.16:0/1794615003 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7fe7580382e0 msgr2=0x7fe75803a7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1748504461 2024-04-22T15:39:55.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.770+0000 7fe76dd9b700 1 --2- 172.21.15.16:0/1794615003 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7fe7580382e0 0x7fe75803a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:39:55.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.971+0000 7fe76dd9b700 1 -- 172.21.15.16:0/1794615003 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7fe7580382e0 msgr2=0x7fe75803a7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1748504461 2024-04-22T15:39:55.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:48.971+0000 7fe76dd9b700 1 --2- 172.21.15.16:0/1794615003 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7fe7580382e0 0x7fe75803a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T15:39:55.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:49.372+0000 7fe76dd9b700 1 -- 172.21.15.16:0/1794615003 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7fe7580382e0 msgr2=0x7fe75803a7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1748504461 2024-04-22T15:39:55.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:49.372+0000 7fe76dd9b700 1 --2- 172.21.15.16:0/1794615003 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7fe7580382e0 0x7fe75803a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T15:39:55.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:50.173+0000 7fe76dd9b700 1 -- 172.21.15.16:0/1794615003 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7fe7580382e0 msgr2=0x7fe75803a7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1748504461 2024-04-22T15:39:55.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:50.173+0000 7fe76dd9b700 1 --2- 172.21.15.16:0/1794615003 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7fe7580382e0 0x7fe75803a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T15:39:55.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:51.773+0000 7fe76dd9b700 1 -- 172.21.15.16:0/1794615003 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7fe7580382e0 msgr2=0x7fe75803a7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1748504461 2024-04-22T15:39:55.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:51.773+0000 7fe76dd9b700 1 --2- 172.21.15.16:0/1794615003 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7fe7580382e0 0x7fe75803a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-22T15:39:55.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:53.481+0000 7fe75f7fe700 1 -- 172.21.15.16:0/1794615003 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mgrmap(e 6) v1 ==== 45050+0+0 (secure 0 0 0) 0x7fe76402dce0 con 0x7fe770106560 2024-04-22T15:39:55.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:53.481+0000 7fe75f7fe700 1 -- 172.21.15.16:0/1794615003 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7fe7580382e0 msgr2=0x7fe75803a7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T15:39:55.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:53.481+0000 7fe75f7fe700 1 --2- 172.21.15.16:0/1794615003 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7fe7580382e0 0x7fe75803a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:55.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:54.482+0000 7fe75f7fe700 1 -- 172.21.15.16:0/1794615003 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 7) v1 ==== 45177+0+0 (secure 0 0 0) 0x7fe76404ded0 con 0x7fe770106560 2024-04-22T15:39:55.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:54.482+0000 7fe75f7fe700 1 --2- 172.21.15.16:0/1794615003 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fe75803bc60 0x7fe75803e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:55.099 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:54.482+0000 7fe75f7fe700 1 -- 172.21.15.16:0/1794615003 --> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe75803aeb0 con 0x7fe75803bc60 2024-04-22T15:39:55.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:54.482+0000 7fe76dd9b700 1 --2- 172.21.15.16:0/1794615003 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fe75803bc60 0x7fe75803e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:55.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:54.483+0000 7fe76dd9b700 1 --2- 172.21.15.16:0/1794615003 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fe75803bc60 0x7fe75803e050 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe760003a10 tx=0x7fe760009290 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:55.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:54.484+0000 7fe75f7fe700 1 -- 172.21.15.16:0/1794615003 <== mgr.14118 v2:172.21.15.16:6800/3431522404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fe75803aeb0 con 0x7fe75803bc60 2024-04-22T15:39:55.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:54.490+0000 7fe774ccb700 1 -- 172.21.15.16:0/1794615003 --> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fe77019b500 con 0x7fe75803bc60 2024-04-22T15:39:55.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:54.490+0000 7fe75f7fe700 1 -- 172.21.15.16:0/1794615003 <== mgr.14118 v2:172.21.15.16:6800/3431522404 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fe77019b500 con 0x7fe75803bc60 2024-04-22T15:39:55.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:54.490+0000 7fe774ccb700 1 -- 172.21.15.16:0/1794615003 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fe75803bc60 msgr2=0x7fe75803e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:55.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:54.490+0000 7fe774ccb700 1 --2- 172.21.15.16:0/1794615003 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fe75803bc60 0x7fe75803e050 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe760003a10 tx=0x7fe760009290 comp rx=0 tx=0).stop 2024-04-22T15:39:55.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:54.490+0000 7fe774ccb700 1 -- 172.21.15.16:0/1794615003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe770106560 msgr2=0x7fe77019a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:55.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:54.490+0000 7fe774ccb700 1 --2- 172.21.15.16:0/1794615003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe770106560 0x7fe77019a1c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe764006af0 tx=0x7fe764011a90 comp rx=0 tx=0).stop 2024-04-22T15:39:55.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:54.490+0000 7fe774ccb700 1 -- 172.21.15.16:0/1794615003 shutdown_connections 2024-04-22T15:39:55.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:54.490+0000 7fe774ccb700 1 --2- 172.21.15.16:0/1794615003 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fe75803bc60 0x7fe75803e050 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:55.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:54.490+0000 7fe774ccb700 1 --2- 172.21.15.16:0/1794615003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe770106560 0x7fe77019a1c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:55.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:54.491+0000 7fe774ccb700 1 --2- 172.21.15.16:0/1794615003 >> [v2:172.21.15.16:6800/1748504461,v1:172.21.15.16:6801/1748504461] conn(0x7fe7580382e0 0x7fe75803a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:55.100 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:54.491+0000 7fe774ccb700 1 -- 172.21.15.16:0/1794615003 >> 172.21.15.16:0/1794615003 conn(0x7fe770101b40 msgr2=0x7fe770102870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:55.101 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:54.491+0000 7fe774ccb700 1 -- 172.21.15.16:0/1794615003 shutdown_connections 2024-04-22T15:39:55.101 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:54.491+0000 7fe774ccb700 1 -- 172.21.15.16:0/1794615003 wait complete. 2024-04-22T15:39:55.101 INFO:teuthology.orchestra.run.smithi016.stdout:mgr epoch 5 is available 2024-04-22T15:39:55.101 INFO:teuthology.orchestra.run.smithi016.stdout:Setting orchestrator backend to cephadm... 2024-04-22T15:39:55.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:55 smithi016 ceph-mon[26620]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-22T15:39:55.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:55 smithi016 ceph-mon[26620]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-22T15:39:55.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:55 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:39:56.408 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.505+0000 7f4792650700 1 Processor -- start 2024-04-22T15:39:56.408 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.505+0000 7f4792650700 1 -- start start 2024-04-22T15:39:56.408 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.506+0000 7f4792650700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f478c107020 0x7f478c109410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:56.408 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.506+0000 7f4792650700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f478c074640 con 0x7f478c107020 2024-04-22T15:39:56.408 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.506+0000 7f478bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f478c107020 0x7f478c109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:56.408 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.506+0000 7f478bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f478c107020 0x7f478c109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48314/0 (socket says 172.21.15.16:48314) 2024-04-22T15:39:56.408 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.506+0000 7f478bfff700 1 -- 172.21.15.16:0/1312950877 learned_addr learned my addr 172.21.15.16:0/1312950877 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:56.409 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.506+0000 7f478bfff700 1 -- 172.21.15.16:0/1312950877 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f478c109950 con 0x7f478c107020 2024-04-22T15:39:56.409 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.506+0000 7f478bfff700 1 --2- 172.21.15.16:0/1312950877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f478c107020 0x7f478c109410 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f477c0098a0 tx=0x7f477c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=351a9a23f691b42d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:56.409 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.506+0000 7f478affd700 1 -- 172.21.15.16:0/1312950877 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f477c004180 con 0x7f478c107020 2024-04-22T15:39:56.409 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.506+0000 7f478affd700 1 -- 172.21.15.16:0/1312950877 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f477c0042e0 con 0x7f478c107020 2024-04-22T15:39:56.409 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.506+0000 7f478affd700 1 -- 172.21.15.16:0/1312950877 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f477c011a50 con 0x7f478c107020 2024-04-22T15:39:56.409 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.507+0000 7f4792650700 1 -- 172.21.15.16:0/1312950877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f478c107020 msgr2=0x7f478c109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:56.409 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.507+0000 7f4792650700 1 --2- 172.21.15.16:0/1312950877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f478c107020 0x7f478c109410 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f477c0098a0 tx=0x7f477c009bb0 comp rx=0 tx=0).stop 2024-04-22T15:39:56.409 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.507+0000 7f4792650700 1 -- 172.21.15.16:0/1312950877 shutdown_connections 2024-04-22T15:39:56.409 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.507+0000 7f4792650700 1 --2- 172.21.15.16:0/1312950877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f478c107020 0x7f478c109410 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:56.409 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.507+0000 7f4792650700 1 -- 172.21.15.16:0/1312950877 >> 172.21.15.16:0/1312950877 conn(0x7f478c1009f0 msgr2=0x7f478c102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:56.409 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.507+0000 7f4792650700 1 -- 172.21.15.16:0/1312950877 shutdown_connections 2024-04-22T15:39:56.409 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.507+0000 7f4792650700 1 -- 172.21.15.16:0/1312950877 wait complete. 2024-04-22T15:39:56.409 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.508+0000 7f4792650700 1 Processor -- start 2024-04-22T15:39:56.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.508+0000 7f4792650700 1 -- start start 2024-04-22T15:39:56.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.508+0000 7f4792650700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f478c107020 0x7f478c19a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:56.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.508+0000 7f4792650700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f478c19a680 con 0x7f478c107020 2024-04-22T15:39:56.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.508+0000 7f478bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f478c107020 0x7f478c19a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:56.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.508+0000 7f478bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f478c107020 0x7f478c19a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48330/0 (socket says 172.21.15.16:48330) 2024-04-22T15:39:56.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.508+0000 7f478bfff700 1 -- 172.21.15.16:0/2378889202 learned_addr learned my addr 172.21.15.16:0/2378889202 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:56.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.508+0000 7f478bfff700 1 -- 172.21.15.16:0/2378889202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477c009550 con 0x7f478c107020 2024-04-22T15:39:56.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.508+0000 7f478bfff700 1 --2- 172.21.15.16:0/2378889202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f478c107020 0x7f478c19a140 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f477c0119e0 tx=0x7f477c011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:56.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.509+0000 7f47897fa700 1 -- 172.21.15.16:0/2378889202 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f477c003b90 con 0x7f478c107020 2024-04-22T15:39:56.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.509+0000 7f47897fa700 1 -- 172.21.15.16:0/2378889202 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f477c01a8a0 con 0x7f478c107020 2024-04-22T15:39:56.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.509+0000 7f4792650700 1 -- 172.21.15.16:0/2378889202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f478c19a880 con 0x7f478c107020 2024-04-22T15:39:56.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.509+0000 7f47897fa700 1 -- 172.21.15.16:0/2378889202 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f477c019930 con 0x7f478c107020 2024-04-22T15:39:56.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.509+0000 7f4792650700 1 -- 172.21.15.16:0/2378889202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f478c19ad20 con 0x7f478c107020 2024-04-22T15:39:56.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.509+0000 7f47897fa700 1 -- 172.21.15.16:0/2378889202 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 7) v1 ==== 45177+0+0 (secure 0 0 0) 0x7f477c019ba0 con 0x7f478c107020 2024-04-22T15:39:56.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.510+0000 7f47897fa700 1 --2- 172.21.15.16:0/2378889202 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f4774038210 0x7f477403a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:56.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.510+0000 7f47897fa700 1 -- 172.21.15.16:0/2378889202 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f477c04c550 con 0x7f478c107020 2024-04-22T15:39:56.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.510+0000 7f478b7fe700 1 --2- 172.21.15.16:0/2378889202 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f4774038210 0x7f477403a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:56.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.510+0000 7f4792650700 1 -- 172.21.15.16:0/2378889202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f478c0623c0 con 0x7f478c107020 2024-04-22T15:39:56.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.510+0000 7f478b7fe700 1 --2- 172.21.15.16:0/2378889202 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f4774038210 0x7f477403a6d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4780006fd0 tx=0x7f4780006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:56.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.515+0000 7f47897fa700 1 -- 172.21.15.16:0/2378889202 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f477c02b930 con 0x7f478c107020 2024-04-22T15:39:56.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.536+0000 7f47897fa700 1 -- 172.21.15.16:0/2378889202 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f477c02c3a0 con 0x7f478c107020 2024-04-22T15:39:56.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.736+0000 7f4792650700 1 -- 172.21.15.16:0/2378889202 --> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f478c19d7a0 con 0x7f4774038210 2024-04-22T15:39:56.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.742+0000 7f47897fa700 1 -- 172.21.15.16:0/2378889202 <== mgr.14118 v2:172.21.15.16:6800/3431522404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f478c19d7a0 con 0x7f4774038210 2024-04-22T15:39:56.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.744+0000 7f4792650700 1 -- 172.21.15.16:0/2378889202 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f4774038210 msgr2=0x7f477403a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:56.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.744+0000 7f4792650700 1 --2- 172.21.15.16:0/2378889202 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f4774038210 0x7f477403a6d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4780006fd0 tx=0x7f4780006e40 comp rx=0 tx=0).stop 2024-04-22T15:39:56.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.744+0000 7f4792650700 1 -- 172.21.15.16:0/2378889202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f478c107020 msgr2=0x7f478c19a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:56.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.744+0000 7f4792650700 1 --2- 172.21.15.16:0/2378889202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f478c107020 0x7f478c19a140 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f477c0119e0 tx=0x7f477c011ac0 comp rx=0 tx=0).stop 2024-04-22T15:39:56.412 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.745+0000 7f4792650700 1 -- 172.21.15.16:0/2378889202 shutdown_connections 2024-04-22T15:39:56.412 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.745+0000 7f4792650700 1 --2- 172.21.15.16:0/2378889202 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f4774038210 0x7f477403a6d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:56.412 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.745+0000 7f4792650700 1 --2- 172.21.15.16:0/2378889202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f478c107020 0x7f478c19a140 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:56.412 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.745+0000 7f4792650700 1 -- 172.21.15.16:0/2378889202 >> 172.21.15.16:0/2378889202 conn(0x7f478c1009f0 msgr2=0x7f478c101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:56.412 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.745+0000 7f4792650700 1 -- 172.21.15.16:0/2378889202 shutdown_connections 2024-04-22T15:39:56.412 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:55.745+0000 7f4792650700 1 -- 172.21.15.16:0/2378889202 wait complete. 2024-04-22T15:39:56.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:56 smithi016 ceph-mon[26620]: [22/Apr/2024:15:39:54] ENGINE Bus STARTING 2024-04-22T15:39:56.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:56 smithi016 ceph-mon[26620]: [22/Apr/2024:15:39:55] ENGINE Serving on http://172.21.15.16:8765 2024-04-22T15:39:56.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:56 smithi016 ceph-mon[26620]: [22/Apr/2024:15:39:55] ENGINE Serving on https://172.21.15.16:7150 2024-04-22T15:39:56.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:56 smithi016 ceph-mon[26620]: [22/Apr/2024:15:39:55] ENGINE Bus STARTED 2024-04-22T15:39:56.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:56 smithi016 ceph-mon[26620]: mgrmap e8: smithi016.rnsnnr(active, since 2s) 2024-04-22T15:39:56.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:56 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:39:56.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:56 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:39:57.574 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:57 smithi016 ceph-mon[26620]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:39:57.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-22T15:39:57.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.805+0000 7f7d8c76e700 1 Processor -- start 2024-04-22T15:39:57.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.805+0000 7f7d8c76e700 1 -- start start 2024-04-22T15:39:57.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.805+0000 7f7d8c76e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d84107020 0x7f7d84109410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:57.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.805+0000 7f7d8c76e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d84074640 con 0x7f7d84107020 2024-04-22T15:39:57.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.805+0000 7f7d8a50a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d84107020 0x7f7d84109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:57.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.805+0000 7f7d8a50a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d84107020 0x7f7d84109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48334/0 (socket says 172.21.15.16:48334) 2024-04-22T15:39:57.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.805+0000 7f7d8a50a700 1 -- 172.21.15.16:0/3847798476 learned_addr learned my addr 172.21.15.16:0/3847798476 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:57.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.806+0000 7f7d8a50a700 1 -- 172.21.15.16:0/3847798476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d84109950 con 0x7f7d84107020 2024-04-22T15:39:57.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.806+0000 7f7d8a50a700 1 --2- 172.21.15.16:0/3847798476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d84107020 0x7f7d84109410 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7d800098a0 tx=0x7f7d80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af3d257cf1de8e13 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:57.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.806+0000 7f7d89508700 1 -- 172.21.15.16:0/3847798476 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d80004180 con 0x7f7d84107020 2024-04-22T15:39:57.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.806+0000 7f7d89508700 1 -- 172.21.15.16:0/3847798476 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7d800042e0 con 0x7f7d84107020 2024-04-22T15:39:57.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.806+0000 7f7d89508700 1 -- 172.21.15.16:0/3847798476 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d80011a50 con 0x7f7d84107020 2024-04-22T15:39:57.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.806+0000 7f7d8c76e700 1 -- 172.21.15.16:0/3847798476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d84107020 msgr2=0x7f7d84109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:57.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.806+0000 7f7d8c76e700 1 --2- 172.21.15.16:0/3847798476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d84107020 0x7f7d84109410 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7d800098a0 tx=0x7f7d80009bb0 comp rx=0 tx=0).stop 2024-04-22T15:39:57.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.807+0000 7f7d8c76e700 1 -- 172.21.15.16:0/3847798476 shutdown_connections 2024-04-22T15:39:57.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.807+0000 7f7d8c76e700 1 --2- 172.21.15.16:0/3847798476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d84107020 0x7f7d84109410 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:57.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.807+0000 7f7d8c76e700 1 -- 172.21.15.16:0/3847798476 >> 172.21.15.16:0/3847798476 conn(0x7f7d841009f0 msgr2=0x7f7d84102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:57.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.807+0000 7f7d8c76e700 1 -- 172.21.15.16:0/3847798476 shutdown_connections 2024-04-22T15:39:57.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.807+0000 7f7d8c76e700 1 -- 172.21.15.16:0/3847798476 wait complete. 2024-04-22T15:39:57.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.807+0000 7f7d8c76e700 1 Processor -- start 2024-04-22T15:39:57.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.807+0000 7f7d8c76e700 1 -- start start 2024-04-22T15:39:57.585 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.808+0000 7f7d8c76e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d84107020 0x7f7d8419a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:57.585 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.808+0000 7f7d8c76e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8419a680 con 0x7f7d84107020 2024-04-22T15:39:57.585 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.808+0000 7f7d8a50a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d84107020 0x7f7d8419a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:57.585 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.808+0000 7f7d8a50a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d84107020 0x7f7d8419a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48344/0 (socket says 172.21.15.16:48344) 2024-04-22T15:39:57.585 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.808+0000 7f7d8a50a700 1 -- 172.21.15.16:0/1906706241 learned_addr learned my addr 172.21.15.16:0/1906706241 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:57.585 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.808+0000 7f7d8a50a700 1 -- 172.21.15.16:0/1906706241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d80009550 con 0x7f7d84107020 2024-04-22T15:39:57.585 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.808+0000 7f7d8a50a700 1 --2- 172.21.15.16:0/1906706241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d84107020 0x7f7d8419a140 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7d800119e0 tx=0x7f7d80011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:57.585 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.808+0000 7f7d7b7fe700 1 -- 172.21.15.16:0/1906706241 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d80003b90 con 0x7f7d84107020 2024-04-22T15:39:57.585 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.808+0000 7f7d7b7fe700 1 -- 172.21.15.16:0/1906706241 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7d8001a8a0 con 0x7f7d84107020 2024-04-22T15:39:57.585 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.808+0000 7f7d8c76e700 1 -- 172.21.15.16:0/1906706241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d8419a880 con 0x7f7d84107020 2024-04-22T15:39:57.585 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.808+0000 7f7d7b7fe700 1 -- 172.21.15.16:0/1906706241 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d80019930 con 0x7f7d84107020 2024-04-22T15:39:57.585 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.808+0000 7f7d8c76e700 1 -- 172.21.15.16:0/1906706241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d8419ad20 con 0x7f7d84107020 2024-04-22T15:39:57.585 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.809+0000 7f7d7b7fe700 1 -- 172.21.15.16:0/1906706241 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f7d80019ba0 con 0x7f7d84107020 2024-04-22T15:39:57.585 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.809+0000 7f7d7b7fe700 1 --2- 172.21.15.16:0/1906706241 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f7d70038330 0x7f7d7003a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:57.586 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.809+0000 7f7d7b7fe700 1 -- 172.21.15.16:0/1906706241 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7d8004c6b0 con 0x7f7d84107020 2024-04-22T15:39:57.586 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.809+0000 7f7d8c76e700 1 -- 172.21.15.16:0/1906706241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d840623c0 con 0x7f7d84107020 2024-04-22T15:39:57.586 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.809+0000 7f7d89d09700 1 --2- 172.21.15.16:0/1906706241 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f7d70038330 0x7f7d7003a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:57.586 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.810+0000 7f7d89d09700 1 --2- 172.21.15.16:0/1906706241 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f7d70038330 0x7f7d7003a7f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7d74006fd0 tx=0x7f7d74006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:57.586 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:56.814+0000 7f7d7b7fe700 1 -- 172.21.15.16:0/1906706241 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7d8002d440 con 0x7f7d84107020 2024-04-22T15:39:57.586 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.014+0000 7f7d8c76e700 1 -- 172.21.15.16:0/1906706241 --> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d8419d7a0 con 0x7f7d70038330 2024-04-22T15:39:57.586 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.015+0000 7f7d7b7fe700 1 -- 172.21.15.16:0/1906706241 <== mgr.14118 v2:172.21.15.16:6800/3431522404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f7d8419d7a0 con 0x7f7d70038330 2024-04-22T15:39:57.586 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.018+0000 7f7d8c76e700 1 -- 172.21.15.16:0/1906706241 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f7d70038330 msgr2=0x7f7d7003a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:57.586 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.018+0000 7f7d8c76e700 1 --2- 172.21.15.16:0/1906706241 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f7d70038330 0x7f7d7003a7f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7d74006fd0 tx=0x7f7d74006e40 comp rx=0 tx=0).stop 2024-04-22T15:39:57.586 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.018+0000 7f7d8c76e700 1 -- 172.21.15.16:0/1906706241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d84107020 msgr2=0x7f7d8419a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:57.586 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.018+0000 7f7d8c76e700 1 --2- 172.21.15.16:0/1906706241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d84107020 0x7f7d8419a140 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7d800119e0 tx=0x7f7d80011ac0 comp rx=0 tx=0).stop 2024-04-22T15:39:57.587 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.018+0000 7f7d8c76e700 1 -- 172.21.15.16:0/1906706241 shutdown_connections 2024-04-22T15:39:57.587 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.018+0000 7f7d8c76e700 1 --2- 172.21.15.16:0/1906706241 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f7d70038330 0x7f7d7003a7f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:57.587 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.018+0000 7f7d8c76e700 1 --2- 172.21.15.16:0/1906706241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d84107020 0x7f7d8419a140 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:57.587 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.018+0000 7f7d8c76e700 1 -- 172.21.15.16:0/1906706241 >> 172.21.15.16:0/1906706241 conn(0x7f7d841009f0 msgr2=0x7f7d84101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:57.587 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.018+0000 7f7d8c76e700 1 -- 172.21.15.16:0/1906706241 shutdown_connections 2024-04-22T15:39:57.587 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.018+0000 7f7d8c76e700 1 -- 172.21.15.16:0/1906706241 wait complete. 2024-04-22T15:39:57.587 INFO:teuthology.orchestra.run.smithi016.stdout:Generating ssh key... 2024-04-22T15:39:58.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:58 smithi016 ceph-mon[26620]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:39:58.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:58 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:39:58.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:58 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:39:58.946 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.979+0000 7fbde86a4700 1 Processor -- start 2024-04-22T15:39:58.946 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.979+0000 7fbde86a4700 1 -- start start 2024-04-22T15:39:58.946 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.980+0000 7fbde86a4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbde007be60 0x7fbde007a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:58.946 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.980+0000 7fbde86a4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde007a890 con 0x7fbde007be60 2024-04-22T15:39:58.946 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.980+0000 7fbde6440700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbde007be60 0x7fbde007a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:58.946 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.980+0000 7fbde6440700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbde007be60 0x7fbde007a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48350/0 (socket says 172.21.15.16:48350) 2024-04-22T15:39:58.946 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.980+0000 7fbde6440700 1 -- 172.21.15.16:0/2302550435 learned_addr learned my addr 172.21.15.16:0/2302550435 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:58.946 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.980+0000 7fbde6440700 1 -- 172.21.15.16:0/2302550435 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde007a9d0 con 0x7fbde007be60 2024-04-22T15:39:58.947 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.980+0000 7fbde6440700 1 --2- 172.21.15.16:0/2302550435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbde007be60 0x7fbde007a2c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbddc0098a0 tx=0x7fbddc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f39e0b744b8fb934 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:58.947 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.980+0000 7fbde543e700 1 -- 172.21.15.16:0/2302550435 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbddc004180 con 0x7fbde007be60 2024-04-22T15:39:58.947 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.980+0000 7fbde543e700 1 -- 172.21.15.16:0/2302550435 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbddc0042e0 con 0x7fbde007be60 2024-04-22T15:39:58.947 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.980+0000 7fbde543e700 1 -- 172.21.15.16:0/2302550435 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbddc011a50 con 0x7fbde007be60 2024-04-22T15:39:58.947 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.981+0000 7fbde86a4700 1 -- 172.21.15.16:0/2302550435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbde007be60 msgr2=0x7fbde007a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:58.947 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.981+0000 7fbde86a4700 1 --2- 172.21.15.16:0/2302550435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbde007be60 0x7fbde007a2c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbddc0098a0 tx=0x7fbddc009bb0 comp rx=0 tx=0).stop 2024-04-22T15:39:58.947 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.981+0000 7fbde86a4700 1 -- 172.21.15.16:0/2302550435 shutdown_connections 2024-04-22T15:39:58.947 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.981+0000 7fbde86a4700 1 --2- 172.21.15.16:0/2302550435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbde007be60 0x7fbde007a2c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:58.947 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.981+0000 7fbde86a4700 1 -- 172.21.15.16:0/2302550435 >> 172.21.15.16:0/2302550435 conn(0x7fbde0101b40 msgr2=0x7fbde0103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:58.947 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.981+0000 7fbde86a4700 1 -- 172.21.15.16:0/2302550435 shutdown_connections 2024-04-22T15:39:58.947 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.981+0000 7fbde86a4700 1 -- 172.21.15.16:0/2302550435 wait complete. 2024-04-22T15:39:58.948 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.982+0000 7fbde86a4700 1 Processor -- start 2024-04-22T15:39:58.948 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.982+0000 7fbde86a4700 1 -- start start 2024-04-22T15:39:58.948 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.982+0000 7fbde86a4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbde007be60 0x7fbde019c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:58.948 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.982+0000 7fbde86a4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde019c910 con 0x7fbde007be60 2024-04-22T15:39:58.948 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.982+0000 7fbde6440700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbde007be60 0x7fbde019c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:58.948 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.982+0000 7fbde6440700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbde007be60 0x7fbde019c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48362/0 (socket says 172.21.15.16:48362) 2024-04-22T15:39:58.948 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.982+0000 7fbde6440700 1 -- 172.21.15.16:0/1663992698 learned_addr learned my addr 172.21.15.16:0/1663992698 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:39:58.948 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.982+0000 7fbde6440700 1 -- 172.21.15.16:0/1663992698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbddc009550 con 0x7fbde007be60 2024-04-22T15:39:58.948 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.983+0000 7fbde6440700 1 --2- 172.21.15.16:0/1663992698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbde007be60 0x7fbde019c3d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbddc003f80 tx=0x7fbddc011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:58.948 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.983+0000 7fbdd77fe700 1 -- 172.21.15.16:0/1663992698 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbddc003a80 con 0x7fbde007be60 2024-04-22T15:39:58.948 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.983+0000 7fbdd77fe700 1 -- 172.21.15.16:0/1663992698 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbddc003be0 con 0x7fbde007be60 2024-04-22T15:39:58.948 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.983+0000 7fbde86a4700 1 -- 172.21.15.16:0/1663992698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde019cb10 con 0x7fbde007be60 2024-04-22T15:39:58.949 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.983+0000 7fbdd77fe700 1 -- 172.21.15.16:0/1663992698 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbddc019820 con 0x7fbde007be60 2024-04-22T15:39:58.949 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.983+0000 7fbde86a4700 1 -- 172.21.15.16:0/1663992698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde019cfb0 con 0x7fbde007be60 2024-04-22T15:39:58.949 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.984+0000 7fbdd77fe700 1 -- 172.21.15.16:0/1663992698 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fbddc019aa0 con 0x7fbde007be60 2024-04-22T15:39:58.949 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.984+0000 7fbde86a4700 1 -- 172.21.15.16:0/1663992698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbde010c8e0 con 0x7fbde007be60 2024-04-22T15:39:58.949 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.984+0000 7fbdd77fe700 1 --2- 172.21.15.16:0/1663992698 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fbdcc0382d0 0x7fbdcc03a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:39:58.949 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.984+0000 7fbdd77fe700 1 -- 172.21.15.16:0/1663992698 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fbddc04c580 con 0x7fbde007be60 2024-04-22T15:39:58.949 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.984+0000 7fbde5c3f700 1 --2- 172.21.15.16:0/1663992698 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fbdcc0382d0 0x7fbdcc03a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:39:58.949 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.985+0000 7fbde5c3f700 1 --2- 172.21.15.16:0/1663992698 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fbdcc0382d0 0x7fbdcc03a790 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0006fd0 tx=0x7fbdd0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:39:58.949 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:57.989+0000 7fbdd77fe700 1 -- 172.21.15.16:0/1663992698 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fbddc023b90 con 0x7fbde007be60 2024-04-22T15:39:58.949 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:58.186+0000 7fbde86a4700 1 -- 172.21.15.16:0/1663992698 --> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fbde0102fe0 con 0x7fbdcc0382d0 2024-04-22T15:39:58.949 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:58.367+0000 7fbdd77fe700 1 -- 172.21.15.16:0/1663992698 <== mgr.14118 v2:172.21.15.16:6800/3431522404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbde0102fe0 con 0x7fbdcc0382d0 2024-04-22T15:39:58.949 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:58.369+0000 7fbde86a4700 1 -- 172.21.15.16:0/1663992698 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fbdcc0382d0 msgr2=0x7fbdcc03a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:58.950 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:58.369+0000 7fbde86a4700 1 --2- 172.21.15.16:0/1663992698 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fbdcc0382d0 0x7fbdcc03a790 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0006fd0 tx=0x7fbdd0006e40 comp rx=0 tx=0).stop 2024-04-22T15:39:58.950 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:58.369+0000 7fbde86a4700 1 -- 172.21.15.16:0/1663992698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbde007be60 msgr2=0x7fbde019c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:39:58.950 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:58.369+0000 7fbde86a4700 1 --2- 172.21.15.16:0/1663992698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbde007be60 0x7fbde019c3d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbddc003f80 tx=0x7fbddc011ab0 comp rx=0 tx=0).stop 2024-04-22T15:39:58.950 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:58.369+0000 7fbde86a4700 1 -- 172.21.15.16:0/1663992698 shutdown_connections 2024-04-22T15:39:58.950 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:58.369+0000 7fbde86a4700 1 --2- 172.21.15.16:0/1663992698 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fbdcc0382d0 0x7fbdcc03a790 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:58.950 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:58.369+0000 7fbde86a4700 1 --2- 172.21.15.16:0/1663992698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbde007be60 0x7fbde019c3d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:39:58.950 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:58.369+0000 7fbde86a4700 1 -- 172.21.15.16:0/1663992698 >> 172.21.15.16:0/1663992698 conn(0x7fbde0101b40 msgr2=0x7fbde01028d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:39:58.950 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:58.369+0000 7fbde86a4700 1 -- 172.21.15.16:0/1663992698 shutdown_connections 2024-04-22T15:39:58.950 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:58.369+0000 7fbde86a4700 1 -- 172.21.15.16:0/1663992698 wait complete. 2024-04-22T15:39:59.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:59 smithi016 ceph-mon[26620]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:39:59.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:39:59 smithi016 ceph-mon[26620]: Generating ssh key... 2024-04-22T15:40:00.281 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDgwd3lqMl8Pw2fyQ3Yu3jAfGhJJ0nbJXvFNmm0PPe3caGzhNSMo72g9z5E7MV02q7sdBHmpo4dvW30aQKZGFOthk5tN+cA7U0+dkvz0hMLB+ld1JF48ldJYZf/5KF1NPVQVBWHwwOeJOK4Ojzivp/14d96JsM8fgDfA+v27vXTQ4otCnY8JAhuecxbdae8/jfVvBkl0G7V2izpWbu2pk/vt2YHeukTfD2e99yK0H39Y8FQCr4Zlz4aqO7QZ/pVeCiYXCwB0vnNVsGjAqeOWwIDPEB77+xPrLYSauuHf9oM4kfam4XP3+o1xwvagMx8fPd643BpCHSwdSSSfzCJL6auGHCIyDl5KlYDBHlmtMoWUTQnIc9H4zWL2VqkMAB38ggJKEl/UhucAFGvnrOAH190zTUSfv7Z0Y4P7RbjnLc+9ozeW49t6iK57LtxUXO3k3MIwIF9vUuJJc3/1r9FUC1oUfzAbPJjc17lpzQDzjZk157X2HSM4vVlMmuMBpJszjE= ceph-67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T15:40:00.281 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.485+0000 7f2efec24700 1 Processor -- start 2024-04-22T15:40:00.281 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.485+0000 7f2efec24700 1 -- start start 2024-04-22T15:40:00.281 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.485+0000 7f2efec24700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ef8105470 0x7f2ef8105890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:00.281 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.485+0000 7f2efec24700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef8105e60 con 0x7f2ef8105470 2024-04-22T15:40:00.281 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.485+0000 7f2efc9c0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ef8105470 0x7f2ef8105890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:00.281 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.485+0000 7f2efc9c0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ef8105470 0x7f2ef8105890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48364/0 (socket says 172.21.15.16:48364) 2024-04-22T15:40:00.281 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.485+0000 7f2efc9c0700 1 -- 172.21.15.16:0/650783870 learned_addr learned my addr 172.21.15.16:0/650783870 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:00.282 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.485+0000 7f2efc9c0700 1 -- 172.21.15.16:0/650783870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef8106670 con 0x7f2ef8105470 2024-04-22T15:40:00.282 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.486+0000 7f2efc9c0700 1 --2- 172.21.15.16:0/650783870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ef8105470 0x7f2ef8105890 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2ee00098a0 tx=0x7f2ee0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f8a6171ede1387 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:00.282 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.486+0000 7f2ef77fe700 1 -- 172.21.15.16:0/650783870 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ee0004180 con 0x7f2ef8105470 2024-04-22T15:40:00.282 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.486+0000 7f2ef77fe700 1 -- 172.21.15.16:0/650783870 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2ee00042e0 con 0x7f2ef8105470 2024-04-22T15:40:00.282 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.486+0000 7f2ef77fe700 1 -- 172.21.15.16:0/650783870 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ee0011a50 con 0x7f2ef8105470 2024-04-22T15:40:00.282 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.486+0000 7f2efec24700 1 -- 172.21.15.16:0/650783870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ef8105470 msgr2=0x7f2ef8105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:00.282 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.486+0000 7f2efec24700 1 --2- 172.21.15.16:0/650783870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ef8105470 0x7f2ef8105890 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2ee00098a0 tx=0x7f2ee0009bb0 comp rx=0 tx=0).stop 2024-04-22T15:40:00.282 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.486+0000 7f2efec24700 1 -- 172.21.15.16:0/650783870 shutdown_connections 2024-04-22T15:40:00.282 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.486+0000 7f2efec24700 1 --2- 172.21.15.16:0/650783870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ef8105470 0x7f2ef8105890 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:00.282 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.486+0000 7f2efec24700 1 -- 172.21.15.16:0/650783870 >> 172.21.15.16:0/650783870 conn(0x7f2ef81009f0 msgr2=0x7f2ef8102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:00.282 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.487+0000 7f2efec24700 1 -- 172.21.15.16:0/650783870 shutdown_connections 2024-04-22T15:40:00.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.487+0000 7f2efec24700 1 -- 172.21.15.16:0/650783870 wait complete. 2024-04-22T15:40:00.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.487+0000 7f2efec24700 1 Processor -- start 2024-04-22T15:40:00.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.487+0000 7f2efec24700 1 -- start start 2024-04-22T15:40:00.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.487+0000 7f2efec24700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ef8105470 0x7f2ef819a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:00.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.487+0000 7f2efec24700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef819a6e0 con 0x7f2ef8105470 2024-04-22T15:40:00.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.488+0000 7f2efc9c0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ef8105470 0x7f2ef819a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:00.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.488+0000 7f2efc9c0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ef8105470 0x7f2ef819a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48372/0 (socket says 172.21.15.16:48372) 2024-04-22T15:40:00.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.488+0000 7f2efc9c0700 1 -- 172.21.15.16:0/1950911766 learned_addr learned my addr 172.21.15.16:0/1950911766 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:00.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.488+0000 7f2efc9c0700 1 -- 172.21.15.16:0/1950911766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee0009550 con 0x7f2ef8105470 2024-04-22T15:40:00.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.488+0000 7f2efc9c0700 1 --2- 172.21.15.16:0/1950911766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ef8105470 0x7f2ef819a1a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0000c00 tx=0x7f2ee0011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:00.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.488+0000 7f2ef5ffb700 1 -- 172.21.15.16:0/1950911766 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ee0003b90 con 0x7f2ef8105470 2024-04-22T15:40:00.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.488+0000 7f2ef5ffb700 1 -- 172.21.15.16:0/1950911766 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2ee00248b0 con 0x7f2ef8105470 2024-04-22T15:40:00.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.488+0000 7f2efec24700 1 -- 172.21.15.16:0/1950911766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef819a8e0 con 0x7f2ef8105470 2024-04-22T15:40:00.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.488+0000 7f2ef5ffb700 1 -- 172.21.15.16:0/1950911766 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ee001a8e0 con 0x7f2ef8105470 2024-04-22T15:40:00.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.488+0000 7f2efec24700 1 -- 172.21.15.16:0/1950911766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef819ad80 con 0x7f2ef8105470 2024-04-22T15:40:00.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.489+0000 7f2ef5ffb700 1 -- 172.21.15.16:0/1950911766 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f2ee001aba0 con 0x7f2ef8105470 2024-04-22T15:40:00.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.489+0000 7f2efec24700 1 -- 172.21.15.16:0/1950911766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef80623c0 con 0x7f2ef8105470 2024-04-22T15:40:00.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.490+0000 7f2ef5ffb700 1 --2- 172.21.15.16:0/1950911766 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f2ee4038320 0x7f2ee403a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:00.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.490+0000 7f2ef5ffb700 1 -- 172.21.15.16:0/1950911766 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2ee004c590 con 0x7f2ef8105470 2024-04-22T15:40:00.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.490+0000 7f2ef7fff700 1 --2- 172.21.15.16:0/1950911766 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f2ee4038320 0x7f2ee403a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:00.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.490+0000 7f2ef7fff700 1 --2- 172.21.15.16:0/1950911766 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f2ee4038320 0x7f2ee403a7e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8006fd0 tx=0x7f2ee8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:00.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.494+0000 7f2ef5ffb700 1 -- 172.21.15.16:0/1950911766 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2ee00189a0 con 0x7f2ef8105470 2024-04-22T15:40:00.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.697+0000 7f2efec24700 1 -- 172.21.15.16:0/1950911766 --> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ef8102b40 con 0x7f2ee4038320 2024-04-22T15:40:00.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.697+0000 7f2ef5ffb700 1 -- 172.21.15.16:0/1950911766 <== mgr.14118 v2:172.21.15.16:6800/3431522404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f2ef8102b40 con 0x7f2ee4038320 2024-04-22T15:40:00.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.699+0000 7f2efec24700 1 -- 172.21.15.16:0/1950911766 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f2ee4038320 msgr2=0x7f2ee403a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:00.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.699+0000 7f2efec24700 1 --2- 172.21.15.16:0/1950911766 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f2ee4038320 0x7f2ee403a7e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8006fd0 tx=0x7f2ee8006e40 comp rx=0 tx=0).stop 2024-04-22T15:40:00.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.699+0000 7f2efec24700 1 -- 172.21.15.16:0/1950911766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ef8105470 msgr2=0x7f2ef819a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:00.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.699+0000 7f2efec24700 1 --2- 172.21.15.16:0/1950911766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ef8105470 0x7f2ef819a1a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0000c00 tx=0x7f2ee0011ab0 comp rx=0 tx=0).stop 2024-04-22T15:40:00.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.699+0000 7f2efec24700 1 -- 172.21.15.16:0/1950911766 shutdown_connections 2024-04-22T15:40:00.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.699+0000 7f2efec24700 1 --2- 172.21.15.16:0/1950911766 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f2ee4038320 0x7f2ee403a7e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:00.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.699+0000 7f2efec24700 1 --2- 172.21.15.16:0/1950911766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ef8105470 0x7f2ef819a1a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:00.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.699+0000 7f2efec24700 1 -- 172.21.15.16:0/1950911766 >> 172.21.15.16:0/1950911766 conn(0x7f2ef81009f0 msgr2=0x7f2ef80740b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:00.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.699+0000 7f2efec24700 1 -- 172.21.15.16:0/1950911766 shutdown_connections 2024-04-22T15:40:00.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:39:59.699+0000 7f2efec24700 1 -- 172.21.15.16:0/1950911766 wait complete. 2024-04-22T15:40:00.285 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-22T15:40:00.285 INFO:teuthology.orchestra.run.smithi016.stdout:Adding key to root@localhost authorized_keys... 2024-04-22T15:40:00.285 INFO:teuthology.orchestra.run.smithi016.stdout:Adding host smithi016... 2024-04-22T15:40:01.884 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:01 smithi016 ceph-mon[26620]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:02.852 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:02 smithi016 ceph-mon[26620]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi016", "addr": "172.21.15.16", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:02.852 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:02 smithi016 ceph-mon[26620]: Deploying cephadm binary to smithi016 2024-04-22T15:40:03.988 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Added host 'smithi016' with addr '172.21.15.16' 2024-04-22T15:40:03.988 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.754+0000 7f5c96272700 1 Processor -- start 2024-04-22T15:40:03.988 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.754+0000 7f5c96272700 1 -- start start 2024-04-22T15:40:03.988 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.754+0000 7f5c96272700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c90105450 0x7f5c90105870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:03.988 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.754+0000 7f5c96272700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c90105e40 con 0x7f5c90105450 2024-04-22T15:40:03.988 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.754+0000 7f5c8ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c90105450 0x7f5c90105870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:03.989 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.755+0000 7f5c8ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c90105450 0x7f5c90105870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48378/0 (socket says 172.21.15.16:48378) 2024-04-22T15:40:03.989 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.755+0000 7f5c8ffff700 1 -- 172.21.15.16:0/450967278 learned_addr learned my addr 172.21.15.16:0/450967278 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:03.989 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.755+0000 7f5c8ffff700 1 -- 172.21.15.16:0/450967278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c90106650 con 0x7f5c90105450 2024-04-22T15:40:03.989 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.755+0000 7f5c8ffff700 1 --2- 172.21.15.16:0/450967278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c90105450 0x7f5c90105870 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5c780098a0 tx=0x7f5c78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de818adf9b86397a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:03.989 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.755+0000 7f5c8effd700 1 -- 172.21.15.16:0/450967278 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c78004180 con 0x7f5c90105450 2024-04-22T15:40:03.989 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.755+0000 7f5c8effd700 1 -- 172.21.15.16:0/450967278 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5c780042e0 con 0x7f5c90105450 2024-04-22T15:40:03.989 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.755+0000 7f5c8effd700 1 -- 172.21.15.16:0/450967278 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c78011a50 con 0x7f5c90105450 2024-04-22T15:40:03.989 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.756+0000 7f5c96272700 1 -- 172.21.15.16:0/450967278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c90105450 msgr2=0x7f5c90105870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:03.989 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.756+0000 7f5c96272700 1 --2- 172.21.15.16:0/450967278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c90105450 0x7f5c90105870 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5c780098a0 tx=0x7f5c78009bb0 comp rx=0 tx=0).stop 2024-04-22T15:40:03.989 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.756+0000 7f5c96272700 1 -- 172.21.15.16:0/450967278 shutdown_connections 2024-04-22T15:40:03.989 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.756+0000 7f5c96272700 1 --2- 172.21.15.16:0/450967278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c90105450 0x7f5c90105870 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:03.989 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.756+0000 7f5c96272700 1 -- 172.21.15.16:0/450967278 >> 172.21.15.16:0/450967278 conn(0x7f5c901009b0 msgr2=0x7f5c90102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:03.989 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.756+0000 7f5c96272700 1 -- 172.21.15.16:0/450967278 shutdown_connections 2024-04-22T15:40:03.990 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.756+0000 7f5c96272700 1 -- 172.21.15.16:0/450967278 wait complete. 2024-04-22T15:40:03.990 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.756+0000 7f5c96272700 1 Processor -- start 2024-04-22T15:40:03.990 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.757+0000 7f5c96272700 1 -- start start 2024-04-22T15:40:03.990 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.757+0000 7f5c96272700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c90105450 0x7f5c90198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:03.990 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.757+0000 7f5c96272700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c90198580 con 0x7f5c90105450 2024-04-22T15:40:03.990 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.757+0000 7f5c8ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c90105450 0x7f5c90198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:03.990 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.757+0000 7f5c8ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c90105450 0x7f5c90198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48392/0 (socket says 172.21.15.16:48392) 2024-04-22T15:40:03.990 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.757+0000 7f5c8ffff700 1 -- 172.21.15.16:0/1367950944 learned_addr learned my addr 172.21.15.16:0/1367950944 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:03.990 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.757+0000 7f5c8ffff700 1 -- 172.21.15.16:0/1367950944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c78009550 con 0x7f5c90105450 2024-04-22T15:40:03.990 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.757+0000 7f5c8ffff700 1 --2- 172.21.15.16:0/1367950944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c90105450 0x7f5c90198040 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5c78009520 tx=0x7f5c78011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:03.990 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.758+0000 7f5c8d7fa700 1 -- 172.21.15.16:0/1367950944 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c78003a80 con 0x7f5c90105450 2024-04-22T15:40:03.990 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.758+0000 7f5c96272700 1 -- 172.21.15.16:0/1367950944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c90198780 con 0x7f5c90105450 2024-04-22T15:40:03.990 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.758+0000 7f5c8d7fa700 1 -- 172.21.15.16:0/1367950944 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5c78003be0 con 0x7f5c90105450 2024-04-22T15:40:03.990 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.758+0000 7f5c8d7fa700 1 -- 172.21.15.16:0/1367950944 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c780197b0 con 0x7f5c90105450 2024-04-22T15:40:03.991 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.758+0000 7f5c96272700 1 -- 172.21.15.16:0/1367950944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c90198c20 con 0x7f5c90105450 2024-04-22T15:40:03.991 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.759+0000 7f5c8d7fa700 1 -- 172.21.15.16:0/1367950944 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f5c78019a70 con 0x7f5c90105450 2024-04-22T15:40:03.991 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.759+0000 7f5c96272700 1 -- 172.21.15.16:0/1367950944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c90191ce0 con 0x7f5c90105450 2024-04-22T15:40:03.991 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.759+0000 7f5c8d7fa700 1 --2- 172.21.15.16:0/1367950944 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f5c7c03a4f0 0x7f5c7c03c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:03.991 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.759+0000 7f5c8d7fa700 1 -- 172.21.15.16:0/1367950944 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5c7804c690 con 0x7f5c90105450 2024-04-22T15:40:03.991 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.759+0000 7f5c8f7fe700 1 --2- 172.21.15.16:0/1367950944 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f5c7c03a4f0 0x7f5c7c03c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:03.991 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.760+0000 7f5c8f7fe700 1 --2- 172.21.15.16:0/1367950944 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f5c7c03a4f0 0x7f5c7c03c9b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5c80006fd0 tx=0x7f5c80006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:03.991 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.764+0000 7f5c8d7fa700 1 -- 172.21.15.16:0/1367950944 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5c7802b750 con 0x7f5c90105450 2024-04-22T15:40:03.991 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:00.966+0000 7f5c96272700 1 -- 172.21.15.16:0/1367950944 --> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi016", "addr": "172.21.15.16", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c900611d0 con 0x7f5c7c03a4f0 2024-04-22T15:40:03.991 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:03.420+0000 7f5c8d7fa700 1 -- 172.21.15.16:0/1367950944 <== mgr.14118 v2:172.21.15.16:6800/3431522404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f5c900611d0 con 0x7f5c7c03a4f0 2024-04-22T15:40:03.991 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:03.422+0000 7f5c96272700 1 -- 172.21.15.16:0/1367950944 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f5c7c03a4f0 msgr2=0x7f5c7c03c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:03.991 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:03.422+0000 7f5c96272700 1 --2- 172.21.15.16:0/1367950944 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f5c7c03a4f0 0x7f5c7c03c9b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5c80006fd0 tx=0x7f5c80006e40 comp rx=0 tx=0).stop 2024-04-22T15:40:03.991 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:03.422+0000 7f5c96272700 1 -- 172.21.15.16:0/1367950944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c90105450 msgr2=0x7f5c90198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:03.991 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:03.422+0000 7f5c96272700 1 --2- 172.21.15.16:0/1367950944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c90105450 0x7f5c90198040 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5c78009520 tx=0x7f5c78011ab0 comp rx=0 tx=0).stop 2024-04-22T15:40:03.992 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:03.423+0000 7f5c96272700 1 -- 172.21.15.16:0/1367950944 shutdown_connections 2024-04-22T15:40:03.992 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:03.423+0000 7f5c96272700 1 --2- 172.21.15.16:0/1367950944 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f5c7c03a4f0 0x7f5c7c03c9b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:03.992 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:03.423+0000 7f5c96272700 1 --2- 172.21.15.16:0/1367950944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c90105450 0x7f5c90198040 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:03.992 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:03.423+0000 7f5c96272700 1 -- 172.21.15.16:0/1367950944 >> 172.21.15.16:0/1367950944 conn(0x7f5c901009b0 msgr2=0x7f5c9018ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:03.992 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:03.423+0000 7f5c96272700 1 -- 172.21.15.16:0/1367950944 shutdown_connections 2024-04-22T15:40:03.992 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:03.423+0000 7f5c96272700 1 -- 172.21.15.16:0/1367950944 wait complete. 2024-04-22T15:40:03.992 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying mon service with default placement... 2024-04-22T15:40:04.527 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:04 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:04.527 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:04 smithi016 ceph-mon[26620]: Added host smithi016 2024-04-22T15:40:04.527 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:04 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:40:05.333 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-22T15:40:05.333 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.444+0000 7fa9ebfbb700 1 Processor -- start 2024-04-22T15:40:05.333 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.444+0000 7fa9ebfbb700 1 -- start start 2024-04-22T15:40:05.333 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.444+0000 7fa9ebfbb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9e4106560 0x7fa9e4106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:05.333 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.444+0000 7fa9ebfbb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e4106f50 con 0x7fa9e4106560 2024-04-22T15:40:05.333 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.444+0000 7fa9e9d57700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9e4106560 0x7fa9e4106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:05.333 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.444+0000 7fa9e9d57700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9e4106560 0x7fa9e4106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43448/0 (socket says 172.21.15.16:43448) 2024-04-22T15:40:05.333 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.444+0000 7fa9e9d57700 1 -- 172.21.15.16:0/3776088915 learned_addr learned my addr 172.21.15.16:0/3776088915 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:05.333 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.444+0000 7fa9e9d57700 1 -- 172.21.15.16:0/3776088915 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e4107760 con 0x7fa9e4106560 2024-04-22T15:40:05.334 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.445+0000 7fa9e9d57700 1 --2- 172.21.15.16:0/3776088915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9e4106560 0x7fa9e4106980 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc009e80 tx=0x7fa9dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c69a7195f0c7f391 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:05.334 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.445+0000 7fa9e8d55700 1 -- 172.21.15.16:0/3776088915 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9dc00e070 con 0x7fa9e4106560 2024-04-22T15:40:05.334 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.445+0000 7fa9e8d55700 1 -- 172.21.15.16:0/3776088915 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa9dc003e90 con 0x7fa9e4106560 2024-04-22T15:40:05.334 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.445+0000 7fa9e8d55700 1 -- 172.21.15.16:0/3776088915 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9dc004380 con 0x7fa9e4106560 2024-04-22T15:40:05.334 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.445+0000 7fa9ebfbb700 1 -- 172.21.15.16:0/3776088915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9e4106560 msgr2=0x7fa9e4106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:05.334 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.445+0000 7fa9ebfbb700 1 --2- 172.21.15.16:0/3776088915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9e4106560 0x7fa9e4106980 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc009e80 tx=0x7fa9dc00b330 comp rx=0 tx=0).stop 2024-04-22T15:40:05.334 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.445+0000 7fa9ebfbb700 1 -- 172.21.15.16:0/3776088915 shutdown_connections 2024-04-22T15:40:05.334 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.445+0000 7fa9ebfbb700 1 --2- 172.21.15.16:0/3776088915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9e4106560 0x7fa9e4106980 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:05.334 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.445+0000 7fa9ebfbb700 1 -- 172.21.15.16:0/3776088915 >> 172.21.15.16:0/3776088915 conn(0x7fa9e4101b40 msgr2=0x7fa9e4103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:05.334 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.445+0000 7fa9ebfbb700 1 -- 172.21.15.16:0/3776088915 shutdown_connections 2024-04-22T15:40:05.334 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.445+0000 7fa9ebfbb700 1 -- 172.21.15.16:0/3776088915 wait complete. 2024-04-22T15:40:05.334 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.446+0000 7fa9ebfbb700 1 Processor -- start 2024-04-22T15:40:05.334 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.446+0000 7fa9ebfbb700 1 -- start start 2024-04-22T15:40:05.334 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.446+0000 7fa9ebfbb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9e4106560 0x7fa9e407bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:05.335 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.446+0000 7fa9ebfbb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e407c170 con 0x7fa9e4106560 2024-04-22T15:40:05.335 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.446+0000 7fa9e9d57700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9e4106560 0x7fa9e407bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:05.335 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.446+0000 7fa9e9d57700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9e4106560 0x7fa9e407bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43464/0 (socket says 172.21.15.16:43464) 2024-04-22T15:40:05.335 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.446+0000 7fa9e9d57700 1 -- 172.21.15.16:0/4126580116 learned_addr learned my addr 172.21.15.16:0/4126580116 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:05.335 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.447+0000 7fa9e9d57700 1 -- 172.21.15.16:0/4126580116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9dc009b30 con 0x7fa9e4106560 2024-04-22T15:40:05.335 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.447+0000 7fa9e9d57700 1 --2- 172.21.15.16:0/4126580116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9e4106560 0x7fa9e407bc30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc00d010 tx=0x7fa9dc004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:05.335 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.447+0000 7fa9e2ffd700 1 -- 172.21.15.16:0/4126580116 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9dc00e040 con 0x7fa9e4106560 2024-04-22T15:40:05.335 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.447+0000 7fa9e2ffd700 1 -- 172.21.15.16:0/4126580116 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa9dc003c30 con 0x7fa9e4106560 2024-04-22T15:40:05.335 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.447+0000 7fa9ebfbb700 1 -- 172.21.15.16:0/4126580116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e407a280 con 0x7fa9e4106560 2024-04-22T15:40:05.335 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.447+0000 7fa9e2ffd700 1 -- 172.21.15.16:0/4126580116 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9dc01b790 con 0x7fa9e4106560 2024-04-22T15:40:05.335 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.447+0000 7fa9ebfbb700 1 -- 172.21.15.16:0/4126580116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e407a780 con 0x7fa9e4106560 2024-04-22T15:40:05.335 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.448+0000 7fa9e2ffd700 1 -- 172.21.15.16:0/4126580116 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fa9dc019040 con 0x7fa9e4106560 2024-04-22T15:40:05.335 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.448+0000 7fa9ebfbb700 1 -- 172.21.15.16:0/4126580116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e40623c0 con 0x7fa9e4106560 2024-04-22T15:40:05.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.448+0000 7fa9e2ffd700 1 --2- 172.21.15.16:0/4126580116 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fa9cc038330 0x7fa9cc03a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:05.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.448+0000 7fa9e2ffd700 1 -- 172.21.15.16:0/4126580116 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa9dc04be80 con 0x7fa9e4106560 2024-04-22T15:40:05.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.448+0000 7fa9e9556700 1 --2- 172.21.15.16:0/4126580116 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fa9cc038330 0x7fa9cc03a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:05.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.449+0000 7fa9e9556700 1 --2- 172.21.15.16:0/4126580116 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fa9cc038330 0x7fa9cc03a7f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8006fd0 tx=0x7fa9d8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:05.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.453+0000 7fa9e2ffd700 1 -- 172.21.15.16:0/4126580116 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa9dc01a390 con 0x7fa9e4106560 2024-04-22T15:40:05.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.667+0000 7fa9ebfbb700 1 -- 172.21.15.16:0/4126580116 --> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9e4102f90 con 0x7fa9cc038330 2024-04-22T15:40:05.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.671+0000 7fa9e2ffd700 1 -- 172.21.15.16:0/4126580116 <== mgr.14118 v2:172.21.15.16:6800/3431522404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa9e4102f90 con 0x7fa9cc038330 2024-04-22T15:40:05.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.673+0000 7fa9ebfbb700 1 -- 172.21.15.16:0/4126580116 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fa9cc038330 msgr2=0x7fa9cc03a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:05.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.673+0000 7fa9ebfbb700 1 --2- 172.21.15.16:0/4126580116 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fa9cc038330 0x7fa9cc03a7f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8006fd0 tx=0x7fa9d8006e40 comp rx=0 tx=0).stop 2024-04-22T15:40:05.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.674+0000 7fa9ebfbb700 1 -- 172.21.15.16:0/4126580116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9e4106560 msgr2=0x7fa9e407bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:05.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.674+0000 7fa9ebfbb700 1 --2- 172.21.15.16:0/4126580116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9e4106560 0x7fa9e407bc30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc00d010 tx=0x7fa9dc004430 comp rx=0 tx=0).stop 2024-04-22T15:40:05.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.674+0000 7fa9ebfbb700 1 -- 172.21.15.16:0/4126580116 shutdown_connections 2024-04-22T15:40:05.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.674+0000 7fa9ebfbb700 1 --2- 172.21.15.16:0/4126580116 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fa9cc038330 0x7fa9cc03a7f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:05.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.674+0000 7fa9ebfbb700 1 --2- 172.21.15.16:0/4126580116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9e4106560 0x7fa9e407bc30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:05.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.674+0000 7fa9ebfbb700 1 -- 172.21.15.16:0/4126580116 >> 172.21.15.16:0/4126580116 conn(0x7fa9e4101b40 msgr2=0x7fa9e4102880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:05.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.674+0000 7fa9ebfbb700 1 -- 172.21.15.16:0/4126580116 shutdown_connections 2024-04-22T15:40:05.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:04.674+0000 7fa9ebfbb700 1 -- 172.21.15.16:0/4126580116 wait complete. 2024-04-22T15:40:05.337 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying mgr service with default placement... 2024-04-22T15:40:05.961 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:05 smithi016 ceph-mon[26620]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:05.961 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:05 smithi016 ceph-mon[26620]: Saving service mon spec with placement count:5 2024-04-22T15:40:05.961 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:05 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:07.526 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:07 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:07.526 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:07 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:07.526 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:07 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:07.534 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-22T15:40:07.534 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.715+0000 7f35efb2d700 1 Processor -- start 2024-04-22T15:40:07.534 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.716+0000 7f35efb2d700 1 -- start start 2024-04-22T15:40:07.534 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.716+0000 7f35efb2d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35e8104e30 0x7f35e8105250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:07.534 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.716+0000 7f35efb2d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e8079e10 con 0x7f35e8104e30 2024-04-22T15:40:07.534 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.716+0000 7f35ed8c9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35e8104e30 0x7f35e8105250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:07.535 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.716+0000 7f35ed8c9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35e8104e30 0x7f35e8105250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43470/0 (socket says 172.21.15.16:43470) 2024-04-22T15:40:07.535 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.716+0000 7f35ed8c9700 1 -- 172.21.15.16:0/3746206823 learned_addr learned my addr 172.21.15.16:0/3746206823 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:07.535 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.716+0000 7f35ed8c9700 1 -- 172.21.15.16:0/3746206823 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e8105790 con 0x7f35e8104e30 2024-04-22T15:40:07.535 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.717+0000 7f35ed8c9700 1 --2- 172.21.15.16:0/3746206823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35e8104e30 0x7f35e8105250 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f35e40098a0 tx=0x7f35e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=936ebf4fc9eb9fe6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:07.535 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.717+0000 7f35ec8c7700 1 -- 172.21.15.16:0/3746206823 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35e4004180 con 0x7f35e8104e30 2024-04-22T15:40:07.535 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.717+0000 7f35ec8c7700 1 -- 172.21.15.16:0/3746206823 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f35e40042e0 con 0x7f35e8104e30 2024-04-22T15:40:07.535 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.717+0000 7f35ec8c7700 1 -- 172.21.15.16:0/3746206823 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35e4011af0 con 0x7f35e8104e30 2024-04-22T15:40:07.535 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.717+0000 7f35efb2d700 1 -- 172.21.15.16:0/3746206823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35e8104e30 msgr2=0x7f35e8105250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:07.535 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.717+0000 7f35efb2d700 1 --2- 172.21.15.16:0/3746206823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35e8104e30 0x7f35e8105250 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f35e40098a0 tx=0x7f35e4009bb0 comp rx=0 tx=0).stop 2024-04-22T15:40:07.535 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.718+0000 7f35efb2d700 1 -- 172.21.15.16:0/3746206823 shutdown_connections 2024-04-22T15:40:07.535 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.718+0000 7f35efb2d700 1 --2- 172.21.15.16:0/3746206823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35e8104e30 0x7f35e8105250 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:07.535 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.718+0000 7f35efb2d700 1 -- 172.21.15.16:0/3746206823 >> 172.21.15.16:0/3746206823 conn(0x7f35e81009b0 msgr2=0x7f35e8102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:07.535 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.718+0000 7f35efb2d700 1 -- 172.21.15.16:0/3746206823 shutdown_connections 2024-04-22T15:40:07.535 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.718+0000 7f35efb2d700 1 -- 172.21.15.16:0/3746206823 wait complete. 2024-04-22T15:40:07.536 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.718+0000 7f35efb2d700 1 Processor -- start 2024-04-22T15:40:07.536 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.718+0000 7f35efb2d700 1 -- start start 2024-04-22T15:40:07.536 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.719+0000 7f35efb2d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35e8104e30 0x7f35e819c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:07.536 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.719+0000 7f35efb2d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e819c920 con 0x7f35e8104e30 2024-04-22T15:40:07.536 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.719+0000 7f35ed8c9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35e8104e30 0x7f35e819c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:07.536 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.719+0000 7f35ed8c9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35e8104e30 0x7f35e819c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43480/0 (socket says 172.21.15.16:43480) 2024-04-22T15:40:07.536 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.719+0000 7f35ed8c9700 1 -- 172.21.15.16:0/3211369603 learned_addr learned my addr 172.21.15.16:0/3211369603 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:07.536 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.719+0000 7f35ed8c9700 1 -- 172.21.15.16:0/3211369603 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e4009550 con 0x7f35e8104e30 2024-04-22T15:40:07.536 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.719+0000 7f35ed8c9700 1 --2- 172.21.15.16:0/3211369603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35e8104e30 0x7f35e819c3e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f35e4011670 tx=0x7f35e4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:07.536 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.719+0000 7f35deffd700 1 -- 172.21.15.16:0/3211369603 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35e4003a40 con 0x7f35e8104e30 2024-04-22T15:40:07.536 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.719+0000 7f35deffd700 1 -- 172.21.15.16:0/3211369603 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f35e4003ba0 con 0x7f35e8104e30 2024-04-22T15:40:07.536 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.719+0000 7f35deffd700 1 -- 172.21.15.16:0/3211369603 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35e4024820 con 0x7f35e8104e30 2024-04-22T15:40:07.536 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.719+0000 7f35efb2d700 1 -- 172.21.15.16:0/3211369603 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e819cb20 con 0x7f35e8104e30 2024-04-22T15:40:07.536 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.720+0000 7f35efb2d700 1 -- 172.21.15.16:0/3211369603 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e819cfc0 con 0x7f35e8104e30 2024-04-22T15:40:07.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.720+0000 7f35deffd700 1 -- 172.21.15.16:0/3211369603 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f35e4024a90 con 0x7f35e8104e30 2024-04-22T15:40:07.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.721+0000 7f35deffd700 1 --2- 172.21.15.16:0/3211369603 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f35d4038330 0x7f35d403a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:07.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.721+0000 7f35deffd700 1 -- 172.21.15.16:0/3211369603 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f35e404c420 con 0x7f35e8104e30 2024-04-22T15:40:07.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.721+0000 7f35efb2d700 1 -- 172.21.15.16:0/3211369603 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e8195d20 con 0x7f35e8104e30 2024-04-22T15:40:07.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.721+0000 7f35ed0c8700 1 --2- 172.21.15.16:0/3211369603 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f35d4038330 0x7f35d403a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:07.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.721+0000 7f35ed0c8700 1 --2- 172.21.15.16:0/3211369603 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f35d4038330 0x7f35d403a7f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f35d8006fd0 tx=0x7f35d8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:07.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.728+0000 7f35deffd700 1 -- 172.21.15.16:0/3211369603 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f35e40187c0 con 0x7f35e8104e30 2024-04-22T15:40:07.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.939+0000 7f35efb2d700 1 -- 172.21.15.16:0/3211369603 --> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f35e80611d0 con 0x7f35d4038330 2024-04-22T15:40:07.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.943+0000 7f35deffd700 1 -- 172.21.15.16:0/3211369603 <== mgr.14118 v2:172.21.15.16:6800/3431522404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f35e80611d0 con 0x7f35d4038330 2024-04-22T15:40:07.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.945+0000 7f35efb2d700 1 -- 172.21.15.16:0/3211369603 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f35d4038330 msgr2=0x7f35d403a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:07.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.945+0000 7f35efb2d700 1 --2- 172.21.15.16:0/3211369603 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f35d4038330 0x7f35d403a7f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f35d8006fd0 tx=0x7f35d8006e40 comp rx=0 tx=0).stop 2024-04-22T15:40:07.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.945+0000 7f35efb2d700 1 -- 172.21.15.16:0/3211369603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35e8104e30 msgr2=0x7f35e819c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:07.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.945+0000 7f35efb2d700 1 --2- 172.21.15.16:0/3211369603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35e8104e30 0x7f35e819c3e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f35e4011670 tx=0x7f35e4011aa0 comp rx=0 tx=0).stop 2024-04-22T15:40:07.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.945+0000 7f35efb2d700 1 -- 172.21.15.16:0/3211369603 shutdown_connections 2024-04-22T15:40:07.538 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.945+0000 7f35efb2d700 1 --2- 172.21.15.16:0/3211369603 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f35d4038330 0x7f35d403a7f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:07.538 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.945+0000 7f35efb2d700 1 --2- 172.21.15.16:0/3211369603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35e8104e30 0x7f35e819c3e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:07.538 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.945+0000 7f35efb2d700 1 -- 172.21.15.16:0/3211369603 >> 172.21.15.16:0/3211369603 conn(0x7f35e81009b0 msgr2=0x7f35e81909f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:07.538 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.946+0000 7f35efb2d700 1 -- 172.21.15.16:0/3211369603 shutdown_connections 2024-04-22T15:40:07.538 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:06.946+0000 7f35efb2d700 1 -- 172.21.15.16:0/3211369603 wait complete. 2024-04-22T15:40:07.538 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying crash service with default placement... 2024-04-22T15:40:08.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:08 smithi016 ceph-mon[26620]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:08.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:08 smithi016 ceph-mon[26620]: Saving service mgr spec with placement count:2 2024-04-22T15:40:08.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:08 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:08.794 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-04-22T15:40:08.794 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.974+0000 7f6125d7f700 1 Processor -- start 2024-04-22T15:40:08.795 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.974+0000 7f6125d7f700 1 -- start start 2024-04-22T15:40:08.795 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.974+0000 7f6125d7f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6120105460 0x7f6120105880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:08.795 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.974+0000 7f6125d7f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6120105e50 con 0x7f6120105460 2024-04-22T15:40:08.795 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.974+0000 7f611f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6120105460 0x7f6120105880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:08.795 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.975+0000 7f611f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6120105460 0x7f6120105880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43496/0 (socket says 172.21.15.16:43496) 2024-04-22T15:40:08.795 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.975+0000 7f611f7fe700 1 -- 172.21.15.16:0/2740053143 learned_addr learned my addr 172.21.15.16:0/2740053143 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:08.795 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.975+0000 7f611f7fe700 1 -- 172.21.15.16:0/2740053143 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6120106660 con 0x7f6120105460 2024-04-22T15:40:08.795 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.975+0000 7f611f7fe700 1 --2- 172.21.15.16:0/2740053143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6120105460 0x7f6120105880 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f61100098a0 tx=0x7f6110009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb4f57713eb22f3a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:08.795 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.975+0000 7f611effd700 1 -- 172.21.15.16:0/2740053143 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6110004180 con 0x7f6120105460 2024-04-22T15:40:08.795 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.975+0000 7f611effd700 1 -- 172.21.15.16:0/2740053143 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f61100042e0 con 0x7f6120105460 2024-04-22T15:40:08.795 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.975+0000 7f611effd700 1 -- 172.21.15.16:0/2740053143 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6110011af0 con 0x7f6120105460 2024-04-22T15:40:08.796 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.976+0000 7f6125d7f700 1 -- 172.21.15.16:0/2740053143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6120105460 msgr2=0x7f6120105880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:08.796 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.976+0000 7f6125d7f700 1 --2- 172.21.15.16:0/2740053143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6120105460 0x7f6120105880 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f61100098a0 tx=0x7f6110009bb0 comp rx=0 tx=0).stop 2024-04-22T15:40:08.796 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.976+0000 7f6125d7f700 1 -- 172.21.15.16:0/2740053143 shutdown_connections 2024-04-22T15:40:08.796 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.976+0000 7f6125d7f700 1 --2- 172.21.15.16:0/2740053143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6120105460 0x7f6120105880 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:08.796 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.976+0000 7f6125d7f700 1 -- 172.21.15.16:0/2740053143 >> 172.21.15.16:0/2740053143 conn(0x7f6120100a20 msgr2=0x7f6120102e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:08.796 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.976+0000 7f6125d7f700 1 -- 172.21.15.16:0/2740053143 shutdown_connections 2024-04-22T15:40:08.796 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.976+0000 7f6125d7f700 1 -- 172.21.15.16:0/2740053143 wait complete. 2024-04-22T15:40:08.796 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.976+0000 7f6125d7f700 1 Processor -- start 2024-04-22T15:40:08.796 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.976+0000 7f6125d7f700 1 -- start start 2024-04-22T15:40:08.796 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.977+0000 7f6125d7f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6120105460 0x7f6120190650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:08.796 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.977+0000 7f6125d7f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6120190b90 con 0x7f6120105460 2024-04-22T15:40:08.796 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.977+0000 7f611f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6120105460 0x7f6120190650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:08.796 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.977+0000 7f611f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6120105460 0x7f6120190650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43498/0 (socket says 172.21.15.16:43498) 2024-04-22T15:40:08.797 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.977+0000 7f611f7fe700 1 -- 172.21.15.16:0/3465928284 learned_addr learned my addr 172.21.15.16:0/3465928284 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:08.797 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.977+0000 7f611f7fe700 1 -- 172.21.15.16:0/3465928284 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6110009550 con 0x7f6120105460 2024-04-22T15:40:08.797 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.977+0000 7f611f7fe700 1 --2- 172.21.15.16:0/3465928284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6120105460 0x7f6120190650 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6110011670 tx=0x7f6110011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:08.797 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.977+0000 7f611d7fa700 1 -- 172.21.15.16:0/3465928284 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61100039f0 con 0x7f6120105460 2024-04-22T15:40:08.797 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.977+0000 7f611d7fa700 1 -- 172.21.15.16:0/3465928284 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6110003b50 con 0x7f6120105460 2024-04-22T15:40:08.797 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.978+0000 7f6125d7f700 1 -- 172.21.15.16:0/3465928284 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f612018eca0 con 0x7f6120105460 2024-04-22T15:40:08.797 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.978+0000 7f611d7fa700 1 -- 172.21.15.16:0/3465928284 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61100247c0 con 0x7f6120105460 2024-04-22T15:40:08.797 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.978+0000 7f6125d7f700 1 -- 172.21.15.16:0/3465928284 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f612018f140 con 0x7f6120105460 2024-04-22T15:40:08.797 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.978+0000 7f611d7fa700 1 -- 172.21.15.16:0/3465928284 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f611002d3f0 con 0x7f6120105460 2024-04-22T15:40:08.797 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.978+0000 7f6125d7f700 1 -- 172.21.15.16:0/3465928284 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61200623c0 con 0x7f6120105460 2024-04-22T15:40:08.797 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.979+0000 7f611d7fa700 1 --2- 172.21.15.16:0/3465928284 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f610c038320 0x7f610c03a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:08.797 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.979+0000 7f611d7fa700 1 -- 172.21.15.16:0/3465928284 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f611004c3f0 con 0x7f6120105460 2024-04-22T15:40:08.798 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.979+0000 7f6116ffd700 1 --2- 172.21.15.16:0/3465928284 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f610c038320 0x7f610c03a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:08.798 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.979+0000 7f6116ffd700 1 --2- 172.21.15.16:0/3465928284 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f610c038320 0x7f610c03a7e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6108006fd0 tx=0x7f6108006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:08.798 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:07.983+0000 7f611d7fa700 1 -- 172.21.15.16:0/3465928284 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6110010970 con 0x7f6120105460 2024-04-22T15:40:08.798 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:08.195+0000 7f6125d7f700 1 -- 172.21.15.16:0/3465928284 --> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f6120109860 con 0x7f610c038320 2024-04-22T15:40:08.798 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:08.199+0000 7f611d7fa700 1 -- 172.21.15.16:0/3465928284 <== mgr.14118 v2:172.21.15.16:6800/3431522404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f6120109860 con 0x7f610c038320 2024-04-22T15:40:08.798 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:08.201+0000 7f6125d7f700 1 -- 172.21.15.16:0/3465928284 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f610c038320 msgr2=0x7f610c03a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:08.798 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:08.201+0000 7f6125d7f700 1 --2- 172.21.15.16:0/3465928284 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f610c038320 0x7f610c03a7e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6108006fd0 tx=0x7f6108006e40 comp rx=0 tx=0).stop 2024-04-22T15:40:08.798 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:08.201+0000 7f6125d7f700 1 -- 172.21.15.16:0/3465928284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6120105460 msgr2=0x7f6120190650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:08.798 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:08.201+0000 7f6125d7f700 1 --2- 172.21.15.16:0/3465928284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6120105460 0x7f6120190650 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6110011670 tx=0x7f6110011b50 comp rx=0 tx=0).stop 2024-04-22T15:40:08.798 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:08.202+0000 7f6125d7f700 1 -- 172.21.15.16:0/3465928284 shutdown_connections 2024-04-22T15:40:08.798 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:08.202+0000 7f6125d7f700 1 --2- 172.21.15.16:0/3465928284 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f610c038320 0x7f610c03a7e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:08.799 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:08.202+0000 7f6125d7f700 1 --2- 172.21.15.16:0/3465928284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6120105460 0x7f6120190650 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:08.799 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:08.202+0000 7f6125d7f700 1 -- 172.21.15.16:0/3465928284 >> 172.21.15.16:0/3465928284 conn(0x7f6120100a20 msgr2=0x7f6120079900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:08.799 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:08.202+0000 7f6125d7f700 1 -- 172.21.15.16:0/3465928284 shutdown_connections 2024-04-22T15:40:08.799 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:08.202+0000 7f6125d7f700 1 -- 172.21.15.16:0/3465928284 wait complete. 2024-04-22T15:40:08.799 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying ceph-exporter service with default placement... 2024-04-22T15:40:09.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:09 smithi016 ceph-mon[26620]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:09.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:09 smithi016 ceph-mon[26620]: Saving service crash spec with placement * 2024-04-22T15:40:09.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:09 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:10.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-04-22T15:40:10.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.433+0000 7ff57b450700 1 Processor -- start 2024-04-22T15:40:10.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.433+0000 7ff57b450700 1 -- start start 2024-04-22T15:40:10.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.433+0000 7ff57b450700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff574105470 0x7ff574105890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:10.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.433+0000 7ff57b450700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff574105e60 con 0x7ff574105470 2024-04-22T15:40:10.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.434+0000 7ff5791ec700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff574105470 0x7ff574105890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:10.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.434+0000 7ff5791ec700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff574105470 0x7ff574105890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43502/0 (socket says 172.21.15.16:43502) 2024-04-22T15:40:10.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.434+0000 7ff5791ec700 1 -- 172.21.15.16:0/1010020984 learned_addr learned my addr 172.21.15.16:0/1010020984 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:10.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.434+0000 7ff5791ec700 1 -- 172.21.15.16:0/1010020984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff574106670 con 0x7ff574105470 2024-04-22T15:40:10.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.434+0000 7ff5791ec700 1 --2- 172.21.15.16:0/1010020984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff574105470 0x7ff574105890 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff5700098a0 tx=0x7ff570009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ab19ba059fadd66 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:10.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.434+0000 7ff56bfff700 1 -- 172.21.15.16:0/1010020984 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff570004180 con 0x7ff574105470 2024-04-22T15:40:10.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.434+0000 7ff56bfff700 1 -- 172.21.15.16:0/1010020984 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff5700042e0 con 0x7ff574105470 2024-04-22T15:40:10.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.434+0000 7ff56bfff700 1 -- 172.21.15.16:0/1010020984 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff570011af0 con 0x7ff574105470 2024-04-22T15:40:10.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.435+0000 7ff57b450700 1 -- 172.21.15.16:0/1010020984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff574105470 msgr2=0x7ff574105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:10.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.435+0000 7ff57b450700 1 --2- 172.21.15.16:0/1010020984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff574105470 0x7ff574105890 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff5700098a0 tx=0x7ff570009bb0 comp rx=0 tx=0).stop 2024-04-22T15:40:10.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.435+0000 7ff57b450700 1 -- 172.21.15.16:0/1010020984 shutdown_connections 2024-04-22T15:40:10.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.435+0000 7ff57b450700 1 --2- 172.21.15.16:0/1010020984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff574105470 0x7ff574105890 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:10.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.435+0000 7ff57b450700 1 -- 172.21.15.16:0/1010020984 >> 172.21.15.16:0/1010020984 conn(0x7ff5741009f0 msgr2=0x7ff574102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:10.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.435+0000 7ff57b450700 1 -- 172.21.15.16:0/1010020984 shutdown_connections 2024-04-22T15:40:10.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.436+0000 7ff57b450700 1 -- 172.21.15.16:0/1010020984 wait complete. 2024-04-22T15:40:10.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.436+0000 7ff57b450700 1 Processor -- start 2024-04-22T15:40:10.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.436+0000 7ff57b450700 1 -- start start 2024-04-22T15:40:10.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.436+0000 7ff57b450700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff574105470 0x7ff574074300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:10.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.436+0000 7ff57b450700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57419cf00 con 0x7ff574105470 2024-04-22T15:40:10.263 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.436+0000 7ff5791ec700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff574105470 0x7ff574074300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:10.263 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.436+0000 7ff5791ec700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff574105470 0x7ff574074300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43518/0 (socket says 172.21.15.16:43518) 2024-04-22T15:40:10.263 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.436+0000 7ff5791ec700 1 -- 172.21.15.16:0/2708058259 learned_addr learned my addr 172.21.15.16:0/2708058259 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:10.263 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.437+0000 7ff5791ec700 1 -- 172.21.15.16:0/2708058259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff570009550 con 0x7ff574105470 2024-04-22T15:40:10.263 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.437+0000 7ff5791ec700 1 --2- 172.21.15.16:0/2708058259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff574105470 0x7ff574074300 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff570006ba0 tx=0x7ff570011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:10.263 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.437+0000 7ff56a7fc700 1 -- 172.21.15.16:0/2708058259 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff570003b00 con 0x7ff574105470 2024-04-22T15:40:10.263 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.437+0000 7ff57b450700 1 -- 172.21.15.16:0/2708058259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff57419d040 con 0x7ff574105470 2024-04-22T15:40:10.263 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.437+0000 7ff56a7fc700 1 -- 172.21.15.16:0/2708058259 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff57001a8e0 con 0x7ff574105470 2024-04-22T15:40:10.263 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.437+0000 7ff56a7fc700 1 -- 172.21.15.16:0/2708058259 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff570019990 con 0x7ff574105470 2024-04-22T15:40:10.263 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.437+0000 7ff57b450700 1 -- 172.21.15.16:0/2708058259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff57419d4e0 con 0x7ff574105470 2024-04-22T15:40:10.263 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.438+0000 7ff56a7fc700 1 -- 172.21.15.16:0/2708058259 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7ff570019c50 con 0x7ff574105470 2024-04-22T15:40:10.263 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.438+0000 7ff57b450700 1 -- 172.21.15.16:0/2708058259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5740623c0 con 0x7ff574105470 2024-04-22T15:40:10.264 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.439+0000 7ff56a7fc700 1 --2- 172.21.15.16:0/2708058259 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7ff5600382d0 0x7ff56003a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:10.264 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.439+0000 7ff56a7fc700 1 -- 172.21.15.16:0/2708058259 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff57004c5a0 con 0x7ff574105470 2024-04-22T15:40:10.264 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.439+0000 7ff5789eb700 1 --2- 172.21.15.16:0/2708058259 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7ff5600382d0 0x7ff56003a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:10.264 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.439+0000 7ff5789eb700 1 --2- 172.21.15.16:0/2708058259 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7ff5600382d0 0x7ff56003a790 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff564006fd0 tx=0x7ff564006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:10.264 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.443+0000 7ff56a7fc700 1 -- 172.21.15.16:0/2708058259 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff57002d440 con 0x7ff574105470 2024-04-22T15:40:10.264 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.657+0000 7ff57b450700 1 -- 172.21.15.16:0/2708058259 --> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7ff574101e60 con 0x7ff5600382d0 2024-04-22T15:40:10.264 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.662+0000 7ff56a7fc700 1 -- 172.21.15.16:0/2708058259 <== mgr.14118 v2:172.21.15.16:6800/3431522404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7ff574101e60 con 0x7ff5600382d0 2024-04-22T15:40:10.264 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.664+0000 7ff57b450700 1 -- 172.21.15.16:0/2708058259 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7ff5600382d0 msgr2=0x7ff56003a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:10.264 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.664+0000 7ff57b450700 1 --2- 172.21.15.16:0/2708058259 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7ff5600382d0 0x7ff56003a790 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff564006fd0 tx=0x7ff564006e40 comp rx=0 tx=0).stop 2024-04-22T15:40:10.264 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.664+0000 7ff57b450700 1 -- 172.21.15.16:0/2708058259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff574105470 msgr2=0x7ff574074300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:10.265 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.664+0000 7ff57b450700 1 --2- 172.21.15.16:0/2708058259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff574105470 0x7ff574074300 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff570006ba0 tx=0x7ff570011b50 comp rx=0 tx=0).stop 2024-04-22T15:40:10.265 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.664+0000 7ff57b450700 1 -- 172.21.15.16:0/2708058259 shutdown_connections 2024-04-22T15:40:10.265 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.664+0000 7ff57b450700 1 --2- 172.21.15.16:0/2708058259 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7ff5600382d0 0x7ff56003a790 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:10.265 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.664+0000 7ff57b450700 1 --2- 172.21.15.16:0/2708058259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff574105470 0x7ff574074300 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:10.265 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.664+0000 7ff57b450700 1 -- 172.21.15.16:0/2708058259 >> 172.21.15.16:0/2708058259 conn(0x7ff5741009f0 msgr2=0x7ff574101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:10.265 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.664+0000 7ff57b450700 1 -- 172.21.15.16:0/2708058259 shutdown_connections 2024-04-22T15:40:10.265 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:09.664+0000 7ff57b450700 1 -- 172.21.15.16:0/2708058259 wait complete. 2024-04-22T15:40:10.265 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying prometheus service with default placement... 2024-04-22T15:40:10.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:10 smithi016 ceph-mon[26620]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:10.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:10 smithi016 ceph-mon[26620]: Saving service ceph-exporter spec with placement * 2024-04-22T15:40:10.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:10 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:10.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:10 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:11.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-04-22T15:40:11.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.739+0000 7fa351036700 1 Processor -- start 2024-04-22T15:40:11.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.739+0000 7fa351036700 1 -- start start 2024-04-22T15:40:11.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.739+0000 7fa351036700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa34c107010 0x7fa34c109400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:11.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.739+0000 7fa351036700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c079e50 con 0x7fa34c107010 2024-04-22T15:40:11.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.739+0000 7fa34ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa34c107010 0x7fa34c109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:11.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.739+0000 7fa34ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa34c107010 0x7fa34c109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43520/0 (socket says 172.21.15.16:43520) 2024-04-22T15:40:11.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.739+0000 7fa34ad9d700 1 -- 172.21.15.16:0/1626730777 learned_addr learned my addr 172.21.15.16:0/1626730777 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:11.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.740+0000 7fa34ad9d700 1 -- 172.21.15.16:0/1626730777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa34c109940 con 0x7fa34c107010 2024-04-22T15:40:11.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.740+0000 7fa34ad9d700 1 --2- 172.21.15.16:0/1626730777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa34c107010 0x7fa34c109400 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa344009e80 tx=0x7fa34400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83857f6f54cef589 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:11.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.740+0000 7fa349d9b700 1 -- 172.21.15.16:0/1626730777 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa34400e070 con 0x7fa34c107010 2024-04-22T15:40:11.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.740+0000 7fa349d9b700 1 -- 172.21.15.16:0/1626730777 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa34400b8c0 con 0x7fa34c107010 2024-04-22T15:40:11.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.740+0000 7fa349d9b700 1 -- 172.21.15.16:0/1626730777 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3440044d0 con 0x7fa34c107010 2024-04-22T15:40:11.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.741+0000 7fa351036700 1 -- 172.21.15.16:0/1626730777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa34c107010 msgr2=0x7fa34c109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:11.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.741+0000 7fa351036700 1 --2- 172.21.15.16:0/1626730777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa34c107010 0x7fa34c109400 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa344009e80 tx=0x7fa34400b330 comp rx=0 tx=0).stop 2024-04-22T15:40:11.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.741+0000 7fa351036700 1 -- 172.21.15.16:0/1626730777 shutdown_connections 2024-04-22T15:40:11.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.741+0000 7fa351036700 1 --2- 172.21.15.16:0/1626730777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa34c107010 0x7fa34c109400 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:11.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.741+0000 7fa351036700 1 -- 172.21.15.16:0/1626730777 >> 172.21.15.16:0/1626730777 conn(0x7fa34c100a20 msgr2=0x7fa34c102e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:11.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.741+0000 7fa351036700 1 -- 172.21.15.16:0/1626730777 shutdown_connections 2024-04-22T15:40:11.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.741+0000 7fa351036700 1 -- 172.21.15.16:0/1626730777 wait complete. 2024-04-22T15:40:11.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.742+0000 7fa351036700 1 Processor -- start 2024-04-22T15:40:11.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.742+0000 7fa351036700 1 -- start start 2024-04-22T15:40:11.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.742+0000 7fa351036700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa34c107010 0x7fa34c19c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:11.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.742+0000 7fa351036700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c19c950 con 0x7fa34c107010 2024-04-22T15:40:11.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.742+0000 7fa34ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa34c107010 0x7fa34c19c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:11.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.742+0000 7fa34ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa34c107010 0x7fa34c19c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43530/0 (socket says 172.21.15.16:43530) 2024-04-22T15:40:11.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.742+0000 7fa34ad9d700 1 -- 172.21.15.16:0/266409150 learned_addr learned my addr 172.21.15.16:0/266409150 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:11.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.743+0000 7fa34ad9d700 1 -- 172.21.15.16:0/266409150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa344009b30 con 0x7fa34c107010 2024-04-22T15:40:11.597 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.743+0000 7fa34ad9d700 1 --2- 172.21.15.16:0/266409150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa34c107010 0x7fa34c19c410 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa344009e50 tx=0x7fa3440049f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:11.597 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.743+0000 7fa343fff700 1 -- 172.21.15.16:0/266409150 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa34400e040 con 0x7fa34c107010 2024-04-22T15:40:11.597 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.743+0000 7fa351036700 1 -- 172.21.15.16:0/266409150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa34c19cb50 con 0x7fa34c107010 2024-04-22T15:40:11.597 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.743+0000 7fa343fff700 1 -- 172.21.15.16:0/266409150 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa34400bcc0 con 0x7fa34c107010 2024-04-22T15:40:11.597 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.743+0000 7fa343fff700 1 -- 172.21.15.16:0/266409150 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa344012670 con 0x7fa34c107010 2024-04-22T15:40:11.597 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.743+0000 7fa351036700 1 -- 172.21.15.16:0/266409150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa34c19d050 con 0x7fa34c107010 2024-04-22T15:40:11.597 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.744+0000 7fa351036700 1 -- 172.21.15.16:0/266409150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa34c10a6d0 con 0x7fa34c107010 2024-04-22T15:40:11.597 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.744+0000 7fa343fff700 1 -- 172.21.15.16:0/266409150 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fa344019040 con 0x7fa34c107010 2024-04-22T15:40:11.597 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.744+0000 7fa343fff700 1 --2- 172.21.15.16:0/266409150 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fa3380382e0 0x7fa33803a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:11.597 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.744+0000 7fa343fff700 1 -- 172.21.15.16:0/266409150 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa34404bf00 con 0x7fa34c107010 2024-04-22T15:40:11.597 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.745+0000 7fa34a59c700 1 --2- 172.21.15.16:0/266409150 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fa3380382e0 0x7fa33803a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:11.597 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.745+0000 7fa34a59c700 1 --2- 172.21.15.16:0/266409150 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fa3380382e0 0x7fa33803a7a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa334006fd0 tx=0x7fa334006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:11.597 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.749+0000 7fa343fff700 1 -- 172.21.15.16:0/266409150 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa344011750 con 0x7fa34c107010 2024-04-22T15:40:11.598 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.978+0000 7fa351036700 1 -- 172.21.15.16:0/266409150 --> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fa34c101e70 con 0x7fa3380382e0 2024-04-22T15:40:11.598 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.982+0000 7fa343fff700 1 -- 172.21.15.16:0/266409150 <== mgr.14118 v2:172.21.15.16:6800/3431522404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fa34c101e70 con 0x7fa3380382e0 2024-04-22T15:40:11.598 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.984+0000 7fa351036700 1 -- 172.21.15.16:0/266409150 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fa3380382e0 msgr2=0x7fa33803a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:11.598 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.984+0000 7fa351036700 1 --2- 172.21.15.16:0/266409150 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fa3380382e0 0x7fa33803a7a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa334006fd0 tx=0x7fa334006e40 comp rx=0 tx=0).stop 2024-04-22T15:40:11.598 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.984+0000 7fa351036700 1 -- 172.21.15.16:0/266409150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa34c107010 msgr2=0x7fa34c19c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:11.598 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.984+0000 7fa351036700 1 --2- 172.21.15.16:0/266409150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa34c107010 0x7fa34c19c410 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa344009e50 tx=0x7fa3440049f0 comp rx=0 tx=0).stop 2024-04-22T15:40:11.598 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.984+0000 7fa351036700 1 -- 172.21.15.16:0/266409150 shutdown_connections 2024-04-22T15:40:11.598 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.984+0000 7fa351036700 1 --2- 172.21.15.16:0/266409150 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fa3380382e0 0x7fa33803a7a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:11.598 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.984+0000 7fa351036700 1 --2- 172.21.15.16:0/266409150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa34c107010 0x7fa34c19c410 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:11.598 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.984+0000 7fa351036700 1 -- 172.21.15.16:0/266409150 >> 172.21.15.16:0/266409150 conn(0x7fa34c100a20 msgr2=0x7fa34c101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:11.598 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.985+0000 7fa351036700 1 -- 172.21.15.16:0/266409150 shutdown_connections 2024-04-22T15:40:11.598 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:10.985+0000 7fa351036700 1 -- 172.21.15.16:0/266409150 wait complete. 2024-04-22T15:40:11.598 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying grafana service with default placement... 2024-04-22T15:40:12.357 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:11 smithi016 ceph-mon[26620]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:12.357 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:11 smithi016 ceph-mon[26620]: Saving service prometheus spec with placement count:1 2024-04-22T15:40:12.358 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:11 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:13.689 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-04-22T15:40:13.689 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.856+0000 7ff1bf479700 1 Processor -- start 2024-04-22T15:40:13.689 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.856+0000 7ff1bf479700 1 -- start start 2024-04-22T15:40:13.689 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.856+0000 7ff1bf479700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1b8106560 0x7ff1b8106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:13.689 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.856+0000 7ff1bf479700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b8106f50 con 0x7ff1b8106560 2024-04-22T15:40:13.689 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.856+0000 7ff1bd215700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1b8106560 0x7ff1b8106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:13.689 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.856+0000 7ff1bd215700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1b8106560 0x7ff1b8106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58296/0 (socket says 172.21.15.16:58296) 2024-04-22T15:40:13.689 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.856+0000 7ff1bd215700 1 -- 172.21.15.16:0/4136564451 learned_addr learned my addr 172.21.15.16:0/4136564451 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:13.690 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.857+0000 7ff1bd215700 1 -- 172.21.15.16:0/4136564451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b8107760 con 0x7ff1b8106560 2024-04-22T15:40:13.690 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.857+0000 7ff1bd215700 1 --2- 172.21.15.16:0/4136564451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1b8106560 0x7ff1b8106980 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff1b40098a0 tx=0x7ff1b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93360c7936042a2b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:13.690 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.857+0000 7ff1affff700 1 -- 172.21.15.16:0/4136564451 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1b4004180 con 0x7ff1b8106560 2024-04-22T15:40:13.690 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.857+0000 7ff1affff700 1 -- 172.21.15.16:0/4136564451 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff1b40042e0 con 0x7ff1b8106560 2024-04-22T15:40:13.690 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.857+0000 7ff1affff700 1 -- 172.21.15.16:0/4136564451 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1b4011af0 con 0x7ff1b8106560 2024-04-22T15:40:13.690 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.857+0000 7ff1bf479700 1 -- 172.21.15.16:0/4136564451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1b8106560 msgr2=0x7ff1b8106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:13.690 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.857+0000 7ff1bf479700 1 --2- 172.21.15.16:0/4136564451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1b8106560 0x7ff1b8106980 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff1b40098a0 tx=0x7ff1b4009bb0 comp rx=0 tx=0).stop 2024-04-22T15:40:13.690 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.858+0000 7ff1bf479700 1 -- 172.21.15.16:0/4136564451 shutdown_connections 2024-04-22T15:40:13.690 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.858+0000 7ff1bf479700 1 --2- 172.21.15.16:0/4136564451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1b8106560 0x7ff1b8106980 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:13.690 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.858+0000 7ff1bf479700 1 -- 172.21.15.16:0/4136564451 >> 172.21.15.16:0/4136564451 conn(0x7ff1b8101b40 msgr2=0x7ff1b8103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:13.690 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.858+0000 7ff1bf479700 1 -- 172.21.15.16:0/4136564451 shutdown_connections 2024-04-22T15:40:13.690 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.858+0000 7ff1bf479700 1 -- 172.21.15.16:0/4136564451 wait complete. 2024-04-22T15:40:13.690 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.858+0000 7ff1bf479700 1 Processor -- start 2024-04-22T15:40:13.691 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.858+0000 7ff1bf479700 1 -- start start 2024-04-22T15:40:13.691 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.858+0000 7ff1bf479700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1b8106560 0x7ff1b807bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:13.691 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.858+0000 7ff1bf479700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b807c170 con 0x7ff1b8106560 2024-04-22T15:40:13.691 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.859+0000 7ff1bd215700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1b8106560 0x7ff1b807bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:13.691 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.859+0000 7ff1bd215700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1b8106560 0x7ff1b807bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58312/0 (socket says 172.21.15.16:58312) 2024-04-22T15:40:13.691 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.859+0000 7ff1bd215700 1 -- 172.21.15.16:0/1478951768 learned_addr learned my addr 172.21.15.16:0/1478951768 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:13.691 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.859+0000 7ff1bd215700 1 -- 172.21.15.16:0/1478951768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b4009550 con 0x7ff1b8106560 2024-04-22T15:40:13.691 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.859+0000 7ff1bd215700 1 --2- 172.21.15.16:0/1478951768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1b8106560 0x7ff1b807bc30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4011670 tx=0x7ff1b4011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:13.691 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.859+0000 7ff1ae7fc700 1 -- 172.21.15.16:0/1478951768 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1b40039f0 con 0x7ff1b8106560 2024-04-22T15:40:13.691 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.859+0000 7ff1ae7fc700 1 -- 172.21.15.16:0/1478951768 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff1b4003b50 con 0x7ff1b8106560 2024-04-22T15:40:13.691 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.859+0000 7ff1ae7fc700 1 -- 172.21.15.16:0/1478951768 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1b40247c0 con 0x7ff1b8106560 2024-04-22T15:40:13.691 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.859+0000 7ff1bf479700 1 -- 172.21.15.16:0/1478951768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1b807a280 con 0x7ff1b8106560 2024-04-22T15:40:13.691 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.859+0000 7ff1bf479700 1 -- 172.21.15.16:0/1478951768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1b807a720 con 0x7ff1b8106560 2024-04-22T15:40:13.691 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.860+0000 7ff1ae7fc700 1 -- 172.21.15.16:0/1478951768 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7ff1b401ac00 con 0x7ff1b8106560 2024-04-22T15:40:13.692 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.860+0000 7ff1bf479700 1 -- 172.21.15.16:0/1478951768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1b80623c0 con 0x7ff1b8106560 2024-04-22T15:40:13.692 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.860+0000 7ff1ae7fc700 1 --2- 172.21.15.16:0/1478951768 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7ff1a4038370 0x7ff1a403a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:13.692 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.861+0000 7ff1ae7fc700 1 -- 172.21.15.16:0/1478951768 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff1b404c3a0 con 0x7ff1b8106560 2024-04-22T15:40:13.692 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.861+0000 7ff1bca14700 1 --2- 172.21.15.16:0/1478951768 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7ff1a4038370 0x7ff1a403a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:13.692 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.861+0000 7ff1bca14700 1 --2- 172.21.15.16:0/1478951768 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7ff1a4038370 0x7ff1a403a830 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8006fd0 tx=0x7ff1a8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:13.692 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:12.865+0000 7ff1ae7fc700 1 -- 172.21.15.16:0/1478951768 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff1b401a430 con 0x7ff1b8106560 2024-04-22T15:40:13.692 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:13.077+0000 7ff1bf479700 1 -- 172.21.15.16:0/1478951768 --> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1b810a960 con 0x7ff1a4038370 2024-04-22T15:40:13.692 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:13.081+0000 7ff1ae7fc700 1 -- 172.21.15.16:0/1478951768 <== mgr.14118 v2:172.21.15.16:6800/3431522404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7ff1b810a960 con 0x7ff1a4038370 2024-04-22T15:40:13.692 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:13.084+0000 7ff1bf479700 1 -- 172.21.15.16:0/1478951768 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7ff1a4038370 msgr2=0x7ff1a403a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:13.692 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:13.084+0000 7ff1bf479700 1 --2- 172.21.15.16:0/1478951768 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7ff1a4038370 0x7ff1a403a830 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8006fd0 tx=0x7ff1a8006e40 comp rx=0 tx=0).stop 2024-04-22T15:40:13.692 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:13.084+0000 7ff1bf479700 1 -- 172.21.15.16:0/1478951768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1b8106560 msgr2=0x7ff1b807bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:13.692 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:13.084+0000 7ff1bf479700 1 --2- 172.21.15.16:0/1478951768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1b8106560 0x7ff1b807bc30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4011670 tx=0x7ff1b4011b50 comp rx=0 tx=0).stop 2024-04-22T15:40:13.692 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:13.084+0000 7ff1bf479700 1 -- 172.21.15.16:0/1478951768 shutdown_connections 2024-04-22T15:40:13.692 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:13.084+0000 7ff1bf479700 1 --2- 172.21.15.16:0/1478951768 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7ff1a4038370 0x7ff1a403a830 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:13.692 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:13.084+0000 7ff1bf479700 1 --2- 172.21.15.16:0/1478951768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1b8106560 0x7ff1b807bc30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:13.692 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:13.084+0000 7ff1bf479700 1 -- 172.21.15.16:0/1478951768 >> 172.21.15.16:0/1478951768 conn(0x7ff1b8101b40 msgr2=0x7ff1b8102920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:13.693 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:13.084+0000 7ff1bf479700 1 -- 172.21.15.16:0/1478951768 shutdown_connections 2024-04-22T15:40:13.693 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:13.084+0000 7ff1bf479700 1 -- 172.21.15.16:0/1478951768 wait complete. 2024-04-22T15:40:13.693 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying node-exporter service with default placement... 2024-04-22T15:40:14.312 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:14 smithi016 ceph-mon[26620]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:14.313 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:14 smithi016 ceph-mon[26620]: Saving service grafana spec with placement count:1 2024-04-22T15:40:14.313 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:14 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:14.313 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:14 smithi016 ceph-mon[26620]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:40:14.870 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-04-22T15:40:14.870 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.089+0000 7f629d785700 1 Processor -- start 2024-04-22T15:40:14.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.089+0000 7f629d785700 1 -- start start 2024-04-22T15:40:14.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.090+0000 7f629d785700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6298106fe0 0x7f62981093d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:14.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.090+0000 7f629d785700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6298074680 con 0x7f6298106fe0 2024-04-22T15:40:14.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.090+0000 7f6296ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6298106fe0 0x7f62981093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:14.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.090+0000 7f6296ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6298106fe0 0x7f62981093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58316/0 (socket says 172.21.15.16:58316) 2024-04-22T15:40:14.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.090+0000 7f6296ffd700 1 -- 172.21.15.16:0/1681878577 learned_addr learned my addr 172.21.15.16:0/1681878577 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:14.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.090+0000 7f6296ffd700 1 -- 172.21.15.16:0/1681878577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6298109910 con 0x7f6298106fe0 2024-04-22T15:40:14.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.090+0000 7f6296ffd700 1 --2- 172.21.15.16:0/1681878577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6298106fe0 0x7f62981093d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6290009e80 tx=0x7f629000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98bf4f878d01bbe4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:14.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.090+0000 7f6295ffb700 1 -- 172.21.15.16:0/1681878577 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f629000e070 con 0x7f6298106fe0 2024-04-22T15:40:14.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.090+0000 7f6295ffb700 1 -- 172.21.15.16:0/1681878577 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f629000b8c0 con 0x7f6298106fe0 2024-04-22T15:40:14.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.090+0000 7f6295ffb700 1 -- 172.21.15.16:0/1681878577 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62900044d0 con 0x7f6298106fe0 2024-04-22T15:40:14.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.091+0000 7f629d785700 1 -- 172.21.15.16:0/1681878577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6298106fe0 msgr2=0x7f62981093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:14.871 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.091+0000 7f629d785700 1 --2- 172.21.15.16:0/1681878577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6298106fe0 0x7f62981093d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6290009e80 tx=0x7f629000b330 comp rx=0 tx=0).stop 2024-04-22T15:40:14.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.091+0000 7f629d785700 1 -- 172.21.15.16:0/1681878577 shutdown_connections 2024-04-22T15:40:14.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.091+0000 7f629d785700 1 --2- 172.21.15.16:0/1681878577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6298106fe0 0x7f62981093d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:14.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.091+0000 7f629d785700 1 -- 172.21.15.16:0/1681878577 >> 172.21.15.16:0/1681878577 conn(0x7f62981009f0 msgr2=0x7f6298102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:14.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.091+0000 7f629d785700 1 -- 172.21.15.16:0/1681878577 shutdown_connections 2024-04-22T15:40:14.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.091+0000 7f629d785700 1 -- 172.21.15.16:0/1681878577 wait complete. 2024-04-22T15:40:14.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.092+0000 7f629d785700 1 Processor -- start 2024-04-22T15:40:14.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.092+0000 7f629d785700 1 -- start start 2024-04-22T15:40:14.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.092+0000 7f629d785700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6298106fe0 0x7f6298106940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:14.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.092+0000 7f629d785700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6298106e80 con 0x7f6298106fe0 2024-04-22T15:40:14.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.092+0000 7f6296ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6298106fe0 0x7f6298106940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:14.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.092+0000 7f6296ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6298106fe0 0x7f6298106940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58322/0 (socket says 172.21.15.16:58322) 2024-04-22T15:40:14.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.092+0000 7f6296ffd700 1 -- 172.21.15.16:0/1049119839 learned_addr learned my addr 172.21.15.16:0/1049119839 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:14.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.092+0000 7f6296ffd700 1 -- 172.21.15.16:0/1049119839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6290009b30 con 0x7f6298106fe0 2024-04-22T15:40:14.872 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.093+0000 7f6296ffd700 1 --2- 172.21.15.16:0/1049119839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6298106fe0 0x7f6298106940 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6290004070 tx=0x7f62900049f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:14.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.093+0000 7f628ffff700 1 -- 172.21.15.16:0/1049119839 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f629000e040 con 0x7f6298106fe0 2024-04-22T15:40:14.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.093+0000 7f629d785700 1 -- 172.21.15.16:0/1049119839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6298104f90 con 0x7f6298106fe0 2024-04-22T15:40:14.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.093+0000 7f628ffff700 1 -- 172.21.15.16:0/1049119839 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f629000be00 con 0x7f6298106fe0 2024-04-22T15:40:14.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.093+0000 7f628ffff700 1 -- 172.21.15.16:0/1049119839 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6290012770 con 0x7f6298106fe0 2024-04-22T15:40:14.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.093+0000 7f629d785700 1 -- 172.21.15.16:0/1049119839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6298105490 con 0x7f6298106fe0 2024-04-22T15:40:14.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.094+0000 7f628ffff700 1 -- 172.21.15.16:0/1049119839 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f6290019040 con 0x7f6298106fe0 2024-04-22T15:40:14.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.094+0000 7f629d785700 1 -- 172.21.15.16:0/1049119839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f629810a700 con 0x7f6298106fe0 2024-04-22T15:40:14.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.094+0000 7f628ffff700 1 --2- 172.21.15.16:0/1049119839 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f6284038320 0x7f628403a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:14.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.094+0000 7f628ffff700 1 -- 172.21.15.16:0/1049119839 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f629004bd90 con 0x7f6298106fe0 2024-04-22T15:40:14.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.094+0000 7f62967fc700 1 --2- 172.21.15.16:0/1049119839 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f6284038320 0x7f628403a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:14.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.094+0000 7f62967fc700 1 --2- 172.21.15.16:0/1049119839 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f6284038320 0x7f628403a7e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6280006fd0 tx=0x7f6280006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:14.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.099+0000 7f628ffff700 1 -- 172.21.15.16:0/1049119839 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6290011750 con 0x7f6298106fe0 2024-04-22T15:40:14.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.309+0000 7f629d785700 1 -- 172.21.15.16:0/1049119839 --> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f6298101df0 con 0x7f6284038320 2024-04-22T15:40:14.873 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.314+0000 7f628ffff700 1 -- 172.21.15.16:0/1049119839 <== mgr.14118 v2:172.21.15.16:6800/3431522404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f6298101df0 con 0x7f6284038320 2024-04-22T15:40:14.874 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.316+0000 7f629d785700 1 -- 172.21.15.16:0/1049119839 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f6284038320 msgr2=0x7f628403a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:14.874 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.316+0000 7f629d785700 1 --2- 172.21.15.16:0/1049119839 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f6284038320 0x7f628403a7e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6280006fd0 tx=0x7f6280006e40 comp rx=0 tx=0).stop 2024-04-22T15:40:14.874 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.316+0000 7f629d785700 1 -- 172.21.15.16:0/1049119839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6298106fe0 msgr2=0x7f6298106940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:14.874 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.316+0000 7f629d785700 1 --2- 172.21.15.16:0/1049119839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6298106fe0 0x7f6298106940 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6290004070 tx=0x7f62900049f0 comp rx=0 tx=0).stop 2024-04-22T15:40:14.874 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.316+0000 7f629d785700 1 -- 172.21.15.16:0/1049119839 shutdown_connections 2024-04-22T15:40:14.874 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.316+0000 7f629d785700 1 --2- 172.21.15.16:0/1049119839 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f6284038320 0x7f628403a7e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:14.874 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.316+0000 7f629d785700 1 --2- 172.21.15.16:0/1049119839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6298106fe0 0x7f6298106940 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:14.874 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.316+0000 7f629d785700 1 -- 172.21.15.16:0/1049119839 >> 172.21.15.16:0/1049119839 conn(0x7f62981009f0 msgr2=0x7f62981016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:14.874 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.316+0000 7f629d785700 1 -- 172.21.15.16:0/1049119839 shutdown_connections 2024-04-22T15:40:14.874 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:14.316+0000 7f629d785700 1 -- 172.21.15.16:0/1049119839 wait complete. 2024-04-22T15:40:14.874 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying alertmanager service with default placement... 2024-04-22T15:40:15.501 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:15 smithi016 ceph-mon[26620]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:15.501 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:15 smithi016 ceph-mon[26620]: Saving service node-exporter spec with placement * 2024-04-22T15:40:15.501 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:15 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:16.103 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-04-22T15:40:16.103 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.274+0000 7fd618afb700 1 Processor -- start 2024-04-22T15:40:16.103 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.274+0000 7fd618afb700 1 -- start start 2024-04-22T15:40:16.103 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.275+0000 7fd618afb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd614104830 0x7fd614104c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:16.103 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.275+0000 7fd618afb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd614105220 con 0x7fd614104830 2024-04-22T15:40:16.103 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.275+0000 7fd61259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd614104830 0x7fd614104c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:16.103 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.275+0000 7fd61259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd614104830 0x7fd614104c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58324/0 (socket says 172.21.15.16:58324) 2024-04-22T15:40:16.103 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.275+0000 7fd61259c700 1 -- 172.21.15.16:0/2514878379 learned_addr learned my addr 172.21.15.16:0/2514878379 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:16.103 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.275+0000 7fd61259c700 1 -- 172.21.15.16:0/2514878379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd614105360 con 0x7fd614104830 2024-04-22T15:40:16.103 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.275+0000 7fd61259c700 1 --2- 172.21.15.16:0/2514878379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd614104830 0x7fd614104c50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd6040098a0 tx=0x7fd604009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd2dcff020aa02db server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:16.103 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.275+0000 7fd61159a700 1 -- 172.21.15.16:0/2514878379 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd604004180 con 0x7fd614104830 2024-04-22T15:40:16.104 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.276+0000 7fd61159a700 1 -- 172.21.15.16:0/2514878379 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd6040042e0 con 0x7fd614104830 2024-04-22T15:40:16.104 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.276+0000 7fd61159a700 1 -- 172.21.15.16:0/2514878379 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd604011af0 con 0x7fd614104830 2024-04-22T15:40:16.104 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.276+0000 7fd618afb700 1 -- 172.21.15.16:0/2514878379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd614104830 msgr2=0x7fd614104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:16.104 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.276+0000 7fd618afb700 1 --2- 172.21.15.16:0/2514878379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd614104830 0x7fd614104c50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd6040098a0 tx=0x7fd604009bb0 comp rx=0 tx=0).stop 2024-04-22T15:40:16.104 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.276+0000 7fd618afb700 1 -- 172.21.15.16:0/2514878379 shutdown_connections 2024-04-22T15:40:16.104 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.276+0000 7fd618afb700 1 --2- 172.21.15.16:0/2514878379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd614104830 0x7fd614104c50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:16.104 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.276+0000 7fd618afb700 1 -- 172.21.15.16:0/2514878379 >> 172.21.15.16:0/2514878379 conn(0x7fd614100190 msgr2=0x7fd6141025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:16.104 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.276+0000 7fd618afb700 1 -- 172.21.15.16:0/2514878379 shutdown_connections 2024-04-22T15:40:16.104 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.276+0000 7fd618afb700 1 -- 172.21.15.16:0/2514878379 wait complete. 2024-04-22T15:40:16.104 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.277+0000 7fd618afb700 1 Processor -- start 2024-04-22T15:40:16.104 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.277+0000 7fd618afb700 1 -- start start 2024-04-22T15:40:16.104 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.278+0000 7fd618afb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd614104830 0x7fd61419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:16.104 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.278+0000 7fd618afb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd61419a730 con 0x7fd614104830 2024-04-22T15:40:16.104 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.278+0000 7fd61259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd614104830 0x7fd61419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:16.105 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.278+0000 7fd61259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd614104830 0x7fd61419a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58326/0 (socket says 172.21.15.16:58326) 2024-04-22T15:40:16.105 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.278+0000 7fd61259c700 1 -- 172.21.15.16:0/3294152600 learned_addr learned my addr 172.21.15.16:0/3294152600 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:16.105 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.278+0000 7fd61259c700 1 -- 172.21.15.16:0/3294152600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd604009550 con 0x7fd614104830 2024-04-22T15:40:16.105 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.278+0000 7fd61259c700 1 --2- 172.21.15.16:0/3294152600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd614104830 0x7fd61419a1f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd604000c00 tx=0x7fd604011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:16.105 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.278+0000 7fd6037fe700 1 -- 172.21.15.16:0/3294152600 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd604003a70 con 0x7fd614104830 2024-04-22T15:40:16.105 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.278+0000 7fd6037fe700 1 -- 172.21.15.16:0/3294152600 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd604003bd0 con 0x7fd614104830 2024-04-22T15:40:16.105 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.278+0000 7fd618afb700 1 -- 172.21.15.16:0/3294152600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd61419a930 con 0x7fd614104830 2024-04-22T15:40:16.105 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.278+0000 7fd618afb700 1 -- 172.21.15.16:0/3294152600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd61419add0 con 0x7fd614104830 2024-04-22T15:40:16.105 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.278+0000 7fd6037fe700 1 -- 172.21.15.16:0/3294152600 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6040238c0 con 0x7fd614104830 2024-04-22T15:40:16.105 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.279+0000 7fd6037fe700 1 -- 172.21.15.16:0/3294152600 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fd604023b30 con 0x7fd614104830 2024-04-22T15:40:16.105 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.279+0000 7fd618afb700 1 -- 172.21.15.16:0/3294152600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6140623c0 con 0x7fd614104830 2024-04-22T15:40:16.105 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.280+0000 7fd6037fe700 1 --2- 172.21.15.16:0/3294152600 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fd5fc038370 0x7fd5fc03a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:16.105 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.280+0000 7fd6037fe700 1 -- 172.21.15.16:0/3294152600 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd60404c170 con 0x7fd614104830 2024-04-22T15:40:16.105 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.280+0000 7fd611d9b700 1 --2- 172.21.15.16:0/3294152600 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fd5fc038370 0x7fd5fc03a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:16.106 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.280+0000 7fd611d9b700 1 --2- 172.21.15.16:0/3294152600 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fd5fc038370 0x7fd5fc03a830 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd608006fd0 tx=0x7fd608006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:16.106 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.285+0000 7fd6037fe700 1 -- 172.21.15.16:0/3294152600 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd604010c20 con 0x7fd614104830 2024-04-22T15:40:16.106 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.498+0000 7fd618afb700 1 -- 172.21.15.16:0/3294152600 --> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fd614101640 con 0x7fd5fc038370 2024-04-22T15:40:16.106 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.502+0000 7fd6037fe700 1 -- 172.21.15.16:0/3294152600 <== mgr.14118 v2:172.21.15.16:6800/3431522404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fd614101640 con 0x7fd5fc038370 2024-04-22T15:40:16.106 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.505+0000 7fd618afb700 1 -- 172.21.15.16:0/3294152600 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fd5fc038370 msgr2=0x7fd5fc03a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:16.106 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.505+0000 7fd618afb700 1 --2- 172.21.15.16:0/3294152600 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fd5fc038370 0x7fd5fc03a830 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd608006fd0 tx=0x7fd608006e40 comp rx=0 tx=0).stop 2024-04-22T15:40:16.106 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.505+0000 7fd618afb700 1 -- 172.21.15.16:0/3294152600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd614104830 msgr2=0x7fd61419a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:16.106 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.505+0000 7fd618afb700 1 --2- 172.21.15.16:0/3294152600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd614104830 0x7fd61419a1f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd604000c00 tx=0x7fd604011b50 comp rx=0 tx=0).stop 2024-04-22T15:40:16.106 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.505+0000 7fd618afb700 1 -- 172.21.15.16:0/3294152600 shutdown_connections 2024-04-22T15:40:16.106 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.505+0000 7fd618afb700 1 --2- 172.21.15.16:0/3294152600 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fd5fc038370 0x7fd5fc03a830 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:16.106 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.505+0000 7fd618afb700 1 --2- 172.21.15.16:0/3294152600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd614104830 0x7fd61419a1f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:16.106 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.505+0000 7fd618afb700 1 -- 172.21.15.16:0/3294152600 >> 172.21.15.16:0/3294152600 conn(0x7fd614100190 msgr2=0x7fd614100f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:16.106 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.505+0000 7fd618afb700 1 -- 172.21.15.16:0/3294152600 shutdown_connections 2024-04-22T15:40:16.107 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:15.505+0000 7fd618afb700 1 -- 172.21.15.16:0/3294152600 wait complete. 2024-04-22T15:40:16.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:16 smithi016 ceph-mon[26620]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:40:16.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:16 smithi016 ceph-mon[26620]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:16.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:16 smithi016 ceph-mon[26620]: Saving service alertmanager spec with placement count:1 2024-04-22T15:40:16.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:16 smithi016 ceph-mon[26620]: from='mgr.14118 172.21.15.16:0/1210457757' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:17.455 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.533+0000 7f7ccb3ac700 1 Processor -- start 2024-04-22T15:40:17.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.534+0000 7f7ccb3ac700 1 -- start start 2024-04-22T15:40:17.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.534+0000 7f7ccb3ac700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cc4105470 0x7f7cc4105890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:17.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.534+0000 7f7ccb3ac700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc4105e60 con 0x7f7cc4105470 2024-04-22T15:40:17.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.534+0000 7f7cc9148700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cc4105470 0x7f7cc4105890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:17.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.534+0000 7f7cc9148700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cc4105470 0x7f7cc4105890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58328/0 (socket says 172.21.15.16:58328) 2024-04-22T15:40:17.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.534+0000 7f7cc9148700 1 -- 172.21.15.16:0/3181202604 learned_addr learned my addr 172.21.15.16:0/3181202604 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:17.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.534+0000 7f7cc9148700 1 -- 172.21.15.16:0/3181202604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc4106670 con 0x7f7cc4105470 2024-04-22T15:40:17.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.534+0000 7f7cc9148700 1 --2- 172.21.15.16:0/3181202604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cc4105470 0x7f7cc4105890 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7cc00098a0 tx=0x7f7cc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37e01482514571d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:17.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.535+0000 7f7cbbfff700 1 -- 172.21.15.16:0/3181202604 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cc0004180 con 0x7f7cc4105470 2024-04-22T15:40:17.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.535+0000 7f7cbbfff700 1 -- 172.21.15.16:0/3181202604 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7cc00042e0 con 0x7f7cc4105470 2024-04-22T15:40:17.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.535+0000 7f7cbbfff700 1 -- 172.21.15.16:0/3181202604 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cc0011af0 con 0x7f7cc4105470 2024-04-22T15:40:17.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.535+0000 7f7ccb3ac700 1 -- 172.21.15.16:0/3181202604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cc4105470 msgr2=0x7f7cc4105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:17.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.535+0000 7f7ccb3ac700 1 --2- 172.21.15.16:0/3181202604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cc4105470 0x7f7cc4105890 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7cc00098a0 tx=0x7f7cc0009bb0 comp rx=0 tx=0).stop 2024-04-22T15:40:17.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.535+0000 7f7ccb3ac700 1 -- 172.21.15.16:0/3181202604 shutdown_connections 2024-04-22T15:40:17.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.535+0000 7f7ccb3ac700 1 --2- 172.21.15.16:0/3181202604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cc4105470 0x7f7cc4105890 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:17.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.535+0000 7f7ccb3ac700 1 -- 172.21.15.16:0/3181202604 >> 172.21.15.16:0/3181202604 conn(0x7f7cc41009f0 msgr2=0x7f7cc4102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:17.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.535+0000 7f7ccb3ac700 1 -- 172.21.15.16:0/3181202604 shutdown_connections 2024-04-22T15:40:17.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.535+0000 7f7ccb3ac700 1 -- 172.21.15.16:0/3181202604 wait complete. 2024-04-22T15:40:17.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.536+0000 7f7ccb3ac700 1 Processor -- start 2024-04-22T15:40:17.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.536+0000 7f7ccb3ac700 1 -- start start 2024-04-22T15:40:17.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.536+0000 7f7ccb3ac700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cc4105470 0x7f7cc419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:17.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.536+0000 7f7ccb3ac700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc419a750 con 0x7f7cc4105470 2024-04-22T15:40:17.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.536+0000 7f7cc9148700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cc4105470 0x7f7cc419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:17.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.536+0000 7f7cc9148700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cc4105470 0x7f7cc419a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58342/0 (socket says 172.21.15.16:58342) 2024-04-22T15:40:17.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.536+0000 7f7cc9148700 1 -- 172.21.15.16:0/2687447282 learned_addr learned my addr 172.21.15.16:0/2687447282 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:17.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.537+0000 7f7cc9148700 1 -- 172.21.15.16:0/2687447282 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc0009550 con 0x7f7cc4105470 2024-04-22T15:40:17.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.537+0000 7f7cc9148700 1 --2- 172.21.15.16:0/2687447282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cc4105470 0x7f7cc419a210 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0006ba0 tx=0x7f7cc0011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:17.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.537+0000 7f7cba7fc700 1 -- 172.21.15.16:0/2687447282 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cc0003b00 con 0x7f7cc4105470 2024-04-22T15:40:17.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.537+0000 7f7cba7fc700 1 -- 172.21.15.16:0/2687447282 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7cc001a8e0 con 0x7f7cc4105470 2024-04-22T15:40:17.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.537+0000 7f7ccb3ac700 1 -- 172.21.15.16:0/2687447282 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cc419a950 con 0x7f7cc4105470 2024-04-22T15:40:17.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.537+0000 7f7cba7fc700 1 -- 172.21.15.16:0/2687447282 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cc0019990 con 0x7f7cc4105470 2024-04-22T15:40:17.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.537+0000 7f7ccb3ac700 1 -- 172.21.15.16:0/2687447282 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cc419adf0 con 0x7f7cc4105470 2024-04-22T15:40:17.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.538+0000 7f7cba7fc700 1 -- 172.21.15.16:0/2687447282 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f7cc0019c50 con 0x7f7cc4105470 2024-04-22T15:40:17.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.538+0000 7f7ccb3ac700 1 -- 172.21.15.16:0/2687447282 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cc40623c0 con 0x7f7cc4105470 2024-04-22T15:40:17.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.538+0000 7f7cba7fc700 1 --2- 172.21.15.16:0/2687447282 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f7cb00382d0 0x7f7cb003a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:17.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.538+0000 7f7cba7fc700 1 -- 172.21.15.16:0/2687447282 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7cc004c5c0 con 0x7f7cc4105470 2024-04-22T15:40:17.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.538+0000 7f7cc8947700 1 --2- 172.21.15.16:0/2687447282 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f7cb00382d0 0x7f7cb003a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:17.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.539+0000 7f7cc8947700 1 --2- 172.21.15.16:0/2687447282 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f7cb00382d0 0x7f7cb003a790 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4006fd0 tx=0x7f7cb4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:17.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.543+0000 7f7cba7fc700 1 -- 172.21.15.16:0/2687447282 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7cc0018530 con 0x7f7cc4105470 2024-04-22T15:40:17.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.740+0000 7f7ccb3ac700 1 -- 172.21.15.16:0/2687447282 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f7cc419dcf0 con 0x7f7cc4105470 2024-04-22T15:40:17.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.743+0000 7f7cba7fc700 1 -- 172.21.15.16:0/2687447282 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f7cc0018350 con 0x7f7cc4105470 2024-04-22T15:40:17.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.745+0000 7f7ccb3ac700 1 -- 172.21.15.16:0/2687447282 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f7cb00382d0 msgr2=0x7f7cb003a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:17.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.745+0000 7f7ccb3ac700 1 --2- 172.21.15.16:0/2687447282 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f7cb00382d0 0x7f7cb003a790 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4006fd0 tx=0x7f7cb4006e40 comp rx=0 tx=0).stop 2024-04-22T15:40:17.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.745+0000 7f7ccb3ac700 1 -- 172.21.15.16:0/2687447282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cc4105470 msgr2=0x7f7cc419a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:17.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.745+0000 7f7ccb3ac700 1 --2- 172.21.15.16:0/2687447282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cc4105470 0x7f7cc419a210 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0006ba0 tx=0x7f7cc0011b50 comp rx=0 tx=0).stop 2024-04-22T15:40:17.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.745+0000 7f7ccb3ac700 1 -- 172.21.15.16:0/2687447282 shutdown_connections 2024-04-22T15:40:17.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.745+0000 7f7ccb3ac700 1 --2- 172.21.15.16:0/2687447282 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f7cb00382d0 0x7f7cb003a790 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:17.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.745+0000 7f7ccb3ac700 1 --2- 172.21.15.16:0/2687447282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cc4105470 0x7f7cc419a210 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:17.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.745+0000 7f7ccb3ac700 1 -- 172.21.15.16:0/2687447282 >> 172.21.15.16:0/2687447282 conn(0x7f7cc41009f0 msgr2=0x7f7cc4079900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:17.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.745+0000 7f7ccb3ac700 1 -- 172.21.15.16:0/2687447282 shutdown_connections 2024-04-22T15:40:17.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:16.745+0000 7f7ccb3ac700 1 -- 172.21.15.16:0/2687447282 wait complete. 2024-04-22T15:40:17.904 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:17 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/2687447282' entity='client.admin' 2024-04-22T15:40:17.905 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:17 smithi016 ceph-mon[26620]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:40:18.799 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.942+0000 7fc92e571700 1 Processor -- start 2024-04-22T15:40:18.799 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.942+0000 7fc92e571700 1 -- start start 2024-04-22T15:40:18.799 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.943+0000 7fc92e571700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc928107040 0x7fc928109430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:18.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.943+0000 7fc92e571700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc928074620 con 0x7fc928107040 2024-04-22T15:40:18.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.943+0000 7fc927fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc928107040 0x7fc928109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:18.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.943+0000 7fc927fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc928107040 0x7fc928109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58358/0 (socket says 172.21.15.16:58358) 2024-04-22T15:40:18.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.943+0000 7fc927fff700 1 -- 172.21.15.16:0/2588754948 learned_addr learned my addr 172.21.15.16:0/2588754948 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:18.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.943+0000 7fc927fff700 1 -- 172.21.15.16:0/2588754948 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc928109970 con 0x7fc928107040 2024-04-22T15:40:18.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.943+0000 7fc927fff700 1 --2- 172.21.15.16:0/2588754948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc928107040 0x7fc928109430 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc9180098a0 tx=0x7fc918009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f8391760f78a58 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:18.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.943+0000 7fc926ffd700 1 -- 172.21.15.16:0/2588754948 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc918004180 con 0x7fc928107040 2024-04-22T15:40:18.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.943+0000 7fc926ffd700 1 -- 172.21.15.16:0/2588754948 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc9180042e0 con 0x7fc928107040 2024-04-22T15:40:18.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.943+0000 7fc926ffd700 1 -- 172.21.15.16:0/2588754948 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc918011af0 con 0x7fc928107040 2024-04-22T15:40:18.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.944+0000 7fc92e571700 1 -- 172.21.15.16:0/2588754948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc928107040 msgr2=0x7fc928109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:18.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.944+0000 7fc92e571700 1 --2- 172.21.15.16:0/2588754948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc928107040 0x7fc928109430 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc9180098a0 tx=0x7fc918009bb0 comp rx=0 tx=0).stop 2024-04-22T15:40:18.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.944+0000 7fc92e571700 1 -- 172.21.15.16:0/2588754948 shutdown_connections 2024-04-22T15:40:18.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.944+0000 7fc92e571700 1 --2- 172.21.15.16:0/2588754948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc928107040 0x7fc928109430 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:18.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.944+0000 7fc92e571700 1 -- 172.21.15.16:0/2588754948 >> 172.21.15.16:0/2588754948 conn(0x7fc9281009f0 msgr2=0x7fc928102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:18.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.944+0000 7fc92e571700 1 -- 172.21.15.16:0/2588754948 shutdown_connections 2024-04-22T15:40:18.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.944+0000 7fc92e571700 1 -- 172.21.15.16:0/2588754948 wait complete. 2024-04-22T15:40:18.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.945+0000 7fc92e571700 1 Processor -- start 2024-04-22T15:40:18.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.945+0000 7fc92e571700 1 -- start start 2024-04-22T15:40:18.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.945+0000 7fc92e571700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc928107040 0x7fc92819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:18.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.945+0000 7fc92e571700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc92819a6a0 con 0x7fc928107040 2024-04-22T15:40:18.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.945+0000 7fc927fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc928107040 0x7fc92819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:18.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.945+0000 7fc927fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc928107040 0x7fc92819a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58366/0 (socket says 172.21.15.16:58366) 2024-04-22T15:40:18.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.945+0000 7fc927fff700 1 -- 172.21.15.16:0/597621585 learned_addr learned my addr 172.21.15.16:0/597621585 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:18.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.946+0000 7fc927fff700 1 -- 172.21.15.16:0/597621585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc918009550 con 0x7fc928107040 2024-04-22T15:40:18.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.946+0000 7fc927fff700 1 --2- 172.21.15.16:0/597621585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc928107040 0x7fc92819a160 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc918011ad0 tx=0x7fc918011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:18.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.946+0000 7fc9257fa700 1 -- 172.21.15.16:0/597621585 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc918003ab0 con 0x7fc928107040 2024-04-22T15:40:18.802 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.946+0000 7fc9257fa700 1 -- 172.21.15.16:0/597621585 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc918003c10 con 0x7fc928107040 2024-04-22T15:40:18.802 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.946+0000 7fc9257fa700 1 -- 172.21.15.16:0/597621585 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9180198e0 con 0x7fc928107040 2024-04-22T15:40:18.802 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.946+0000 7fc92e571700 1 -- 172.21.15.16:0/597621585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc92819a8a0 con 0x7fc928107040 2024-04-22T15:40:18.802 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.946+0000 7fc92e571700 1 -- 172.21.15.16:0/597621585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc92819ad40 con 0x7fc928107040 2024-04-22T15:40:18.802 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.947+0000 7fc9257fa700 1 -- 172.21.15.16:0/597621585 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fc918019b50 con 0x7fc928107040 2024-04-22T15:40:18.802 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.947+0000 7fc92e571700 1 -- 172.21.15.16:0/597621585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc92810a700 con 0x7fc928107040 2024-04-22T15:40:18.802 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.947+0000 7fc9257fa700 1 --2- 172.21.15.16:0/597621585 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fc910038370 0x7fc91003a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:18.802 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.947+0000 7fc9257fa700 1 -- 172.21.15.16:0/597621585 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc91804c120 con 0x7fc928107040 2024-04-22T15:40:18.802 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.947+0000 7fc9277fe700 1 --2- 172.21.15.16:0/597621585 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fc910038370 0x7fc91003a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:18.802 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.948+0000 7fc9277fe700 1 --2- 172.21.15.16:0/597621585 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fc910038370 0x7fc91003a830 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc91c006fd0 tx=0x7fc91c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:18.802 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:17.952+0000 7fc9257fa700 1 -- 172.21.15.16:0/597621585 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc91801a430 con 0x7fc928107040 2024-04-22T15:40:18.802 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:18.150+0000 7fc92e571700 1 -- 172.21.15.16:0/597621585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fc92819db20 con 0x7fc928107040 2024-04-22T15:40:18.803 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:18.154+0000 7fc9257fa700 1 -- 172.21.15.16:0/597621585 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fc91802c570 con 0x7fc928107040 2024-04-22T15:40:18.803 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:18.156+0000 7fc92e571700 1 -- 172.21.15.16:0/597621585 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fc910038370 msgr2=0x7fc91003a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:18.803 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:18.156+0000 7fc92e571700 1 --2- 172.21.15.16:0/597621585 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fc910038370 0x7fc91003a830 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc91c006fd0 tx=0x7fc91c006e40 comp rx=0 tx=0).stop 2024-04-22T15:40:18.803 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:18.156+0000 7fc92e571700 1 -- 172.21.15.16:0/597621585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc928107040 msgr2=0x7fc92819a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:18.803 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:18.156+0000 7fc92e571700 1 --2- 172.21.15.16:0/597621585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc928107040 0x7fc92819a160 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc918011ad0 tx=0x7fc918011bb0 comp rx=0 tx=0).stop 2024-04-22T15:40:18.803 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:18.156+0000 7fc92e571700 1 -- 172.21.15.16:0/597621585 shutdown_connections 2024-04-22T15:40:18.803 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:18.156+0000 7fc92e571700 1 --2- 172.21.15.16:0/597621585 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fc910038370 0x7fc91003a830 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:18.803 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:18.156+0000 7fc92e571700 1 --2- 172.21.15.16:0/597621585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc928107040 0x7fc92819a160 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:18.803 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:18.156+0000 7fc92e571700 1 -- 172.21.15.16:0/597621585 >> 172.21.15.16:0/597621585 conn(0x7fc9281009f0 msgr2=0x7fc928101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:18.803 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:18.156+0000 7fc92e571700 1 -- 172.21.15.16:0/597621585 shutdown_connections 2024-04-22T15:40:18.803 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:18.156+0000 7fc92e571700 1 -- 172.21.15.16:0/597621585 wait complete. 2024-04-22T15:40:18.803 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling the dashboard module... 2024-04-22T15:40:19.205 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:19 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/597621585' entity='client.admin' 2024-04-22T15:40:20.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:20 smithi016 ceph-mon[26620]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:40:20.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:20 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/2849930550' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-22T15:40:21.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.313+0000 7fdfacd70700 1 Processor -- start 2024-04-22T15:40:21.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.313+0000 7fdfacd70700 1 -- start start 2024-04-22T15:40:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.313+0000 7fdfacd70700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfa8106ab0 0x7fdfa8106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.313+0000 7fdfacd70700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa81074a0 con 0x7fdfa8106ab0 2024-04-22T15:40:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.313+0000 7fdfa659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfa8106ab0 0x7fdfa8106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.313+0000 7fdfa659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfa8106ab0 0x7fdfa8106ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58368/0 (socket says 172.21.15.16:58368) 2024-04-22T15:40:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.313+0000 7fdfa659c700 1 -- 172.21.15.16:0/1960689053 learned_addr learned my addr 172.21.15.16:0/1960689053 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.314+0000 7fdfa659c700 1 -- 172.21.15.16:0/1960689053 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa81075e0 con 0x7fdfa8106ab0 2024-04-22T15:40:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.314+0000 7fdfa659c700 1 --2- 172.21.15.16:0/1960689053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfa8106ab0 0x7fdfa8106ed0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdf980098a0 tx=0x7fdf98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0b596e7a0b96b94 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.314+0000 7fdfa559a700 1 -- 172.21.15.16:0/1960689053 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf98004180 con 0x7fdfa8106ab0 2024-04-22T15:40:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.314+0000 7fdfa559a700 1 -- 172.21.15.16:0/1960689053 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdf980042e0 con 0x7fdfa8106ab0 2024-04-22T15:40:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.314+0000 7fdfa559a700 1 -- 172.21.15.16:0/1960689053 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf98011af0 con 0x7fdfa8106ab0 2024-04-22T15:40:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.315+0000 7fdfacd70700 1 -- 172.21.15.16:0/1960689053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfa8106ab0 msgr2=0x7fdfa8106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.315+0000 7fdfacd70700 1 --2- 172.21.15.16:0/1960689053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfa8106ab0 0x7fdfa8106ed0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdf980098a0 tx=0x7fdf98009bb0 comp rx=0 tx=0).stop 2024-04-22T15:40:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.315+0000 7fdfacd70700 1 -- 172.21.15.16:0/1960689053 shutdown_connections 2024-04-22T15:40:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.315+0000 7fdfacd70700 1 --2- 172.21.15.16:0/1960689053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfa8106ab0 0x7fdfa8106ed0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.315+0000 7fdfacd70700 1 -- 172.21.15.16:0/1960689053 >> 172.21.15.16:0/1960689053 conn(0x7fdfa8100110 msgr2=0x7fdfa8102590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.315+0000 7fdfacd70700 1 -- 172.21.15.16:0/1960689053 shutdown_connections 2024-04-22T15:40:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.315+0000 7fdfacd70700 1 -- 172.21.15.16:0/1960689053 wait complete. 2024-04-22T15:40:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.316+0000 7fdfacd70700 1 Processor -- start 2024-04-22T15:40:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.316+0000 7fdfacd70700 1 -- start start 2024-04-22T15:40:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.316+0000 7fdfacd70700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfa8106ab0 0x7fdfa819c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.316+0000 7fdfacd70700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa819c970 con 0x7fdfa8106ab0 2024-04-22T15:40:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.316+0000 7fdfa659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfa8106ab0 0x7fdfa819c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.316+0000 7fdfa659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfa8106ab0 0x7fdfa819c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58374/0 (socket says 172.21.15.16:58374) 2024-04-22T15:40:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.316+0000 7fdfa659c700 1 -- 172.21.15.16:0/2849930550 learned_addr learned my addr 172.21.15.16:0/2849930550 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.316+0000 7fdfa659c700 1 -- 172.21.15.16:0/2849930550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf98009550 con 0x7fdfa8106ab0 2024-04-22T15:40:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.316+0000 7fdfa659c700 1 --2- 172.21.15.16:0/2849930550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfa8106ab0 0x7fdfa819c430 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdf98011670 tx=0x7fdf98011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.317+0000 7fdf977fe700 1 -- 172.21.15.16:0/2849930550 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf98003a40 con 0x7fdfa8106ab0 2024-04-22T15:40:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.317+0000 7fdfacd70700 1 -- 172.21.15.16:0/2849930550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa819cb70 con 0x7fdfa8106ab0 2024-04-22T15:40:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.317+0000 7fdf977fe700 1 -- 172.21.15.16:0/2849930550 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdf98003ba0 con 0x7fdfa8106ab0 2024-04-22T15:40:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.317+0000 7fdf977fe700 1 -- 172.21.15.16:0/2849930550 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf98024790 con 0x7fdfa8106ab0 2024-04-22T15:40:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.317+0000 7fdfacd70700 1 -- 172.21.15.16:0/2849930550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa819d010 con 0x7fdfa8106ab0 2024-04-22T15:40:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.317+0000 7fdf977fe700 1 -- 172.21.15.16:0/2849930550 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fdf9802d3f0 con 0x7fdfa8106ab0 2024-04-22T15:40:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.318+0000 7fdf977fe700 1 --2- 172.21.15.16:0/2849930550 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fdf90038320 0x7fdf9003a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.318+0000 7fdf977fe700 1 -- 172.21.15.16:0/2849930550 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fdf9804c570 con 0x7fdfa8106ab0 2024-04-22T15:40:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.318+0000 7fdfa5d9b700 1 --2- 172.21.15.16:0/2849930550 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fdf90038320 0x7fdf9003a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.318+0000 7fdfacd70700 1 -- 172.21.15.16:0/2849930550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa8196110 con 0x7fdfa8106ab0 2024-04-22T15:40:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.318+0000 7fdfa5d9b700 1 --2- 172.21.15.16:0/2849930550 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fdf90038320 0x7fdf9003a7e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c006fd0 tx=0x7fdf9c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.323+0000 7fdf977fe700 1 -- 172.21.15.16:0/2849930550 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fdf98010c00 con 0x7fdfa8106ab0 2024-04-22T15:40:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:19.578+0000 7fdfacd70700 1 -- 172.21.15.16:0/2849930550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fdfa80623c0 con 0x7fdfa8106ab0 2024-04-22T15:40:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:20.158+0000 7fdf977fe700 1 -- 172.21.15.16:0/2849930550 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7fdf98019500 con 0x7fdfa8106ab0 2024-04-22T15:40:21.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:20.159+0000 7fdf977fe700 1 -- 172.21.15.16:0/2849930550 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fdf98021d80 con 0x7fdfa8106ab0 2024-04-22T15:40:21.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:20.162+0000 7fdfacd70700 1 -- 172.21.15.16:0/2849930550 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fdf90038320 msgr2=0x7fdf9003a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:21.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:20.162+0000 7fdfacd70700 1 --2- 172.21.15.16:0/2849930550 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fdf90038320 0x7fdf9003a7e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c006fd0 tx=0x7fdf9c006e40 comp rx=0 tx=0).stop 2024-04-22T15:40:21.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:20.162+0000 7fdfacd70700 1 -- 172.21.15.16:0/2849930550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfa8106ab0 msgr2=0x7fdfa819c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:21.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:20.162+0000 7fdfacd70700 1 --2- 172.21.15.16:0/2849930550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfa8106ab0 0x7fdfa819c430 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdf98011670 tx=0x7fdf98011b50 comp rx=0 tx=0).stop 2024-04-22T15:40:21.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:20.162+0000 7fdfacd70700 1 -- 172.21.15.16:0/2849930550 shutdown_connections 2024-04-22T15:40:21.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:20.162+0000 7fdfacd70700 1 --2- 172.21.15.16:0/2849930550 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7fdf90038320 0x7fdf9003a7e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:21.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:20.162+0000 7fdfacd70700 1 --2- 172.21.15.16:0/2849930550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfa8106ab0 0x7fdfa819c430 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:21.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:20.162+0000 7fdfacd70700 1 -- 172.21.15.16:0/2849930550 >> 172.21.15.16:0/2849930550 conn(0x7fdfa8100110 msgr2=0x7fdfa8193250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:21.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:20.162+0000 7fdfacd70700 1 -- 172.21.15.16:0/2849930550 shutdown_connections 2024-04-22T15:40:21.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:20.162+0000 7fdfacd70700 1 -- 172.21.15.16:0/2849930550 wait complete. 2024-04-22T15:40:21.261 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:21 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/2849930550' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-22T15:40:21.261 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:21 smithi016 ceph-mon[26620]: mgrmap e9: smithi016.rnsnnr(active, since 26s) 2024-04-22T15:40:22.315 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:22 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/2577912891' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T15:40:22.324 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-22T15:40:22.324 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-22T15:40:22.324 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T15:40:22.324 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "active_name": "smithi016.rnsnnr", 2024-04-22T15:40:22.324 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-22T15:40:22.324 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-22T15:40:22.324 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.485+0000 7f11e3356700 1 Processor -- start 2024-04-22T15:40:22.324 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.485+0000 7f11e3356700 1 -- start start 2024-04-22T15:40:22.324 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.486+0000 7f11e3356700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc105470 0x7f11dc105890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:22.324 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.486+0000 7f11e3356700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc105e60 con 0x7f11dc105470 2024-04-22T15:40:22.324 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.490+0000 7f11e10f2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc105470 0x7f11dc105890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:22.325 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.490+0000 7f11e10f2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc105470 0x7f11dc105890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58402/0 (socket says 172.21.15.16:58402) 2024-04-22T15:40:22.325 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.491+0000 7f11e10f2700 1 -- 172.21.15.16:0/3716685688 learned_addr learned my addr 172.21.15.16:0/3716685688 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:22.325 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.491+0000 7f11e10f2700 1 -- 172.21.15.16:0/3716685688 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11dc106670 con 0x7f11dc105470 2024-04-22T15:40:22.325 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.491+0000 7f11e10f2700 1 --2- 172.21.15.16:0/3716685688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc105470 0x7f11dc105890 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f11d80098a0 tx=0x7f11d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f6879902a698e25 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:22.325 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.491+0000 7f11d3fff700 1 -- 172.21.15.16:0/3716685688 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11d8004180 con 0x7f11dc105470 2024-04-22T15:40:22.325 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.491+0000 7f11d3fff700 1 -- 172.21.15.16:0/3716685688 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f11d80042e0 con 0x7f11dc105470 2024-04-22T15:40:22.325 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.491+0000 7f11d3fff700 1 -- 172.21.15.16:0/3716685688 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11d8004180 con 0x7f11dc105470 2024-04-22T15:40:22.325 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.492+0000 7f11e3356700 1 -- 172.21.15.16:0/3716685688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc105470 msgr2=0x7f11dc105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:22.325 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.492+0000 7f11e3356700 1 --2- 172.21.15.16:0/3716685688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc105470 0x7f11dc105890 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f11d80098a0 tx=0x7f11d8009bb0 comp rx=0 tx=0).stop 2024-04-22T15:40:22.325 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.492+0000 7f11e3356700 1 -- 172.21.15.16:0/3716685688 shutdown_connections 2024-04-22T15:40:22.326 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.492+0000 7f11e3356700 1 --2- 172.21.15.16:0/3716685688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc105470 0x7f11dc105890 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:22.326 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.492+0000 7f11e3356700 1 -- 172.21.15.16:0/3716685688 >> 172.21.15.16:0/3716685688 conn(0x7f11dc1009f0 msgr2=0x7f11dc102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:22.326 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.492+0000 7f11e3356700 1 -- 172.21.15.16:0/3716685688 shutdown_connections 2024-04-22T15:40:22.326 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.492+0000 7f11e3356700 1 -- 172.21.15.16:0/3716685688 wait complete. 2024-04-22T15:40:22.326 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.493+0000 7f11e3356700 1 Processor -- start 2024-04-22T15:40:22.326 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.493+0000 7f11e3356700 1 -- start start 2024-04-22T15:40:22.326 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.493+0000 7f11e3356700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc105470 0x7f11dc19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:22.326 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.493+0000 7f11e3356700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc19a6f0 con 0x7f11dc105470 2024-04-22T15:40:22.326 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.493+0000 7f11e10f2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc105470 0x7f11dc19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:22.326 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.493+0000 7f11e10f2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc105470 0x7f11dc19a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58418/0 (socket says 172.21.15.16:58418) 2024-04-22T15:40:22.326 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.493+0000 7f11e10f2700 1 -- 172.21.15.16:0/2577912891 learned_addr learned my addr 172.21.15.16:0/2577912891 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:22.326 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.494+0000 7f11e10f2700 1 -- 172.21.15.16:0/2577912891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d8009550 con 0x7f11dc105470 2024-04-22T15:40:22.327 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.494+0000 7f11e10f2700 1 --2- 172.21.15.16:0/2577912891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc105470 0x7f11dc19a1b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f11d8011cd0 tx=0x7f11d8011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:22.327 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.494+0000 7f11d27fc700 1 -- 172.21.15.16:0/2577912891 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11d8004180 con 0x7f11dc105470 2024-04-22T15:40:22.327 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.494+0000 7f11d27fc700 1 -- 172.21.15.16:0/2577912891 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f11d80039f0 con 0x7f11dc105470 2024-04-22T15:40:22.327 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.494+0000 7f11e3356700 1 -- 172.21.15.16:0/2577912891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11dc19a8f0 con 0x7f11dc105470 2024-04-22T15:40:22.327 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.494+0000 7f11d27fc700 1 -- 172.21.15.16:0/2577912891 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11d801a6f0 con 0x7f11dc105470 2024-04-22T15:40:22.327 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.494+0000 7f11e3356700 1 -- 172.21.15.16:0/2577912891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11dc19ad90 con 0x7f11dc105470 2024-04-22T15:40:22.327 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.495+0000 7f11e3356700 1 -- 172.21.15.16:0/2577912891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11dc193f70 con 0x7f11dc105470 2024-04-22T15:40:22.327 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.495+0000 7f11d27fc700 1 -- 172.21.15.16:0/2577912891 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f11d8023af0 con 0x7f11dc105470 2024-04-22T15:40:22.327 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.495+0000 7f11d27fc700 1 --2- 172.21.15.16:0/2577912891 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f11c8038340 0x7f11c803a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:22.327 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.495+0000 7f11d27fc700 1 -- 172.21.15.16:0/2577912891 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f11d804c2c0 con 0x7f11dc105470 2024-04-22T15:40:22.327 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.495+0000 7f11e08f1700 1 -- 172.21.15.16:0/2577912891 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f11c8038340 msgr2=0x7f11c803a800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/3431522404 2024-04-22T15:40:22.327 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.496+0000 7f11e08f1700 1 --2- 172.21.15.16:0/2577912891 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f11c8038340 0x7f11c803a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:40:22.327 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.500+0000 7f11d27fc700 1 -- 172.21.15.16:0/2577912891 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f11d80225a0 con 0x7f11dc105470 2024-04-22T15:40:22.328 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.697+0000 7f11e08f1700 1 -- 172.21.15.16:0/2577912891 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f11c8038340 msgr2=0x7f11c803a800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/3431522404 2024-04-22T15:40:22.328 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.697+0000 7f11e08f1700 1 --2- 172.21.15.16:0/2577912891 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f11c8038340 0x7f11c803a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T15:40:22.328 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.760+0000 7f11e3356700 1 -- 172.21.15.16:0/2577912891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f11dc061a40 con 0x7f11dc105470 2024-04-22T15:40:22.328 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.760+0000 7f11d27fc700 1 -- 172.21.15.16:0/2577912891 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f11d801e080 con 0x7f11dc105470 2024-04-22T15:40:22.328 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.762+0000 7f11e3356700 1 -- 172.21.15.16:0/2577912891 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f11c8038340 msgr2=0x7f11c803a800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T15:40:22.328 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.762+0000 7f11e3356700 1 --2- 172.21.15.16:0/2577912891 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f11c8038340 0x7f11c803a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:22.328 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.762+0000 7f11e3356700 1 -- 172.21.15.16:0/2577912891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc105470 msgr2=0x7f11dc19a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:22.328 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.762+0000 7f11e3356700 1 --2- 172.21.15.16:0/2577912891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc105470 0x7f11dc19a1b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f11d8011cd0 tx=0x7f11d8011db0 comp rx=0 tx=0).stop 2024-04-22T15:40:22.328 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.762+0000 7f11e3356700 1 -- 172.21.15.16:0/2577912891 shutdown_connections 2024-04-22T15:40:22.328 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.762+0000 7f11e3356700 1 --2- 172.21.15.16:0/2577912891 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7f11c8038340 0x7f11c803a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:22.328 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.762+0000 7f11e3356700 1 --2- 172.21.15.16:0/2577912891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc105470 0x7f11dc19a1b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:22.328 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.762+0000 7f11e3356700 1 -- 172.21.15.16:0/2577912891 >> 172.21.15.16:0/2577912891 conn(0x7f11dc1009f0 msgr2=0x7f11dc1016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:22.328 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.762+0000 7f11e3356700 1 -- 172.21.15.16:0/2577912891 shutdown_connections 2024-04-22T15:40:22.328 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:21.762+0000 7f11e3356700 1 -- 172.21.15.16:0/2577912891 wait complete. 2024-04-22T15:40:22.329 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for the mgr to restart... 2024-04-22T15:40:22.329 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr epoch 9... 2024-04-22T15:40:27.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:27 smithi016 ceph-mon[26620]: Active manager daemon smithi016.rnsnnr restarted 2024-04-22T15:40:27.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:27 smithi016 ceph-mon[26620]: Activating manager daemon smithi016.rnsnnr 2024-04-22T15:40:27.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:27 smithi016 ceph-mon[26620]: osdmap e3: 0 total, 0 up, 0 in 2024-04-22T15:40:27.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:27 smithi016 ceph-mon[26620]: mgrmap e10: smithi016.rnsnnr(active, starting, since 0.00315084s) 2024-04-22T15:40:27.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:27 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-04-22T15:40:27.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:27 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr metadata", "who": "smithi016.rnsnnr", "id": "smithi016.rnsnnr"}]: dispatch 2024-04-22T15:40:27.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:27 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T15:40:27.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:27 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T15:40:27.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:27 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T15:40:27.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:27 smithi016 ceph-mon[26620]: Manager daemon smithi016.rnsnnr is now available 2024-04-22T15:40:27.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:27 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:40:27.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:27 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:40:27.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:27 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi016.rnsnnr/mirror_snapshot_schedule"}]: dispatch 2024-04-22T15:40:28.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:28 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi016.rnsnnr/trash_purge_schedule"}]: dispatch 2024-04-22T15:40:28.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:28 smithi016 ceph-mon[26620]: mgrmap e11: smithi016.rnsnnr(active, since 1.00579s) 2024-04-22T15:40:28.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-22T15:40:28.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-22T15:40:28.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-22T15:40:28.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-22T15:40:28.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.718+0000 7eff81ade700 1 Processor -- start 2024-04-22T15:40:28.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.718+0000 7eff81ade700 1 -- start start 2024-04-22T15:40:28.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.718+0000 7eff81ade700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff7c1070c0 0x7eff7c1094b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:28.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.718+0000 7eff81ade700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7c0745a0 con 0x7eff7c1070c0 2024-04-22T15:40:28.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.718+0000 7eff7b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff7c1070c0 0x7eff7c1094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:28.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.718+0000 7eff7b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff7c1070c0 0x7eff7c1094b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38014/0 (socket says 172.21.15.16:38014) 2024-04-22T15:40:28.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.718+0000 7eff7b7fe700 1 -- 172.21.15.16:0/2121981757 learned_addr learned my addr 172.21.15.16:0/2121981757 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:28.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.719+0000 7eff7b7fe700 1 -- 172.21.15.16:0/2121981757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7c0746e0 con 0x7eff7c1070c0 2024-04-22T15:40:28.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.719+0000 7eff7b7fe700 1 --2- 172.21.15.16:0/2121981757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff7c1070c0 0x7eff7c1094b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7eff6c0098a0 tx=0x7eff6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191e5d7c4c9e16e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:28.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.719+0000 7eff7a7fc700 1 -- 172.21.15.16:0/2121981757 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff6c004180 con 0x7eff7c1070c0 2024-04-22T15:40:28.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.719+0000 7eff7a7fc700 1 -- 172.21.15.16:0/2121981757 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7eff6c0042e0 con 0x7eff7c1070c0 2024-04-22T15:40:28.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.719+0000 7eff7a7fc700 1 -- 172.21.15.16:0/2121981757 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff6c011af0 con 0x7eff7c1070c0 2024-04-22T15:40:28.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.720+0000 7eff81ade700 1 -- 172.21.15.16:0/2121981757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff7c1070c0 msgr2=0x7eff7c1094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:28.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.720+0000 7eff81ade700 1 --2- 172.21.15.16:0/2121981757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff7c1070c0 0x7eff7c1094b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7eff6c0098a0 tx=0x7eff6c009bb0 comp rx=0 tx=0).stop 2024-04-22T15:40:28.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.720+0000 7eff81ade700 1 -- 172.21.15.16:0/2121981757 shutdown_connections 2024-04-22T15:40:28.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.720+0000 7eff81ade700 1 --2- 172.21.15.16:0/2121981757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff7c1070c0 0x7eff7c1094b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:28.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.720+0000 7eff81ade700 1 -- 172.21.15.16:0/2121981757 >> 172.21.15.16:0/2121981757 conn(0x7eff7c1009f0 msgr2=0x7eff7c102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:28.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.720+0000 7eff81ade700 1 -- 172.21.15.16:0/2121981757 shutdown_connections 2024-04-22T15:40:28.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.720+0000 7eff81ade700 1 -- 172.21.15.16:0/2121981757 wait complete. 2024-04-22T15:40:28.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.720+0000 7eff81ade700 1 Processor -- start 2024-04-22T15:40:28.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.721+0000 7eff81ade700 1 -- start start 2024-04-22T15:40:28.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.721+0000 7eff81ade700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff7c1070c0 0x7eff7c106a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:28.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.721+0000 7eff81ade700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7c106f60 con 0x7eff7c1070c0 2024-04-22T15:40:28.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.721+0000 7eff7b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff7c1070c0 0x7eff7c106a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:28.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.721+0000 7eff7b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff7c1070c0 0x7eff7c106a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38026/0 (socket says 172.21.15.16:38026) 2024-04-22T15:40:28.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.721+0000 7eff7b7fe700 1 -- 172.21.15.16:0/1407614042 learned_addr learned my addr 172.21.15.16:0/1407614042 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:28.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.721+0000 7eff7b7fe700 1 -- 172.21.15.16:0/1407614042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff6c009550 con 0x7eff7c1070c0 2024-04-22T15:40:28.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.721+0000 7eff7b7fe700 1 --2- 172.21.15.16:0/1407614042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff7c1070c0 0x7eff7c106a20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7eff6c011ad0 tx=0x7eff6c011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:28.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.721+0000 7eff78ff9700 1 -- 172.21.15.16:0/1407614042 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff6c003ab0 con 0x7eff7c1070c0 2024-04-22T15:40:28.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.721+0000 7eff81ade700 1 -- 172.21.15.16:0/1407614042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff7c105070 con 0x7eff7c1070c0 2024-04-22T15:40:28.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.722+0000 7eff78ff9700 1 -- 172.21.15.16:0/1407614042 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7eff6c003c10 con 0x7eff7c1070c0 2024-04-22T15:40:28.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.722+0000 7eff78ff9700 1 -- 172.21.15.16:0/1407614042 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff6c0198e0 con 0x7eff7c1070c0 2024-04-22T15:40:28.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.722+0000 7eff81ade700 1 -- 172.21.15.16:0/1407614042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff7c105510 con 0x7eff7c1070c0 2024-04-22T15:40:28.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.722+0000 7eff78ff9700 1 -- 172.21.15.16:0/1407614042 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7eff6c019b70 con 0x7eff7c1070c0 2024-04-22T15:40:28.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.723+0000 7eff78ff9700 1 --2- 172.21.15.16:0/1407614042 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7eff640383c0 0x7eff6403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:28.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.723+0000 7eff81ade700 1 -- 172.21.15.16:0/1407614042 --> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7eff7c10a710 con 0x7eff640383c0 2024-04-22T15:40:28.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.723+0000 7eff78ff9700 1 -- 172.21.15.16:0/1407614042 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7eff6c04c200 con 0x7eff7c1070c0 2024-04-22T15:40:28.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.723+0000 7eff7affd700 1 -- 172.21.15.16:0/1407614042 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7eff640383c0 msgr2=0x7eff6403a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/3431522404 2024-04-22T15:40:28.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.723+0000 7eff7affd700 1 --2- 172.21.15.16:0/1407614042 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7eff640383c0 0x7eff6403a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:40:28.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.924+0000 7eff7affd700 1 -- 172.21.15.16:0/1407614042 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7eff640383c0 msgr2=0x7eff6403a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/3431522404 2024-04-22T15:40:28.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:22.924+0000 7eff7affd700 1 --2- 172.21.15.16:0/1407614042 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7eff640383c0 0x7eff6403a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T15:40:28.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:23.325+0000 7eff7affd700 1 -- 172.21.15.16:0/1407614042 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7eff640383c0 msgr2=0x7eff6403a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/3431522404 2024-04-22T15:40:28.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:23.325+0000 7eff7affd700 1 --2- 172.21.15.16:0/1407614042 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7eff640383c0 0x7eff6403a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T15:40:28.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:24.126+0000 7eff7affd700 1 -- 172.21.15.16:0/1407614042 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7eff640383c0 msgr2=0x7eff6403a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/3431522404 2024-04-22T15:40:28.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:24.126+0000 7eff7affd700 1 --2- 172.21.15.16:0/1407614042 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7eff640383c0 0x7eff6403a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T15:40:28.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:25.727+0000 7eff7affd700 1 -- 172.21.15.16:0/1407614042 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7eff640383c0 msgr2=0x7eff6403a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/3431522404 2024-04-22T15:40:28.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:25.727+0000 7eff7affd700 1 --2- 172.21.15.16:0/1407614042 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7eff640383c0 0x7eff6403a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-22T15:40:28.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:27.108+0000 7eff78ff9700 1 -- 172.21.15.16:0/1407614042 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mgrmap(e 10) v1 ==== 45063+0+0 (secure 0 0 0) 0x7eff6c02ad00 con 0x7eff7c1070c0 2024-04-22T15:40:28.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:27.108+0000 7eff78ff9700 1 -- 172.21.15.16:0/1407614042 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7eff640383c0 msgr2=0x7eff6403a880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T15:40:28.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:27.108+0000 7eff78ff9700 1 --2- 172.21.15.16:0/1407614042 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7eff640383c0 0x7eff6403a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:28.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:28.109+0000 7eff78ff9700 1 -- 172.21.15.16:0/1407614042 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 11) v1 ==== 45190+0+0 (secure 0 0 0) 0x7eff6c00ecf0 con 0x7eff7c1070c0 2024-04-22T15:40:28.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:28.109+0000 7eff78ff9700 1 --2- 172.21.15.16:0/1407614042 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7eff6403bc80 0x7eff6403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:28.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:28.110+0000 7eff78ff9700 1 -- 172.21.15.16:0/1407614042 --> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7eff7c10a710 con 0x7eff6403bc80 2024-04-22T15:40:28.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:28.110+0000 7eff7affd700 1 --2- 172.21.15.16:0/1407614042 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7eff6403bc80 0x7eff6403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:28.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:28.111+0000 7eff7affd700 1 --2- 172.21.15.16:0/1407614042 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7eff6403bc80 0x7eff6403e070 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7eff70003a10 tx=0x7eff70009270 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:28.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:28.111+0000 7eff78ff9700 1 -- 172.21.15.16:0/1407614042 <== mgr.14162 v2:172.21.15.16:6800/3972982132 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7eff7c10a710 con 0x7eff6403bc80 2024-04-22T15:40:28.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:28.117+0000 7eff81ade700 1 -- 172.21.15.16:0/1407614042 --> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7eff7c1065c0 con 0x7eff6403bc80 2024-04-22T15:40:28.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:28.118+0000 7eff78ff9700 1 -- 172.21.15.16:0/1407614042 <== mgr.14162 v2:172.21.15.16:6800/3972982132 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7eff7c1065c0 con 0x7eff6403bc80 2024-04-22T15:40:28.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:28.118+0000 7eff81ade700 1 -- 172.21.15.16:0/1407614042 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7eff6403bc80 msgr2=0x7eff6403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:28.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:28.118+0000 7eff81ade700 1 --2- 172.21.15.16:0/1407614042 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7eff6403bc80 0x7eff6403e070 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7eff70003a10 tx=0x7eff70009270 comp rx=0 tx=0).stop 2024-04-22T15:40:28.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:28.118+0000 7eff81ade700 1 -- 172.21.15.16:0/1407614042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff7c1070c0 msgr2=0x7eff7c106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:28.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:28.118+0000 7eff81ade700 1 --2- 172.21.15.16:0/1407614042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff7c1070c0 0x7eff7c106a20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7eff6c011ad0 tx=0x7eff6c011bb0 comp rx=0 tx=0).stop 2024-04-22T15:40:28.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:28.118+0000 7eff81ade700 1 -- 172.21.15.16:0/1407614042 shutdown_connections 2024-04-22T15:40:28.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:28.118+0000 7eff81ade700 1 --2- 172.21.15.16:0/1407614042 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7eff6403bc80 0x7eff6403e070 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:28.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:28.118+0000 7eff81ade700 1 --2- 172.21.15.16:0/1407614042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff7c1070c0 0x7eff7c106a20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:28.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:28.118+0000 7eff81ade700 1 --2- 172.21.15.16:0/1407614042 >> [v2:172.21.15.16:6800/3431522404,v1:172.21.15.16:6801/3431522404] conn(0x7eff640383c0 0x7eff6403a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:28.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:28.118+0000 7eff81ade700 1 -- 172.21.15.16:0/1407614042 >> 172.21.15.16:0/1407614042 conn(0x7eff7c1009f0 msgr2=0x7eff7c101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:28.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:28.118+0000 7eff81ade700 1 -- 172.21.15.16:0/1407614042 shutdown_connections 2024-04-22T15:40:28.941 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:28.118+0000 7eff81ade700 1 -- 172.21.15.16:0/1407614042 wait complete. 2024-04-22T15:40:28.941 INFO:teuthology.orchestra.run.smithi016.stdout:mgr epoch 9 is available 2024-04-22T15:40:28.941 INFO:teuthology.orchestra.run.smithi016.stdout:Generating a dashboard self-signed certificate... 2024-04-22T15:40:29.336 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:29 smithi016 ceph-mon[26620]: [22/Apr/2024:15:40:28] ENGINE Bus STARTING 2024-04-22T15:40:29.336 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:29 smithi016 ceph-mon[26620]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-22T15:40:29.337 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:29 smithi016 ceph-mon[26620]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-22T15:40:29.337 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:29 smithi016 ceph-mon[26620]: [22/Apr/2024:15:40:28] ENGINE Serving on http://172.21.15.16:8765 2024-04-22T15:40:29.337 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:29 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:30.215 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-22T15:40:30.215 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.354+0000 7fd000d0a700 1 Processor -- start 2024-04-22T15:40:30.215 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.354+0000 7fd000d0a700 1 -- start start 2024-04-22T15:40:30.215 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.354+0000 7fd000d0a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcffc104860 0x7fcffc104c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:30.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.354+0000 7fd000d0a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc105250 con 0x7fcffc104860 2024-04-22T15:40:30.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.355+0000 7fcffa59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcffc104860 0x7fcffc104c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:30.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.355+0000 7fcffa59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcffc104860 0x7fcffc104c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38108/0 (socket says 172.21.15.16:38108) 2024-04-22T15:40:30.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.355+0000 7fcffa59c700 1 -- 172.21.15.16:0/3275386637 learned_addr learned my addr 172.21.15.16:0/3275386637 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:30.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.355+0000 7fcffa59c700 1 -- 172.21.15.16:0/3275386637 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcffc1077f0 con 0x7fcffc104860 2024-04-22T15:40:30.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.355+0000 7fcffa59c700 1 --2- 172.21.15.16:0/3275386637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcffc104860 0x7fcffc104c80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcfec0098a0 tx=0x7fcfec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5433416c9179ac5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:30.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.355+0000 7fcff959a700 1 -- 172.21.15.16:0/3275386637 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfec004180 con 0x7fcffc104860 2024-04-22T15:40:30.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.356+0000 7fcff959a700 1 -- 172.21.15.16:0/3275386637 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcfec0042e0 con 0x7fcffc104860 2024-04-22T15:40:30.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.356+0000 7fcff959a700 1 -- 172.21.15.16:0/3275386637 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfec011af0 con 0x7fcffc104860 2024-04-22T15:40:30.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.356+0000 7fd000d0a700 1 -- 172.21.15.16:0/3275386637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcffc104860 msgr2=0x7fcffc104c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:30.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.356+0000 7fd000d0a700 1 --2- 172.21.15.16:0/3275386637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcffc104860 0x7fcffc104c80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcfec0098a0 tx=0x7fcfec009bb0 comp rx=0 tx=0).stop 2024-04-22T15:40:30.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.356+0000 7fd000d0a700 1 -- 172.21.15.16:0/3275386637 shutdown_connections 2024-04-22T15:40:30.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.356+0000 7fd000d0a700 1 --2- 172.21.15.16:0/3275386637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcffc104860 0x7fcffc104c80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:30.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.356+0000 7fd000d0a700 1 -- 172.21.15.16:0/3275386637 >> 172.21.15.16:0/3275386637 conn(0x7fcffc100190 msgr2=0x7fcffc1025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:30.217 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.356+0000 7fd000d0a700 1 -- 172.21.15.16:0/3275386637 shutdown_connections 2024-04-22T15:40:30.217 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.357+0000 7fd000d0a700 1 -- 172.21.15.16:0/3275386637 wait complete. 2024-04-22T15:40:30.217 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.357+0000 7fd000d0a700 1 Processor -- start 2024-04-22T15:40:30.217 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.357+0000 7fd000d0a700 1 -- start start 2024-04-22T15:40:30.217 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.357+0000 7fd000d0a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcffc104860 0x7fcffc19a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:30.217 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.357+0000 7fd000d0a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc19a770 con 0x7fcffc104860 2024-04-22T15:40:30.217 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.358+0000 7fcffa59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcffc104860 0x7fcffc19a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:30.217 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.358+0000 7fcffa59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcffc104860 0x7fcffc19a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38124/0 (socket says 172.21.15.16:38124) 2024-04-22T15:40:30.217 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.358+0000 7fcffa59c700 1 -- 172.21.15.16:0/691178071 learned_addr learned my addr 172.21.15.16:0/691178071 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:30.217 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.358+0000 7fcffa59c700 1 -- 172.21.15.16:0/691178071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfec009550 con 0x7fcffc104860 2024-04-22T15:40:30.217 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.358+0000 7fcffa59c700 1 --2- 172.21.15.16:0/691178071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcffc104860 0x7fcffc19a230 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcfec006ba0 tx=0x7fcfec011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:30.217 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.358+0000 7fcff37fe700 1 -- 172.21.15.16:0/691178071 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfec003b00 con 0x7fcffc104860 2024-04-22T15:40:30.217 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.358+0000 7fd000d0a700 1 -- 172.21.15.16:0/691178071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcffc19a970 con 0x7fcffc104860 2024-04-22T15:40:30.217 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.358+0000 7fcff37fe700 1 -- 172.21.15.16:0/691178071 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcfec01a8e0 con 0x7fcffc104860 2024-04-22T15:40:30.218 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.358+0000 7fcff37fe700 1 -- 172.21.15.16:0/691178071 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfec019990 con 0x7fcffc104860 2024-04-22T15:40:30.218 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.358+0000 7fd000d0a700 1 -- 172.21.15.16:0/691178071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcffc19ae10 con 0x7fcffc104860 2024-04-22T15:40:30.218 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.360+0000 7fcff37fe700 1 -- 172.21.15.16:0/691178071 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 11) v1 ==== 45190+0+0 (secure 0 0 0) 0x7fcfec019c50 con 0x7fcffc104860 2024-04-22T15:40:30.218 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.360+0000 7fcff37fe700 1 --2- 172.21.15.16:0/691178071 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fcfe80381f0 0x7fcfe803a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:30.218 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.360+0000 7fd000d0a700 1 -- 172.21.15.16:0/691178071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcffc0623c0 con 0x7fcffc104860 2024-04-22T15:40:30.218 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.360+0000 7fcff37fe700 1 -- 172.21.15.16:0/691178071 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcfec04c750 con 0x7fcffc104860 2024-04-22T15:40:30.218 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.360+0000 7fcff9d9b700 1 --2- 172.21.15.16:0/691178071 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fcfe80381f0 0x7fcfe803a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:30.218 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.360+0000 7fcff9d9b700 1 --2- 172.21.15.16:0/691178071 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fcfe80381f0 0x7fcfe803a6b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4006fd0 tx=0x7fcfe4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:30.218 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.365+0000 7fcff37fe700 1 -- 172.21.15.16:0/691178071 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fcfec02b700 con 0x7fcffc104860 2024-04-22T15:40:30.218 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.561+0000 7fcff37fe700 1 -- 172.21.15.16:0/691178071 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 12) v1 ==== 45296+0+0 (secure 0 0 0) 0x7fcfec02c3a0 con 0x7fcffc104860 2024-04-22T15:40:30.218 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.575+0000 7fd000d0a700 1 -- 172.21.15.16:0/691178071 --> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fcffc101660 con 0x7fcfe80381f0 2024-04-22T15:40:30.218 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.639+0000 7fcff37fe700 1 -- 172.21.15.16:0/691178071 <== mgr.14162 v2:172.21.15.16:6800/3972982132 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fcffc101660 con 0x7fcfe80381f0 2024-04-22T15:40:30.219 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.641+0000 7fd000d0a700 1 -- 172.21.15.16:0/691178071 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fcfe80381f0 msgr2=0x7fcfe803a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:30.219 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.641+0000 7fd000d0a700 1 --2- 172.21.15.16:0/691178071 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fcfe80381f0 0x7fcfe803a6b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4006fd0 tx=0x7fcfe4006e40 comp rx=0 tx=0).stop 2024-04-22T15:40:30.219 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.641+0000 7fd000d0a700 1 -- 172.21.15.16:0/691178071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcffc104860 msgr2=0x7fcffc19a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:30.219 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.641+0000 7fd000d0a700 1 --2- 172.21.15.16:0/691178071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcffc104860 0x7fcffc19a230 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcfec006ba0 tx=0x7fcfec011b50 comp rx=0 tx=0).stop 2024-04-22T15:40:30.219 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.641+0000 7fd000d0a700 1 -- 172.21.15.16:0/691178071 shutdown_connections 2024-04-22T15:40:30.219 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.641+0000 7fd000d0a700 1 --2- 172.21.15.16:0/691178071 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fcfe80381f0 0x7fcfe803a6b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:30.219 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.641+0000 7fd000d0a700 1 --2- 172.21.15.16:0/691178071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcffc104860 0x7fcffc19a230 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:30.219 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.641+0000 7fd000d0a700 1 -- 172.21.15.16:0/691178071 >> 172.21.15.16:0/691178071 conn(0x7fcffc100190 msgr2=0x7fcffc100f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:30.219 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.641+0000 7fd000d0a700 1 -- 172.21.15.16:0/691178071 shutdown_connections 2024-04-22T15:40:30.219 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:29.642+0000 7fd000d0a700 1 -- 172.21.15.16:0/691178071 wait complete. 2024-04-22T15:40:30.219 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial admin user... 2024-04-22T15:40:30.333 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:30 smithi016 ceph-mon[26620]: [22/Apr/2024:15:40:28] ENGINE Serving on https://172.21.15.16:7150 2024-04-22T15:40:30.333 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:30 smithi016 ceph-mon[26620]: [22/Apr/2024:15:40:28] ENGINE Bus STARTED 2024-04-22T15:40:30.333 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:30 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:30.333 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:30 smithi016 ceph-mon[26620]: mgrmap e12: smithi016.rnsnnr(active, since 2s) 2024-04-22T15:40:30.333 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:30 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:30.333 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:30 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:31.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:31 smithi016 ceph-mon[26620]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:31.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:31 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:31.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$Rz2WOXeSXu2NXqVh8GqOCO7Gid5.gZX1kqpohYDrnlxv61IdF6AFC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713800431, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-22T15:40:31.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.625+0000 7f69922b3700 1 Processor -- start 2024-04-22T15:40:31.828 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.625+0000 7f69922b3700 1 -- start start 2024-04-22T15:40:31.828 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.625+0000 7f69922b3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f698c105470 0x7f698c105890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:31.828 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.625+0000 7f69922b3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698c105e60 con 0x7f698c105470 2024-04-22T15:40:31.828 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.626+0000 7f698bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f698c105470 0x7f698c105890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:31.828 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.626+0000 7f698bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f698c105470 0x7f698c105890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38138/0 (socket says 172.21.15.16:38138) 2024-04-22T15:40:31.828 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.626+0000 7f698bfff700 1 -- 172.21.15.16:0/1165558930 learned_addr learned my addr 172.21.15.16:0/1165558930 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:31.828 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.626+0000 7f698bfff700 1 -- 172.21.15.16:0/1165558930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f698c106670 con 0x7f698c105470 2024-04-22T15:40:31.828 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.626+0000 7f698bfff700 1 --2- 172.21.15.16:0/1165558930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f698c105470 0x7f698c105890 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f697c0098a0 tx=0x7f697c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1f7d95845e0e76b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:31.828 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.626+0000 7f698affd700 1 -- 172.21.15.16:0/1165558930 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f697c004180 con 0x7f698c105470 2024-04-22T15:40:31.828 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.626+0000 7f698affd700 1 -- 172.21.15.16:0/1165558930 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f697c0042e0 con 0x7f698c105470 2024-04-22T15:40:31.828 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.626+0000 7f698affd700 1 -- 172.21.15.16:0/1165558930 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f697c011af0 con 0x7f698c105470 2024-04-22T15:40:31.828 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.627+0000 7f69922b3700 1 -- 172.21.15.16:0/1165558930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f698c105470 msgr2=0x7f698c105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:31.829 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.627+0000 7f69922b3700 1 --2- 172.21.15.16:0/1165558930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f698c105470 0x7f698c105890 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f697c0098a0 tx=0x7f697c009bb0 comp rx=0 tx=0).stop 2024-04-22T15:40:31.829 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.627+0000 7f69922b3700 1 -- 172.21.15.16:0/1165558930 shutdown_connections 2024-04-22T15:40:31.829 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.627+0000 7f69922b3700 1 --2- 172.21.15.16:0/1165558930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f698c105470 0x7f698c105890 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:31.829 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.627+0000 7f69922b3700 1 -- 172.21.15.16:0/1165558930 >> 172.21.15.16:0/1165558930 conn(0x7f698c1009f0 msgr2=0x7f698c102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:31.829 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.627+0000 7f69922b3700 1 -- 172.21.15.16:0/1165558930 shutdown_connections 2024-04-22T15:40:31.829 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.627+0000 7f69922b3700 1 -- 172.21.15.16:0/1165558930 wait complete. 2024-04-22T15:40:31.829 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.628+0000 7f69922b3700 1 Processor -- start 2024-04-22T15:40:31.829 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.628+0000 7f69922b3700 1 -- start start 2024-04-22T15:40:31.829 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.628+0000 7f69922b3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f698c105470 0x7f698c190660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:31.829 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.628+0000 7f69922b3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698c190ba0 con 0x7f698c105470 2024-04-22T15:40:31.829 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.628+0000 7f698bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f698c105470 0x7f698c190660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:31.829 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.628+0000 7f698bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f698c105470 0x7f698c190660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38154/0 (socket says 172.21.15.16:38154) 2024-04-22T15:40:31.829 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.628+0000 7f698bfff700 1 -- 172.21.15.16:0/2968421102 learned_addr learned my addr 172.21.15.16:0/2968421102 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:31.829 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.628+0000 7f698bfff700 1 -- 172.21.15.16:0/2968421102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697c009550 con 0x7f698c105470 2024-04-22T15:40:31.829 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.629+0000 7f698bfff700 1 --2- 172.21.15.16:0/2968421102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f698c105470 0x7f698c190660 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f697c006ba0 tx=0x7f697c011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:31.830 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.629+0000 7f69897fa700 1 -- 172.21.15.16:0/2968421102 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f697c003b00 con 0x7f698c105470 2024-04-22T15:40:31.830 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.629+0000 7f69897fa700 1 -- 172.21.15.16:0/2968421102 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f697c01a8e0 con 0x7f698c105470 2024-04-22T15:40:31.830 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.629+0000 7f69922b3700 1 -- 172.21.15.16:0/2968421102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f698c18ecb0 con 0x7f698c105470 2024-04-22T15:40:31.830 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.629+0000 7f69897fa700 1 -- 172.21.15.16:0/2968421102 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f697c019990 con 0x7f698c105470 2024-04-22T15:40:31.830 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.629+0000 7f69922b3700 1 -- 172.21.15.16:0/2968421102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f698c18f150 con 0x7f698c105470 2024-04-22T15:40:31.830 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.630+0000 7f69897fa700 1 -- 172.21.15.16:0/2968421102 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f697c019c00 con 0x7f698c105470 2024-04-22T15:40:31.830 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.630+0000 7f69897fa700 1 --2- 172.21.15.16:0/2968421102 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f6974038370 0x7f697403a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:31.830 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.630+0000 7f69922b3700 1 -- 172.21.15.16:0/2968421102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f698c0623c0 con 0x7f698c105470 2024-04-22T15:40:31.830 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.630+0000 7f69897fa700 1 -- 172.21.15.16:0/2968421102 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f697c04c6a0 con 0x7f698c105470 2024-04-22T15:40:31.830 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.630+0000 7f698b7fe700 1 --2- 172.21.15.16:0/2968421102 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f6974038370 0x7f697403a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:31.830 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.630+0000 7f698b7fe700 1 --2- 172.21.15.16:0/2968421102 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f6974038370 0x7f697403a830 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6980006fd0 tx=0x7f6980006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:31.830 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.635+0000 7f69897fa700 1 -- 172.21.15.16:0/2968421102 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f697c02b700 con 0x7f698c105470 2024-04-22T15:40:31.830 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:30.850+0000 7f69922b3700 1 -- 172.21.15.16:0/2968421102 --> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f698c0621e0 con 0x7f6974038370 2024-04-22T15:40:31.831 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:31.096+0000 7f69897fa700 1 -- 172.21.15.16:0/2968421102 <== mgr.14162 v2:172.21.15.16:6800/3972982132 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f698c0621e0 con 0x7f6974038370 2024-04-22T15:40:31.831 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:31.098+0000 7f69922b3700 1 -- 172.21.15.16:0/2968421102 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f6974038370 msgr2=0x7f697403a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:31.831 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:31.098+0000 7f69922b3700 1 --2- 172.21.15.16:0/2968421102 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f6974038370 0x7f697403a830 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6980006fd0 tx=0x7f6980006e40 comp rx=0 tx=0).stop 2024-04-22T15:40:31.831 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:31.098+0000 7f69922b3700 1 -- 172.21.15.16:0/2968421102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f698c105470 msgr2=0x7f698c190660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:31.831 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:31.098+0000 7f69922b3700 1 --2- 172.21.15.16:0/2968421102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f698c105470 0x7f698c190660 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f697c006ba0 tx=0x7f697c011b50 comp rx=0 tx=0).stop 2024-04-22T15:40:31.831 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:31.098+0000 7f69922b3700 1 -- 172.21.15.16:0/2968421102 shutdown_connections 2024-04-22T15:40:31.831 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:31.098+0000 7f69922b3700 1 --2- 172.21.15.16:0/2968421102 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f6974038370 0x7f697403a830 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:31.831 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:31.098+0000 7f69922b3700 1 --2- 172.21.15.16:0/2968421102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f698c105470 0x7f698c190660 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:31.831 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:31.098+0000 7f69922b3700 1 -- 172.21.15.16:0/2968421102 >> 172.21.15.16:0/2968421102 conn(0x7f698c1009f0 msgr2=0x7f698c0740f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:31.831 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:31.098+0000 7f69922b3700 1 -- 172.21.15.16:0/2968421102 shutdown_connections 2024-04-22T15:40:31.831 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:31.098+0000 7f69922b3700 1 -- 172.21.15.16:0/2968421102 wait complete. 2024-04-22T15:40:31.831 INFO:teuthology.orchestra.run.smithi016.stdout:Fetching dashboard port number... 2024-04-22T15:40:32.237 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:32 smithi016 ceph-mon[26620]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:33.384 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:33 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/2944963034' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-22T15:40:33.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 8443 2024-04-22T15:40:33.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.252+0000 7fb4631a2700 1 Processor -- start 2024-04-22T15:40:33.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.252+0000 7fb4631a2700 1 -- start start 2024-04-22T15:40:33.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.253+0000 7fb4631a2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c104e50 0x7fb45c105270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:33.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.253+0000 7fb4631a2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45c079df0 con 0x7fb45c104e50 2024-04-22T15:40:33.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.253+0000 7fb460f3e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c104e50 0x7fb45c105270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:33.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.253+0000 7fb460f3e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c104e50 0x7fb45c105270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34204/0 (socket says 172.21.15.16:34204) 2024-04-22T15:40:33.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.253+0000 7fb460f3e700 1 -- 172.21.15.16:0/302055862 learned_addr learned my addr 172.21.15.16:0/302055862 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:33.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.253+0000 7fb460f3e700 1 -- 172.21.15.16:0/302055862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45c079f30 con 0x7fb45c104e50 2024-04-22T15:40:33.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.253+0000 7fb460f3e700 1 --2- 172.21.15.16:0/302055862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c104e50 0x7fb45c105270 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb4500098a0 tx=0x7fb450009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3b9ace2f726e829 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:33.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.253+0000 7fb45b7fe700 1 -- 172.21.15.16:0/302055862 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb450004180 con 0x7fb45c104e50 2024-04-22T15:40:33.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.254+0000 7fb45b7fe700 1 -- 172.21.15.16:0/302055862 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb4500042e0 con 0x7fb45c104e50 2024-04-22T15:40:33.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.254+0000 7fb45b7fe700 1 -- 172.21.15.16:0/302055862 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb450011af0 con 0x7fb45c104e50 2024-04-22T15:40:33.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.254+0000 7fb4631a2700 1 -- 172.21.15.16:0/302055862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c104e50 msgr2=0x7fb45c105270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:33.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.254+0000 7fb4631a2700 1 --2- 172.21.15.16:0/302055862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c104e50 0x7fb45c105270 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb4500098a0 tx=0x7fb450009bb0 comp rx=0 tx=0).stop 2024-04-22T15:40:33.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.254+0000 7fb4631a2700 1 -- 172.21.15.16:0/302055862 shutdown_connections 2024-04-22T15:40:33.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.254+0000 7fb4631a2700 1 --2- 172.21.15.16:0/302055862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c104e50 0x7fb45c105270 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:33.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.254+0000 7fb4631a2700 1 -- 172.21.15.16:0/302055862 >> 172.21.15.16:0/302055862 conn(0x7fb45c1009b0 msgr2=0x7fb45c102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:33.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.254+0000 7fb4631a2700 1 -- 172.21.15.16:0/302055862 shutdown_connections 2024-04-22T15:40:33.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.254+0000 7fb4631a2700 1 -- 172.21.15.16:0/302055862 wait complete. 2024-04-22T15:40:33.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.255+0000 7fb4631a2700 1 Processor -- start 2024-04-22T15:40:33.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.255+0000 7fb4631a2700 1 -- start start 2024-04-22T15:40:33.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.255+0000 7fb4631a2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c104e50 0x7fb45c19c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:33.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.255+0000 7fb4631a2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45c19c960 con 0x7fb45c104e50 2024-04-22T15:40:33.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.256+0000 7fb460f3e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c104e50 0x7fb45c19c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:33.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.256+0000 7fb460f3e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c104e50 0x7fb45c19c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34218/0 (socket says 172.21.15.16:34218) 2024-04-22T15:40:33.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.256+0000 7fb460f3e700 1 -- 172.21.15.16:0/2944963034 learned_addr learned my addr 172.21.15.16:0/2944963034 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:33.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.256+0000 7fb460f3e700 1 -- 172.21.15.16:0/2944963034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb450009550 con 0x7fb45c104e50 2024-04-22T15:40:33.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.256+0000 7fb460f3e700 1 --2- 172.21.15.16:0/2944963034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c104e50 0x7fb45c19c420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb450011670 tx=0x7fb450011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:33.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.256+0000 7fb459ffb700 1 -- 172.21.15.16:0/2944963034 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb450003a40 con 0x7fb45c104e50 2024-04-22T15:40:33.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.256+0000 7fb459ffb700 1 -- 172.21.15.16:0/2944963034 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb450003ba0 con 0x7fb45c104e50 2024-04-22T15:40:33.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.256+0000 7fb4631a2700 1 -- 172.21.15.16:0/2944963034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb45c19cb60 con 0x7fb45c104e50 2024-04-22T15:40:33.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.256+0000 7fb459ffb700 1 -- 172.21.15.16:0/2944963034 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb450024820 con 0x7fb45c104e50 2024-04-22T15:40:33.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.257+0000 7fb4631a2700 1 -- 172.21.15.16:0/2944963034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb45c19d000 con 0x7fb45c104e50 2024-04-22T15:40:33.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.257+0000 7fb4631a2700 1 -- 172.21.15.16:0/2944963034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb45c195c90 con 0x7fb45c104e50 2024-04-22T15:40:33.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.258+0000 7fb459ffb700 1 -- 172.21.15.16:0/2944963034 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 45296+0+0 (secure 0 0 0) 0x7fb450024a90 con 0x7fb45c104e50 2024-04-22T15:40:33.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.258+0000 7fb459ffb700 1 --2- 172.21.15.16:0/2944963034 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fb44c038380 0x7fb44c03a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:33.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.258+0000 7fb459ffb700 1 -- 172.21.15.16:0/2944963034 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb45004c520 con 0x7fb45c104e50 2024-04-22T15:40:33.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.258+0000 7fb45bfff700 1 --2- 172.21.15.16:0/2944963034 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fb44c038380 0x7fb44c03a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:33.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.258+0000 7fb45bfff700 1 --2- 172.21.15.16:0/2944963034 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fb44c038380 0x7fb44c03a840 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb448006fd0 tx=0x7fb448006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:33.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.263+0000 7fb459ffb700 1 -- 172.21.15.16:0/2944963034 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb450022310 con 0x7fb45c104e50 2024-04-22T15:40:33.396 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.466+0000 7fb4631a2700 1 -- 172.21.15.16:0/2944963034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fb45c02cc70 con 0x7fb45c104e50 2024-04-22T15:40:33.396 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.466+0000 7fb459ffb700 1 -- 172.21.15.16:0/2944963034 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fb450018b40 con 0x7fb45c104e50 2024-04-22T15:40:33.396 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.468+0000 7fb4631a2700 1 -- 172.21.15.16:0/2944963034 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fb44c038380 msgr2=0x7fb44c03a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:33.396 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.468+0000 7fb4631a2700 1 --2- 172.21.15.16:0/2944963034 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fb44c038380 0x7fb44c03a840 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb448006fd0 tx=0x7fb448006e40 comp rx=0 tx=0).stop 2024-04-22T15:40:33.396 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.468+0000 7fb4631a2700 1 -- 172.21.15.16:0/2944963034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c104e50 msgr2=0x7fb45c19c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:33.396 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.468+0000 7fb4631a2700 1 --2- 172.21.15.16:0/2944963034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c104e50 0x7fb45c19c420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb450011670 tx=0x7fb450011aa0 comp rx=0 tx=0).stop 2024-04-22T15:40:33.396 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.468+0000 7fb4631a2700 1 -- 172.21.15.16:0/2944963034 shutdown_connections 2024-04-22T15:40:33.396 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.468+0000 7fb4631a2700 1 --2- 172.21.15.16:0/2944963034 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fb44c038380 0x7fb44c03a840 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:33.396 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.468+0000 7fb4631a2700 1 --2- 172.21.15.16:0/2944963034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c104e50 0x7fb45c19c420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:33.396 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.468+0000 7fb4631a2700 1 -- 172.21.15.16:0/2944963034 >> 172.21.15.16:0/2944963034 conn(0x7fb45c1009b0 msgr2=0x7fb45c190a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:33.396 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.469+0000 7fb4631a2700 1 -- 172.21.15.16:0/2944963034 shutdown_connections 2024-04-22T15:40:33.396 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:32.469+0000 7fb4631a2700 1 -- 172.21.15.16:0/2944963034 wait complete. 2024-04-22T15:40:33.403 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T15:40:33.403 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-04-22T15:40:33.411 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T15:40:33.411 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-04-22T15:40:33.411 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-04-22T15:40:33.411 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-22T15:40:33.413 INFO:teuthology.orchestra.run.smithi016.stdout:Ceph Dashboard is now available at: 2024-04-22T15:40:33.413 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:40:33.413 INFO:teuthology.orchestra.run.smithi016.stdout: URL: https://smithi016:8443/ 2024-04-22T15:40:33.413 INFO:teuthology.orchestra.run.smithi016.stdout: User: admin 2024-04-22T15:40:33.413 INFO:teuthology.orchestra.run.smithi016.stdout: Password: 1qouddz77a 2024-04-22T15:40:33.413 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:40:33.413 INFO:teuthology.orchestra.run.smithi016.stdout:Saving cluster configuration to /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config directory 2024-04-22T15:40:33.414 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling autotune for osd_memory_target 2024-04-22T15:40:34.390 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:34 smithi016 ceph-mon[26620]: mgrmap e13: smithi016.rnsnnr(active, since 6s) 2024-04-22T15:40:34.649 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.832+0000 7fc98b4b5700 1 Processor -- start 2024-04-22T15:40:34.649 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.832+0000 7fc98b4b5700 1 -- start start 2024-04-22T15:40:34.649 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.832+0000 7fc98b4b5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc984106fd0 0x7fc9841093c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:34.649 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.832+0000 7fc98b4b5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc984079e90 con 0x7fc984106fd0 2024-04-22T15:40:34.649 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.832+0000 7fc989251700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc984106fd0 0x7fc9841093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:34.649 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.832+0000 7fc989251700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc984106fd0 0x7fc9841093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34234/0 (socket says 172.21.15.16:34234) 2024-04-22T15:40:34.649 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.832+0000 7fc989251700 1 -- 172.21.15.16:0/2690397353 learned_addr learned my addr 172.21.15.16:0/2690397353 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:34.649 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.832+0000 7fc989251700 1 -- 172.21.15.16:0/2690397353 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc984109900 con 0x7fc984106fd0 2024-04-22T15:40:34.649 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.833+0000 7fc989251700 1 --2- 172.21.15.16:0/2690397353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc984106fd0 0x7fc9841093c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc9800098a0 tx=0x7fc980009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcf701e885645c52 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:34.650 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.833+0000 7fc97bfff700 1 -- 172.21.15.16:0/2690397353 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc980004180 con 0x7fc984106fd0 2024-04-22T15:40:34.650 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.833+0000 7fc97bfff700 1 -- 172.21.15.16:0/2690397353 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc9800042e0 con 0x7fc984106fd0 2024-04-22T15:40:34.650 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.833+0000 7fc97bfff700 1 -- 172.21.15.16:0/2690397353 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc980011af0 con 0x7fc984106fd0 2024-04-22T15:40:34.650 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.833+0000 7fc98b4b5700 1 -- 172.21.15.16:0/2690397353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc984106fd0 msgr2=0x7fc9841093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:34.650 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.833+0000 7fc98b4b5700 1 --2- 172.21.15.16:0/2690397353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc984106fd0 0x7fc9841093c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc9800098a0 tx=0x7fc980009bb0 comp rx=0 tx=0).stop 2024-04-22T15:40:34.650 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.833+0000 7fc98b4b5700 1 -- 172.21.15.16:0/2690397353 shutdown_connections 2024-04-22T15:40:34.650 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.833+0000 7fc98b4b5700 1 --2- 172.21.15.16:0/2690397353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc984106fd0 0x7fc9841093c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:34.650 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.833+0000 7fc98b4b5700 1 -- 172.21.15.16:0/2690397353 >> 172.21.15.16:0/2690397353 conn(0x7fc984100a20 msgr2=0x7fc984102e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:34.650 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.834+0000 7fc98b4b5700 1 -- 172.21.15.16:0/2690397353 shutdown_connections 2024-04-22T15:40:34.650 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.834+0000 7fc98b4b5700 1 -- 172.21.15.16:0/2690397353 wait complete. 2024-04-22T15:40:34.650 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.834+0000 7fc98b4b5700 1 Processor -- start 2024-04-22T15:40:34.650 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.834+0000 7fc98b4b5700 1 -- start start 2024-04-22T15:40:34.650 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.834+0000 7fc98b4b5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc984106fd0 0x7fc98419a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:34.651 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.834+0000 7fc98b4b5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98419a760 con 0x7fc984106fd0 2024-04-22T15:40:34.651 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.835+0000 7fc989251700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc984106fd0 0x7fc98419a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:34.651 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.835+0000 7fc989251700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc984106fd0 0x7fc98419a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34246/0 (socket says 172.21.15.16:34246) 2024-04-22T15:40:34.651 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.835+0000 7fc989251700 1 -- 172.21.15.16:0/1504357249 learned_addr learned my addr 172.21.15.16:0/1504357249 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:34.651 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.835+0000 7fc989251700 1 -- 172.21.15.16:0/1504357249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc980009550 con 0x7fc984106fd0 2024-04-22T15:40:34.651 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.835+0000 7fc989251700 1 --2- 172.21.15.16:0/1504357249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc984106fd0 0x7fc98419a220 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc980011670 tx=0x7fc980011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:34.651 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.835+0000 7fc97a7fc700 1 -- 172.21.15.16:0/1504357249 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc980003a40 con 0x7fc984106fd0 2024-04-22T15:40:34.651 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.835+0000 7fc97a7fc700 1 -- 172.21.15.16:0/1504357249 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc980003ba0 con 0x7fc984106fd0 2024-04-22T15:40:34.651 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.835+0000 7fc98b4b5700 1 -- 172.21.15.16:0/1504357249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc98419a960 con 0x7fc984106fd0 2024-04-22T15:40:34.651 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.835+0000 7fc97a7fc700 1 -- 172.21.15.16:0/1504357249 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc980024790 con 0x7fc984106fd0 2024-04-22T15:40:34.651 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.835+0000 7fc98b4b5700 1 -- 172.21.15.16:0/1504357249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc98419ae00 con 0x7fc984106fd0 2024-04-22T15:40:34.651 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.836+0000 7fc97a7fc700 1 -- 172.21.15.16:0/1504357249 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7fc98001a430 con 0x7fc984106fd0 2024-04-22T15:40:34.652 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.836+0000 7fc98b4b5700 1 -- 172.21.15.16:0/1504357249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc98410a6f0 con 0x7fc984106fd0 2024-04-22T15:40:34.652 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.836+0000 7fc97a7fc700 1 --2- 172.21.15.16:0/1504357249 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fc970038460 0x7fc97003a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:34.652 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.836+0000 7fc97a7fc700 1 -- 172.21.15.16:0/1504357249 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc98004c670 con 0x7fc984106fd0 2024-04-22T15:40:34.652 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.837+0000 7fc988a50700 1 --2- 172.21.15.16:0/1504357249 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fc970038460 0x7fc97003a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:34.652 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.837+0000 7fc988a50700 1 --2- 172.21.15.16:0/1504357249 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fc970038460 0x7fc97003a920 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc974006fd0 tx=0x7fc974006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:34.652 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:33.841+0000 7fc97a7fc700 1 -- 172.21.15.16:0/1504357249 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc980010380 con 0x7fc984106fd0 2024-04-22T15:40:34.652 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:34.041+0000 7fc98b4b5700 1 -- 172.21.15.16:0/1504357249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fc98419dc40 con 0x7fc984106fd0 2024-04-22T15:40:34.652 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:34.042+0000 7fc97a7fc700 1 -- 172.21.15.16:0/1504357249 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fc980018b40 con 0x7fc984106fd0 2024-04-22T15:40:34.652 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:34.043+0000 7fc98b4b5700 1 -- 172.21.15.16:0/1504357249 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fc970038460 msgr2=0x7fc97003a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:34.652 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:34.043+0000 7fc98b4b5700 1 --2- 172.21.15.16:0/1504357249 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fc970038460 0x7fc97003a920 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc974006fd0 tx=0x7fc974006e40 comp rx=0 tx=0).stop 2024-04-22T15:40:34.652 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:34.043+0000 7fc98b4b5700 1 -- 172.21.15.16:0/1504357249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc984106fd0 msgr2=0x7fc98419a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:34.652 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:34.043+0000 7fc98b4b5700 1 --2- 172.21.15.16:0/1504357249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc984106fd0 0x7fc98419a220 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc980011670 tx=0x7fc980011b50 comp rx=0 tx=0).stop 2024-04-22T15:40:34.652 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:34.044+0000 7fc98b4b5700 1 -- 172.21.15.16:0/1504357249 shutdown_connections 2024-04-22T15:40:34.652 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:34.044+0000 7fc98b4b5700 1 --2- 172.21.15.16:0/1504357249 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fc970038460 0x7fc97003a920 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:34.653 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:34.044+0000 7fc98b4b5700 1 --2- 172.21.15.16:0/1504357249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc984106fd0 0x7fc98419a220 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:34.653 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:34.044+0000 7fc98b4b5700 1 -- 172.21.15.16:0/1504357249 >> 172.21.15.16:0/1504357249 conn(0x7fc984100a20 msgr2=0x7fc9841017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:34.653 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:34.044+0000 7fc98b4b5700 1 -- 172.21.15.16:0/1504357249 shutdown_connections 2024-04-22T15:40:34.653 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:34.044+0000 7fc98b4b5700 1 -- 172.21.15.16:0/1504357249 wait complete. 2024-04-22T15:40:35.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.050+0000 7fa9fcf2f700 1 Processor -- start 2024-04-22T15:40:35.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.050+0000 7fa9fcf2f700 1 -- start start 2024-04-22T15:40:35.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.050+0000 7fa9fcf2f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f8107090 0x7fa9f8109480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:35.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.050+0000 7fa9fcf2f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f8079dd0 con 0x7fa9f8107090 2024-04-22T15:40:35.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.050+0000 7fa9f659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f8107090 0x7fa9f8109480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:35.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.050+0000 7fa9f659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f8107090 0x7fa9f8109480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34256/0 (socket says 172.21.15.16:34256) 2024-04-22T15:40:35.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.050+0000 7fa9f659c700 1 -- 172.21.15.16:0/3016824090 learned_addr learned my addr 172.21.15.16:0/3016824090 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:35.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.050+0000 7fa9f659c700 1 -- 172.21.15.16:0/3016824090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f8079f10 con 0x7fa9f8107090 2024-04-22T15:40:35.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.051+0000 7fa9f659c700 1 --2- 172.21.15.16:0/3016824090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f8107090 0x7fa9f8109480 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa9e80098a0 tx=0x7fa9e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf25d7e33e6cf596 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:35.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.051+0000 7fa9f559a700 1 -- 172.21.15.16:0/3016824090 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9e8004180 con 0x7fa9f8107090 2024-04-22T15:40:35.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.051+0000 7fa9f559a700 1 -- 172.21.15.16:0/3016824090 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa9e80042e0 con 0x7fa9f8107090 2024-04-22T15:40:35.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.051+0000 7fa9f559a700 1 -- 172.21.15.16:0/3016824090 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9e8011af0 con 0x7fa9f8107090 2024-04-22T15:40:35.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.051+0000 7fa9fcf2f700 1 -- 172.21.15.16:0/3016824090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f8107090 msgr2=0x7fa9f8109480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:35.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.051+0000 7fa9fcf2f700 1 --2- 172.21.15.16:0/3016824090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f8107090 0x7fa9f8109480 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa9e80098a0 tx=0x7fa9e8009bb0 comp rx=0 tx=0).stop 2024-04-22T15:40:35.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.051+0000 7fa9fcf2f700 1 -- 172.21.15.16:0/3016824090 shutdown_connections 2024-04-22T15:40:35.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.051+0000 7fa9fcf2f700 1 --2- 172.21.15.16:0/3016824090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f8107090 0x7fa9f8109480 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:35.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.051+0000 7fa9fcf2f700 1 -- 172.21.15.16:0/3016824090 >> 172.21.15.16:0/3016824090 conn(0x7fa9f8100a20 msgr2=0x7fa9f8102e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:35.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.052+0000 7fa9fcf2f700 1 -- 172.21.15.16:0/3016824090 shutdown_connections 2024-04-22T15:40:35.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.052+0000 7fa9fcf2f700 1 -- 172.21.15.16:0/3016824090 wait complete. 2024-04-22T15:40:35.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.052+0000 7fa9fcf2f700 1 Processor -- start 2024-04-22T15:40:35.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.052+0000 7fa9fcf2f700 1 -- start start 2024-04-22T15:40:35.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.052+0000 7fa9fcf2f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f8107090 0x7fa9f819a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:35.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.052+0000 7fa9fcf2f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f819a750 con 0x7fa9f8107090 2024-04-22T15:40:35.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.053+0000 7fa9f659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f8107090 0x7fa9f819a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:35.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.053+0000 7fa9f659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f8107090 0x7fa9f819a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34272/0 (socket says 172.21.15.16:34272) 2024-04-22T15:40:35.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.053+0000 7fa9f659c700 1 -- 172.21.15.16:0/2188237035 learned_addr learned my addr 172.21.15.16:0/2188237035 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:35.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.053+0000 7fa9f659c700 1 -- 172.21.15.16:0/2188237035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e8009550 con 0x7fa9f8107090 2024-04-22T15:40:35.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.053+0000 7fa9f659c700 1 --2- 172.21.15.16:0/2188237035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f8107090 0x7fa9f819a210 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8011670 tx=0x7fa9e8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:35.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.053+0000 7fa9e77fe700 1 -- 172.21.15.16:0/2188237035 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9e8003a40 con 0x7fa9f8107090 2024-04-22T15:40:35.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.053+0000 7fa9e77fe700 1 -- 172.21.15.16:0/2188237035 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa9e8003ba0 con 0x7fa9f8107090 2024-04-22T15:40:35.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.053+0000 7fa9fcf2f700 1 -- 172.21.15.16:0/2188237035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f819a950 con 0x7fa9f8107090 2024-04-22T15:40:35.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.053+0000 7fa9e77fe700 1 -- 172.21.15.16:0/2188237035 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9e8024820 con 0x7fa9f8107090 2024-04-22T15:40:35.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.053+0000 7fa9fcf2f700 1 -- 172.21.15.16:0/2188237035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f819adf0 con 0x7fa9f8107090 2024-04-22T15:40:35.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.054+0000 7fa9e77fe700 1 -- 172.21.15.16:0/2188237035 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7fa9e8024ae0 con 0x7fa9f8107090 2024-04-22T15:40:35.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.054+0000 7fa9fcf2f700 1 -- 172.21.15.16:0/2188237035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9f810a6e0 con 0x7fa9f8107090 2024-04-22T15:40:35.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.054+0000 7fa9e77fe700 1 --2- 172.21.15.16:0/2188237035 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fa9e00383d0 0x7fa9e003a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:35.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.054+0000 7fa9e77fe700 1 -- 172.21.15.16:0/2188237035 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa9e804c5f0 con 0x7fa9f8107090 2024-04-22T15:40:35.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.055+0000 7fa9f5d9b700 1 --2- 172.21.15.16:0/2188237035 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fa9e00383d0 0x7fa9e003a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:35.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.055+0000 7fa9f5d9b700 1 --2- 172.21.15.16:0/2188237035 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fa9e00383d0 0x7fa9e003a890 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec006fd0 tx=0x7fa9ec006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:35.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.059+0000 7fa9e77fe700 1 -- 172.21.15.16:0/2188237035 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa9e80187c0 con 0x7fa9f8107090 2024-04-22T15:40:35.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.366+0000 7fa9fcf2f700 1 -- 172.21.15.16:0/2188237035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fa9f819dbd0 con 0x7fa9f8107090 2024-04-22T15:40:35.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.368+0000 7fa9e77fe700 1 -- 172.21.15.16:0/2188237035 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v34)=0 set mgr/dashboard/cluster/status v34) v1 ==== 153+0+0 (secure 0 0 0) 0x7fa9e800be60 con 0x7fa9f8107090 2024-04-22T15:40:35.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.370+0000 7fa9fcf2f700 1 -- 172.21.15.16:0/2188237035 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fa9e00383d0 msgr2=0x7fa9e003a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:35.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.370+0000 7fa9fcf2f700 1 --2- 172.21.15.16:0/2188237035 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fa9e00383d0 0x7fa9e003a890 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec006fd0 tx=0x7fa9ec006e40 comp rx=0 tx=0).stop 2024-04-22T15:40:35.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.370+0000 7fa9fcf2f700 1 -- 172.21.15.16:0/2188237035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f8107090 msgr2=0x7fa9f819a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:35.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.370+0000 7fa9fcf2f700 1 --2- 172.21.15.16:0/2188237035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f8107090 0x7fa9f819a210 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8011670 tx=0x7fa9e8011aa0 comp rx=0 tx=0).stop 2024-04-22T15:40:35.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.370+0000 7fa9fcf2f700 1 -- 172.21.15.16:0/2188237035 shutdown_connections 2024-04-22T15:40:35.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.370+0000 7fa9fcf2f700 1 --2- 172.21.15.16:0/2188237035 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fa9e00383d0 0x7fa9e003a890 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:35.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.370+0000 7fa9fcf2f700 1 --2- 172.21.15.16:0/2188237035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f8107090 0x7fa9f819a210 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:35.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.370+0000 7fa9fcf2f700 1 -- 172.21.15.16:0/2188237035 >> 172.21.15.16:0/2188237035 conn(0x7fa9f8100a20 msgr2=0x7fa9f81017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:35.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.370+0000 7fa9fcf2f700 1 -- 172.21.15.16:0/2188237035 shutdown_connections 2024-04-22T15:40:35.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-22T15:40:35.370+0000 7fa9fcf2f700 1 -- 172.21.15.16:0/2188237035 wait complete. 2024-04-22T15:40:35.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-22T15:40:35.959 INFO:teuthology.orchestra.run.smithi016.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-22T15:40:35.959 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:40:35.959 INFO:teuthology.orchestra.run.smithi016.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-22T15:40:35.960 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:40:35.960 INFO:teuthology.orchestra.run.smithi016.stdout:Or, if you are only running a single cluster on this host: 2024-04-22T15:40:35.960 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:40:35.960 INFO:teuthology.orchestra.run.smithi016.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-22T15:40:35.960 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:40:35.960 INFO:teuthology.orchestra.run.smithi016.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-22T15:40:35.961 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:40:35.961 INFO:teuthology.orchestra.run.smithi016.stdout: ceph telemetry on 2024-04-22T15:40:35.961 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:40:35.961 INFO:teuthology.orchestra.run.smithi016.stdout:For more information see: 2024-04-22T15:40:35.961 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:40:35.961 INFO:teuthology.orchestra.run.smithi016.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-22T15:40:35.961 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:40:35.961 INFO:teuthology.orchestra.run.smithi016.stdout:Bootstrap complete. 2024-04-22T15:40:35.986 INFO:tasks.cephadm:Fetching config... 2024-04-22T15:40:35.986 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-22T15:40:35.986 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-22T15:40:36.042 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-22T15:40:36.042 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-22T15:40:36.042 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-22T15:40:36.096 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-22T15:40:36.097 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-22T15:40:36.097 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/keyring of=/dev/stdout 2024-04-22T15:40:36.161 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-22T15:40:36.161 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-22T15:40:36.162 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-22T15:40:36.216 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-22T15:40:36.217 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDgwd3lqMl8Pw2fyQ3Yu3jAfGhJJ0nbJXvFNmm0PPe3caGzhNSMo72g9z5E7MV02q7sdBHmpo4dvW30aQKZGFOthk5tN+cA7U0+dkvz0hMLB+ld1JF48ldJYZf/5KF1NPVQVBWHwwOeJOK4Ojzivp/14d96JsM8fgDfA+v27vXTQ4otCnY8JAhuecxbdae8/jfVvBkl0G7V2izpWbu2pk/vt2YHeukTfD2e99yK0H39Y8FQCr4Zlz4aqO7QZ/pVeCiYXCwB0vnNVsGjAqeOWwIDPEB77+xPrLYSauuHf9oM4kfam4XP3+o1xwvagMx8fPd643BpCHSwdSSSfzCJL6auGHCIyDl5KlYDBHlmtMoWUTQnIc9H4zWL2VqkMAB38ggJKEl/UhucAFGvnrOAH190zTUSfv7Z0Y4P7RbjnLc+9ozeW49t6iK57LtxUXO3k3MIwIF9vUuJJc3/1r9FUC1oUfzAbPJjc17lpzQDzjZk157X2HSM4vVlMmuMBpJszjE= ceph-67ab3a80-00be-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T15:40:36.297 INFO:teuthology.orchestra.run.smithi016.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDgwd3lqMl8Pw2fyQ3Yu3jAfGhJJ0nbJXvFNmm0PPe3caGzhNSMo72g9z5E7MV02q7sdBHmpo4dvW30aQKZGFOthk5tN+cA7U0+dkvz0hMLB+ld1JF48ldJYZf/5KF1NPVQVBWHwwOeJOK4Ojzivp/14d96JsM8fgDfA+v27vXTQ4otCnY8JAhuecxbdae8/jfVvBkl0G7V2izpWbu2pk/vt2YHeukTfD2e99yK0H39Y8FQCr4Zlz4aqO7QZ/pVeCiYXCwB0vnNVsGjAqeOWwIDPEB77+xPrLYSauuHf9oM4kfam4XP3+o1xwvagMx8fPd643BpCHSwdSSSfzCJL6auGHCIyDl5KlYDBHlmtMoWUTQnIc9H4zWL2VqkMAB38ggJKEl/UhucAFGvnrOAH190zTUSfv7Z0Y4P7RbjnLc+9ozeW49t6iK57LtxUXO3k3MIwIF9vUuJJc3/1r9FUC1oUfzAbPJjc17lpzQDzjZk157X2HSM4vVlMmuMBpJszjE= ceph-67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T15:40:36.308 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDgwd3lqMl8Pw2fyQ3Yu3jAfGhJJ0nbJXvFNmm0PPe3caGzhNSMo72g9z5E7MV02q7sdBHmpo4dvW30aQKZGFOthk5tN+cA7U0+dkvz0hMLB+ld1JF48ldJYZf/5KF1NPVQVBWHwwOeJOK4Ojzivp/14d96JsM8fgDfA+v27vXTQ4otCnY8JAhuecxbdae8/jfVvBkl0G7V2izpWbu2pk/vt2YHeukTfD2e99yK0H39Y8FQCr4Zlz4aqO7QZ/pVeCiYXCwB0vnNVsGjAqeOWwIDPEB77+xPrLYSauuHf9oM4kfam4XP3+o1xwvagMx8fPd643BpCHSwdSSSfzCJL6auGHCIyDl5KlYDBHlmtMoWUTQnIc9H4zWL2VqkMAB38ggJKEl/UhucAFGvnrOAH190zTUSfv7Z0Y4P7RbjnLc+9ozeW49t6iK57LtxUXO3k3MIwIF9vUuJJc3/1r9FUC1oUfzAbPJjc17lpzQDzjZk157X2HSM4vVlMmuMBpJszjE= ceph-67ab3a80-00be-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T15:40:36.350 INFO:teuthology.orchestra.run.smithi027.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDgwd3lqMl8Pw2fyQ3Yu3jAfGhJJ0nbJXvFNmm0PPe3caGzhNSMo72g9z5E7MV02q7sdBHmpo4dvW30aQKZGFOthk5tN+cA7U0+dkvz0hMLB+ld1JF48ldJYZf/5KF1NPVQVBWHwwOeJOK4Ojzivp/14d96JsM8fgDfA+v27vXTQ4otCnY8JAhuecxbdae8/jfVvBkl0G7V2izpWbu2pk/vt2YHeukTfD2e99yK0H39Y8FQCr4Zlz4aqO7QZ/pVeCiYXCwB0vnNVsGjAqeOWwIDPEB77+xPrLYSauuHf9oM4kfam4XP3+o1xwvagMx8fPd643BpCHSwdSSSfzCJL6auGHCIyDl5KlYDBHlmtMoWUTQnIc9H4zWL2VqkMAB38ggJKEl/UhucAFGvnrOAH190zTUSfv7Z0Y4P7RbjnLc+9ozeW49t6iK57LtxUXO3k3MIwIF9vUuJJc3/1r9FUC1oUfzAbPJjc17lpzQDzjZk157X2HSM4vVlMmuMBpJszjE= ceph-67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T15:40:36.363 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-22T15:40:36.566 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:40:36.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:36 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/2188237035' entity='client.admin' 2024-04-22T15:40:38.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.400+0000 7fae7ec28700 1 -- 172.21.15.16:0/3195761760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae780fef10 msgr2=0x7fae780ff330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:38.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.400+0000 7fae7ec28700 1 --2- 172.21.15.16:0/3195761760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae780fef10 0x7fae780ff330 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fae68009910 tx=0x7fae68009c20 comp rx=0 tx=0).stop 2024-04-22T15:40:38.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.400+0000 7fae7ec28700 1 -- 172.21.15.16:0/3195761760 shutdown_connections 2024-04-22T15:40:38.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.400+0000 7fae7ec28700 1 --2- 172.21.15.16:0/3195761760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae780fef10 0x7fae780ff330 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:38.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.400+0000 7fae7ec28700 1 -- 172.21.15.16:0/3195761760 >> 172.21.15.16:0/3195761760 conn(0x7fae780fa490 msgr2=0x7fae780fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:38.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.400+0000 7fae7ec28700 1 -- 172.21.15.16:0/3195761760 shutdown_connections 2024-04-22T15:40:38.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.400+0000 7fae7ec28700 1 -- 172.21.15.16:0/3195761760 wait complete. 2024-04-22T15:40:38.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.401+0000 7fae7ec28700 1 Processor -- start 2024-04-22T15:40:38.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.401+0000 7fae7ec28700 1 -- start start 2024-04-22T15:40:38.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.401+0000 7fae7ec28700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae780fef10 0x7fae781948b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:38.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.401+0000 7fae7ec28700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae78194df0 con 0x7fae780fef10 2024-04-22T15:40:38.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.401+0000 7fae7c9c4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae780fef10 0x7fae781948b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:38.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.401+0000 7fae7c9c4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae780fef10 0x7fae781948b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34296/0 (socket says 172.21.15.16:34296) 2024-04-22T15:40:38.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.401+0000 7fae7c9c4700 1 -- 172.21.15.16:0/2124476833 learned_addr learned my addr 172.21.15.16:0/2124476833 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:38.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.401+0000 7fae7c9c4700 1 -- 172.21.15.16:0/2124476833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae680095f0 con 0x7fae780fef10 2024-04-22T15:40:38.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.402+0000 7fae7c9c4700 1 --2- 172.21.15.16:0/2124476833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae780fef10 0x7fae781948b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fae680092e0 tx=0x7fae68009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:38.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.402+0000 7fae75ffb700 1 -- 172.21.15.16:0/2124476833 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae6801d070 con 0x7fae780fef10 2024-04-22T15:40:38.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.402+0000 7fae75ffb700 1 -- 172.21.15.16:0/2124476833 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fae6800be10 con 0x7fae780fef10 2024-04-22T15:40:38.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.402+0000 7fae7ec28700 1 -- 172.21.15.16:0/2124476833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae78194ff0 con 0x7fae780fef10 2024-04-22T15:40:38.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.402+0000 7fae75ffb700 1 -- 172.21.15.16:0/2124476833 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae6800f460 con 0x7fae780fef10 2024-04-22T15:40:38.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.402+0000 7fae7ec28700 1 -- 172.21.15.16:0/2124476833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae78195490 con 0x7fae780fef10 2024-04-22T15:40:38.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.403+0000 7fae75ffb700 1 -- 172.21.15.16:0/2124476833 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7fae6800f6d0 con 0x7fae780fef10 2024-04-22T15:40:38.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.403+0000 7fae7ec28700 1 -- 172.21.15.16:0/2124476833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae78104cc0 con 0x7fae780fef10 2024-04-22T15:40:38.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.403+0000 7fae75ffb700 1 --2- 172.21.15.16:0/2124476833 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fae600383c0 0x7fae6003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:38.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.403+0000 7fae75ffb700 1 -- 172.21.15.16:0/2124476833 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fae6804c480 con 0x7fae780fef10 2024-04-22T15:40:38.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.403+0000 7fae77fff700 1 --2- 172.21.15.16:0/2124476833 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fae600383c0 0x7fae6003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:38.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.403+0000 7fae77fff700 1 --2- 172.21.15.16:0/2124476833 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fae600383c0 0x7fae6003a880 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fae6c006fd0 tx=0x7fae6c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:38.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.408+0000 7fae75ffb700 1 -- 172.21.15.16:0/2124476833 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fae68005c00 con 0x7fae780fef10 2024-04-22T15:40:38.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.605+0000 7fae7ec28700 1 -- 172.21.15.16:0/2124476833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fae78198270 con 0x7fae780fef10 2024-04-22T15:40:38.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.608+0000 7fae75ffb700 1 -- 172.21.15.16:0/2124476833 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fae68027020 con 0x7fae780fef10 2024-04-22T15:40:38.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.610+0000 7fae7ec28700 1 -- 172.21.15.16:0/2124476833 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fae600383c0 msgr2=0x7fae6003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:38.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.610+0000 7fae7ec28700 1 --2- 172.21.15.16:0/2124476833 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fae600383c0 0x7fae6003a880 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fae6c006fd0 tx=0x7fae6c006e40 comp rx=0 tx=0).stop 2024-04-22T15:40:38.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.610+0000 7fae7ec28700 1 -- 172.21.15.16:0/2124476833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae780fef10 msgr2=0x7fae781948b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:38.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.610+0000 7fae7ec28700 1 --2- 172.21.15.16:0/2124476833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae780fef10 0x7fae781948b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fae680092e0 tx=0x7fae68009e60 comp rx=0 tx=0).stop 2024-04-22T15:40:38.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.610+0000 7fae7ec28700 1 -- 172.21.15.16:0/2124476833 shutdown_connections 2024-04-22T15:40:38.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.610+0000 7fae7ec28700 1 --2- 172.21.15.16:0/2124476833 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fae600383c0 0x7fae6003a880 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:38.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.610+0000 7fae7ec28700 1 --2- 172.21.15.16:0/2124476833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae780fef10 0x7fae781948b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:38.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.610+0000 7fae7ec28700 1 -- 172.21.15.16:0/2124476833 >> 172.21.15.16:0/2124476833 conn(0x7fae780fa490 msgr2=0x7fae780fb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:38.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.610+0000 7fae7ec28700 1 -- 172.21.15.16:0/2124476833 shutdown_connections 2024-04-22T15:40:38.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:38.610+0000 7fae7ec28700 1 -- 172.21.15.16:0/2124476833 wait complete. 2024-04-22T15:40:39.308 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-22T15:40:39.308 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-22T15:40:39.804 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:39 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/2124476833' entity='client.admin' 2024-04-22T15:40:39.864 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:40:42.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.039+0000 7f7bca7c7700 1 -- 172.21.15.16:0/911566462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bc40ff780 msgr2=0x7f7bc40ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:42.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.039+0000 7f7bca7c7700 1 --2- 172.21.15.16:0/911566462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bc40ff780 0x7f7bc40ffba0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4009910 tx=0x7f7bb4009c20 comp rx=0 tx=0).stop 2024-04-22T15:40:42.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.039+0000 7f7bca7c7700 1 -- 172.21.15.16:0/911566462 shutdown_connections 2024-04-22T15:40:42.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.039+0000 7f7bca7c7700 1 --2- 172.21.15.16:0/911566462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bc40ff780 0x7f7bc40ffba0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.039+0000 7f7bca7c7700 1 -- 172.21.15.16:0/911566462 >> 172.21.15.16:0/911566462 conn(0x7f7bc40fad20 msgr2=0x7f7bc40fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.039+0000 7f7bca7c7700 1 -- 172.21.15.16:0/911566462 shutdown_connections 2024-04-22T15:40:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.039+0000 7f7bca7c7700 1 -- 172.21.15.16:0/911566462 wait complete. 2024-04-22T15:40:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.040+0000 7f7bca7c7700 1 Processor -- start 2024-04-22T15:40:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.040+0000 7f7bca7c7700 1 -- start start 2024-04-22T15:40:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.040+0000 7f7bca7c7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bc40ff780 0x7f7bc40fccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.040+0000 7f7bca7c7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc40fd200 con 0x7f7bc40ff780 2024-04-22T15:40:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.040+0000 7f7bc3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bc40ff780 0x7f7bc40fccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.040+0000 7f7bc3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bc40ff780 0x7f7bc40fccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33734/0 (socket says 172.21.15.16:33734) 2024-04-22T15:40:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.040+0000 7f7bc3fff700 1 -- 172.21.15.16:0/1477607443 learned_addr learned my addr 172.21.15.16:0/1477607443 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.040+0000 7f7bc3fff700 1 -- 172.21.15.16:0/1477607443 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb40095f0 con 0x7f7bc40ff780 2024-04-22T15:40:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.040+0000 7f7bc3fff700 1 --2- 172.21.15.16:0/1477607443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bc40ff780 0x7f7bc40fccc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4005950 tx=0x7f7bb4009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.041+0000 7f7bc17fa700 1 -- 172.21.15.16:0/1477607443 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bb401d070 con 0x7f7bc40ff780 2024-04-22T15:40:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.041+0000 7f7bc17fa700 1 -- 172.21.15.16:0/1477607443 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7bb400be10 con 0x7f7bc40ff780 2024-04-22T15:40:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.041+0000 7f7bc17fa700 1 -- 172.21.15.16:0/1477607443 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bb400f460 con 0x7f7bc40ff780 2024-04-22T15:40:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.041+0000 7f7bca7c7700 1 -- 172.21.15.16:0/1477607443 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc40fb310 con 0x7f7bc40ff780 2024-04-22T15:40:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.041+0000 7f7bca7c7700 1 -- 172.21.15.16:0/1477607443 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc40fb7b0 con 0x7f7bc40ff780 2024-04-22T15:40:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.042+0000 7f7bc17fa700 1 -- 172.21.15.16:0/1477607443 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f7bb400b960 con 0x7f7bc40ff780 2024-04-22T15:40:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.042+0000 7f7bca7c7700 1 -- 172.21.15.16:0/1477607443 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc40623c0 con 0x7f7bc40ff780 2024-04-22T15:40:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.042+0000 7f7bc17fa700 1 --2- 172.21.15.16:0/1477607443 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f7bac0383c0 0x7f7bac03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.042+0000 7f7bc17fa700 1 -- 172.21.15.16:0/1477607443 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7bb404c3e0 con 0x7f7bc40ff780 2024-04-22T15:40:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.042+0000 7f7bc37fe700 1 --2- 172.21.15.16:0/1477607443 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f7bac0383c0 0x7f7bac03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.042+0000 7f7bc37fe700 1 --2- 172.21.15.16:0/1477607443 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f7bac0383c0 0x7f7bac03a880 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8006fd0 tx=0x7f7bb8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.047+0000 7f7bc17fa700 1 -- 172.21.15.16:0/1477607443 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7bb402a360 con 0x7f7bc40ff780 2024-04-22T15:40:42.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.255+0000 7f7bca7c7700 1 -- 172.21.15.16:0/1477607443 --> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f7bc40fba80 con 0x7f7bac0383c0 2024-04-22T15:40:42.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.258+0000 7f7bc17fa700 1 -- 172.21.15.16:0/1477607443 <== mgr.14162 v2:172.21.15.16:6800/3972982132 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7bc40fba80 con 0x7f7bac0383c0 2024-04-22T15:40:42.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.260+0000 7f7bca7c7700 1 -- 172.21.15.16:0/1477607443 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f7bac0383c0 msgr2=0x7f7bac03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:42.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.260+0000 7f7bca7c7700 1 --2- 172.21.15.16:0/1477607443 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f7bac0383c0 0x7f7bac03a880 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8006fd0 tx=0x7f7bb8006e40 comp rx=0 tx=0).stop 2024-04-22T15:40:42.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.260+0000 7f7bca7c7700 1 -- 172.21.15.16:0/1477607443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bc40ff780 msgr2=0x7f7bc40fccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:42.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.260+0000 7f7bca7c7700 1 --2- 172.21.15.16:0/1477607443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bc40ff780 0x7f7bc40fccc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4005950 tx=0x7f7bb4009e60 comp rx=0 tx=0).stop 2024-04-22T15:40:42.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.261+0000 7f7bca7c7700 1 -- 172.21.15.16:0/1477607443 shutdown_connections 2024-04-22T15:40:42.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.261+0000 7f7bca7c7700 1 --2- 172.21.15.16:0/1477607443 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f7bac0383c0 0x7f7bac03a880 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:42.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.261+0000 7f7bca7c7700 1 --2- 172.21.15.16:0/1477607443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bc40ff780 0x7f7bc40fccc0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:42.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.261+0000 7f7bca7c7700 1 -- 172.21.15.16:0/1477607443 >> 172.21.15.16:0/1477607443 conn(0x7f7bc40fad20 msgr2=0x7f7bc4068820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:42.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.261+0000 7f7bca7c7700 1 -- 172.21.15.16:0/1477607443 shutdown_connections 2024-04-22T15:40:42.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:42.261+0000 7f7bca7c7700 1 -- 172.21.15.16:0/1477607443 wait complete. 2024-04-22T15:40:42.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:42 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:42.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:42 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:42.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:42 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd/host:smithi016", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:40:42.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:42 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:42.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:42 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi016", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:40:42.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:42 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi016", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-22T15:40:42.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:42 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:40:42.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:42 smithi016 ceph-mon[26620]: Deploying daemon ceph-exporter.smithi016 on smithi016 2024-04-22T15:40:43.236 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi027 2024-04-22T15:40:43.236 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-22T15:40:43.236 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/etc/ceph/ceph.conf 2024-04-22T15:40:43.260 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-22T15:40:43.260 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:40:43.317 INFO:tasks.cephadm:Adding host smithi027 to orchestrator... 2024-04-22T15:40:43.317 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph orch host add smithi027 2024-04-22T15:40:43.618 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:43 smithi016 ceph-mon[26620]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:43.618 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:43 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:43.711 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:40:45.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.308+0000 7f4dc9359700 1 -- 172.21.15.16:0/1335107864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dc40ff780 msgr2=0x7f4dc40ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:45.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.308+0000 7f4dc9359700 1 --2- 172.21.15.16:0/1335107864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dc40ff780 0x7f4dc40ffba0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc009910 tx=0x7f4dbc009c20 comp rx=0 tx=0).stop 2024-04-22T15:40:45.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.308+0000 7f4dc9359700 1 -- 172.21.15.16:0/1335107864 shutdown_connections 2024-04-22T15:40:45.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.308+0000 7f4dc9359700 1 --2- 172.21.15.16:0/1335107864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dc40ff780 0x7f4dc40ffba0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:45.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.308+0000 7f4dc9359700 1 -- 172.21.15.16:0/1335107864 >> 172.21.15.16:0/1335107864 conn(0x7f4dc40fad20 msgr2=0x7f4dc40fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:45.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.308+0000 7f4dc9359700 1 -- 172.21.15.16:0/1335107864 shutdown_connections 2024-04-22T15:40:45.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.308+0000 7f4dc9359700 1 -- 172.21.15.16:0/1335107864 wait complete. 2024-04-22T15:40:45.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.309+0000 7f4dc9359700 1 Processor -- start 2024-04-22T15:40:45.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.309+0000 7f4dc9359700 1 -- start start 2024-04-22T15:40:45.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.309+0000 7f4dc9359700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dc40ff780 0x7f4dc41959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:45.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.309+0000 7f4dc9359700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc4195ee0 con 0x7f4dc40ff780 2024-04-22T15:40:45.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.309+0000 7f4dc2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dc40ff780 0x7f4dc41959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:45.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.309+0000 7f4dc2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dc40ff780 0x7f4dc41959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33748/0 (socket says 172.21.15.16:33748) 2024-04-22T15:40:45.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.309+0000 7f4dc2ffd700 1 -- 172.21.15.16:0/3025295023 learned_addr learned my addr 172.21.15.16:0/3025295023 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:45.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.309+0000 7f4dc2ffd700 1 -- 172.21.15.16:0/3025295023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dbc0095f0 con 0x7f4dc40ff780 2024-04-22T15:40:45.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.310+0000 7f4dc2ffd700 1 --2- 172.21.15.16:0/3025295023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dc40ff780 0x7f4dc41959a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc005b40 tx=0x7f4dbc004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:45.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.310+0000 7f4dbbfff700 1 -- 172.21.15.16:0/3025295023 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4dbc01d070 con 0x7f4dc40ff780 2024-04-22T15:40:45.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.310+0000 7f4dc9359700 1 -- 172.21.15.16:0/3025295023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc41960e0 con 0x7f4dc40ff780 2024-04-22T15:40:45.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.310+0000 7f4dbbfff700 1 -- 172.21.15.16:0/3025295023 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4dbc00be10 con 0x7f4dc40ff780 2024-04-22T15:40:45.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.310+0000 7f4dbbfff700 1 -- 172.21.15.16:0/3025295023 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4dbc00f460 con 0x7f4dc40ff780 2024-04-22T15:40:45.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.310+0000 7f4dc9359700 1 -- 172.21.15.16:0/3025295023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc4196580 con 0x7f4dc40ff780 2024-04-22T15:40:45.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.311+0000 7f4dbbfff700 1 -- 172.21.15.16:0/3025295023 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f4dbc00f6d0 con 0x7f4dc40ff780 2024-04-22T15:40:45.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.311+0000 7f4dc9359700 1 -- 172.21.15.16:0/3025295023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc418f3b0 con 0x7f4dc40ff780 2024-04-22T15:40:45.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.311+0000 7f4dbbfff700 1 --2- 172.21.15.16:0/3025295023 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f4db00383c0 0x7f4db003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:45.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.311+0000 7f4dbbfff700 1 -- 172.21.15.16:0/3025295023 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4dbc04c500 con 0x7f4dc40ff780 2024-04-22T15:40:45.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.311+0000 7f4dc27fc700 1 --2- 172.21.15.16:0/3025295023 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f4db00383c0 0x7f4db003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:45.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.311+0000 7f4dc27fc700 1 --2- 172.21.15.16:0/3025295023 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f4db00383c0 0x7f4db003a880 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4dac006fd0 tx=0x7f4dac006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:45.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.316+0000 7f4dbbfff700 1 -- 172.21.15.16:0/3025295023 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4dbc02a360 con 0x7f4dc40ff780 2024-04-22T15:40:45.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:45.524+0000 7f4dc9359700 1 -- 172.21.15.16:0/3025295023 --> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi027", "target": ["mon-mgr", ""]}) v1 -- 0x7f4dc40611d0 con 0x7f4db00383c0 2024-04-22T15:40:46.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:46 smithi016 ceph-mon[26620]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi027", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:46.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:46 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:46.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:46 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:46.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:46 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:46.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:46 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:46.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:46 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi016", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:40:46.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:46 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi016", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-22T15:40:46.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:46 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:40:46.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:46 smithi016 ceph-mon[26620]: Deploying daemon crash.smithi016 on smithi016 2024-04-22T15:40:46.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:46 smithi016 ceph-mon[26620]: Deploying cephadm binary to smithi027 2024-04-22T15:40:48.097 INFO:teuthology.orchestra.run.smithi016.stdout:Added host 'smithi027' with addr '172.21.15.27' 2024-04-22T15:40:48.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:48.094+0000 7f4dbbfff700 1 -- 172.21.15.16:0/3025295023 <== mgr.14162 v2:172.21.15.16:6800/3972982132 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f4dc40611d0 con 0x7f4db00383c0 2024-04-22T15:40:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:48.096+0000 7f4dc9359700 1 -- 172.21.15.16:0/3025295023 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f4db00383c0 msgr2=0x7f4db003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:48.096+0000 7f4dc9359700 1 --2- 172.21.15.16:0/3025295023 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f4db00383c0 0x7f4db003a880 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4dac006fd0 tx=0x7f4dac006e40 comp rx=0 tx=0).stop 2024-04-22T15:40:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:48.097+0000 7f4dc9359700 1 -- 172.21.15.16:0/3025295023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dc40ff780 msgr2=0x7f4dc41959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:48.097+0000 7f4dc9359700 1 --2- 172.21.15.16:0/3025295023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dc40ff780 0x7f4dc41959a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc005b40 tx=0x7f4dbc004740 comp rx=0 tx=0).stop 2024-04-22T15:40:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:48.097+0000 7f4dc9359700 1 -- 172.21.15.16:0/3025295023 shutdown_connections 2024-04-22T15:40:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:48.097+0000 7f4dc9359700 1 --2- 172.21.15.16:0/3025295023 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f4db00383c0 0x7f4db003a880 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:48.097+0000 7f4dc9359700 1 --2- 172.21.15.16:0/3025295023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dc40ff780 0x7f4dc41959a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:48.097+0000 7f4dc9359700 1 -- 172.21.15.16:0/3025295023 >> 172.21.15.16:0/3025295023 conn(0x7f4dc40fad20 msgr2=0x7f4dc4068820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:48.097+0000 7f4dc9359700 1 -- 172.21.15.16:0/3025295023 shutdown_connections 2024-04-22T15:40:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:48.097+0000 7f4dc9359700 1 -- 172.21.15.16:0/3025295023 wait complete. 2024-04-22T15:40:48.395 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:48 smithi016 ceph-mon[26620]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:40:48.395 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:48 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:48.395 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:48 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:48.897 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-22T15:40:49.112 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:40:49.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:49 smithi016 ceph-mon[26620]: Added host smithi027 2024-04-22T15:40:49.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:49 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:49.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:49 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:49.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:49 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:49.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:49 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:50.272 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:50 smithi016 ceph-mon[26620]: Deploying daemon node-exporter.smithi016 on smithi016 2024-04-22T15:40:50.273 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:50 smithi016 ceph-mon[26620]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:40:50.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.811+0000 7f75dd406700 1 -- 172.21.15.16:0/2458927059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75d8101310 msgr2=0x7f75d8103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:50.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.811+0000 7f75dd406700 1 --2- 172.21.15.16:0/2458927059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75d8101310 0x7f75d8103700 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f75c8009910 tx=0x7f75c8009c20 comp rx=0 tx=0).stop 2024-04-22T15:40:50.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.811+0000 7f75dd406700 1 -- 172.21.15.16:0/2458927059 shutdown_connections 2024-04-22T15:40:50.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.811+0000 7f75dd406700 1 --2- 172.21.15.16:0/2458927059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75d8101310 0x7f75d8103700 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:50.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.811+0000 7f75dd406700 1 -- 172.21.15.16:0/2458927059 >> 172.21.15.16:0/2458927059 conn(0x7f75d80fad20 msgr2=0x7f75d80fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:50.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.811+0000 7f75dd406700 1 -- 172.21.15.16:0/2458927059 shutdown_connections 2024-04-22T15:40:50.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.811+0000 7f75dd406700 1 -- 172.21.15.16:0/2458927059 wait complete. 2024-04-22T15:40:50.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.812+0000 7f75dd406700 1 Processor -- start 2024-04-22T15:40:50.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.812+0000 7f75dd406700 1 -- start start 2024-04-22T15:40:50.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.812+0000 7f75dd406700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75d8101310 0x7f75d8195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:50.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.813+0000 7f75dd406700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d8195ec0 con 0x7f75d8101310 2024-04-22T15:40:50.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.813+0000 7f75d6ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75d8101310 0x7f75d8195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:50.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.813+0000 7f75d6ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75d8101310 0x7f75d8195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33798/0 (socket says 172.21.15.16:33798) 2024-04-22T15:40:50.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.813+0000 7f75d6ffd700 1 -- 172.21.15.16:0/2916544650 learned_addr learned my addr 172.21.15.16:0/2916544650 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:50.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.813+0000 7f75d6ffd700 1 -- 172.21.15.16:0/2916544650 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c80095f0 con 0x7f75d8101310 2024-04-22T15:40:50.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.813+0000 7f75d6ffd700 1 --2- 172.21.15.16:0/2916544650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75d8101310 0x7f75d8195980 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f75c80092e0 tx=0x7f75c8009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:50.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.813+0000 7f75bffff700 1 -- 172.21.15.16:0/2916544650 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75c801d070 con 0x7f75d8101310 2024-04-22T15:40:50.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.813+0000 7f75bffff700 1 -- 172.21.15.16:0/2916544650 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f75c800be10 con 0x7f75d8101310 2024-04-22T15:40:50.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.813+0000 7f75dd406700 1 -- 172.21.15.16:0/2916544650 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d81960c0 con 0x7f75d8101310 2024-04-22T15:40:50.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.813+0000 7f75bffff700 1 -- 172.21.15.16:0/2916544650 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75c800f460 con 0x7f75d8101310 2024-04-22T15:40:50.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.813+0000 7f75dd406700 1 -- 172.21.15.16:0/2916544650 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d8196560 con 0x7f75d8101310 2024-04-22T15:40:50.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.814+0000 7f75bffff700 1 -- 172.21.15.16:0/2916544650 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f75c800f5c0 con 0x7f75d8101310 2024-04-22T15:40:50.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.814+0000 7f75dd406700 1 -- 172.21.15.16:0/2916544650 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d818f4a0 con 0x7f75d8101310 2024-04-22T15:40:50.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.814+0000 7f75bffff700 1 --2- 172.21.15.16:0/2916544650 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f75c0038410 0x7f75c003a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:50.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.814+0000 7f75bffff700 1 -- 172.21.15.16:0/2916544650 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f75c804c520 con 0x7f75d8101310 2024-04-22T15:40:50.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.815+0000 7f75d67fc700 1 --2- 172.21.15.16:0/2916544650 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f75c0038410 0x7f75c003a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:50.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.815+0000 7f75d67fc700 1 --2- 172.21.15.16:0/2916544650 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f75c0038410 0x7f75c003a8d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f75cc006fd0 tx=0x7f75cc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:50.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:50.819+0000 7f75bffff700 1 -- 172.21.15.16:0/2916544650 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f75c8005c00 con 0x7f75d8101310 2024-04-22T15:40:51.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:51.034+0000 7f75dd406700 1 -- 172.21.15.16:0/2916544650 --> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f75d80611d0 con 0x7f75c0038410 2024-04-22T15:40:51.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:51.035+0000 7f75bffff700 1 -- 172.21.15.16:0/2916544650 <== mgr.14162 v2:172.21.15.16:6800/3972982132 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f75d80611d0 con 0x7f75c0038410 2024-04-22T15:40:51.037 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:40:51.037 INFO:teuthology.orchestra.run.smithi016.stdout:[{"addr": "172.21.15.16", "hostname": "smithi016", "labels": [], "status": ""}, {"addr": "172.21.15.27", "hostname": "smithi027", "labels": [], "status": ""}] 2024-04-22T15:40:51.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:51.037+0000 7f75dd406700 1 -- 172.21.15.16:0/2916544650 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f75c0038410 msgr2=0x7f75c003a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:51.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:51.037+0000 7f75dd406700 1 --2- 172.21.15.16:0/2916544650 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f75c0038410 0x7f75c003a8d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f75cc006fd0 tx=0x7f75cc006e40 comp rx=0 tx=0).stop 2024-04-22T15:40:51.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:51.037+0000 7f75dd406700 1 -- 172.21.15.16:0/2916544650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75d8101310 msgr2=0x7f75d8195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:51.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:51.037+0000 7f75dd406700 1 --2- 172.21.15.16:0/2916544650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75d8101310 0x7f75d8195980 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f75c80092e0 tx=0x7f75c8009e60 comp rx=0 tx=0).stop 2024-04-22T15:40:51.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:51.037+0000 7f75dd406700 1 -- 172.21.15.16:0/2916544650 shutdown_connections 2024-04-22T15:40:51.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:51.037+0000 7f75dd406700 1 --2- 172.21.15.16:0/2916544650 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f75c0038410 0x7f75c003a8d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:51.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:51.037+0000 7f75dd406700 1 --2- 172.21.15.16:0/2916544650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75d8101310 0x7f75d8195980 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:51.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:51.037+0000 7f75dd406700 1 -- 172.21.15.16:0/2916544650 >> 172.21.15.16:0/2916544650 conn(0x7f75d80fad20 msgr2=0x7f75d80fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:51.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:51.037+0000 7f75dd406700 1 -- 172.21.15.16:0/2916544650 shutdown_connections 2024-04-22T15:40:51.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:51.037+0000 7f75dd406700 1 -- 172.21.15.16:0/2916544650 wait complete. 2024-04-22T15:40:51.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:51 smithi016 ceph-mon[26620]: from='client.14191 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T15:40:51.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:51 smithi016 ceph-mon[26620]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:40:51.780 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-22T15:40:51.780 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-22T15:40:51.990 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:40:53.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:53 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:54.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.310+0000 7fb769771700 1 -- 172.21.15.16:0/3444127650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb764101310 msgr2=0x7fb764103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:54.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.310+0000 7fb769771700 1 --2- 172.21.15.16:0/3444127650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb764101310 0x7fb764103700 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb74c009910 tx=0x7fb74c009c20 comp rx=0 tx=0).stop 2024-04-22T15:40:54.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.310+0000 7fb769771700 1 -- 172.21.15.16:0/3444127650 shutdown_connections 2024-04-22T15:40:54.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.310+0000 7fb769771700 1 --2- 172.21.15.16:0/3444127650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb764101310 0x7fb764103700 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:54.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.310+0000 7fb769771700 1 -- 172.21.15.16:0/3444127650 >> 172.21.15.16:0/3444127650 conn(0x7fb7640fad20 msgr2=0x7fb7640fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:54.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.310+0000 7fb769771700 1 -- 172.21.15.16:0/3444127650 shutdown_connections 2024-04-22T15:40:54.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.310+0000 7fb769771700 1 -- 172.21.15.16:0/3444127650 wait complete. 2024-04-22T15:40:54.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.311+0000 7fb769771700 1 Processor -- start 2024-04-22T15:40:54.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.311+0000 7fb769771700 1 -- start start 2024-04-22T15:40:54.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.311+0000 7fb769771700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb764101310 0x7fb764195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:54.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.311+0000 7fb769771700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb764195ec0 con 0x7fb764101310 2024-04-22T15:40:54.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.312+0000 7fb762ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb764101310 0x7fb764195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:54.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.312+0000 7fb762ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb764101310 0x7fb764195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53410/0 (socket says 172.21.15.16:53410) 2024-04-22T15:40:54.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.312+0000 7fb762ffd700 1 -- 172.21.15.16:0/3514681996 learned_addr learned my addr 172.21.15.16:0/3514681996 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:40:54.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.312+0000 7fb762ffd700 1 -- 172.21.15.16:0/3514681996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74c0095f0 con 0x7fb764101310 2024-04-22T15:40:54.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.312+0000 7fb762ffd700 1 --2- 172.21.15.16:0/3514681996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb764101310 0x7fb764195980 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb74c005b40 tx=0x7fb74c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:54.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.312+0000 7fb75bfff700 1 -- 172.21.15.16:0/3514681996 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb74c01d070 con 0x7fb764101310 2024-04-22T15:40:54.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.312+0000 7fb75bfff700 1 -- 172.21.15.16:0/3514681996 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb74c00be10 con 0x7fb764101310 2024-04-22T15:40:54.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.312+0000 7fb769771700 1 -- 172.21.15.16:0/3514681996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7641960c0 con 0x7fb764101310 2024-04-22T15:40:54.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.312+0000 7fb75bfff700 1 -- 172.21.15.16:0/3514681996 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb74c00f460 con 0x7fb764101310 2024-04-22T15:40:54.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.312+0000 7fb769771700 1 -- 172.21.15.16:0/3514681996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb764196560 con 0x7fb764101310 2024-04-22T15:40:54.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.313+0000 7fb75bfff700 1 -- 172.21.15.16:0/3514681996 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7fb74c00f6d0 con 0x7fb764101310 2024-04-22T15:40:54.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.313+0000 7fb75bfff700 1 --2- 172.21.15.16:0/3514681996 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fb7500383c0 0x7fb75003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:54.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.313+0000 7fb75bfff700 1 -- 172.21.15.16:0/3514681996 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb74c04c3c0 con 0x7fb764101310 2024-04-22T15:40:54.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.313+0000 7fb7627fc700 1 --2- 172.21.15.16:0/3514681996 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fb7500383c0 0x7fb75003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:54.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.313+0000 7fb769771700 1 -- 172.21.15.16:0/3514681996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb76418f4a0 con 0x7fb764101310 2024-04-22T15:40:54.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.314+0000 7fb7627fc700 1 --2- 172.21.15.16:0/3514681996 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fb7500383c0 0x7fb75003a880 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb754006fd0 tx=0x7fb754006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:54.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.318+0000 7fb75bfff700 1 -- 172.21.15.16:0/3514681996 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb74c027070 con 0x7fb764101310 2024-04-22T15:40:54.455 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:54 smithi016 ceph-mon[26620]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:40:54.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:54.524+0000 7fb769771700 1 -- 172.21.15.16:0/3514681996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fb7640623c0 con 0x7fb764101310 2024-04-22T15:40:55.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:55.172+0000 7fb75bfff700 1 -- 172.21.15.16:0/3514681996 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fb74c02c440 con 0x7fb764101310 2024-04-22T15:40:55.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:55.175+0000 7fb769771700 1 -- 172.21.15.16:0/3514681996 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fb7500383c0 msgr2=0x7fb75003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:55.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:55.175+0000 7fb769771700 1 --2- 172.21.15.16:0/3514681996 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fb7500383c0 0x7fb75003a880 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb754006fd0 tx=0x7fb754006e40 comp rx=0 tx=0).stop 2024-04-22T15:40:55.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:55.175+0000 7fb769771700 1 -- 172.21.15.16:0/3514681996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb764101310 msgr2=0x7fb764195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:55.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:55.175+0000 7fb769771700 1 --2- 172.21.15.16:0/3514681996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb764101310 0x7fb764195980 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb74c005b40 tx=0x7fb74c004740 comp rx=0 tx=0).stop 2024-04-22T15:40:55.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:55.175+0000 7fb769771700 1 -- 172.21.15.16:0/3514681996 shutdown_connections 2024-04-22T15:40:55.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:55.175+0000 7fb769771700 1 --2- 172.21.15.16:0/3514681996 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fb7500383c0 0x7fb75003a880 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:55.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:55.175+0000 7fb769771700 1 --2- 172.21.15.16:0/3514681996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb764101310 0x7fb764195980 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:55.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:55.175+0000 7fb769771700 1 -- 172.21.15.16:0/3514681996 >> 172.21.15.16:0/3514681996 conn(0x7fb7640fad20 msgr2=0x7fb7640fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:55.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:55.175+0000 7fb769771700 1 -- 172.21.15.16:0/3514681996 shutdown_connections 2024-04-22T15:40:55.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:40:55.175+0000 7fb769771700 1 -- 172.21.15.16:0/3514681996 wait complete. 2024-04-22T15:40:55.179 INFO:teuthology.orchestra.run.smithi016.stderr:adjusted tunables profile to default 2024-04-22T15:40:55.481 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:55 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/3514681996' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-22T15:40:55.481 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:55 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:55.482 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:55 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:55.482 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:55 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:55.482 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:55 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:55.784 INFO:tasks.cephadm:Adding mon.smithi016 on smithi016 2024-04-22T15:40:55.784 INFO:tasks.cephadm:Adding mon.smithi027 on smithi027 2024-04-22T15:40:55.784 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph orch apply mon '2;smithi016:172.21.15.16=smithi016;smithi027:172.21.15.27=smithi027' 2024-04-22T15:40:55.995 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:40:56.043 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:40:56.511 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:56 smithi016 ceph-mon[26620]: Deploying daemon alertmanager.smithi016 on smithi016 2024-04-22T15:40:56.511 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:56 smithi016 ceph-mon[26620]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:40:56.511 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:56 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/3514681996' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-22T15:40:56.511 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:56 smithi016 ceph-mon[26620]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T15:40:57.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.464+0000 7f46b616a700 1 -- 172.21.15.27:0/755919140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46b00ff780 msgr2=0x7f46b00ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:57.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.464+0000 7f46b616a700 1 --2- 172.21.15.27:0/755919140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46b00ff780 0x7f46b00ffba0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f46a0009910 tx=0x7f46a0009c20 comp rx=0 tx=0).stop 2024-04-22T15:40:57.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.464+0000 7f46b616a700 1 -- 172.21.15.27:0/755919140 shutdown_connections 2024-04-22T15:40:57.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.464+0000 7f46b616a700 1 --2- 172.21.15.27:0/755919140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46b00ff780 0x7f46b00ffba0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:57.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.464+0000 7f46b616a700 1 -- 172.21.15.27:0/755919140 >> 172.21.15.27:0/755919140 conn(0x7f46b00fad20 msgr2=0x7f46b00fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:57.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.464+0000 7f46b616a700 1 -- 172.21.15.27:0/755919140 shutdown_connections 2024-04-22T15:40:57.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.464+0000 7f46b616a700 1 -- 172.21.15.27:0/755919140 wait complete. 2024-04-22T15:40:57.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.464+0000 7f46b616a700 1 Processor -- start 2024-04-22T15:40:57.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.464+0000 7f46b616a700 1 -- start start 2024-04-22T15:40:57.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.465+0000 7f46b616a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46b00ff780 0x7f46b0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:57.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.465+0000 7f46b616a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b0198670 con 0x7f46b00ff780 2024-04-22T15:40:57.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.465+0000 7f46af7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46b00ff780 0x7f46b0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:57.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.465+0000 7f46af7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46b00ff780 0x7f46b0069420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.27:38336/0 (socket says 172.21.15.27:38336) 2024-04-22T15:40:57.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.465+0000 7f46af7fe700 1 -- 172.21.15.27:0/1764977703 learned_addr learned my addr 172.21.15.27:0/1764977703 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-22T15:40:57.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.465+0000 7f46af7fe700 1 -- 172.21.15.27:0/1764977703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a00095f0 con 0x7f46b00ff780 2024-04-22T15:40:57.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.465+0000 7f46af7fe700 1 --2- 172.21.15.27:0/1764977703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46b00ff780 0x7f46b0069420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f46a0005950 tx=0x7f46a0009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:57.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.465+0000 7f46acff9700 1 -- 172.21.15.27:0/1764977703 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46a001d070 con 0x7f46b00ff780 2024-04-22T15:40:57.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.465+0000 7f46acff9700 1 -- 172.21.15.27:0/1764977703 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f46a000be10 con 0x7f46b00ff780 2024-04-22T15:40:57.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.465+0000 7f46b616a700 1 -- 172.21.15.27:0/1764977703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b01987b0 con 0x7f46b00ff780 2024-04-22T15:40:57.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.465+0000 7f46acff9700 1 -- 172.21.15.27:0/1764977703 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46a000f460 con 0x7f46b00ff780 2024-04-22T15:40:57.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.465+0000 7f46b616a700 1 -- 172.21.15.27:0/1764977703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b0198c50 con 0x7f46b00ff780 2024-04-22T15:40:57.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.466+0000 7f46acff9700 1 -- 172.21.15.27:0/1764977703 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f46a000b960 con 0x7f46b00ff780 2024-04-22T15:40:57.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.466+0000 7f46acff9700 1 --2- 172.21.15.27:0/1764977703 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f4698038410 0x7f469803a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:40:57.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.466+0000 7f46b616a700 1 -- 172.21.15.27:0/1764977703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46b018e3a0 con 0x7f46b00ff780 2024-04-22T15:40:57.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.466+0000 7f46acff9700 1 -- 172.21.15.27:0/1764977703 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f46a004c490 con 0x7f46b00ff780 2024-04-22T15:40:57.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.467+0000 7f46aeffd700 1 --2- 172.21.15.27:0/1764977703 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f4698038410 0x7f469803a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:40:57.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.467+0000 7f46aeffd700 1 --2- 172.21.15.27:0/1764977703 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f4698038410 0x7f469803a8d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f46a4006fd0 tx=0x7f46a4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:40:57.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.472+0000 7f46acff9700 1 -- 172.21.15.27:0/1764977703 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f46a002bba0 con 0x7f46b00ff780 2024-04-22T15:40:57.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.682+0000 7f46b616a700 1 -- 172.21.15.27:0/1764977703 --> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi016:172.21.15.16=smithi016;smithi027:172.21.15.27=smithi027", "target": ["mon-mgr", ""]}) v1 -- 0x7f46b0061340 con 0x7f4698038410 2024-04-22T15:40:57.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.689+0000 7f46acff9700 1 -- 172.21.15.27:0/1764977703 <== mgr.14162 v2:172.21.15.16:6800/3972982132 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f46b0061340 con 0x7f4698038410 2024-04-22T15:40:57.692 INFO:teuthology.orchestra.run.smithi027.stdout:Scheduled mon update... 2024-04-22T15:40:57.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.690+0000 7f46b616a700 1 -- 172.21.15.27:0/1764977703 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f4698038410 msgr2=0x7f469803a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:57.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.690+0000 7f46b616a700 1 --2- 172.21.15.27:0/1764977703 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f4698038410 0x7f469803a8d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f46a4006fd0 tx=0x7f46a4006e40 comp rx=0 tx=0).stop 2024-04-22T15:40:57.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.690+0000 7f46b616a700 1 -- 172.21.15.27:0/1764977703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46b00ff780 msgr2=0x7f46b0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:40:57.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.690+0000 7f46b616a700 1 --2- 172.21.15.27:0/1764977703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46b00ff780 0x7f46b0069420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f46a0005950 tx=0x7f46a0009e60 comp rx=0 tx=0).stop 2024-04-22T15:40:57.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.691+0000 7f46b616a700 1 -- 172.21.15.27:0/1764977703 shutdown_connections 2024-04-22T15:40:57.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.691+0000 7f46b616a700 1 --2- 172.21.15.27:0/1764977703 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f4698038410 0x7f469803a8d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:57.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.691+0000 7f46b616a700 1 --2- 172.21.15.27:0/1764977703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46b00ff780 0x7f46b0069420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:40:57.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.691+0000 7f46b616a700 1 -- 172.21.15.27:0/1764977703 >> 172.21.15.27:0/1764977703 conn(0x7f46b00fad20 msgr2=0x7f46b00fb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:40:57.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.691+0000 7f46b616a700 1 -- 172.21.15.27:0/1764977703 shutdown_connections 2024-04-22T15:40:57.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:40:57.691+0000 7f46b616a700 1 -- 172.21.15.27:0/1764977703 wait complete. 2024-04-22T15:40:58.350 DEBUG:teuthology.orchestra.run.smithi027:mon.smithi027> sudo journalctl -f -n 0 -u ceph-67ab3a80-00be-11ef-bc93-c7b262605968@mon.smithi027.service 2024-04-22T15:40:58.352 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:40:58.352 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:40:58.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:58 smithi016 ceph-mon[26620]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:40:58.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:58 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:58.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:58 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:40:58.694 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:40:58.743 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:40:59.647 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:59 smithi016 ceph-mon[26620]: from='client.14195 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi016:172.21.15.16=smithi016;smithi027:172.21.15.27=smithi027", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:40:59.648 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:40:59 smithi016 ceph-mon[26620]: Saving service mon spec with placement smithi016:172.21.15.16=smithi016;smithi027:172.21.15.27=smithi027;count:2 2024-04-22T15:41:00.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.253+0000 7f16cd1fe700 1 -- 172.21.15.27:0/3109256168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c80ff790 msgr2=0x7f16c80ffbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:00.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.253+0000 7f16cd1fe700 1 --2- 172.21.15.27:0/3109256168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c80ff790 0x7f16c80ffbb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f16b8009910 tx=0x7f16b8009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:00.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.253+0000 7f16cd1fe700 1 -- 172.21.15.27:0/3109256168 shutdown_connections 2024-04-22T15:41:00.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.253+0000 7f16cd1fe700 1 --2- 172.21.15.27:0/3109256168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c80ff790 0x7f16c80ffbb0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:00.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.253+0000 7f16cd1fe700 1 -- 172.21.15.27:0/3109256168 >> 172.21.15.27:0/3109256168 conn(0x7f16c80fad30 msgr2=0x7f16c80fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:00.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.253+0000 7f16cd1fe700 1 -- 172.21.15.27:0/3109256168 shutdown_connections 2024-04-22T15:41:00.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.253+0000 7f16cd1fe700 1 -- 172.21.15.27:0/3109256168 wait complete. 2024-04-22T15:41:00.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.254+0000 7f16cd1fe700 1 Processor -- start 2024-04-22T15:41:00.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.254+0000 7f16cd1fe700 1 -- start start 2024-04-22T15:41:00.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.254+0000 7f16cd1fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c80ff790 0x7f16c81959b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:00.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.254+0000 7f16cd1fe700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c8195ef0 con 0x7f16c80ff790 2024-04-22T15:41:00.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.255+0000 7f16c6d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c80ff790 0x7f16c81959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:00.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.255+0000 7f16c6d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c80ff790 0x7f16c81959b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.27:38362/0 (socket says 172.21.15.27:38362) 2024-04-22T15:41:00.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.255+0000 7f16c6d9d700 1 -- 172.21.15.27:0/733834676 learned_addr learned my addr 172.21.15.27:0/733834676 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-22T15:41:00.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.255+0000 7f16c6d9d700 1 -- 172.21.15.27:0/733834676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b80095f0 con 0x7f16c80ff790 2024-04-22T15:41:00.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.255+0000 7f16c6d9d700 1 --2- 172.21.15.27:0/733834676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c80ff790 0x7f16c81959b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f16b8005b40 tx=0x7f16b8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:00.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.255+0000 7f16affff700 1 -- 172.21.15.27:0/733834676 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16b801d070 con 0x7f16c80ff790 2024-04-22T15:41:00.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.255+0000 7f16cd1fe700 1 -- 172.21.15.27:0/733834676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c81960f0 con 0x7f16c80ff790 2024-04-22T15:41:00.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.255+0000 7f16affff700 1 -- 172.21.15.27:0/733834676 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f16b800be10 con 0x7f16c80ff790 2024-04-22T15:41:00.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.255+0000 7f16affff700 1 -- 172.21.15.27:0/733834676 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16b800f460 con 0x7f16c80ff790 2024-04-22T15:41:00.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.256+0000 7f16cd1fe700 1 -- 172.21.15.27:0/733834676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c8196590 con 0x7f16c80ff790 2024-04-22T15:41:00.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.256+0000 7f16cd1fe700 1 -- 172.21.15.27:0/733834676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c818f3c0 con 0x7f16c80ff790 2024-04-22T15:41:00.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.256+0000 7f16affff700 1 -- 172.21.15.27:0/733834676 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f16b800f6d0 con 0x7f16c80ff790 2024-04-22T15:41:00.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.257+0000 7f16affff700 1 --2- 172.21.15.27:0/733834676 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f16b00383c0 0x7f16b003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:00.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.257+0000 7f16affff700 1 -- 172.21.15.27:0/733834676 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f16b804c600 con 0x7f16c80ff790 2024-04-22T15:41:00.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.257+0000 7f16c659c700 1 --2- 172.21.15.27:0/733834676 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f16b00383c0 0x7f16b003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:00.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.257+0000 7f16c659c700 1 --2- 172.21.15.27:0/733834676 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f16b00383c0 0x7f16b003a880 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f16bc006fd0 tx=0x7f16bc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:00.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.262+0000 7f16affff700 1 -- 172.21.15.27:0/733834676 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f16b802a360 con 0x7f16c80ff790 2024-04-22T15:41:00.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:00 smithi016 ceph-mon[26620]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:41:00.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.539+0000 7f16cd1fe700 1 -- 172.21.15.27:0/733834676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16c802cc70 con 0x7f16c80ff790 2024-04-22T15:41:00.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.540+0000 7f16affff700 1 -- 172.21.15.27:0/733834676 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f16b8027030 con 0x7f16c80ff790 2024-04-22T15:41:00.543 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:41:00.543 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":1,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","modified":"2024-04-22T15:39:17.702487Z","created":"2024-04-22T15:39:17.702487Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:41:00.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.542+0000 7f16cd1fe700 1 -- 172.21.15.27:0/733834676 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f16b00383c0 msgr2=0x7f16b003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:00.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.542+0000 7f16cd1fe700 1 --2- 172.21.15.27:0/733834676 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f16b00383c0 0x7f16b003a880 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f16bc006fd0 tx=0x7f16bc006e40 comp rx=0 tx=0).stop 2024-04-22T15:41:00.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.542+0000 7f16cd1fe700 1 -- 172.21.15.27:0/733834676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c80ff790 msgr2=0x7f16c81959b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:00.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.542+0000 7f16cd1fe700 1 --2- 172.21.15.27:0/733834676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c80ff790 0x7f16c81959b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f16b8005b40 tx=0x7f16b8004740 comp rx=0 tx=0).stop 2024-04-22T15:41:00.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.542+0000 7f16cd1fe700 1 -- 172.21.15.27:0/733834676 shutdown_connections 2024-04-22T15:41:00.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.542+0000 7f16cd1fe700 1 --2- 172.21.15.27:0/733834676 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f16b00383c0 0x7f16b003a880 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:00.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.542+0000 7f16cd1fe700 1 --2- 172.21.15.27:0/733834676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c80ff790 0x7f16c81959b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:00.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.542+0000 7f16cd1fe700 1 -- 172.21.15.27:0/733834676 >> 172.21.15.27:0/733834676 conn(0x7f16c80fad30 msgr2=0x7f16c8069110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:00.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.542+0000 7f16cd1fe700 1 -- 172.21.15.27:0/733834676 shutdown_connections 2024-04-22T15:41:00.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:00.542+0000 7f16cd1fe700 1 -- 172.21.15.27:0/733834676 wait complete. 2024-04-22T15:41:00.546 INFO:teuthology.orchestra.run.smithi027.stderr:dumped monmap epoch 1 2024-04-22T15:41:01.256 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:01 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/733834676' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:41:02.235 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:41:02.236 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:41:02.394 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:02 smithi016 ceph-mon[26620]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:41:02.444 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:02.492 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:03.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.884+0000 7f31a9ad0700 1 -- 172.21.15.27:0/1421578379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a4101320 msgr2=0x7f31a4103710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:03.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.884+0000 7f31a9ad0700 1 --2- 172.21.15.27:0/1421578379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a4101320 0x7f31a4103710 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3194009910 tx=0x7f3194009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:03.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.884+0000 7f31a9ad0700 1 -- 172.21.15.27:0/1421578379 shutdown_connections 2024-04-22T15:41:03.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.884+0000 7f31a9ad0700 1 --2- 172.21.15.27:0/1421578379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a4101320 0x7f31a4103710 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:03.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.884+0000 7f31a9ad0700 1 -- 172.21.15.27:0/1421578379 >> 172.21.15.27:0/1421578379 conn(0x7f31a40fad30 msgr2=0x7f31a40fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:03.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.884+0000 7f31a9ad0700 1 -- 172.21.15.27:0/1421578379 shutdown_connections 2024-04-22T15:41:03.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.884+0000 7f31a9ad0700 1 -- 172.21.15.27:0/1421578379 wait complete. 2024-04-22T15:41:03.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.884+0000 7f31a9ad0700 1 Processor -- start 2024-04-22T15:41:03.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.885+0000 7f31a9ad0700 1 -- start start 2024-04-22T15:41:03.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.885+0000 7f31a9ad0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a4101320 0x7f31a4100c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:03.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.885+0000 7f31a9ad0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a41011c0 con 0x7f31a4101320 2024-04-22T15:41:03.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.885+0000 7f31a37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a4101320 0x7f31a4100c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:03.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.885+0000 7f31a37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a4101320 0x7f31a4100c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.27:42960/0 (socket says 172.21.15.27:42960) 2024-04-22T15:41:03.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.885+0000 7f31a37fe700 1 -- 172.21.15.27:0/3764949263 learned_addr learned my addr 172.21.15.27:0/3764949263 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-22T15:41:03.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.885+0000 7f31a37fe700 1 -- 172.21.15.27:0/3764949263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31940095f0 con 0x7f31a4101320 2024-04-22T15:41:03.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.885+0000 7f31a37fe700 1 --2- 172.21.15.27:0/3764949263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a4101320 0x7f31a4100c80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3194005b40 tx=0x7f3194004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:03.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.886+0000 7f31a0ff9700 1 -- 172.21.15.27:0/3764949263 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f319401d070 con 0x7f31a4101320 2024-04-22T15:41:03.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.886+0000 7f31a0ff9700 1 -- 172.21.15.27:0/3764949263 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f319400be10 con 0x7f31a4101320 2024-04-22T15:41:03.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.886+0000 7f31a9ad0700 1 -- 172.21.15.27:0/3764949263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a40ff2d0 con 0x7f31a4101320 2024-04-22T15:41:03.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.886+0000 7f31a0ff9700 1 -- 172.21.15.27:0/3764949263 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f319400f460 con 0x7f31a4101320 2024-04-22T15:41:03.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.886+0000 7f31a9ad0700 1 -- 172.21.15.27:0/3764949263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a40ff770 con 0x7f31a4101320 2024-04-22T15:41:03.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.887+0000 7f31a0ff9700 1 -- 172.21.15.27:0/3764949263 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f319400f6d0 con 0x7f31a4101320 2024-04-22T15:41:03.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.887+0000 7f31a9ad0700 1 -- 172.21.15.27:0/3764949263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a40623c0 con 0x7f31a4101320 2024-04-22T15:41:03.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.887+0000 7f31a0ff9700 1 --2- 172.21.15.27:0/3764949263 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f318c0383c0 0x7f318c03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:03.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.887+0000 7f31a0ff9700 1 -- 172.21.15.27:0/3764949263 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f319404c550 con 0x7f31a4101320 2024-04-22T15:41:03.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.887+0000 7f31a2ffd700 1 --2- 172.21.15.27:0/3764949263 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f318c0383c0 0x7f318c03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:03.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.887+0000 7f31a2ffd700 1 --2- 172.21.15.27:0/3764949263 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f318c0383c0 0x7f318c03a880 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3198006fd0 tx=0x7f3198006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:03.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:03.892+0000 7f31a0ff9700 1 -- 172.21.15.27:0/3764949263 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3194020ba0 con 0x7f31a4101320 2024-04-22T15:41:04.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:04.171+0000 7f31a9ad0700 1 -- 172.21.15.27:0/3764949263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f31a418ee90 con 0x7f31a4101320 2024-04-22T15:41:04.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:04.171+0000 7f31a0ff9700 1 -- 172.21.15.27:0/3764949263 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f3194027030 con 0x7f31a4101320 2024-04-22T15:41:04.174 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:41:04.174 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":1,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","modified":"2024-04-22T15:39:17.702487Z","created":"2024-04-22T15:39:17.702487Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:41:04.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:04.173+0000 7f31a9ad0700 1 -- 172.21.15.27:0/3764949263 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f318c0383c0 msgr2=0x7f318c03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:04.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:04.173+0000 7f31a9ad0700 1 --2- 172.21.15.27:0/3764949263 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f318c0383c0 0x7f318c03a880 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3198006fd0 tx=0x7f3198006e40 comp rx=0 tx=0).stop 2024-04-22T15:41:04.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:04.173+0000 7f31a9ad0700 1 -- 172.21.15.27:0/3764949263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a4101320 msgr2=0x7f31a4100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:04.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:04.173+0000 7f31a9ad0700 1 --2- 172.21.15.27:0/3764949263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a4101320 0x7f31a4100c80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3194005b40 tx=0x7f3194004740 comp rx=0 tx=0).stop 2024-04-22T15:41:04.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:04.173+0000 7f31a9ad0700 1 -- 172.21.15.27:0/3764949263 shutdown_connections 2024-04-22T15:41:04.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:04.173+0000 7f31a9ad0700 1 --2- 172.21.15.27:0/3764949263 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f318c0383c0 0x7f318c03a880 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:04.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:04.173+0000 7f31a9ad0700 1 --2- 172.21.15.27:0/3764949263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a4101320 0x7f31a4100c80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:04.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:04.173+0000 7f31a9ad0700 1 -- 172.21.15.27:0/3764949263 >> 172.21.15.27:0/3764949263 conn(0x7f31a40fad30 msgr2=0x7f31a40fb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:04.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:04.174+0000 7f31a9ad0700 1 -- 172.21.15.27:0/3764949263 shutdown_connections 2024-04-22T15:41:04.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:04.174+0000 7f31a9ad0700 1 -- 172.21.15.27:0/3764949263 wait complete. 2024-04-22T15:41:04.177 INFO:teuthology.orchestra.run.smithi027.stderr:dumped monmap epoch 1 2024-04-22T15:41:04.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:04 smithi016 ceph-mon[26620]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:41:04.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:04 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:04.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:04 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:04.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:04 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:04.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:04 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:04.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:04 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:04.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:04 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:04.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:04 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:04.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:04 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:04.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:04 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T15:41:04.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:04 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:04.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:04 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/3764949263' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:41:05.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:05 smithi016 ceph-mon[26620]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-22T15:41:05.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:05 smithi016 ceph-mon[26620]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T15:41:05.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:05 smithi016 ceph-mon[26620]: Deploying daemon grafana.smithi016 on smithi016 2024-04-22T15:41:05.740 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:41:05.740 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:41:05.949 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:05.998 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:06.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:06 smithi016 ceph-mon[26620]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:41:07.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.450+0000 7fa5aee20700 1 -- 172.21.15.27:0/1876095985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5a8101370 msgr2=0x7fa5a8103760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:07.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.450+0000 7fa5aee20700 1 --2- 172.21.15.27:0/1876095985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5a8101370 0x7fa5a8103760 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa59c009910 tx=0x7fa59c009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:07.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.451+0000 7fa5aee20700 1 -- 172.21.15.27:0/1876095985 shutdown_connections 2024-04-22T15:41:07.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.451+0000 7fa5aee20700 1 --2- 172.21.15.27:0/1876095985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5a8101370 0x7fa5a8103760 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:07.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.451+0000 7fa5aee20700 1 -- 172.21.15.27:0/1876095985 >> 172.21.15.27:0/1876095985 conn(0x7fa5a80fad20 msgr2=0x7fa5a80fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:07.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.451+0000 7fa5aee20700 1 -- 172.21.15.27:0/1876095985 shutdown_connections 2024-04-22T15:41:07.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.451+0000 7fa5aee20700 1 -- 172.21.15.27:0/1876095985 wait complete. 2024-04-22T15:41:07.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.451+0000 7fa5aee20700 1 Processor -- start 2024-04-22T15:41:07.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.451+0000 7fa5aee20700 1 -- start start 2024-04-22T15:41:07.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.451+0000 7fa5aee20700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5a8101370 0x7fa5a8195930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:07.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.451+0000 7fa5aee20700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a8195e70 con 0x7fa5a8101370 2024-04-22T15:41:07.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.451+0000 7fa5acbbc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5a8101370 0x7fa5a8195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:07.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.452+0000 7fa5acbbc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5a8101370 0x7fa5a8195930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.27:42974/0 (socket says 172.21.15.27:42974) 2024-04-22T15:41:07.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.452+0000 7fa5acbbc700 1 -- 172.21.15.27:0/3695475958 learned_addr learned my addr 172.21.15.27:0/3695475958 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-22T15:41:07.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.452+0000 7fa5acbbc700 1 -- 172.21.15.27:0/3695475958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa59c0095f0 con 0x7fa5a8101370 2024-04-22T15:41:07.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.452+0000 7fa5acbbc700 1 --2- 172.21.15.27:0/3695475958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5a8101370 0x7fa5a8195930 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa59c0092e0 tx=0x7fa59c009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:07.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.452+0000 7fa5a5ffb700 1 -- 172.21.15.27:0/3695475958 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa59c01d070 con 0x7fa5a8101370 2024-04-22T15:41:07.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.452+0000 7fa5aee20700 1 -- 172.21.15.27:0/3695475958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5a8196070 con 0x7fa5a8101370 2024-04-22T15:41:07.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.452+0000 7fa5a5ffb700 1 -- 172.21.15.27:0/3695475958 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa59c00be10 con 0x7fa5a8101370 2024-04-22T15:41:07.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.452+0000 7fa5a5ffb700 1 -- 172.21.15.27:0/3695475958 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa59c00f460 con 0x7fa5a8101370 2024-04-22T15:41:07.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.452+0000 7fa5aee20700 1 -- 172.21.15.27:0/3695475958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5a8196510 con 0x7fa5a8101370 2024-04-22T15:41:07.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.453+0000 7fa5a5ffb700 1 -- 172.21.15.27:0/3695475958 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7fa59c00f6d0 con 0x7fa5a8101370 2024-04-22T15:41:07.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.453+0000 7fa5a5ffb700 1 --2- 172.21.15.27:0/3695475958 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fa5980383c0 0x7fa59803a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:07.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.453+0000 7fa5aee20700 1 -- 172.21.15.27:0/3695475958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5a818f4a0 con 0x7fa5a8101370 2024-04-22T15:41:07.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.453+0000 7fa5a5ffb700 1 -- 172.21.15.27:0/3695475958 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa59c04c520 con 0x7fa5a8101370 2024-04-22T15:41:07.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.454+0000 7fa5a7fff700 1 --2- 172.21.15.27:0/3695475958 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fa5980383c0 0x7fa59803a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:07.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.454+0000 7fa5a7fff700 1 --2- 172.21.15.27:0/3695475958 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fa5980383c0 0x7fa59803a880 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa594006fd0 tx=0x7fa594006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:07.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.459+0000 7fa5a5ffb700 1 -- 172.21.15.27:0/3695475958 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa59c027030 con 0x7fa5a8101370 2024-04-22T15:41:07.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.736+0000 7fa5aee20700 1 -- 172.21.15.27:0/3695475958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5a802cc70 con 0x7fa5a8101370 2024-04-22T15:41:07.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.737+0000 7fa5a5ffb700 1 -- 172.21.15.27:0/3695475958 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fa59c020bd0 con 0x7fa5a8101370 2024-04-22T15:41:07.740 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:41:07.740 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":1,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","modified":"2024-04-22T15:39:17.702487Z","created":"2024-04-22T15:39:17.702487Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:41:07.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.739+0000 7fa5aee20700 1 -- 172.21.15.27:0/3695475958 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fa5980383c0 msgr2=0x7fa59803a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:07.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.739+0000 7fa5aee20700 1 --2- 172.21.15.27:0/3695475958 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fa5980383c0 0x7fa59803a880 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa594006fd0 tx=0x7fa594006e40 comp rx=0 tx=0).stop 2024-04-22T15:41:07.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.739+0000 7fa5aee20700 1 -- 172.21.15.27:0/3695475958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5a8101370 msgr2=0x7fa5a8195930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:07.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.739+0000 7fa5aee20700 1 --2- 172.21.15.27:0/3695475958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5a8101370 0x7fa5a8195930 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa59c0092e0 tx=0x7fa59c009e60 comp rx=0 tx=0).stop 2024-04-22T15:41:07.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.739+0000 7fa5aee20700 1 -- 172.21.15.27:0/3695475958 shutdown_connections 2024-04-22T15:41:07.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.739+0000 7fa5aee20700 1 --2- 172.21.15.27:0/3695475958 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fa5980383c0 0x7fa59803a880 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:07.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.739+0000 7fa5aee20700 1 --2- 172.21.15.27:0/3695475958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5a8101370 0x7fa5a8195930 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:07.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.739+0000 7fa5aee20700 1 -- 172.21.15.27:0/3695475958 >> 172.21.15.27:0/3695475958 conn(0x7fa5a80fad20 msgr2=0x7fa5a80fb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:07.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.739+0000 7fa5aee20700 1 -- 172.21.15.27:0/3695475958 shutdown_connections 2024-04-22T15:41:07.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:07.739+0000 7fa5aee20700 1 -- 172.21.15.27:0/3695475958 wait complete. 2024-04-22T15:41:07.743 INFO:teuthology.orchestra.run.smithi027.stderr:dumped monmap epoch 1 2024-04-22T15:41:08.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:08 smithi016 ceph-mon[26620]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:41:08.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:08 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:08.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:08 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/3695475958' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:41:09.263 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:41:09.263 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:41:09.460 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:09.513 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:10.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:10 smithi016 ceph-mon[26620]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:41:10.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.851+0000 7f602b8c0700 1 -- 172.21.15.27:0/3571547704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6024101310 msgr2=0x7f6024103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:10.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.852+0000 7f602b8c0700 1 --2- 172.21.15.27:0/3571547704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6024101310 0x7f6024103700 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6020009910 tx=0x7f6020009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:10.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.852+0000 7f602b8c0700 1 -- 172.21.15.27:0/3571547704 shutdown_connections 2024-04-22T15:41:10.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.852+0000 7f602b8c0700 1 --2- 172.21.15.27:0/3571547704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6024101310 0x7f6024103700 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:10.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.852+0000 7f602b8c0700 1 -- 172.21.15.27:0/3571547704 >> 172.21.15.27:0/3571547704 conn(0x7f60240fad20 msgr2=0x7f60240fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:10.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.852+0000 7f602b8c0700 1 -- 172.21.15.27:0/3571547704 shutdown_connections 2024-04-22T15:41:10.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.852+0000 7f602b8c0700 1 -- 172.21.15.27:0/3571547704 wait complete. 2024-04-22T15:41:10.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.852+0000 7f602b8c0700 1 Processor -- start 2024-04-22T15:41:10.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.852+0000 7f602b8c0700 1 -- start start 2024-04-22T15:41:10.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.852+0000 7f602b8c0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6024101310 0x7f6024195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:10.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.852+0000 7f602b8c0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6024195ec0 con 0x7f6024101310 2024-04-22T15:41:10.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.852+0000 7f602965c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6024101310 0x7f6024195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:10.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.853+0000 7f602965c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6024101310 0x7f6024195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.27:42978/0 (socket says 172.21.15.27:42978) 2024-04-22T15:41:10.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.853+0000 7f602965c700 1 -- 172.21.15.27:0/2227749399 learned_addr learned my addr 172.21.15.27:0/2227749399 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-22T15:41:10.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.853+0000 7f602965c700 1 -- 172.21.15.27:0/2227749399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60200095f0 con 0x7f6024101310 2024-04-22T15:41:10.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.853+0000 7f602965c700 1 --2- 172.21.15.27:0/2227749399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6024101310 0x7f6024195980 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f60200092e0 tx=0x7f6020009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:10.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.853+0000 7f601a7fc700 1 -- 172.21.15.27:0/2227749399 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f602001d070 con 0x7f6024101310 2024-04-22T15:41:10.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.853+0000 7f602b8c0700 1 -- 172.21.15.27:0/2227749399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60241960c0 con 0x7f6024101310 2024-04-22T15:41:10.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.853+0000 7f601a7fc700 1 -- 172.21.15.27:0/2227749399 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f602000be10 con 0x7f6024101310 2024-04-22T15:41:10.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.853+0000 7f601a7fc700 1 -- 172.21.15.27:0/2227749399 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f602000f460 con 0x7f6024101310 2024-04-22T15:41:10.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.853+0000 7f602b8c0700 1 -- 172.21.15.27:0/2227749399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6024196560 con 0x7f6024101310 2024-04-22T15:41:10.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.854+0000 7f601a7fc700 1 -- 172.21.15.27:0/2227749399 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f602000b960 con 0x7f6024101310 2024-04-22T15:41:10.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.854+0000 7f602b8c0700 1 -- 172.21.15.27:0/2227749399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f602418f4a0 con 0x7f6024101310 2024-04-22T15:41:10.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.854+0000 7f601a7fc700 1 --2- 172.21.15.27:0/2227749399 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f60100383c0 0x7f601003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:10.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.854+0000 7f601a7fc700 1 -- 172.21.15.27:0/2227749399 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f602004c570 con 0x7f6024101310 2024-04-22T15:41:10.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.854+0000 7f6028e5b700 1 --2- 172.21.15.27:0/2227749399 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f60100383c0 0x7f601003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:10.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.855+0000 7f6028e5b700 1 --2- 172.21.15.27:0/2227749399 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f60100383c0 0x7f601003a880 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6014006fd0 tx=0x7f6014006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:10.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:10.859+0000 7f601a7fc700 1 -- 172.21.15.27:0/2227749399 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6020005c00 con 0x7f6024101310 2024-04-22T15:41:11.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:11.139+0000 7f602b8c0700 1 -- 172.21.15.27:0/2227749399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f602402cc70 con 0x7f6024101310 2024-04-22T15:41:11.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:11.139+0000 7f601a7fc700 1 -- 172.21.15.27:0/2227749399 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f6020027070 con 0x7f6024101310 2024-04-22T15:41:11.142 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:41:11.143 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":1,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","modified":"2024-04-22T15:39:17.702487Z","created":"2024-04-22T15:39:17.702487Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:41:11.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:11.141+0000 7f602b8c0700 1 -- 172.21.15.27:0/2227749399 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f60100383c0 msgr2=0x7f601003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:11.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:11.141+0000 7f602b8c0700 1 --2- 172.21.15.27:0/2227749399 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f60100383c0 0x7f601003a880 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6014006fd0 tx=0x7f6014006e40 comp rx=0 tx=0).stop 2024-04-22T15:41:11.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:11.141+0000 7f602b8c0700 1 -- 172.21.15.27:0/2227749399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6024101310 msgr2=0x7f6024195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:11.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:11.141+0000 7f602b8c0700 1 --2- 172.21.15.27:0/2227749399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6024101310 0x7f6024195980 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f60200092e0 tx=0x7f6020009e60 comp rx=0 tx=0).stop 2024-04-22T15:41:11.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:11.141+0000 7f602b8c0700 1 -- 172.21.15.27:0/2227749399 shutdown_connections 2024-04-22T15:41:11.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:11.141+0000 7f602b8c0700 1 --2- 172.21.15.27:0/2227749399 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f60100383c0 0x7f601003a880 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:11.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:11.141+0000 7f602b8c0700 1 --2- 172.21.15.27:0/2227749399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6024101310 0x7f6024195980 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:11.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:11.141+0000 7f602b8c0700 1 -- 172.21.15.27:0/2227749399 >> 172.21.15.27:0/2227749399 conn(0x7f60240fad20 msgr2=0x7f60240fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:11.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:11.141+0000 7f602b8c0700 1 -- 172.21.15.27:0/2227749399 shutdown_connections 2024-04-22T15:41:11.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:11.141+0000 7f602b8c0700 1 -- 172.21.15.27:0/2227749399 wait complete. 2024-04-22T15:41:11.145 INFO:teuthology.orchestra.run.smithi027.stderr:dumped monmap epoch 1 2024-04-22T15:41:11.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:11 smithi016 ceph-mon[26620]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:41:11.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:11 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/2227749399' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:41:12.866 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:41:12.867 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:41:13.075 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:13.123 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:14.567 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:14 smithi016 ceph-mon[26620]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:41:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.579+0000 7fc112269700 1 -- 172.21.15.27:0/1527259937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc10c0ff780 msgr2=0x7fc10c0ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.579+0000 7fc112269700 1 --2- 172.21.15.27:0/1527259937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc10c0ff780 0x7fc10c0ffba0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc100009910 tx=0x7fc100009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.579+0000 7fc112269700 1 -- 172.21.15.27:0/1527259937 shutdown_connections 2024-04-22T15:41:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.579+0000 7fc112269700 1 --2- 172.21.15.27:0/1527259937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc10c0ff780 0x7fc10c0ffba0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.579+0000 7fc112269700 1 -- 172.21.15.27:0/1527259937 >> 172.21.15.27:0/1527259937 conn(0x7fc10c0fad20 msgr2=0x7fc10c0fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.579+0000 7fc112269700 1 -- 172.21.15.27:0/1527259937 shutdown_connections 2024-04-22T15:41:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.579+0000 7fc112269700 1 -- 172.21.15.27:0/1527259937 wait complete. 2024-04-22T15:41:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.580+0000 7fc112269700 1 Processor -- start 2024-04-22T15:41:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.580+0000 7fc112269700 1 -- start start 2024-04-22T15:41:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.580+0000 7fc112269700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc10c0fccd0 0x7fc10c0fb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.580+0000 7fc112269700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc10c0fd0f0 con 0x7fc10c0fccd0 2024-04-22T15:41:14.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.580+0000 7fc10bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc10c0fccd0 0x7fc10c0fb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:14.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.580+0000 7fc10bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc10c0fccd0 0x7fc10c0fb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.27:43218/0 (socket says 172.21.15.27:43218) 2024-04-22T15:41:14.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.580+0000 7fc10bfff700 1 -- 172.21.15.27:0/1085843294 learned_addr learned my addr 172.21.15.27:0/1085843294 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-22T15:41:14.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.581+0000 7fc10bfff700 1 -- 172.21.15.27:0/1085843294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1000095f0 con 0x7fc10c0fccd0 2024-04-22T15:41:14.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.581+0000 7fc10bfff700 1 --2- 172.21.15.27:0/1085843294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc10c0fccd0 0x7fc10c0fb360 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc10c1007e0 tx=0x7fc100009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:14.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.581+0000 7fc1097fa700 1 -- 172.21.15.27:0/1085843294 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc10001d070 con 0x7fc10c0fccd0 2024-04-22T15:41:14.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.581+0000 7fc112269700 1 -- 172.21.15.27:0/1085843294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc10c0fb8a0 con 0x7fc10c0fccd0 2024-04-22T15:41:14.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.581+0000 7fc1097fa700 1 -- 172.21.15.27:0/1085843294 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc10000b750 con 0x7fc10c0fccd0 2024-04-22T15:41:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.581+0000 7fc1097fa700 1 -- 172.21.15.27:0/1085843294 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1000038c0 con 0x7fc10c0fccd0 2024-04-22T15:41:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.581+0000 7fc112269700 1 -- 172.21.15.27:0/1085843294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc10c0fbd40 con 0x7fc10c0fccd0 2024-04-22T15:41:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.582+0000 7fc1097fa700 1 -- 172.21.15.27:0/1085843294 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7fc100003a40 con 0x7fc10c0fccd0 2024-04-22T15:41:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.582+0000 7fc112269700 1 -- 172.21.15.27:0/1085843294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc10c18f3b0 con 0x7fc10c0fccd0 2024-04-22T15:41:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.582+0000 7fc1097fa700 1 --2- 172.21.15.27:0/1085843294 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fc0f40383c0 0x7fc0f403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.582+0000 7fc1097fa700 1 -- 172.21.15.27:0/1085843294 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc10004bf20 con 0x7fc10c0fccd0 2024-04-22T15:41:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.583+0000 7fc10b7fe700 1 --2- 172.21.15.27:0/1085843294 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fc0f40383c0 0x7fc0f403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.583+0000 7fc10b7fe700 1 --2- 172.21.15.27:0/1085843294 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fc0f40383c0 0x7fc0f403a880 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc006fd0 tx=0x7fc0fc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:14.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.588+0000 7fc1097fa700 1 -- 172.21.15.27:0/1085843294 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc10000b8c0 con 0x7fc10c0fccd0 2024-04-22T15:41:14.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.866+0000 7fc112269700 1 -- 172.21.15.27:0/1085843294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc10c105670 con 0x7fc10c0fccd0 2024-04-22T15:41:14.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.867+0000 7fc1097fa700 1 -- 172.21.15.27:0/1085843294 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fc100026020 con 0x7fc10c0fccd0 2024-04-22T15:41:14.870 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:41:14.871 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":1,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","modified":"2024-04-22T15:39:17.702487Z","created":"2024-04-22T15:39:17.702487Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:41:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.869+0000 7fc112269700 1 -- 172.21.15.27:0/1085843294 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fc0f40383c0 msgr2=0x7fc0f403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.869+0000 7fc112269700 1 --2- 172.21.15.27:0/1085843294 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fc0f40383c0 0x7fc0f403a880 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc006fd0 tx=0x7fc0fc006e40 comp rx=0 tx=0).stop 2024-04-22T15:41:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.869+0000 7fc112269700 1 -- 172.21.15.27:0/1085843294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc10c0fccd0 msgr2=0x7fc10c0fb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.869+0000 7fc112269700 1 --2- 172.21.15.27:0/1085843294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc10c0fccd0 0x7fc10c0fb360 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc10c1007e0 tx=0x7fc100009e60 comp rx=0 tx=0).stop 2024-04-22T15:41:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.869+0000 7fc112269700 1 -- 172.21.15.27:0/1085843294 shutdown_connections 2024-04-22T15:41:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.869+0000 7fc112269700 1 --2- 172.21.15.27:0/1085843294 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fc0f40383c0 0x7fc0f403a880 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.869+0000 7fc112269700 1 --2- 172.21.15.27:0/1085843294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc10c0fccd0 0x7fc10c0fb360 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.869+0000 7fc112269700 1 -- 172.21.15.27:0/1085843294 >> 172.21.15.27:0/1085843294 conn(0x7fc10c0fad20 msgr2=0x7fc10c068820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.869+0000 7fc112269700 1 -- 172.21.15.27:0/1085843294 shutdown_connections 2024-04-22T15:41:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:14.869+0000 7fc112269700 1 -- 172.21.15.27:0/1085843294 wait complete. 2024-04-22T15:41:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:dumped monmap epoch 1 2024-04-22T15:41:15.577 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:15 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/1085843294' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:41:16.311 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:41:16.311 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:41:16.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:16 smithi016 ceph-mon[26620]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:41:16.521 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:17.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.942+0000 7f58ff8e5700 1 -- 172.21.15.27:0/544392347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58f80ff790 msgr2=0x7f58f80ffbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:17.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.942+0000 7f58ff8e5700 1 --2- 172.21.15.27:0/544392347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58f80ff790 0x7f58f80ffbb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f58e8009910 tx=0x7f58e8009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:17.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.942+0000 7f58ff8e5700 1 -- 172.21.15.27:0/544392347 shutdown_connections 2024-04-22T15:41:17.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.942+0000 7f58ff8e5700 1 --2- 172.21.15.27:0/544392347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58f80ff790 0x7f58f80ffbb0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:17.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.942+0000 7f58ff8e5700 1 -- 172.21.15.27:0/544392347 >> 172.21.15.27:0/544392347 conn(0x7f58f80fad30 msgr2=0x7f58f80fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:17.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.942+0000 7f58ff8e5700 1 -- 172.21.15.27:0/544392347 shutdown_connections 2024-04-22T15:41:17.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.942+0000 7f58ff8e5700 1 -- 172.21.15.27:0/544392347 wait complete. 2024-04-22T15:41:17.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.942+0000 7f58ff8e5700 1 Processor -- start 2024-04-22T15:41:17.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.943+0000 7f58ff8e5700 1 -- start start 2024-04-22T15:41:17.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.943+0000 7f58ff8e5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58f80ff790 0x7f58f8069400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:17.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.943+0000 7f58ff8e5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f81986d0 con 0x7f58f80ff790 2024-04-22T15:41:17.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.943+0000 7f58fd681700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58f80ff790 0x7f58f8069400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:17.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.943+0000 7f58fd681700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58f80ff790 0x7f58f8069400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.27:43240/0 (socket says 172.21.15.27:43240) 2024-04-22T15:41:17.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.943+0000 7f58fd681700 1 -- 172.21.15.27:0/2147140078 learned_addr learned my addr 172.21.15.27:0/2147140078 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-22T15:41:17.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.943+0000 7f58fd681700 1 -- 172.21.15.27:0/2147140078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e80095f0 con 0x7f58f80ff790 2024-04-22T15:41:17.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.944+0000 7f58fd681700 1 --2- 172.21.15.27:0/2147140078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58f80ff790 0x7f58f8069400 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f58e8005950 tx=0x7f58e8009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:17.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.944+0000 7f58ee7fc700 1 -- 172.21.15.27:0/2147140078 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58e801d070 con 0x7f58f80ff790 2024-04-22T15:41:17.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.944+0000 7f58ff8e5700 1 -- 172.21.15.27:0/2147140078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f8198810 con 0x7f58f80ff790 2024-04-22T15:41:17.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.944+0000 7f58ee7fc700 1 -- 172.21.15.27:0/2147140078 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f58e800be10 con 0x7f58f80ff790 2024-04-22T15:41:17.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.944+0000 7f58ee7fc700 1 -- 172.21.15.27:0/2147140078 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58e800f460 con 0x7f58f80ff790 2024-04-22T15:41:17.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.944+0000 7f58ff8e5700 1 -- 172.21.15.27:0/2147140078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f8198cb0 con 0x7f58f80ff790 2024-04-22T15:41:17.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.945+0000 7f58ee7fc700 1 -- 172.21.15.27:0/2147140078 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f58e800b960 con 0x7f58f80ff790 2024-04-22T15:41:17.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.945+0000 7f58ff8e5700 1 -- 172.21.15.27:0/2147140078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58f818e3b0 con 0x7f58f80ff790 2024-04-22T15:41:17.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.945+0000 7f58ee7fc700 1 --2- 172.21.15.27:0/2147140078 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f58e40383c0 0x7f58e403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:17.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.945+0000 7f58ee7fc700 1 -- 172.21.15.27:0/2147140078 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f58e804c4f0 con 0x7f58f80ff790 2024-04-22T15:41:17.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.945+0000 7f58fce80700 1 --2- 172.21.15.27:0/2147140078 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f58e40383c0 0x7f58e403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:17.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.946+0000 7f58fce80700 1 --2- 172.21.15.27:0/2147140078 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f58e40383c0 0x7f58e403a880 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f58f4006fd0 tx=0x7f58f4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:17.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:17.951+0000 7f58ee7fc700 1 -- 172.21.15.27:0/2147140078 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f58e802bd00 con 0x7f58f80ff790 2024-04-22T15:41:18.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:18.230+0000 7f58ff8e5700 1 -- 172.21.15.27:0/2147140078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f58f802cc70 con 0x7f58f80ff790 2024-04-22T15:41:18.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:18.231+0000 7f58ee7fc700 1 -- 172.21.15.27:0/2147140078 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f58e8027030 con 0x7f58f80ff790 2024-04-22T15:41:18.234 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:41:18.234 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":1,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","modified":"2024-04-22T15:39:17.702487Z","created":"2024-04-22T15:39:17.702487Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:41:18.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:18.233+0000 7f58ff8e5700 1 -- 172.21.15.27:0/2147140078 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f58e40383c0 msgr2=0x7f58e403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:18.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:18.233+0000 7f58ff8e5700 1 --2- 172.21.15.27:0/2147140078 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f58e40383c0 0x7f58e403a880 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f58f4006fd0 tx=0x7f58f4006e40 comp rx=0 tx=0).stop 2024-04-22T15:41:18.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:18.233+0000 7f58ff8e5700 1 -- 172.21.15.27:0/2147140078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58f80ff790 msgr2=0x7f58f8069400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:18.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:18.233+0000 7f58ff8e5700 1 --2- 172.21.15.27:0/2147140078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58f80ff790 0x7f58f8069400 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f58e8005950 tx=0x7f58e8009e60 comp rx=0 tx=0).stop 2024-04-22T15:41:18.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:18.233+0000 7f58ff8e5700 1 -- 172.21.15.27:0/2147140078 shutdown_connections 2024-04-22T15:41:18.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:18.233+0000 7f58ff8e5700 1 --2- 172.21.15.27:0/2147140078 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f58e40383c0 0x7f58e403a880 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:18.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:18.233+0000 7f58ff8e5700 1 --2- 172.21.15.27:0/2147140078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58f80ff790 0x7f58f8069400 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:18.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:18.233+0000 7f58ff8e5700 1 -- 172.21.15.27:0/2147140078 >> 172.21.15.27:0/2147140078 conn(0x7f58f80fad30 msgr2=0x7f58f80fb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:18.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:18.233+0000 7f58ff8e5700 1 -- 172.21.15.27:0/2147140078 shutdown_connections 2024-04-22T15:41:18.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:18.233+0000 7f58ff8e5700 1 -- 172.21.15.27:0/2147140078 wait complete. 2024-04-22T15:41:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:dumped monmap epoch 1 2024-04-22T15:41:18.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:18 smithi016 ceph-mon[26620]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:41:19.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:19 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/2147140078' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:41:19.834 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:41:19.835 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:41:20.041 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:20.090 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:20.404 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:20 smithi016 ceph-mon[26620]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:41:21.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.359+0000 7f84bc875700 1 -- 172.21.15.27:0/4250911681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84b81008a0 msgr2=0x7f84b8100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:21.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.359+0000 7f84bc875700 1 --2- 172.21.15.27:0/4250911681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84b81008a0 0x7f84b8100cc0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f84ac009910 tx=0x7f84ac009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:21.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.359+0000 7f84bc875700 1 -- 172.21.15.27:0/4250911681 shutdown_connections 2024-04-22T15:41:21.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.359+0000 7f84bc875700 1 --2- 172.21.15.27:0/4250911681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84b81008a0 0x7f84b8100cc0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:21.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.359+0000 7f84bc875700 1 -- 172.21.15.27:0/4250911681 >> 172.21.15.27:0/4250911681 conn(0x7f84b80fbe20 msgr2=0x7f84b80fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:21.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.359+0000 7f84bc875700 1 -- 172.21.15.27:0/4250911681 shutdown_connections 2024-04-22T15:41:21.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.359+0000 7f84bc875700 1 -- 172.21.15.27:0/4250911681 wait complete. 2024-04-22T15:41:21.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.360+0000 7f84bc875700 1 Processor -- start 2024-04-22T15:41:21.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.360+0000 7f84bc875700 1 -- start start 2024-04-22T15:41:21.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.360+0000 7f84bc875700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84b81008a0 0x7f84b81959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:21.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.360+0000 7f84bc875700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b8195ee0 con 0x7f84b81008a0 2024-04-22T15:41:21.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.360+0000 7f84b659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84b81008a0 0x7f84b81959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:21.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.360+0000 7f84b659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84b81008a0 0x7f84b81959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.27:43256/0 (socket says 172.21.15.27:43256) 2024-04-22T15:41:21.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.360+0000 7f84b659c700 1 -- 172.21.15.27:0/3682903652 learned_addr learned my addr 172.21.15.27:0/3682903652 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-22T15:41:21.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.360+0000 7f84b659c700 1 -- 172.21.15.27:0/3682903652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84ac0095f0 con 0x7f84b81008a0 2024-04-22T15:41:21.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.361+0000 7f84b659c700 1 --2- 172.21.15.27:0/3682903652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84b81008a0 0x7f84b81959a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f84ac005950 tx=0x7f84ac009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:21.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.361+0000 7f84a77fe700 1 -- 172.21.15.27:0/3682903652 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84ac01d070 con 0x7f84b81008a0 2024-04-22T15:41:21.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.361+0000 7f84bc875700 1 -- 172.21.15.27:0/3682903652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b81960e0 con 0x7f84b81008a0 2024-04-22T15:41:21.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.361+0000 7f84a77fe700 1 -- 172.21.15.27:0/3682903652 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f84ac00be10 con 0x7f84b81008a0 2024-04-22T15:41:21.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.361+0000 7f84a77fe700 1 -- 172.21.15.27:0/3682903652 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84ac00f460 con 0x7f84b81008a0 2024-04-22T15:41:21.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.361+0000 7f84bc875700 1 -- 172.21.15.27:0/3682903652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b8196580 con 0x7f84b81008a0 2024-04-22T15:41:21.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.362+0000 7f84a77fe700 1 -- 172.21.15.27:0/3682903652 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f84ac0227c0 con 0x7f84b81008a0 2024-04-22T15:41:21.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.362+0000 7f84bc875700 1 -- 172.21.15.27:0/3682903652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84b818f4c0 con 0x7f84b81008a0 2024-04-22T15:41:21.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.362+0000 7f84a77fe700 1 --2- 172.21.15.27:0/3682903652 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f84a00383c0 0x7f84a003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:21.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.362+0000 7f84a77fe700 1 -- 172.21.15.27:0/3682903652 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f84ac04c400 con 0x7f84b81008a0 2024-04-22T15:41:21.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.362+0000 7f84b5d9b700 1 --2- 172.21.15.27:0/3682903652 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f84a00383c0 0x7f84a003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:21.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.363+0000 7f84b5d9b700 1 --2- 172.21.15.27:0/3682903652 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f84a00383c0 0x7f84a003a880 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f84a8006fd0 tx=0x7f84a8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:21.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.368+0000 7f84a77fe700 1 -- 172.21.15.27:0/3682903652 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f84ac02bba0 con 0x7f84b81008a0 2024-04-22T15:41:21.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.651+0000 7f84bc875700 1 -- 172.21.15.27:0/3682903652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84b802cc70 con 0x7f84b81008a0 2024-04-22T15:41:21.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.651+0000 7f84a77fe700 1 -- 172.21.15.27:0/3682903652 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f84ac027070 con 0x7f84b81008a0 2024-04-22T15:41:21.655 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:41:21.655 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":1,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","modified":"2024-04-22T15:39:17.702487Z","created":"2024-04-22T15:39:17.702487Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:41:21.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.653+0000 7f84bc875700 1 -- 172.21.15.27:0/3682903652 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f84a00383c0 msgr2=0x7f84a003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:21.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.653+0000 7f84bc875700 1 --2- 172.21.15.27:0/3682903652 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f84a00383c0 0x7f84a003a880 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f84a8006fd0 tx=0x7f84a8006e40 comp rx=0 tx=0).stop 2024-04-22T15:41:21.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.653+0000 7f84bc875700 1 -- 172.21.15.27:0/3682903652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84b81008a0 msgr2=0x7f84b81959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:21.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.653+0000 7f84bc875700 1 --2- 172.21.15.27:0/3682903652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84b81008a0 0x7f84b81959a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f84ac005950 tx=0x7f84ac009e60 comp rx=0 tx=0).stop 2024-04-22T15:41:21.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.653+0000 7f84bc875700 1 -- 172.21.15.27:0/3682903652 shutdown_connections 2024-04-22T15:41:21.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.653+0000 7f84bc875700 1 --2- 172.21.15.27:0/3682903652 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f84a00383c0 0x7f84a003a880 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:21.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.653+0000 7f84bc875700 1 --2- 172.21.15.27:0/3682903652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84b81008a0 0x7f84b81959a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:21.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.653+0000 7f84bc875700 1 -- 172.21.15.27:0/3682903652 >> 172.21.15.27:0/3682903652 conn(0x7f84b80fbe20 msgr2=0x7f84b80fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:21.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.653+0000 7f84bc875700 1 -- 172.21.15.27:0/3682903652 shutdown_connections 2024-04-22T15:41:21.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:21.653+0000 7f84bc875700 1 -- 172.21.15.27:0/3682903652 wait complete. 2024-04-22T15:41:21.658 INFO:teuthology.orchestra.run.smithi027.stderr:dumped monmap epoch 1 2024-04-22T15:41:22.260 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:21 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:22.260 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:21 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:22.260 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:21 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:22.260 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:21 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:22.260 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:21 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:22.260 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:21 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:22.260 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:21 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:22.260 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:21 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:22.260 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:21 smithi016 ceph-mon[26620]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:41:22.260 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:21 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/3682903652' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:41:23.199 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:41:23.200 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:41:23.411 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:23.458 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:23.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:23 smithi016 ceph-mon[26620]: Deploying daemon prometheus.smithi016 on smithi016 2024-04-22T15:41:23.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:23 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:24.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:24 smithi016 ceph-mon[26620]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:41:24.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.905+0000 7fb31e7a3700 1 -- 172.21.15.27:0/3037688774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3181013b0 msgr2=0x7fb3181037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:24.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.905+0000 7fb31e7a3700 1 --2- 172.21.15.27:0/3037688774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3181013b0 0x7fb3181037a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb308009910 tx=0x7fb308009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:24.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.905+0000 7fb31e7a3700 1 -- 172.21.15.27:0/3037688774 shutdown_connections 2024-04-22T15:41:24.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.905+0000 7fb31e7a3700 1 --2- 172.21.15.27:0/3037688774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3181013b0 0x7fb3181037a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:24.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.905+0000 7fb31e7a3700 1 -- 172.21.15.27:0/3037688774 >> 172.21.15.27:0/3037688774 conn(0x7fb3180fad20 msgr2=0x7fb3180fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:24.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.906+0000 7fb31e7a3700 1 -- 172.21.15.27:0/3037688774 shutdown_connections 2024-04-22T15:41:24.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.906+0000 7fb31e7a3700 1 -- 172.21.15.27:0/3037688774 wait complete. 2024-04-22T15:41:24.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.906+0000 7fb31e7a3700 1 Processor -- start 2024-04-22T15:41:24.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.906+0000 7fb31e7a3700 1 -- start start 2024-04-22T15:41:24.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.906+0000 7fb31e7a3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3181013b0 0x7fb3181958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:24.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.906+0000 7fb31e7a3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb318195e20 con 0x7fb3181013b0 2024-04-22T15:41:24.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.907+0000 7fb317fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3181013b0 0x7fb3181958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:24.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.907+0000 7fb317fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3181013b0 0x7fb3181958e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.27:56776/0 (socket says 172.21.15.27:56776) 2024-04-22T15:41:24.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.907+0000 7fb317fff700 1 -- 172.21.15.27:0/1014526346 learned_addr learned my addr 172.21.15.27:0/1014526346 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-22T15:41:24.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.907+0000 7fb317fff700 1 -- 172.21.15.27:0/1014526346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3080095f0 con 0x7fb3181013b0 2024-04-22T15:41:24.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.907+0000 7fb317fff700 1 --2- 172.21.15.27:0/1014526346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3181013b0 0x7fb3181958e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb308005b40 tx=0x7fb308004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:24.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.907+0000 7fb3157fa700 1 -- 172.21.15.27:0/1014526346 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb30801d070 con 0x7fb3181013b0 2024-04-22T15:41:24.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.907+0000 7fb3157fa700 1 -- 172.21.15.27:0/1014526346 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb30800be10 con 0x7fb3181013b0 2024-04-22T15:41:24.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.907+0000 7fb31e7a3700 1 -- 172.21.15.27:0/1014526346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb318196020 con 0x7fb3181013b0 2024-04-22T15:41:24.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.907+0000 7fb3157fa700 1 -- 172.21.15.27:0/1014526346 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb30800f460 con 0x7fb3181013b0 2024-04-22T15:41:24.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.907+0000 7fb31e7a3700 1 -- 172.21.15.27:0/1014526346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3181964c0 con 0x7fb3181013b0 2024-04-22T15:41:24.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.908+0000 7fb3157fa700 1 -- 172.21.15.27:0/1014526346 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7fb30800b960 con 0x7fb3181013b0 2024-04-22T15:41:24.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.908+0000 7fb31e7a3700 1 -- 172.21.15.27:0/1014526346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb31818f4a0 con 0x7fb3181013b0 2024-04-22T15:41:24.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.908+0000 7fb3157fa700 1 --2- 172.21.15.27:0/1014526346 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fb3040383c0 0x7fb30403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:24.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.909+0000 7fb3157fa700 1 -- 172.21.15.27:0/1014526346 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb30804c3f0 con 0x7fb3181013b0 2024-04-22T15:41:24.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.909+0000 7fb3177fe700 1 --2- 172.21.15.27:0/1014526346 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fb3040383c0 0x7fb30403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:24.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.909+0000 7fb3177fe700 1 --2- 172.21.15.27:0/1014526346 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fb3040383c0 0x7fb30403a880 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb300006fd0 tx=0x7fb300006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:24.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:24.913+0000 7fb3157fa700 1 -- 172.21.15.27:0/1014526346 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb30802a360 con 0x7fb3181013b0 2024-04-22T15:41:25.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:25.192+0000 7fb31e7a3700 1 -- 172.21.15.27:0/1014526346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb31802cc70 con 0x7fb3181013b0 2024-04-22T15:41:25.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:25.192+0000 7fb3157fa700 1 -- 172.21.15.27:0/1014526346 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fb308027030 con 0x7fb3181013b0 2024-04-22T15:41:25.196 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:41:25.196 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":1,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","modified":"2024-04-22T15:39:17.702487Z","created":"2024-04-22T15:39:17.702487Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:41:25.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:25.194+0000 7fb31e7a3700 1 -- 172.21.15.27:0/1014526346 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fb3040383c0 msgr2=0x7fb30403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:25.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:25.194+0000 7fb31e7a3700 1 --2- 172.21.15.27:0/1014526346 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fb3040383c0 0x7fb30403a880 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb300006fd0 tx=0x7fb300006e40 comp rx=0 tx=0).stop 2024-04-22T15:41:25.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:25.194+0000 7fb31e7a3700 1 -- 172.21.15.27:0/1014526346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3181013b0 msgr2=0x7fb3181958e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:25.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:25.194+0000 7fb31e7a3700 1 --2- 172.21.15.27:0/1014526346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3181013b0 0x7fb3181958e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb308005b40 tx=0x7fb308004740 comp rx=0 tx=0).stop 2024-04-22T15:41:25.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:25.194+0000 7fb31e7a3700 1 -- 172.21.15.27:0/1014526346 shutdown_connections 2024-04-22T15:41:25.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:25.194+0000 7fb31e7a3700 1 --2- 172.21.15.27:0/1014526346 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fb3040383c0 0x7fb30403a880 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:25.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:25.194+0000 7fb31e7a3700 1 --2- 172.21.15.27:0/1014526346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3181013b0 0x7fb3181958e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:25.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:25.194+0000 7fb31e7a3700 1 -- 172.21.15.27:0/1014526346 >> 172.21.15.27:0/1014526346 conn(0x7fb3180fad20 msgr2=0x7fb3180fb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:25.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:25.194+0000 7fb31e7a3700 1 -- 172.21.15.27:0/1014526346 shutdown_connections 2024-04-22T15:41:25.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:25.194+0000 7fb31e7a3700 1 -- 172.21.15.27:0/1014526346 wait complete. 2024-04-22T15:41:25.199 INFO:teuthology.orchestra.run.smithi027.stderr:dumped monmap epoch 1 2024-04-22T15:41:26.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:26 smithi016 ceph-mon[26620]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:41:26.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:26 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/1014526346' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:41:26.764 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:41:26.765 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:41:26.974 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:27.032 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:28.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.475+0000 7f1547c98700 1 -- 172.21.15.27:0/1455827982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1540100d20 msgr2=0x7f1540101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:28.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.475+0000 7f1547c98700 1 --2- 172.21.15.27:0/1455827982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1540100d20 0x7f1540101140 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f153c009910 tx=0x7f153c009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:28.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.475+0000 7f1547c98700 1 -- 172.21.15.27:0/1455827982 shutdown_connections 2024-04-22T15:41:28.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.475+0000 7f1547c98700 1 --2- 172.21.15.27:0/1455827982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1540100d20 0x7f1540101140 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.475+0000 7f1547c98700 1 -- 172.21.15.27:0/1455827982 >> 172.21.15.27:0/1455827982 conn(0x7f15400fa4b0 msgr2=0x7f15400fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.475+0000 7f1547c98700 1 -- 172.21.15.27:0/1455827982 shutdown_connections 2024-04-22T15:41:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.475+0000 7f1547c98700 1 -- 172.21.15.27:0/1455827982 wait complete. 2024-04-22T15:41:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.476+0000 7f1547c98700 1 Processor -- start 2024-04-22T15:41:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.476+0000 7f1547c98700 1 -- start start 2024-04-22T15:41:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.476+0000 7f1547c98700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1540100d20 0x7f15400fc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.476+0000 7f1547c98700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15400fc990 con 0x7f1540100d20 2024-04-22T15:41:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.476+0000 7f1545a34700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1540100d20 0x7f15400fc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.476+0000 7f1545a34700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1540100d20 0x7f15400fc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.27:56786/0 (socket says 172.21.15.27:56786) 2024-04-22T15:41:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.476+0000 7f1545a34700 1 -- 172.21.15.27:0/3910955398 learned_addr learned my addr 172.21.15.27:0/3910955398 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-22T15:41:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.476+0000 7f1545a34700 1 -- 172.21.15.27:0/3910955398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153c0095f0 con 0x7f1540100d20 2024-04-22T15:41:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.477+0000 7f1545a34700 1 --2- 172.21.15.27:0/3910955398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1540100d20 0x7f15400fc450 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f153c005b40 tx=0x7f153c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.477+0000 7f1536ffd700 1 -- 172.21.15.27:0/3910955398 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f153c01d070 con 0x7f1540100d20 2024-04-22T15:41:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.477+0000 7f1536ffd700 1 -- 172.21.15.27:0/3910955398 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f153c00be10 con 0x7f1540100d20 2024-04-22T15:41:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.477+0000 7f1547c98700 1 -- 172.21.15.27:0/3910955398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15400faaa0 con 0x7f1540100d20 2024-04-22T15:41:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.477+0000 7f1536ffd700 1 -- 172.21.15.27:0/3910955398 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f153c00f460 con 0x7f1540100d20 2024-04-22T15:41:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.477+0000 7f1547c98700 1 -- 172.21.15.27:0/3910955398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15400faf40 con 0x7f1540100d20 2024-04-22T15:41:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.478+0000 7f1536ffd700 1 -- 172.21.15.27:0/3910955398 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f153c00b960 con 0x7f1540100d20 2024-04-22T15:41:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.478+0000 7f1547c98700 1 -- 172.21.15.27:0/3910955398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15400623c0 con 0x7f1540100d20 2024-04-22T15:41:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.478+0000 7f1536ffd700 1 --2- 172.21.15.27:0/3910955398 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f152c0383c0 0x7f152c03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.478+0000 7f1536ffd700 1 -- 172.21.15.27:0/3910955398 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f153c04c480 con 0x7f1540100d20 2024-04-22T15:41:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.478+0000 7f1545233700 1 --2- 172.21.15.27:0/3910955398 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f152c0383c0 0x7f152c03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.478+0000 7f1545233700 1 --2- 172.21.15.27:0/3910955398 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f152c0383c0 0x7f152c03a880 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1530006fd0 tx=0x7f1530006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:28.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.483+0000 7f1536ffd700 1 -- 172.21.15.27:0/3910955398 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f153c02a360 con 0x7f1540100d20 2024-04-22T15:41:28.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:28 smithi016 ceph-mon[26620]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:41:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.759+0000 7f1547c98700 1 -- 172.21.15.27:0/3910955398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f154018eda0 con 0x7f1540100d20 2024-04-22T15:41:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.760+0000 7f1536ffd700 1 -- 172.21.15.27:0/3910955398 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f153c027030 con 0x7f1540100d20 2024-04-22T15:41:28.763 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:41:28.764 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":1,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","modified":"2024-04-22T15:39:17.702487Z","created":"2024-04-22T15:39:17.702487Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:41:28.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.762+0000 7f1547c98700 1 -- 172.21.15.27:0/3910955398 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f152c0383c0 msgr2=0x7f152c03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:28.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.762+0000 7f1547c98700 1 --2- 172.21.15.27:0/3910955398 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f152c0383c0 0x7f152c03a880 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1530006fd0 tx=0x7f1530006e40 comp rx=0 tx=0).stop 2024-04-22T15:41:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.762+0000 7f1547c98700 1 -- 172.21.15.27:0/3910955398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1540100d20 msgr2=0x7f15400fc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.762+0000 7f1547c98700 1 --2- 172.21.15.27:0/3910955398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1540100d20 0x7f15400fc450 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f153c005b40 tx=0x7f153c004740 comp rx=0 tx=0).stop 2024-04-22T15:41:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.762+0000 7f1547c98700 1 -- 172.21.15.27:0/3910955398 shutdown_connections 2024-04-22T15:41:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.762+0000 7f1547c98700 1 --2- 172.21.15.27:0/3910955398 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f152c0383c0 0x7f152c03a880 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.762+0000 7f1547c98700 1 --2- 172.21.15.27:0/3910955398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1540100d20 0x7f15400fc450 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.762+0000 7f1547c98700 1 -- 172.21.15.27:0/3910955398 >> 172.21.15.27:0/3910955398 conn(0x7f15400fa4b0 msgr2=0x7f15400ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.762+0000 7f1547c98700 1 -- 172.21.15.27:0/3910955398 shutdown_connections 2024-04-22T15:41:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:28.762+0000 7f1547c98700 1 -- 172.21.15.27:0/3910955398 wait complete. 2024-04-22T15:41:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:dumped monmap epoch 1 2024-04-22T15:41:29.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:29 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/3910955398' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:41:30.378 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:30 smithi016 ceph-mon[26620]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:41:30.392 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:41:30.392 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:41:30.602 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:30.650 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:31.792 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:31 smithi016 ceph-mon[26620]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:41:32.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.076+0000 7ff9edd0e700 1 -- 172.21.15.27:0/3135933067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e81008a0 msgr2=0x7ff9e8100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:32.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.076+0000 7ff9edd0e700 1 --2- 172.21.15.27:0/3135933067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e81008a0 0x7ff9e8100cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0009910 tx=0x7ff9e0009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:32.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.076+0000 7ff9edd0e700 1 -- 172.21.15.27:0/3135933067 shutdown_connections 2024-04-22T15:41:32.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.076+0000 7ff9edd0e700 1 --2- 172.21.15.27:0/3135933067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e81008a0 0x7ff9e8100cc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:32.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.076+0000 7ff9edd0e700 1 -- 172.21.15.27:0/3135933067 >> 172.21.15.27:0/3135933067 conn(0x7ff9e80fbe20 msgr2=0x7ff9e80fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:32.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.076+0000 7ff9edd0e700 1 -- 172.21.15.27:0/3135933067 shutdown_connections 2024-04-22T15:41:32.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.076+0000 7ff9edd0e700 1 -- 172.21.15.27:0/3135933067 wait complete. 2024-04-22T15:41:32.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.077+0000 7ff9edd0e700 1 Processor -- start 2024-04-22T15:41:32.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.077+0000 7ff9edd0e700 1 -- start start 2024-04-22T15:41:32.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.077+0000 7ff9edd0e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e81008a0 0x7ff9e8195930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:32.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.077+0000 7ff9edd0e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e8195e70 con 0x7ff9e81008a0 2024-04-22T15:41:32.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.077+0000 7ff9e77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e81008a0 0x7ff9e8195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:32.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.077+0000 7ff9e77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e81008a0 0x7ff9e8195930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.27:56812/0 (socket says 172.21.15.27:56812) 2024-04-22T15:41:32.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.077+0000 7ff9e77fe700 1 -- 172.21.15.27:0/4212038092 learned_addr learned my addr 172.21.15.27:0/4212038092 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-22T15:41:32.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.078+0000 7ff9e77fe700 1 -- 172.21.15.27:0/4212038092 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e00095f0 con 0x7ff9e81008a0 2024-04-22T15:41:32.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.078+0000 7ff9e77fe700 1 --2- 172.21.15.27:0/4212038092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e81008a0 0x7ff9e8195930 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0005950 tx=0x7ff9e0009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:32.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.078+0000 7ff9e4ff9700 1 -- 172.21.15.27:0/4212038092 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9e001d070 con 0x7ff9e81008a0 2024-04-22T15:41:32.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.078+0000 7ff9e4ff9700 1 -- 172.21.15.27:0/4212038092 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff9e000be10 con 0x7ff9e81008a0 2024-04-22T15:41:32.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.078+0000 7ff9edd0e700 1 -- 172.21.15.27:0/4212038092 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e8196070 con 0x7ff9e81008a0 2024-04-22T15:41:32.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.078+0000 7ff9e4ff9700 1 -- 172.21.15.27:0/4212038092 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9e000f460 con 0x7ff9e81008a0 2024-04-22T15:41:32.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.078+0000 7ff9edd0e700 1 -- 172.21.15.27:0/4212038092 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e8196510 con 0x7ff9e81008a0 2024-04-22T15:41:32.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.079+0000 7ff9e4ff9700 1 -- 172.21.15.27:0/4212038092 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7ff9e000b960 con 0x7ff9e81008a0 2024-04-22T15:41:32.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.079+0000 7ff9edd0e700 1 -- 172.21.15.27:0/4212038092 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e818f4c0 con 0x7ff9e81008a0 2024-04-22T15:41:32.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.079+0000 7ff9e4ff9700 1 --2- 172.21.15.27:0/4212038092 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7ff9d40383c0 0x7ff9d403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:32.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.079+0000 7ff9e4ff9700 1 -- 172.21.15.27:0/4212038092 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff9e004c3c0 con 0x7ff9e81008a0 2024-04-22T15:41:32.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.079+0000 7ff9e6ffd700 1 --2- 172.21.15.27:0/4212038092 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7ff9d40383c0 0x7ff9d403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:32.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.080+0000 7ff9e6ffd700 1 --2- 172.21.15.27:0/4212038092 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7ff9d40383c0 0x7ff9d403a880 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0006fd0 tx=0x7ff9d0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:32.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.084+0000 7ff9e4ff9700 1 -- 172.21.15.27:0/4212038092 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff9e002bba0 con 0x7ff9e81008a0 2024-04-22T15:41:32.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.365+0000 7ff9edd0e700 1 -- 172.21.15.27:0/4212038092 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9e802cc70 con 0x7ff9e81008a0 2024-04-22T15:41:32.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.366+0000 7ff9e4ff9700 1 -- 172.21.15.27:0/4212038092 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7ff9e0027030 con 0x7ff9e81008a0 2024-04-22T15:41:32.369 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:41:32.369 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":1,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","modified":"2024-04-22T15:39:17.702487Z","created":"2024-04-22T15:39:17.702487Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:41:32.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.368+0000 7ff9edd0e700 1 -- 172.21.15.27:0/4212038092 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7ff9d40383c0 msgr2=0x7ff9d403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:32.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.368+0000 7ff9edd0e700 1 --2- 172.21.15.27:0/4212038092 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7ff9d40383c0 0x7ff9d403a880 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0006fd0 tx=0x7ff9d0006e40 comp rx=0 tx=0).stop 2024-04-22T15:41:32.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.368+0000 7ff9edd0e700 1 -- 172.21.15.27:0/4212038092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e81008a0 msgr2=0x7ff9e8195930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:32.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.368+0000 7ff9edd0e700 1 --2- 172.21.15.27:0/4212038092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e81008a0 0x7ff9e8195930 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0005950 tx=0x7ff9e0009e60 comp rx=0 tx=0).stop 2024-04-22T15:41:32.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.368+0000 7ff9edd0e700 1 -- 172.21.15.27:0/4212038092 shutdown_connections 2024-04-22T15:41:32.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.368+0000 7ff9edd0e700 1 --2- 172.21.15.27:0/4212038092 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7ff9d40383c0 0x7ff9d403a880 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:32.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.368+0000 7ff9edd0e700 1 --2- 172.21.15.27:0/4212038092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e81008a0 0x7ff9e8195930 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:32.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.368+0000 7ff9edd0e700 1 -- 172.21.15.27:0/4212038092 >> 172.21.15.27:0/4212038092 conn(0x7ff9e80fbe20 msgr2=0x7ff9e80fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:32.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.368+0000 7ff9edd0e700 1 -- 172.21.15.27:0/4212038092 shutdown_connections 2024-04-22T15:41:32.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:32.368+0000 7ff9edd0e700 1 -- 172.21.15.27:0/4212038092 wait complete. 2024-04-22T15:41:32.372 INFO:teuthology.orchestra.run.smithi027.stderr:dumped monmap epoch 1 2024-04-22T15:41:32.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:32 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/4212038092' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:41:32.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:32 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:32.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:32 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:32.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:32 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:32.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:32 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-22T15:41:33.785 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:33 smithi016 ceph-mon[26620]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:41:33.785 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:33 smithi016 ceph-mon[26620]: from='mgr.14162 172.21.15.16:0/4162252149' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-22T15:41:33.785 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:33 smithi016 ceph-mon[26620]: mgrmap e14: smithi016.rnsnnr(active, since 66s) 2024-04-22T15:41:33.953 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:41:33.953 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:41:34.165 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:34.221 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:35.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.571+0000 7f1fb644c700 1 -- 172.21.15.27:0/3668791776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb01013b0 msgr2=0x7f1fb01037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:35.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.571+0000 7f1fb644c700 1 --2- 172.21.15.27:0/3668791776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb01013b0 0x7f1fb01037a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0009910 tx=0x7f1fa0009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:35.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.571+0000 7f1fb644c700 1 -- 172.21.15.27:0/3668791776 shutdown_connections 2024-04-22T15:41:35.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.571+0000 7f1fb644c700 1 --2- 172.21.15.27:0/3668791776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb01013b0 0x7f1fb01037a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:35.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.571+0000 7f1fb644c700 1 -- 172.21.15.27:0/3668791776 >> 172.21.15.27:0/3668791776 conn(0x7f1fb00fad20 msgr2=0x7f1fb00fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:35.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.572+0000 7f1fb644c700 1 -- 172.21.15.27:0/3668791776 shutdown_connections 2024-04-22T15:41:35.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.572+0000 7f1fb644c700 1 -- 172.21.15.27:0/3668791776 wait complete. 2024-04-22T15:41:35.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.572+0000 7f1fb644c700 1 Processor -- start 2024-04-22T15:41:35.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.572+0000 7f1fb644c700 1 -- start start 2024-04-22T15:41:35.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.572+0000 7f1fb644c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb01013b0 0x7f1fb0197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:35.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.572+0000 7f1fb644c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb0198090 con 0x7f1fb01013b0 2024-04-22T15:41:35.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.572+0000 7f1faffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb01013b0 0x7f1fb0197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:35.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.572+0000 7f1faffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb01013b0 0x7f1fb0197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.27:44074/0 (socket says 172.21.15.27:44074) 2024-04-22T15:41:35.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.573+0000 7f1faffff700 1 -- 172.21.15.27:0/3086656072 learned_addr learned my addr 172.21.15.27:0/3086656072 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-22T15:41:35.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.573+0000 7f1faffff700 1 -- 172.21.15.27:0/3086656072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa00095f0 con 0x7f1fb01013b0 2024-04-22T15:41:35.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.573+0000 7f1faffff700 1 --2- 172.21.15.27:0/3086656072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb01013b0 0x7f1fb0197b50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0005b40 tx=0x7f1fa0004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:35.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.573+0000 7f1fadffb700 1 -- 172.21.15.27:0/3086656072 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fa001d070 con 0x7f1fb01013b0 2024-04-22T15:41:35.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.573+0000 7f1fadffb700 1 -- 172.21.15.27:0/3086656072 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1fa000be10 con 0x7f1fb01013b0 2024-04-22T15:41:35.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.573+0000 7f1fb644c700 1 -- 172.21.15.27:0/3086656072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fb0198290 con 0x7f1fb01013b0 2024-04-22T15:41:35.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.573+0000 7f1fadffb700 1 -- 172.21.15.27:0/3086656072 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fa000f460 con 0x7f1fb01013b0 2024-04-22T15:41:35.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.573+0000 7f1fb644c700 1 -- 172.21.15.27:0/3086656072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fb0198730 con 0x7f1fb01013b0 2024-04-22T15:41:35.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.574+0000 7f1fadffb700 1 -- 172.21.15.27:0/3086656072 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 14) v1 ==== 45353+0+0 (secure 0 0 0) 0x7f1fa00227e0 con 0x7f1fb01013b0 2024-04-22T15:41:35.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.574+0000 7f1fb644c700 1 -- 172.21.15.27:0/3086656072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fb01918a0 con 0x7f1fb01013b0 2024-04-22T15:41:35.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.574+0000 7f1fadffb700 1 --2- 172.21.15.27:0/3086656072 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f1f9c038410 0x7f1f9c03a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:35.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.575+0000 7f1fadffb700 1 -- 172.21.15.27:0/3086656072 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1fa004c420 con 0x7f1fb01013b0 2024-04-22T15:41:35.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.575+0000 7f1fa7fff700 1 -- 172.21.15.27:0/3086656072 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f1f9c038410 msgr2=0x7f1f9c03a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/3972982132 2024-04-22T15:41:35.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.575+0000 7f1fa7fff700 1 --2- 172.21.15.27:0/3086656072 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f1f9c038410 0x7f1f9c03a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:41:35.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.579+0000 7f1fadffb700 1 -- 172.21.15.27:0/3086656072 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1fa002bba0 con 0x7f1fb01013b0 2024-04-22T15:41:35.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.776+0000 7f1fa7fff700 1 -- 172.21.15.27:0/3086656072 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f1f9c038410 msgr2=0x7f1f9c03a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/3972982132 2024-04-22T15:41:35.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.776+0000 7f1fa7fff700 1 --2- 172.21.15.27:0/3086656072 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f1f9c038410 0x7f1f9c03a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T15:41:35.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.864+0000 7f1fb644c700 1 -- 172.21.15.27:0/3086656072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1fb002cc70 con 0x7f1fb01013b0 2024-04-22T15:41:35.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.864+0000 7f1fadffb700 1 -- 172.21.15.27:0/3086656072 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f1fa0027030 con 0x7f1fb01013b0 2024-04-22T15:41:35.868 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:41:35.868 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":1,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","modified":"2024-04-22T15:39:17.702487Z","created":"2024-04-22T15:39:17.702487Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:41:35.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.866+0000 7f1fb644c700 1 -- 172.21.15.27:0/3086656072 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f1f9c038410 msgr2=0x7f1f9c03a8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T15:41:35.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.866+0000 7f1fb644c700 1 --2- 172.21.15.27:0/3086656072 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f1f9c038410 0x7f1f9c03a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:35.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.866+0000 7f1fb644c700 1 -- 172.21.15.27:0/3086656072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb01013b0 msgr2=0x7f1fb0197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:35.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.866+0000 7f1fb644c700 1 --2- 172.21.15.27:0/3086656072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb01013b0 0x7f1fb0197b50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0005b40 tx=0x7f1fa0004740 comp rx=0 tx=0).stop 2024-04-22T15:41:35.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.866+0000 7f1fb644c700 1 -- 172.21.15.27:0/3086656072 shutdown_connections 2024-04-22T15:41:35.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.866+0000 7f1fb644c700 1 --2- 172.21.15.27:0/3086656072 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7f1f9c038410 0x7f1f9c03a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:35.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.866+0000 7f1fb644c700 1 --2- 172.21.15.27:0/3086656072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb01013b0 0x7f1fb0197b50 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:35.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.866+0000 7f1fb644c700 1 -- 172.21.15.27:0/3086656072 >> 172.21.15.27:0/3086656072 conn(0x7f1fb00fad20 msgr2=0x7f1fb00fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:35.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.867+0000 7f1fb644c700 1 -- 172.21.15.27:0/3086656072 shutdown_connections 2024-04-22T15:41:35.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:35.867+0000 7f1fb644c700 1 -- 172.21.15.27:0/3086656072 wait complete. 2024-04-22T15:41:35.871 INFO:teuthology.orchestra.run.smithi027.stderr:dumped monmap epoch 1 2024-04-22T15:41:36.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:36 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/3086656072' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:41:37.415 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:41:37.416 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:41:37.631 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:37.688 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:39.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.184+0000 7fe766bf5700 1 -- 172.21.15.27:0/4289071046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7601008a0 msgr2=0x7fe760100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.184+0000 7fe766bf5700 1 --2- 172.21.15.27:0/4289071046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7601008a0 0x7fe760100cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe754009910 tx=0x7fe754009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.184+0000 7fe766bf5700 1 -- 172.21.15.27:0/4289071046 shutdown_connections 2024-04-22T15:41:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.184+0000 7fe766bf5700 1 --2- 172.21.15.27:0/4289071046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7601008a0 0x7fe760100cc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.184+0000 7fe766bf5700 1 -- 172.21.15.27:0/4289071046 >> 172.21.15.27:0/4289071046 conn(0x7fe7600fbe20 msgr2=0x7fe7600fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.184+0000 7fe766bf5700 1 -- 172.21.15.27:0/4289071046 shutdown_connections 2024-04-22T15:41:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.184+0000 7fe766bf5700 1 -- 172.21.15.27:0/4289071046 wait complete. 2024-04-22T15:41:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.185+0000 7fe766bf5700 1 Processor -- start 2024-04-22T15:41:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.185+0000 7fe766bf5700 1 -- start start 2024-04-22T15:41:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.185+0000 7fe766bf5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7601008a0 0x7fe7601959f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.185+0000 7fe766bf5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe760195f30 con 0x7fe7601008a0 2024-04-22T15:41:39.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.185+0000 7fe764991700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7601008a0 0x7fe7601959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:39.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.185+0000 7fe764991700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7601008a0 0x7fe7601959f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.27:44102/0 (socket says 172.21.15.27:44102) 2024-04-22T15:41:39.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.185+0000 7fe764991700 1 -- 172.21.15.27:0/3452717543 learned_addr learned my addr 172.21.15.27:0/3452717543 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-22T15:41:39.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.185+0000 7fe764991700 1 -- 172.21.15.27:0/3452717543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7540095f0 con 0x7fe7601008a0 2024-04-22T15:41:39.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.186+0000 7fe764991700 1 --2- 172.21.15.27:0/3452717543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7601008a0 0x7fe7601959f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe754005b40 tx=0x7fe754004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:39.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.186+0000 7fe75dffb700 1 -- 172.21.15.27:0/3452717543 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe75401d070 con 0x7fe7601008a0 2024-04-22T15:41:39.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.186+0000 7fe766bf5700 1 -- 172.21.15.27:0/3452717543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe760196130 con 0x7fe7601008a0 2024-04-22T15:41:39.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.186+0000 7fe75dffb700 1 -- 172.21.15.27:0/3452717543 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe75400be10 con 0x7fe7601008a0 2024-04-22T15:41:39.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.186+0000 7fe75dffb700 1 -- 172.21.15.27:0/3452717543 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe75400f460 con 0x7fe7601008a0 2024-04-22T15:41:39.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.186+0000 7fe766bf5700 1 -- 172.21.15.27:0/3452717543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7601965d0 con 0x7fe7601008a0 2024-04-22T15:41:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.187+0000 7fe75dffb700 1 -- 172.21.15.27:0/3452717543 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 14) v1 ==== 45353+0+0 (secure 0 0 0) 0x7fe75400f6d0 con 0x7fe7601008a0 2024-04-22T15:41:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.187+0000 7fe766bf5700 1 -- 172.21.15.27:0/3452717543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76018f4c0 con 0x7fe7601008a0 2024-04-22T15:41:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.187+0000 7fe75dffb700 1 --2- 172.21.15.27:0/3452717543 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fe750038410 0x7fe75003a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.187+0000 7fe75dffb700 1 -- 172.21.15.27:0/3452717543 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe75404c5b0 con 0x7fe7601008a0 2024-04-22T15:41:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.187+0000 7fe75ffff700 1 -- 172.21.15.27:0/3452717543 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fe750038410 msgr2=0x7fe75003a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/3972982132 2024-04-22T15:41:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.187+0000 7fe75ffff700 1 --2- 172.21.15.27:0/3452717543 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fe750038410 0x7fe75003a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:41:39.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.196+0000 7fe75dffb700 1 -- 172.21.15.27:0/3452717543 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe754020550 con 0x7fe7601008a0 2024-04-22T15:41:39.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.388+0000 7fe75ffff700 1 -- 172.21.15.27:0/3452717543 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fe750038410 msgr2=0x7fe75003a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/3972982132 2024-04-22T15:41:39.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.388+0000 7fe75ffff700 1 --2- 172.21.15.27:0/3452717543 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fe750038410 0x7fe75003a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T15:41:39.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.475+0000 7fe766bf5700 1 -- 172.21.15.27:0/3452717543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe76002cc70 con 0x7fe7601008a0 2024-04-22T15:41:39.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.476+0000 7fe75dffb700 1 -- 172.21.15.27:0/3452717543 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fe754027070 con 0x7fe7601008a0 2024-04-22T15:41:39.480 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:41:39.480 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":1,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","modified":"2024-04-22T15:39:17.702487Z","created":"2024-04-22T15:39:17.702487Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:41:39.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.478+0000 7fe766bf5700 1 -- 172.21.15.27:0/3452717543 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fe750038410 msgr2=0x7fe75003a8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T15:41:39.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.478+0000 7fe766bf5700 1 --2- 172.21.15.27:0/3452717543 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fe750038410 0x7fe75003a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:39.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.478+0000 7fe766bf5700 1 -- 172.21.15.27:0/3452717543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7601008a0 msgr2=0x7fe7601959f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:39.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.478+0000 7fe766bf5700 1 --2- 172.21.15.27:0/3452717543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7601008a0 0x7fe7601959f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe754005b40 tx=0x7fe754004740 comp rx=0 tx=0).stop 2024-04-22T15:41:39.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.478+0000 7fe766bf5700 1 -- 172.21.15.27:0/3452717543 shutdown_connections 2024-04-22T15:41:39.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.478+0000 7fe766bf5700 1 --2- 172.21.15.27:0/3452717543 >> [v2:172.21.15.16:6800/3972982132,v1:172.21.15.16:6801/3972982132] conn(0x7fe750038410 0x7fe75003a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:39.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.478+0000 7fe766bf5700 1 --2- 172.21.15.27:0/3452717543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7601008a0 0x7fe7601959f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:39.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.478+0000 7fe766bf5700 1 -- 172.21.15.27:0/3452717543 >> 172.21.15.27:0/3452717543 conn(0x7fe7600fbe20 msgr2=0x7fe7600fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:39.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.478+0000 7fe766bf5700 1 -- 172.21.15.27:0/3452717543 shutdown_connections 2024-04-22T15:41:39.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:39.478+0000 7fe766bf5700 1 -- 172.21.15.27:0/3452717543 wait complete. 2024-04-22T15:41:39.483 INFO:teuthology.orchestra.run.smithi027.stderr:dumped monmap epoch 1 2024-04-22T15:41:39.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:39 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/3452717543' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:41:40.758 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:40 smithi016 ceph-mon[26620]: Active manager daemon smithi016.rnsnnr restarted 2024-04-22T15:41:40.758 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:40 smithi016 ceph-mon[26620]: Activating manager daemon smithi016.rnsnnr 2024-04-22T15:41:40.758 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:40 smithi016 ceph-mon[26620]: osdmap e5: 0 total, 0 up, 0 in 2024-04-22T15:41:40.758 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:40 smithi016 ceph-mon[26620]: mgrmap e15: smithi016.rnsnnr(active, starting, since 0.00309959s) 2024-04-22T15:41:40.758 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-04-22T15:41:40.758 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr metadata", "who": "smithi016.rnsnnr", "id": "smithi016.rnsnnr"}]: dispatch 2024-04-22T15:41:40.759 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T15:41:40.759 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T15:41:40.759 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T15:41:40.759 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:40 smithi016 ceph-mon[26620]: Manager daemon smithi016.rnsnnr is now available 2024-04-22T15:41:40.759 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:41:40.759 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:41:40.759 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:40.759 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:41:40.759 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi016.rnsnnr/mirror_snapshot_schedule"}]: dispatch 2024-04-22T15:41:40.759 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi016.rnsnnr/trash_purge_schedule"}]: dispatch 2024-04-22T15:41:40.959 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:41:40.960 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:41:41.160 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:41.209 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:42.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:42 smithi016 ceph-mon[26620]: [22/Apr/2024:15:41:40] ENGINE Bus STARTING 2024-04-22T15:41:42.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:42 smithi016 ceph-mon[26620]: [22/Apr/2024:15:41:40] ENGINE Serving on https://172.21.15.16:7150 2024-04-22T15:41:42.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:42 smithi016 ceph-mon[26620]: [22/Apr/2024:15:41:41] ENGINE Serving on http://172.21.15.16:8765 2024-04-22T15:41:42.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:42 smithi016 ceph-mon[26620]: [22/Apr/2024:15:41:41] ENGINE Bus STARTED 2024-04-22T15:41:42.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:42 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:42.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:42 smithi016 ceph-mon[26620]: mgrmap e16: smithi016.rnsnnr(active, since 1.00549s) 2024-04-22T15:41:42.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.979+0000 7fdfc0892700 1 -- 172.21.15.27:0/2386705512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfbc101320 msgr2=0x7fdfbc103710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:42.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.979+0000 7fdfc0892700 1 --2- 172.21.15.27:0/2386705512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfbc101320 0x7fdfbc103710 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdfac009910 tx=0x7fdfac009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:42.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.979+0000 7fdfc0892700 1 -- 172.21.15.27:0/2386705512 shutdown_connections 2024-04-22T15:41:42.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.979+0000 7fdfc0892700 1 --2- 172.21.15.27:0/2386705512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfbc101320 0x7fdfbc103710 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:42.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.979+0000 7fdfc0892700 1 -- 172.21.15.27:0/2386705512 >> 172.21.15.27:0/2386705512 conn(0x7fdfbc0fad30 msgr2=0x7fdfbc0fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:42.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.979+0000 7fdfc0892700 1 -- 172.21.15.27:0/2386705512 shutdown_connections 2024-04-22T15:41:42.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.979+0000 7fdfc0892700 1 -- 172.21.15.27:0/2386705512 wait complete. 2024-04-22T15:41:42.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.980+0000 7fdfc0892700 1 Processor -- start 2024-04-22T15:41:42.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.980+0000 7fdfc0892700 1 -- start start 2024-04-22T15:41:42.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.980+0000 7fdfc0892700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfbc101320 0x7fdfbc195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:42.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.980+0000 7fdfc0892700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfbc195ea0 con 0x7fdfbc101320 2024-04-22T15:41:42.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.980+0000 7fdfba59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfbc101320 0x7fdfbc195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:42.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.980+0000 7fdfba59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfbc101320 0x7fdfbc195960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.27:36948/0 (socket says 172.21.15.27:36948) 2024-04-22T15:41:42.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.980+0000 7fdfba59c700 1 -- 172.21.15.27:0/145520373 learned_addr learned my addr 172.21.15.27:0/145520373 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-22T15:41:42.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.981+0000 7fdfba59c700 1 -- 172.21.15.27:0/145520373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfac0095f0 con 0x7fdfbc101320 2024-04-22T15:41:42.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.981+0000 7fdfba59c700 1 --2- 172.21.15.27:0/145520373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfbc101320 0x7fdfbc195960 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdfac005b40 tx=0x7fdfac004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:42.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.981+0000 7fdfab7fe700 1 -- 172.21.15.27:0/145520373 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfac01d070 con 0x7fdfbc101320 2024-04-22T15:41:42.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.981+0000 7fdfab7fe700 1 -- 172.21.15.27:0/145520373 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdfac00be10 con 0x7fdfbc101320 2024-04-22T15:41:42.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.981+0000 7fdfab7fe700 1 -- 172.21.15.27:0/145520373 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfac00f460 con 0x7fdfbc101320 2024-04-22T15:41:42.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.981+0000 7fdfc0892700 1 -- 172.21.15.27:0/145520373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfbc1960a0 con 0x7fdfbc101320 2024-04-22T15:41:42.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.981+0000 7fdfc0892700 1 -- 172.21.15.27:0/145520373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfbc196540 con 0x7fdfbc101320 2024-04-22T15:41:42.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.982+0000 7fdfc0892700 1 -- 172.21.15.27:0/145520373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfbc18f4b0 con 0x7fdfbc101320 2024-04-22T15:41:42.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.982+0000 7fdfab7fe700 1 -- 172.21.15.27:0/145520373 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 45204+0+0 (secure 0 0 0) 0x7fdfac00b960 con 0x7fdfbc101320 2024-04-22T15:41:42.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.983+0000 7fdfab7fe700 1 --2- 172.21.15.27:0/145520373 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fdfa40382b0 0x7fdfa403a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:42.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.983+0000 7fdfab7fe700 1 -- 172.21.15.27:0/145520373 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fdfac04c200 con 0x7fdfbc101320 2024-04-22T15:41:42.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.983+0000 7fdfb9d9b700 1 --2- 172.21.15.27:0/145520373 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fdfa40382b0 0x7fdfa403a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:42.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.983+0000 7fdfb9d9b700 1 --2- 172.21.15.27:0/145520373 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fdfa40382b0 0x7fdfa403a770 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdfb0006fd0 tx=0x7fdfb0006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:42.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:42.987+0000 7fdfab7fe700 1 -- 172.21.15.27:0/145520373 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fdfac02bba0 con 0x7fdfbc101320 2024-04-22T15:41:43.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:43.105+0000 7fdfab7fe700 1 -- 172.21.15.27:0/145520373 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7fdfac0227e0 con 0x7fdfbc101320 2024-04-22T15:41:43.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:43.271+0000 7fdfc0892700 1 -- 172.21.15.27:0/145520373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdfbc02cc70 con 0x7fdfbc101320 2024-04-22T15:41:43.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:43.272+0000 7fdfab7fe700 1 -- 172.21.15.27:0/145520373 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fdfac027090 con 0x7fdfbc101320 2024-04-22T15:41:43.274 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:41:43.274 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":1,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","modified":"2024-04-22T15:39:17.702487Z","created":"2024-04-22T15:39:17.702487Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:41:43.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:43.274+0000 7fdfc0892700 1 -- 172.21.15.27:0/145520373 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fdfa40382b0 msgr2=0x7fdfa403a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:43.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:43.274+0000 7fdfc0892700 1 --2- 172.21.15.27:0/145520373 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fdfa40382b0 0x7fdfa403a770 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdfb0006fd0 tx=0x7fdfb0006e40 comp rx=0 tx=0).stop 2024-04-22T15:41:43.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:43.274+0000 7fdfc0892700 1 -- 172.21.15.27:0/145520373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfbc101320 msgr2=0x7fdfbc195960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:43.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:43.274+0000 7fdfc0892700 1 --2- 172.21.15.27:0/145520373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfbc101320 0x7fdfbc195960 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdfac005b40 tx=0x7fdfac004740 comp rx=0 tx=0).stop 2024-04-22T15:41:43.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:43.274+0000 7fdfc0892700 1 -- 172.21.15.27:0/145520373 shutdown_connections 2024-04-22T15:41:43.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:43.274+0000 7fdfc0892700 1 --2- 172.21.15.27:0/145520373 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fdfa40382b0 0x7fdfa403a770 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:43.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:43.274+0000 7fdfc0892700 1 --2- 172.21.15.27:0/145520373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfbc101320 0x7fdfbc195960 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:43.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:43.274+0000 7fdfc0892700 1 -- 172.21.15.27:0/145520373 >> 172.21.15.27:0/145520373 conn(0x7fdfbc0fad30 msgr2=0x7fdfbc0fb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:43.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:43.274+0000 7fdfc0892700 1 -- 172.21.15.27:0/145520373 shutdown_connections 2024-04-22T15:41:43.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:43.274+0000 7fdfc0892700 1 -- 172.21.15.27:0/145520373 wait complete. 2024-04-22T15:41:43.278 INFO:teuthology.orchestra.run.smithi027.stderr:dumped monmap epoch 1 2024-04-22T15:41:43.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:43 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:43.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:43 smithi016 ceph-mon[26620]: mgrmap e17: smithi016.rnsnnr(active, since 2s) 2024-04-22T15:41:43.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:43 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/145520373' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:41:44.694 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:44 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:44.694 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:44 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:44.694 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:44 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:44.865 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:41:44.865 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:41:45.356 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:45.413 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T15:41:46.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:46 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:46.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:46 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:46.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:46 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd/host:smithi016", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:41:47.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.639+0000 7f52033ad700 1 -- 172.21.15.27:0/2481852469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51fc105720 msgr2=0x7f51fc105b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:47.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.639+0000 7f52033ad700 1 --2- 172.21.15.27:0/2481852469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51fc105720 0x7f51fc105b00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f51f8009910 tx=0x7f51f8009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:47.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.639+0000 7f52033ad700 1 -- 172.21.15.27:0/2481852469 shutdown_connections 2024-04-22T15:41:47.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.639+0000 7f52033ad700 1 --2- 172.21.15.27:0/2481852469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51fc105720 0x7f51fc105b00 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:47.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.639+0000 7f52033ad700 1 -- 172.21.15.27:0/2481852469 >> 172.21.15.27:0/2481852469 conn(0x7f51fc0fb250 msgr2=0x7f51fc0fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:47.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.639+0000 7f52033ad700 1 -- 172.21.15.27:0/2481852469 shutdown_connections 2024-04-22T15:41:47.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.639+0000 7f52033ad700 1 -- 172.21.15.27:0/2481852469 wait complete. 2024-04-22T15:41:47.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.639+0000 7f52033ad700 1 Processor -- start 2024-04-22T15:41:47.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.639+0000 7f52033ad700 1 -- start start 2024-04-22T15:41:47.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.640+0000 7f52033ad700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51fc105720 0x7f51fc196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:47.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.640+0000 7f52033ad700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51fc1970f0 con 0x7f51fc105720 2024-04-22T15:41:47.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.640+0000 7f5201149700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51fc105720 0x7f51fc196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:47.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.640+0000 7f5201149700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51fc105720 0x7f51fc196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.27:36952/0 (socket says 172.21.15.27:36952) 2024-04-22T15:41:47.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.640+0000 7f5201149700 1 -- 172.21.15.27:0/2078921097 learned_addr learned my addr 172.21.15.27:0/2078921097 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-22T15:41:47.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.640+0000 7f5201149700 1 -- 172.21.15.27:0/2078921097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f80095f0 con 0x7f51fc105720 2024-04-22T15:41:47.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.640+0000 7f5201149700 1 --2- 172.21.15.27:0/2078921097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51fc105720 0x7f51fc196bb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f51f8005b40 tx=0x7f51f8004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:47.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.640+0000 7f51f27fc700 1 -- 172.21.15.27:0/2078921097 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51f801d070 con 0x7f51fc105720 2024-04-22T15:41:47.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.640+0000 7f51f27fc700 1 -- 172.21.15.27:0/2078921097 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f51f80044d0 con 0x7f51fc105720 2024-04-22T15:41:47.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.640+0000 7f52033ad700 1 -- 172.21.15.27:0/2078921097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51fc1972f0 con 0x7f51fc105720 2024-04-22T15:41:47.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.640+0000 7f51f27fc700 1 -- 172.21.15.27:0/2078921097 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51f8003910 con 0x7f51fc105720 2024-04-22T15:41:47.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.640+0000 7f52033ad700 1 -- 172.21.15.27:0/2078921097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51fc197770 con 0x7f51fc105720 2024-04-22T15:41:47.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.641+0000 7f51f27fc700 1 -- 172.21.15.27:0/2078921097 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7f51f8003b40 con 0x7f51fc105720 2024-04-22T15:41:47.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.641+0000 7f51f27fc700 1 --2- 172.21.15.27:0/2078921097 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f51e8038500 0x7f51e803a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:47.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.641+0000 7f52033ad700 1 -- 172.21.15.27:0/2078921097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51fc1039d0 con 0x7f51fc105720 2024-04-22T15:41:47.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.642+0000 7f51f27fc700 1 -- 172.21.15.27:0/2078921097 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f51f804c560 con 0x7f51fc105720 2024-04-22T15:41:47.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.642+0000 7f5200948700 1 --2- 172.21.15.27:0/2078921097 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f51e8038500 0x7f51e803a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:47.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.642+0000 7f5200948700 1 --2- 172.21.15.27:0/2078921097 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f51e8038500 0x7f51e803a9c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f51ec006fd0 tx=0x7f51ec006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:47.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.647+0000 7f51f27fc700 1 -- 172.21.15.27:0/2078921097 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f51f802a950 con 0x7f51fc105720 2024-04-22T15:41:47.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:47 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:47.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:47 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:47.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:47 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:47.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:47 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:47.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:47 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd/host:smithi027", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:41:47.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:47 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:41:47.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:47 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:41:47.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:47 smithi016 ceph-mon[26620]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-22T15:41:47.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:47 smithi016 ceph-mon[26620]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-22T15:41:47.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:47 smithi016 ceph-mon[26620]: Updating smithi027:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:41:47.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:47 smithi016 ceph-mon[26620]: Updating smithi016:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:41:47.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:47 smithi016 ceph-mon[26620]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:41:47.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:47 smithi016 ceph-mon[26620]: Updating smithi027:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:41:47.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:47 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:47.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:47 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.927+0000 7f52033ad700 1 -- 172.21.15.27:0/2078921097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f51fc0623c0 con 0x7f51fc105720 2024-04-22T15:41:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.928+0000 7f51f27fc700 1 -- 172.21.15.27:0/2078921097 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f51f802a360 con 0x7f51fc105720 2024-04-22T15:41:47.930 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:41:47.931 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":1,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","modified":"2024-04-22T15:39:17.702487Z","created":"2024-04-22T15:39:17.702487Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:41:47.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.930+0000 7f52033ad700 1 -- 172.21.15.27:0/2078921097 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f51e8038500 msgr2=0x7f51e803a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:47.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.930+0000 7f52033ad700 1 --2- 172.21.15.27:0/2078921097 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f51e8038500 0x7f51e803a9c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f51ec006fd0 tx=0x7f51ec006e40 comp rx=0 tx=0).stop 2024-04-22T15:41:47.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.930+0000 7f52033ad700 1 -- 172.21.15.27:0/2078921097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51fc105720 msgr2=0x7f51fc196bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:47.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.930+0000 7f52033ad700 1 --2- 172.21.15.27:0/2078921097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51fc105720 0x7f51fc196bb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f51f8005b40 tx=0x7f51f8004c80 comp rx=0 tx=0).stop 2024-04-22T15:41:47.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.930+0000 7f52033ad700 1 -- 172.21.15.27:0/2078921097 shutdown_connections 2024-04-22T15:41:47.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.930+0000 7f52033ad700 1 --2- 172.21.15.27:0/2078921097 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f51e8038500 0x7f51e803a9c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:47.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.930+0000 7f52033ad700 1 --2- 172.21.15.27:0/2078921097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51fc105720 0x7f51fc196bb0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:47.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.930+0000 7f52033ad700 1 -- 172.21.15.27:0/2078921097 >> 172.21.15.27:0/2078921097 conn(0x7f51fc0fb250 msgr2=0x7f51fc107b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:47.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.930+0000 7f52033ad700 1 -- 172.21.15.27:0/2078921097 shutdown_connections 2024-04-22T15:41:47.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:47.930+0000 7f52033ad700 1 -- 172.21.15.27:0/2078921097 wait complete. 2024-04-22T15:41:47.933 INFO:teuthology.orchestra.run.smithi027.stderr:dumped monmap epoch 1 2024-04-22T15:41:48.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:48 smithi016 ceph-mon[26620]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:41:48.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:48 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/2078921097' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:41:48.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:48 smithi016 ceph-mon[26620]: Updating smithi016:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:41:48.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:48 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:48.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:48 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:48.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:48 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:48.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:48 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi027", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:41:48.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:48 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi027", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-22T15:41:48.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:48 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:41:48.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:48 smithi016 ceph-mon[26620]: Deploying daemon ceph-exporter.smithi027 on smithi027 2024-04-22T15:41:49.518 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:41:49.518 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:41:49.741 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:41:51.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.464+0000 7fa5ccd96700 1 -- 172.21.15.27:0/3186117903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5c8105700 msgr2=0x7fa5c8105ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:51.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.464+0000 7fa5ccd96700 1 --2- 172.21.15.27:0/3186117903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5c8105700 0x7fa5c8105ae0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8009910 tx=0x7fa5b8009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:51.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.464+0000 7fa5ccd96700 1 -- 172.21.15.27:0/3186117903 shutdown_connections 2024-04-22T15:41:51.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.464+0000 7fa5ccd96700 1 --2- 172.21.15.27:0/3186117903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5c8105700 0x7fa5c8105ae0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:51.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.464+0000 7fa5ccd96700 1 -- 172.21.15.27:0/3186117903 >> 172.21.15.27:0/3186117903 conn(0x7fa5c80fb210 msgr2=0x7fa5c80fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:51.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.464+0000 7fa5ccd96700 1 -- 172.21.15.27:0/3186117903 shutdown_connections 2024-04-22T15:41:51.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.464+0000 7fa5ccd96700 1 -- 172.21.15.27:0/3186117903 wait complete. 2024-04-22T15:41:51.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.465+0000 7fa5ccd96700 1 Processor -- start 2024-04-22T15:41:51.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.465+0000 7fa5ccd96700 1 -- start start 2024-04-22T15:41:51.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.465+0000 7fa5ccd96700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5c8105700 0x7fa5c80689f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:51.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.465+0000 7fa5ccd96700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c8068f30 con 0x7fa5c8105700 2024-04-22T15:41:51.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.466+0000 7fa5c659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5c8105700 0x7fa5c80689f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:51.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.466+0000 7fa5c659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5c8105700 0x7fa5c80689f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.27:36970/0 (socket says 172.21.15.27:36970) 2024-04-22T15:41:51.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.466+0000 7fa5c659c700 1 -- 172.21.15.27:0/721597630 learned_addr learned my addr 172.21.15.27:0/721597630 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-22T15:41:51.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.466+0000 7fa5c659c700 1 -- 172.21.15.27:0/721597630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b80095f0 con 0x7fa5c8105700 2024-04-22T15:41:51.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.466+0000 7fa5c659c700 1 --2- 172.21.15.27:0/721597630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5c8105700 0x7fa5c80689f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8005b40 tx=0x7fa5b8004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:51.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.466+0000 7fa5b77fe700 1 -- 172.21.15.27:0/721597630 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5b801d070 con 0x7fa5c8105700 2024-04-22T15:41:51.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.466+0000 7fa5ccd96700 1 -- 172.21.15.27:0/721597630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c8069130 con 0x7fa5c8105700 2024-04-22T15:41:51.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.466+0000 7fa5b77fe700 1 -- 172.21.15.27:0/721597630 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa5b80044d0 con 0x7fa5c8105700 2024-04-22T15:41:51.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.466+0000 7fa5b77fe700 1 -- 172.21.15.27:0/721597630 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5b8003910 con 0x7fa5c8105700 2024-04-22T15:41:51.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.466+0000 7fa5ccd96700 1 -- 172.21.15.27:0/721597630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c80695b0 con 0x7fa5c8105700 2024-04-22T15:41:51.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.467+0000 7fa5ccd96700 1 -- 172.21.15.27:0/721597630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c81039b0 con 0x7fa5c8105700 2024-04-22T15:41:51.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.467+0000 7fa5b77fe700 1 -- 172.21.15.27:0/721597630 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7fa5b800b960 con 0x7fa5c8105700 2024-04-22T15:41:51.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.467+0000 7fa5b77fe700 1 --2- 172.21.15.27:0/721597630 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fa5b00384b0 0x7fa5b003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:51.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.467+0000 7fa5b77fe700 1 -- 172.21.15.27:0/721597630 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fa5b804c5b0 con 0x7fa5c8105700 2024-04-22T15:41:51.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.468+0000 7fa5c5d9b700 1 --2- 172.21.15.27:0/721597630 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fa5b00384b0 0x7fa5b003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:51.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.468+0000 7fa5c5d9b700 1 --2- 172.21.15.27:0/721597630 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fa5b00384b0 0x7fa5b003a970 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc006fd0 tx=0x7fa5bc006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:51.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.473+0000 7fa5b77fe700 1 -- 172.21.15.27:0/721597630 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa5b802a540 con 0x7fa5c8105700 2024-04-22T15:41:51.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.765+0000 7fa5ccd96700 1 -- 172.21.15.27:0/721597630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5c802cc70 con 0x7fa5c8105700 2024-04-22T15:41:51.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.776+0000 7fa5b77fe700 1 -- 172.21.15.27:0/721597630 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fa5b802a360 con 0x7fa5c8105700 2024-04-22T15:41:51.778 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:41:51.779 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":1,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","modified":"2024-04-22T15:39:17.702487Z","created":"2024-04-22T15:39:17.702487Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:41:51.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.778+0000 7fa5ccd96700 1 -- 172.21.15.27:0/721597630 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fa5b00384b0 msgr2=0x7fa5b003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:51.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.778+0000 7fa5ccd96700 1 --2- 172.21.15.27:0/721597630 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fa5b00384b0 0x7fa5b003a970 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc006fd0 tx=0x7fa5bc006e40 comp rx=0 tx=0).stop 2024-04-22T15:41:51.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.778+0000 7fa5ccd96700 1 -- 172.21.15.27:0/721597630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5c8105700 msgr2=0x7fa5c80689f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:51.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.778+0000 7fa5ccd96700 1 --2- 172.21.15.27:0/721597630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5c8105700 0x7fa5c80689f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8005b40 tx=0x7fa5b8004c80 comp rx=0 tx=0).stop 2024-04-22T15:41:51.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.778+0000 7fa5ccd96700 1 -- 172.21.15.27:0/721597630 shutdown_connections 2024-04-22T15:41:51.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.778+0000 7fa5ccd96700 1 --2- 172.21.15.27:0/721597630 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fa5b00384b0 0x7fa5b003a970 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:51.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.778+0000 7fa5ccd96700 1 --2- 172.21.15.27:0/721597630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5c8105700 0x7fa5c80689f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:51.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.778+0000 7fa5ccd96700 1 -- 172.21.15.27:0/721597630 >> 172.21.15.27:0/721597630 conn(0x7fa5c80fb210 msgr2=0x7fa5c8107af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:51.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.778+0000 7fa5ccd96700 1 -- 172.21.15.27:0/721597630 shutdown_connections 2024-04-22T15:41:51.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:51.778+0000 7fa5ccd96700 1 -- 172.21.15.27:0/721597630 wait complete. 2024-04-22T15:41:51.782 INFO:teuthology.orchestra.run.smithi027.stderr:dumped monmap epoch 1 2024-04-22T15:41:52.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:51 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/721597630' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:41:53.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:52 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:53.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:52 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:53.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:52 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:53.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:52 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:53.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:52 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi027", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:41:53.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:52 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi027", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-22T15:41:53.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:52 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:41:53.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:52 smithi016 ceph-mon[26620]: Deploying daemon crash.smithi027 on smithi027 2024-04-22T15:41:53.479 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:41:53.479 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:41:53.685 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:41:55.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.532+0000 7f60e621f700 1 -- 172.21.15.27:0/1879768356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60e0106820 msgr2=0x7f60e0106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:55.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.532+0000 7f60e621f700 1 --2- 172.21.15.27:0/1879768356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60e0106820 0x7f60e0106c00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f60d0009910 tx=0x7f60d0009c20 comp rx=0 tx=0).stop 2024-04-22T15:41:55.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.532+0000 7f60e621f700 1 -- 172.21.15.27:0/1879768356 shutdown_connections 2024-04-22T15:41:55.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.532+0000 7f60e621f700 1 --2- 172.21.15.27:0/1879768356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60e0106820 0x7f60e0106c00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:55.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.532+0000 7f60e621f700 1 -- 172.21.15.27:0/1879768356 >> 172.21.15.27:0/1879768356 conn(0x7f60e0075290 msgr2=0x7f60e00756a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:55.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.532+0000 7f60e621f700 1 -- 172.21.15.27:0/1879768356 shutdown_connections 2024-04-22T15:41:55.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.532+0000 7f60e621f700 1 -- 172.21.15.27:0/1879768356 wait complete. 2024-04-22T15:41:55.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.533+0000 7f60e621f700 1 Processor -- start 2024-04-22T15:41:55.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.533+0000 7f60e621f700 1 -- start start 2024-04-22T15:41:55.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.533+0000 7f60e621f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60e0106820 0x7f60e00ff800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:55.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.533+0000 7f60e621f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e01017d0 con 0x7f60e0106820 2024-04-22T15:41:55.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.533+0000 7f60df7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60e0106820 0x7f60e00ff800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:55.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.534+0000 7f60df7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60e0106820 0x7f60e00ff800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.27:51960/0 (socket says 172.21.15.27:51960) 2024-04-22T15:41:55.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.534+0000 7f60df7fe700 1 -- 172.21.15.27:0/1093888042 learned_addr learned my addr 172.21.15.27:0/1093888042 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-22T15:41:55.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.534+0000 7f60df7fe700 1 -- 172.21.15.27:0/1093888042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d00095f0 con 0x7f60e0106820 2024-04-22T15:41:55.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.534+0000 7f60df7fe700 1 --2- 172.21.15.27:0/1093888042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60e0106820 0x7f60e00ff800 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f60d0005b40 tx=0x7f60d0004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:55.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.534+0000 7f60dcff9700 1 -- 172.21.15.27:0/1093888042 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60d001d070 con 0x7f60e0106820 2024-04-22T15:41:55.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.534+0000 7f60dcff9700 1 -- 172.21.15.27:0/1093888042 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f60d00044d0 con 0x7f60e0106820 2024-04-22T15:41:55.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.534+0000 7f60e621f700 1 -- 172.21.15.27:0/1093888042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e00ffe00 con 0x7f60e0106820 2024-04-22T15:41:55.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.534+0000 7f60dcff9700 1 -- 172.21.15.27:0/1093888042 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60d0003910 con 0x7f60e0106820 2024-04-22T15:41:55.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.534+0000 7f60e621f700 1 -- 172.21.15.27:0/1093888042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e0100280 con 0x7f60e0106820 2024-04-22T15:41:55.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.535+0000 7f60e621f700 1 -- 172.21.15.27:0/1093888042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60e0105ad0 con 0x7f60e0106820 2024-04-22T15:41:55.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.535+0000 7f60dcff9700 1 -- 172.21.15.27:0/1093888042 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7f60d0003b40 con 0x7f60e0106820 2024-04-22T15:41:55.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.535+0000 7f60dcff9700 1 --2- 172.21.15.27:0/1093888042 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f60c8038500 0x7f60c803a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:41:55.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.535+0000 7f60dcff9700 1 -- 172.21.15.27:0/1093888042 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f60d004c500 con 0x7f60e0106820 2024-04-22T15:41:55.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.536+0000 7f60deffd700 1 --2- 172.21.15.27:0/1093888042 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f60c8038500 0x7f60c803a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:41:55.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.536+0000 7f60deffd700 1 --2- 172.21.15.27:0/1093888042 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f60c8038500 0x7f60c803a9c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f60d4006fd0 tx=0x7f60d4006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:41:55.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.541+0000 7f60dcff9700 1 -- 172.21.15.27:0/1093888042 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f60d002a540 con 0x7f60e0106820 2024-04-22T15:41:55.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.830+0000 7f60e621f700 1 -- 172.21.15.27:0/1093888042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f60e0100530 con 0x7f60e0106820 2024-04-22T15:41:55.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.831+0000 7f60dcff9700 1 -- 172.21.15.27:0/1093888042 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f60d002a360 con 0x7f60e0106820 2024-04-22T15:41:55.835 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:41:55.835 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":1,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","modified":"2024-04-22T15:39:17.702487Z","created":"2024-04-22T15:39:17.702487Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:41:55.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.833+0000 7f60e621f700 1 -- 172.21.15.27:0/1093888042 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f60c8038500 msgr2=0x7f60c803a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:55.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.833+0000 7f60e621f700 1 --2- 172.21.15.27:0/1093888042 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f60c8038500 0x7f60c803a9c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f60d4006fd0 tx=0x7f60d4006e40 comp rx=0 tx=0).stop 2024-04-22T15:41:55.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.833+0000 7f60e621f700 1 -- 172.21.15.27:0/1093888042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60e0106820 msgr2=0x7f60e00ff800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:41:55.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.833+0000 7f60e621f700 1 --2- 172.21.15.27:0/1093888042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60e0106820 0x7f60e00ff800 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f60d0005b40 tx=0x7f60d0004c80 comp rx=0 tx=0).stop 2024-04-22T15:41:55.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.833+0000 7f60e621f700 1 -- 172.21.15.27:0/1093888042 shutdown_connections 2024-04-22T15:41:55.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.833+0000 7f60e621f700 1 --2- 172.21.15.27:0/1093888042 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f60c8038500 0x7f60c803a9c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:55.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.833+0000 7f60e621f700 1 --2- 172.21.15.27:0/1093888042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60e0106820 0x7f60e00ff800 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:41:55.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.833+0000 7f60e621f700 1 -- 172.21.15.27:0/1093888042 >> 172.21.15.27:0/1093888042 conn(0x7f60e0075290 msgr2=0x7f60e00fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:41:55.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.833+0000 7f60e621f700 1 -- 172.21.15.27:0/1093888042 shutdown_connections 2024-04-22T15:41:55.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:55.833+0000 7f60e621f700 1 -- 172.21.15.27:0/1093888042 wait complete. 2024-04-22T15:41:55.837 INFO:teuthology.orchestra.run.smithi027.stderr:dumped monmap epoch 1 2024-04-22T15:41:56.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:56 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:56.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:56 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:41:56.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:56 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:56.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:56 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:56.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:56 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:56.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:56 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:41:56.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:56 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/1093888042' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:41:57.441 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:41:57.442 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:41:57.648 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:41:57.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:41:57 smithi016 ceph-mon[26620]: Deploying daemon node-exporter.smithi027 on smithi027 2024-04-22T15:42:00.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.983+0000 7febd4749700 1 -- 172.21.15.27:0/1314571540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febcc106810 msgr2=0x7febcc106bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:00.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.983+0000 7febd4749700 1 --2- 172.21.15.27:0/1314571540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febcc106810 0x7febcc106bf0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7febbc00b0a0 tx=0x7febbc00b3b0 comp rx=0 tx=0).stop 2024-04-22T15:42:00.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.983+0000 7febd4749700 1 -- 172.21.15.27:0/1314571540 shutdown_connections 2024-04-22T15:42:00.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.983+0000 7febd4749700 1 --2- 172.21.15.27:0/1314571540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febcc106810 0x7febcc106bf0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:00.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.983+0000 7febd4749700 1 -- 172.21.15.27:0/1314571540 >> 172.21.15.27:0/1314571540 conn(0x7febcc0fc320 msgr2=0x7febcc0fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:00.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.984+0000 7febd4749700 1 -- 172.21.15.27:0/1314571540 shutdown_connections 2024-04-22T15:42:00.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.984+0000 7febd4749700 1 -- 172.21.15.27:0/1314571540 wait complete. 2024-04-22T15:42:00.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.984+0000 7febd4749700 1 Processor -- start 2024-04-22T15:42:00.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.984+0000 7febd4749700 1 -- start start 2024-04-22T15:42:00.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.984+0000 7febd4749700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febcc106810 0x7febcc19bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:00.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.984+0000 7febd4749700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febcc19c010 con 0x7febcc106810 2024-04-22T15:42:00.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.985+0000 7febd24e5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febcc106810 0x7febcc19bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:00.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.985+0000 7febd24e5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febcc106810 0x7febcc19bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.27:51998/0 (socket says 172.21.15.27:51998) 2024-04-22T15:42:00.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.985+0000 7febd24e5700 1 -- 172.21.15.27:0/1066202179 learned_addr learned my addr 172.21.15.27:0/1066202179 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-22T15:42:00.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.985+0000 7febd24e5700 1 -- 172.21.15.27:0/1066202179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febbc009d00 con 0x7febcc106810 2024-04-22T15:42:00.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.985+0000 7febd24e5700 1 --2- 172.21.15.27:0/1066202179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febcc106810 0x7febcc19bad0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7febbc00d010 tx=0x7febbc003ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:00.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.985+0000 7febcb7fe700 1 -- 172.21.15.27:0/1066202179 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febbc00e040 con 0x7febcc106810 2024-04-22T15:42:00.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.985+0000 7febd4749700 1 -- 172.21.15.27:0/1066202179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febcc19c210 con 0x7febcc106810 2024-04-22T15:42:00.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.985+0000 7febcb7fe700 1 -- 172.21.15.27:0/1066202179 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7febbc0044f0 con 0x7febcc106810 2024-04-22T15:42:00.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.985+0000 7febcb7fe700 1 -- 172.21.15.27:0/1066202179 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febbc01d920 con 0x7febcc106810 2024-04-22T15:42:00.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.985+0000 7febd4749700 1 -- 172.21.15.27:0/1066202179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febcc195b50 con 0x7febcc106810 2024-04-22T15:42:00.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.986+0000 7febd4749700 1 -- 172.21.15.27:0/1066202179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febcc0623c0 con 0x7febcc106810 2024-04-22T15:42:00.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.986+0000 7febcb7fe700 1 -- 172.21.15.27:0/1066202179 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7febbc004040 con 0x7febcc106810 2024-04-22T15:42:00.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.986+0000 7febcb7fe700 1 --2- 172.21.15.27:0/1066202179 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7febb40384b0 0x7febb403a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:00.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.987+0000 7febcb7fe700 1 -- 172.21.15.27:0/1066202179 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7febbc04c4e0 con 0x7febcc106810 2024-04-22T15:42:00.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.987+0000 7febd1ce4700 1 --2- 172.21.15.27:0/1066202179 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7febb40384b0 0x7febb403a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:00.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.987+0000 7febd1ce4700 1 --2- 172.21.15.27:0/1066202179 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7febb40384b0 0x7febb403a970 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7febc0006fd0 tx=0x7febc0006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:00.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:41:59.992+0000 7febcb7fe700 1 -- 172.21.15.27:0/1066202179 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7febbc011570 con 0x7febcc106810 2024-04-22T15:42:00.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:00.272+0000 7febd4749700 1 -- 172.21.15.27:0/1066202179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7febcc1965d0 con 0x7febcc106810 2024-04-22T15:42:00.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:00.273+0000 7febcb7fe700 1 -- 172.21.15.27:0/1066202179 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7febbc011390 con 0x7febcc106810 2024-04-22T15:42:00.277 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:42:00.277 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":1,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","modified":"2024-04-22T15:39:17.702487Z","created":"2024-04-22T15:39:17.702487Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:42:00.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:00.275+0000 7febd4749700 1 -- 172.21.15.27:0/1066202179 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7febb40384b0 msgr2=0x7febb403a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:00.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:00.275+0000 7febd4749700 1 --2- 172.21.15.27:0/1066202179 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7febb40384b0 0x7febb403a970 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7febc0006fd0 tx=0x7febc0006e40 comp rx=0 tx=0).stop 2024-04-22T15:42:00.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:00.275+0000 7febd4749700 1 -- 172.21.15.27:0/1066202179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febcc106810 msgr2=0x7febcc19bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:00.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:00.275+0000 7febd4749700 1 --2- 172.21.15.27:0/1066202179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febcc106810 0x7febcc19bad0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7febbc00d010 tx=0x7febbc003ad0 comp rx=0 tx=0).stop 2024-04-22T15:42:00.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:00.275+0000 7febd4749700 1 -- 172.21.15.27:0/1066202179 shutdown_connections 2024-04-22T15:42:00.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:00.275+0000 7febd4749700 1 --2- 172.21.15.27:0/1066202179 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7febb40384b0 0x7febb403a970 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:00.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:00.275+0000 7febd4749700 1 --2- 172.21.15.27:0/1066202179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febcc106810 0x7febcc19bad0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:00.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:00.275+0000 7febd4749700 1 -- 172.21.15.27:0/1066202179 >> 172.21.15.27:0/1066202179 conn(0x7febcc0fc320 msgr2=0x7febcc108c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:00.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:00.275+0000 7febd4749700 1 -- 172.21.15.27:0/1066202179 shutdown_connections 2024-04-22T15:42:00.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:00.275+0000 7febd4749700 1 -- 172.21.15.27:0/1066202179 wait complete. 2024-04-22T15:42:00.280 INFO:teuthology.orchestra.run.smithi027.stderr:dumped monmap epoch 1 2024-04-22T15:42:00.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:00 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/1066202179' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:42:01.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:01 smithi016 ceph-mon[26620]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:01.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:01 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:01.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:01 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:01.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:01 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:01.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:01 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:01.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:01 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:01.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:01 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi027.ukdgzs", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:42:01.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:01 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi027.ukdgzs", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T15:42:01.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:01 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:42:01.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:01 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:02.211 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:42:02.211 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:42:02.439 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:42:02.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:02 smithi016 ceph-mon[26620]: Deploying daemon mgr.smithi027.ukdgzs on smithi027 2024-04-22T15:42:03.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:03 smithi016 ceph-mon[26620]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:03.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.980+0000 7f2406b8a700 1 -- 172.21.15.27:0/2232188095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2400106820 msgr2=0x7f2400106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:03.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.980+0000 7f2406b8a700 1 --2- 172.21.15.27:0/2232188095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2400106820 0x7f2400106c00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f23e8009910 tx=0x7f23e8009c20 comp rx=0 tx=0).stop 2024-04-22T15:42:03.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.981+0000 7f2406b8a700 1 -- 172.21.15.27:0/2232188095 shutdown_connections 2024-04-22T15:42:03.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.981+0000 7f2406b8a700 1 --2- 172.21.15.27:0/2232188095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2400106820 0x7f2400106c00 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:03.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.981+0000 7f2406b8a700 1 -- 172.21.15.27:0/2232188095 >> 172.21.15.27:0/2232188095 conn(0x7f2400075290 msgr2=0x7f24000756a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:03.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.981+0000 7f2406b8a700 1 -- 172.21.15.27:0/2232188095 shutdown_connections 2024-04-22T15:42:03.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.981+0000 7f2406b8a700 1 -- 172.21.15.27:0/2232188095 wait complete. 2024-04-22T15:42:03.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.981+0000 7f2406b8a700 1 Processor -- start 2024-04-22T15:42:03.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.981+0000 7f2406b8a700 1 -- start start 2024-04-22T15:42:03.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.982+0000 7f2406b8a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2400106820 0x7f240019bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:03.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.982+0000 7f2406b8a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240019c060 con 0x7f2400106820 2024-04-22T15:42:03.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.982+0000 7f2404926700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2400106820 0x7f240019bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:03.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.982+0000 7f2404926700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2400106820 0x7f240019bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.27:53226/0 (socket says 172.21.15.27:53226) 2024-04-22T15:42:03.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.982+0000 7f2404926700 1 -- 172.21.15.27:0/2456545044 learned_addr learned my addr 172.21.15.27:0/2456545044 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-22T15:42:03.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.982+0000 7f2404926700 1 -- 172.21.15.27:0/2456545044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e80095f0 con 0x7f2400106820 2024-04-22T15:42:03.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.982+0000 7f2404926700 1 --2- 172.21.15.27:0/2456545044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2400106820 0x7f240019bb20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f23e8005b40 tx=0x7f23e8004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:03.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.982+0000 7f23fdffb700 1 -- 172.21.15.27:0/2456545044 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23e801d070 con 0x7f2400106820 2024-04-22T15:42:03.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.983+0000 7f2406b8a700 1 -- 172.21.15.27:0/2456545044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f240019c260 con 0x7f2400106820 2024-04-22T15:42:03.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.983+0000 7f23fdffb700 1 -- 172.21.15.27:0/2456545044 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f23e80044d0 con 0x7f2400106820 2024-04-22T15:42:03.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.983+0000 7f23fdffb700 1 -- 172.21.15.27:0/2456545044 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23e8003910 con 0x7f2400106820 2024-04-22T15:42:03.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.983+0000 7f2406b8a700 1 -- 172.21.15.27:0/2456545044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2400195b10 con 0x7f2400106820 2024-04-22T15:42:03.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.984+0000 7f2406b8a700 1 -- 172.21.15.27:0/2456545044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24000623c0 con 0x7f2400106820 2024-04-22T15:42:03.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.984+0000 7f23fdffb700 1 -- 172.21.15.27:0/2456545044 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7f23e800b960 con 0x7f2400106820 2024-04-22T15:42:03.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.984+0000 7f23fdffb700 1 --2- 172.21.15.27:0/2456545044 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f23ec0384b0 0x7f23ec03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:03.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.984+0000 7f23fdffb700 1 -- 172.21.15.27:0/2456545044 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f23e804c4f0 con 0x7f2400106820 2024-04-22T15:42:03.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.984+0000 7f23fffff700 1 --2- 172.21.15.27:0/2456545044 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f23ec0384b0 0x7f23ec03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:03.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.985+0000 7f23fffff700 1 --2- 172.21.15.27:0/2456545044 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f23ec0384b0 0x7f23ec03a970 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f23f0006fd0 tx=0x7f23f0006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:03.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:03.989+0000 7f23fdffb700 1 -- 172.21.15.27:0/2456545044 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f23e802a540 con 0x7f2400106820 2024-04-22T15:42:04.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:04.276+0000 7f2406b8a700 1 -- 172.21.15.27:0/2456545044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f24001965a0 con 0x7f2400106820 2024-04-22T15:42:04.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:04.277+0000 7f23fdffb700 1 -- 172.21.15.27:0/2456545044 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f23e802a360 con 0x7f2400106820 2024-04-22T15:42:04.281 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:42:04.281 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":1,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","modified":"2024-04-22T15:39:17.702487Z","created":"2024-04-22T15:39:17.702487Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:42:04.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:04.279+0000 7f2406b8a700 1 -- 172.21.15.27:0/2456545044 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f23ec0384b0 msgr2=0x7f23ec03a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:04.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:04.279+0000 7f2406b8a700 1 --2- 172.21.15.27:0/2456545044 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f23ec0384b0 0x7f23ec03a970 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f23f0006fd0 tx=0x7f23f0006e40 comp rx=0 tx=0).stop 2024-04-22T15:42:04.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:04.279+0000 7f2406b8a700 1 -- 172.21.15.27:0/2456545044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2400106820 msgr2=0x7f240019bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:04.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:04.279+0000 7f2406b8a700 1 --2- 172.21.15.27:0/2456545044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2400106820 0x7f240019bb20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f23e8005b40 tx=0x7f23e8004c80 comp rx=0 tx=0).stop 2024-04-22T15:42:04.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:04.279+0000 7f2406b8a700 1 -- 172.21.15.27:0/2456545044 shutdown_connections 2024-04-22T15:42:04.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:04.279+0000 7f2406b8a700 1 --2- 172.21.15.27:0/2456545044 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f23ec0384b0 0x7f23ec03a970 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:04.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:04.279+0000 7f2406b8a700 1 --2- 172.21.15.27:0/2456545044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2400106820 0x7f240019bb20 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:04.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:04.279+0000 7f2406b8a700 1 -- 172.21.15.27:0/2456545044 >> 172.21.15.27:0/2456545044 conn(0x7f2400075290 msgr2=0x7f24000fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:04.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:04.279+0000 7f2406b8a700 1 -- 172.21.15.27:0/2456545044 shutdown_connections 2024-04-22T15:42:04.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:04.279+0000 7f2406b8a700 1 -- 172.21.15.27:0/2456545044 wait complete. 2024-04-22T15:42:04.284 INFO:teuthology.orchestra.run.smithi027.stderr:dumped monmap epoch 1 2024-04-22T15:42:04.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:04 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/2456545044' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:42:05.792 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:05 smithi016 ceph-mon[26620]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:05.793 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:05 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:05.793 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:05 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:05.793 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:05 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:05.793 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:05 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:05.793 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:05 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:42:05.793 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:05 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:05.793 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:05 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:06.005 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:42:06.005 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:42:06.757 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:42:06.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:06 smithi016 ceph-mon[26620]: Deploying daemon mon.smithi027 on smithi027 2024-04-22T15:42:07.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:07 smithi016 ceph-mon[26620]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:08.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.902+0000 7f9f72421700 1 -- 172.21.15.27:0/1039763574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f6c105780 msgr2=0x7f9f6c105b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:08.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.902+0000 7f9f72421700 1 --2- 172.21.15.27:0/1039763574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f6c105780 0x7f9f6c105b60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c009910 tx=0x7f9f5c009c20 comp rx=0 tx=0).stop 2024-04-22T15:42:08.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.902+0000 7f9f72421700 1 -- 172.21.15.27:0/1039763574 shutdown_connections 2024-04-22T15:42:08.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.902+0000 7f9f72421700 1 --2- 172.21.15.27:0/1039763574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f6c105780 0x7f9f6c105b60 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:08.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.902+0000 7f9f72421700 1 -- 172.21.15.27:0/1039763574 >> 172.21.15.27:0/1039763574 conn(0x7f9f6c075290 msgr2=0x7f9f6c0756a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:08.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.902+0000 7f9f72421700 1 -- 172.21.15.27:0/1039763574 shutdown_connections 2024-04-22T15:42:08.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.902+0000 7f9f72421700 1 -- 172.21.15.27:0/1039763574 wait complete. 2024-04-22T15:42:08.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.903+0000 7f9f72421700 1 Processor -- start 2024-04-22T15:42:08.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.903+0000 7f9f72421700 1 -- start start 2024-04-22T15:42:08.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.903+0000 7f9f72421700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f6c105780 0x7f9f6c067f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:08.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.903+0000 7f9f72421700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c1a41e0 con 0x7f9f6c105780 2024-04-22T15:42:08.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.904+0000 7f9f6bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f6c105780 0x7f9f6c067f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:08.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.904+0000 7f9f6bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f6c105780 0x7f9f6c067f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.27:53254/0 (socket says 172.21.15.27:53254) 2024-04-22T15:42:08.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.904+0000 7f9f6bfff700 1 -- 172.21.15.27:0/1681213525 learned_addr learned my addr 172.21.15.27:0/1681213525 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-22T15:42:08.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.904+0000 7f9f6bfff700 1 -- 172.21.15.27:0/1681213525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f5c0095f0 con 0x7f9f6c105780 2024-04-22T15:42:08.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.904+0000 7f9f6bfff700 1 --2- 172.21.15.27:0/1681213525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f6c105780 0x7f9f6c067f90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c005b40 tx=0x7f9f5c004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:08.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.904+0000 7f9f697fa700 1 -- 172.21.15.27:0/1681213525 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f5c01d070 con 0x7f9f6c105780 2024-04-22T15:42:08.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.904+0000 7f9f697fa700 1 -- 172.21.15.27:0/1681213525 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9f5c0044d0 con 0x7f9f6c105780 2024-04-22T15:42:08.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.904+0000 7f9f72421700 1 -- 172.21.15.27:0/1681213525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f6c1a43e0 con 0x7f9f6c105780 2024-04-22T15:42:08.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.904+0000 7f9f697fa700 1 -- 172.21.15.27:0/1681213525 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f5c003930 con 0x7f9f6c105780 2024-04-22T15:42:08.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.905+0000 7f9f72421700 1 -- 172.21.15.27:0/1681213525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f6c0686a0 con 0x7f9f6c105780 2024-04-22T15:42:08.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.906+0000 7f9f72421700 1 -- 172.21.15.27:0/1681213525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f6c1039c0 con 0x7f9f6c105780 2024-04-22T15:42:08.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.906+0000 7f9f697fa700 1 -- 172.21.15.27:0/1681213525 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7f9f5c003bf0 con 0x7f9f6c105780 2024-04-22T15:42:08.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.906+0000 7f9f697fa700 1 --2- 172.21.15.27:0/1681213525 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f9f580384b0 0x7f9f5803a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:08.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.906+0000 7f9f697fa700 1 -- 172.21.15.27:0/1681213525 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f9f5c04c470 con 0x7f9f6c105780 2024-04-22T15:42:08.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.906+0000 7f9f6b7fe700 1 --2- 172.21.15.27:0/1681213525 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f9f580384b0 0x7f9f5803a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:08.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.907+0000 7f9f6b7fe700 1 --2- 172.21.15.27:0/1681213525 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f9f580384b0 0x7f9f5803a970 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9f6400ad30 tx=0x7f9f640093f0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:08.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:08.911+0000 7f9f697fa700 1 -- 172.21.15.27:0/1681213525 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9f5c02a950 con 0x7f9f6c105780 2024-04-22T15:42:09.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:09.215+0000 7f9f72421700 1 -- 172.21.15.27:0/1681213525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9f6c0623c0 con 0x7f9f6c105780 2024-04-22T15:42:09.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:09.216+0000 7f9f697fa700 1 -- 172.21.15.27:0/1681213525 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f9f5c02a360 con 0x7f9f6c105780 2024-04-22T15:42:09.220 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:42:09.220 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":1,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","modified":"2024-04-22T15:39:17.702487Z","created":"2024-04-22T15:39:17.702487Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T15:42:09.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:09.218+0000 7f9f72421700 1 -- 172.21.15.27:0/1681213525 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f9f580384b0 msgr2=0x7f9f5803a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:09.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:09.218+0000 7f9f72421700 1 --2- 172.21.15.27:0/1681213525 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f9f580384b0 0x7f9f5803a970 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9f6400ad30 tx=0x7f9f640093f0 comp rx=0 tx=0).stop 2024-04-22T15:42:09.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:09.218+0000 7f9f72421700 1 -- 172.21.15.27:0/1681213525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f6c105780 msgr2=0x7f9f6c067f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:09.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:09.218+0000 7f9f72421700 1 --2- 172.21.15.27:0/1681213525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f6c105780 0x7f9f6c067f90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c005b40 tx=0x7f9f5c004c80 comp rx=0 tx=0).stop 2024-04-22T15:42:09.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:09.218+0000 7f9f72421700 1 -- 172.21.15.27:0/1681213525 shutdown_connections 2024-04-22T15:42:09.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:09.218+0000 7f9f72421700 1 --2- 172.21.15.27:0/1681213525 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f9f580384b0 0x7f9f5803a970 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:09.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:09.218+0000 7f9f72421700 1 --2- 172.21.15.27:0/1681213525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f6c105780 0x7f9f6c067f90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:09.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:09.218+0000 7f9f72421700 1 -- 172.21.15.27:0/1681213525 >> 172.21.15.27:0/1681213525 conn(0x7f9f6c075290 msgr2=0x7f9f6c0fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:09.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:09.218+0000 7f9f72421700 1 -- 172.21.15.27:0/1681213525 shutdown_connections 2024-04-22T15:42:09.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:09.218+0000 7f9f72421700 1 -- 172.21.15.27:0/1681213525 wait complete. 2024-04-22T15:42:09.223 INFO:teuthology.orchestra.run.smithi027.stderr:dumped monmap epoch 1 2024-04-22T15:42:09.327 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 podman[34886]: 2024-04-22 15:42:09.301678603 +0000 UTC m=+0.359522001 container start 9f21ce7fa7efbcb8069a76aea9e8eb14b09826943ef4eb05c8f00ff39f6f02e8 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi027, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-22T15:42:09.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T15:42:09.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable), process ceph-mon, pid 2 2024-04-22T15:42:09.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: pidfile_write: ignore empty --pid-file 2024-04-22T15:42:09.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: load: jerasure load: lrc 2024-04-22T15:42:09.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: RocksDB version: 7.9.2 2024-04-22T15:42:09.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Git sha 0 2024-04-22T15:42:09.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Compile date 2023-12-11 22:07:34 2024-04-22T15:42:09.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: DB SUMMARY 2024-04-22T15:42:09.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: DB Session ID: JZ3D9OVZXCV8W046QA7N 2024-04-22T15:42:09.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: CURRENT file: CURRENT 2024-04-22T15:42:09.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T15:42:09.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-22T15:42:09.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi027/store.db dir, Total Num: 0, files: 2024-04-22T15:42:09.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi027/store.db: 000004.log size: 511 ; 2024-04-22T15:42:09.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.error_if_exists: 0 2024-04-22T15:42:09.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.create_if_missing: 0 2024-04-22T15:42:09.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.paranoid_checks: 1 2024-04-22T15:42:09.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-22T15:42:09.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T15:42:09.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-22T15:42:09.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.env: 0x5600e69a7720 2024-04-22T15:42:09.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.fs: PosixFileSystem 2024-04-22T15:42:09.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.info_log: 0x5600e86dd2e0 2024-04-22T15:42:09.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T15:42:09.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.statistics: (nil) 2024-04-22T15:42:09.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.use_fsync: 0 2024-04-22T15:42:09.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_log_file_size: 0 2024-04-22T15:42:09.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T15:42:09.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T15:42:09.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T15:42:09.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T15:42:09.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.allow_fallocate: 1 2024-04-22T15:42:09.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T15:42:09.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T15:42:09.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.use_direct_reads: 0 2024-04-22T15:42:09.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T15:42:09.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T15:42:09.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.db_log_dir: 2024-04-22T15:42:09.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.wal_dir: 2024-04-22T15:42:09.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T15:42:09.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T15:42:09.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T15:42:09.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T15:42:09.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T15:42:09.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T15:42:09.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T15:42:09.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T15:42:09.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.write_buffer_manager: 0x5600e794e140 2024-04-22T15:42:09.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T15:42:09.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T15:42:09.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T15:42:09.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T15:42:09.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T15:42:09.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T15:42:09.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T15:42:09.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T15:42:09.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.unordered_write: 0 2024-04-22T15:42:09.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T15:42:09.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T15:42:09.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T15:42:09.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T15:42:09.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.row_cache: None 2024-04-22T15:42:09.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.wal_filter: None 2024-04-22T15:42:09.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T15:42:09.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T15:42:09.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.two_write_queues: 0 2024-04-22T15:42:09.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T15:42:09.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.wal_compression: 0 2024-04-22T15:42:09.593 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.atomic_flush: 0 2024-04-22T15:42:09.593 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T15:42:09.593 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T15:42:09.593 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T15:42:09.593 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.log_readahead_size: 0 2024-04-22T15:42:09.593 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T15:42:09.593 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T15:42:09.593 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T15:42:09.593 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T15:42:09.593 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T15:42:09.593 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T15:42:09.593 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-22T15:42:09.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_background_jobs: 2 2024-04-22T15:42:09.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_background_compactions: -1 2024-04-22T15:42:09.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_subcompactions: 1 2024-04-22T15:42:09.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T15:42:09.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T15:42:09.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T15:42:09.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T15:42:09.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T15:42:09.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T15:42:09.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T15:42:09.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T15:42:09.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_open_files: -1 2024-04-22T15:42:09.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T15:42:09.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T15:42:09.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T15:42:09.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T15:42:09.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_background_flushes: -1 2024-04-22T15:42:09.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Compression algorithms supported: 2024-04-22T15:42:09.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T15:42:09.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: kZSTD supported: 0 2024-04-22T15:42:09.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: kXpressCompression supported: 0 2024-04-22T15:42:09.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T15:42:09.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: kZlibCompression supported: 1 2024-04-22T15:42:09.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: kSnappyCompression supported: 1 2024-04-22T15:42:09.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: kLZ4Compression supported: 1 2024-04-22T15:42:09.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: kBZip2Compression supported: 0 2024-04-22T15:42:09.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T15:42:09.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-22T15:42:09.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi027/store.db/MANIFEST-000005 2024-04-22T15:42:09.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-22T15:42:09.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T15:42:09.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.merge_operator: 2024-04-22T15:42:09.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.compaction_filter: None 2024-04-22T15:42:09.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T15:42:09.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T15:42:09.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T15:42:09.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T15:42:09.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5600e86dd3e0) 2024-04-22T15:42:09.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: cache_index_and_filter_blocks: 1 2024-04-22T15:42:09.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T15:42:09.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T15:42:09.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: pin_top_level_index_and_filter: 1 2024-04-22T15:42:09.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: index_type: 0 2024-04-22T15:42:09.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: data_block_index_type: 0 2024-04-22T15:42:09.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: index_shortening: 1 2024-04-22T15:42:09.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T15:42:09.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: checksum: 4 2024-04-22T15:42:09.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: no_block_cache: 0 2024-04-22T15:42:09.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_cache: 0x5600e79ef350 2024-04-22T15:42:09.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_cache_name: BinnedLRUCache 2024-04-22T15:42:09.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_cache_options: 2024-04-22T15:42:09.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: capacity : 536870912 2024-04-22T15:42:09.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: num_shard_bits : 4 2024-04-22T15:42:09.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: strict_capacity_limit : 0 2024-04-22T15:42:09.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: high_pri_pool_ratio: 0.000 2024-04-22T15:42:09.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_cache_compressed: (nil) 2024-04-22T15:42:09.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: persistent_cache: (nil) 2024-04-22T15:42:09.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_size: 4096 2024-04-22T15:42:09.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_size_deviation: 10 2024-04-22T15:42:09.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_restart_interval: 16 2024-04-22T15:42:09.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: index_block_restart_interval: 1 2024-04-22T15:42:09.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: metadata_block_size: 4096 2024-04-22T15:42:09.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: partition_filters: 0 2024-04-22T15:42:09.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: use_delta_encoding: 1 2024-04-22T15:42:09.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: filter_policy: bloomfilter 2024-04-22T15:42:09.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: whole_key_filtering: 1 2024-04-22T15:42:09.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: verify_compression: 0 2024-04-22T15:42:09.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: read_amp_bytes_per_bit: 0 2024-04-22T15:42:09.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: format_version: 5 2024-04-22T15:42:09.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: enable_index_compression: 1 2024-04-22T15:42:09.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_align: 0 2024-04-22T15:42:09.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: max_auto_readahead_size: 262144 2024-04-22T15:42:09.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: prepopulate_block_cache: 0 2024-04-22T15:42:09.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: initial_auto_readahead_size: 8192 2024-04-22T15:42:09.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: num_file_reads_for_auto_readahead: 2 2024-04-22T15:42:09.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T15:42:09.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T15:42:09.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.compression: NoCompression 2024-04-22T15:42:09.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T15:42:09.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T15:42:09.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T15:42:09.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.num_levels: 7 2024-04-22T15:42:09.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T15:42:09.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T15:42:09.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T15:42:09.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T15:42:09.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T15:42:09.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T15:42:09.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T15:42:09.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T15:42:09.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T15:42:09.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T15:42:09.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-22T15:42:09.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-22T15:42:09.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T15:42:09.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T15:42:09.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T15:42:09.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T15:42:09.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T15:42:09.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-22T15:42:09.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T15:42:09.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T15:42:09.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-22T15:42:09.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T15:42:09.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T15:42:09.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T15:42:09.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T15:42:09.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T15:42:09.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T15:42:09.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T15:42:09.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T15:42:09.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T15:42:09.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T15:42:09.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T15:42:09.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T15:42:09.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T15:42:09.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T15:42:09.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T15:42:09.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T15:42:09.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T15:42:09.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-22T15:42:09.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.arena_block_size: 1048576 2024-04-22T15:42:09.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T15:42:09.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T15:42:09.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T15:42:09.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T15:42:09.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T15:42:09.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T15:42:09.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T15:42:09.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T15:42:09.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T15:42:09.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T15:42:09.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T15:42:09.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T15:42:09.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T15:42:09.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.table_properties_collectors: 2024-04-22T15:42:09.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.inplace_update_support: 0 2024-04-22T15:42:09.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T15:42:09.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T15:42:09.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T15:42:09.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T15:42:09.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.bloom_locality: 0 2024-04-22T15:42:09.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.max_successive_merges: 0 2024-04-22T15:42:09.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T15:42:09.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T15:42:09.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T15:42:09.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T15:42:09.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.ttl: 2592000 2024-04-22T15:42:09.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T15:42:09.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-22T15:42:09.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-22T15:42:09.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.enable_blob_files: false 2024-04-22T15:42:09.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.min_blob_size: 0 2024-04-22T15:42:09.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T15:42:09.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T15:42:09.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T15:42:09.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T15:42:09.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-22T15:42:09.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-22T15:42:09.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.blob_file_starting_level: 0 2024-04-22T15:42:09.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-22T15:42:09.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi027/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-22T15:42:09.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-22T15:42:09.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7cea9a7e-595e-4cf5-874d-5063d0cedd5d 2024-04-22T15:42:09.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713800529331020, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-22T15:42:09.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-22T15:42:09.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713800529331441, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713800529, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7cea9a7e-595e-4cf5-874d-5063d0cedd5d", "db_session_id": "JZ3D9OVZXCV8W046QA7N", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-22T15:42:09.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713800529331522, "job": 1, "event": "recovery_finished"} 2024-04-22T15:42:09.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-22T15:42:09.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi027/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T15:42:09.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5600e7a8c000 2024-04-22T15:42:09.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: DB pointer 0x5600e7a78000 2024-04-22T15:42:09.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-22T15:42:09.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-22T15:42:09.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ** DB Stats ** 2024-04-22T15:42:09.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T15:42:09.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T15:42:09.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T15:42:09.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T15:42:09.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T15:42:09.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T15:42:09.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T15:42:09.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-22T15:42:09.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ** Compaction Stats [default] ** 2024-04-22T15:42:09.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T15:42:09.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-22T15:42:09.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T15:42:09.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T15:42:09.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T15:42:09.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-22T15:42:09.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ** Compaction Stats [default] ** 2024-04-22T15:42:09.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T15:42:09.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T15:42:09.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T15:42:09.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-22T15:42:09.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-22T15:42:09.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-22T15:42:09.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T15:42:09.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T15:42:09.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T15:42:09.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T15:42:09.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T15:42:09.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T15:42:09.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T15:42:09.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T15:42:09.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T15:42:09.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Block cache BinnedLRUCache@0x5600e79ef350#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-22T15:42:09.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-22T15:42:09.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-22T15:42:09.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T15:42:09.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: mon.smithi027 does not exist in monmap, will attempt to join an existing cluster 2024-04-22T15:42:09.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: using public_addr v2:172.21.15.27:0/0 -> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] 2024-04-22T15:42:09.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: starting mon.smithi027 rank -1 at public addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] at bind addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi027 fsid 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T15:42:09.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: mon.smithi027@-1(???) e0 preinit fsid 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T15:42:09.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: mon.smithi027@-1(synchronizing).mds e1 new map 2024-04-22T15:42:09.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: mon.smithi027@-1(synchronizing).mds e1 print_map 2024-04-22T15:42:09.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: e1 2024-04-22T15:42:09.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:42:09.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:42:09.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: legacy client fscid: -1 2024-04-22T15:42:09.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-22T15:42:09.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: No filesystems configured 2024-04-22T15:42:09.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: mon.smithi027@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-22T15:42:09.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: mon.smithi027@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-22T15:42:09.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: mon.smithi027@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-22T15:42:09.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: mon.smithi027@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-22T15:42:09.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: mon.smithi027@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-22T15:42:09.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: mon.smithi027@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-22T15:42:09.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: mon.smithi027@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-04-22T15:42:09.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: mon.smithi027@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T15:42:09.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: mon.smithi027@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T15:42:09.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: mon.smithi027@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T15:42:09.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: mon.smithi027@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T15:42:09.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:09.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:09.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:09.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:09.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:09.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:09.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi027.ukdgzs", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:42:09.612 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi027.ukdgzs", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T15:42:09.612 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:42:09.612 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:09.612 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: Deploying daemon mgr.smithi027.ukdgzs on smithi027 2024-04-22T15:42:09.612 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:09.612 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: from='client.? 172.21.15.27:0/2456545044' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:42:09.612 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:09.612 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:09.612 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:09.612 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:09.612 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:09.612 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:42:09.612 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:09.612 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:09.613 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: Deploying daemon mon.smithi027 on smithi027 2024-04-22T15:42:09.613 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:09.613 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: mon.smithi027@-1(synchronizing).paxosservice(auth 1..8) refresh upgraded, format 0 -> 3 2024-04-22T15:42:09.613 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: mon.smithi027@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-04-22T15:42:09.613 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: expand_channel_meta expand map: {default=false} 2024-04-22T15:42:09.613 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: expand_channel_meta from 'false' to 'false' 2024-04-22T15:42:09.613 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: expand_channel_meta expanded map: {default=false} 2024-04-22T15:42:09.613 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: expand_channel_meta expand map: {default=info} 2024-04-22T15:42:09.613 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: expand_channel_meta from 'info' to 'info' 2024-04-22T15:42:09.613 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: expand_channel_meta expanded map: {default=info} 2024-04-22T15:42:09.613 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: expand_channel_meta expand map: {default=daemon} 2024-04-22T15:42:09.613 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: expand_channel_meta from 'daemon' to 'daemon' 2024-04-22T15:42:09.613 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: expand_channel_meta expanded map: {default=daemon} 2024-04-22T15:42:09.613 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: expand_channel_meta expand map: {default=debug} 2024-04-22T15:42:09.613 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: expand_channel_meta from 'debug' to 'debug' 2024-04-22T15:42:09.614 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 ceph-mon[34922]: expand_channel_meta expanded map: {default=debug} 2024-04-22T15:42:09.927 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 bash[34886]: 9f21ce7fa7efbcb8069a76aea9e8eb14b09826943ef4eb05c8f00ff39f6f02e8 2024-04-22T15:42:09.927 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:09 smithi027 systemd[1]: Started Ceph mon.smithi027 for 67ab3a80-00be-11ef-bc93-c7b262605968. 2024-04-22T15:42:10.985 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T15:42:10.985 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T15:42:11.195 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi027/config 2024-04-22T15:42:14.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-04-22T15:42:14.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: mon.smithi016 calling monitor election 2024-04-22T15:42:14.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:42:14.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:42:14.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:14.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:42:14.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:42:14.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: mon.smithi027 calling monitor election 2024-04-22T15:42:14.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.27:0/2992446063' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.ukdgzs/crt"}]: dispatch 2024-04-22T15:42:14.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:42:14.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:14.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:42:14.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:42:14.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: mon.smithi016 is new leader, mons smithi016,smithi027 in quorum (ranks 0,1) 2024-04-22T15:42:14.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: monmap e2: 2 mons at {smithi016=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],smithi027=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-22T15:42:14.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: fsmap 2024-04-22T15:42:14.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: osdmap e5: 0 total, 0 up, 0 in 2024-04-22T15:42:14.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: mgrmap e17: smithi016.rnsnnr(active, since 33s) 2024-04-22T15:42:14.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: Standby manager daemon smithi027.ukdgzs started 2024-04-22T15:42:14.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.27:0/2992446063' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:42:14.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: overall HEALTH_OK 2024-04-22T15:42:14.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.27:0/2992446063' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.ukdgzs/key"}]: dispatch 2024-04-22T15:42:14.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:14.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.27:0/2992446063' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:42:14.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:14.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:14.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:14.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:14 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:14.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-04-22T15:42:14.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: mon.smithi016 calling monitor election 2024-04-22T15:42:14.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:42:14.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:42:14.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:14.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:42:14.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:42:14.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: mon.smithi027 calling monitor election 2024-04-22T15:42:14.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: from='mgr.? 172.21.15.27:0/2992446063' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.ukdgzs/crt"}]: dispatch 2024-04-22T15:42:14.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:42:14.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:14.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:42:14.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:42:14.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: mon.smithi016 is new leader, mons smithi016,smithi027 in quorum (ranks 0,1) 2024-04-22T15:42:14.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: monmap e2: 2 mons at {smithi016=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],smithi027=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-22T15:42:14.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: fsmap 2024-04-22T15:42:14.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: osdmap e5: 0 total, 0 up, 0 in 2024-04-22T15:42:14.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: mgrmap e17: smithi016.rnsnnr(active, since 33s) 2024-04-22T15:42:14.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: Standby manager daemon smithi027.ukdgzs started 2024-04-22T15:42:14.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: from='mgr.? 172.21.15.27:0/2992446063' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:42:14.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: overall HEALTH_OK 2024-04-22T15:42:14.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: from='mgr.? 172.21.15.27:0/2992446063' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.ukdgzs/key"}]: dispatch 2024-04-22T15:42:14.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:14.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: from='mgr.? 172.21.15.27:0/2992446063' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:42:14.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:14.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:14.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:14.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:14 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:15.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.570+0000 7fbd147f9700 1 -- 172.21.15.27:0/324639899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcf80035c0 msgr2=0x7fbcf8005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:15.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.570+0000 7fbd147f9700 1 --2- 172.21.15.27:0/324639899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcf80035c0 0x7fbcf8005a50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c0ff770 tx=0x7fbd0800feb0 comp rx=0 tx=0).stop 2024-04-22T15:42:15.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.571+0000 7fbd147f9700 1 -- 172.21.15.27:0/324639899 shutdown_connections 2024-04-22T15:42:15.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.571+0000 7fbd147f9700 1 --2- 172.21.15.27:0/324639899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcf80035c0 0x7fbcf8005a50 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:15.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.571+0000 7fbd147f9700 1 -- 172.21.15.27:0/324639899 >> 172.21.15.27:0/324639899 conn(0x7fbd0c0fb350 msgr2=0x7fbd0c0fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:15.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.571+0000 7fbd147f9700 1 -- 172.21.15.27:0/324639899 shutdown_connections 2024-04-22T15:42:15.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.571+0000 7fbd147f9700 1 -- 172.21.15.27:0/324639899 wait complete. 2024-04-22T15:42:15.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.572+0000 7fbd147f9700 1 Processor -- start 2024-04-22T15:42:15.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.572+0000 7fbd147f9700 1 -- start start 2024-04-22T15:42:15.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.572+0000 7fbd147f9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcf80035c0 0x7fbd0c196e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:15.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.572+0000 7fbd147f9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c197340 0x7fbd0c1977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:15.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.572+0000 7fbd147f9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c19fe70 con 0x7fbcf80035c0 2024-04-22T15:42:15.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.572+0000 7fbd147f9700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c19ffb0 con 0x7fbd0c197340 2024-04-22T15:42:15.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.572+0000 7fbd11d94700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c197340 0x7fbd0c1977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:15.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.572+0000 7fbd11d94700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c197340 0x7fbd0c1977c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38110/0 (socket says 172.21.15.27:38110) 2024-04-22T15:42:15.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.572+0000 7fbd11d94700 1 -- 172.21.15.27:0/4182495234 learned_addr learned my addr 172.21.15.27:0/4182495234 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-22T15:42:15.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.572+0000 7fbd12595700 1 --2- 172.21.15.27:0/4182495234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcf80035c0 0x7fbd0c196e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:15.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.572+0000 7fbd11d94700 1 -- 172.21.15.27:0/4182495234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c197340 msgr2=0x7fbd0c1977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-22T15:42:15.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.572+0000 7fbd11d94700 1 -- 172.21.15.27:0/4182495234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c197340 msgr2=0x7fbd0c1977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T15:42:15.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.572+0000 7fbd11d94700 1 --2- 172.21.15.27:0/4182495234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c197340 0x7fbd0c1977c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T15:42:15.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.572+0000 7fbd11d94700 1 --2- 172.21.15.27:0/4182495234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c197340 0x7fbd0c1977c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:42:15.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.573+0000 7fbd12595700 1 -- 172.21.15.27:0/4182495234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c197340 msgr2=0x7fbd0c1977c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T15:42:15.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.573+0000 7fbd12595700 1 --2- 172.21.15.27:0/4182495234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c197340 0x7fbd0c1977c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:15.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.573+0000 7fbd12595700 1 -- 172.21.15.27:0/4182495234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd0800f900 con 0x7fbcf80035c0 2024-04-22T15:42:15.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.573+0000 7fbd12595700 1 --2- 172.21.15.27:0/4182495234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcf80035c0 0x7fbd0c196e00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbd0800f800 tx=0x7fbd08011590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:15.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.573+0000 7fbd037fe700 1 -- 172.21.15.27:0/4182495234 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd08013b30 con 0x7fbcf80035c0 2024-04-22T15:42:15.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.573+0000 7fbd037fe700 1 -- 172.21.15.27:0/4182495234 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbd08004b70 con 0x7fbcf80035c0 2024-04-22T15:42:15.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.573+0000 7fbd147f9700 1 -- 172.21.15.27:0/4182495234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd0c1a01b0 con 0x7fbcf80035c0 2024-04-22T15:42:15.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.573+0000 7fbd147f9700 1 -- 172.21.15.27:0/4182495234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd0c1a06d0 con 0x7fbcf80035c0 2024-04-22T15:42:15.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.573+0000 7fbd037fe700 1 -- 172.21.15.27:0/4182495234 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd08011c60 con 0x7fbcf80035c0 2024-04-22T15:42:15.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.574+0000 7fbd147f9700 1 -- 172.21.15.27:0/4182495234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf8002130 con 0x7fbcf80035c0 2024-04-22T15:42:15.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.575+0000 7fbd037fe700 1 -- 172.21.15.27:0/4182495234 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 90256+0+0 (secure 0 0 0) 0x7fbd08003680 con 0x7fbcf80035c0 2024-04-22T15:42:15.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.575+0000 7fbd037fe700 1 --2- 172.21.15.27:0/4182495234 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fbcf006c5b0 0x7fbcf006ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:15.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.575+0000 7fbd037fe700 1 -- 172.21.15.27:0/4182495234 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fbd0808d160 con 0x7fbcf80035c0 2024-04-22T15:42:15.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.575+0000 7fbd11d94700 1 --2- 172.21.15.27:0/4182495234 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fbcf006c5b0 0x7fbcf006ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:15.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.576+0000 7fbd11d94700 1 --2- 172.21.15.27:0/4182495234 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fbcf006c5b0 0x7fbcf006ea70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c069e60 tx=0x7fbcfc008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:15.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.580+0000 7fbd037fe700 1 -- 172.21.15.27:0/4182495234 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fbd08058b10 con 0x7fbcf80035c0 2024-04-22T15:42:15.808 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:15 smithi016 ceph-mon[26620]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:15.808 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:15 smithi016 ceph-mon[26620]: mgrmap e18: smithi016.rnsnnr(active, since 34s), standbys: smithi027.ukdgzs 2024-04-22T15:42:15.808 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:15 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr metadata", "who": "smithi027.ukdgzs", "id": "smithi027.ukdgzs"}]: dispatch 2024-04-22T15:42:15.808 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:15 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:42:15.808 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:15 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:42:15.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:15 smithi027 ceph-mon[34922]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:15.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:15 smithi027 ceph-mon[34922]: mgrmap e18: smithi016.rnsnnr(active, since 34s), standbys: smithi027.ukdgzs 2024-04-22T15:42:15.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:15 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr metadata", "who": "smithi027.ukdgzs", "id": "smithi027.ukdgzs"}]: dispatch 2024-04-22T15:42:15.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:15 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:42:15.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:15 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:42:15.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.874+0000 7fbd147f9700 1 -- 172.21.15.27:0/4182495234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbcf8003010 con 0x7fbcf80035c0 2024-04-22T15:42:15.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.875+0000 7fbd037fe700 1 -- 172.21.15.27:0/4182495234 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1018 (secure 0 0 0) 0x7fbd08031410 con 0x7fbcf80035c0 2024-04-22T15:42:15.878 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-22T15:42:15.878 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":2,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","modified":"2024-04-22T15:42:09.358969Z","created":"2024-04-22T15:39:17.702487Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-22T15:42:15.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.877+0000 7fbd147f9700 1 -- 172.21.15.27:0/4182495234 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fbcf006c5b0 msgr2=0x7fbcf006ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.877+0000 7fbd147f9700 1 --2- 172.21.15.27:0/4182495234 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fbcf006c5b0 0x7fbcf006ea70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c069e60 tx=0x7fbcfc008040 comp rx=0 tx=0).stop 2024-04-22T15:42:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.877+0000 7fbd147f9700 1 -- 172.21.15.27:0/4182495234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcf80035c0 msgr2=0x7fbd0c196e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.877+0000 7fbd147f9700 1 --2- 172.21.15.27:0/4182495234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcf80035c0 0x7fbd0c196e00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbd0800f800 tx=0x7fbd08011590 comp rx=0 tx=0).stop 2024-04-22T15:42:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.877+0000 7fbd147f9700 1 -- 172.21.15.27:0/4182495234 shutdown_connections 2024-04-22T15:42:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.877+0000 7fbd147f9700 1 --2- 172.21.15.27:0/4182495234 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fbcf006c5b0 0x7fbcf006ea70 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.877+0000 7fbd147f9700 1 --2- 172.21.15.27:0/4182495234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcf80035c0 0x7fbd0c196e00 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.877+0000 7fbd147f9700 1 --2- 172.21.15.27:0/4182495234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c197340 0x7fbd0c1977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.877+0000 7fbd147f9700 1 -- 172.21.15.27:0/4182495234 >> 172.21.15.27:0/4182495234 conn(0x7fbd0c0fb350 msgr2=0x7fbd0c18ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.877+0000 7fbd147f9700 1 -- 172.21.15.27:0/4182495234 shutdown_connections 2024-04-22T15:42:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:42:15.877+0000 7fbd147f9700 1 -- 172.21.15.27:0/4182495234 wait complete. 2024-04-22T15:42:15.882 INFO:teuthology.orchestra.run.smithi027.stderr:dumped monmap epoch 2 2024-04-22T15:42:16.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:16 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:16.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:16 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/4182495234' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:42:16.696 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:16 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:16.697 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:16 smithi027 ceph-mon[34922]: from='client.? 172.21.15.27:0/4182495234' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T15:42:16.722 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-22T15:42:16.722 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-22T15:42:16.945 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:42:17.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:17 smithi016 ceph-mon[26620]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:17.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:17 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:17.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:17 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:17.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:17 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:17.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:17 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:42:17.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:17 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:17.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:17 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:17.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:17 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:17.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:17 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:17.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:17 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:17.810 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:17 smithi027 ceph-mon[34922]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:17.810 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:17 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:17.810 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:17 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:17.810 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:17 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:17.811 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:17 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:42:17.811 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:17 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:17.811 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:17 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:17.811 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:17 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:17.811 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:17 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:17.811 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:17 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:18.638 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:18 smithi016 ceph-mon[26620]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-22T15:42:18.638 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:18 smithi016 ceph-mon[26620]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-22T15:42:18.638 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:18 smithi016 ceph-mon[26620]: Updating smithi016:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:42:18.638 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:18 smithi016 ceph-mon[26620]: Updating smithi027:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:42:18.638 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:18 smithi016 ceph-mon[26620]: Reconfiguring mon.smithi016 (unknown last config time)... 2024-04-22T15:42:18.639 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:18 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:42:18.639 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:18 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T15:42:18.639 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:18 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:18.639 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:18 smithi016 ceph-mon[26620]: Reconfiguring daemon mon.smithi016 on smithi016 2024-04-22T15:42:18.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:18 smithi027 ceph-mon[34922]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-22T15:42:18.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:18 smithi027 ceph-mon[34922]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-22T15:42:18.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:18 smithi027 ceph-mon[34922]: Updating smithi016:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:42:18.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:18 smithi027 ceph-mon[34922]: Updating smithi027:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:42:18.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:18 smithi027 ceph-mon[34922]: Reconfiguring mon.smithi016 (unknown last config time)... 2024-04-22T15:42:18.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:18 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:42:18.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:18 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T15:42:18.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:18 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:18.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:18 smithi027 ceph-mon[34922]: Reconfiguring daemon mon.smithi016 on smithi016 2024-04-22T15:42:19.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.584+0000 7f2f7c8ac700 1 -- 172.21.15.16:0/2116402062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f780fff50 msgr2=0x7f2f78105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:19.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.584+0000 7f2f7c8ac700 1 --2- 172.21.15.16:0/2116402062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f780fff50 0x7f2f78105840 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c009910 tx=0x7f2f6c009c20 comp rx=0 tx=0).stop 2024-04-22T15:42:19.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.584+0000 7f2f7c8ac700 1 -- 172.21.15.16:0/2116402062 shutdown_connections 2024-04-22T15:42:19.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.584+0000 7f2f7c8ac700 1 --2- 172.21.15.16:0/2116402062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f780fff50 0x7f2f78105840 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:19.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.584+0000 7f2f7c8ac700 1 --2- 172.21.15.16:0/2116402062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f780ff5a0 0x7f2f780ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:19.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.584+0000 7f2f7c8ac700 1 -- 172.21.15.16:0/2116402062 >> 172.21.15.16:0/2116402062 conn(0x7f2f78074e30 msgr2=0x7f2f78075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:19.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.584+0000 7f2f7c8ac700 1 -- 172.21.15.16:0/2116402062 shutdown_connections 2024-04-22T15:42:19.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.584+0000 7f2f7c8ac700 1 -- 172.21.15.16:0/2116402062 wait complete. 2024-04-22T15:42:19.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.584+0000 7f2f7c8ac700 1 Processor -- start 2024-04-22T15:42:19.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.585+0000 7f2f7c8ac700 1 -- start start 2024-04-22T15:42:19.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.585+0000 7f2f7c8ac700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f780ff5a0 0x7f2f7819c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:19.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.585+0000 7f2f7c8ac700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f780fff50 0x7f2f7819cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:19.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.585+0000 7f2f7c8ac700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7819d2a0 con 0x7f2f780ff5a0 2024-04-22T15:42:19.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.585+0000 7f2f7c8ac700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f78196750 con 0x7f2f780fff50 2024-04-22T15:42:19.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.585+0000 7f2f7659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f780ff5a0 0x7f2f7819c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:19.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.585+0000 7f2f7659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f780ff5a0 0x7f2f7819c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60184/0 (socket says 172.21.15.16:60184) 2024-04-22T15:42:19.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.585+0000 7f2f7659c700 1 -- 172.21.15.16:0/1422719585 learned_addr learned my addr 172.21.15.16:0/1422719585 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:42:19.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.585+0000 7f2f75d9b700 1 --2- 172.21.15.16:0/1422719585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f780fff50 0x7f2f7819cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:19.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.585+0000 7f2f7659c700 1 -- 172.21.15.16:0/1422719585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f780fff50 msgr2=0x7f2f7819cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:19.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.585+0000 7f2f7659c700 1 --2- 172.21.15.16:0/1422719585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f780fff50 0x7f2f7819cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:19.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.585+0000 7f2f7659c700 1 -- 172.21.15.16:0/1422719585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6c0095f0 con 0x7f2f780ff5a0 2024-04-22T15:42:19.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.585+0000 7f2f7659c700 1 --2- 172.21.15.16:0/1422719585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f780ff5a0 0x7f2f7819c6d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2f6800b4b0 tx=0x7f2f6800b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:19.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.586+0000 7f2f677fe700 1 -- 172.21.15.16:0/1422719585 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f68011800 con 0x7f2f780ff5a0 2024-04-22T15:42:19.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.586+0000 7f2f7c8ac700 1 -- 172.21.15.16:0/1422719585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f78196a30 con 0x7f2f780ff5a0 2024-04-22T15:42:19.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.586+0000 7f2f677fe700 1 -- 172.21.15.16:0/1422719585 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2f68011e10 con 0x7f2f780ff5a0 2024-04-22T15:42:19.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.586+0000 7f2f677fe700 1 -- 172.21.15.16:0/1422719585 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f6800f2e0 con 0x7f2f780ff5a0 2024-04-22T15:42:19.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.586+0000 7f2f7c8ac700 1 -- 172.21.15.16:0/1422719585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f78196f80 con 0x7f2f780ff5a0 2024-04-22T15:42:19.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.587+0000 7f2f7c8ac700 1 -- 172.21.15.16:0/1422719585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f58005320 con 0x7f2f780ff5a0 2024-04-22T15:42:19.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.587+0000 7f2f677fe700 1 -- 172.21.15.16:0/1422719585 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 90256+0+0 (secure 0 0 0) 0x7f2f6800f510 con 0x7f2f780ff5a0 2024-04-22T15:42:19.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.588+0000 7f2f677fe700 1 --2- 172.21.15.16:0/1422719585 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f2f6006c470 0x7f2f6006e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:19.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.588+0000 7f2f677fe700 1 -- 172.21.15.16:0/1422719585 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f2f6808a200 con 0x7f2f780ff5a0 2024-04-22T15:42:19.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.588+0000 7f2f75d9b700 1 --2- 172.21.15.16:0/1422719585 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f2f6006c470 0x7f2f6006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:19.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.588+0000 7f2f75d9b700 1 --2- 172.21.15.16:0/1422719585 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f2f6006c470 0x7f2f6006e930 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2f78197bc0 tx=0x7f2f6c01a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:19.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.592+0000 7f2f677fe700 1 -- 172.21.15.16:0/1422719585 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2f68059550 con 0x7f2f780ff5a0 2024-04-22T15:42:19.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:19 smithi016 ceph-mon[26620]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:19.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:19 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:19.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:19 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:19.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:19 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi016.rnsnnr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:42:19.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:19 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:42:19.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:19 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.800+0000 7f2f7c8ac700 1 -- 172.21.15.16:0/1422719585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f2f580059f0 con 0x7f2f780ff5a0 2024-04-22T15:42:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.801+0000 7f2f677fe700 1 -- 172.21.15.16:0/1422719585 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+223 (secure 0 0 0) 0x7f2f68015090 con 0x7f2f780ff5a0 2024-04-22T15:42:19.804 INFO:teuthology.orchestra.run.smithi016.stdout:# minimal ceph.conf for 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T15:42:19.804 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2024-04-22T15:42:19.804 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T15:42:19.804 INFO:teuthology.orchestra.run.smithi016.stdout: mon_host = [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] 2024-04-22T15:42:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.803+0000 7f2f7c8ac700 1 -- 172.21.15.16:0/1422719585 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f2f6006c470 msgr2=0x7f2f6006e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.803+0000 7f2f7c8ac700 1 --2- 172.21.15.16:0/1422719585 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f2f6006c470 0x7f2f6006e930 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2f78197bc0 tx=0x7f2f6c01a040 comp rx=0 tx=0).stop 2024-04-22T15:42:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.803+0000 7f2f7c8ac700 1 -- 172.21.15.16:0/1422719585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f780ff5a0 msgr2=0x7f2f7819c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.803+0000 7f2f7c8ac700 1 --2- 172.21.15.16:0/1422719585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f780ff5a0 0x7f2f7819c6d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2f6800b4b0 tx=0x7f2f6800b7c0 comp rx=0 tx=0).stop 2024-04-22T15:42:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.803+0000 7f2f7c8ac700 1 -- 172.21.15.16:0/1422719585 shutdown_connections 2024-04-22T15:42:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.803+0000 7f2f7c8ac700 1 --2- 172.21.15.16:0/1422719585 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f2f6006c470 0x7f2f6006e930 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.803+0000 7f2f7c8ac700 1 --2- 172.21.15.16:0/1422719585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f780ff5a0 0x7f2f7819c6d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.803+0000 7f2f7c8ac700 1 --2- 172.21.15.16:0/1422719585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f780fff50 0x7f2f7819cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.803+0000 7f2f7c8ac700 1 -- 172.21.15.16:0/1422719585 >> 172.21.15.16:0/1422719585 conn(0x7f2f78074e30 msgr2=0x7f2f780fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.803+0000 7f2f7c8ac700 1 -- 172.21.15.16:0/1422719585 shutdown_connections 2024-04-22T15:42:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:19.803+0000 7f2f7c8ac700 1 -- 172.21.15.16:0/1422719585 wait complete. 2024-04-22T15:42:19.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:19 smithi027 ceph-mon[34922]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:19.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:19 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:19.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:19 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:19.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:19 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi016.rnsnnr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:42:19.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:19 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:42:19.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:19 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:20.815 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:20 smithi016 ceph-mon[26620]: Reconfiguring mgr.smithi016.rnsnnr (unknown last config time)... 2024-04-22T15:42:20.815 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:20 smithi016 ceph-mon[26620]: Reconfiguring daemon mgr.smithi016.rnsnnr on smithi016 2024-04-22T15:42:20.816 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:20 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/1422719585' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:20.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:20 smithi027 ceph-mon[34922]: Reconfiguring mgr.smithi016.rnsnnr (unknown last config time)... 2024-04-22T15:42:20.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:20 smithi027 ceph-mon[34922]: Reconfiguring daemon mgr.smithi016.rnsnnr on smithi016 2024-04-22T15:42:20.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:20 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/1422719585' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:20.875 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-22T15:42:20.875 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-22T15:42:20.875 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T15:42:20.906 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-22T15:42:20.906 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:42:20.973 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-22T15:42:20.974 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T15:42:21.006 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-22T15:42:21.007 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:42:21.077 INFO:tasks.cephadm:Deploying OSDs... 2024-04-22T15:42:21.077 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-22T15:42:21.077 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T15:42:21.095 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T15:42:21.095 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_1 2024-04-22T15:42:21.153 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T15:42:21.153 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T15:42:21.153 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-22T15:42:21.153 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T15:42:21.153 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T15:42:21.153 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-22 15:41:31.716351964 +0000 2024-04-22T15:42:21.153 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-22 15:36:02.501297887 +0000 2024-04-22T15:42:21.154 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-22 15:36:02.501297887 +0000 2024-04-22T15:42:21.154 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-22 15:36:02.501297887 +0000 2024-04-22T15:42:21.154 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T15:42:21.220 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-22T15:42:21.220 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-22T15:42:21.220 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000120778 s, 4.2 MB/s 2024-04-22T15:42:21.221 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T15:42:21.279 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_2 2024-04-22T15:42:21.336 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T15:42:21.336 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T15:42:21.337 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-22T15:42:21.337 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T15:42:21.337 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T15:42:21.337 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-22 15:41:31.716351964 +0000 2024-04-22T15:42:21.337 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-22 15:36:02.839292360 +0000 2024-04-22T15:42:21.337 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-22 15:36:02.839292360 +0000 2024-04-22T15:42:21.337 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-22 15:36:02.839292360 +0000 2024-04-22T15:42:21.337 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T15:42:21.402 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-22T15:42:21.402 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-22T15:42:21.403 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000116542 s, 4.4 MB/s 2024-04-22T15:42:21.403 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T15:42:21.462 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_3 2024-04-22T15:42:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T15:42:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T15:42:21.519 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-22T15:42:21.519 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T15:42:21.519 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T15:42:21.519 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-22 15:41:31.716351964 +0000 2024-04-22T15:42:21.519 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-22 15:36:03.161287094 +0000 2024-04-22T15:42:21.519 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-22 15:36:03.161287094 +0000 2024-04-22T15:42:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-22 15:36:03.161287094 +0000 2024-04-22T15:42:21.520 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T15:42:21.582 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-22T15:42:21.582 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-22T15:42:21.582 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.00013864 s, 3.7 MB/s 2024-04-22T15:42:21.583 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T15:42:21.640 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_4 2024-04-22T15:42:21.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:21 smithi016 ceph-mon[26620]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:21.681 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T15:42:21.681 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T15:42:21.681 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-22T15:42:21.681 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T15:42:21.681 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T15:42:21.681 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-22 15:41:31.717351948 +0000 2024-04-22T15:42:21.681 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-22 15:36:03.487281763 +0000 2024-04-22T15:42:21.681 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-22 15:36:03.487281763 +0000 2024-04-22T15:42:21.681 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-22 15:36:03.487281763 +0000 2024-04-22T15:42:21.682 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T15:42:21.744 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-22T15:42:21.744 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-22T15:42:21.744 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000107476 s, 4.8 MB/s 2024-04-22T15:42:21.745 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T15:42:21.802 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-22T15:42:21.802 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T15:42:21.821 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T15:42:21.822 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_1 2024-04-22T15:42:21.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:21 smithi027 ceph-mon[34922]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:21.853 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T15:42:21.854 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T15:42:21.854 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-22T15:42:21.854 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T15:42:21.854 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T15:42:21.854 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-22 15:42:08.440613754 +0000 2024-04-22T15:42:21.854 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-22 15:35:58.387604281 +0000 2024-04-22T15:42:21.854 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-22 15:35:58.387604281 +0000 2024-04-22T15:42:21.854 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-04-22 15:35:58.387604281 +0000 2024-04-22T15:42:21.854 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T15:42:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-04-22T15:42:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-04-22T15:42:21.923 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000235206 s, 2.2 MB/s 2024-04-22T15:42:21.924 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T15:42:21.983 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_2 2024-04-22T15:42:22.041 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T15:42:22.041 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T15:42:22.041 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-22T15:42:22.041 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T15:42:22.041 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T15:42:22.041 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-22 15:42:08.441613729 +0000 2024-04-22T15:42:22.041 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-22 15:35:58.724595603 +0000 2024-04-22T15:42:22.041 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-22 15:35:58.724595603 +0000 2024-04-22T15:42:22.041 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-04-22 15:35:58.724595603 +0000 2024-04-22T15:42:22.041 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T15:42:22.107 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-04-22T15:42:22.107 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-04-22T15:42:22.107 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.00020457 s, 2.5 MB/s 2024-04-22T15:42:22.108 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T15:42:22.165 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_3 2024-04-22T15:42:22.222 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T15:42:22.222 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T15:42:22.222 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-22T15:42:22.223 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T15:42:22.223 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T15:42:22.223 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-22 15:42:08.441613729 +0000 2024-04-22T15:42:22.223 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-22 15:35:59.069586719 +0000 2024-04-22T15:42:22.223 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-22 15:35:59.069586719 +0000 2024-04-22T15:42:22.223 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-04-22 15:35:59.069586719 +0000 2024-04-22T15:42:22.223 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T15:42:22.288 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-04-22T15:42:22.288 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-04-22T15:42:22.288 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000199484 s, 2.6 MB/s 2024-04-22T15:42:22.290 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T15:42:22.348 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_4 2024-04-22T15:42:22.404 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T15:42:22.404 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T15:42:22.404 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-22T15:42:22.404 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T15:42:22.404 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T15:42:22.405 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-22 15:42:08.442613704 +0000 2024-04-22T15:42:22.405 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-22 15:35:59.423577604 +0000 2024-04-22T15:42:22.405 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-22 15:35:59.423577604 +0000 2024-04-22T15:42:22.405 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-04-22 15:35:59.423577604 +0000 2024-04-22T15:42:22.405 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T15:42:22.468 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-04-22T15:42:22.469 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-04-22T15:42:22.469 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000220016 s, 2.3 MB/s 2024-04-22T15:42:22.470 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T15:42:22.527 INFO:tasks.cephadm:Deploying osd.0 on smithi016 with /dev/vg_nvme/lv_4... 2024-04-22T15:42:22.527 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T15:42:22.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:22 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:22.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:22 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:22.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:22 smithi027 ceph-mon[34922]: Reconfiguring ceph-exporter.smithi016 (monmap changed)... 2024-04-22T15:42:22.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:22 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi016", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:42:22.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:22 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:22.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:22 smithi027 ceph-mon[34922]: Reconfiguring daemon ceph-exporter.smithi016 on smithi016 2024-04-22T15:42:22.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:22 smithi027 ceph-mon[34922]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:22.632 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:22 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:22.633 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:22 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:22.633 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:22 smithi016 ceph-mon[26620]: Reconfiguring ceph-exporter.smithi016 (monmap changed)... 2024-04-22T15:42:22.633 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:22 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi016", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:42:22.633 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:22 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:22.633 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:22 smithi016 ceph-mon[26620]: Reconfiguring daemon ceph-exporter.smithi016 on smithi016 2024-04-22T15:42:22.633 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:22 smithi016 ceph-mon[26620]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:22.976 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:42:24.024 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:23 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:24.024 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:23 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:24.024 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:23 smithi016 ceph-mon[26620]: Reconfiguring crash.smithi016 (monmap changed)... 2024-04-22T15:42:24.025 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:23 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi016", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:42:24.025 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:23 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:24.025 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:23 smithi016 ceph-mon[26620]: Reconfiguring daemon crash.smithi016 on smithi016 2024-04-22T15:42:24.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:23 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:24.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:23 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:24.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:23 smithi027 ceph-mon[34922]: Reconfiguring crash.smithi016 (monmap changed)... 2024-04-22T15:42:24.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:23 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi016", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:42:24.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:23 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:24.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:23 smithi027 ceph-mon[34922]: Reconfiguring daemon crash.smithi016 on smithi016 2024-04-22T15:42:25.079 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:24 smithi016 ceph-mon[26620]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:25.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:24 smithi027 ceph-mon[34922]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:25.954 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T15:42:25.954 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T15:42:25.954 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-04-22T15:42:25.954 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-04-22T15:42:25.954 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00843583 s, 1.2 GB/s 2024-04-22T15:42:25.954 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-04-22T15:42:26.330 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:26 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:26.330 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:26 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:26.330 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:26 smithi016 ceph-mon[26620]: Reconfiguring alertmanager.smithi016 (dependencies changed)... 2024-04-22T15:42:26.330 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:26 smithi016 ceph-mon[26620]: Reconfiguring daemon alertmanager.smithi016 on smithi016 2024-04-22T15:42:26.330 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:26 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:42:26.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:26 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:26.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:26 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:26.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:26 smithi027 ceph-mon[34922]: Reconfiguring alertmanager.smithi016 (dependencies changed)... 2024-04-22T15:42:26.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:26 smithi027 ceph-mon[34922]: Reconfiguring daemon alertmanager.smithi016 on smithi016 2024-04-22T15:42:26.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:26 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:42:26.992 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_4 2024-04-22T15:42:27.310 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:27 smithi016 ceph-mon[26620]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:27.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:27 smithi027 ceph-mon[34922]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:28.784 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:42:29.632 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:29 smithi016 ceph-mon[26620]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:29.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:29 smithi027 ceph-mon[34922]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.254+0000 7ff742d93700 1 -- 172.21.15.16:0/2492007150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff73c104170 msgr2=0x7ff73c106560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.254+0000 7ff742d93700 1 --2- 172.21.15.16:0/2492007150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff73c104170 0x7ff73c106560 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff728009910 tx=0x7ff728009c20 comp rx=0 tx=0).stop 2024-04-22T15:42:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.254+0000 7ff742d93700 1 -- 172.21.15.16:0/2492007150 shutdown_connections 2024-04-22T15:42:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.254+0000 7ff742d93700 1 --2- 172.21.15.16:0/2492007150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff73c104170 0x7ff73c106560 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.254+0000 7ff742d93700 1 --2- 172.21.15.16:0/2492007150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff73c101840 0x7ff73c103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.254+0000 7ff742d93700 1 -- 172.21.15.16:0/2492007150 >> 172.21.15.16:0/2492007150 conn(0x7ff73c0fb190 msgr2=0x7ff73c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.254+0000 7ff742d93700 1 -- 172.21.15.16:0/2492007150 shutdown_connections 2024-04-22T15:42:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.254+0000 7ff742d93700 1 -- 172.21.15.16:0/2492007150 wait complete. 2024-04-22T15:42:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.254+0000 7ff742d93700 1 Processor -- start 2024-04-22T15:42:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.255+0000 7ff742d93700 1 -- start start 2024-04-22T15:42:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.255+0000 7ff742d93700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff73c101840 0x7ff73c196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.255+0000 7ff742d93700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff73c104170 0x7ff73c1969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.255+0000 7ff742d93700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c196fd0 con 0x7ff73c104170 2024-04-22T15:42:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.255+0000 7ff742d93700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c197110 con 0x7ff73c101840 2024-04-22T15:42:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.255+0000 7ff73bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff73c104170 0x7ff73c1969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.255+0000 7ff73bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff73c104170 0x7ff73c1969b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60822/0 (socket says 172.21.15.16:60822) 2024-04-22T15:42:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.255+0000 7ff73bfff700 1 -- 172.21.15.16:0/3131314354 learned_addr learned my addr 172.21.15.16:0/3131314354 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:42:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.255+0000 7ff740b2f700 1 --2- 172.21.15.16:0/3131314354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff73c101840 0x7ff73c196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.255+0000 7ff73bfff700 1 -- 172.21.15.16:0/3131314354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff73c101840 msgr2=0x7ff73c196470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.255+0000 7ff73bfff700 1 --2- 172.21.15.16:0/3131314354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff73c101840 0x7ff73c196470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.255+0000 7ff73bfff700 1 -- 172.21.15.16:0/3131314354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7280095f0 con 0x7ff73c104170 2024-04-22T15:42:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.255+0000 7ff73bfff700 1 --2- 172.21.15.16:0/3131314354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff73c104170 0x7ff73c1969b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff72800b5c0 tx=0x7ff72800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.256+0000 7ff739ffb700 1 -- 172.21.15.16:0/3131314354 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff72801d070 con 0x7ff73c104170 2024-04-22T15:42:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.256+0000 7ff739ffb700 1 -- 172.21.15.16:0/3131314354 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff72800f460 con 0x7ff73c104170 2024-04-22T15:42:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.256+0000 7ff742d93700 1 -- 172.21.15.16:0/3131314354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff73c19bb60 con 0x7ff73c104170 2024-04-22T15:42:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.256+0000 7ff739ffb700 1 -- 172.21.15.16:0/3131314354 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff728021640 con 0x7ff73c104170 2024-04-22T15:42:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.256+0000 7ff742d93700 1 -- 172.21.15.16:0/3131314354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff73c19c050 con 0x7ff73c104170 2024-04-22T15:42:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.257+0000 7ff742d93700 1 -- 172.21.15.16:0/3131314354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff73c109810 con 0x7ff73c104170 2024-04-22T15:42:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.258+0000 7ff739ffb700 1 -- 172.21.15.16:0/3131314354 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 90256+0+0 (secure 0 0 0) 0x7ff72800fa60 con 0x7ff73c104170 2024-04-22T15:42:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.258+0000 7ff739ffb700 1 --2- 172.21.15.16:0/3131314354 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff72c06c470 0x7ff72c06e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:42:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.258+0000 7ff739ffb700 1 -- 172.21.15.16:0/3131314354 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7ff72808b860 con 0x7ff73c104170 2024-04-22T15:42:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.258+0000 7ff740b2f700 1 --2- 172.21.15.16:0/3131314354 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff72c06c470 0x7ff72c06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:42:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.259+0000 7ff740b2f700 1 --2- 172.21.15.16:0/3131314354 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff72c06c470 0x7ff72c06e930 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff730005950 tx=0x7ff73000b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:42:30.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.264+0000 7ff739ffb700 1 -- 172.21.15.16:0/3131314354 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff72805ac30 con 0x7ff73c104170 2024-04-22T15:42:30.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:30.472+0000 7ff742d93700 1 -- 172.21.15.16:0/3131314354 --> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff73c104de0 con 0x7ff72c06c470 2024-04-22T15:42:31.613 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:31 smithi016 ceph-mon[26620]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:31.613 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:31 smithi016 ceph-mon[26620]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:42:31.613 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:31 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T15:42:31.613 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:31 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T15:42:31.613 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:31 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:31.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:31 smithi027 ceph-mon[34922]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:31.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:31 smithi027 ceph-mon[34922]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:42:31.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:31 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T15:42:31.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:31 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T15:42:31.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:31 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:33.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:33 smithi016 ceph-mon[26620]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:33.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:33 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:33.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:33 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:33.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:33 smithi027 ceph-mon[34922]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:33.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:33 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:33.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:33 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:34.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:34 smithi027 ceph-mon[34922]: Reconfiguring grafana.smithi016 (dependencies changed)... 2024-04-22T15:42:34.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:34 smithi027 ceph-mon[34922]: Reconfiguring daemon grafana.smithi016 on smithi016 2024-04-22T15:42:34.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:34 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/3817430773' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67bd69ee-bd8c-4670-927a-f5fbcd92c9bd"}]: dispatch 2024-04-22T15:42:34.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:34 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/3817430773' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "67bd69ee-bd8c-4670-927a-f5fbcd92c9bd"}]': finished 2024-04-22T15:42:34.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:34 smithi027 ceph-mon[34922]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T15:42:34.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:34 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:42:34.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:34 smithi016 ceph-mon[26620]: Reconfiguring grafana.smithi016 (dependencies changed)... 2024-04-22T15:42:34.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:34 smithi016 ceph-mon[26620]: Reconfiguring daemon grafana.smithi016 on smithi016 2024-04-22T15:42:34.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:34 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/3817430773' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67bd69ee-bd8c-4670-927a-f5fbcd92c9bd"}]: dispatch 2024-04-22T15:42:34.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:34 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/3817430773' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "67bd69ee-bd8c-4670-927a-f5fbcd92c9bd"}]': finished 2024-04-22T15:42:34.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:34 smithi016 ceph-mon[26620]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T15:42:34.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:34 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:42:35.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:35 smithi027 ceph-mon[34922]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:35.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:35 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/2402645077' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T15:42:35.837 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:35 smithi016 ceph-mon[26620]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:35.837 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:35 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/2402645077' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T15:42:36.890 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:36 smithi027 ceph-mon[34922]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:36.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:36 smithi016 ceph-mon[26620]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:39.505 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:39 smithi016 ceph-mon[26620]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:39.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:39 smithi027 ceph-mon[34922]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:40.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:42:40.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:40 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:42:41.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:41 smithi027 ceph-mon[34922]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:41.866 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:41 smithi016 ceph-mon[26620]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:43.623 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:43 smithi016 ceph-mon[26620]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:43.623 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:43 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:43.623 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:43 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:43.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:43 smithi027 ceph-mon[34922]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:43.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:43 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:43.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:43 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:44.727 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:44 smithi016 ceph-mon[26620]: Reconfiguring prometheus.smithi016 (dependencies changed)... 2024-04-22T15:42:44.727 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:44 smithi016 ceph-mon[26620]: Reconfiguring daemon prometheus.smithi016 on smithi016 2024-04-22T15:42:44.727 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:44 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T15:42:44.727 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:44 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:44.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:44 smithi027 ceph-mon[34922]: Reconfiguring prometheus.smithi016 (dependencies changed)... 2024-04-22T15:42:44.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:44 smithi027 ceph-mon[34922]: Reconfiguring daemon prometheus.smithi016 on smithi016 2024-04-22T15:42:44.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:44 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T15:42:44.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:44 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:45.763 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:45 smithi016 ceph-mon[26620]: Deploying daemon osd.0 on smithi016 2024-04-22T15:42:45.763 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:45 smithi016 ceph-mon[26620]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:45.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:45 smithi027 ceph-mon[34922]: Deploying daemon osd.0 on smithi016 2024-04-22T15:42:45.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:45 smithi027 ceph-mon[34922]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:46.883 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:46 smithi016 ceph-mon[26620]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:46.903 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:46 smithi027 ceph-mon[34922]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:49.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:49 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:49.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:49 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:49.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:49 smithi016 ceph-mon[26620]: Reconfiguring ceph-exporter.smithi027 (monmap changed)... 2024-04-22T15:42:49.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:49 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi027", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:42:49.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:49 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:49.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:49 smithi016 ceph-mon[26620]: Reconfiguring daemon ceph-exporter.smithi027 on smithi027 2024-04-22T15:42:49.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:49 smithi016 ceph-mon[26620]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:49.494 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:49 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:49.494 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:49 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:49.495 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:49 smithi027 ceph-mon[34922]: Reconfiguring ceph-exporter.smithi027 (monmap changed)... 2024-04-22T15:42:49.495 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:49 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi027", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:42:49.495 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:49 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:49.495 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:49 smithi027 ceph-mon[34922]: Reconfiguring daemon ceph-exporter.smithi027 on smithi027 2024-04-22T15:42:49.495 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:49 smithi027 ceph-mon[34922]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:50.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:50 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:50.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:50 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:50.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:50 smithi027 ceph-mon[34922]: Reconfiguring crash.smithi027 (monmap changed)... 2024-04-22T15:42:50.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:50 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi027", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:42:50.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:50 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:50.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:50 smithi027 ceph-mon[34922]: Reconfiguring daemon crash.smithi027 on smithi027 2024-04-22T15:42:50.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:50 smithi027 ceph-mon[34922]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:50.595 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:50 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:50.595 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:50 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:50.595 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:50 smithi016 ceph-mon[26620]: Reconfiguring crash.smithi027 (monmap changed)... 2024-04-22T15:42:50.595 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:50 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi027", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:42:50.595 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:50 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:50.596 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:50 smithi016 ceph-mon[26620]: Reconfiguring daemon crash.smithi027 on smithi027 2024-04-22T15:42:50.596 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:50 smithi016 ceph-mon[26620]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:52.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:51 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:52.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:51 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:52.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:51 smithi027 ceph-mon[34922]: Reconfiguring mgr.smithi027.ukdgzs (monmap changed)... 2024-04-22T15:42:52.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:51 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi027.ukdgzs", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:42:52.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:51 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:42:52.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:51 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:52.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:51 smithi027 ceph-mon[34922]: Reconfiguring daemon mgr.smithi027.ukdgzs on smithi027 2024-04-22T15:42:52.376 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:51 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:52.376 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:51 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:52.376 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:51 smithi016 ceph-mon[26620]: Reconfiguring mgr.smithi027.ukdgzs (monmap changed)... 2024-04-22T15:42:52.376 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:51 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi027.ukdgzs", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:42:52.376 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:51 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:42:52.376 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:51 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:52.376 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:51 smithi016 ceph-mon[26620]: Reconfiguring daemon mgr.smithi027.ukdgzs on smithi027 2024-04-22T15:42:53.656 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:53 smithi016 ceph-mon[26620]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:53.656 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:53 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:53.656 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:53 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:53.656 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:53 smithi016 ceph-mon[26620]: Reconfiguring mon.smithi027 (monmap changed)... 2024-04-22T15:42:53.657 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:53 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:42:53.657 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:53 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T15:42:53.657 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:53 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:53.657 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:53 smithi016 ceph-mon[26620]: Reconfiguring daemon mon.smithi027 on smithi027 2024-04-22T15:42:53.657 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:53 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:53.657 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:53 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:53.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:53 smithi027 ceph-mon[34922]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:53.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:53 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:53.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:53 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:53.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:53 smithi027 ceph-mon[34922]: Reconfiguring mon.smithi027 (monmap changed)... 2024-04-22T15:42:53.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:53 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:42:53.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:53 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T15:42:53.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:53 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:42:53.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:53 smithi027 ceph-mon[34922]: Reconfiguring daemon mon.smithi027 on smithi027 2024-04-22T15:42:53.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:53 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:53.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:53 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:55.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:54 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:55.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:54 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:55.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:54 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T15:42:55.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:54 smithi027 ceph-mon[34922]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T15:42:55.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:54 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016:9093"}]: dispatch 2024-04-22T15:42:55.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:54 smithi027 ceph-mon[34922]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016:9093"}]: dispatch 2024-04-22T15:42:55.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:54 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:55.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:54 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T15:42:55.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:54 smithi027 ceph-mon[34922]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T15:42:55.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:54 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi016:3000"}]: dispatch 2024-04-22T15:42:55.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:54 smithi027 ceph-mon[34922]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi016:3000"}]: dispatch 2024-04-22T15:42:55.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:54 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:55.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:54 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:42:55.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:54 smithi027 ceph-mon[34922]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:42:55.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:54 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi016:9095"}]: dispatch 2024-04-22T15:42:55.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:54 smithi027 ceph-mon[34922]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi016:9095"}]: dispatch 2024-04-22T15:42:55.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:54 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:55.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:54 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:42:55.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:54 smithi027 ceph-mon[34922]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:55.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:54 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:55.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:54 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:55.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:54 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T15:42:55.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:54 smithi016 ceph-mon[26620]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T15:42:55.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:54 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016:9093"}]: dispatch 2024-04-22T15:42:55.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:54 smithi016 ceph-mon[26620]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016:9093"}]: dispatch 2024-04-22T15:42:55.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:54 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:55.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:54 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T15:42:55.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:54 smithi016 ceph-mon[26620]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T15:42:55.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:54 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi016:3000"}]: dispatch 2024-04-22T15:42:55.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:54 smithi016 ceph-mon[26620]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi016:3000"}]: dispatch 2024-04-22T15:42:55.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:54 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:55.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:54 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:42:55.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:54 smithi016 ceph-mon[26620]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:42:55.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:54 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi016:9095"}]: dispatch 2024-04-22T15:42:55.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:54 smithi016 ceph-mon[26620]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi016:9095"}]: dispatch 2024-04-22T15:42:55.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:54 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:55.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:54 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:42:55.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:54 smithi016 ceph-mon[26620]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:56.058 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:56 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:42:56.058 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:56 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:56.058 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:56 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:56.102 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 0 on host 'smithi016' 2024-04-22T15:42:56.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:56.099+0000 7ff739ffb700 1 -- 172.21.15.16:0/3131314354 <== mgr.14217 v2:172.21.15.16:6800/1057234399 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff73c104de0 con 0x7ff72c06c470 2024-04-22T15:42:56.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:56.101+0000 7ff742d93700 1 -- 172.21.15.16:0/3131314354 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff72c06c470 msgr2=0x7ff72c06e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:56.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:56.101+0000 7ff742d93700 1 --2- 172.21.15.16:0/3131314354 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff72c06c470 0x7ff72c06e930 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff730005950 tx=0x7ff73000b410 comp rx=0 tx=0).stop 2024-04-22T15:42:56.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:56.101+0000 7ff742d93700 1 -- 172.21.15.16:0/3131314354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff73c104170 msgr2=0x7ff73c1969b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:42:56.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:56.101+0000 7ff742d93700 1 --2- 172.21.15.16:0/3131314354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff73c104170 0x7ff73c1969b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff72800b5c0 tx=0x7ff72800bd70 comp rx=0 tx=0).stop 2024-04-22T15:42:56.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:56.101+0000 7ff742d93700 1 -- 172.21.15.16:0/3131314354 shutdown_connections 2024-04-22T15:42:56.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:56.101+0000 7ff742d93700 1 --2- 172.21.15.16:0/3131314354 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff72c06c470 0x7ff72c06e930 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:56.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:56.101+0000 7ff742d93700 1 --2- 172.21.15.16:0/3131314354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff73c101840 0x7ff73c196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:56.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:56.101+0000 7ff742d93700 1 --2- 172.21.15.16:0/3131314354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff73c104170 0x7ff73c1969b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:42:56.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:56.101+0000 7ff742d93700 1 -- 172.21.15.16:0/3131314354 >> 172.21.15.16:0/3131314354 conn(0x7ff73c0fb190 msgr2=0x7ff73c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:42:56.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:56.101+0000 7ff742d93700 1 -- 172.21.15.16:0/3131314354 shutdown_connections 2024-04-22T15:42:56.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:42:56.101+0000 7ff742d93700 1 -- 172.21.15.16:0/3131314354 wait complete. 2024-04-22T15:42:56.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:56 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:42:56.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:56 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:56.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:56 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:56.707 DEBUG:teuthology.orchestra.run.smithi016:osd.0> sudo journalctl -f -n 0 -u ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.0.service 2024-04-22T15:42:56.710 INFO:tasks.cephadm:Deploying osd.1 on smithi016 with /dev/vg_nvme/lv_3... 2024-04-22T15:42:56.710 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T15:42:57.295 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:42:57.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:57 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:57.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:57 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:57.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:57 smithi027 ceph-mon[34922]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:57.414 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:57 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:57.415 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:57 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:42:57.415 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:57 smithi016 ceph-mon[26620]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:57.415 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:42:57 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0[45810]: 2024-04-22T15:42:57.161+0000 7f0823106640 -1 osd.0 0 log_to_monitors true 2024-04-22T15:42:58.345 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:58 smithi016 ceph-mon[26620]: from='osd.0 [v2:172.21.15.16:6802/1210635569,v1:172.21.15.16:6803/1210635569]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T15:42:58.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:58 smithi027 ceph-mon[34922]: from='osd.0 [v2:172.21.15.16:6802/1210635569,v1:172.21.15.16:6803/1210635569]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T15:42:59.109 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:59 smithi016 ceph-mon[26620]: from='osd.0 [v2:172.21.15.16:6802/1210635569,v1:172.21.15.16:6803/1210635569]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T15:42:59.109 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:59 smithi016 ceph-mon[26620]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T15:42:59.109 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:59 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:42:59.109 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:59 smithi016 ceph-mon[26620]: from='osd.0 [v2:172.21.15.16:6802/1210635569,v1:172.21.15.16:6803/1210635569]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-22T15:42:59.109 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:42:59 smithi016 ceph-mon[26620]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:59.415 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:42:59 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0[45810]: 2024-04-22T15:42:59.105+0000 7f0817f69700 -1 osd.0 0 waiting for initial osdmap 2024-04-22T15:42:59.415 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:42:59 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0[45810]: 2024-04-22T15:42:59.108+0000 7f081255b700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T15:42:59.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:59 smithi027 ceph-mon[34922]: from='osd.0 [v2:172.21.15.16:6802/1210635569,v1:172.21.15.16:6803/1210635569]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T15:42:59.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:59 smithi027 ceph-mon[34922]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T15:42:59.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:59 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:42:59.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:59 smithi027 ceph-mon[34922]: from='osd.0 [v2:172.21.15.16:6802/1210635569,v1:172.21.15.16:6803/1210635569]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-22T15:42:59.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:42:59 smithi027 ceph-mon[34922]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T15:42:59.896 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T15:42:59.897 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T15:42:59.897 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-04-22T15:42:59.897 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-04-22T15:42:59.897 INFO:teuthology.orchestra.run.smithi016.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00964067 s, 1.1 GB/s 2024-04-22T15:42:59.897 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-04-22T15:43:00.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:00 smithi016 ceph-mon[26620]: from='osd.0 [v2:172.21.15.16:6802/1210635569,v1:172.21.15.16:6803/1210635569]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-22T15:43:00.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:00 smithi016 ceph-mon[26620]: osdmap e8: 1 total, 0 up, 1 in 2024-04-22T15:43:00.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:00 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:43:00.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:00 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:43:00.558 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_3 2024-04-22T15:43:00.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:00 smithi027 ceph-mon[34922]: from='osd.0 [v2:172.21.15.16:6802/1210635569,v1:172.21.15.16:6803/1210635569]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-22T15:43:00.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:00 smithi027 ceph-mon[34922]: osdmap e8: 1 total, 0 up, 1 in 2024-04-22T15:43:00.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:00 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:43:00.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:00 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:43:00.954 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:43:01.397 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:01 smithi016 ceph-mon[26620]: purged_snaps scrub starts 2024-04-22T15:43:01.398 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:01 smithi016 ceph-mon[26620]: purged_snaps scrub ok 2024-04-22T15:43:01.398 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:01 smithi016 ceph-mon[26620]: osd.0 [v2:172.21.15.16:6802/1210635569,v1:172.21.15.16:6803/1210635569] boot 2024-04-22T15:43:01.398 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:01 smithi016 ceph-mon[26620]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T15:43:01.398 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:01 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:43:01.398 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:01 smithi016 ceph-mon[26620]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:01.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:01 smithi027 ceph-mon[34922]: purged_snaps scrub starts 2024-04-22T15:43:01.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:01 smithi027 ceph-mon[34922]: purged_snaps scrub ok 2024-04-22T15:43:01.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:01 smithi027 ceph-mon[34922]: osd.0 [v2:172.21.15.16:6802/1210635569,v1:172.21.15.16:6803/1210635569] boot 2024-04-22T15:43:01.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:01 smithi027 ceph-mon[34922]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T15:43:01.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:01 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:43:01.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:01 smithi027 ceph-mon[34922]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:02.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:02 smithi027 ceph-mon[34922]: osdmap e10: 1 total, 1 up, 1 in 2024-04-22T15:43:02.341 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:02 smithi016 ceph-mon[26620]: osdmap e10: 1 total, 1 up, 1 in 2024-04-22T15:43:03.307 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:03 smithi016 ceph-mon[26620]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:03.307 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:03 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:03.308 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:03 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:03.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.315+0000 7f5aa3714700 1 -- 172.21.15.16:0/1858325464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a9c1040e0 msgr2=0x7f5a9c1064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:03.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.315+0000 7f5aa3714700 1 --2- 172.21.15.16:0/1858325464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a9c1040e0 0x7f5a9c1064d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c009910 tx=0x7f5a8c009c20 comp rx=0 tx=0).stop 2024-04-22T15:43:03.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.315+0000 7f5aa3714700 1 -- 172.21.15.16:0/1858325464 shutdown_connections 2024-04-22T15:43:03.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.315+0000 7f5aa3714700 1 --2- 172.21.15.16:0/1858325464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a9c1040e0 0x7f5a9c1064d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:03.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.315+0000 7f5aa3714700 1 --2- 172.21.15.16:0/1858325464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a9c1017b0 0x7f5a9c103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:03.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.315+0000 7f5aa3714700 1 -- 172.21.15.16:0/1858325464 >> 172.21.15.16:0/1858325464 conn(0x7f5a9c0fb160 msgr2=0x7f5a9c0fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:43:03.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.315+0000 7f5aa3714700 1 -- 172.21.15.16:0/1858325464 shutdown_connections 2024-04-22T15:43:03.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.315+0000 7f5aa3714700 1 -- 172.21.15.16:0/1858325464 wait complete. 2024-04-22T15:43:03.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.315+0000 7f5aa3714700 1 Processor -- start 2024-04-22T15:43:03.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.315+0000 7f5aa3714700 1 -- start start 2024-04-22T15:43:03.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.316+0000 7f5aa3714700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a9c1017b0 0x7f5a9c198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:03.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.316+0000 7f5aa3714700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a9c1040e0 0x7f5a9c198ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:03.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.316+0000 7f5aa3714700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9c1994e0 con 0x7f5a9c1040e0 2024-04-22T15:43:03.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.316+0000 7f5aa3714700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9c199620 con 0x7f5a9c1017b0 2024-04-22T15:43:03.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.316+0000 7f5aa0caf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a9c1040e0 0x7f5a9c198ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:03.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.316+0000 7f5aa0caf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a9c1040e0 0x7f5a9c198ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40886/0 (socket says 172.21.15.16:40886) 2024-04-22T15:43:03.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.316+0000 7f5aa0caf700 1 -- 172.21.15.16:0/982997717 learned_addr learned my addr 172.21.15.16:0/982997717 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:43:03.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.316+0000 7f5aa14b0700 1 --2- 172.21.15.16:0/982997717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a9c1017b0 0x7f5a9c198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:03.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.316+0000 7f5aa0caf700 1 -- 172.21.15.16:0/982997717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a9c1017b0 msgr2=0x7f5a9c198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:03.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.316+0000 7f5aa0caf700 1 --2- 172.21.15.16:0/982997717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a9c1017b0 0x7f5a9c198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:03.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.316+0000 7f5aa0caf700 1 -- 172.21.15.16:0/982997717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8c0095f0 con 0x7f5a9c1040e0 2024-04-22T15:43:03.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.316+0000 7f5aa0caf700 1 --2- 172.21.15.16:0/982997717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a9c1040e0 0x7f5a9c198ec0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c006010 tx=0x7f5a8c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:43:03.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.316+0000 7f5a927fc700 1 -- 172.21.15.16:0/982997717 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a8c01d070 con 0x7f5a9c1040e0 2024-04-22T15:43:03.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.316+0000 7f5aa3714700 1 -- 172.21.15.16:0/982997717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a9c19e070 con 0x7f5a9c1040e0 2024-04-22T15:43:03.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.316+0000 7f5a927fc700 1 -- 172.21.15.16:0/982997717 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5a8c00f460 con 0x7f5a9c1040e0 2024-04-22T15:43:03.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.316+0000 7f5a927fc700 1 -- 172.21.15.16:0/982997717 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a8c017630 con 0x7f5a9c1040e0 2024-04-22T15:43:03.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.316+0000 7f5aa3714700 1 -- 172.21.15.16:0/982997717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a9c19e560 con 0x7f5a9c1040e0 2024-04-22T15:43:03.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.317+0000 7f5aa3714700 1 -- 172.21.15.16:0/982997717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a9c068af0 con 0x7f5a9c1040e0 2024-04-22T15:43:03.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.318+0000 7f5a927fc700 1 -- 172.21.15.16:0/982997717 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 90256+0+0 (secure 0 0 0) 0x7f5a8c003a60 con 0x7f5a9c1040e0 2024-04-22T15:43:03.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.318+0000 7f5a927fc700 1 --2- 172.21.15.16:0/982997717 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f5a8806c460 0x7f5a8806e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:03.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.318+0000 7f5a927fc700 1 -- 172.21.15.16:0/982997717 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1982+0+0 (secure 0 0 0) 0x7f5a8c08be40 con 0x7f5a9c1040e0 2024-04-22T15:43:03.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.318+0000 7f5aa14b0700 1 --2- 172.21.15.16:0/982997717 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f5a8806c460 0x7f5a8806e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:03.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.319+0000 7f5aa14b0700 1 --2- 172.21.15.16:0/982997717 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f5a8806c460 0x7f5a8806e920 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5a98005950 tx=0x7f5a980058e0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:43:03.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.324+0000 7f5a927fc700 1 -- 172.21.15.16:0/982997717 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5a8c05af30 con 0x7f5a9c1040e0 2024-04-22T15:43:03.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:03.547+0000 7f5aa3714700 1 -- 172.21.15.16:0/982997717 --> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a9c100e70 con 0x7f5a8806c460 2024-04-22T15:43:03.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:03 smithi027 ceph-mon[34922]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:03.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:03 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:03.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:03 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:04.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:04 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T15:43:04.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:04 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T15:43:04.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:04 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:43:04.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:04 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T15:43:04.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:04 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T15:43:04.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:04 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:43:05.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:05 smithi016 ceph-mon[26620]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:43:05.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:05 smithi016 ceph-mon[26620]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:05.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:05 smithi027 ceph-mon[34922]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:43:05.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:05 smithi027 ceph-mon[34922]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:06.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:06 smithi016 ceph-mon[26620]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:06.929 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:06 smithi027 ceph-mon[34922]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:07.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:07 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/1622469638' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f51a9e4d-fb0c-4f62-9328-e23eea863213"}]: dispatch 2024-04-22T15:43:07.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:07 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/1622469638' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f51a9e4d-fb0c-4f62-9328-e23eea863213"}]': finished 2024-04-22T15:43:07.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:07 smithi016 ceph-mon[26620]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T15:43:07.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:07 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:43:07.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:07 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/2900280890' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T15:43:08.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:07 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/1622469638' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f51a9e4d-fb0c-4f62-9328-e23eea863213"}]: dispatch 2024-04-22T15:43:08.107 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:07 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/1622469638' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f51a9e4d-fb0c-4f62-9328-e23eea863213"}]': finished 2024-04-22T15:43:08.107 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:07 smithi027 ceph-mon[34922]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T15:43:08.107 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:07 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:43:08.107 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:07 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/2900280890' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T15:43:08.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:08 smithi016 ceph-mon[26620]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:09.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:08 smithi027 ceph-mon[34922]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:10.772 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:10 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:43:10.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:10 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:43:11.735 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:11 smithi016 ceph-mon[26620]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:11.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:11 smithi027 ceph-mon[34922]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:13.591 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:13 smithi016 ceph-mon[26620]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:13.592 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:13 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:13.592 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:13 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:13.592 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:13 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:43:13.592 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:13 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:13.592 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:13 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:43:13.592 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:13 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:43:13.592 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:13 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:13.592 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:13 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:43:13.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:13 smithi027 ceph-mon[34922]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:13.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:13 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:13.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:13 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:13.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:13 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:43:13.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:13 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:13.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:13 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:43:13.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:13 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:43:13.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:13 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:13.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:13 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:43:14.754 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:14 smithi027 ceph-mon[34922]: Detected new or changed devices on smithi016 2024-04-22T15:43:14.754 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:14 smithi027 ceph-mon[34922]: Adjusting osd_memory_target on smithi016 to 11845M 2024-04-22T15:43:14.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:14 smithi016 ceph-mon[26620]: Detected new or changed devices on smithi016 2024-04-22T15:43:14.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:14 smithi016 ceph-mon[26620]: Adjusting osd_memory_target on smithi016 to 11845M 2024-04-22T15:43:15.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:15 smithi016 ceph-mon[26620]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:15.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:15 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T15:43:15.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:15 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:43:15.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:15 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:15.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:15 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:15.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:15 smithi027 ceph-mon[34922]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:15.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:15 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T15:43:15.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:15 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:43:15.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:15 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:15.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:15 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:16.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:16 smithi016 ceph-mon[26620]: Deploying daemon osd.1 on smithi016 2024-04-22T15:43:16.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:16 smithi027 ceph-mon[34922]: Deploying daemon osd.1 on smithi016 2024-04-22T15:43:17.554 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:17 smithi016 ceph-mon[26620]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:17.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:17 smithi027 ceph-mon[34922]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:19.744 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:19 smithi016 ceph-mon[26620]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:19.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:19 smithi027 ceph-mon[34922]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:21.157 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:20 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:21.157 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:20 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:21.157 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:20 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:43:21.157 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:20 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:43:21.158 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:20 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:21.158 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:20 smithi016 ceph-mon[26620]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:21.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:20 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:21.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:20 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:21.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:20 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:43:21.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:20 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:43:21.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:20 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:21.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:20 smithi027 ceph-mon[34922]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:23.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:23 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:23.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:23 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:23.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:23 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:43:23.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:23 smithi027 ceph-mon[34922]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:23.603 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:23 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:23.603 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:23 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:23.603 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:23 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:43:23.603 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:23 smithi016 ceph-mon[26620]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:25.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:25.259+0000 7f5a927fc700 1 -- 172.21.15.16:0/982997717 <== mgr.14217 v2:172.21.15.16:6800/1057234399 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5a9c100e70 con 0x7f5a8806c460 2024-04-22T15:43:25.263 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 1 on host 'smithi016' 2024-04-22T15:43:25.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:25.262+0000 7f5aa3714700 1 -- 172.21.15.16:0/982997717 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f5a8806c460 msgr2=0x7f5a8806e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:25.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:25.262+0000 7f5aa3714700 1 --2- 172.21.15.16:0/982997717 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f5a8806c460 0x7f5a8806e920 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5a98005950 tx=0x7f5a980058e0 comp rx=0 tx=0).stop 2024-04-22T15:43:25.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:25.262+0000 7f5aa3714700 1 -- 172.21.15.16:0/982997717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a9c1040e0 msgr2=0x7f5a9c198ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:25.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:25.262+0000 7f5aa3714700 1 --2- 172.21.15.16:0/982997717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a9c1040e0 0x7f5a9c198ec0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c006010 tx=0x7f5a8c00be20 comp rx=0 tx=0).stop 2024-04-22T15:43:25.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:25.262+0000 7f5aa3714700 1 -- 172.21.15.16:0/982997717 shutdown_connections 2024-04-22T15:43:25.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:25.262+0000 7f5aa3714700 1 --2- 172.21.15.16:0/982997717 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f5a8806c460 0x7f5a8806e920 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:25.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:25.262+0000 7f5aa3714700 1 --2- 172.21.15.16:0/982997717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a9c1017b0 0x7f5a9c198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:25.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:25.262+0000 7f5aa3714700 1 --2- 172.21.15.16:0/982997717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a9c1040e0 0x7f5a9c198ec0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:25.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:25.262+0000 7f5aa3714700 1 -- 172.21.15.16:0/982997717 >> 172.21.15.16:0/982997717 conn(0x7f5a9c0fb160 msgr2=0x7f5a9c100760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:43:25.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:25.262+0000 7f5aa3714700 1 -- 172.21.15.16:0/982997717 shutdown_connections 2024-04-22T15:43:25.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:25.262+0000 7f5aa3714700 1 -- 172.21.15.16:0/982997717 wait complete. 2024-04-22T15:43:25.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:25 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:25.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:25 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:25.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:25 smithi027 ceph-mon[34922]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:25.592 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:25 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:25.592 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:25 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:25.592 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:25 smithi016 ceph-mon[26620]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:25.905 DEBUG:teuthology.orchestra.run.smithi016:osd.1> sudo journalctl -f -n 0 -u ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.1.service 2024-04-22T15:43:25.908 INFO:tasks.cephadm:Deploying osd.2 on smithi016 with /dev/vg_nvme/lv_2... 2024-04-22T15:43:25.908 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T15:43:26.463 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:26 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:26.463 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:26 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:26.463 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:26 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:43:26.463 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:26 smithi016 ceph-mon[26620]: from='osd.1 [v2:172.21.15.16:6810/3974144580,v1:172.21.15.16:6811/3974144580]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T15:43:26.463 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:43:26 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1[53304]: 2024-04-22T15:43:26.170+0000 7f29dd499640 -1 osd.1 0 log_to_monitors true 2024-04-22T15:43:26.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:26 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:26.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:26 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:26.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:26 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:43:26.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:26 smithi027 ceph-mon[34922]: from='osd.1 [v2:172.21.15.16:6810/3974144580,v1:172.21.15.16:6811/3974144580]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T15:43:26.683 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:43:27.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:27 smithi016 ceph-mon[26620]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:27.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:27 smithi016 ceph-mon[26620]: from='osd.1 [v2:172.21.15.16:6810/3974144580,v1:172.21.15.16:6811/3974144580]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T15:43:27.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:27 smithi016 ceph-mon[26620]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T15:43:27.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:27 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:43:27.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:27 smithi016 ceph-mon[26620]: from='osd.1 [v2:172.21.15.16:6810/3974144580,v1:172.21.15.16:6811/3974144580]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-22T15:43:27.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:27 smithi027 ceph-mon[34922]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:27.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:27 smithi027 ceph-mon[34922]: from='osd.1 [v2:172.21.15.16:6810/3974144580,v1:172.21.15.16:6811/3974144580]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T15:43:27.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:27 smithi027 ceph-mon[34922]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T15:43:27.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:27 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:43:27.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:27 smithi027 ceph-mon[34922]: from='osd.1 [v2:172.21.15.16:6810/3974144580,v1:172.21.15.16:6811/3974144580]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-22T15:43:27.917 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:43:27 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1[53304]: 2024-04-22T15:43:27.611+0000 7f29d22fc700 -1 osd.1 0 waiting for initial osdmap 2024-04-22T15:43:27.917 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:43:27 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1[53304]: 2024-04-22T15:43:27.616+0000 7f29ca8ea700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T15:43:28.809 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:28 smithi016 ceph-mon[26620]: from='osd.1 [v2:172.21.15.16:6810/3974144580,v1:172.21.15.16:6811/3974144580]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-22T15:43:28.809 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:28 smithi016 ceph-mon[26620]: osdmap e13: 2 total, 1 up, 2 in 2024-04-22T15:43:28.809 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:28 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:43:28.810 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:28 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:43:28.810 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:28 smithi016 ceph-mon[26620]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:29.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:28 smithi027 ceph-mon[34922]: from='osd.1 [v2:172.21.15.16:6810/3974144580,v1:172.21.15.16:6811/3974144580]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-22T15:43:29.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:28 smithi027 ceph-mon[34922]: osdmap e13: 2 total, 1 up, 2 in 2024-04-22T15:43:29.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:28 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:43:29.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:28 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:43:29.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:28 smithi027 ceph-mon[34922]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T15:43:29.683 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T15:43:29.683 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T15:43:29.683 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-04-22T15:43:29.683 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-04-22T15:43:29.683 INFO:teuthology.orchestra.run.smithi016.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00984363 s, 1.1 GB/s 2024-04-22T15:43:29.683 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-04-22T15:43:29.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:29 smithi016 ceph-mon[26620]: purged_snaps scrub starts 2024-04-22T15:43:29.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:29 smithi016 ceph-mon[26620]: purged_snaps scrub ok 2024-04-22T15:43:29.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:29 smithi016 ceph-mon[26620]: osd.1 [v2:172.21.15.16:6810/3974144580,v1:172.21.15.16:6811/3974144580] boot 2024-04-22T15:43:29.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:29 smithi016 ceph-mon[26620]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T15:43:29.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:29 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:43:30.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:29 smithi027 ceph-mon[34922]: purged_snaps scrub starts 2024-04-22T15:43:30.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:29 smithi027 ceph-mon[34922]: purged_snaps scrub ok 2024-04-22T15:43:30.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:29 smithi027 ceph-mon[34922]: osd.1 [v2:172.21.15.16:6810/3974144580,v1:172.21.15.16:6811/3974144580] boot 2024-04-22T15:43:30.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:29 smithi027 ceph-mon[34922]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T15:43:30.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:29 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:43:30.629 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:30 smithi016 ceph-mon[26620]: osdmap e15: 2 total, 2 up, 2 in 2024-04-22T15:43:30.630 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:30 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:30.630 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:30 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:30.630 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:30 smithi016 ceph-mon[26620]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:30.651 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_2 2024-04-22T15:43:31.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:30 smithi027 ceph-mon[34922]: osdmap e15: 2 total, 2 up, 2 in 2024-04-22T15:43:31.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:30 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:31.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:30 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:31.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:30 smithi027 ceph-mon[34922]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:31.753 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:43:33.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:33 smithi016 ceph-mon[26620]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:33.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:33 smithi027 ceph-mon[34922]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:33.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.947+0000 7f65136ec700 1 -- 172.21.15.16:0/3371876989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f650c1040d0 msgr2=0x7f650c1064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:33.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.947+0000 7f65136ec700 1 --2- 172.21.15.16:0/3371876989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f650c1040d0 0x7f650c1064c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f64fc009910 tx=0x7f64fc009c20 comp rx=0 tx=0).stop 2024-04-22T15:43:33.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.947+0000 7f65136ec700 1 -- 172.21.15.16:0/3371876989 shutdown_connections 2024-04-22T15:43:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.947+0000 7f65136ec700 1 --2- 172.21.15.16:0/3371876989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f650c1040d0 0x7f650c1064c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.947+0000 7f65136ec700 1 --2- 172.21.15.16:0/3371876989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f650c1017a0 0x7f650c103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.947+0000 7f65136ec700 1 -- 172.21.15.16:0/3371876989 >> 172.21.15.16:0/3371876989 conn(0x7f650c0fb150 msgr2=0x7f650c0fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:43:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.947+0000 7f65136ec700 1 -- 172.21.15.16:0/3371876989 shutdown_connections 2024-04-22T15:43:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.947+0000 7f65136ec700 1 -- 172.21.15.16:0/3371876989 wait complete. 2024-04-22T15:43:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.948+0000 7f65136ec700 1 Processor -- start 2024-04-22T15:43:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.948+0000 7f65136ec700 1 -- start start 2024-04-22T15:43:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.948+0000 7f65136ec700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f650c1017a0 0x7f650c196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.948+0000 7f65136ec700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f650c1040d0 0x7f650c196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.948+0000 7f65136ec700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650c197070 con 0x7f650c1040d0 2024-04-22T15:43:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.948+0000 7f65136ec700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650c1971b0 con 0x7f650c1017a0 2024-04-22T15:43:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.948+0000 7f6510c87700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f650c1040d0 0x7f650c196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.948+0000 7f6510c87700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f650c1040d0 0x7f650c196a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53634/0 (socket says 172.21.15.16:53634) 2024-04-22T15:43:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.948+0000 7f6510c87700 1 -- 172.21.15.16:0/3704176496 learned_addr learned my addr 172.21.15.16:0/3704176496 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:43:33.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.949+0000 7f6511488700 1 --2- 172.21.15.16:0/3704176496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f650c1017a0 0x7f650c196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:33.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.949+0000 7f6510c87700 1 -- 172.21.15.16:0/3704176496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f650c1017a0 msgr2=0x7f650c196510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:33.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.949+0000 7f6510c87700 1 --2- 172.21.15.16:0/3704176496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f650c1017a0 0x7f650c196510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:33.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.949+0000 7f6510c87700 1 -- 172.21.15.16:0/3704176496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64fc0095f0 con 0x7f650c1040d0 2024-04-22T15:43:33.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.949+0000 7f6510c87700 1 --2- 172.21.15.16:0/3704176496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f650c1040d0 0x7f650c196a50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f64fc006010 tx=0x7f64fc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:43:33.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.949+0000 7f65027fc700 1 -- 172.21.15.16:0/3704176496 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64fc01d070 con 0x7f650c1040d0 2024-04-22T15:43:33.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.949+0000 7f65136ec700 1 -- 172.21.15.16:0/3704176496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f650c19bc00 con 0x7f650c1040d0 2024-04-22T15:43:33.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.949+0000 7f65027fc700 1 -- 172.21.15.16:0/3704176496 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f64fc00f460 con 0x7f650c1040d0 2024-04-22T15:43:33.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.950+0000 7f65027fc700 1 -- 172.21.15.16:0/3704176496 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64fc021640 con 0x7f650c1040d0 2024-04-22T15:43:33.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.950+0000 7f65136ec700 1 -- 172.21.15.16:0/3704176496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f650c19c0f0 con 0x7f650c1040d0 2024-04-22T15:43:33.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.951+0000 7f65027fc700 1 -- 172.21.15.16:0/3704176496 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 90256+0+0 (secure 0 0 0) 0x7f64fc02b440 con 0x7f650c1040d0 2024-04-22T15:43:33.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.951+0000 7f65136ec700 1 -- 172.21.15.16:0/3704176496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f650c1906a0 con 0x7f650c1040d0 2024-04-22T15:43:33.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.951+0000 7f65027fc700 1 --2- 172.21.15.16:0/3704176496 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f64f806c460 0x7f64f806e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:33.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.951+0000 7f65027fc700 1 -- 172.21.15.16:0/3704176496 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2414+0+0 (secure 0 0 0) 0x7f64fc08bf60 con 0x7f650c1040d0 2024-04-22T15:43:33.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.951+0000 7f6511488700 1 --2- 172.21.15.16:0/3704176496 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f64f806c460 0x7f64f806e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:33.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.952+0000 7f6511488700 1 --2- 172.21.15.16:0/3704176496 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f64f806c460 0x7f64f806e920 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6508006fd0 tx=0x7f6508008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:43:33.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:33.959+0000 7f65027fc700 1 -- 172.21.15.16:0/3704176496 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f64fc05afc0 con 0x7f650c1040d0 2024-04-22T15:43:34.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:34.170+0000 7f65136ec700 1 -- 172.21.15.16:0/3704176496 --> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f650c0611d0 con 0x7f64f806c460 2024-04-22T15:43:34.581 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:34 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T15:43:34.582 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:34 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T15:43:34.582 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:34 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:43:34.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:34 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T15:43:34.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:34 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T15:43:34.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:34 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:43:35.689 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:35 smithi027 ceph-mon[34922]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:43:35.690 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:35 smithi027 ceph-mon[34922]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:35.690 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:35 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:35.690 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:35 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:35.690 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:35 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:43:35.690 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:35 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:43:35.690 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:35 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:35.690 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:35 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:43:35.690 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:35 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:43:35.690 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:35 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:35.690 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:35 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:43:35.707 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:35 smithi016 ceph-mon[26620]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:43:35.707 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:35 smithi016 ceph-mon[26620]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:35.707 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:35 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:35.707 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:35 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:35.707 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:35 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:43:35.708 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:35 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:43:35.708 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:35 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:35.708 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:35 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:43:35.708 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:35 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:43:35.708 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:35 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:35.708 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:35 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:43:36.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:36 smithi027 ceph-mon[34922]: Adjusting osd_memory_target on smithi016 to 5922M 2024-04-22T15:43:36.762 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:36 smithi016 ceph-mon[26620]: Adjusting osd_memory_target on smithi016 to 5922M 2024-04-22T15:43:37.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:37 smithi016 ceph-mon[26620]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:37.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:37 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:37.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:37 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:37.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:37 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/3598576960' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e59fdb03-e04e-4afd-a58e-0b58945d3765"}]: dispatch 2024-04-22T15:43:37.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:37 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/3598576960' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e59fdb03-e04e-4afd-a58e-0b58945d3765"}]': finished 2024-04-22T15:43:37.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:37 smithi016 ceph-mon[26620]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T15:43:37.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:37 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:43:37.832 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:37 smithi027 ceph-mon[34922]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:37.832 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:37 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:37.832 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:37 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:37.832 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:37 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/3598576960' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e59fdb03-e04e-4afd-a58e-0b58945d3765"}]: dispatch 2024-04-22T15:43:37.833 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:37 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/3598576960' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e59fdb03-e04e-4afd-a58e-0b58945d3765"}]': finished 2024-04-22T15:43:37.833 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:37 smithi027 ceph-mon[34922]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T15:43:37.833 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:37 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:43:38.754 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:38 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/249757689' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T15:43:38.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:38 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/249757689' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T15:43:39.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:39 smithi027 ceph-mon[34922]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:39.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:39 smithi016 ceph-mon[26620]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:41.290 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:41 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:41.290 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:41 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:41.290 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:41 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:43:41.290 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:41 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:43:41.291 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:41 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:41.291 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:41 smithi016 ceph-mon[26620]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:41.291 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:41 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:43:41.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:41 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:41.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:41 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:41.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:41 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:43:41.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:41 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:43:41.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:41 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:41.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:41 smithi027 ceph-mon[34922]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:41.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:41 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:43:43.688 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:43 smithi016 ceph-mon[26620]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:43.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:43 smithi027 ceph-mon[34922]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:45.531 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:45 smithi016 ceph-mon[26620]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:45.532 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:45 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T15:43:45.532 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:45 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:43:45.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:45 smithi027 ceph-mon[34922]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:45.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:45 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T15:43:45.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:45 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:43:46.601 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:46 smithi016 ceph-mon[26620]: Deploying daemon osd.2 on smithi016 2024-04-22T15:43:46.798 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:46 smithi027 ceph-mon[34922]: Deploying daemon osd.2 on smithi016 2024-04-22T15:43:47.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:47 smithi027 ceph-mon[34922]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:47.777 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:47 smithi016 ceph-mon[26620]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:49.602 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:49 smithi016 ceph-mon[26620]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:49.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:49 smithi027 ceph-mon[34922]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:50.697 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:50 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:43:50.697 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:50 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:50.697 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:50 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:50.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:50 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:43:50.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:50 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:50.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:50 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:51.718 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:51 smithi016 ceph-mon[26620]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:51.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:51 smithi027 ceph-mon[34922]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:53.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:53 smithi016 ceph-mon[26620]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:53.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:53.823+0000 7f65027fc700 1 -- 172.21.15.16:0/3704176496 <== mgr.14217 v2:172.21.15.16:6800/1057234399 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f650c0611d0 con 0x7f64f806c460 2024-04-22T15:43:53.826 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 2 on host 'smithi016' 2024-04-22T15:43:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:53.825+0000 7f65136ec700 1 -- 172.21.15.16:0/3704176496 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f64f806c460 msgr2=0x7f64f806e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:53.825+0000 7f65136ec700 1 --2- 172.21.15.16:0/3704176496 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f64f806c460 0x7f64f806e920 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6508006fd0 tx=0x7f6508008040 comp rx=0 tx=0).stop 2024-04-22T15:43:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:53.825+0000 7f65136ec700 1 -- 172.21.15.16:0/3704176496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f650c1040d0 msgr2=0x7f650c196a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:53.825+0000 7f65136ec700 1 --2- 172.21.15.16:0/3704176496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f650c1040d0 0x7f650c196a50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f64fc006010 tx=0x7f64fc00be20 comp rx=0 tx=0).stop 2024-04-22T15:43:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:53.826+0000 7f65136ec700 1 -- 172.21.15.16:0/3704176496 shutdown_connections 2024-04-22T15:43:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:53.826+0000 7f65136ec700 1 --2- 172.21.15.16:0/3704176496 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f64f806c460 0x7f64f806e920 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:53.826+0000 7f65136ec700 1 --2- 172.21.15.16:0/3704176496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f650c1017a0 0x7f650c196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:53.826+0000 7f65136ec700 1 --2- 172.21.15.16:0/3704176496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f650c1040d0 0x7f650c196a50 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:53.826+0000 7f65136ec700 1 -- 172.21.15.16:0/3704176496 >> 172.21.15.16:0/3704176496 conn(0x7f650c0fb150 msgr2=0x7f650c0fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:43:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:53.826+0000 7f65136ec700 1 -- 172.21.15.16:0/3704176496 shutdown_connections 2024-04-22T15:43:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:43:53.826+0000 7f65136ec700 1 -- 172.21.15.16:0/3704176496 wait complete. 2024-04-22T15:43:53.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:53 smithi027 ceph-mon[34922]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:54.564 DEBUG:teuthology.orchestra.run.smithi016:osd.2> sudo journalctl -f -n 0 -u ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.2.service 2024-04-22T15:43:54.567 INFO:tasks.cephadm:Deploying osd.3 on smithi027 with /dev/vg_nvme/lv_4... 2024-04-22T15:43:54.567 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T15:43:54.789 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi027/config 2024-04-22T15:43:54.910 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:54 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:54.910 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:54 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:54.910 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:54 smithi027 ceph-mon[34922]: from='osd.2 [v2:172.21.15.16:6818/2760833880,v1:172.21.15.16:6819/2760833880]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T15:43:54.910 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:54 smithi027 ceph-mon[34922]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:54.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:54 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:54.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:54 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:54.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:54 smithi016 ceph-mon[26620]: from='osd.2 [v2:172.21.15.16:6818/2760833880,v1:172.21.15.16:6819/2760833880]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T15:43:54.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:54 smithi016 ceph-mon[26620]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:56.050 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:55 smithi016 ceph-mon[26620]: from='osd.2 [v2:172.21.15.16:6818/2760833880,v1:172.21.15.16:6819/2760833880]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T15:43:56.050 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:55 smithi016 ceph-mon[26620]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T15:43:56.051 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:55 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:43:56.051 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:55 smithi016 ceph-mon[26620]: from='osd.2 [v2:172.21.15.16:6818/2760833880,v1:172.21.15.16:6819/2760833880]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-22T15:43:56.051 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:55 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:43:56.051 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:43:55 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2[60684]: 2024-04-22T15:43:55.792+0000 7f9aaffff700 -1 osd.2 0 waiting for initial osdmap 2024-04-22T15:43:56.051 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:43:55 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2[60684]: 2024-04-22T15:43:55.797+0000 7f9aaa5f1700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T15:43:56.053 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:55 smithi027 ceph-mon[34922]: from='osd.2 [v2:172.21.15.16:6818/2760833880,v1:172.21.15.16:6819/2760833880]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T15:43:56.053 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:55 smithi027 ceph-mon[34922]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T15:43:56.053 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:55 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:43:56.054 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:55 smithi027 ceph-mon[34922]: from='osd.2 [v2:172.21.15.16:6818/2760833880,v1:172.21.15.16:6819/2760833880]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-22T15:43:56.054 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:55 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:43:56.557 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T15:43:56.557 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T15:43:56.557 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-04-22T15:43:56.557 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-04-22T15:43:56.557 INFO:teuthology.orchestra.run.smithi027.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0120767 s, 868 MB/s 2024-04-22T15:43:56.557 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-04-22T15:43:56.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:56 smithi027 ceph-mon[34922]: from='osd.2 [v2:172.21.15.16:6818/2760833880,v1:172.21.15.16:6819/2760833880]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-22T15:43:56.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:56 smithi027 ceph-mon[34922]: osdmap e18: 3 total, 2 up, 3 in 2024-04-22T15:43:56.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:56 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:43:56.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:56 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:43:56.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:56 smithi027 ceph-mon[34922]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:57.093 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_4 2024-04-22T15:43:57.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:56 smithi016 ceph-mon[26620]: from='osd.2 [v2:172.21.15.16:6818/2760833880,v1:172.21.15.16:6819/2760833880]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-22T15:43:57.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:56 smithi016 ceph-mon[26620]: osdmap e18: 3 total, 2 up, 3 in 2024-04-22T15:43:57.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:56 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:43:57.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:56 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:43:57.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:56 smithi016 ceph-mon[26620]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T15:43:57.317 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi027/config 2024-04-22T15:43:57.915 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:57 smithi016 ceph-mon[26620]: purged_snaps scrub starts 2024-04-22T15:43:57.915 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:57 smithi016 ceph-mon[26620]: purged_snaps scrub ok 2024-04-22T15:43:57.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:57 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:43:57.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:57 smithi016 ceph-mon[26620]: osd.2 [v2:172.21.15.16:6818/2760833880,v1:172.21.15.16:6819/2760833880] boot 2024-04-22T15:43:57.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:57 smithi016 ceph-mon[26620]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T15:43:57.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:57 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:43:57.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:57 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:57.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:57 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:58.070 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:57 smithi027 ceph-mon[34922]: purged_snaps scrub starts 2024-04-22T15:43:58.070 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:57 smithi027 ceph-mon[34922]: purged_snaps scrub ok 2024-04-22T15:43:58.070 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:57 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:43:58.070 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:57 smithi027 ceph-mon[34922]: osd.2 [v2:172.21.15.16:6818/2760833880,v1:172.21.15.16:6819/2760833880] boot 2024-04-22T15:43:58.070 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:57 smithi027 ceph-mon[34922]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T15:43:58.070 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:57 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:43:58.070 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:57 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:58.070 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:57 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:43:58.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.808+0000 7ff232a5c700 1 -- 172.21.15.27:0/1317026966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff22c101eb0 msgr2=0x7ff22c102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:58.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.808+0000 7ff232a5c700 1 --2- 172.21.15.27:0/1317026966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff22c101eb0 0x7ff22c102310 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff220009910 tx=0x7ff220009c20 comp rx=0 tx=0).stop 2024-04-22T15:43:58.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.808+0000 7ff232a5c700 1 -- 172.21.15.27:0/1317026966 shutdown_connections 2024-04-22T15:43:58.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.808+0000 7ff232a5c700 1 --2- 172.21.15.27:0/1317026966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff22c101eb0 0x7ff22c102310 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:58.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.808+0000 7ff232a5c700 1 --2- 172.21.15.27:0/1317026966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff22c100cb0 0x7ff22c1010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:58.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.808+0000 7ff232a5c700 1 -- 172.21.15.27:0/1317026966 >> 172.21.15.27:0/1317026966 conn(0x7ff22c0fc230 msgr2=0x7ff22c0fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:43:58.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.808+0000 7ff232a5c700 1 -- 172.21.15.27:0/1317026966 shutdown_connections 2024-04-22T15:43:58.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.808+0000 7ff232a5c700 1 -- 172.21.15.27:0/1317026966 wait complete. 2024-04-22T15:43:58.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.809+0000 7ff232a5c700 1 Processor -- start 2024-04-22T15:43:58.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.809+0000 7ff232a5c700 1 -- start start 2024-04-22T15:43:58.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.809+0000 7ff232a5c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff22c100cb0 0x7ff22c196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:58.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.809+0000 7ff232a5c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff22c101eb0 0x7ff22c1969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:58.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.809+0000 7ff232a5c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22c196fe0 con 0x7ff22c101eb0 2024-04-22T15:43:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.809+0000 7ff232a5c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22c197120 con 0x7ff22c100cb0 2024-04-22T15:43:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.809+0000 7ff22bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff22c100cb0 0x7ff22c196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.809+0000 7ff22bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff22c100cb0 0x7ff22c196480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41058/0 (socket says 172.21.15.27:41058) 2024-04-22T15:43:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.809+0000 7ff22bfff700 1 -- 172.21.15.27:0/1540091756 learned_addr learned my addr 172.21.15.27:0/1540091756 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-22T15:43:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.809+0000 7ff22b7fe700 1 --2- 172.21.15.27:0/1540091756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff22c101eb0 0x7ff22c1969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.810+0000 7ff22bfff700 1 -- 172.21.15.27:0/1540091756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff22c101eb0 msgr2=0x7ff22c1969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:43:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.810+0000 7ff22bfff700 1 --2- 172.21.15.27:0/1540091756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff22c101eb0 0x7ff22c1969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:43:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.810+0000 7ff22bfff700 1 -- 172.21.15.27:0/1540091756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2200095f0 con 0x7ff22c100cb0 2024-04-22T15:43:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.810+0000 7ff22bfff700 1 --2- 172.21.15.27:0/1540091756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff22c100cb0 0x7ff22c196480 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff21c00d900 tx=0x7ff21c00dc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:43:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.810+0000 7ff2297fa700 1 -- 172.21.15.27:0/1540091756 <== mon.1 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff21c0041f0 con 0x7ff22c100cb0 2024-04-22T15:43:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.810+0000 7ff2297fa700 1 -- 172.21.15.27:0/1540091756 <== mon.1 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff21c004350 con 0x7ff22c100cb0 2024-04-22T15:43:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.810+0000 7ff2297fa700 1 -- 172.21.15.27:0/1540091756 <== mon.1 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff21c003d20 con 0x7ff22c100cb0 2024-04-22T15:43:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.810+0000 7ff232a5c700 1 -- 172.21.15.27:0/1540091756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff22c19bbd0 con 0x7ff22c100cb0 2024-04-22T15:43:58.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.810+0000 7ff232a5c700 1 -- 172.21.15.27:0/1540091756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff22c19c120 con 0x7ff22c100cb0 2024-04-22T15:43:58.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.811+0000 7ff232a5c700 1 -- 172.21.15.27:0/1540091756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff22c109810 con 0x7ff22c100cb0 2024-04-22T15:43:58.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.811+0000 7ff2297fa700 1 -- 172.21.15.27:0/1540091756 <== mon.1 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 90256+0+0 (secure 0 0 0) 0x7ff21c009730 con 0x7ff22c100cb0 2024-04-22T15:43:58.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.812+0000 7ff2297fa700 1 --2- 172.21.15.27:0/1540091756 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff21406c460 0x7ff21406e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:43:58.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.812+0000 7ff2297fa700 1 -- 172.21.15.27:0/1540091756 <== mon.1 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 2846+0+0 (secure 0 0 0) 0x7ff21c08aaa0 con 0x7ff22c100cb0 2024-04-22T15:43:58.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.812+0000 7ff22b7fe700 1 --2- 172.21.15.27:0/1540091756 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff21406c460 0x7ff21406e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:43:58.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.812+0000 7ff22b7fe700 1 --2- 172.21.15.27:0/1540091756 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff21406c460 0x7ff21406e920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff220006010 tx=0x7ff22000b540 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:43:58.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:58.817+0000 7ff2297fa700 1 -- 172.21.15.27:0/1540091756 <== mon.1 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff21c059960 con 0x7ff22c100cb0 2024-04-22T15:43:59.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:43:59.025+0000 7ff232a5c700 1 -- 172.21.15.27:0/1540091756 --> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff22c106800 con 0x7ff21406c460 2024-04-22T15:43:59.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:59 smithi027 ceph-mon[34922]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T15:43:59.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:59 smithi027 ceph-mon[34922]: pgmap v77: 0 pgs: ; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:43:59.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:59 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-22T15:43:59.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:59 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T15:43:59.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:59 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T15:43:59.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:43:59 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:43:59.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:59 smithi016 ceph-mon[26620]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T15:43:59.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:59 smithi016 ceph-mon[26620]: pgmap v77: 0 pgs: ; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:43:59.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:59 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-22T15:43:59.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:59 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T15:43:59.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:59 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T15:43:59.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:43:59 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:44:00.268 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:00 smithi016 ceph-mon[26620]: from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:44:00.269 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:00 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-22T15:44:00.269 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:00 smithi016 ceph-mon[26620]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T15:44:00.269 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:00 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T15:44:00.419 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:00 smithi027 ceph-mon[34922]: from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:44:00.419 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:00 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-22T15:44:00.419 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:00 smithi027 ceph-mon[34922]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T15:44:00.419 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:00 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T15:44:01.417 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:44:01 smithi016 sudo[65498]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T15:44:01.417 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:44:01 smithi016 sudo[65498]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T15:44:01.417 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:44:01 smithi016 sudo[65498]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T15:44:01.417 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:44:01 smithi016 sudo[65498]: pam_unix(sudo:session): session closed for user root 2024-04-22T15:44:01.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:01 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T15:44:01.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:01 smithi016 ceph-mon[26620]: osdmap e22: 3 total, 3 up, 3 in 2024-04-22T15:44:01.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:01 smithi016 ceph-mon[26620]: Detected new or changed devices on smithi016 2024-04-22T15:44:01.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:01 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:01.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:01 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:01.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:01 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:44:01.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:01 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:44:01.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:01 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:44:01.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:01 smithi016 ceph-mon[26620]: Adjusting osd_memory_target on smithi016 to 3948M 2024-04-22T15:44:01.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:01 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:01.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:01 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:44:01.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:01 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:44:01.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:01 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:01.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:01 smithi016 ceph-mon[26620]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:44:01.419 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:01 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:44:01.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:01 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T15:44:01.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:01 smithi027 ceph-mon[34922]: osdmap e22: 3 total, 3 up, 3 in 2024-04-22T15:44:01.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:01 smithi027 ceph-mon[34922]: Detected new or changed devices on smithi016 2024-04-22T15:44:01.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:01 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:01.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:01 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:01.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:01 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:44:01.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:01 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:44:01.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:01 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:44:01.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:01 smithi027 ceph-mon[34922]: Adjusting osd_memory_target on smithi016 to 3948M 2024-04-22T15:44:01.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:01 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:01.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:01 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:44:01.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:01 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:44:01.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:01 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:01.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:01 smithi027 ceph-mon[34922]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:44:01.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:01 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:44:02.044 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:44:01 smithi016 sudo[65556]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T15:44:02.044 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:44:01 smithi016 sudo[65556]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T15:44:02.045 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:44:01 smithi016 sudo[65556]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T15:44:02.045 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:44:01 smithi016 sudo[65556]: pam_unix(sudo:session): session closed for user root 2024-04-22T15:44:02.333 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:44:02 smithi016 sudo[65625]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T15:44:02.333 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:44:02 smithi016 sudo[65625]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T15:44:02.333 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:44:02 smithi016 sudo[65625]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T15:44:02.333 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:44:02 smithi016 sudo[65625]: pam_unix(sudo:session): session closed for user root 2024-04-22T15:44:02.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:02 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/42224518' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa25b413-d004-4455-8d7b-0ced7f880a7a"}]: dispatch 2024-04-22T15:44:02.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:02 smithi016 ceph-mon[26620]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa25b413-d004-4455-8d7b-0ced7f880a7a"}]: dispatch 2024-04-22T15:44:02.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:02 smithi016 ceph-mon[26620]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa25b413-d004-4455-8d7b-0ced7f880a7a"}]': finished 2024-04-22T15:44:02.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:02 smithi016 ceph-mon[26620]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T15:44:02.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:02 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:44:02.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:02 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:02.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:02 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/800064099' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T15:44:02.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:02 smithi027 ceph-mon[34922]: from='client.? 172.21.15.27:0/42224518' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa25b413-d004-4455-8d7b-0ced7f880a7a"}]: dispatch 2024-04-22T15:44:02.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:02 smithi027 ceph-mon[34922]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa25b413-d004-4455-8d7b-0ced7f880a7a"}]: dispatch 2024-04-22T15:44:02.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:02 smithi027 ceph-mon[34922]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa25b413-d004-4455-8d7b-0ced7f880a7a"}]': finished 2024-04-22T15:44:02.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:02 smithi027 ceph-mon[34922]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T15:44:02.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:02 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:44:02.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:02 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:02.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:02 smithi027 ceph-mon[34922]: from='client.? 172.21.15.27:0/800064099' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T15:44:03.029 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:44:02 smithi016 sudo[65673]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T15:44:03.029 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:44:02 smithi016 sudo[65673]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T15:44:03.029 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:44:02 smithi016 sudo[65673]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T15:44:03.030 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:44:02 smithi016 sudo[65673]: pam_unix(sudo:session): session closed for user root 2024-04-22T15:44:03.370 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:44:03 smithi016 sudo[65704]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T15:44:03.370 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:44:03 smithi016 sudo[65704]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T15:44:03.370 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:44:03 smithi016 sudo[65704]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T15:44:03.370 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:44:03 smithi016 sudo[65704]: pam_unix(sudo:session): session closed for user root 2024-04-22T15:44:03.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:03.371+0000 7ff2297fa700 1 -- 172.21.15.27:0/1540091756 <== mon.1 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7ff21c0562a0 con 0x7ff22c100cb0 2024-04-22T15:44:03.648 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:03 smithi016 ceph-mon[26620]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:44:03.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:03 smithi027 ceph-mon[34922]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:44:03.916 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:44:03 smithi016 sudo[65770]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T15:44:03.917 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:44:03 smithi016 sudo[65770]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T15:44:03.917 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:44:03 smithi016 sudo[65770]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T15:44:03.917 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:44:03 smithi016 sudo[65770]: pam_unix(sudo:session): session closed for user root 2024-04-22T15:44:04.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:04 smithi016 sudo[65833]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T15:44:04.429 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:04 smithi016 sudo[65833]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T15:44:04.429 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:04 smithi016 sudo[65833]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T15:44:04.429 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:04 smithi016 sudo[65833]: pam_unix(sudo:session): session closed for user root 2024-04-22T15:44:04.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:04 smithi027 ceph-mon[34922]: mgrmap e19: smithi016.rnsnnr(active, since 2m), standbys: smithi027.ukdgzs 2024-04-22T15:44:04.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:04 smithi027 ceph-mon[34922]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T15:44:04.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:04 smithi016 sudo[65864]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T15:44:04.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:04 smithi016 sudo[65864]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T15:44:04.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:04 smithi016 sudo[65864]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T15:44:04.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:04 smithi016 sudo[65864]: pam_unix(sudo:session): session closed for user root 2024-04-22T15:44:04.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:04 smithi016 ceph-mon[26620]: mgrmap e19: smithi016.rnsnnr(active, since 2m), standbys: smithi027.ukdgzs 2024-04-22T15:44:04.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:04 smithi016 ceph-mon[26620]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T15:44:05.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:05 smithi027 sudo[40087]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T15:44:05.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:05 smithi027 sudo[40087]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T15:44:05.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:05 smithi027 sudo[40087]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T15:44:05.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:05 smithi027 sudo[40087]: pam_unix(sudo:session): session closed for user root 2024-04-22T15:44:05.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:05 smithi027 sudo[40203]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T15:44:05.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:05 smithi027 sudo[40203]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T15:44:05.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:05 smithi027 sudo[40203]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T15:44:05.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:05 smithi027 sudo[40203]: pam_unix(sudo:session): session closed for user root 2024-04-22T15:44:05.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:05 smithi027 ceph-mon[34922]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:44:05.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:05 smithi027 ceph-mon[34922]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T15:44:05.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:05 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-04-22T15:44:05.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:05 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:44:05.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:05 smithi027 ceph-mon[34922]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T15:44:05.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:05 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-04-22T15:44:05.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:05 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:44:05.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:05 smithi027 ceph-mon[34922]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T15:44:05.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:05 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:05.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:05 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:05.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:05 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:05.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:05 smithi016 ceph-mon[26620]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:44:05.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:05 smithi016 ceph-mon[26620]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T15:44:05.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:05 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-04-22T15:44:05.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:05 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:44:05.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:05 smithi016 ceph-mon[26620]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T15:44:05.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:05 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-04-22T15:44:05.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:05 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:44:05.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:05 smithi016 ceph-mon[26620]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T15:44:05.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:05 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:05.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:05 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:05.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:05 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:06.881 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:06 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:44:06.881 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:06 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:44:06.881 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:06 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:06.881 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:06 smithi027 ceph-mon[34922]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:44:06.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:06 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:44:06.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:06 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:44:06.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:06 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:06.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:06 smithi016 ceph-mon[26620]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:44:09.760 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:09 smithi027 ceph-mon[34922]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:44:09.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:09 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T15:44:09.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:09 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:44:09.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:09 smithi016 ceph-mon[26620]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:44:09.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:09 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T15:44:09.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:09 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:44:10.713 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:10 smithi027 ceph-mon[34922]: Deploying daemon osd.3 on smithi027 2024-04-22T15:44:10.713 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:10 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:44:10.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:10 smithi016 ceph-mon[26620]: Deploying daemon osd.3 on smithi027 2024-04-22T15:44:10.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:10 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:44:11.653 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:11 smithi027 ceph-mon[34922]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-22T15:44:11.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:11 smithi016 ceph-mon[26620]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-22T15:44:13.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:13 smithi027 ceph-mon[34922]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-22T15:44:13.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:13 smithi016 ceph-mon[26620]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-22T15:44:15.528 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:15 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:15.528 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:15 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:15.528 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:15 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:44:15.528 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:15 smithi027 ceph-mon[34922]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-22T15:44:15.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:15 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:15.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:15 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:15.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:15 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:44:15.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:15 smithi016 ceph-mon[26620]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-22T15:44:16.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:16 smithi027 ceph-mon[34922]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-22T15:44:16.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:16 smithi016 ceph-mon[26620]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-22T15:44:17.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:17.188+0000 7ff2297fa700 1 -- 172.21.15.27:0/1540091756 <== mgr.14217 v2:172.21.15.16:6800/1057234399 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff22c106800 con 0x7ff21406c460 2024-04-22T15:44:17.192 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 3 on host 'smithi027' 2024-04-22T15:44:17.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:17.191+0000 7ff232a5c700 1 -- 172.21.15.27:0/1540091756 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff21406c460 msgr2=0x7ff21406e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:17.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:17.191+0000 7ff232a5c700 1 --2- 172.21.15.27:0/1540091756 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff21406c460 0x7ff21406e920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff220006010 tx=0x7ff22000b540 comp rx=0 tx=0).stop 2024-04-22T15:44:17.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:17.191+0000 7ff232a5c700 1 -- 172.21.15.27:0/1540091756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff22c100cb0 msgr2=0x7ff22c196480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:17.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:17.191+0000 7ff232a5c700 1 --2- 172.21.15.27:0/1540091756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff22c100cb0 0x7ff22c196480 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff21c00d900 tx=0x7ff21c00dc10 comp rx=0 tx=0).stop 2024-04-22T15:44:17.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:17.191+0000 7ff232a5c700 1 -- 172.21.15.27:0/1540091756 shutdown_connections 2024-04-22T15:44:17.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:17.191+0000 7ff232a5c700 1 --2- 172.21.15.27:0/1540091756 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff21406c460 0x7ff21406e920 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:17.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:17.191+0000 7ff232a5c700 1 --2- 172.21.15.27:0/1540091756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff22c100cb0 0x7ff22c196480 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:17.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:17.191+0000 7ff232a5c700 1 --2- 172.21.15.27:0/1540091756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff22c101eb0 0x7ff22c1969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:17.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:17.191+0000 7ff232a5c700 1 -- 172.21.15.27:0/1540091756 >> 172.21.15.27:0/1540091756 conn(0x7ff22c0fc230 msgr2=0x7ff22c1050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:17.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:17.191+0000 7ff232a5c700 1 -- 172.21.15.27:0/1540091756 shutdown_connections 2024-04-22T15:44:17.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:17.191+0000 7ff232a5c700 1 -- 172.21.15.27:0/1540091756 wait complete. 2024-04-22T15:44:17.766 DEBUG:teuthology.orchestra.run.smithi027:osd.3> sudo journalctl -f -n 0 -u ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.3.service 2024-04-22T15:44:17.768 INFO:tasks.cephadm:Deploying osd.4 on smithi027 with /dev/vg_nvme/lv_3... 2024-04-22T15:44:17.768 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T15:44:18.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:18 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:18.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:18 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:18.337 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:44:18 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3[40990]: 2024-04-22T15:44:18.151+0000 7f8e420a2640 -1 osd.3 0 log_to_monitors true 2024-04-22T15:44:18.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:18 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:18.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:18 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:18.526 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi027/config 2024-04-22T15:44:19.232 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:19 smithi027 ceph-mon[34922]: from='osd.3 [v2:172.21.15.27:6800/286252834,v1:172.21.15.27:6801/286252834]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T15:44:19.232 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:19 smithi027 ceph-mon[34922]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T15:44:19.232 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:19 smithi027 ceph-mon[34922]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-22T15:44:19.232 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:19 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:19.232 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:19 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:19.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:19 smithi016 ceph-mon[26620]: from='osd.3 [v2:172.21.15.27:6800/286252834,v1:172.21.15.27:6801/286252834]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T15:44:19.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:19 smithi016 ceph-mon[26620]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T15:44:19.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:19 smithi016 ceph-mon[26620]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-22T15:44:19.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:19 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:19.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:19 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:20.321 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T15:44:20.321 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T15:44:20.321 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-04-22T15:44:20.321 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-04-22T15:44:20.321 INFO:teuthology.orchestra.run.smithi027.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0139478 s, 752 MB/s 2024-04-22T15:44:20.321 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-04-22T15:44:20.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:20 smithi016 ceph-mon[26620]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T15:44:20.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:20 smithi016 ceph-mon[26620]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T15:44:20.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:20 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:44:20.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:20 smithi016 ceph-mon[26620]: from='osd.3 [v2:172.21.15.27:6800/286252834,v1:172.21.15.27:6801/286252834]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-22T15:44:20.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:20 smithi016 ceph-mon[26620]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-22T15:44:20.464 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:20 smithi027 ceph-mon[34922]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T15:44:20.464 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:20 smithi027 ceph-mon[34922]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T15:44:20.464 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:20 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:44:20.464 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:20 smithi027 ceph-mon[34922]: from='osd.3 [v2:172.21.15.27:6800/286252834,v1:172.21.15.27:6801/286252834]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-22T15:44:20.464 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:20 smithi027 ceph-mon[34922]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-22T15:44:20.465 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:44:20 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3[40990]: 2024-04-22T15:44:20.157+0000 7f8e36f05700 -1 osd.3 0 waiting for initial osdmap 2024-04-22T15:44:20.465 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:44:20 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3[40990]: 2024-04-22T15:44:20.163+0000 7f8e2f4f3700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T15:44:21.243 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:21 smithi027 ceph-mon[34922]: purged_snaps scrub starts 2024-04-22T15:44:21.243 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:21 smithi027 ceph-mon[34922]: purged_snaps scrub ok 2024-04-22T15:44:21.243 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:21 smithi027 ceph-mon[34922]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-22T15:44:21.243 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:21 smithi027 ceph-mon[34922]: osdmap e25: 4 total, 3 up, 4 in 2024-04-22T15:44:21.243 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:21 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:44:21.243 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:21 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:44:21.243 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:21 smithi027 ceph-mon[34922]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:44:21.244 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:21 smithi027 ceph-mon[34922]: osd.3 [v2:172.21.15.27:6800/286252834,v1:172.21.15.27:6801/286252834] boot 2024-04-22T15:44:21.244 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:21 smithi027 ceph-mon[34922]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T15:44:21.244 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:21 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:44:21.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:21 smithi016 ceph-mon[26620]: purged_snaps scrub starts 2024-04-22T15:44:21.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:21 smithi016 ceph-mon[26620]: purged_snaps scrub ok 2024-04-22T15:44:21.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:21 smithi016 ceph-mon[26620]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-22T15:44:21.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:21 smithi016 ceph-mon[26620]: osdmap e25: 4 total, 3 up, 4 in 2024-04-22T15:44:21.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:21 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:44:21.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:21 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:44:21.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:21 smithi016 ceph-mon[26620]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-22T15:44:21.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:21 smithi016 ceph-mon[26620]: osd.3 [v2:172.21.15.27:6800/286252834,v1:172.21.15.27:6801/286252834] boot 2024-04-22T15:44:21.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:21 smithi016 ceph-mon[26620]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T15:44:21.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:21 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:44:22.113 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_3 2024-04-22T15:44:22.523 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi027/config 2024-04-22T15:44:22.878 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:22 smithi027 ceph-mon[34922]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T15:44:22.878 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:22 smithi027 ceph-mon[34922]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T15:44:22.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:22 smithi016 ceph-mon[26620]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T15:44:22.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:22 smithi016 ceph-mon[26620]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T15:44:23.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:23 smithi016 ceph-mon[26620]: osdmap e28: 4 total, 4 up, 4 in 2024-04-22T15:44:23.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:23 smithi016 ceph-mon[26620]: Detected new or changed devices on smithi027 2024-04-22T15:44:23.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:23 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:23.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:23 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:23.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:23 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:44:23.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:23 smithi016 ceph-mon[26620]: Adjusting osd_memory_target on smithi027 to 14912M 2024-04-22T15:44:23.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:23 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:23.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:23 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:44:23.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:23 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:44:23.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:23 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:23.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:23 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:44:23.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:23 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:44:23.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:23 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:44:23.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:23 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:23.934 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:23 smithi027 ceph-mon[34922]: osdmap e28: 4 total, 4 up, 4 in 2024-04-22T15:44:23.934 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:23 smithi027 ceph-mon[34922]: Detected new or changed devices on smithi027 2024-04-22T15:44:23.935 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:23 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:23.935 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:23 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:23.935 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:23 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:44:23.935 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:23 smithi027 ceph-mon[34922]: Adjusting osd_memory_target on smithi027 to 14912M 2024-04-22T15:44:23.935 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:23 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:23.935 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:23 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:44:23.935 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:23 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:44:23.935 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:23 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:23.935 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:23 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:44:23.935 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:23 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:44:23.935 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:23 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:44:23.935 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:23 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:24.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.434+0000 7f50650a6700 1 -- 172.21.15.27:0/2915805020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5060101840 msgr2=0x7f5060103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:24.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.434+0000 7f50650a6700 1 --2- 172.21.15.27:0/2915805020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5060101840 0x7f5060103c30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f505000b0a0 tx=0x7f505000b3b0 comp rx=0 tx=0).stop 2024-04-22T15:44:24.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.434+0000 7f50650a6700 1 -- 172.21.15.27:0/2915805020 shutdown_connections 2024-04-22T15:44:24.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.434+0000 7f50650a6700 1 --2- 172.21.15.27:0/2915805020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5060104170 0x7f5060106560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:24.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.434+0000 7f50650a6700 1 --2- 172.21.15.27:0/2915805020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5060101840 0x7f5060103c30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:24.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.434+0000 7f50650a6700 1 -- 172.21.15.27:0/2915805020 >> 172.21.15.27:0/2915805020 conn(0x7f50600fb190 msgr2=0x7f50600fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:24.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.434+0000 7f50650a6700 1 -- 172.21.15.27:0/2915805020 shutdown_connections 2024-04-22T15:44:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.434+0000 7f50650a6700 1 -- 172.21.15.27:0/2915805020 wait complete. 2024-04-22T15:44:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.434+0000 7f50650a6700 1 Processor -- start 2024-04-22T15:44:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.435+0000 7f50650a6700 1 -- start start 2024-04-22T15:44:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.435+0000 7f50650a6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5060101840 0x7f50601964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.435+0000 7f50650a6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5060104170 0x7f50601969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.435+0000 7f50650a6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5060197000 con 0x7f5060101840 2024-04-22T15:44:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.435+0000 7f50650a6700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5060197140 con 0x7f5060104170 2024-04-22T15:44:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.435+0000 7f505e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5060104170 0x7f50601969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.435+0000 7f505e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5060104170 0x7f50601969e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53644/0 (socket says 172.21.15.27:53644) 2024-04-22T15:44:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.435+0000 7f505e59c700 1 -- 172.21.15.27:0/560311711 learned_addr learned my addr 172.21.15.27:0/560311711 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-22T15:44:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.435+0000 7f505ed9d700 1 --2- 172.21.15.27:0/560311711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5060101840 0x7f50601964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.435+0000 7f505e59c700 1 -- 172.21.15.27:0/560311711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5060101840 msgr2=0x7f50601964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.435+0000 7f505e59c700 1 --2- 172.21.15.27:0/560311711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5060101840 0x7f50601964a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.435+0000 7f505e59c700 1 -- 172.21.15.27:0/560311711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5050009d00 con 0x7f5060104170 2024-04-22T15:44:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.435+0000 7f505ed9d700 1 --2- 172.21.15.27:0/560311711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5060101840 0x7f50601964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:44:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.436+0000 7f505e59c700 1 --2- 172.21.15.27:0/560311711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5060104170 0x7f50601969e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f504800d900 tx=0x7f504800dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.436+0000 7f5057fff700 1 -- 172.21.15.27:0/560311711 <== mon.1 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50480041f0 con 0x7f5060104170 2024-04-22T15:44:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.436+0000 7f50650a6700 1 -- 172.21.15.27:0/560311711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f506019bbf0 con 0x7f5060104170 2024-04-22T15:44:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.436+0000 7f50650a6700 1 -- 172.21.15.27:0/560311711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f506019c140 con 0x7f5060104170 2024-04-22T15:44:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.436+0000 7f5057fff700 1 -- 172.21.15.27:0/560311711 <== mon.1 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5048004ce0 con 0x7f5060104170 2024-04-22T15:44:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.436+0000 7f5057fff700 1 -- 172.21.15.27:0/560311711 <== mon.1 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f504800b750 con 0x7f5060104170 2024-04-22T15:44:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.437+0000 7f50650a6700 1 -- 172.21.15.27:0/560311711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5060190620 con 0x7f5060104170 2024-04-22T15:44:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.438+0000 7f5057fff700 1 -- 172.21.15.27:0/560311711 <== mon.1 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f5048004350 con 0x7f5060104170 2024-04-22T15:44:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.438+0000 7f5057fff700 1 --2- 172.21.15.27:0/560311711 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f504c06c4e0 0x7f504c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.438+0000 7f5057fff700 1 -- 172.21.15.27:0/560311711 <== mon.1 v2:172.21.15.27:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3774+0+0 (secure 0 0 0) 0x7f504808b090 con 0x7f5060104170 2024-04-22T15:44:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.438+0000 7f505ed9d700 1 --2- 172.21.15.27:0/560311711 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f504c06c4e0 0x7f504c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.439+0000 7f505ed9d700 1 --2- 172.21.15.27:0/560311711 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f504c06c4e0 0x7f504c06e9a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f505000b830 tx=0x7f5050006210 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:24.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.443+0000 7f5057fff700 1 -- 172.21.15.27:0/560311711 <== mon.1 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f50480599a0 con 0x7f5060104170 2024-04-22T15:44:24.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:24.651+0000 7f50650a6700 1 -- 172.21.15.27:0/560311711 --> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f50600611d0 con 0x7f504c06c4e0 2024-04-22T15:44:24.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:24 smithi016 ceph-mon[26620]: osdmap e29: 4 total, 4 up, 4 in 2024-04-22T15:44:24.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:24 smithi016 ceph-mon[26620]: pgmap v99: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-22T15:44:25.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:24 smithi027 ceph-mon[34922]: osdmap e29: 4 total, 4 up, 4 in 2024-04-22T15:44:25.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:24 smithi027 ceph-mon[34922]: pgmap v99: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-22T15:44:25.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:25 smithi027 ceph-mon[34922]: osdmap e30: 4 total, 4 up, 4 in 2024-04-22T15:44:25.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:25 smithi027 ceph-mon[34922]: from='client.24161 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:44:25.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:25 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T15:44:25.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:25 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T15:44:25.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:25 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:44:25.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:25 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:44:25.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:25 smithi016 ceph-mon[26620]: osdmap e30: 4 total, 4 up, 4 in 2024-04-22T15:44:25.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:25 smithi016 ceph-mon[26620]: from='client.24161 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:44:25.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:25 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T15:44:25.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:25 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T15:44:25.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:25 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:44:25.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:25 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:44:26.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:26 smithi016 ceph-mon[26620]: pgmap v101: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-04-22T15:44:27.055 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:26 smithi027 ceph-mon[34922]: pgmap v101: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-04-22T15:44:27.856 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:27 smithi027 ceph-mon[34922]: from='client.? 172.21.15.27:0/79473675' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b1eda0d-1627-4c49-93b1-1a78896f660c"}]: dispatch 2024-04-22T15:44:27.856 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:27 smithi027 ceph-mon[34922]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b1eda0d-1627-4c49-93b1-1a78896f660c"}]: dispatch 2024-04-22T15:44:27.856 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:27 smithi027 ceph-mon[34922]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b1eda0d-1627-4c49-93b1-1a78896f660c"}]': finished 2024-04-22T15:44:27.856 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:27 smithi027 ceph-mon[34922]: osdmap e31: 5 total, 4 up, 5 in 2024-04-22T15:44:27.857 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:27 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:44:27.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:27 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/79473675' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b1eda0d-1627-4c49-93b1-1a78896f660c"}]: dispatch 2024-04-22T15:44:27.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:27 smithi016 ceph-mon[26620]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b1eda0d-1627-4c49-93b1-1a78896f660c"}]: dispatch 2024-04-22T15:44:27.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:27 smithi016 ceph-mon[26620]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b1eda0d-1627-4c49-93b1-1a78896f660c"}]': finished 2024-04-22T15:44:27.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:27 smithi016 ceph-mon[26620]: osdmap e31: 5 total, 4 up, 5 in 2024-04-22T15:44:27.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:27 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:44:28.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:28 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/4129175551' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T15:44:28.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:28 smithi016 ceph-mon[26620]: pgmap v103: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-04-22T15:44:29.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:28 smithi027 ceph-mon[34922]: from='client.? 172.21.15.27:0/4129175551' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T15:44:29.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:28 smithi027 ceph-mon[34922]: pgmap v103: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-04-22T15:44:31.730 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:31 smithi027 ceph-mon[34922]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T15:44:31.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:31 smithi016 ceph-mon[26620]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T15:44:33.817 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:33 smithi027 ceph-mon[34922]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T15:44:33.865 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:33 smithi016 ceph-mon[26620]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T15:44:35.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:35 smithi027 ceph-mon[34922]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T15:44:35.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:35 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T15:44:35.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:35 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:44:35.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:35 smithi027 ceph-mon[34922]: Deploying daemon osd.4 on smithi027 2024-04-22T15:44:35.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:35 smithi016 ceph-mon[26620]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T15:44:35.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:35 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T15:44:35.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:35 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:44:35.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:35 smithi016 ceph-mon[26620]: Deploying daemon osd.4 on smithi027 2024-04-22T15:44:36.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:36 smithi027 ceph-mon[34922]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T15:44:36.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:36 smithi016 ceph-mon[26620]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T15:44:39.548 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:39 smithi027 ceph-mon[34922]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T15:44:39.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:39 smithi016 ceph-mon[26620]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T15:44:40.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:40.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:40.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:44:40.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:40 smithi016 ceph-mon[26620]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T15:44:40.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:44:40.922 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:40 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:40.922 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:40 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:40.922 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:40 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:44:40.922 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:40 smithi027 ceph-mon[34922]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T15:44:40.922 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:40 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:44:42.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:42.667+0000 7f5057fff700 1 -- 172.21.15.27:0/560311711 <== mgr.14217 v2:172.21.15.16:6800/1057234399 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f50600611d0 con 0x7f504c06c4e0 2024-04-22T15:44:42.671 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 4 on host 'smithi027' 2024-04-22T15:44:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:42.670+0000 7f50650a6700 1 -- 172.21.15.27:0/560311711 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f504c06c4e0 msgr2=0x7f504c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:42.670+0000 7f50650a6700 1 --2- 172.21.15.27:0/560311711 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f504c06c4e0 0x7f504c06e9a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f505000b830 tx=0x7f5050006210 comp rx=0 tx=0).stop 2024-04-22T15:44:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:42.670+0000 7f50650a6700 1 -- 172.21.15.27:0/560311711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5060104170 msgr2=0x7f50601969e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:42.670+0000 7f50650a6700 1 --2- 172.21.15.27:0/560311711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5060104170 0x7f50601969e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f504800d900 tx=0x7f504800dcc0 comp rx=0 tx=0).stop 2024-04-22T15:44:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:42.670+0000 7f50650a6700 1 -- 172.21.15.27:0/560311711 shutdown_connections 2024-04-22T15:44:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:42.670+0000 7f50650a6700 1 --2- 172.21.15.27:0/560311711 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f504c06c4e0 0x7f504c06e9a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:42.670+0000 7f50650a6700 1 --2- 172.21.15.27:0/560311711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5060101840 0x7f50601964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:42.670+0000 7f50650a6700 1 --2- 172.21.15.27:0/560311711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5060104170 0x7f50601969e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:42.670+0000 7f50650a6700 1 -- 172.21.15.27:0/560311711 >> 172.21.15.27:0/560311711 conn(0x7f50600fb190 msgr2=0x7f50600fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:42.670+0000 7f50650a6700 1 -- 172.21.15.27:0/560311711 shutdown_connections 2024-04-22T15:44:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:42.670+0000 7f50650a6700 1 -- 172.21.15.27:0/560311711 wait complete. 2024-04-22T15:44:43.260 DEBUG:teuthology.orchestra.run.smithi027:osd.4> sudo journalctl -f -n 0 -u ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.4.service 2024-04-22T15:44:43.262 INFO:tasks.cephadm:Deploying osd.5 on smithi027 with /dev/vg_nvme/lv_2... 2024-04-22T15:44:43.263 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T15:44:43.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:43 smithi027 ceph-mon[34922]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T15:44:43.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:43 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:43.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:43 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:43.587 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:44:43 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[47751]: 2024-04-22T15:44:43.471+0000 7f3225812640 -1 osd.4 0 log_to_monitors true 2024-04-22T15:44:43.877 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi027/config 2024-04-22T15:44:43.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:43 smithi016 ceph-mon[26620]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T15:44:43.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:43 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:43.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:43 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:44.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:44 smithi027 ceph-mon[34922]: from='osd.4 [v2:172.21.15.27:6808/691669816,v1:172.21.15.27:6809/691669816]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T15:44:44.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:44 smithi027 ceph-mon[34922]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T15:44:44.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:44 smithi016 ceph-mon[26620]: from='osd.4 [v2:172.21.15.27:6808/691669816,v1:172.21.15.27:6809/691669816]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T15:44:44.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:44 smithi016 ceph-mon[26620]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T15:44:45.818 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:45 smithi016 ceph-mon[26620]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T15:44:45.818 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:45 smithi016 ceph-mon[26620]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T15:44:45.819 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:45 smithi016 ceph-mon[26620]: osdmap e32: 5 total, 4 up, 5 in 2024-04-22T15:44:45.819 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:45 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:44:45.819 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:45 smithi016 ceph-mon[26620]: from='osd.4 [v2:172.21.15.27:6808/691669816,v1:172.21.15.27:6809/691669816]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-22T15:44:45.819 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:45 smithi016 ceph-mon[26620]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-22T15:44:45.819 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:45 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:45.819 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:45 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:45.836 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:44:45 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[47751]: 2024-04-22T15:44:45.477+0000 7f321a675700 -1 osd.4 0 waiting for initial osdmap 2024-04-22T15:44:45.836 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:44:45 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[47751]: 2024-04-22T15:44:45.484+0000 7f3214c67700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T15:44:45.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:45 smithi027 ceph-mon[34922]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T15:44:45.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:45 smithi027 ceph-mon[34922]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T15:44:45.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:45 smithi027 ceph-mon[34922]: osdmap e32: 5 total, 4 up, 5 in 2024-04-22T15:44:45.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:45 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:44:45.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:45 smithi027 ceph-mon[34922]: from='osd.4 [v2:172.21.15.27:6808/691669816,v1:172.21.15.27:6809/691669816]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-22T15:44:45.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:45 smithi027 ceph-mon[34922]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-22T15:44:45.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:45 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:45.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:45 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:46.313 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T15:44:46.313 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T15:44:46.314 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-04-22T15:44:46.314 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-04-22T15:44:46.314 INFO:teuthology.orchestra.run.smithi027.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0120422 s, 871 MB/s 2024-04-22T15:44:46.314 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-04-22T15:44:46.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:46 smithi027 ceph-mon[34922]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-22T15:44:46.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:46 smithi027 ceph-mon[34922]: osdmap e33: 5 total, 4 up, 5 in 2024-04-22T15:44:46.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:46 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:44:46.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:46 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:44:46.829 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_2 2024-04-22T15:44:46.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:46 smithi016 ceph-mon[26620]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-22T15:44:46.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:46 smithi016 ceph-mon[26620]: osdmap e33: 5 total, 4 up, 5 in 2024-04-22T15:44:46.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:46 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:44:46.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:46 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:44:47.280 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi027/config 2024-04-22T15:44:47.740 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:47 smithi027 ceph-mon[34922]: purged_snaps scrub starts 2024-04-22T15:44:47.740 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:47 smithi027 ceph-mon[34922]: purged_snaps scrub ok 2024-04-22T15:44:47.740 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:47 smithi027 ceph-mon[34922]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T15:44:47.740 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:47 smithi027 ceph-mon[34922]: osd.4 [v2:172.21.15.27:6808/691669816,v1:172.21.15.27:6809/691669816] boot 2024-04-22T15:44:47.740 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:47 smithi027 ceph-mon[34922]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T15:44:47.740 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:47 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:44:47.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:47 smithi016 ceph-mon[26620]: purged_snaps scrub starts 2024-04-22T15:44:47.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:47 smithi016 ceph-mon[26620]: purged_snaps scrub ok 2024-04-22T15:44:47.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:47 smithi016 ceph-mon[26620]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T15:44:47.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:47 smithi016 ceph-mon[26620]: osd.4 [v2:172.21.15.27:6808/691669816,v1:172.21.15.27:6809/691669816] boot 2024-04-22T15:44:47.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:47 smithi016 ceph-mon[26620]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T15:44:47.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:47 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:44:48.755 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:48 smithi027 ceph-mon[34922]: osdmap e35: 5 total, 5 up, 5 in 2024-04-22T15:44:48.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:48 smithi016 ceph-mon[26620]: osdmap e35: 5 total, 5 up, 5 in 2024-04-22T15:44:49.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.536+0000 7fd90e0ed700 1 -- 172.21.15.27:0/3478027227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd908100dc0 msgr2=0x7fd908101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:49.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.536+0000 7fd90e0ed700 1 --2- 172.21.15.27:0/3478027227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd908100dc0 0x7fd908101220 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0009910 tx=0x7fd8f0009c20 comp rx=0 tx=0).stop 2024-04-22T15:44:49.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.536+0000 7fd90e0ed700 1 -- 172.21.15.27:0/3478027227 shutdown_connections 2024-04-22T15:44:49.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.536+0000 7fd90e0ed700 1 --2- 172.21.15.27:0/3478027227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd908100dc0 0x7fd908101220 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:49.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.536+0000 7fd90e0ed700 1 --2- 172.21.15.27:0/3478027227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9080ffbc0 0x7fd9080fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:49.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.536+0000 7fd90e0ed700 1 -- 172.21.15.27:0/3478027227 >> 172.21.15.27:0/3478027227 conn(0x7fd9080fb160 msgr2=0x7fd9080fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:44:49.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.536+0000 7fd90e0ed700 1 -- 172.21.15.27:0/3478027227 shutdown_connections 2024-04-22T15:44:49.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.536+0000 7fd90e0ed700 1 -- 172.21.15.27:0/3478027227 wait complete. 2024-04-22T15:44:49.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.536+0000 7fd90e0ed700 1 Processor -- start 2024-04-22T15:44:49.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.536+0000 7fd90e0ed700 1 -- start start 2024-04-22T15:44:49.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.537+0000 7fd90e0ed700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9080ffbc0 0x7fd9080699d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:49.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.537+0000 7fd90e0ed700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd908100dc0 0x7fd90819b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:49.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.537+0000 7fd90e0ed700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90819bac0 con 0x7fd9080ffbc0 2024-04-22T15:44:49.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.537+0000 7fd90e0ed700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90819bc00 con 0x7fd908100dc0 2024-04-22T15:44:49.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.537+0000 7fd8fffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd908100dc0 0x7fd90819b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:49.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.537+0000 7fd8fffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd908100dc0 0x7fd90819b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50230/0 (socket says 172.21.15.27:50230) 2024-04-22T15:44:49.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.537+0000 7fd8fffff700 1 -- 172.21.15.27:0/367640007 learned_addr learned my addr 172.21.15.27:0/367640007 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-22T15:44:49.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.537+0000 7fd9077fe700 1 --2- 172.21.15.27:0/367640007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9080ffbc0 0x7fd9080699d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:49.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.537+0000 7fd8fffff700 1 -- 172.21.15.27:0/367640007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9080ffbc0 msgr2=0x7fd9080699d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:44:49.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.537+0000 7fd8fffff700 1 --2- 172.21.15.27:0/367640007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9080ffbc0 0x7fd9080699d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:44:49.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.537+0000 7fd8fffff700 1 -- 172.21.15.27:0/367640007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f00095f0 con 0x7fd908100dc0 2024-04-22T15:44:49.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.537+0000 7fd8fffff700 1 --2- 172.21.15.27:0/367640007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd908100dc0 0x7fd90819b4a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0006010 tx=0x7fd8f000bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:49.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.537+0000 7fd9057fa700 1 -- 172.21.15.27:0/367640007 <== mon.1 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8f001d070 con 0x7fd908100dc0 2024-04-22T15:44:49.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.537+0000 7fd9057fa700 1 -- 172.21.15.27:0/367640007 <== mon.1 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd8f000f460 con 0x7fd908100dc0 2024-04-22T15:44:49.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.537+0000 7fd9057fa700 1 -- 172.21.15.27:0/367640007 <== mon.1 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8f0021640 con 0x7fd908100dc0 2024-04-22T15:44:49.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.537+0000 7fd90e0ed700 1 -- 172.21.15.27:0/367640007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90819be80 con 0x7fd908100dc0 2024-04-22T15:44:49.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.538+0000 7fd90e0ed700 1 -- 172.21.15.27:0/367640007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90819c340 con 0x7fd908100dc0 2024-04-22T15:44:49.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.539+0000 7fd90e0ed700 1 -- 172.21.15.27:0/367640007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd908068af0 con 0x7fd908100dc0 2024-04-22T15:44:49.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.539+0000 7fd9057fa700 1 -- 172.21.15.27:0/367640007 <== mon.1 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fd8f000fab0 con 0x7fd908100dc0 2024-04-22T15:44:49.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.539+0000 7fd9057fa700 1 --2- 172.21.15.27:0/367640007 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fd8f406c4e0 0x7fd8f406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:44:49.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.539+0000 7fd9057fa700 1 -- 172.21.15.27:0/367640007 <== mon.1 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4206+0+0 (secure 0 0 0) 0x7fd8f008c580 con 0x7fd908100dc0 2024-04-22T15:44:49.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.540+0000 7fd9077fe700 1 --2- 172.21.15.27:0/367640007 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fd8f406c4e0 0x7fd8f406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:44:49.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.540+0000 7fd9077fe700 1 --2- 172.21.15.27:0/367640007 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fd8f406c4e0 0x7fd8f406e9a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd908100c20 tx=0x7fd8f800b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:44:49.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.544+0000 7fd9057fa700 1 -- 172.21.15.27:0/367640007 <== mon.1 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd8f005ace0 con 0x7fd908100dc0 2024-04-22T15:44:49.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:49 smithi027 ceph-mon[34922]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:44:49.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:49 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:49.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:49 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:49.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:49 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:44:49.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:49 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:44:49.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:49 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:49.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:49 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:44:49.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:49 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:44:49.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:49 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:49.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:49 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:44:49.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:49 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:44:49.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:49 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:44:49.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:49 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:49.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:44:49.754+0000 7fd90e0ed700 1 -- 172.21.15.27:0/367640007 --> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd908105830 con 0x7fd8f406c4e0 2024-04-22T15:44:49.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:49 smithi016 ceph-mon[26620]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:44:49.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:49 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:49.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:49 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:49.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:49 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:44:49.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:49 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:44:49.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:49 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:49.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:49 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:44:49.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:49 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:44:49.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:49 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:49.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:49 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:44:49.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:49 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:44:49.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:49 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:44:49.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:49 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:44:50.765 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:50 smithi027 ceph-mon[34922]: Detected new or changed devices on smithi027 2024-04-22T15:44:50.765 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:50 smithi027 ceph-mon[34922]: Adjusting osd_memory_target on smithi027 to 7456M 2024-04-22T15:44:50.765 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:50 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T15:44:50.765 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:50 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T15:44:50.765 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:50 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:44:50.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:50 smithi016 ceph-mon[26620]: Detected new or changed devices on smithi027 2024-04-22T15:44:50.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:50 smithi016 ceph-mon[26620]: Adjusting osd_memory_target on smithi027 to 7456M 2024-04-22T15:44:50.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:50 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T15:44:50.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:50 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T15:44:50.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:50 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:44:51.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:51 smithi027 ceph-mon[34922]: from='client.24179 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:44:51.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:51 smithi027 ceph-mon[34922]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:44:51.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:51 smithi016 ceph-mon[26620]: from='client.24179 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:44:51.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:51 smithi016 ceph-mon[26620]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:44:52.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:52 smithi027 ceph-mon[34922]: from='client.? 172.21.15.27:0/4053403548' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58cebe34-0821-458f-86b5-7d460829d994"}]: dispatch 2024-04-22T15:44:52.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:52 smithi027 ceph-mon[34922]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58cebe34-0821-458f-86b5-7d460829d994"}]: dispatch 2024-04-22T15:44:52.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:52 smithi027 ceph-mon[34922]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "58cebe34-0821-458f-86b5-7d460829d994"}]': finished 2024-04-22T15:44:52.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:52 smithi027 ceph-mon[34922]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T15:44:52.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:52 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:44:52.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:52 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/4053403548' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58cebe34-0821-458f-86b5-7d460829d994"}]: dispatch 2024-04-22T15:44:52.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:52 smithi016 ceph-mon[26620]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58cebe34-0821-458f-86b5-7d460829d994"}]: dispatch 2024-04-22T15:44:52.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:52 smithi016 ceph-mon[26620]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "58cebe34-0821-458f-86b5-7d460829d994"}]': finished 2024-04-22T15:44:52.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:52 smithi016 ceph-mon[26620]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T15:44:52.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:52 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:44:53.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:53 smithi027 ceph-mon[34922]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:44:53.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:53 smithi027 ceph-mon[34922]: from='client.? 172.21.15.27:0/4033844600' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T15:44:53.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:53 smithi016 ceph-mon[26620]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:44:53.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:53 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/4033844600' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T15:44:55.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:55 smithi027 ceph-mon[34922]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:44:55.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:55 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:44:55.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:55 smithi016 ceph-mon[26620]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:44:55.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:55 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:44:56.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:56 smithi016 ceph-mon[26620]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:44:57.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:56 smithi027 ceph-mon[34922]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:44:59.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:59 smithi027 ceph-mon[34922]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:44:59.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:59 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T15:44:59.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:44:59 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:44:59.921 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:59 smithi016 ceph-mon[26620]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:44:59.922 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:59 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T15:44:59.922 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:44:59 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:45:00.818 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:00 smithi027 ceph-mon[34922]: Deploying daemon osd.5 on smithi027 2024-04-22T15:45:00.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:00 smithi016 ceph-mon[26620]: Deploying daemon osd.5 on smithi027 2024-04-22T15:45:01.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:01 smithi027 ceph-mon[34922]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:45:01.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:01 smithi016 ceph-mon[26620]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:45:03.828 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:03 smithi027 ceph-mon[34922]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:45:03.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:03 smithi016 ceph-mon[26620]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:45:05.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:05 smithi027 ceph-mon[34922]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:45:05.889 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:05 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:45:05.889 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:05 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:45:05.889 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:05 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:45:05.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:05 smithi016 ceph-mon[26620]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:45:05.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:05 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:45:05.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:05 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:45:05.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:05 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:45:06.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:06 smithi027 ceph-mon[34922]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:45:06.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:06 smithi016 ceph-mon[26620]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:45:08.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:08.526+0000 7fd9057fa700 1 -- 172.21.15.27:0/367640007 <== mgr.14217 v2:172.21.15.16:6800/1057234399 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd908105830 con 0x7fd8f406c4e0 2024-04-22T15:45:08.529 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 5 on host 'smithi027' 2024-04-22T15:45:08.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:08.528+0000 7fd90e0ed700 1 -- 172.21.15.27:0/367640007 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fd8f406c4e0 msgr2=0x7fd8f406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:08.528+0000 7fd90e0ed700 1 --2- 172.21.15.27:0/367640007 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fd8f406c4e0 0x7fd8f406e9a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd908100c20 tx=0x7fd8f800b410 comp rx=0 tx=0).stop 2024-04-22T15:45:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:08.528+0000 7fd90e0ed700 1 -- 172.21.15.27:0/367640007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd908100dc0 msgr2=0x7fd90819b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:08.528+0000 7fd90e0ed700 1 --2- 172.21.15.27:0/367640007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd908100dc0 0x7fd90819b4a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0006010 tx=0x7fd8f000bd70 comp rx=0 tx=0).stop 2024-04-22T15:45:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:08.528+0000 7fd90e0ed700 1 -- 172.21.15.27:0/367640007 shutdown_connections 2024-04-22T15:45:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:08.528+0000 7fd90e0ed700 1 --2- 172.21.15.27:0/367640007 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fd8f406c4e0 0x7fd8f406e9a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:08.528+0000 7fd90e0ed700 1 --2- 172.21.15.27:0/367640007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9080ffbc0 0x7fd9080699d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:08.528+0000 7fd90e0ed700 1 --2- 172.21.15.27:0/367640007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd908100dc0 0x7fd90819b4a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:08.528+0000 7fd90e0ed700 1 -- 172.21.15.27:0/367640007 >> 172.21.15.27:0/367640007 conn(0x7fd9080fb160 msgr2=0x7fd9081040d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:08.528+0000 7fd90e0ed700 1 -- 172.21.15.27:0/367640007 shutdown_connections 2024-04-22T15:45:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:08.528+0000 7fd90e0ed700 1 -- 172.21.15.27:0/367640007 wait complete. 2024-04-22T15:45:09.356 DEBUG:teuthology.orchestra.run.smithi027:osd.5> sudo journalctl -f -n 0 -u ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.5.service 2024-04-22T15:45:09.359 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-04-22T15:45:09.359 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-22T15:45:09.592 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:45:09.740 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:09 smithi016 ceph-mon[26620]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:45:09.740 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:09 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:45:09.740 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:09 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:45:09.740 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:09 smithi016 ceph-mon[26620]: from='osd.5 [v2:172.21.15.27:6816/1237795056,v1:172.21.15.27:6817/1237795056]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T15:45:09.740 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:09 smithi016 ceph-mon[26620]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T15:45:09.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:09 smithi027 ceph-mon[34922]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:45:09.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:09 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:45:09.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:09 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:45:09.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:09 smithi027 ceph-mon[34922]: from='osd.5 [v2:172.21.15.27:6816/1237795056,v1:172.21.15.27:6817/1237795056]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T15:45:09.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:09 smithi027 ceph-mon[34922]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T15:45:10.730 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:10 smithi016 ceph-mon[26620]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T15:45:10.730 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:10 smithi016 ceph-mon[26620]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T15:45:10.730 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:10 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:45:10.730 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:10 smithi016 ceph-mon[26620]: from='osd.5 [v2:172.21.15.27:6816/1237795056,v1:172.21.15.27:6817/1237795056]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-22T15:45:10.730 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:10 smithi016 ceph-mon[26620]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-22T15:45:10.730 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:10 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:45:10.731 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:10 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:45:10.731 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:10 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:45:10.764 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:10 smithi027 ceph-mon[34922]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T15:45:10.764 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:10 smithi027 ceph-mon[34922]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T15:45:10.764 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:10 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:45:10.764 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:10 smithi027 ceph-mon[34922]: from='osd.5 [v2:172.21.15.27:6816/1237795056,v1:172.21.15.27:6817/1237795056]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-22T15:45:10.764 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:10 smithi027 ceph-mon[34922]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-22T15:45:10.765 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:10 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:45:10.765 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:10 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:45:10.765 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:10 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:45:10.765 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:45:10 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[54558]: 2024-04-22T15:45:10.503+0000 7f17c8f43700 -1 osd.5 0 waiting for initial osdmap 2024-04-22T15:45:10.765 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:45:10 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[54558]: 2024-04-22T15:45:10.515+0000 7f17c3d36700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T15:45:11.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.213+0000 7f9f2d81e700 1 -- 172.21.15.16:0/2050048391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f28101840 msgr2=0x7f9f28103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:11.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.213+0000 7f9f2d81e700 1 --2- 172.21.15.16:0/2050048391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f28101840 0x7f9f28103c30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9f1800b0a0 tx=0x7f9f1800b3b0 comp rx=0 tx=0).stop 2024-04-22T15:45:11.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.213+0000 7f9f2d81e700 1 -- 172.21.15.16:0/2050048391 shutdown_connections 2024-04-22T15:45:11.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.213+0000 7f9f2d81e700 1 --2- 172.21.15.16:0/2050048391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f28104170 0x7f9f28106560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:11.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.213+0000 7f9f2d81e700 1 --2- 172.21.15.16:0/2050048391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f28101840 0x7f9f28103c30 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:11.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.213+0000 7f9f2d81e700 1 -- 172.21.15.16:0/2050048391 >> 172.21.15.16:0/2050048391 conn(0x7f9f280fb190 msgr2=0x7f9f280fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:11.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.213+0000 7f9f2d81e700 1 -- 172.21.15.16:0/2050048391 shutdown_connections 2024-04-22T15:45:11.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.213+0000 7f9f2d81e700 1 -- 172.21.15.16:0/2050048391 wait complete. 2024-04-22T15:45:11.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.213+0000 7f9f2d81e700 1 Processor -- start 2024-04-22T15:45:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.214+0000 7f9f2d81e700 1 -- start start 2024-04-22T15:45:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.214+0000 7f9f2d81e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f28101840 0x7f9f28196570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.214+0000 7f9f2d81e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f28104170 0x7f9f28196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.214+0000 7f9f2d81e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f281970d0 con 0x7f9f28101840 2024-04-22T15:45:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.214+0000 7f9f2d81e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f28197210 con 0x7f9f28104170 2024-04-22T15:45:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.214+0000 7f9f26ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f28101840 0x7f9f28196570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.214+0000 7f9f26ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f28101840 0x7f9f28196570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57292/0 (socket says 172.21.15.16:57292) 2024-04-22T15:45:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.214+0000 7f9f26ffd700 1 -- 172.21.15.16:0/1043148376 learned_addr learned my addr 172.21.15.16:0/1043148376 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:45:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.214+0000 7f9f267fc700 1 --2- 172.21.15.16:0/1043148376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f28104170 0x7f9f28196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.214+0000 7f9f26ffd700 1 -- 172.21.15.16:0/1043148376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f28104170 msgr2=0x7f9f28196ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.214+0000 7f9f26ffd700 1 --2- 172.21.15.16:0/1043148376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f28104170 0x7f9f28196ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.214+0000 7f9f26ffd700 1 -- 172.21.15.16:0/1043148376 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f18009d00 con 0x7f9f28101840 2024-04-22T15:45:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.214+0000 7f9f26ffd700 1 --2- 172.21.15.16:0/1043148376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f28101840 0x7f9f28196570 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9f180077b0 tx=0x7f9f18003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.214+0000 7f9f2c81c700 1 -- 172.21.15.16:0/1043148376 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f1800e040 con 0x7f9f28101840 2024-04-22T15:45:11.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.215+0000 7f9f2c81c700 1 -- 172.21.15.16:0/1043148376 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9f1800ce60 con 0x7f9f28101840 2024-04-22T15:45:11.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.215+0000 7f9f2d81e700 1 -- 172.21.15.16:0/1043148376 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f2819bc60 con 0x7f9f28101840 2024-04-22T15:45:11.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.215+0000 7f9f2c81c700 1 -- 172.21.15.16:0/1043148376 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f1801daf0 con 0x7f9f28101840 2024-04-22T15:45:11.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.215+0000 7f9f2d81e700 1 -- 172.21.15.16:0/1043148376 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f2819c150 con 0x7f9f28101840 2024-04-22T15:45:11.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.215+0000 7f9f2d81e700 1 -- 172.21.15.16:0/1043148376 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f28109890 con 0x7f9f28101840 2024-04-22T15:45:11.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.216+0000 7f9f2c81c700 1 -- 172.21.15.16:0/1043148376 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f9f18004250 con 0x7f9f28101840 2024-04-22T15:45:11.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.216+0000 7f9f2c81c700 1 --2- 172.21.15.16:0/1043148376 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f9f1406c4e0 0x7f9f1406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:11.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.216+0000 7f9f2c81c700 1 -- 172.21.15.16:0/1043148376 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4358+0+0 (secure 0 0 0) 0x7f9f1808ce90 con 0x7f9f28101840 2024-04-22T15:45:11.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.217+0000 7f9f267fc700 1 --2- 172.21.15.16:0/1043148376 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f9f1406c4e0 0x7f9f1406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:11.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.217+0000 7f9f267fc700 1 --2- 172.21.15.16:0/1043148376 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f9f1406c4e0 0x7f9f1406e9a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9f10006fd0 tx=0x7f9f10008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:11.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.221+0000 7f9f2c81c700 1 -- 172.21.15.16:0/1043148376 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9f1805b550 con 0x7f9f28101840 2024-04-22T15:45:11.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.421+0000 7f9f2d81e700 1 -- 172.21.15.16:0/1043148376 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f9f2819c7c0 con 0x7f9f28101840 2024-04-22T15:45:11.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.421+0000 7f9f2c81c700 1 -- 172.21.15.16:0/1043148376 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v38) v1 ==== 74+0+130 (secure 0 0 0) 0x7f9f1805b0e0 con 0x7f9f28101840 2024-04-22T15:45:11.425 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:45:11.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.423+0000 7f9f2d81e700 1 -- 172.21.15.16:0/1043148376 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f9f1406c4e0 msgr2=0x7f9f1406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:11.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.423+0000 7f9f2d81e700 1 --2- 172.21.15.16:0/1043148376 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f9f1406c4e0 0x7f9f1406e9a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9f10006fd0 tx=0x7f9f10008040 comp rx=0 tx=0).stop 2024-04-22T15:45:11.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.423+0000 7f9f2d81e700 1 -- 172.21.15.16:0/1043148376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f28101840 msgr2=0x7f9f28196570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:11.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.423+0000 7f9f2d81e700 1 --2- 172.21.15.16:0/1043148376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f28101840 0x7f9f28196570 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9f180077b0 tx=0x7f9f18003ce0 comp rx=0 tx=0).stop 2024-04-22T15:45:11.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.424+0000 7f9f2d81e700 1 -- 172.21.15.16:0/1043148376 shutdown_connections 2024-04-22T15:45:11.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.424+0000 7f9f2d81e700 1 --2- 172.21.15.16:0/1043148376 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f9f1406c4e0 0x7f9f1406e9a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:11.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.424+0000 7f9f2d81e700 1 --2- 172.21.15.16:0/1043148376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f28101840 0x7f9f28196570 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:11.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.424+0000 7f9f2d81e700 1 --2- 172.21.15.16:0/1043148376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f28104170 0x7f9f28196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:11.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.424+0000 7f9f2d81e700 1 -- 172.21.15.16:0/1043148376 >> 172.21.15.16:0/1043148376 conn(0x7f9f280fb190 msgr2=0x7f9f280fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:11.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.424+0000 7f9f2d81e700 1 -- 172.21.15.16:0/1043148376 shutdown_connections 2024-04-22T15:45:11.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:11.424+0000 7f9f2d81e700 1 -- 172.21.15.16:0/1043148376 wait complete. 2024-04-22T15:45:11.694 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:11 smithi016 ceph-mon[26620]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:45:11.694 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:11 smithi016 ceph-mon[26620]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-22T15:45:11.694 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:11 smithi016 ceph-mon[26620]: osdmap e38: 6 total, 5 up, 6 in 2024-04-22T15:45:11.694 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:11 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:45:11.694 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:11 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:45:11.694 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:11 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/1043148376' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T15:45:11.730 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:11 smithi027 ceph-mon[34922]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:45:11.731 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:11 smithi027 ceph-mon[34922]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-22T15:45:11.731 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:11 smithi027 ceph-mon[34922]: osdmap e38: 6 total, 5 up, 6 in 2024-04-22T15:45:11.731 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:11 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:45:11.731 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:11 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:45:11.731 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:11 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/1043148376' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T15:45:12.139 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":38,"num_osds":6,"num_up_osds":5,"osd_up_since":1713800686,"num_in_osds":6,"osd_in_since":1713800692,"num_remapped_pgs":0} 2024-04-22T15:45:12.552 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:12 smithi027 ceph-mon[34922]: purged_snaps scrub starts 2024-04-22T15:45:12.552 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:12 smithi027 ceph-mon[34922]: purged_snaps scrub ok 2024-04-22T15:45:12.552 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:12 smithi027 ceph-mon[34922]: osd.5 [v2:172.21.15.27:6816/1237795056,v1:172.21.15.27:6817/1237795056] boot 2024-04-22T15:45:12.552 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:12 smithi027 ceph-mon[34922]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T15:45:12.552 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:12 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:45:12.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:12 smithi016 ceph-mon[26620]: purged_snaps scrub starts 2024-04-22T15:45:12.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:12 smithi016 ceph-mon[26620]: purged_snaps scrub ok 2024-04-22T15:45:12.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:12 smithi016 ceph-mon[26620]: osd.5 [v2:172.21.15.27:6816/1237795056,v1:172.21.15.27:6817/1237795056] boot 2024-04-22T15:45:12.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:12 smithi016 ceph-mon[26620]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T15:45:12.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:12 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:45:13.140 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-22T15:45:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:45:13.787 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:13 smithi016 ceph-mon[26620]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:45:13.788 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:13 smithi016 ceph-mon[26620]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T15:45:13.788 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:13 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:45:13.788 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:13 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:45:13.788 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:13 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:45:13.788 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:13 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:45:13.788 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:13 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:45:13.788 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:13 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:45:13.788 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:13 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:45:13.788 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:13 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:45:13.788 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:13 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:45:13.788 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:13 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:45:13.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:13 smithi027 ceph-mon[34922]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T15:45:13.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:13 smithi027 ceph-mon[34922]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T15:45:13.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:13 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:45:13.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:13 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:45:13.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:13 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:45:13.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:13 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:45:13.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:13 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:45:13.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:13 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:45:13.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:13 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:45:13.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:13 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:45:13.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:13 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:45:13.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:13 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:45:14.818 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:14 smithi016 ceph-mon[26620]: Detected new or changed devices on smithi027 2024-04-22T15:45:14.818 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:14 smithi016 ceph-mon[26620]: Adjusting osd_memory_target on smithi027 to 4970M 2024-04-22T15:45:14.818 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:14 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:45:14.819 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:14 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:45:14.819 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:14 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:45:14.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:14 smithi027 ceph-mon[34922]: Detected new or changed devices on smithi027 2024-04-22T15:45:14.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:14 smithi027 ceph-mon[34922]: Adjusting osd_memory_target on smithi027 to 4970M 2024-04-22T15:45:14.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:14 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:45:14.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:14 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:45:14.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:14 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:45:15.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.011+0000 7f3994610700 1 -- 172.21.15.16:0/450972758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f398c104080 msgr2=0x7f398c106470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:15.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.011+0000 7f3994610700 1 --2- 172.21.15.16:0/450972758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f398c104080 0x7f398c106470 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f397c009910 tx=0x7f397c009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:15.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.011+0000 7f3994610700 1 -- 172.21.15.16:0/450972758 shutdown_connections 2024-04-22T15:45:15.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.011+0000 7f3994610700 1 --2- 172.21.15.16:0/450972758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f398c104080 0x7f398c106470 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:15.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.011+0000 7f3994610700 1 --2- 172.21.15.16:0/450972758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f398c101750 0x7f398c103b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:15.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.011+0000 7f3994610700 1 -- 172.21.15.16:0/450972758 >> 172.21.15.16:0/450972758 conn(0x7f398c0fb160 msgr2=0x7f398c0fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:15.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.011+0000 7f3994610700 1 -- 172.21.15.16:0/450972758 shutdown_connections 2024-04-22T15:45:15.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.011+0000 7f3994610700 1 -- 172.21.15.16:0/450972758 wait complete. 2024-04-22T15:45:15.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.011+0000 7f3994610700 1 Processor -- start 2024-04-22T15:45:15.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.012+0000 7f3994610700 1 -- start start 2024-04-22T15:45:15.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.012+0000 7f3994610700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f398c101750 0x7f398c1967a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:15.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.012+0000 7f3994610700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f398c104080 0x7f398c196ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:15.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.012+0000 7f3994610700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398c197300 con 0x7f398c104080 2024-04-22T15:45:15.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.012+0000 7f3994610700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398c197440 con 0x7f398c101750 2024-04-22T15:45:15.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.012+0000 7f3991bab700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f398c104080 0x7f398c196ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:15.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.012+0000 7f3991bab700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f398c104080 0x7f398c196ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52964/0 (socket says 172.21.15.16:52964) 2024-04-22T15:45:15.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.012+0000 7f3991bab700 1 -- 172.21.15.16:0/760568762 learned_addr learned my addr 172.21.15.16:0/760568762 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:45:15.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.012+0000 7f39923ac700 1 --2- 172.21.15.16:0/760568762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f398c101750 0x7f398c1967a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:15.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.012+0000 7f3991bab700 1 -- 172.21.15.16:0/760568762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f398c101750 msgr2=0x7f398c1967a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:15.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.012+0000 7f3991bab700 1 --2- 172.21.15.16:0/760568762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f398c101750 0x7f398c1967a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:15.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.012+0000 7f3991bab700 1 -- 172.21.15.16:0/760568762 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f397c0095f0 con 0x7f398c104080 2024-04-22T15:45:15.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.012+0000 7f3991bab700 1 --2- 172.21.15.16:0/760568762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f398c104080 0x7f398c196ce0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f397c006010 tx=0x7f397c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:15.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.012+0000 7f39837fe700 1 -- 172.21.15.16:0/760568762 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f397c01d070 con 0x7f398c104080 2024-04-22T15:45:15.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.012+0000 7f39837fe700 1 -- 172.21.15.16:0/760568762 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f397c00f460 con 0x7f398c104080 2024-04-22T15:45:15.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.013+0000 7f3994610700 1 -- 172.21.15.16:0/760568762 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f398c19be90 con 0x7f398c104080 2024-04-22T15:45:15.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.013+0000 7f39837fe700 1 -- 172.21.15.16:0/760568762 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f397c021640 con 0x7f398c104080 2024-04-22T15:45:15.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.013+0000 7f3994610700 1 -- 172.21.15.16:0/760568762 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f398c19c350 con 0x7f398c104080 2024-04-22T15:45:15.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.014+0000 7f3994610700 1 -- 172.21.15.16:0/760568762 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f398c190630 con 0x7f398c104080 2024-04-22T15:45:15.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.014+0000 7f39837fe700 1 -- 172.21.15.16:0/760568762 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f397c00fab0 con 0x7f398c104080 2024-04-22T15:45:15.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.014+0000 7f39837fe700 1 --2- 172.21.15.16:0/760568762 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f397806c4e0 0x7f397806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:15.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.014+0000 7f39837fe700 1 -- 172.21.15.16:0/760568762 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f397c08c6c0 con 0x7f398c104080 2024-04-22T15:45:15.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.014+0000 7f39923ac700 1 --2- 172.21.15.16:0/760568762 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f397806c4e0 0x7f397806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:15.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.015+0000 7f39923ac700 1 --2- 172.21.15.16:0/760568762 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f397806c4e0 0x7f397806e9a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3988005950 tx=0x7f398800b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:15.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.019+0000 7f39837fe700 1 -- 172.21.15.16:0/760568762 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f397c05ac70 con 0x7f398c104080 2024-04-22T15:45:15.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.225+0000 7f3994610700 1 -- 172.21.15.16:0/760568762 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f398c068af0 con 0x7f398c104080 2024-04-22T15:45:15.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.226+0000 7f39837fe700 1 -- 172.21.15.16:0/760568762 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7f397c05a800 con 0x7f398c104080 2024-04-22T15:45:15.229 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:45:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.228+0000 7f3994610700 1 -- 172.21.15.16:0/760568762 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f397806c4e0 msgr2=0x7f397806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.228+0000 7f3994610700 1 --2- 172.21.15.16:0/760568762 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f397806c4e0 0x7f397806e9a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3988005950 tx=0x7f398800b410 comp rx=0 tx=0).stop 2024-04-22T15:45:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.228+0000 7f3994610700 1 -- 172.21.15.16:0/760568762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f398c104080 msgr2=0x7f398c196ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.228+0000 7f3994610700 1 --2- 172.21.15.16:0/760568762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f398c104080 0x7f398c196ce0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f397c006010 tx=0x7f397c00bd70 comp rx=0 tx=0).stop 2024-04-22T15:45:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.228+0000 7f3994610700 1 -- 172.21.15.16:0/760568762 shutdown_connections 2024-04-22T15:45:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.228+0000 7f3994610700 1 --2- 172.21.15.16:0/760568762 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f397806c4e0 0x7f397806e9a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.228+0000 7f3994610700 1 --2- 172.21.15.16:0/760568762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f398c101750 0x7f398c1967a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.228+0000 7f3994610700 1 --2- 172.21.15.16:0/760568762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f398c104080 0x7f398c196ce0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.228+0000 7f3994610700 1 -- 172.21.15.16:0/760568762 >> 172.21.15.16:0/760568762 conn(0x7f398c0fb160 msgr2=0x7f398c0fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.228+0000 7f3994610700 1 -- 172.21.15.16:0/760568762 shutdown_connections 2024-04-22T15:45:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:15.228+0000 7f3994610700 1 -- 172.21.15.16:0/760568762 wait complete. 2024-04-22T15:45:15.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:15 smithi027 ceph-mon[34922]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:15.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:15 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/760568762' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T15:45:15.891 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:15 smithi016 ceph-mon[26620]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:15.891 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:15 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/760568762' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T15:45:15.914 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1713800711,"num_in_osds":6,"osd_in_since":1713800692,"num_remapped_pgs":0} 2024-04-22T15:45:15.914 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T15:45:16.125 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:45:16.881 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:16 smithi016 ceph-mon[26620]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:17.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:16 smithi027 ceph-mon[34922]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:17.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.731+0000 7fdc45dd3700 1 -- 172.21.15.16:0/4206036569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc40101e60 msgr2=0x7fdc40102240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:17.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.731+0000 7fdc45dd3700 1 --2- 172.21.15.16:0/4206036569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc40101e60 0x7fdc40102240 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdc30009910 tx=0x7fdc30009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:17.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.731+0000 7fdc45dd3700 1 -- 172.21.15.16:0/4206036569 shutdown_connections 2024-04-22T15:45:17.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.731+0000 7fdc45dd3700 1 --2- 172.21.15.16:0/4206036569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40102780 0x7fdc4010ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.731+0000 7fdc45dd3700 1 --2- 172.21.15.16:0/4206036569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc40101e60 0x7fdc40102240 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.731+0000 7fdc45dd3700 1 -- 172.21.15.16:0/4206036569 >> 172.21.15.16:0/4206036569 conn(0x7fdc400fb650 msgr2=0x7fdc400fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.731+0000 7fdc45dd3700 1 -- 172.21.15.16:0/4206036569 shutdown_connections 2024-04-22T15:45:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.731+0000 7fdc45dd3700 1 -- 172.21.15.16:0/4206036569 wait complete. 2024-04-22T15:45:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.731+0000 7fdc45dd3700 1 Processor -- start 2024-04-22T15:45:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.732+0000 7fdc45dd3700 1 -- start start 2024-04-22T15:45:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.732+0000 7fdc45dd3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40101e60 0x7fdc400ffac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.732+0000 7fdc45dd3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc40102780 0x7fdc40100000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.732+0000 7fdc45dd3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc40100540 con 0x7fdc40102780 2024-04-22T15:45:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.732+0000 7fdc45dd3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc40100680 con 0x7fdc40101e60 2024-04-22T15:45:17.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.732+0000 7fdc3effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc40102780 0x7fdc40100000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:17.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.732+0000 7fdc3effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc40102780 0x7fdc40100000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52970/0 (socket says 172.21.15.16:52970) 2024-04-22T15:45:17.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.732+0000 7fdc3effd700 1 -- 172.21.15.16:0/1323969074 learned_addr learned my addr 172.21.15.16:0/1323969074 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:45:17.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.732+0000 7fdc3f7fe700 1 --2- 172.21.15.16:0/1323969074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40101e60 0x7fdc400ffac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:17.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.732+0000 7fdc3effd700 1 -- 172.21.15.16:0/1323969074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40101e60 msgr2=0x7fdc400ffac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:17.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.732+0000 7fdc3effd700 1 --2- 172.21.15.16:0/1323969074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40101e60 0x7fdc400ffac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:17.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.732+0000 7fdc3effd700 1 -- 172.21.15.16:0/1323969074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc300095f0 con 0x7fdc40102780 2024-04-22T15:45:17.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.732+0000 7fdc3effd700 1 --2- 172.21.15.16:0/1323969074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc40102780 0x7fdc40100000 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdc3400c8f0 tx=0x7fdc3400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:17.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.733+0000 7fdc3cff9700 1 -- 172.21.15.16:0/1323969074 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc340078a0 con 0x7fdc40102780 2024-04-22T15:45:17.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.733+0000 7fdc45dd3700 1 -- 172.21.15.16:0/1323969074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc401a2aa0 con 0x7fdc40102780 2024-04-22T15:45:17.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.733+0000 7fdc3cff9700 1 -- 172.21.15.16:0/1323969074 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdc3400f450 con 0x7fdc40102780 2024-04-22T15:45:17.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.733+0000 7fdc3cff9700 1 -- 172.21.15.16:0/1323969074 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc34018610 con 0x7fdc40102780 2024-04-22T15:45:17.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.733+0000 7fdc45dd3700 1 -- 172.21.15.16:0/1323969074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc401a2e50 con 0x7fdc40102780 2024-04-22T15:45:17.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.734+0000 7fdc45dd3700 1 -- 172.21.15.16:0/1323969074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc40108310 con 0x7fdc40102780 2024-04-22T15:45:17.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.734+0000 7fdc3cff9700 1 -- 172.21.15.16:0/1323969074 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fdc3400f5c0 con 0x7fdc40102780 2024-04-22T15:45:17.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.735+0000 7fdc3cff9700 1 --2- 172.21.15.16:0/1323969074 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fdc2806c4e0 0x7fdc2806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:17.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.735+0000 7fdc3cff9700 1 -- 172.21.15.16:0/1323969074 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fdc3408b3d0 con 0x7fdc40102780 2024-04-22T15:45:17.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.735+0000 7fdc3f7fe700 1 --2- 172.21.15.16:0/1323969074 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fdc2806c4e0 0x7fdc2806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:17.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.735+0000 7fdc3f7fe700 1 --2- 172.21.15.16:0/1323969074 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fdc2806c4e0 0x7fdc2806e9a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdc30006010 tx=0x7fdc3001a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:17.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.739+0000 7fdc3cff9700 1 -- 172.21.15.16:0/1323969074 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fdc34055ef0 con 0x7fdc40102780 2024-04-22T15:45:17.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.938+0000 7fdc45dd3700 1 -- 172.21.15.16:0/1323969074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc40100840 con 0x7fdc40102780 2024-04-22T15:45:17.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.938+0000 7fdc3cff9700 1 -- 172.21.15.16:0/1323969074 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11289 (secure 0 0 0) 0x7fdc40100840 con 0x7fdc40102780 2024-04-22T15:45:17.942 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:45:17.942 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":40,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","created":"2024-04-22T15:39:20.727181+0000","modified":"2024-04-22T15:45:12.502671+0000","last_up_change":"2024-04-22T15:45:11.497701+0000","last_in_change":"2024-04-22T15:44:52.054297+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T15:43:58.896563+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"67bd69ee-bd8c-4670-927a-f5fbcd92c9bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1210635569},{"type":"v1","addr":"172.21.15.16:6803","nonce":1210635569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1210635569},{"type":"v1","addr":"172.21.15.16:6805","nonce":1210635569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1210635569},{"type":"v1","addr":"172.21.15.16:6809","nonce":1210635569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1210635569},{"type":"v1","addr":"172.21.15.16:6807","nonce":1210635569}]},"public_addr":"172.21.15.16:6803/1210635569","cluster_addr":"172.21.15.16:6805/1210635569","heartbeat_back_addr":"172.21.15.16:6809/1210635569","heartbeat_front_addr":"172.21.15.16:6807/1210635569","state":["exists","up"]},{"osd":1,"uuid":"f51a9e4d-fb0c-4f62-9328-e23eea863213","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3974144580},{"type":"v1","addr":"172.21.15.16:6811","nonce":3974144580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3974144580},{"type":"v1","addr":"172.21.15.16:6813","nonce":3974144580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3974144580},{"type":"v1","addr":"172.21.15.16:6817","nonce":3974144580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3974144580},{"type":"v1","addr":"172.21.15.16:6815","nonce":3974144580}]},"public_addr":"172.21.15.16:6811/3974144580","cluster_addr":"172.21.15.16:6813/3974144580","heartbeat_back_addr":"172.21.15.16:6817/3974144580","heartbeat_front_addr":"172.21.15.16:6815/3974144580","state":["exists","up"]},{"osd":2,"uuid":"e59fdb03-e04e-4afd-a58e-0b58945d3765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2760833880},{"type":"v1","addr":"172.21.15.16:6819","nonce":2760833880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2760833880},{"type":"v1","addr":"172.21.15.16:6821","nonce":2760833880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2760833880},{"type":"v1","addr":"172.21.15.16:6825","nonce":2760833880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2760833880},{"type":"v1","addr":"172.21.15.16:6823","nonce":2760833880}]},"public_addr":"172.21.15.16:6819/2760833880","cluster_addr":"172.21.15.16:6821/2760833880","heartbeat_back_addr":"172.21.15.16:6825/2760833880","heartbeat_front_addr":"172.21.15.16:6823/2760833880","state":["exists","up"]},{"osd":3,"uuid":"fa25b413-d004-4455-8d7b-0ced7f880a7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":286252834},{"type":"v1","addr":"172.21.15.27:6801","nonce":286252834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":286252834},{"type":"v1","addr":"172.21.15.27:6803","nonce":286252834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":286252834},{"type":"v1","addr":"172.21.15.27:6807","nonce":286252834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":286252834},{"type":"v1","addr":"172.21.15.27:6805","nonce":286252834}]},"public_addr":"172.21.15.27:6801/286252834","cluster_addr":"172.21.15.27:6803/286252834","heartbeat_back_addr":"172.21.15.27:6807/286252834","heartbeat_front_addr":"172.21.15.27:6805/286252834","state":["exists","up"]},{"osd":4,"uuid":"6b1eda0d-1627-4c49-93b1-1a78896f660c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":691669816},{"type":"v1","addr":"172.21.15.27:6809","nonce":691669816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":691669816},{"type":"v1","addr":"172.21.15.27:6811","nonce":691669816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":691669816},{"type":"v1","addr":"172.21.15.27:6815","nonce":691669816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":691669816},{"type":"v1","addr":"172.21.15.27:6813","nonce":691669816}]},"public_addr":"172.21.15.27:6809/691669816","cluster_addr":"172.21.15.27:6811/691669816","heartbeat_back_addr":"172.21.15.27:6815/691669816","heartbeat_front_addr":"172.21.15.27:6813/691669816","state":["exists","up"]},{"osd":5,"uuid":"58cebe34-0821-458f-86b5-7d460829d994","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1237795056},{"type":"v1","addr":"172.21.15.27:6817","nonce":1237795056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1237795056},{"type":"v1","addr":"172.21.15.27:6819","nonce":1237795056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1237795056},{"type":"v1","addr":"172.21.15.27:6823","nonce":1237795056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1237795056},{"type":"v1","addr":"172.21.15.27:6821","nonce":1237795056}]},"public_addr":"172.21.15.27:6817/1237795056","cluster_addr":"172.21.15.27:6819/1237795056","heartbeat_back_addr":"172.21.15.27:6823/1237795056","heartbeat_front_addr":"172.21.15.27:6821/1237795056","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:42:58.192450+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:43:27.129522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:43:54.750854+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:44:19.111479+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:44:44.487429+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:45:09.605548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/690437556":"2024-04-23T15:41:40.376957+0000","172.21.15.16:6800/3972982132":"2024-04-23T15:41:40.376957+0000","172.21.15.16:0/2002585335":"2024-04-23T15:40:27.107187+0000","172.21.15.16:0/4021274304":"2024-04-23T15:41:40.376957+0000","172.21.15.16:0/4172787384":"2024-04-23T15:40:27.107187+0000","172.21.15.16:0/955466902":"2024-04-23T15:40:27.107187+0000","172.21.15.16:6801/1748504461":"2024-04-23T15:39:53.479948+0000","172.21.15.16:6801/3431522404":"2024-04-23T15:40:27.107187+0000","172.21.15.16:6800/1748504461":"2024-04-23T15:39:53.479948+0000","172.21.15.16:0/2570865319":"2024-04-23T15:39:53.479948+0000","172.21.15.16:0/2048283809":"2024-04-23T15:39:53.479948+0000","172.21.15.16:0/4252231167":"2024-04-23T15:41:40.376957+0000","172.21.15.16:0/2348469475":"2024-04-23T15:39:53.479948+0000","172.21.15.16:6801/3972982132":"2024-04-23T15:41:40.376957+0000","172.21.15.16:6800/3431522404":"2024-04-23T15:40:27.107187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T15:45:17.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.940+0000 7fdc45dd3700 1 -- 172.21.15.16:0/1323969074 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fdc2806c4e0 msgr2=0x7fdc2806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:17.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.940+0000 7fdc45dd3700 1 --2- 172.21.15.16:0/1323969074 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fdc2806c4e0 0x7fdc2806e9a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdc30006010 tx=0x7fdc3001a040 comp rx=0 tx=0).stop 2024-04-22T15:45:17.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.940+0000 7fdc45dd3700 1 -- 172.21.15.16:0/1323969074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc40102780 msgr2=0x7fdc40100000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:17.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.940+0000 7fdc45dd3700 1 --2- 172.21.15.16:0/1323969074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc40102780 0x7fdc40100000 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdc3400c8f0 tx=0x7fdc3400cc00 comp rx=0 tx=0).stop 2024-04-22T15:45:17.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.941+0000 7fdc45dd3700 1 -- 172.21.15.16:0/1323969074 shutdown_connections 2024-04-22T15:45:17.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.941+0000 7fdc45dd3700 1 --2- 172.21.15.16:0/1323969074 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fdc2806c4e0 0x7fdc2806e9a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:17.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.941+0000 7fdc45dd3700 1 --2- 172.21.15.16:0/1323969074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40101e60 0x7fdc400ffac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:17.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.941+0000 7fdc45dd3700 1 --2- 172.21.15.16:0/1323969074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc40102780 0x7fdc40100000 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:17.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.941+0000 7fdc45dd3700 1 -- 172.21.15.16:0/1323969074 >> 172.21.15.16:0/1323969074 conn(0x7fdc400fb650 msgr2=0x7fdc40105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:17.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.941+0000 7fdc45dd3700 1 -- 172.21.15.16:0/1323969074 shutdown_connections 2024-04-22T15:45:17.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:17.941+0000 7fdc45dd3700 1 -- 172.21.15.16:0/1323969074 wait complete. 2024-04-22T15:45:18.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:17 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/1323969074' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T15:45:18.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:17 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/1323969074' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T15:45:18.547 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T15:43:58.896563+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T15:45:18.548 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-22T15:45:18.759 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:45:19.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:18 smithi016 ceph-mon[26620]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:19.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:18 smithi027 ceph-mon[34922]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:20.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.170+0000 7f8494998700 1 -- 172.21.15.16:0/492042977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8490100e90 msgr2=0x7f8490101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:20.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.170+0000 7f8494998700 1 --2- 172.21.15.16:0/492042977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8490100e90 0x7f8490101270 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8488009910 tx=0x7f8488009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:20.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.170+0000 7f8494998700 1 -- 172.21.15.16:0/492042977 shutdown_connections 2024-04-22T15:45:20.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.170+0000 7f8494998700 1 --2- 172.21.15.16:0/492042977 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8490101840 0x7f8490105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:20.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.170+0000 7f8494998700 1 --2- 172.21.15.16:0/492042977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8490100e90 0x7f8490101270 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:20.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.170+0000 7f8494998700 1 -- 172.21.15.16:0/492042977 >> 172.21.15.16:0/492042977 conn(0x7f84900fc740 msgr2=0x7f84900feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:20.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.170+0000 7f8494998700 1 -- 172.21.15.16:0/492042977 shutdown_connections 2024-04-22T15:45:20.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.170+0000 7f8494998700 1 -- 172.21.15.16:0/492042977 wait complete. 2024-04-22T15:45:20.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.171+0000 7f8494998700 1 Processor -- start 2024-04-22T15:45:20.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.171+0000 7f8494998700 1 -- start start 2024-04-22T15:45:20.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.171+0000 7f8494998700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8490100e90 0x7f8490073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:20.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.171+0000 7f8494998700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8490101840 0x7f84900735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:20.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.171+0000 7f8494998700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8490073b10 con 0x7f8490100e90 2024-04-22T15:45:20.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.171+0000 7f8494998700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8490073c50 con 0x7f8490101840 2024-04-22T15:45:20.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.171+0000 7f848e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8490100e90 0x7f8490073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:20.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.171+0000 7f848e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8490100e90 0x7f8490073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52978/0 (socket says 172.21.15.16:52978) 2024-04-22T15:45:20.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.171+0000 7f848e59c700 1 -- 172.21.15.16:0/1366350208 learned_addr learned my addr 172.21.15.16:0/1366350208 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:45:20.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.172+0000 7f848e59c700 1 -- 172.21.15.16:0/1366350208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8490101840 msgr2=0x7f84900735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:20.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.172+0000 7f848dd9b700 1 --2- 172.21.15.16:0/1366350208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8490101840 0x7f84900735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:20.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.172+0000 7f848e59c700 1 --2- 172.21.15.16:0/1366350208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8490101840 0x7f84900735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:20.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.172+0000 7f848e59c700 1 -- 172.21.15.16:0/1366350208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84880095f0 con 0x7f8490100e90 2024-04-22T15:45:20.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.172+0000 7f848dd9b700 1 --2- 172.21.15.16:0/1366350208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8490101840 0x7f84900735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:45:20.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.172+0000 7f848e59c700 1 --2- 172.21.15.16:0/1366350208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8490100e90 0x7f8490073090 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f848800b5c0 tx=0x7f848800be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:20.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.172+0000 7f84877fe700 1 -- 172.21.15.16:0/1366350208 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f848801d070 con 0x7f8490100e90 2024-04-22T15:45:20.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.172+0000 7f84877fe700 1 -- 172.21.15.16:0/1366350208 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8488022470 con 0x7f8490100e90 2024-04-22T15:45:20.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.172+0000 7f84877fe700 1 -- 172.21.15.16:0/1366350208 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f848800f670 con 0x7f8490100e90 2024-04-22T15:45:20.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.172+0000 7f8494998700 1 -- 172.21.15.16:0/1366350208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8490073ed0 con 0x7f8490100e90 2024-04-22T15:45:20.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.172+0000 7f8494998700 1 -- 172.21.15.16:0/1366350208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84901a2dc0 con 0x7f8490100e90 2024-04-22T15:45:20.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.173+0000 7f8494998700 1 -- 172.21.15.16:0/1366350208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8490109230 con 0x7f8490100e90 2024-04-22T15:45:20.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.174+0000 7f84877fe700 1 -- 172.21.15.16:0/1366350208 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f8488004c30 con 0x7f8490100e90 2024-04-22T15:45:20.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.174+0000 7f84877fe700 1 --2- 172.21.15.16:0/1366350208 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f847c06c4e0 0x7f847c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:20.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.174+0000 7f84877fe700 1 -- 172.21.15.16:0/1366350208 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f848808ca10 con 0x7f8490100e90 2024-04-22T15:45:20.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.174+0000 7f848dd9b700 1 --2- 172.21.15.16:0/1366350208 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f847c06c4e0 0x7f847c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:20.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.175+0000 7f848dd9b700 1 --2- 172.21.15.16:0/1366350208 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f847c06c4e0 0x7f847c06e9a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8478005fd0 tx=0x7f8478005dc0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:20.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.179+0000 7f84877fe700 1 -- 172.21.15.16:0/1366350208 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8488057660 con 0x7f8490100e90 2024-04-22T15:45:20.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.375+0000 7f8494998700 1 -- 172.21.15.16:0/1366350208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8490108c10 con 0x7f8490100e90 2024-04-22T15:45:20.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.376+0000 7f84877fe700 1 -- 172.21.15.16:0/1366350208 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8488027020 con 0x7f8490100e90 2024-04-22T15:45:20.379 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-22T15:45:20.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.378+0000 7f8494998700 1 -- 172.21.15.16:0/1366350208 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f847c06c4e0 msgr2=0x7f847c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:20.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.378+0000 7f8494998700 1 --2- 172.21.15.16:0/1366350208 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f847c06c4e0 0x7f847c06e9a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8478005fd0 tx=0x7f8478005dc0 comp rx=0 tx=0).stop 2024-04-22T15:45:20.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.378+0000 7f8494998700 1 -- 172.21.15.16:0/1366350208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8490100e90 msgr2=0x7f8490073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:20.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.378+0000 7f8494998700 1 --2- 172.21.15.16:0/1366350208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8490100e90 0x7f8490073090 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f848800b5c0 tx=0x7f848800be00 comp rx=0 tx=0).stop 2024-04-22T15:45:20.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.378+0000 7f8494998700 1 -- 172.21.15.16:0/1366350208 shutdown_connections 2024-04-22T15:45:20.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.378+0000 7f8494998700 1 --2- 172.21.15.16:0/1366350208 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f847c06c4e0 0x7f847c06e9a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:20.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.378+0000 7f8494998700 1 --2- 172.21.15.16:0/1366350208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8490100e90 0x7f8490073090 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:20.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.378+0000 7f8494998700 1 --2- 172.21.15.16:0/1366350208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8490101840 0x7f84900735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:20.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.378+0000 7f8494998700 1 -- 172.21.15.16:0/1366350208 >> 172.21.15.16:0/1366350208 conn(0x7f84900fc740 msgr2=0x7f84900fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:20.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.378+0000 7f8494998700 1 -- 172.21.15.16:0/1366350208 shutdown_connections 2024-04-22T15:45:20.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:20.378+0000 7f8494998700 1 -- 172.21.15.16:0/1366350208 wait complete. 2024-04-22T15:45:20.721 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:20 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/1366350208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T15:45:20.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:20 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/1366350208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T15:45:21.073 INFO:tasks.cephadm:Setting up client nodes... 2024-04-22T15:45:21.074 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-22T15:45:21.290 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:45:21.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:21 smithi016 ceph-mon[26620]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:21.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:21 smithi027 ceph-mon[34922]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:22.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.881+0000 7fce06ae5700 1 -- 172.21.15.16:0/3944097747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce000fed00 msgr2=0x7fce000ff160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:22.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.881+0000 7fce06ae5700 1 --2- 172.21.15.16:0/3944097747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce000fed00 0x7fce000ff160 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fcdec009910 tx=0x7fcdec009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:22.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.881+0000 7fce06ae5700 1 -- 172.21.15.16:0/3944097747 shutdown_connections 2024-04-22T15:45:22.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.881+0000 7fce06ae5700 1 --2- 172.21.15.16:0/3944097747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce000fed00 0x7fce000ff160 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:22.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.881+0000 7fce06ae5700 1 --2- 172.21.15.16:0/3944097747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce00102280 0x7fce001026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:22.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.881+0000 7fce06ae5700 1 -- 172.21.15.16:0/3944097747 >> 172.21.15.16:0/3944097747 conn(0x7fce000fa8c0 msgr2=0x7fce000fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:22.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.881+0000 7fce06ae5700 1 -- 172.21.15.16:0/3944097747 shutdown_connections 2024-04-22T15:45:22.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.881+0000 7fce06ae5700 1 -- 172.21.15.16:0/3944097747 wait complete. 2024-04-22T15:45:22.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.881+0000 7fce06ae5700 1 Processor -- start 2024-04-22T15:45:22.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.881+0000 7fce06ae5700 1 -- start start 2024-04-22T15:45:22.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.882+0000 7fce06ae5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce000fed00 0x7fce00196550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:22.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.882+0000 7fce06ae5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce00102280 0x7fce00196a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:22.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.882+0000 7fce06ae5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce001970b0 con 0x7fce00102280 2024-04-22T15:45:22.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.882+0000 7fce06ae5700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce001971f0 con 0x7fce000fed00 2024-04-22T15:45:22.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.882+0000 7fcdfffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce00102280 0x7fce00196a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:22.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.882+0000 7fcdfffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce00102280 0x7fce00196a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47374/0 (socket says 172.21.15.16:47374) 2024-04-22T15:45:22.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.882+0000 7fcdfffff700 1 -- 172.21.15.16:0/1195203103 learned_addr learned my addr 172.21.15.16:0/1195203103 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:45:22.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.882+0000 7fce04881700 1 --2- 172.21.15.16:0/1195203103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce000fed00 0x7fce00196550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:22.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.882+0000 7fcdfffff700 1 -- 172.21.15.16:0/1195203103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce000fed00 msgr2=0x7fce00196550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:22.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.882+0000 7fcdfffff700 1 --2- 172.21.15.16:0/1195203103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce000fed00 0x7fce00196550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:22.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.882+0000 7fcdfffff700 1 -- 172.21.15.16:0/1195203103 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdec0095f0 con 0x7fce00102280 2024-04-22T15:45:22.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.882+0000 7fcdfffff700 1 --2- 172.21.15.16:0/1195203103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce00102280 0x7fce00196a90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcdec0092e0 tx=0x7fcdec0055f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:22.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.882+0000 7fcdfdffb700 1 -- 172.21.15.16:0/1195203103 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdec01d070 con 0x7fce00102280 2024-04-22T15:45:22.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.882+0000 7fce06ae5700 1 -- 172.21.15.16:0/1195203103 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce0019bc40 con 0x7fce00102280 2024-04-22T15:45:22.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.883+0000 7fcdfdffb700 1 -- 172.21.15.16:0/1195203103 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcdec00f460 con 0x7fce00102280 2024-04-22T15:45:22.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.883+0000 7fcdfdffb700 1 -- 172.21.15.16:0/1195203103 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdec021640 con 0x7fce00102280 2024-04-22T15:45:22.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.883+0000 7fce06ae5700 1 -- 172.21.15.16:0/1195203103 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce0019c130 con 0x7fce00102280 2024-04-22T15:45:22.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.884+0000 7fce06ae5700 1 -- 172.21.15.16:0/1195203103 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce00109810 con 0x7fce00102280 2024-04-22T15:45:22.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.884+0000 7fcdfdffb700 1 -- 172.21.15.16:0/1195203103 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fcdec00fab0 con 0x7fce00102280 2024-04-22T15:45:22.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.885+0000 7fcdfdffb700 1 --2- 172.21.15.16:0/1195203103 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fcdf006c4e0 0x7fcdf006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:22.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.885+0000 7fcdfdffb700 1 -- 172.21.15.16:0/1195203103 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fcdec08c990 con 0x7fce00102280 2024-04-22T15:45:22.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.885+0000 7fce04881700 1 --2- 172.21.15.16:0/1195203103 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fcdf006c4e0 0x7fcdf006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:22.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.885+0000 7fce04881700 1 --2- 172.21.15.16:0/1195203103 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fcdf006c4e0 0x7fcdf006e9a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fce00068b30 tx=0x7fcdf400a380 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:22.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:22.890+0000 7fcdfdffb700 1 -- 172.21.15.16:0/1195203103 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fcdec05b000 con 0x7fce00102280 2024-04-22T15:45:23.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:23.171+0000 7fce06ae5700 1 -- 172.21.15.16:0/1195203103 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fce00109a00 con 0x7fce00102280 2024-04-22T15:45:23.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:23.175+0000 7fcdfdffb700 1 -- 172.21.15.16:0/1195203103 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7fcdec05ab90 con 0x7fce00102280 2024-04-22T15:45:23.178 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-04-22T15:45:23.178 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAThiZmBVBKChAAtmD2vNoyL59v/vW9DEkTcQ== 2024-04-22T15:45:23.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:23.177+0000 7fce06ae5700 1 -- 172.21.15.16:0/1195203103 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fcdf006c4e0 msgr2=0x7fcdf006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:23.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:23.177+0000 7fce06ae5700 1 --2- 172.21.15.16:0/1195203103 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fcdf006c4e0 0x7fcdf006e9a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fce00068b30 tx=0x7fcdf400a380 comp rx=0 tx=0).stop 2024-04-22T15:45:23.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:23.177+0000 7fce06ae5700 1 -- 172.21.15.16:0/1195203103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce00102280 msgr2=0x7fce00196a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:23.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:23.177+0000 7fce06ae5700 1 --2- 172.21.15.16:0/1195203103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce00102280 0x7fce00196a90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcdec0092e0 tx=0x7fcdec0055f0 comp rx=0 tx=0).stop 2024-04-22T15:45:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:23.177+0000 7fce06ae5700 1 -- 172.21.15.16:0/1195203103 shutdown_connections 2024-04-22T15:45:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:23.177+0000 7fce06ae5700 1 --2- 172.21.15.16:0/1195203103 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fcdf006c4e0 0x7fcdf006e9a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:23.177+0000 7fce06ae5700 1 --2- 172.21.15.16:0/1195203103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce000fed00 0x7fce00196550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:23.177+0000 7fce06ae5700 1 --2- 172.21.15.16:0/1195203103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce00102280 0x7fce00196a90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:23.177+0000 7fce06ae5700 1 -- 172.21.15.16:0/1195203103 >> 172.21.15.16:0/1195203103 conn(0x7fce000fa8c0 msgr2=0x7fce00105250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:23.177+0000 7fce06ae5700 1 -- 172.21.15.16:0/1195203103 shutdown_connections 2024-04-22T15:45:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:23.177+0000 7fce06ae5700 1 -- 172.21.15.16:0/1195203103 wait complete. 2024-04-22T15:45:23.736 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:23 smithi016 ceph-mon[26620]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:23.736 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:23 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/1195203103' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T15:45:23.736 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:23 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/1195203103' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T15:45:23.757 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-22T15:45:23.758 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-22T15:45:23.758 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-22T15:45:23.793 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-22T15:45:23.822 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:23 smithi027 ceph-mon[34922]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:23.822 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:23 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/1195203103' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T15:45:23.822 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:23 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/1195203103' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T15:45:24.018 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi027/config 2024-04-22T15:45:25.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.647+0000 7f750ca92700 1 -- 172.21.15.27:0/1108575420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7508101740 msgr2=0x7f7508103b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.647+0000 7f750ca92700 1 --2- 172.21.15.27:0/1108575420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7508101740 0x7f7508103b30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f750000b0a0 tx=0x7f750000b3b0 comp rx=0 tx=0).stop 2024-04-22T15:45:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.647+0000 7f750ca92700 1 -- 172.21.15.27:0/1108575420 shutdown_connections 2024-04-22T15:45:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.647+0000 7f750ca92700 1 --2- 172.21.15.27:0/1108575420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7508104070 0x7f7508106460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.647+0000 7f750ca92700 1 --2- 172.21.15.27:0/1108575420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7508101740 0x7f7508103b30 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.647+0000 7f750ca92700 1 -- 172.21.15.27:0/1108575420 >> 172.21.15.27:0/1108575420 conn(0x7f75080fb150 msgr2=0x7f75080fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.647+0000 7f750ca92700 1 -- 172.21.15.27:0/1108575420 shutdown_connections 2024-04-22T15:45:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.647+0000 7f750ca92700 1 -- 172.21.15.27:0/1108575420 wait complete. 2024-04-22T15:45:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.648+0000 7f750ca92700 1 Processor -- start 2024-04-22T15:45:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.648+0000 7f750ca92700 1 -- start start 2024-04-22T15:45:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.648+0000 7f750ca92700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7508101740 0x7f7508196490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.648+0000 7f750ca92700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7508104070 0x7f75081969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.648+0000 7f750ca92700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7508196ff0 con 0x7f7508101740 2024-04-22T15:45:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.648+0000 7f750ca92700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7508197130 con 0x7f7508104070 2024-04-22T15:45:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.648+0000 7f7505d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7508104070 0x7f75081969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.648+0000 7f7505d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7508104070 0x7f75081969d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52438/0 (socket says 172.21.15.27:52438) 2024-04-22T15:45:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.648+0000 7f7505d9b700 1 -- 172.21.15.27:0/1778373667 learned_addr learned my addr 172.21.15.27:0/1778373667 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-22T15:45:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.648+0000 7f750659c700 1 --2- 172.21.15.27:0/1778373667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7508101740 0x7f7508196490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.649+0000 7f7505d9b700 1 -- 172.21.15.27:0/1778373667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7508101740 msgr2=0x7f7508196490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.649+0000 7f7505d9b700 1 --2- 172.21.15.27:0/1778373667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7508101740 0x7f7508196490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.649+0000 7f7505d9b700 1 -- 172.21.15.27:0/1778373667 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7500009d00 con 0x7f7508104070 2024-04-22T15:45:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.649+0000 7f7505d9b700 1 --2- 172.21.15.27:0/1778373667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7508104070 0x7f75081969d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f74f000c8a0 tx=0x7f74f000cbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.649+0000 7f74ff7fe700 1 -- 172.21.15.27:0/1778373667 <== mon.1 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74f00078e0 con 0x7f7508104070 2024-04-22T15:45:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.649+0000 7f750ca92700 1 -- 172.21.15.27:0/1778373667 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f750819bbe0 con 0x7f7508104070 2024-04-22T15:45:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.649+0000 7f74ff7fe700 1 -- 172.21.15.27:0/1778373667 <== mon.1 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f74f000f450 con 0x7f7508104070 2024-04-22T15:45:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.649+0000 7f74ff7fe700 1 -- 172.21.15.27:0/1778373667 <== mon.1 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74f000e5e0 con 0x7f7508104070 2024-04-22T15:45:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.649+0000 7f750ca92700 1 -- 172.21.15.27:0/1778373667 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f750819c130 con 0x7f7508104070 2024-04-22T15:45:25.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.650+0000 7f750ca92700 1 -- 172.21.15.27:0/1778373667 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7508109810 con 0x7f7508104070 2024-04-22T15:45:25.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.651+0000 7f74ff7fe700 1 -- 172.21.15.27:0/1778373667 <== mon.1 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f74f000f5c0 con 0x7f7508104070 2024-04-22T15:45:25.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.651+0000 7f74ff7fe700 1 --2- 172.21.15.27:0/1778373667 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f74f406c4e0 0x7f74f406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:25.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.651+0000 7f74ff7fe700 1 -- 172.21.15.27:0/1778373667 <== mon.1 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f74f008b380 con 0x7f7508104070 2024-04-22T15:45:25.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.651+0000 7f750659c700 1 --2- 172.21.15.27:0/1778373667 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f74f406c4e0 0x7f74f406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:25.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.652+0000 7f750659c700 1 --2- 172.21.15.27:0/1778373667 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f74f406c4e0 0x7f74f406e9a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7500000f80 tx=0x7f75000060c0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.656+0000 7f74ff7fe700 1 -- 172.21.15.27:0/1778373667 <== mon.1 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f74f00599b0 con 0x7f7508104070 2024-04-22T15:45:25.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:25 smithi027 ceph-mon[34922]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:25.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:25 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:45:25.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:25 smithi016 ceph-mon[26620]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:25.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:25 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:45:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.931+0000 7f750ca92700 1 -- 172.21.15.27:0/1778373667 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f7508109a00 con 0x7f7508104070 2024-04-22T15:45:25.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.936+0000 7f74ff7fe700 1 -- 172.21.15.27:0/1778373667 <== mon.1 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7f74f0059540 con 0x7f7508104070 2024-04-22T15:45:25.943 INFO:teuthology.orchestra.run.smithi027.stdout:[client.1] 2024-04-22T15:45:25.943 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQAVhiZmkCurNxAA+2/mF8KPV8VfPS7bOIEcCQ== 2024-04-22T15:45:25.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.938+0000 7f750ca92700 1 -- 172.21.15.27:0/1778373667 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f74f406c4e0 msgr2=0x7f74f406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:25.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.938+0000 7f750ca92700 1 --2- 172.21.15.27:0/1778373667 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f74f406c4e0 0x7f74f406e9a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7500000f80 tx=0x7f75000060c0 comp rx=0 tx=0).stop 2024-04-22T15:45:25.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.938+0000 7f750ca92700 1 -- 172.21.15.27:0/1778373667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7508104070 msgr2=0x7f75081969d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:25.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.938+0000 7f750ca92700 1 --2- 172.21.15.27:0/1778373667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7508104070 0x7f75081969d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f74f000c8a0 tx=0x7f74f000cbb0 comp rx=0 tx=0).stop 2024-04-22T15:45:25.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.938+0000 7f750ca92700 1 -- 172.21.15.27:0/1778373667 shutdown_connections 2024-04-22T15:45:25.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.938+0000 7f750ca92700 1 --2- 172.21.15.27:0/1778373667 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f74f406c4e0 0x7f74f406e9a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:25.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.938+0000 7f750ca92700 1 --2- 172.21.15.27:0/1778373667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7508101740 0x7f7508196490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:25.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.938+0000 7f750ca92700 1 --2- 172.21.15.27:0/1778373667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7508104070 0x7f75081969d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:25.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.938+0000 7f750ca92700 1 -- 172.21.15.27:0/1778373667 >> 172.21.15.27:0/1778373667 conn(0x7f75080fb150 msgr2=0x7f75080fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:25.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.938+0000 7f750ca92700 1 -- 172.21.15.27:0/1778373667 shutdown_connections 2024-04-22T15:45:25.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:45:25.938+0000 7f750ca92700 1 -- 172.21.15.27:0/1778373667 wait complete. 2024-04-22T15:45:26.561 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-22T15:45:26.561 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-22T15:45:26.562 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-22T15:45:26.641 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-22T15:45:26.641 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-22T15:45:26.642 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-22T15:45:26.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:26 smithi027 ceph-mon[34922]: from='client.? 172.21.15.27:0/1778373667' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T15:45:26.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:26 smithi027 ceph-mon[34922]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T15:45:26.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:26 smithi027 ceph-mon[34922]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T15:45:26.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:26 smithi027 ceph-mon[34922]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:26.864 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:45:26.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:26 smithi016 ceph-mon[26620]: from='client.? 172.21.15.27:0/1778373667' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T15:45:26.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:26 smithi016 ceph-mon[26620]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T15:45:26.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:26 smithi016 ceph-mon[26620]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T15:45:26.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:26 smithi016 ceph-mon[26620]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:28.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.375+0000 7f59f7221700 1 -- 172.21.15.16:0/4271994032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59f00ff5a0 msgr2=0x7f59f00ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:28.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.375+0000 7f59f7221700 1 --2- 172.21.15.16:0/4271994032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59f00ff5a0 0x7f59f00ff980 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f59e4009910 tx=0x7f59e4009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:28.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.375+0000 7f59f7221700 1 -- 172.21.15.16:0/4271994032 shutdown_connections 2024-04-22T15:45:28.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.375+0000 7f59f7221700 1 --2- 172.21.15.16:0/4271994032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f00fff50 0x7f59f0105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:28.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.375+0000 7f59f7221700 1 --2- 172.21.15.16:0/4271994032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59f00ff5a0 0x7f59f00ff980 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:28.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.375+0000 7f59f7221700 1 -- 172.21.15.16:0/4271994032 >> 172.21.15.16:0/4271994032 conn(0x7f59f0074e30 msgr2=0x7f59f0075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:28.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.376+0000 7f59f7221700 1 -- 172.21.15.16:0/4271994032 shutdown_connections 2024-04-22T15:45:28.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.376+0000 7f59f7221700 1 -- 172.21.15.16:0/4271994032 wait complete. 2024-04-22T15:45:28.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.376+0000 7f59f7221700 1 Processor -- start 2024-04-22T15:45:28.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.376+0000 7f59f7221700 1 -- start start 2024-04-22T15:45:28.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.376+0000 7f59f7221700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59f00ff5a0 0x7f59f0100520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:28.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.376+0000 7f59f7221700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f00fff50 0x7f59f0100a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:28.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.376+0000 7f59f7221700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f0100fa0 con 0x7f59f00ff5a0 2024-04-22T15:45:28.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.377+0000 7f59f7221700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f01010e0 con 0x7f59f00fff50 2024-04-22T15:45:28.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.377+0000 7f59f4fbd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59f00ff5a0 0x7f59f0100520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:28.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.377+0000 7f59f4fbd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59f00ff5a0 0x7f59f0100520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47394/0 (socket says 172.21.15.16:47394) 2024-04-22T15:45:28.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.377+0000 7f59f4fbd700 1 -- 172.21.15.16:0/4122688669 learned_addr learned my addr 172.21.15.16:0/4122688669 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:45:28.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.377+0000 7f59effff700 1 --2- 172.21.15.16:0/4122688669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f00fff50 0x7f59f0100a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:28.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.377+0000 7f59f4fbd700 1 -- 172.21.15.16:0/4122688669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f00fff50 msgr2=0x7f59f0100a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:28.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.377+0000 7f59f4fbd700 1 --2- 172.21.15.16:0/4122688669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f00fff50 0x7f59f0100a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:28.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.377+0000 7f59f4fbd700 1 -- 172.21.15.16:0/4122688669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e40095f0 con 0x7f59f00ff5a0 2024-04-22T15:45:28.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.377+0000 7f59effff700 1 --2- 172.21.15.16:0/4122688669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f00fff50 0x7f59f0100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:45:28.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.377+0000 7f59f4fbd700 1 --2- 172.21.15.16:0/4122688669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59f00ff5a0 0x7f59f0100520 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f59e4006010 tx=0x7f59e400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:28.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.377+0000 7f59edffb700 1 -- 172.21.15.16:0/4122688669 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59e401d070 con 0x7f59f00ff5a0 2024-04-22T15:45:28.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.377+0000 7f59edffb700 1 -- 172.21.15.16:0/4122688669 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f59e400f460 con 0x7f59f00ff5a0 2024-04-22T15:45:28.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.377+0000 7f59edffb700 1 -- 172.21.15.16:0/4122688669 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59e4021640 con 0x7f59f00ff5a0 2024-04-22T15:45:28.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.377+0000 7f59f7221700 1 -- 172.21.15.16:0/4122688669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f0101360 con 0x7f59f00ff5a0 2024-04-22T15:45:28.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.378+0000 7f59f7221700 1 -- 172.21.15.16:0/4122688669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f01a2e30 con 0x7f59f00ff5a0 2024-04-22T15:45:28.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.378+0000 7f59f7221700 1 -- 172.21.15.16:0/4122688669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f0109270 con 0x7f59f00ff5a0 2024-04-22T15:45:28.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.379+0000 7f59edffb700 1 -- 172.21.15.16:0/4122688669 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f59e400faf0 con 0x7f59f00ff5a0 2024-04-22T15:45:28.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.379+0000 7f59edffb700 1 --2- 172.21.15.16:0/4122688669 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f59e006c4e0 0x7f59e006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:28.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.379+0000 7f59edffb700 1 -- 172.21.15.16:0/4122688669 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f59e408c6d0 con 0x7f59f00ff5a0 2024-04-22T15:45:28.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.380+0000 7f59effff700 1 --2- 172.21.15.16:0/4122688669 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f59e006c4e0 0x7f59e006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:28.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.380+0000 7f59effff700 1 --2- 172.21.15.16:0/4122688669 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f59e006c4e0 0x7f59e006e9a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f59f0101d10 tx=0x7f59dc005f50 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:28.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.384+0000 7f59edffb700 1 -- 172.21.15.16:0/4122688669 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f59e4057270 con 0x7f59f00ff5a0 2024-04-22T15:45:28.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.639+0000 7f59f7221700 1 -- 172.21.15.16:0/4122688669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f59f0108c50 con 0x7f59f00ff5a0 2024-04-22T15:45:28.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.643+0000 7f59edffb700 1 -- 172.21.15.16:0/4122688669 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+173034 (secure 0 0 0) 0x7f59e405a890 con 0x7f59f00ff5a0 2024-04-22T15:45:28.649 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:45:28.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.645+0000 7f59f7221700 1 -- 172.21.15.16:0/4122688669 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f59e006c4e0 msgr2=0x7f59e006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:28.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.645+0000 7f59f7221700 1 --2- 172.21.15.16:0/4122688669 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f59e006c4e0 0x7f59e006e9a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f59f0101d10 tx=0x7f59dc005f50 comp rx=0 tx=0).stop 2024-04-22T15:45:28.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.646+0000 7f59f7221700 1 -- 172.21.15.16:0/4122688669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59f00ff5a0 msgr2=0x7f59f0100520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:28.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.646+0000 7f59f7221700 1 --2- 172.21.15.16:0/4122688669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59f00ff5a0 0x7f59f0100520 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f59e4006010 tx=0x7f59e400be20 comp rx=0 tx=0).stop 2024-04-22T15:45:28.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.646+0000 7f59f7221700 1 -- 172.21.15.16:0/4122688669 shutdown_connections 2024-04-22T15:45:28.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.646+0000 7f59f7221700 1 --2- 172.21.15.16:0/4122688669 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f59e006c4e0 0x7f59e006e9a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:28.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.646+0000 7f59f7221700 1 --2- 172.21.15.16:0/4122688669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59f00ff5a0 0x7f59f0100520 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:28.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.646+0000 7f59f7221700 1 --2- 172.21.15.16:0/4122688669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f00fff50 0x7f59f0100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:28.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.646+0000 7f59f7221700 1 -- 172.21.15.16:0/4122688669 >> 172.21.15.16:0/4122688669 conn(0x7f59f0074e30 msgr2=0x7f59f00fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:28.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.646+0000 7f59f7221700 1 -- 172.21.15.16:0/4122688669 shutdown_connections 2024-04-22T15:45:28.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:28.646+0000 7f59f7221700 1 -- 172.21.15.16:0/4122688669 wait complete. 2024-04-22T15:45:29.336 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":19,"active_gid":14217,"active_name":"smithi016.rnsnnr","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1057234399},{"type":"v1","addr":"172.21.15.16:6801","nonce":1057234399}]},"active_addr":"172.21.15.16:6801/1057234399","active_change":"2024-04-22T15:41:40.377066+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":14242,"name":"smithi027.ukdgzs","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.16:8443/","prometheus":"http://172.21.15.16:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":3682380}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":3645737911}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":25024044}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":1878962216}]}]} 2024-04-22T15:45:29.340 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-22T15:45:29.341 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-22T15:45:29.341 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T15:45:29.557 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:45:29.644 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:29 smithi016 ceph-mon[26620]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:29.644 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:29 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/4122688669' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T15:45:29.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:29 smithi027 ceph-mon[34922]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:29.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:29 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/4122688669' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T15:45:30.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.989+0000 7f264b420700 1 -- 172.21.15.16:0/1929230724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2644101de0 msgr2=0x7f26441021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:30.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.989+0000 7f264b420700 1 --2- 172.21.15.16:0/1929230724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2644101de0 0x7f26441021c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2640009910 tx=0x7f2640009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:30.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.989+0000 7f264b420700 1 -- 172.21.15.16:0/1929230724 shutdown_connections 2024-04-22T15:45:30.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.989+0000 7f264b420700 1 --2- 172.21.15.16:0/1929230724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2644102700 0x7f264410abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:30.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.989+0000 7f264b420700 1 --2- 172.21.15.16:0/1929230724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2644101de0 0x7f26441021c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:30.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.989+0000 7f264b420700 1 -- 172.21.15.16:0/1929230724 >> 172.21.15.16:0/1929230724 conn(0x7f26440fb650 msgr2=0x7f26440fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:30.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.989+0000 7f264b420700 1 -- 172.21.15.16:0/1929230724 shutdown_connections 2024-04-22T15:45:30.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.989+0000 7f264b420700 1 -- 172.21.15.16:0/1929230724 wait complete. 2024-04-22T15:45:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.990+0000 7f264b420700 1 Processor -- start 2024-04-22T15:45:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.990+0000 7f264b420700 1 -- start start 2024-04-22T15:45:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.990+0000 7f264b420700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2644101de0 0x7f264419c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.990+0000 7f264b420700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2644102700 0x7f264419cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.990+0000 7f264b420700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f264419d270 con 0x7f2644101de0 2024-04-22T15:45:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.990+0000 7f264b420700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2644196720 con 0x7f2644102700 2024-04-22T15:45:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.991+0000 7f26491bc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2644101de0 0x7f264419c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.991+0000 7f26491bc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2644101de0 0x7f264419c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47408/0 (socket says 172.21.15.16:47408) 2024-04-22T15:45:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.991+0000 7f26491bc700 1 -- 172.21.15.16:0/2235470146 learned_addr learned my addr 172.21.15.16:0/2235470146 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:45:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.991+0000 7f26489bb700 1 --2- 172.21.15.16:0/2235470146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2644102700 0x7f264419cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:30.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.991+0000 7f26491bc700 1 -- 172.21.15.16:0/2235470146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2644102700 msgr2=0x7f264419cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:30.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.991+0000 7f26491bc700 1 --2- 172.21.15.16:0/2235470146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2644102700 0x7f264419cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:30.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.991+0000 7f26491bc700 1 -- 172.21.15.16:0/2235470146 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26400095f0 con 0x7f2644101de0 2024-04-22T15:45:30.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.991+0000 7f26489bb700 1 --2- 172.21.15.16:0/2235470146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2644102700 0x7f264419cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:45:30.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.991+0000 7f26491bc700 1 --2- 172.21.15.16:0/2235470146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2644101de0 0x7f264419c6a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2640006010 tx=0x7f264000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:30.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.991+0000 7f263a7fc700 1 -- 172.21.15.16:0/2235470146 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f264001d070 con 0x7f2644101de0 2024-04-22T15:45:30.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.991+0000 7f263a7fc700 1 -- 172.21.15.16:0/2235470146 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2640004f50 con 0x7f2644101de0 2024-04-22T15:45:30.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.991+0000 7f264b420700 1 -- 172.21.15.16:0/2235470146 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26441969a0 con 0x7f2644101de0 2024-04-22T15:45:30.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.991+0000 7f263a7fc700 1 -- 172.21.15.16:0/2235470146 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2640021640 con 0x7f2644101de0 2024-04-22T15:45:30.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.991+0000 7f264b420700 1 -- 172.21.15.16:0/2235470146 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2644196e90 con 0x7f2644101de0 2024-04-22T15:45:30.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.992+0000 7f264b420700 1 -- 172.21.15.16:0/2235470146 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26441082f0 con 0x7f2644101de0 2024-04-22T15:45:30.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.993+0000 7f263a7fc700 1 -- 172.21.15.16:0/2235470146 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f264000f460 con 0x7f2644101de0 2024-04-22T15:45:30.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.993+0000 7f263a7fc700 1 --2- 172.21.15.16:0/2235470146 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f263006c4e0 0x7f263006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:30.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.993+0000 7f263a7fc700 1 -- 172.21.15.16:0/2235470146 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f264008c740 con 0x7f2644101de0 2024-04-22T15:45:30.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.993+0000 7f26489bb700 1 --2- 172.21.15.16:0/2235470146 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f263006c4e0 0x7f263006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:30.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.994+0000 7f26489bb700 1 --2- 172.21.15.16:0/2235470146 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f263006c4e0 0x7f263006e9a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2644197b90 tx=0x7f26340092c0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:30.998+0000 7f263a7fc700 1 -- 172.21.15.16:0/2235470146 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f264005acf0 con 0x7f2644101de0 2024-04-22T15:45:31.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:31.196+0000 7f264b420700 1 -- 172.21.15.16:0/2235470146 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2644197d30 con 0x7f2644101de0 2024-04-22T15:45:31.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:31.197+0000 7f263a7fc700 1 -- 172.21.15.16:0/2235470146 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11289 (secure 0 0 0) 0x7f2644197d30 con 0x7f2644101de0 2024-04-22T15:45:31.200 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:45:31.200 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":40,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","created":"2024-04-22T15:39:20.727181+0000","modified":"2024-04-22T15:45:12.502671+0000","last_up_change":"2024-04-22T15:45:11.497701+0000","last_in_change":"2024-04-22T15:44:52.054297+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T15:43:58.896563+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"67bd69ee-bd8c-4670-927a-f5fbcd92c9bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1210635569},{"type":"v1","addr":"172.21.15.16:6803","nonce":1210635569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1210635569},{"type":"v1","addr":"172.21.15.16:6805","nonce":1210635569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1210635569},{"type":"v1","addr":"172.21.15.16:6809","nonce":1210635569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1210635569},{"type":"v1","addr":"172.21.15.16:6807","nonce":1210635569}]},"public_addr":"172.21.15.16:6803/1210635569","cluster_addr":"172.21.15.16:6805/1210635569","heartbeat_back_addr":"172.21.15.16:6809/1210635569","heartbeat_front_addr":"172.21.15.16:6807/1210635569","state":["exists","up"]},{"osd":1,"uuid":"f51a9e4d-fb0c-4f62-9328-e23eea863213","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3974144580},{"type":"v1","addr":"172.21.15.16:6811","nonce":3974144580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3974144580},{"type":"v1","addr":"172.21.15.16:6813","nonce":3974144580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3974144580},{"type":"v1","addr":"172.21.15.16:6817","nonce":3974144580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3974144580},{"type":"v1","addr":"172.21.15.16:6815","nonce":3974144580}]},"public_addr":"172.21.15.16:6811/3974144580","cluster_addr":"172.21.15.16:6813/3974144580","heartbeat_back_addr":"172.21.15.16:6817/3974144580","heartbeat_front_addr":"172.21.15.16:6815/3974144580","state":["exists","up"]},{"osd":2,"uuid":"e59fdb03-e04e-4afd-a58e-0b58945d3765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2760833880},{"type":"v1","addr":"172.21.15.16:6819","nonce":2760833880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2760833880},{"type":"v1","addr":"172.21.15.16:6821","nonce":2760833880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2760833880},{"type":"v1","addr":"172.21.15.16:6825","nonce":2760833880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2760833880},{"type":"v1","addr":"172.21.15.16:6823","nonce":2760833880}]},"public_addr":"172.21.15.16:6819/2760833880","cluster_addr":"172.21.15.16:6821/2760833880","heartbeat_back_addr":"172.21.15.16:6825/2760833880","heartbeat_front_addr":"172.21.15.16:6823/2760833880","state":["exists","up"]},{"osd":3,"uuid":"fa25b413-d004-4455-8d7b-0ced7f880a7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":286252834},{"type":"v1","addr":"172.21.15.27:6801","nonce":286252834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":286252834},{"type":"v1","addr":"172.21.15.27:6803","nonce":286252834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":286252834},{"type":"v1","addr":"172.21.15.27:6807","nonce":286252834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":286252834},{"type":"v1","addr":"172.21.15.27:6805","nonce":286252834}]},"public_addr":"172.21.15.27:6801/286252834","cluster_addr":"172.21.15.27:6803/286252834","heartbeat_back_addr":"172.21.15.27:6807/286252834","heartbeat_front_addr":"172.21.15.27:6805/286252834","state":["exists","up"]},{"osd":4,"uuid":"6b1eda0d-1627-4c49-93b1-1a78896f660c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":691669816},{"type":"v1","addr":"172.21.15.27:6809","nonce":691669816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":691669816},{"type":"v1","addr":"172.21.15.27:6811","nonce":691669816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":691669816},{"type":"v1","addr":"172.21.15.27:6815","nonce":691669816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":691669816},{"type":"v1","addr":"172.21.15.27:6813","nonce":691669816}]},"public_addr":"172.21.15.27:6809/691669816","cluster_addr":"172.21.15.27:6811/691669816","heartbeat_back_addr":"172.21.15.27:6815/691669816","heartbeat_front_addr":"172.21.15.27:6813/691669816","state":["exists","up"]},{"osd":5,"uuid":"58cebe34-0821-458f-86b5-7d460829d994","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1237795056},{"type":"v1","addr":"172.21.15.27:6817","nonce":1237795056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1237795056},{"type":"v1","addr":"172.21.15.27:6819","nonce":1237795056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1237795056},{"type":"v1","addr":"172.21.15.27:6823","nonce":1237795056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1237795056},{"type":"v1","addr":"172.21.15.27:6821","nonce":1237795056}]},"public_addr":"172.21.15.27:6817/1237795056","cluster_addr":"172.21.15.27:6819/1237795056","heartbeat_back_addr":"172.21.15.27:6823/1237795056","heartbeat_front_addr":"172.21.15.27:6821/1237795056","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:42:58.192450+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:43:27.129522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:43:54.750854+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:44:19.111479+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:44:44.487429+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:45:09.605548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/690437556":"2024-04-23T15:41:40.376957+0000","172.21.15.16:6800/3972982132":"2024-04-23T15:41:40.376957+0000","172.21.15.16:0/2002585335":"2024-04-23T15:40:27.107187+0000","172.21.15.16:0/4021274304":"2024-04-23T15:41:40.376957+0000","172.21.15.16:0/4172787384":"2024-04-23T15:40:27.107187+0000","172.21.15.16:0/955466902":"2024-04-23T15:40:27.107187+0000","172.21.15.16:6801/1748504461":"2024-04-23T15:39:53.479948+0000","172.21.15.16:6801/3431522404":"2024-04-23T15:40:27.107187+0000","172.21.15.16:6800/1748504461":"2024-04-23T15:39:53.479948+0000","172.21.15.16:0/2570865319":"2024-04-23T15:39:53.479948+0000","172.21.15.16:0/2048283809":"2024-04-23T15:39:53.479948+0000","172.21.15.16:0/4252231167":"2024-04-23T15:41:40.376957+0000","172.21.15.16:0/2348469475":"2024-04-23T15:39:53.479948+0000","172.21.15.16:6801/3972982132":"2024-04-23T15:41:40.376957+0000","172.21.15.16:6800/3431522404":"2024-04-23T15:40:27.107187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T15:45:31.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:31.199+0000 7f264b420700 1 -- 172.21.15.16:0/2235470146 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f263006c4e0 msgr2=0x7f263006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:31.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:31.199+0000 7f264b420700 1 --2- 172.21.15.16:0/2235470146 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f263006c4e0 0x7f263006e9a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2644197b90 tx=0x7f26340092c0 comp rx=0 tx=0).stop 2024-04-22T15:45:31.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:31.199+0000 7f264b420700 1 -- 172.21.15.16:0/2235470146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2644101de0 msgr2=0x7f264419c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:31.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:31.199+0000 7f264b420700 1 --2- 172.21.15.16:0/2235470146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2644101de0 0x7f264419c6a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2640006010 tx=0x7f264000be20 comp rx=0 tx=0).stop 2024-04-22T15:45:31.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:31.199+0000 7f264b420700 1 -- 172.21.15.16:0/2235470146 shutdown_connections 2024-04-22T15:45:31.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:31.199+0000 7f264b420700 1 --2- 172.21.15.16:0/2235470146 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f263006c4e0 0x7f263006e9a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:31.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:31.199+0000 7f264b420700 1 --2- 172.21.15.16:0/2235470146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2644101de0 0x7f264419c6a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:31.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:31.199+0000 7f264b420700 1 --2- 172.21.15.16:0/2235470146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2644102700 0x7f264419cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:31.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:31.199+0000 7f264b420700 1 -- 172.21.15.16:0/2235470146 >> 172.21.15.16:0/2235470146 conn(0x7f26440fb650 msgr2=0x7f2644105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:31.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:31.199+0000 7f264b420700 1 -- 172.21.15.16:0/2235470146 shutdown_connections 2024-04-22T15:45:31.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:31.199+0000 7f264b420700 1 -- 172.21.15.16:0/2235470146 wait complete. 2024-04-22T15:45:31.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:31 smithi027 ceph-mon[34922]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:31.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:31 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/2235470146' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T15:45:31.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:31 smithi016 ceph-mon[26620]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:31.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:31 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/2235470146' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T15:45:32.001 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-22T15:45:32.001 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T15:45:32.219 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:45:33.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:33 smithi016 ceph-mon[26620]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:33.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:33 smithi027 ceph-mon[34922]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:33.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.892+0000 7fadd2283700 1 -- 172.21.15.16:0/3246274338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadcc0ff5a0 msgr2=0x7fadcc0ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:33.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.892+0000 7fadd2283700 1 --2- 172.21.15.16:0/3246274338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadcc0ff5a0 0x7fadcc0ff980 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fadbc009910 tx=0x7fadbc009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:33.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.892+0000 7fadd2283700 1 -- 172.21.15.16:0/3246274338 shutdown_connections 2024-04-22T15:45:33.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.892+0000 7fadd2283700 1 --2- 172.21.15.16:0/3246274338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadcc0fff50 0x7fadcc105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:33.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.892+0000 7fadd2283700 1 --2- 172.21.15.16:0/3246274338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadcc0ff5a0 0x7fadcc0ff980 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:33.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.892+0000 7fadd2283700 1 -- 172.21.15.16:0/3246274338 >> 172.21.15.16:0/3246274338 conn(0x7fadcc074e30 msgr2=0x7fadcc075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:33.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.893+0000 7fadd2283700 1 -- 172.21.15.16:0/3246274338 shutdown_connections 2024-04-22T15:45:33.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.893+0000 7fadd2283700 1 -- 172.21.15.16:0/3246274338 wait complete. 2024-04-22T15:45:33.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.893+0000 7fadd2283700 1 Processor -- start 2024-04-22T15:45:33.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.893+0000 7fadd2283700 1 -- start start 2024-04-22T15:45:33.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.893+0000 7fadd2283700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadcc0ff5a0 0x7fadcc19c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:33.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.893+0000 7fadd2283700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadcc0fff50 0x7fadcc19cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:33.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.893+0000 7fadd2283700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc19d250 con 0x7fadcc0fff50 2024-04-22T15:45:33.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.894+0000 7fadd2283700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc196700 con 0x7fadcc0ff5a0 2024-04-22T15:45:33.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.894+0000 7fadcb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadcc0fff50 0x7fadcc19cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:33.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.894+0000 7fadcbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadcc0ff5a0 0x7fadcc19c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:33.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.894+0000 7fadcb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadcc0fff50 0x7fadcc19cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37330/0 (socket says 172.21.15.16:37330) 2024-04-22T15:45:33.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.894+0000 7fadcbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadcc0ff5a0 0x7fadcc19c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:50140/0 (socket says 172.21.15.16:50140) 2024-04-22T15:45:33.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.894+0000 7fadcb7fe700 1 -- 172.21.15.16:0/1640416305 learned_addr learned my addr 172.21.15.16:0/1640416305 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:45:33.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.894+0000 7fadcb7fe700 1 -- 172.21.15.16:0/1640416305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadcc0ff5a0 msgr2=0x7fadcc19c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:33.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.894+0000 7fadcb7fe700 1 --2- 172.21.15.16:0/1640416305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadcc0ff5a0 0x7fadcc19c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:33.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.894+0000 7fadcb7fe700 1 -- 172.21.15.16:0/1640416305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadbc0095f0 con 0x7fadcc0fff50 2024-04-22T15:45:33.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.894+0000 7fadcb7fe700 1 --2- 172.21.15.16:0/1640416305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadcc0fff50 0x7fadcc19cbc0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fadc000d900 tx=0x7fadc000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:33.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.894+0000 7fadc97fa700 1 -- 172.21.15.16:0/1640416305 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadc0009900 con 0x7fadcc0fff50 2024-04-22T15:45:33.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.894+0000 7fadd2283700 1 -- 172.21.15.16:0/1640416305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadcc1969e0 con 0x7fadcc0fff50 2024-04-22T15:45:33.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.894+0000 7fadc97fa700 1 -- 172.21.15.16:0/1640416305 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fadc0010460 con 0x7fadcc0fff50 2024-04-22T15:45:33.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.894+0000 7fadc97fa700 1 -- 172.21.15.16:0/1640416305 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadc000f5f0 con 0x7fadcc0fff50 2024-04-22T15:45:33.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.894+0000 7fadd2283700 1 -- 172.21.15.16:0/1640416305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadcc196f30 con 0x7fadcc0fff50 2024-04-22T15:45:33.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.895+0000 7fadd2283700 1 -- 172.21.15.16:0/1640416305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadcc109270 con 0x7fadcc0fff50 2024-04-22T15:45:33.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.896+0000 7fadc97fa700 1 -- 172.21.15.16:0/1640416305 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fadc00105d0 con 0x7fadcc0fff50 2024-04-22T15:45:33.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.896+0000 7fadc97fa700 1 --2- 172.21.15.16:0/1640416305 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fadb406c4e0 0x7fadb406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:33.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.897+0000 7fadc97fa700 1 -- 172.21.15.16:0/1640416305 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fadc008b080 con 0x7fadcc0fff50 2024-04-22T15:45:33.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.897+0000 7fadcbfff700 1 --2- 172.21.15.16:0/1640416305 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fadb406c4e0 0x7fadb406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:33.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.897+0000 7fadcbfff700 1 --2- 172.21.15.16:0/1640416305 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fadb406c4e0 0x7fadb406e9a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fadbc006010 tx=0x7fadbc01a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:33.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:33.901+0000 7fadc97fa700 1 -- 172.21.15.16:0/1640416305 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fadc0059630 con 0x7fadcc0fff50 2024-04-22T15:45:34.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:34.100+0000 7fadd2283700 1 -- 172.21.15.16:0/1640416305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fadcc04f2e0 con 0x7fadcc0fff50 2024-04-22T15:45:34.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:34.101+0000 7fadc97fa700 1 -- 172.21.15.16:0/1640416305 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11289 (secure 0 0 0) 0x7fadc00591c0 con 0x7fadcc0fff50 2024-04-22T15:45:34.104 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:45:34.104 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":40,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","created":"2024-04-22T15:39:20.727181+0000","modified":"2024-04-22T15:45:12.502671+0000","last_up_change":"2024-04-22T15:45:11.497701+0000","last_in_change":"2024-04-22T15:44:52.054297+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T15:43:58.896563+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"67bd69ee-bd8c-4670-927a-f5fbcd92c9bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1210635569},{"type":"v1","addr":"172.21.15.16:6803","nonce":1210635569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1210635569},{"type":"v1","addr":"172.21.15.16:6805","nonce":1210635569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1210635569},{"type":"v1","addr":"172.21.15.16:6809","nonce":1210635569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1210635569},{"type":"v1","addr":"172.21.15.16:6807","nonce":1210635569}]},"public_addr":"172.21.15.16:6803/1210635569","cluster_addr":"172.21.15.16:6805/1210635569","heartbeat_back_addr":"172.21.15.16:6809/1210635569","heartbeat_front_addr":"172.21.15.16:6807/1210635569","state":["exists","up"]},{"osd":1,"uuid":"f51a9e4d-fb0c-4f62-9328-e23eea863213","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3974144580},{"type":"v1","addr":"172.21.15.16:6811","nonce":3974144580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3974144580},{"type":"v1","addr":"172.21.15.16:6813","nonce":3974144580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3974144580},{"type":"v1","addr":"172.21.15.16:6817","nonce":3974144580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3974144580},{"type":"v1","addr":"172.21.15.16:6815","nonce":3974144580}]},"public_addr":"172.21.15.16:6811/3974144580","cluster_addr":"172.21.15.16:6813/3974144580","heartbeat_back_addr":"172.21.15.16:6817/3974144580","heartbeat_front_addr":"172.21.15.16:6815/3974144580","state":["exists","up"]},{"osd":2,"uuid":"e59fdb03-e04e-4afd-a58e-0b58945d3765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2760833880},{"type":"v1","addr":"172.21.15.16:6819","nonce":2760833880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2760833880},{"type":"v1","addr":"172.21.15.16:6821","nonce":2760833880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2760833880},{"type":"v1","addr":"172.21.15.16:6825","nonce":2760833880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2760833880},{"type":"v1","addr":"172.21.15.16:6823","nonce":2760833880}]},"public_addr":"172.21.15.16:6819/2760833880","cluster_addr":"172.21.15.16:6821/2760833880","heartbeat_back_addr":"172.21.15.16:6825/2760833880","heartbeat_front_addr":"172.21.15.16:6823/2760833880","state":["exists","up"]},{"osd":3,"uuid":"fa25b413-d004-4455-8d7b-0ced7f880a7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":286252834},{"type":"v1","addr":"172.21.15.27:6801","nonce":286252834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":286252834},{"type":"v1","addr":"172.21.15.27:6803","nonce":286252834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":286252834},{"type":"v1","addr":"172.21.15.27:6807","nonce":286252834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":286252834},{"type":"v1","addr":"172.21.15.27:6805","nonce":286252834}]},"public_addr":"172.21.15.27:6801/286252834","cluster_addr":"172.21.15.27:6803/286252834","heartbeat_back_addr":"172.21.15.27:6807/286252834","heartbeat_front_addr":"172.21.15.27:6805/286252834","state":["exists","up"]},{"osd":4,"uuid":"6b1eda0d-1627-4c49-93b1-1a78896f660c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":691669816},{"type":"v1","addr":"172.21.15.27:6809","nonce":691669816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":691669816},{"type":"v1","addr":"172.21.15.27:6811","nonce":691669816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":691669816},{"type":"v1","addr":"172.21.15.27:6815","nonce":691669816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":691669816},{"type":"v1","addr":"172.21.15.27:6813","nonce":691669816}]},"public_addr":"172.21.15.27:6809/691669816","cluster_addr":"172.21.15.27:6811/691669816","heartbeat_back_addr":"172.21.15.27:6815/691669816","heartbeat_front_addr":"172.21.15.27:6813/691669816","state":["exists","up"]},{"osd":5,"uuid":"58cebe34-0821-458f-86b5-7d460829d994","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1237795056},{"type":"v1","addr":"172.21.15.27:6817","nonce":1237795056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1237795056},{"type":"v1","addr":"172.21.15.27:6819","nonce":1237795056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1237795056},{"type":"v1","addr":"172.21.15.27:6823","nonce":1237795056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1237795056},{"type":"v1","addr":"172.21.15.27:6821","nonce":1237795056}]},"public_addr":"172.21.15.27:6817/1237795056","cluster_addr":"172.21.15.27:6819/1237795056","heartbeat_back_addr":"172.21.15.27:6823/1237795056","heartbeat_front_addr":"172.21.15.27:6821/1237795056","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:42:58.192450+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:43:27.129522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:43:54.750854+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:44:19.111479+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:44:44.487429+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:45:09.605548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/690437556":"2024-04-23T15:41:40.376957+0000","172.21.15.16:6800/3972982132":"2024-04-23T15:41:40.376957+0000","172.21.15.16:0/2002585335":"2024-04-23T15:40:27.107187+0000","172.21.15.16:0/4021274304":"2024-04-23T15:41:40.376957+0000","172.21.15.16:0/4172787384":"2024-04-23T15:40:27.107187+0000","172.21.15.16:0/955466902":"2024-04-23T15:40:27.107187+0000","172.21.15.16:6801/1748504461":"2024-04-23T15:39:53.479948+0000","172.21.15.16:6801/3431522404":"2024-04-23T15:40:27.107187+0000","172.21.15.16:6800/1748504461":"2024-04-23T15:39:53.479948+0000","172.21.15.16:0/2570865319":"2024-04-23T15:39:53.479948+0000","172.21.15.16:0/2048283809":"2024-04-23T15:39:53.479948+0000","172.21.15.16:0/4252231167":"2024-04-23T15:41:40.376957+0000","172.21.15.16:0/2348469475":"2024-04-23T15:39:53.479948+0000","172.21.15.16:6801/3972982132":"2024-04-23T15:41:40.376957+0000","172.21.15.16:6800/3431522404":"2024-04-23T15:40:27.107187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T15:45:34.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:34.103+0000 7fadd2283700 1 -- 172.21.15.16:0/1640416305 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fadb406c4e0 msgr2=0x7fadb406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:34.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:34.103+0000 7fadd2283700 1 --2- 172.21.15.16:0/1640416305 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fadb406c4e0 0x7fadb406e9a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fadbc006010 tx=0x7fadbc01a040 comp rx=0 tx=0).stop 2024-04-22T15:45:34.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:34.103+0000 7fadd2283700 1 -- 172.21.15.16:0/1640416305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadcc0fff50 msgr2=0x7fadcc19cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:34.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:34.103+0000 7fadd2283700 1 --2- 172.21.15.16:0/1640416305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadcc0fff50 0x7fadcc19cbc0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fadc000d900 tx=0x7fadc000dc10 comp rx=0 tx=0).stop 2024-04-22T15:45:34.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:34.103+0000 7fadd2283700 1 -- 172.21.15.16:0/1640416305 shutdown_connections 2024-04-22T15:45:34.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:34.103+0000 7fadd2283700 1 --2- 172.21.15.16:0/1640416305 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fadb406c4e0 0x7fadb406e9a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:34.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:34.103+0000 7fadd2283700 1 --2- 172.21.15.16:0/1640416305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadcc0ff5a0 0x7fadcc19c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:34.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:34.103+0000 7fadd2283700 1 --2- 172.21.15.16:0/1640416305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadcc0fff50 0x7fadcc19cbc0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:34.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:34.103+0000 7fadd2283700 1 -- 172.21.15.16:0/1640416305 >> 172.21.15.16:0/1640416305 conn(0x7fadcc074e30 msgr2=0x7fadcc0fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:34.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:34.104+0000 7fadd2283700 1 -- 172.21.15.16:0/1640416305 shutdown_connections 2024-04-22T15:45:34.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:34.104+0000 7fadd2283700 1 -- 172.21.15.16:0/1640416305 wait complete. 2024-04-22T15:45:34.797 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:34 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/1640416305' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T15:45:34.820 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-22T15:45:34.820 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-22T15:45:34.821 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-22T15:45:34.821 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-22T15:45:34.822 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-22T15:45:34.822 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-22T15:45:34.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:34 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/1640416305' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T15:45:35.102 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:45:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:45:35.140 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:45:35.361 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:45:35.746 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:35 smithi016 ceph-mon[26620]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:35.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:35 smithi027 ceph-mon[34922]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:36.330 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:45:36.330 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:45:36.735 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:36 smithi016 ceph-mon[26620]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:37.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:36 smithi027 ceph-mon[34922]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:39.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:39 smithi027 ceph-mon[34922]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:39.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:39 smithi016 ceph-mon[26620]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:40.562 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:45:40.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:40 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:45:40.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.883+0000 7ff708f78700 1 -- 172.21.15.16:0/2192071945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff704106cd0 msgr2=0x7ff7041070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:40.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.883+0000 7ff708f78700 1 --2- 172.21.15.16:0/2192071945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff704106cd0 0x7ff7041070b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4009910 tx=0x7ff6f4009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:40.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.883+0000 7ff708f78700 1 -- 172.21.15.16:0/2192071945 shutdown_connections 2024-04-22T15:45:40.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.883+0000 7ff708f78700 1 --2- 172.21.15.16:0/2192071945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff704068580 0x7ff7040689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:40.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.883+0000 7ff708f78700 1 --2- 172.21.15.16:0/2192071945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff704106cd0 0x7ff7041070b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:40.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.883+0000 7ff708f78700 1 -- 172.21.15.16:0/2192071945 >> 172.21.15.16:0/2192071945 conn(0x7ff704075670 msgr2=0x7ff704075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:40.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.883+0000 7ff708f78700 1 -- 172.21.15.16:0/2192071945 shutdown_connections 2024-04-22T15:45:40.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.883+0000 7ff708f78700 1 -- 172.21.15.16:0/2192071945 wait complete. 2024-04-22T15:45:40.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.884+0000 7ff708f78700 1 Processor -- start 2024-04-22T15:45:40.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.884+0000 7ff708f78700 1 -- start start 2024-04-22T15:45:40.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.884+0000 7ff708f78700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff704068580 0x7ff7040ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:40.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.884+0000 7ff708f78700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff704106cd0 0x7ff704100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:40.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.884+0000 7ff708f78700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff704100770 con 0x7ff704068580 2024-04-22T15:45:40.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.884+0000 7ff708f78700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7041008b0 con 0x7ff704106cd0 2024-04-22T15:45:40.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.884+0000 7ff70259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff704068580 0x7ff7040ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:40.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.884+0000 7ff70259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff704068580 0x7ff7040ffcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37352/0 (socket says 172.21.15.16:37352) 2024-04-22T15:45:40.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.884+0000 7ff70259c700 1 -- 172.21.15.16:0/1498822569 learned_addr learned my addr 172.21.15.16:0/1498822569 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:45:40.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.885+0000 7ff70259c700 1 -- 172.21.15.16:0/1498822569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff704106cd0 msgr2=0x7ff704100230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:40.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.885+0000 7ff70259c700 1 --2- 172.21.15.16:0/1498822569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff704106cd0 0x7ff704100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:40.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.885+0000 7ff70259c700 1 -- 172.21.15.16:0/1498822569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f40095f0 con 0x7ff704068580 2024-04-22T15:45:40.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.885+0000 7ff70259c700 1 --2- 172.21.15.16:0/1498822569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff704068580 0x7ff7040ffcf0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4006010 tx=0x7ff6f400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:40.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.885+0000 7ff6fb7fe700 1 -- 172.21.15.16:0/1498822569 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6f401d070 con 0x7ff704068580 2024-04-22T15:45:40.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.885+0000 7ff6fb7fe700 1 -- 172.21.15.16:0/1498822569 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff6f400f460 con 0x7ff704068580 2024-04-22T15:45:40.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.885+0000 7ff6fb7fe700 1 -- 172.21.15.16:0/1498822569 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6f4017630 con 0x7ff704068580 2024-04-22T15:45:40.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.885+0000 7ff708f78700 1 -- 172.21.15.16:0/1498822569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff704100b30 con 0x7ff704068580 2024-04-22T15:45:40.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.885+0000 7ff708f78700 1 -- 172.21.15.16:0/1498822569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7041a2e10 con 0x7ff704068580 2024-04-22T15:45:40.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.886+0000 7ff708f78700 1 -- 172.21.15.16:0/1498822569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff70410e120 con 0x7ff704068580 2024-04-22T15:45:40.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.887+0000 7ff6fb7fe700 1 -- 172.21.15.16:0/1498822569 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7ff6f400f5d0 con 0x7ff704068580 2024-04-22T15:45:40.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.887+0000 7ff6fb7fe700 1 --2- 172.21.15.16:0/1498822569 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff6f006c4e0 0x7ff6f006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:40.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.887+0000 7ff6fb7fe700 1 -- 172.21.15.16:0/1498822569 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff6f408ca90 con 0x7ff704068580 2024-04-22T15:45:40.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.888+0000 7ff6fbfff700 1 --2- 172.21.15.16:0/1498822569 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff6f006c4e0 0x7ff6f006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:40.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.888+0000 7ff6fb7fe700 1 --2- 172.21.15.16:0/1498822569 >> [v2:172.21.15.16:6802/1210635569,v1:172.21.15.16:6803/1210635569] conn(0x7ff6f00721c0 0x7ff6f00745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:40.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.888+0000 7ff6fb7fe700 1 -- 172.21.15.16:0/1498822569 --> [v2:172.21.15.16:6802/1210635569,v1:172.21.15.16:6803/1210635569] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff6f0074c90 con 0x7ff6f00721c0 2024-04-22T15:45:40.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.888+0000 7ff6fb7fe700 1 -- 172.21.15.16:0/1498822569 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff6f40912a0 con 0x7ff704068580 2024-04-22T15:45:40.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.888+0000 7ff702d9d700 1 --2- 172.21.15.16:0/1498822569 >> [v2:172.21.15.16:6802/1210635569,v1:172.21.15.16:6803/1210635569] conn(0x7ff6f00721c0 0x7ff6f00745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:40.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.888+0000 7ff6fbfff700 1 --2- 172.21.15.16:0/1498822569 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff6f006c4e0 0x7ff6f006e9a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff7041014e0 tx=0x7ff6ec006d20 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:40.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.888+0000 7ff702d9d700 1 --2- 172.21.15.16:0/1498822569 >> [v2:172.21.15.16:6802/1210635569,v1:172.21.15.16:6803/1210635569] conn(0x7ff6f00721c0 0x7ff6f00745e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:40.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.889+0000 7ff6fb7fe700 1 -- 172.21.15.16:0/1498822569 <== osd.0 v2:172.21.15.16:6802/1210635569 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7ff6f0074c90 con 0x7ff6f00721c0 2024-04-22T15:45:40.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.904+0000 7ff708f78700 1 -- 172.21.15.16:0/1498822569 --> [v2:172.21.15.16:6802/1210635569,v1:172.21.15.16:6803/1210635569] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff70404f2e0 con 0x7ff6f00721c0 2024-04-22T15:45:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.905+0000 7ff6fb7fe700 1 -- 172.21.15.16:0/1498822569 <== osd.0 v2:172.21.15.16:6802/1210635569 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff70404f2e0 con 0x7ff6f00721c0 2024-04-22T15:45:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.905+0000 7ff708f78700 1 -- 172.21.15.16:0/1498822569 >> [v2:172.21.15.16:6802/1210635569,v1:172.21.15.16:6803/1210635569] conn(0x7ff6f00721c0 msgr2=0x7ff6f00745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.905+0000 7ff708f78700 1 --2- 172.21.15.16:0/1498822569 >> [v2:172.21.15.16:6802/1210635569,v1:172.21.15.16:6803/1210635569] conn(0x7ff6f00721c0 0x7ff6f00745e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.905+0000 7ff708f78700 1 -- 172.21.15.16:0/1498822569 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff6f006c4e0 msgr2=0x7ff6f006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.905+0000 7ff708f78700 1 --2- 172.21.15.16:0/1498822569 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff6f006c4e0 0x7ff6f006e9a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff7041014e0 tx=0x7ff6ec006d20 comp rx=0 tx=0).stop 2024-04-22T15:45:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.905+0000 7ff708f78700 1 -- 172.21.15.16:0/1498822569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff704068580 msgr2=0x7ff7040ffcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.905+0000 7ff708f78700 1 --2- 172.21.15.16:0/1498822569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff704068580 0x7ff7040ffcf0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4006010 tx=0x7ff6f400be20 comp rx=0 tx=0).stop 2024-04-22T15:45:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.905+0000 7ff708f78700 1 -- 172.21.15.16:0/1498822569 shutdown_connections 2024-04-22T15:45:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.905+0000 7ff708f78700 1 --2- 172.21.15.16:0/1498822569 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff6f006c4e0 0x7ff6f006e9a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.905+0000 7ff708f78700 1 --2- 172.21.15.16:0/1498822569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff704068580 0x7ff7040ffcf0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.905+0000 7ff708f78700 1 --2- 172.21.15.16:0/1498822569 >> [v2:172.21.15.16:6802/1210635569,v1:172.21.15.16:6803/1210635569] conn(0x7ff6f00721c0 0x7ff6f00745e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.905+0000 7ff708f78700 1 --2- 172.21.15.16:0/1498822569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff704106cd0 0x7ff704100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:40.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.905+0000 7ff708f78700 1 -- 172.21.15.16:0/1498822569 >> 172.21.15.16:0/1498822569 conn(0x7ff704075670 msgr2=0x7ff7040fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:40.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.905+0000 7ff708f78700 1 -- 172.21.15.16:0/1498822569 shutdown_connections 2024-04-22T15:45:40.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:40.905+0000 7ff708f78700 1 -- 172.21.15.16:0/1498822569 wait complete. 2024-04-22T15:45:41.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.351+0000 7f4e5f52e700 1 -- 172.21.15.16:0/2688514420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e580ffc20 msgr2=0x7f4e58100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:41.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.351+0000 7f4e5f52e700 1 --2- 172.21.15.16:0/2688514420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e580ffc20 0x7f4e58100080 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4e54009910 tx=0x7f4e54009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.351+0000 7f4e5f52e700 1 -- 172.21.15.16:0/2688514420 shutdown_connections 2024-04-22T15:45:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.351+0000 7f4e5f52e700 1 --2- 172.21.15.16:0/2688514420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e580ffc20 0x7f4e58100080 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.351+0000 7f4e5f52e700 1 --2- 172.21.15.16:0/2688514420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e58105bd0 0x7f4e58105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.351+0000 7f4e5f52e700 1 -- 172.21.15.16:0/2688514420 >> 172.21.15.16:0/2688514420 conn(0x7f4e58075670 msgr2=0x7f4e58075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.352+0000 7f4e5f52e700 1 -- 172.21.15.16:0/2688514420 shutdown_connections 2024-04-22T15:45:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.352+0000 7f4e5f52e700 1 -- 172.21.15.16:0/2688514420 wait complete. 2024-04-22T15:45:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.352+0000 7f4e5f52e700 1 Processor -- start 2024-04-22T15:45:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.352+0000 7f4e5f52e700 1 -- start start 2024-04-22T15:45:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.352+0000 7f4e5f52e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e580ffc20 0x7f4e5819e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.352+0000 7f4e5f52e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e58105bd0 0x7f4e5819ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.352+0000 7f4e5f52e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5819f3f0 con 0x7f4e58105bd0 2024-04-22T15:45:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.352+0000 7f4e5f52e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e581988a0 con 0x7f4e580ffc20 2024-04-22T15:45:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.352+0000 7f4e5cac9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e58105bd0 0x7f4e5819ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:41.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.352+0000 7f4e5cac9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e58105bd0 0x7f4e5819ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37372/0 (socket says 172.21.15.16:37372) 2024-04-22T15:45:41.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.352+0000 7f4e5cac9700 1 -- 172.21.15.16:0/988718077 learned_addr learned my addr 172.21.15.16:0/988718077 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:45:41.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.353+0000 7f4e5cac9700 1 -- 172.21.15.16:0/988718077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e580ffc20 msgr2=0x7f4e5819e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:41.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.353+0000 7f4e5d2ca700 1 --2- 172.21.15.16:0/988718077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e580ffc20 0x7f4e5819e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:41.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.353+0000 7f4e5cac9700 1 --2- 172.21.15.16:0/988718077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e580ffc20 0x7f4e5819e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:41.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.353+0000 7f4e5cac9700 1 -- 172.21.15.16:0/988718077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e540095f0 con 0x7f4e58105bd0 2024-04-22T15:45:41.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.353+0000 7f4e5cac9700 1 --2- 172.21.15.16:0/988718077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e58105bd0 0x7f4e5819ed60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4e54006010 tx=0x7f4e5400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:41.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.353+0000 7f4e4e7fc700 1 -- 172.21.15.16:0/988718077 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e5401d070 con 0x7f4e58105bd0 2024-04-22T15:45:41.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.353+0000 7f4e4e7fc700 1 -- 172.21.15.16:0/988718077 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4e5400f460 con 0x7f4e58105bd0 2024-04-22T15:45:41.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.353+0000 7f4e5f52e700 1 -- 172.21.15.16:0/988718077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e58198b20 con 0x7f4e58105bd0 2024-04-22T15:45:41.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.353+0000 7f4e4e7fc700 1 -- 172.21.15.16:0/988718077 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e54021640 con 0x7f4e58105bd0 2024-04-22T15:45:41.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.353+0000 7f4e5f52e700 1 -- 172.21.15.16:0/988718077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e58199010 con 0x7f4e58105bd0 2024-04-22T15:45:41.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.354+0000 7f4e5f52e700 1 -- 172.21.15.16:0/988718077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4e5810cfa0 con 0x7f4e58105bd0 2024-04-22T15:45:41.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.355+0000 7f4e4e7fc700 1 -- 172.21.15.16:0/988718077 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f4e5400faf0 con 0x7f4e58105bd0 2024-04-22T15:45:41.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.355+0000 7f4e4e7fc700 1 --2- 172.21.15.16:0/988718077 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4e4406c4e0 0x7f4e4406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:41.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.355+0000 7f4e4e7fc700 1 -- 172.21.15.16:0/988718077 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f4e5408c790 con 0x7f4e58105bd0 2024-04-22T15:45:41.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.355+0000 7f4e5d2ca700 1 --2- 172.21.15.16:0/988718077 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4e4406c4e0 0x7f4e4406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:41.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.355+0000 7f4e4e7fc700 1 --2- 172.21.15.16:0/988718077 >> [v2:172.21.15.16:6810/3974144580,v1:172.21.15.16:6811/3974144580] conn(0x7f4e440721c0 0x7f4e440745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:41.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.355+0000 7f4e4e7fc700 1 -- 172.21.15.16:0/988718077 --> [v2:172.21.15.16:6810/3974144580,v1:172.21.15.16:6811/3974144580] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e44074c90 con 0x7f4e440721c0 2024-04-22T15:45:41.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.355+0000 7f4e4e7fc700 1 -- 172.21.15.16:0/988718077 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e540912a0 con 0x7f4e58105bd0 2024-04-22T15:45:41.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.355+0000 7f4e5dacb700 1 --2- 172.21.15.16:0/988718077 >> [v2:172.21.15.16:6810/3974144580,v1:172.21.15.16:6811/3974144580] conn(0x7f4e440721c0 0x7f4e440745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:41.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.356+0000 7f4e5d2ca700 1 --2- 172.21.15.16:0/988718077 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4e4406c4e0 0x7f4e4406e9a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4e48005950 tx=0x7f4e48005cf0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:41.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.356+0000 7f4e5dacb700 1 --2- 172.21.15.16:0/988718077 >> [v2:172.21.15.16:6810/3974144580,v1:172.21.15.16:6811/3974144580] conn(0x7f4e440721c0 0x7f4e440745e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:41.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.356+0000 7f4e4e7fc700 1 -- 172.21.15.16:0/988718077 <== osd.1 v2:172.21.15.16:6810/3974144580 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f4e44074c90 con 0x7f4e440721c0 2024-04-22T15:45:41.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.372+0000 7f4e5f52e700 1 -- 172.21.15.16:0/988718077 --> [v2:172.21.15.16:6810/3974144580,v1:172.21.15.16:6811/3974144580] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4e58199470 con 0x7f4e440721c0 2024-04-22T15:45:41.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.372+0000 7f4e4e7fc700 1 -- 172.21.15.16:0/988718077 <== osd.1 v2:172.21.15.16:6810/3974144580 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4e58199470 con 0x7f4e440721c0 2024-04-22T15:45:41.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.373+0000 7f4e5f52e700 1 -- 172.21.15.16:0/988718077 >> [v2:172.21.15.16:6810/3974144580,v1:172.21.15.16:6811/3974144580] conn(0x7f4e440721c0 msgr2=0x7f4e440745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:41.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.373+0000 7f4e5f52e700 1 --2- 172.21.15.16:0/988718077 >> [v2:172.21.15.16:6810/3974144580,v1:172.21.15.16:6811/3974144580] conn(0x7f4e440721c0 0x7f4e440745e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:41.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.373+0000 7f4e5f52e700 1 -- 172.21.15.16:0/988718077 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4e4406c4e0 msgr2=0x7f4e4406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:41.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.373+0000 7f4e5f52e700 1 --2- 172.21.15.16:0/988718077 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4e4406c4e0 0x7f4e4406e9a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4e48005950 tx=0x7f4e48005cf0 comp rx=0 tx=0).stop 2024-04-22T15:45:41.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.373+0000 7f4e5f52e700 1 -- 172.21.15.16:0/988718077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e58105bd0 msgr2=0x7f4e5819ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:41.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.373+0000 7f4e5f52e700 1 --2- 172.21.15.16:0/988718077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e58105bd0 0x7f4e5819ed60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4e54006010 tx=0x7f4e5400be20 comp rx=0 tx=0).stop 2024-04-22T15:45:41.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.373+0000 7f4e5f52e700 1 -- 172.21.15.16:0/988718077 shutdown_connections 2024-04-22T15:45:41.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.373+0000 7f4e5f52e700 1 --2- 172.21.15.16:0/988718077 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4e4406c4e0 0x7f4e4406e9a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:41.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.373+0000 7f4e5f52e700 1 --2- 172.21.15.16:0/988718077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e580ffc20 0x7f4e5819e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:41.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.373+0000 7f4e5f52e700 1 --2- 172.21.15.16:0/988718077 >> [v2:172.21.15.16:6810/3974144580,v1:172.21.15.16:6811/3974144580] conn(0x7f4e440721c0 0x7f4e440745e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:41.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.373+0000 7f4e5f52e700 1 --2- 172.21.15.16:0/988718077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e58105bd0 0x7f4e5819ed60 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:41.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.373+0000 7f4e5f52e700 1 -- 172.21.15.16:0/988718077 >> 172.21.15.16:0/988718077 conn(0x7f4e58075670 msgr2=0x7f4e58068740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:41.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.373+0000 7f4e5f52e700 1 -- 172.21.15.16:0/988718077 shutdown_connections 2024-04-22T15:45:41.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:41.373+0000 7f4e5f52e700 1 -- 172.21.15.16:0/988718077 wait complete. 2024-04-22T15:45:41.797 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:41 smithi016 ceph-mon[26620]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:41.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:41 smithi027 ceph-mon[34922]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:42.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.733+0000 7f8f84b92700 1 -- 172.21.15.16:0/374404537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f80102720 msgr2=0x7f8f8010ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:42.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.733+0000 7f8f84b92700 1 --2- 172.21.15.16:0/374404537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f80102720 0x7f8f8010ac10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8f68009910 tx=0x7f8f68009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:42.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.733+0000 7f8f84b92700 1 -- 172.21.15.16:0/374404537 shutdown_connections 2024-04-22T15:45:42.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.733+0000 7f8f84b92700 1 --2- 172.21.15.16:0/374404537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f80102720 0x7f8f8010ac10 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:42.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.733+0000 7f8f84b92700 1 --2- 172.21.15.16:0/374404537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f80101e00 0x7f8f801021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:42.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.733+0000 7f8f84b92700 1 -- 172.21.15.16:0/374404537 >> 172.21.15.16:0/374404537 conn(0x7f8f800fb650 msgr2=0x7f8f800fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:42.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.733+0000 7f8f84b92700 1 -- 172.21.15.16:0/374404537 shutdown_connections 2024-04-22T15:45:42.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.733+0000 7f8f84b92700 1 -- 172.21.15.16:0/374404537 wait complete. 2024-04-22T15:45:42.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.734+0000 7f8f84b92700 1 Processor -- start 2024-04-22T15:45:42.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.734+0000 7f8f84b92700 1 -- start start 2024-04-22T15:45:42.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.734+0000 7f8f84b92700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f80101e00 0x7f8f80196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:42.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.734+0000 7f8f84b92700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f80102720 0x7f8f80196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:42.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.734+0000 7f8f84b92700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f80197530 con 0x7f8f80101e00 2024-04-22T15:45:42.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.734+0000 7f8f84b92700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8019b340 con 0x7f8f80102720 2024-04-22T15:45:42.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.734+0000 7f8f7e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f80101e00 0x7f8f80196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:42.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.734+0000 7f8f7e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f80101e00 0x7f8f80196960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58386/0 (socket says 172.21.15.16:58386) 2024-04-22T15:45:42.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.734+0000 7f8f7e59c700 1 -- 172.21.15.16:0/547853182 learned_addr learned my addr 172.21.15.16:0/547853182 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:45:42.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.734+0000 7f8f7dd9b700 1 --2- 172.21.15.16:0/547853182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f80102720 0x7f8f80196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:42.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.734+0000 7f8f7e59c700 1 -- 172.21.15.16:0/547853182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f80102720 msgr2=0x7f8f80196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:42.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.734+0000 7f8f7e59c700 1 --2- 172.21.15.16:0/547853182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f80102720 0x7f8f80196ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:42.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.734+0000 7f8f7e59c700 1 -- 172.21.15.16:0/547853182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f680095f0 con 0x7f8f80101e00 2024-04-22T15:45:42.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.734+0000 7f8f7dd9b700 1 --2- 172.21.15.16:0/547853182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f80102720 0x7f8f80196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:45:42.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.734+0000 7f8f7e59c700 1 --2- 172.21.15.16:0/547853182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f80101e00 0x7f8f80196960 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8f7000ba50 tx=0x7f8f7000be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:42.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.735+0000 7f8f777fe700 1 -- 172.21.15.16:0/547853182 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f7000ca00 con 0x7f8f80101e00 2024-04-22T15:45:42.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.735+0000 7f8f777fe700 1 -- 172.21.15.16:0/547853182 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8f7000cb60 con 0x7f8f80101e00 2024-04-22T15:45:42.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.735+0000 7f8f84b92700 1 -- 172.21.15.16:0/547853182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f8019b620 con 0x7f8f80101e00 2024-04-22T15:45:42.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.735+0000 7f8f777fe700 1 -- 172.21.15.16:0/547853182 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f70007580 con 0x7f8f80101e00 2024-04-22T15:45:42.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.735+0000 7f8f84b92700 1 -- 172.21.15.16:0/547853182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f8019bb40 con 0x7f8f80101e00 2024-04-22T15:45:42.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.735+0000 7f8f84b92700 1 -- 172.21.15.16:0/547853182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8f8010e150 con 0x7f8f80101e00 2024-04-22T15:45:42.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.736+0000 7f8f777fe700 1 -- 172.21.15.16:0/547853182 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f8f7000ccd0 con 0x7f8f80101e00 2024-04-22T15:45:42.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.736+0000 7f8f777fe700 1 --2- 172.21.15.16:0/547853182 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f8f6c06c4e0 0x7f8f6c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:42.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.737+0000 7f8f777fe700 1 -- 172.21.15.16:0/547853182 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f8f70006e40 con 0x7f8f80101e00 2024-04-22T15:45:42.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.737+0000 7f8f7dd9b700 1 --2- 172.21.15.16:0/547853182 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f8f6c06c4e0 0x7f8f6c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:42.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.737+0000 7f8f777fe700 1 --2- 172.21.15.16:0/547853182 >> [v2:172.21.15.27:6800/286252834,v1:172.21.15.27:6801/286252834] conn(0x7f8f6c0721c0 0x7f8f6c0745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:42.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.737+0000 7f8f777fe700 1 -- 172.21.15.16:0/547853182 --> [v2:172.21.15.27:6800/286252834,v1:172.21.15.27:6801/286252834] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8f6c074c90 con 0x7f8f6c0721c0 2024-04-22T15:45:42.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.737+0000 7f8f777fe700 1 -- 172.21.15.16:0/547853182 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f7008f2a0 con 0x7f8f80101e00 2024-04-22T15:45:42.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.737+0000 7f8f7ed9d700 1 --2- 172.21.15.16:0/547853182 >> [v2:172.21.15.27:6800/286252834,v1:172.21.15.27:6801/286252834] conn(0x7f8f6c0721c0 0x7f8f6c0745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:42.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.737+0000 7f8f7dd9b700 1 --2- 172.21.15.16:0/547853182 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f8f6c06c4e0 0x7f8f6c06e9a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8f68006010 tx=0x7f8f6801a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:42.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.738+0000 7f8f7ed9d700 1 --2- 172.21.15.16:0/547853182 >> [v2:172.21.15.27:6800/286252834,v1:172.21.15.27:6801/286252834] conn(0x7f8f6c0721c0 0x7f8f6c0745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:42.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.738+0000 7f8f777fe700 1 -- 172.21.15.16:0/547853182 <== osd.3 v2:172.21.15.27:6800/286252834 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f8f6c074c90 con 0x7f8f6c0721c0 2024-04-22T15:45:42.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.754+0000 7f8f84b92700 1 -- 172.21.15.16:0/547853182 --> [v2:172.21.15.27:6800/286252834,v1:172.21.15.27:6801/286252834] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8f8004f2e0 con 0x7f8f6c0721c0 2024-04-22T15:45:42.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.754+0000 7f8f777fe700 1 -- 172.21.15.16:0/547853182 <== osd.3 v2:172.21.15.27:6800/286252834 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8f8004f2e0 con 0x7f8f6c0721c0 2024-04-22T15:45:42.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.755+0000 7f8f84b92700 1 -- 172.21.15.16:0/547853182 >> [v2:172.21.15.27:6800/286252834,v1:172.21.15.27:6801/286252834] conn(0x7f8f6c0721c0 msgr2=0x7f8f6c0745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:42.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.755+0000 7f8f84b92700 1 --2- 172.21.15.16:0/547853182 >> [v2:172.21.15.27:6800/286252834,v1:172.21.15.27:6801/286252834] conn(0x7f8f6c0721c0 0x7f8f6c0745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:42.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.755+0000 7f8f84b92700 1 -- 172.21.15.16:0/547853182 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f8f6c06c4e0 msgr2=0x7f8f6c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:42.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.755+0000 7f8f84b92700 1 --2- 172.21.15.16:0/547853182 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f8f6c06c4e0 0x7f8f6c06e9a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8f68006010 tx=0x7f8f6801a040 comp rx=0 tx=0).stop 2024-04-22T15:45:42.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.755+0000 7f8f84b92700 1 -- 172.21.15.16:0/547853182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f80101e00 msgr2=0x7f8f80196960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:42.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.755+0000 7f8f84b92700 1 --2- 172.21.15.16:0/547853182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f80101e00 0x7f8f80196960 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8f7000ba50 tx=0x7f8f7000be10 comp rx=0 tx=0).stop 2024-04-22T15:45:42.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.755+0000 7f8f84b92700 1 -- 172.21.15.16:0/547853182 shutdown_connections 2024-04-22T15:45:42.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.755+0000 7f8f84b92700 1 --2- 172.21.15.16:0/547853182 >> [v2:172.21.15.27:6800/286252834,v1:172.21.15.27:6801/286252834] conn(0x7f8f6c0721c0 0x7f8f6c0745e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:42.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.755+0000 7f8f84b92700 1 --2- 172.21.15.16:0/547853182 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f8f6c06c4e0 0x7f8f6c06e9a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:42.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.755+0000 7f8f84b92700 1 --2- 172.21.15.16:0/547853182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f80101e00 0x7f8f80196960 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:42.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.755+0000 7f8f84b92700 1 --2- 172.21.15.16:0/547853182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f80102720 0x7f8f80196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:42.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.755+0000 7f8f84b92700 1 -- 172.21.15.16:0/547853182 >> 172.21.15.16:0/547853182 conn(0x7f8f800fb650 msgr2=0x7f8f80105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:42.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.755+0000 7f8f84b92700 1 -- 172.21.15.16:0/547853182 shutdown_connections 2024-04-22T15:45:42.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:42.755+0000 7f8f84b92700 1 -- 172.21.15.16:0/547853182 wait complete. 2024-04-22T15:45:43.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:43 smithi016 ceph-mon[26620]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:43.799 INFO:teuthology.orchestra.run.smithi016.stdout:111669149715 2024-04-22T15:45:43.799 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-22T15:45:43.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:43 smithi027 ceph-mon[34922]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:44.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.026+0000 7f3ba2456700 1 -- 172.21.15.16:0/3073525932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b9c101850 msgr2=0x7f3b9c1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:44.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.026+0000 7f3ba2456700 1 --2- 172.21.15.16:0/3073525932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b9c101850 0x7f3b9c1058a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3b90009910 tx=0x7f3b90009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:44.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.027+0000 7f3ba2456700 1 -- 172.21.15.16:0/3073525932 shutdown_connections 2024-04-22T15:45:44.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.027+0000 7f3ba2456700 1 --2- 172.21.15.16:0/3073525932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b9c101850 0x7f3b9c1058a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:44.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.027+0000 7f3ba2456700 1 --2- 172.21.15.16:0/3073525932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b9c100ea0 0x7f3b9c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:44.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.027+0000 7f3ba2456700 1 -- 172.21.15.16:0/3073525932 >> 172.21.15.16:0/3073525932 conn(0x7f3b9c0fc750 msgr2=0x7f3b9c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:44.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.027+0000 7f3ba2456700 1 -- 172.21.15.16:0/3073525932 shutdown_connections 2024-04-22T15:45:44.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.027+0000 7f3ba2456700 1 -- 172.21.15.16:0/3073525932 wait complete. 2024-04-22T15:45:44.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.027+0000 7f3ba2456700 1 Processor -- start 2024-04-22T15:45:44.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.027+0000 7f3ba2456700 1 -- start start 2024-04-22T15:45:44.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.028+0000 7f3ba2456700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b9c100ea0 0x7f3b9c19c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:44.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.028+0000 7f3ba2456700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b9c101850 0x7f3b9c19cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:44.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.028+0000 7f3b9bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b9c100ea0 0x7f3b9c19c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:44.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.028+0000 7f3b9bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b9c100ea0 0x7f3b9c19c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58400/0 (socket says 172.21.15.16:58400) 2024-04-22T15:45:44.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.029+0000 7f3b9bfff700 1 -- 172.21.15.16:0/1791887277 learned_addr learned my addr 172.21.15.16:0/1791887277 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:45:44.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.029+0000 7f3ba2456700 1 -- 172.21.15.16:0/1791887277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c19d210 con 0x7f3b9c100ea0 2024-04-22T15:45:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.029+0000 7f3ba2456700 1 -- 172.21.15.16:0/1791887277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c1966c0 con 0x7f3b9c101850 2024-04-22T15:45:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.029+0000 7f3b9b7fe700 1 --2- 172.21.15.16:0/1791887277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b9c101850 0x7f3b9c19cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.029+0000 7f3b9bfff700 1 -- 172.21.15.16:0/1791887277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b9c101850 msgr2=0x7f3b9c19cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.029+0000 7f3b9bfff700 1 --2- 172.21.15.16:0/1791887277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b9c101850 0x7f3b9c19cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.029+0000 7f3b9bfff700 1 -- 172.21.15.16:0/1791887277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b900095f0 con 0x7f3b9c100ea0 2024-04-22T15:45:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.029+0000 7f3b9b7fe700 1 --2- 172.21.15.16:0/1791887277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b9c101850 0x7f3b9c19cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:45:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.030+0000 7f3b9bfff700 1 --2- 172.21.15.16:0/1791887277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b9c100ea0 0x7f3b9c19c640 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c00e920 tx=0x7f3b8c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.030+0000 7f3b997fa700 1 -- 172.21.15.16:0/1791887277 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b8c0139e0 con 0x7f3b9c100ea0 2024-04-22T15:45:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.030+0000 7f3b997fa700 1 -- 172.21.15.16:0/1791887277 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3b8c00c7c0 con 0x7f3b9c100ea0 2024-04-22T15:45:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.030+0000 7f3ba2456700 1 -- 172.21.15.16:0/1791887277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b9c196940 con 0x7f3b9c100ea0 2024-04-22T15:45:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.030+0000 7f3b997fa700 1 -- 172.21.15.16:0/1791887277 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b8c01b9a0 con 0x7f3b9c100ea0 2024-04-22T15:45:44.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.030+0000 7f3ba2456700 1 -- 172.21.15.16:0/1791887277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b9c196e60 con 0x7f3b9c100ea0 2024-04-22T15:45:44.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.031+0000 7f3ba2456700 1 -- 172.21.15.16:0/1791887277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3b88000ff0 con 0x7f3b9c100ea0 2024-04-22T15:45:44.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.032+0000 7f3b997fa700 1 -- 172.21.15.16:0/1791887277 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f3b8c013b40 con 0x7f3b9c100ea0 2024-04-22T15:45:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.033+0000 7f3b997fa700 1 --2- 172.21.15.16:0/1791887277 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f3b8406c4e0 0x7f3b8406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.033+0000 7f3b997fa700 1 -- 172.21.15.16:0/1791887277 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f3b8c08f850 con 0x7f3b9c100ea0 2024-04-22T15:45:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.033+0000 7f3b9b7fe700 1 --2- 172.21.15.16:0/1791887277 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f3b8406c4e0 0x7f3b8406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.033+0000 7f3b997fa700 1 --2- 172.21.15.16:0/1791887277 >> [v2:172.21.15.27:6816/1237795056,v1:172.21.15.27:6817/1237795056] conn(0x7f3b840721c0 0x7f3b840745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.033+0000 7f3b997fa700 1 -- 172.21.15.16:0/1791887277 --> [v2:172.21.15.27:6816/1237795056,v1:172.21.15.27:6817/1237795056] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3b84074c90 con 0x7f3b840721c0 2024-04-22T15:45:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.033+0000 7f3b997fa700 1 -- 172.21.15.16:0/1791887277 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b8c08fcb0 con 0x7f3b9c100ea0 2024-04-22T15:45:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.033+0000 7f3b9b7fe700 1 --2- 172.21.15.16:0/1791887277 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f3b8406c4e0 0x7f3b8406e9a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3b900059d0 tx=0x7f3b90005940 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.033+0000 7f3ba09f3700 1 --2- 172.21.15.16:0/1791887277 >> [v2:172.21.15.27:6816/1237795056,v1:172.21.15.27:6817/1237795056] conn(0x7f3b840721c0 0x7f3b840745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.034+0000 7f3ba09f3700 1 --2- 172.21.15.16:0/1791887277 >> [v2:172.21.15.27:6816/1237795056,v1:172.21.15.27:6817/1237795056] conn(0x7f3b840721c0 0x7f3b840745e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.035+0000 7f3b997fa700 1 -- 172.21.15.16:0/1791887277 <== osd.5 v2:172.21.15.27:6816/1237795056 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f3b84074c90 con 0x7f3b840721c0 2024-04-22T15:45:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.050+0000 7f3ba2456700 1 -- 172.21.15.16:0/1791887277 --> [v2:172.21.15.27:6816/1237795056,v1:172.21.15.27:6817/1237795056] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3b88002ce0 con 0x7f3b840721c0 2024-04-22T15:45:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.051+0000 7f3b997fa700 1 -- 172.21.15.16:0/1791887277 <== osd.5 v2:172.21.15.27:6816/1237795056 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f3b88002ce0 con 0x7f3b840721c0 2024-04-22T15:45:44.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.051+0000 7f3ba2456700 1 -- 172.21.15.16:0/1791887277 >> [v2:172.21.15.27:6816/1237795056,v1:172.21.15.27:6817/1237795056] conn(0x7f3b840721c0 msgr2=0x7f3b840745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:44.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.051+0000 7f3ba2456700 1 --2- 172.21.15.16:0/1791887277 >> [v2:172.21.15.27:6816/1237795056,v1:172.21.15.27:6817/1237795056] conn(0x7f3b840721c0 0x7f3b840745e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:44.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.051+0000 7f3ba2456700 1 -- 172.21.15.16:0/1791887277 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f3b8406c4e0 msgr2=0x7f3b8406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:44.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.051+0000 7f3ba2456700 1 --2- 172.21.15.16:0/1791887277 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f3b8406c4e0 0x7f3b8406e9a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3b900059d0 tx=0x7f3b90005940 comp rx=0 tx=0).stop 2024-04-22T15:45:44.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.051+0000 7f3ba2456700 1 -- 172.21.15.16:0/1791887277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b9c100ea0 msgr2=0x7f3b9c19c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:44.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.051+0000 7f3ba2456700 1 --2- 172.21.15.16:0/1791887277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b9c100ea0 0x7f3b9c19c640 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c00e920 tx=0x7f3b8c00ec30 comp rx=0 tx=0).stop 2024-04-22T15:45:44.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.051+0000 7f3ba2456700 1 -- 172.21.15.16:0/1791887277 shutdown_connections 2024-04-22T15:45:44.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.051+0000 7f3ba2456700 1 --2- 172.21.15.16:0/1791887277 >> [v2:172.21.15.27:6816/1237795056,v1:172.21.15.27:6817/1237795056] conn(0x7f3b840721c0 0x7f3b840745e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:44.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.051+0000 7f3ba2456700 1 --2- 172.21.15.16:0/1791887277 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f3b8406c4e0 0x7f3b8406e9a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:44.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.051+0000 7f3ba2456700 1 --2- 172.21.15.16:0/1791887277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b9c100ea0 0x7f3b9c19c640 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:44.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.051+0000 7f3ba2456700 1 --2- 172.21.15.16:0/1791887277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b9c101850 0x7f3b9c19cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:44.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.051+0000 7f3ba2456700 1 -- 172.21.15.16:0/1791887277 >> 172.21.15.16:0/1791887277 conn(0x7f3b9c0fc750 msgr2=0x7f3b9c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:44.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.051+0000 7f3ba2456700 1 -- 172.21.15.16:0/1791887277 shutdown_connections 2024-04-22T15:45:44.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:44.051+0000 7f3ba2456700 1 -- 172.21.15.16:0/1791887277 wait complete. 2024-04-22T15:45:44.569 INFO:teuthology.orchestra.run.smithi016.stdout:38654705699 2024-04-22T15:45:44.570 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-22T15:45:45.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.204+0000 7f31d4bd8700 1 -- 172.21.15.16:0/35953049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d0105bd0 msgr2=0x7f31d0105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:45.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.204+0000 7f31d4bd8700 1 --2- 172.21.15.16:0/35953049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d0105bd0 0x7f31d0105fb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f31c0009910 tx=0x7f31c0009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:45.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.204+0000 7f31d4bd8700 1 -- 172.21.15.16:0/35953049 shutdown_connections 2024-04-22T15:45:45.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.204+0000 7f31d4bd8700 1 --2- 172.21.15.16:0/35953049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d00ffc20 0x7f31d0100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:45.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.204+0000 7f31d4bd8700 1 --2- 172.21.15.16:0/35953049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d0105bd0 0x7f31d0105fb0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:45.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.204+0000 7f31d4bd8700 1 -- 172.21.15.16:0/35953049 >> 172.21.15.16:0/35953049 conn(0x7f31d0075670 msgr2=0x7f31d0075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:45.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.204+0000 7f31d4bd8700 1 -- 172.21.15.16:0/35953049 shutdown_connections 2024-04-22T15:45:45.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.204+0000 7f31d4bd8700 1 -- 172.21.15.16:0/35953049 wait complete. 2024-04-22T15:45:45.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.205+0000 7f31d4bd8700 1 Processor -- start 2024-04-22T15:45:45.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.205+0000 7f31d4bd8700 1 -- start start 2024-04-22T15:45:45.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.205+0000 7f31d4bd8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d00ffc20 0x7f31d0195510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:45.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.205+0000 7f31d4bd8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d0105bd0 0x7f31d0195a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:45.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.205+0000 7f31d4bd8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d0195f90 con 0x7f31d0105bd0 2024-04-22T15:45:45.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.205+0000 7f31d4bd8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d0196100 con 0x7f31d00ffc20 2024-04-22T15:45:45.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.205+0000 7f31c7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d0105bd0 0x7f31d0195a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:45.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.205+0000 7f31c7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d0105bd0 0x7f31d0195a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58420/0 (socket says 172.21.15.16:58420) 2024-04-22T15:45:45.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.205+0000 7f31c7fff700 1 -- 172.21.15.16:0/1374189580 learned_addr learned my addr 172.21.15.16:0/1374189580 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:45:45.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.206+0000 7f31ce59c700 1 --2- 172.21.15.16:0/1374189580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d00ffc20 0x7f31d0195510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:45.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.206+0000 7f31c7fff700 1 -- 172.21.15.16:0/1374189580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d00ffc20 msgr2=0x7f31d0195510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:45.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.206+0000 7f31c7fff700 1 --2- 172.21.15.16:0/1374189580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d00ffc20 0x7f31d0195510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:45.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.206+0000 7f31c7fff700 1 -- 172.21.15.16:0/1374189580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c00095f0 con 0x7f31d0105bd0 2024-04-22T15:45:45.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.206+0000 7f31c7fff700 1 --2- 172.21.15.16:0/1374189580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d0105bd0 0x7f31d0195a50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f31b800d900 tx=0x7f31b800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:45.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.206+0000 7f31c77fe700 1 -- 172.21.15.16:0/1374189580 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31b8004a00 con 0x7f31d0105bd0 2024-04-22T15:45:45.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.206+0000 7f31c77fe700 1 -- 172.21.15.16:0/1374189580 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f31b80054d0 con 0x7f31d0105bd0 2024-04-22T15:45:45.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.206+0000 7f31d4bd8700 1 -- 172.21.15.16:0/1374189580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d01a2b80 con 0x7f31d0105bd0 2024-04-22T15:45:45.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.206+0000 7f31c77fe700 1 -- 172.21.15.16:0/1374189580 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31b8009d60 con 0x7f31d0105bd0 2024-04-22T15:45:45.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.206+0000 7f31d4bd8700 1 -- 172.21.15.16:0/1374189580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d01a2ee0 con 0x7f31d0105bd0 2024-04-22T15:45:45.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.207+0000 7f31d4bd8700 1 -- 172.21.15.16:0/1374189580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f31d010cf80 con 0x7f31d0105bd0 2024-04-22T15:45:45.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.207+0000 7f31c77fe700 1 -- 172.21.15.16:0/1374189580 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f31b8005020 con 0x7f31d0105bd0 2024-04-22T15:45:45.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.208+0000 7f31c77fe700 1 --2- 172.21.15.16:0/1374189580 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f31bc06c4e0 0x7f31bc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:45.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.208+0000 7f31c77fe700 1 -- 172.21.15.16:0/1374189580 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f31b808b5f0 con 0x7f31d0105bd0 2024-04-22T15:45:45.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.208+0000 7f31ce59c700 1 --2- 172.21.15.16:0/1374189580 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f31bc06c4e0 0x7f31bc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:45.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.208+0000 7f31c77fe700 1 --2- 172.21.15.16:0/1374189580 >> [v2:172.21.15.16:6818/2760833880,v1:172.21.15.16:6819/2760833880] conn(0x7f31bc0721c0 0x7f31bc0745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:45.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.208+0000 7f31c77fe700 1 -- 172.21.15.16:0/1374189580 --> [v2:172.21.15.16:6818/2760833880,v1:172.21.15.16:6819/2760833880] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f31bc074c90 con 0x7f31bc0721c0 2024-04-22T15:45:45.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.208+0000 7f31c77fe700 1 -- 172.21.15.16:0/1374189580 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f31b808dcc0 con 0x7f31d0105bd0 2024-04-22T15:45:45.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.208+0000 7f31ced9d700 1 --2- 172.21.15.16:0/1374189580 >> [v2:172.21.15.16:6818/2760833880,v1:172.21.15.16:6819/2760833880] conn(0x7f31bc0721c0 0x7f31bc0745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:45.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.208+0000 7f31ce59c700 1 --2- 172.21.15.16:0/1374189580 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f31bc06c4e0 0x7f31bc06e9a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f31c000b5c0 tx=0x7f31c0005960 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:45.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.209+0000 7f31ced9d700 1 --2- 172.21.15.16:0/1374189580 >> [v2:172.21.15.16:6818/2760833880,v1:172.21.15.16:6819/2760833880] conn(0x7f31bc0721c0 0x7f31bc0745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:45.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.209+0000 7f31c77fe700 1 -- 172.21.15.16:0/1374189580 <== osd.2 v2:172.21.15.16:6818/2760833880 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f31bc074c90 con 0x7f31bc0721c0 2024-04-22T15:45:45.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.225+0000 7f31d4bd8700 1 -- 172.21.15.16:0/1374189580 --> [v2:172.21.15.16:6818/2760833880,v1:172.21.15.16:6819/2760833880] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f31d004f2e0 con 0x7f31bc0721c0 2024-04-22T15:45:45.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.225+0000 7f31c77fe700 1 -- 172.21.15.16:0/1374189580 <== osd.2 v2:172.21.15.16:6818/2760833880 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f31d004f2e0 con 0x7f31bc0721c0 2024-04-22T15:45:45.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.225+0000 7f31d4bd8700 1 -- 172.21.15.16:0/1374189580 >> [v2:172.21.15.16:6818/2760833880,v1:172.21.15.16:6819/2760833880] conn(0x7f31bc0721c0 msgr2=0x7f31bc0745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:45.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.225+0000 7f31d4bd8700 1 --2- 172.21.15.16:0/1374189580 >> [v2:172.21.15.16:6818/2760833880,v1:172.21.15.16:6819/2760833880] conn(0x7f31bc0721c0 0x7f31bc0745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:45.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.226+0000 7f31d4bd8700 1 -- 172.21.15.16:0/1374189580 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f31bc06c4e0 msgr2=0x7f31bc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:45.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.226+0000 7f31d4bd8700 1 --2- 172.21.15.16:0/1374189580 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f31bc06c4e0 0x7f31bc06e9a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f31c000b5c0 tx=0x7f31c0005960 comp rx=0 tx=0).stop 2024-04-22T15:45:45.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.226+0000 7f31d4bd8700 1 -- 172.21.15.16:0/1374189580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d0105bd0 msgr2=0x7f31d0195a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:45.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.226+0000 7f31d4bd8700 1 --2- 172.21.15.16:0/1374189580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d0105bd0 0x7f31d0195a50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f31b800d900 tx=0x7f31b800dcc0 comp rx=0 tx=0).stop 2024-04-22T15:45:45.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.226+0000 7f31d4bd8700 1 -- 172.21.15.16:0/1374189580 shutdown_connections 2024-04-22T15:45:45.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.226+0000 7f31d4bd8700 1 --2- 172.21.15.16:0/1374189580 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f31bc06c4e0 0x7f31bc06e9a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:45.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.226+0000 7f31d4bd8700 1 --2- 172.21.15.16:0/1374189580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d00ffc20 0x7f31d0195510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:45.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.226+0000 7f31d4bd8700 1 --2- 172.21.15.16:0/1374189580 >> [v2:172.21.15.16:6818/2760833880,v1:172.21.15.16:6819/2760833880] conn(0x7f31bc0721c0 0x7f31bc0745e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:45.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.226+0000 7f31d4bd8700 1 --2- 172.21.15.16:0/1374189580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d0105bd0 0x7f31d0195a50 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:45.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.226+0000 7f31d4bd8700 1 -- 172.21.15.16:0/1374189580 >> 172.21.15.16:0/1374189580 conn(0x7f31d0075670 msgr2=0x7f31d00fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:45.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.226+0000 7f31d4bd8700 1 -- 172.21.15.16:0/1374189580 shutdown_connections 2024-04-22T15:45:45.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:45.226+0000 7f31d4bd8700 1 -- 172.21.15.16:0/1374189580 wait complete. 2024-04-22T15:45:45.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:45 smithi027 ceph-mon[34922]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:45.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:45 smithi016 ceph-mon[26620]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:45.981 INFO:teuthology.orchestra.run.smithi016.stdout:60129542173 2024-04-22T15:45:45.981 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-22T15:45:46.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.173+0000 7f36f3ac4700 1 -- 172.21.15.16:0/3912811863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36ec0ff5a0 msgr2=0x7f36ec0ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:46.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.173+0000 7f36f3ac4700 1 --2- 172.21.15.16:0/3912811863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36ec0ff5a0 0x7f36ec0ff980 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f36e8009910 tx=0x7f36e8009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:46.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.173+0000 7f36f3ac4700 1 -- 172.21.15.16:0/3912811863 shutdown_connections 2024-04-22T15:45:46.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.173+0000 7f36f3ac4700 1 --2- 172.21.15.16:0/3912811863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36ec0fff50 0x7f36ec105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:46.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.173+0000 7f36f3ac4700 1 --2- 172.21.15.16:0/3912811863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36ec0ff5a0 0x7f36ec0ff980 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:46.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.173+0000 7f36f3ac4700 1 -- 172.21.15.16:0/3912811863 >> 172.21.15.16:0/3912811863 conn(0x7f36ec074e30 msgr2=0x7f36ec075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:46.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.173+0000 7f36f3ac4700 1 -- 172.21.15.16:0/3912811863 shutdown_connections 2024-04-22T15:45:46.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.173+0000 7f36f3ac4700 1 -- 172.21.15.16:0/3912811863 wait complete. 2024-04-22T15:45:46.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.173+0000 7f36f3ac4700 1 Processor -- start 2024-04-22T15:45:46.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.173+0000 7f36f3ac4700 1 -- start start 2024-04-22T15:45:46.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.174+0000 7f36f3ac4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36ec0ff5a0 0x7f36ec100520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:46.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.174+0000 7f36f3ac4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36ec0fff50 0x7f36ec100a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:46.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.174+0000 7f36f3ac4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec100fa0 con 0x7f36ec0ff5a0 2024-04-22T15:45:46.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.174+0000 7f36f3ac4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec1010e0 con 0x7f36ec0fff50 2024-04-22T15:45:46.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.174+0000 7f36f1860700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36ec0ff5a0 0x7f36ec100520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:46.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.174+0000 7f36f1860700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36ec0ff5a0 0x7f36ec100520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58446/0 (socket says 172.21.15.16:58446) 2024-04-22T15:45:46.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.174+0000 7f36f1860700 1 -- 172.21.15.16:0/1077897746 learned_addr learned my addr 172.21.15.16:0/1077897746 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:45:46.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.174+0000 7f36f105f700 1 --2- 172.21.15.16:0/1077897746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36ec0fff50 0x7f36ec100a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:46.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.174+0000 7f36f1860700 1 -- 172.21.15.16:0/1077897746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36ec0fff50 msgr2=0x7f36ec100a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:46.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.174+0000 7f36f1860700 1 --2- 172.21.15.16:0/1077897746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36ec0fff50 0x7f36ec100a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:46.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.174+0000 7f36f1860700 1 -- 172.21.15.16:0/1077897746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e80095f0 con 0x7f36ec0ff5a0 2024-04-22T15:45:46.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.174+0000 7f36f1860700 1 --2- 172.21.15.16:0/1077897746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36ec0ff5a0 0x7f36ec100520 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f36e8006010 tx=0x7f36e800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:46.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.174+0000 7f36e2ffd700 1 -- 172.21.15.16:0/1077897746 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36e801d070 con 0x7f36ec0ff5a0 2024-04-22T15:45:46.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.174+0000 7f36e2ffd700 1 -- 172.21.15.16:0/1077897746 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f36e8004fa0 con 0x7f36ec0ff5a0 2024-04-22T15:45:46.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.174+0000 7f36f3ac4700 1 -- 172.21.15.16:0/1077897746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36ec101360 con 0x7f36ec0ff5a0 2024-04-22T15:45:46.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.175+0000 7f36e2ffd700 1 -- 172.21.15.16:0/1077897746 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36e800f690 con 0x7f36ec0ff5a0 2024-04-22T15:45:46.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.175+0000 7f36f3ac4700 1 -- 172.21.15.16:0/1077897746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36ec1a2e30 con 0x7f36ec0ff5a0 2024-04-22T15:45:46.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.175+0000 7f36f3ac4700 1 -- 172.21.15.16:0/1077897746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f36ec10e0a0 con 0x7f36ec0ff5a0 2024-04-22T15:45:46.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.176+0000 7f36e2ffd700 1 -- 172.21.15.16:0/1077897746 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f36e8022470 con 0x7f36ec0ff5a0 2024-04-22T15:45:46.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.176+0000 7f36e2ffd700 1 --2- 172.21.15.16:0/1077897746 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f36d806c4e0 0x7f36d806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:46.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.177+0000 7f36e2ffd700 1 -- 172.21.15.16:0/1077897746 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f36e808cbf0 con 0x7f36ec0ff5a0 2024-04-22T15:45:46.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.177+0000 7f36f105f700 1 --2- 172.21.15.16:0/1077897746 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f36d806c4e0 0x7f36d806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:46.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.177+0000 7f36e2ffd700 1 --2- 172.21.15.16:0/1077897746 >> [v2:172.21.15.27:6808/691669816,v1:172.21.15.27:6809/691669816] conn(0x7f36d80721c0 0x7f36d80745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:46.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.177+0000 7f36e2ffd700 1 -- 172.21.15.16:0/1077897746 --> [v2:172.21.15.27:6808/691669816,v1:172.21.15.27:6809/691669816] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36d8074c90 con 0x7f36d80721c0 2024-04-22T15:45:46.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.177+0000 7f36e2ffd700 1 -- 172.21.15.16:0/1077897746 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36e80922d0 con 0x7f36ec0ff5a0 2024-04-22T15:45:46.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.177+0000 7f36f105f700 1 --2- 172.21.15.16:0/1077897746 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f36d806c4e0 0x7f36d806e9a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f36ec101d10 tx=0x7f36dc00a300 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:46.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.177+0000 7f36f2061700 1 --2- 172.21.15.16:0/1077897746 >> [v2:172.21.15.27:6808/691669816,v1:172.21.15.27:6809/691669816] conn(0x7f36d80721c0 0x7f36d80745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:46.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.178+0000 7f36f2061700 1 --2- 172.21.15.16:0/1077897746 >> [v2:172.21.15.27:6808/691669816,v1:172.21.15.27:6809/691669816] conn(0x7f36d80721c0 0x7f36d80745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:46.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.178+0000 7f36e2ffd700 1 -- 172.21.15.16:0/1077897746 <== osd.4 v2:172.21.15.27:6808/691669816 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f36d8074c90 con 0x7f36d80721c0 2024-04-22T15:45:46.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.194+0000 7f36f3ac4700 1 -- 172.21.15.16:0/1077897746 --> [v2:172.21.15.27:6808/691669816,v1:172.21.15.27:6809/691669816] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f36ec04f2e0 con 0x7f36d80721c0 2024-04-22T15:45:46.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.195+0000 7f36e2ffd700 1 -- 172.21.15.16:0/1077897746 <== osd.4 v2:172.21.15.27:6808/691669816 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f36ec04f2e0 con 0x7f36d80721c0 2024-04-22T15:45:46.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.195+0000 7f36f3ac4700 1 -- 172.21.15.16:0/1077897746 >> [v2:172.21.15.27:6808/691669816,v1:172.21.15.27:6809/691669816] conn(0x7f36d80721c0 msgr2=0x7f36d80745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:46.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.195+0000 7f36f3ac4700 1 --2- 172.21.15.16:0/1077897746 >> [v2:172.21.15.27:6808/691669816,v1:172.21.15.27:6809/691669816] conn(0x7f36d80721c0 0x7f36d80745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:46.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.195+0000 7f36f3ac4700 1 -- 172.21.15.16:0/1077897746 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f36d806c4e0 msgr2=0x7f36d806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:46.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.195+0000 7f36f3ac4700 1 --2- 172.21.15.16:0/1077897746 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f36d806c4e0 0x7f36d806e9a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f36ec101d10 tx=0x7f36dc00a300 comp rx=0 tx=0).stop 2024-04-22T15:45:46.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.195+0000 7f36f3ac4700 1 -- 172.21.15.16:0/1077897746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36ec0ff5a0 msgr2=0x7f36ec100520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:46.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.195+0000 7f36f3ac4700 1 --2- 172.21.15.16:0/1077897746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36ec0ff5a0 0x7f36ec100520 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f36e8006010 tx=0x7f36e800bd70 comp rx=0 tx=0).stop 2024-04-22T15:45:46.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.195+0000 7f36f3ac4700 1 -- 172.21.15.16:0/1077897746 shutdown_connections 2024-04-22T15:45:46.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.195+0000 7f36f3ac4700 1 --2- 172.21.15.16:0/1077897746 >> [v2:172.21.15.27:6808/691669816,v1:172.21.15.27:6809/691669816] conn(0x7f36d80721c0 0x7f36d80745e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:46.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.195+0000 7f36f3ac4700 1 --2- 172.21.15.16:0/1077897746 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f36d806c4e0 0x7f36d806e9a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:46.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.195+0000 7f36f3ac4700 1 --2- 172.21.15.16:0/1077897746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36ec0ff5a0 0x7f36ec100520 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:46.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.195+0000 7f36f3ac4700 1 --2- 172.21.15.16:0/1077897746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36ec0fff50 0x7f36ec100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:46.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.195+0000 7f36f3ac4700 1 -- 172.21.15.16:0/1077897746 >> 172.21.15.16:0/1077897746 conn(0x7f36ec074e30 msgr2=0x7f36ec0fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:46.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.195+0000 7f36f3ac4700 1 -- 172.21.15.16:0/1077897746 shutdown_connections 2024-04-22T15:45:46.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:46.195+0000 7f36f3ac4700 1 -- 172.21.15.16:0/1077897746 wait complete. 2024-04-22T15:45:46.718 INFO:teuthology.orchestra.run.smithi016.stdout:81604378649 2024-04-22T15:45:46.718 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-22T15:45:46.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:46 smithi016 ceph-mon[26620]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:47.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:46 smithi027 ceph-mon[34922]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:47.292 INFO:teuthology.orchestra.run.smithi016.stdout:167503724553 2024-04-22T15:45:47.293 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-22T15:45:47.714 INFO:teuthology.orchestra.run.smithi016.stdout:146028888078 2024-04-22T15:45:47.715 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-22T15:45:47.750 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:45:47.753 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:45:47.754 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:45:47.758 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:45:49.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:49 smithi027 ceph-mon[34922]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:49.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:49 smithi016 ceph-mon[26620]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:50.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.487+0000 7fbcf7713700 1 -- 172.21.15.16:0/2381570572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcf00ffbf0 msgr2=0x7fbcf00fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:50.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.487+0000 7fbcf7713700 1 --2- 172.21.15.16:0/2381570572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcf00ffbf0 0x7fbcf00fffd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbcec009910 tx=0x7fbcec009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:50.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.487+0000 7fbcf7713700 1 -- 172.21.15.16:0/2381570572 shutdown_connections 2024-04-22T15:45:50.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.487+0000 7fbcf7713700 1 --2- 172.21.15.16:0/2381570572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbcf0100510 0x7fbcf01049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:50.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.487+0000 7fbcf7713700 1 --2- 172.21.15.16:0/2381570572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcf00ffbf0 0x7fbcf00fffd0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:50.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.487+0000 7fbcf7713700 1 -- 172.21.15.16:0/2381570572 >> 172.21.15.16:0/2381570572 conn(0x7fbcf00fb650 msgr2=0x7fbcf00fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:50.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.487+0000 7fbcf7713700 1 -- 172.21.15.16:0/2381570572 shutdown_connections 2024-04-22T15:45:50.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.487+0000 7fbcf7713700 1 -- 172.21.15.16:0/2381570572 wait complete. 2024-04-22T15:45:50.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.488+0000 7fbcf7713700 1 Processor -- start 2024-04-22T15:45:50.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.488+0000 7fbcf7713700 1 -- start start 2024-04-22T15:45:50.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.488+0000 7fbcf7713700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcf00ffbf0 0x7fbcf019cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:50.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.488+0000 7fbcf7713700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbcf0100510 0x7fbcf019d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:50.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.488+0000 7fbcf7713700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf019db80 con 0x7fbcf00ffbf0 2024-04-22T15:45:50.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.488+0000 7fbcf7713700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf01a1910 con 0x7fbcf0100510 2024-04-22T15:45:50.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.488+0000 7fbcf54af700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcf00ffbf0 0x7fbcf019cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:50.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.488+0000 7fbcf54af700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcf00ffbf0 0x7fbcf019cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58470/0 (socket says 172.21.15.16:58470) 2024-04-22T15:45:50.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.488+0000 7fbcf54af700 1 -- 172.21.15.16:0/2750980628 learned_addr learned my addr 172.21.15.16:0/2750980628 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:45:50.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.488+0000 7fbcf4cae700 1 --2- 172.21.15.16:0/2750980628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbcf0100510 0x7fbcf019d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:50.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.488+0000 7fbcf54af700 1 -- 172.21.15.16:0/2750980628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbcf0100510 msgr2=0x7fbcf019d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:50.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.488+0000 7fbcf54af700 1 --2- 172.21.15.16:0/2750980628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbcf0100510 0x7fbcf019d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:50.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.488+0000 7fbcf54af700 1 -- 172.21.15.16:0/2750980628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcec0095f0 con 0x7fbcf00ffbf0 2024-04-22T15:45:50.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.488+0000 7fbcf4cae700 1 --2- 172.21.15.16:0/2750980628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbcf0100510 0x7fbcf019d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:45:50.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.488+0000 7fbcf54af700 1 --2- 172.21.15.16:0/2750980628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcf00ffbf0 0x7fbcf019cf60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbcec006010 tx=0x7fbcec00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:50.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.489+0000 7fbce67fc700 1 -- 172.21.15.16:0/2750980628 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcec01d070 con 0x7fbcf00ffbf0 2024-04-22T15:45:50.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.489+0000 7fbce67fc700 1 -- 172.21.15.16:0/2750980628 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbcec00f460 con 0x7fbcf00ffbf0 2024-04-22T15:45:50.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.489+0000 7fbcf7713700 1 -- 172.21.15.16:0/2750980628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcf01a1b90 con 0x7fbcf00ffbf0 2024-04-22T15:45:50.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.489+0000 7fbce67fc700 1 -- 172.21.15.16:0/2750980628 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcec021640 con 0x7fbcf00ffbf0 2024-04-22T15:45:50.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.489+0000 7fbcf7713700 1 -- 172.21.15.16:0/2750980628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcf01a2080 con 0x7fbcf00ffbf0 2024-04-22T15:45:50.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.490+0000 7fbcf7713700 1 -- 172.21.15.16:0/2750980628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf0102090 con 0x7fbcf00ffbf0 2024-04-22T15:45:50.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.490+0000 7fbce67fc700 1 -- 172.21.15.16:0/2750980628 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fbcec00faf0 con 0x7fbcf00ffbf0 2024-04-22T15:45:50.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.491+0000 7fbce67fc700 1 --2- 172.21.15.16:0/2750980628 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fbcdc06c4e0 0x7fbcdc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:50.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.491+0000 7fbce67fc700 1 -- 172.21.15.16:0/2750980628 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fbcec08c6e0 con 0x7fbcf00ffbf0 2024-04-22T15:45:50.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.491+0000 7fbcf4cae700 1 --2- 172.21.15.16:0/2750980628 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fbcdc06c4e0 0x7fbcdc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:50.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.491+0000 7fbcf4cae700 1 --2- 172.21.15.16:0/2750980628 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fbcdc06c4e0 0x7fbcdc06e9a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbcf019e580 tx=0x7fbce0005f50 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:50.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.495+0000 7fbce67fc700 1 -- 172.21.15.16:0/2750980628 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fbcec057280 con 0x7fbcf00ffbf0 2024-04-22T15:45:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.702+0000 7fbcf7713700 1 -- 172.21.15.16:0/2750980628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fbcf004f2e0 con 0x7fbcf00ffbf0 2024-04-22T15:45:50.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.703+0000 7fbce67fc700 1 -- 172.21.15.16:0/2750980628 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbcec05a8a0 con 0x7fbcf00ffbf0 2024-04-22T15:45:50.706 INFO:teuthology.orchestra.run.smithi016.stdout:38654705701 2024-04-22T15:45:50.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.705+0000 7fbcf7713700 1 -- 172.21.15.16:0/2750980628 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fbcdc06c4e0 msgr2=0x7fbcdc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:50.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.705+0000 7fbcf7713700 1 --2- 172.21.15.16:0/2750980628 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fbcdc06c4e0 0x7fbcdc06e9a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbcf019e580 tx=0x7fbce0005f50 comp rx=0 tx=0).stop 2024-04-22T15:45:50.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.705+0000 7fbcf7713700 1 -- 172.21.15.16:0/2750980628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcf00ffbf0 msgr2=0x7fbcf019cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:50.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.705+0000 7fbcf7713700 1 --2- 172.21.15.16:0/2750980628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcf00ffbf0 0x7fbcf019cf60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbcec006010 tx=0x7fbcec00be20 comp rx=0 tx=0).stop 2024-04-22T15:45:50.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.705+0000 7fbcf7713700 1 -- 172.21.15.16:0/2750980628 shutdown_connections 2024-04-22T15:45:50.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.705+0000 7fbcf7713700 1 --2- 172.21.15.16:0/2750980628 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fbcdc06c4e0 0x7fbcdc06e9a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:50.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.705+0000 7fbcf7713700 1 --2- 172.21.15.16:0/2750980628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcf00ffbf0 0x7fbcf019cf60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:50.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.705+0000 7fbcf7713700 1 --2- 172.21.15.16:0/2750980628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbcf0100510 0x7fbcf019d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:50.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.705+0000 7fbcf7713700 1 -- 172.21.15.16:0/2750980628 >> 172.21.15.16:0/2750980628 conn(0x7fbcf00fb650 msgr2=0x7fbcf0111060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:50.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.705+0000 7fbcf7713700 1 -- 172.21.15.16:0/2750980628 shutdown_connections 2024-04-22T15:45:50.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:50.705+0000 7fbcf7713700 1 -- 172.21.15.16:0/2750980628 wait complete. 2024-04-22T15:45:51.716 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705699 got 38654705701 for osd.0 2024-04-22T15:45:51.716 DEBUG:teuthology.parallel:result is None 2024-04-22T15:45:51.748 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:51 smithi016 ceph-mon[26620]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:51.748 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:51 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/2750980628' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T15:45:51.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:51 smithi027 ceph-mon[34922]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:51.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:51 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/2750980628' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T15:45:53.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.200+0000 7ff9dfde8700 1 -- 172.21.15.16:0/2558713442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d8101de0 msgr2=0x7ff9d81021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:53.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.200+0000 7ff9dfde8700 1 --2- 172.21.15.16:0/2558713442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d8101de0 0x7ff9d81021c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0009910 tx=0x7ff9d0009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.200+0000 7ff9dfde8700 1 -- 172.21.15.16:0/2558713442 shutdown_connections 2024-04-22T15:45:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.200+0000 7ff9dfde8700 1 --2- 172.21.15.16:0/2558713442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9d8102700 0x7ff9d810abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.200+0000 7ff9dfde8700 1 --2- 172.21.15.16:0/2558713442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d8101de0 0x7ff9d81021c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.200+0000 7ff9dfde8700 1 -- 172.21.15.16:0/2558713442 >> 172.21.15.16:0/2558713442 conn(0x7ff9d80fb650 msgr2=0x7ff9d80fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.200+0000 7ff9dfde8700 1 -- 172.21.15.16:0/2558713442 shutdown_connections 2024-04-22T15:45:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.200+0000 7ff9dfde8700 1 -- 172.21.15.16:0/2558713442 wait complete. 2024-04-22T15:45:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.201+0000 7ff9dfde8700 1 Processor -- start 2024-04-22T15:45:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.201+0000 7ff9dfde8700 1 -- start start 2024-04-22T15:45:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.201+0000 7ff9dfde8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d8101de0 0x7ff9d819e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.201+0000 7ff9dfde8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9d8102700 0x7ff9d819edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:53.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.201+0000 7ff9dfde8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d819f3d0 con 0x7ff9d8101de0 2024-04-22T15:45:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.201+0000 7ff9dfde8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d8198910 con 0x7ff9d8102700 2024-04-22T15:45:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.201+0000 7ff9ddb84700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d8101de0 0x7ff9d819e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.201+0000 7ff9ddb84700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d8101de0 0x7ff9d819e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39056/0 (socket says 172.21.15.16:39056) 2024-04-22T15:45:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.201+0000 7ff9ddb84700 1 -- 172.21.15.16:0/2509759750 learned_addr learned my addr 172.21.15.16:0/2509759750 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:45:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.201+0000 7ff9dd383700 1 --2- 172.21.15.16:0/2509759750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9d8102700 0x7ff9d819edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.202+0000 7ff9ddb84700 1 -- 172.21.15.16:0/2509759750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9d8102700 msgr2=0x7ff9d819edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.202+0000 7ff9ddb84700 1 --2- 172.21.15.16:0/2509759750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9d8102700 0x7ff9d819edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.202+0000 7ff9ddb84700 1 -- 172.21.15.16:0/2509759750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d00095f0 con 0x7ff9d8101de0 2024-04-22T15:45:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.202+0000 7ff9ddb84700 1 --2- 172.21.15.16:0/2509759750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d8101de0 0x7ff9d819e890 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0006010 tx=0x7ff9d000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.202+0000 7ff9d6ffd700 1 -- 172.21.15.16:0/2509759750 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9d001d070 con 0x7ff9d8101de0 2024-04-22T15:45:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.202+0000 7ff9dfde8700 1 -- 172.21.15.16:0/2509759750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d8198b90 con 0x7ff9d8101de0 2024-04-22T15:45:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.202+0000 7ff9d6ffd700 1 -- 172.21.15.16:0/2509759750 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff9d0004fa0 con 0x7ff9d8101de0 2024-04-22T15:45:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.202+0000 7ff9d6ffd700 1 -- 172.21.15.16:0/2509759750 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9d000f690 con 0x7ff9d8101de0 2024-04-22T15:45:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.202+0000 7ff9dfde8700 1 -- 172.21.15.16:0/2509759750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d8199080 con 0x7ff9d8101de0 2024-04-22T15:45:53.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.203+0000 7ff9dfde8700 1 -- 172.21.15.16:0/2509759750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d810a6f0 con 0x7ff9d8101de0 2024-04-22T15:45:53.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.204+0000 7ff9d6ffd700 1 -- 172.21.15.16:0/2509759750 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7ff9d0022470 con 0x7ff9d8101de0 2024-04-22T15:45:53.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.204+0000 7ff9d6ffd700 1 --2- 172.21.15.16:0/2509759750 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff9c006c4e0 0x7ff9c006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:53.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.204+0000 7ff9d6ffd700 1 -- 172.21.15.16:0/2509759750 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff9d008cc20 con 0x7ff9d8101de0 2024-04-22T15:45:53.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.204+0000 7ff9dd383700 1 --2- 172.21.15.16:0/2509759750 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff9c006c4e0 0x7ff9c006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:53.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.205+0000 7ff9dd383700 1 --2- 172.21.15.16:0/2509759750 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff9c006c4e0 0x7ff9c006e9a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff9d80fce80 tx=0x7ff9cc00a2d0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:53.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.209+0000 7ff9d6ffd700 1 -- 172.21.15.16:0/2509759750 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff9d005b1d0 con 0x7ff9d8101de0 2024-04-22T15:45:53.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.334+0000 7f9bde0cd700 1 -- 172.21.15.16:0/2060671380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd8106cd0 msgr2=0x7f9bd81070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:53.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.334+0000 7f9bde0cd700 1 --2- 172.21.15.16:0/2060671380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd8106cd0 0x7f9bd81070b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8009910 tx=0x7f9bc8009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:53.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.334+0000 7f9bde0cd700 1 -- 172.21.15.16:0/2060671380 shutdown_connections 2024-04-22T15:45:53.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.334+0000 7f9bde0cd700 1 --2- 172.21.15.16:0/2060671380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd8068580 0x7f9bd80689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:53.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.334+0000 7f9bde0cd700 1 --2- 172.21.15.16:0/2060671380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd8106cd0 0x7f9bd81070b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:53.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.334+0000 7f9bde0cd700 1 -- 172.21.15.16:0/2060671380 >> 172.21.15.16:0/2060671380 conn(0x7f9bd8075670 msgr2=0x7f9bd8075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:53.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.334+0000 7f9bde0cd700 1 -- 172.21.15.16:0/2060671380 shutdown_connections 2024-04-22T15:45:53.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.334+0000 7f9bde0cd700 1 -- 172.21.15.16:0/2060671380 wait complete. 2024-04-22T15:45:53.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.334+0000 7f9bde0cd700 1 Processor -- start 2024-04-22T15:45:53.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.335+0000 7f9bde0cd700 1 -- start start 2024-04-22T15:45:53.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.335+0000 7f9bde0cd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd8068580 0x7f9bd80ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:53.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.335+0000 7f9bde0cd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd8106cd0 0x7f9bd8100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:53.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.335+0000 7f9bde0cd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd8100770 con 0x7f9bd8068580 2024-04-22T15:45:53.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.335+0000 7f9bde0cd700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd81008b0 con 0x7f9bd8106cd0 2024-04-22T15:45:53.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.335+0000 7f9bd77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd8068580 0x7f9bd80ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:53.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.335+0000 7f9bd77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd8068580 0x7f9bd80ffcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39066/0 (socket says 172.21.15.16:39066) 2024-04-22T15:45:53.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.335+0000 7f9bd77fe700 1 -- 172.21.15.16:0/1586175812 learned_addr learned my addr 172.21.15.16:0/1586175812 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:45:53.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.335+0000 7f9bd6ffd700 1 --2- 172.21.15.16:0/1586175812 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd8106cd0 0x7f9bd8100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:53.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.335+0000 7f9bd77fe700 1 -- 172.21.15.16:0/1586175812 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd8106cd0 msgr2=0x7f9bd8100230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:53.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.335+0000 7f9bd77fe700 1 --2- 172.21.15.16:0/1586175812 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd8106cd0 0x7f9bd8100230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:53.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.335+0000 7f9bd77fe700 1 -- 172.21.15.16:0/1586175812 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc80095f0 con 0x7f9bd8068580 2024-04-22T15:45:53.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.336+0000 7f9bd77fe700 1 --2- 172.21.15.16:0/1586175812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd8068580 0x7f9bd80ffcf0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9bc800b5c0 tx=0x7f9bc800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:53.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.336+0000 7f9bd4ff9700 1 -- 172.21.15.16:0/1586175812 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bc801d070 con 0x7f9bd8068580 2024-04-22T15:45:53.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.336+0000 7f9bd4ff9700 1 -- 172.21.15.16:0/1586175812 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9bc8005e00 con 0x7f9bd8068580 2024-04-22T15:45:53.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.336+0000 7f9bd4ff9700 1 -- 172.21.15.16:0/1586175812 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bc8017400 con 0x7f9bd8068580 2024-04-22T15:45:53.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.336+0000 7f9bde0cd700 1 -- 172.21.15.16:0/1586175812 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd8100b30 con 0x7f9bd8068580 2024-04-22T15:45:53.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.336+0000 7f9bde0cd700 1 -- 172.21.15.16:0/1586175812 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd81a2d90 con 0x7f9bd8068580 2024-04-22T15:45:53.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.337+0000 7f9bde0cd700 1 -- 172.21.15.16:0/1586175812 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd81091f0 con 0x7f9bd8068580 2024-04-22T15:45:53.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.337+0000 7f9bd4ff9700 1 -- 172.21.15.16:0/1586175812 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f9bc800f460 con 0x7f9bd8068580 2024-04-22T15:45:53.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.338+0000 7f9bd4ff9700 1 --2- 172.21.15.16:0/1586175812 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f9bc406c4e0 0x7f9bc406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:53.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.338+0000 7f9bd4ff9700 1 -- 172.21.15.16:0/1586175812 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f9bc808c9e0 con 0x7f9bd8068580 2024-04-22T15:45:53.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.338+0000 7f9bd6ffd700 1 --2- 172.21.15.16:0/1586175812 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f9bc406c4e0 0x7f9bc406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:53.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.338+0000 7f9bd6ffd700 1 --2- 172.21.15.16:0/1586175812 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f9bc406c4e0 0x7f9bc406e9a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9bd81014e0 tx=0x7f9bc00073d0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.343+0000 7f9bd4ff9700 1 -- 172.21.15.16:0/1586175812 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9bc8057500 con 0x7f9bd8068580 2024-04-22T15:45:53.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.430+0000 7ff9dfde8700 1 -- 172.21.15.16:0/2509759750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ff9d8199e20 con 0x7ff9d8101de0 2024-04-22T15:45:53.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.430+0000 7ff9d6ffd700 1 -- 172.21.15.16:0/2509759750 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff9d0027020 con 0x7ff9d8101de0 2024-04-22T15:45:53.434 INFO:teuthology.orchestra.run.smithi016.stdout:111669149717 2024-04-22T15:45:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.432+0000 7ff9dfde8700 1 -- 172.21.15.16:0/2509759750 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff9c006c4e0 msgr2=0x7ff9c006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.432+0000 7ff9dfde8700 1 --2- 172.21.15.16:0/2509759750 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff9c006c4e0 0x7ff9c006e9a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff9d80fce80 tx=0x7ff9cc00a2d0 comp rx=0 tx=0).stop 2024-04-22T15:45:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.432+0000 7ff9dfde8700 1 -- 172.21.15.16:0/2509759750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d8101de0 msgr2=0x7ff9d819e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.432+0000 7ff9dfde8700 1 --2- 172.21.15.16:0/2509759750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d8101de0 0x7ff9d819e890 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0006010 tx=0x7ff9d000bd70 comp rx=0 tx=0).stop 2024-04-22T15:45:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.432+0000 7ff9dfde8700 1 -- 172.21.15.16:0/2509759750 shutdown_connections 2024-04-22T15:45:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.432+0000 7ff9dfde8700 1 --2- 172.21.15.16:0/2509759750 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff9c006c4e0 0x7ff9c006e9a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.432+0000 7ff9dfde8700 1 --2- 172.21.15.16:0/2509759750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d8101de0 0x7ff9d819e890 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.432+0000 7ff9dfde8700 1 --2- 172.21.15.16:0/2509759750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9d8102700 0x7ff9d819edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.433+0000 7ff9dfde8700 1 -- 172.21.15.16:0/2509759750 >> 172.21.15.16:0/2509759750 conn(0x7ff9d80fb650 msgr2=0x7ff9d8107830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.433+0000 7ff9dfde8700 1 -- 172.21.15.16:0/2509759750 shutdown_connections 2024-04-22T15:45:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.433+0000 7ff9dfde8700 1 -- 172.21.15.16:0/2509759750 wait complete. 2024-04-22T15:45:53.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.565+0000 7f9bde0cd700 1 -- 172.21.15.16:0/1586175812 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f9bd804f2e0 con 0x7f9bd8068580 2024-04-22T15:45:53.568 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:53 smithi016 ceph-mon[26620]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:53.569 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:53 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/2509759750' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T15:45:53.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.565+0000 7f9bd4ff9700 1 -- 172.21.15.16:0/1586175812 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9bc805ab20 con 0x7f9bd8068580 2024-04-22T15:45:53.569 INFO:teuthology.orchestra.run.smithi016.stdout:81604378650 2024-04-22T15:45:53.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.567+0000 7f9bde0cd700 1 -- 172.21.15.16:0/1586175812 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f9bc406c4e0 msgr2=0x7f9bc406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:53.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.567+0000 7f9bde0cd700 1 --2- 172.21.15.16:0/1586175812 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f9bc406c4e0 0x7f9bc406e9a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9bd81014e0 tx=0x7f9bc00073d0 comp rx=0 tx=0).stop 2024-04-22T15:45:53.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.568+0000 7f9bde0cd700 1 -- 172.21.15.16:0/1586175812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd8068580 msgr2=0x7f9bd80ffcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:53.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.568+0000 7f9bde0cd700 1 --2- 172.21.15.16:0/1586175812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd8068580 0x7f9bd80ffcf0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9bc800b5c0 tx=0x7f9bc800bd70 comp rx=0 tx=0).stop 2024-04-22T15:45:53.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.568+0000 7f9bde0cd700 1 -- 172.21.15.16:0/1586175812 shutdown_connections 2024-04-22T15:45:53.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.568+0000 7f9bde0cd700 1 --2- 172.21.15.16:0/1586175812 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f9bc406c4e0 0x7f9bc406e9a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:53.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.568+0000 7f9bde0cd700 1 --2- 172.21.15.16:0/1586175812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd8068580 0x7f9bd80ffcf0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:53.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.568+0000 7f9bde0cd700 1 --2- 172.21.15.16:0/1586175812 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd8106cd0 0x7f9bd8100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:53.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.568+0000 7f9bde0cd700 1 -- 172.21.15.16:0/1586175812 >> 172.21.15.16:0/1586175812 conn(0x7f9bd8075670 msgr2=0x7f9bd80fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:53.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.568+0000 7f9bde0cd700 1 -- 172.21.15.16:0/1586175812 shutdown_connections 2024-04-22T15:45:53.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:53.568+0000 7f9bde0cd700 1 -- 172.21.15.16:0/1586175812 wait complete. 2024-04-22T15:45:53.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:53 smithi027 ceph-mon[34922]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:53.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:53 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/2509759750' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T15:45:53.860 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:45:54.430 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149715 got 111669149717 for osd.3 2024-04-22T15:45:54.430 DEBUG:teuthology.parallel:result is None 2024-04-22T15:45:54.631 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:45:54.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:54 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/1586175812' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T15:45:54.836 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:54 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/1586175812' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T15:45:55.325 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378649 got 81604378650 for osd.2 2024-04-22T15:45:55.325 DEBUG:teuthology.parallel:result is None 2024-04-22T15:45:55.579 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:55 smithi016 ceph-mon[26620]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:55.579 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:55 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:45:55.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:55 smithi027 ceph-mon[34922]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:55.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:55 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:45:56.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.844+0000 7fce5c844700 1 -- 172.21.15.16:0/2406650443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce58105bd0 msgr2=0x7fce58105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:56.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.844+0000 7fce5c844700 1 --2- 172.21.15.16:0/2406650443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce58105bd0 0x7fce58105fb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fce48009910 tx=0x7fce48009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:56.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.844+0000 7fce5c844700 1 -- 172.21.15.16:0/2406650443 shutdown_connections 2024-04-22T15:45:56.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.844+0000 7fce5c844700 1 --2- 172.21.15.16:0/2406650443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce580ffc20 0x7fce58100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:56.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.844+0000 7fce5c844700 1 --2- 172.21.15.16:0/2406650443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce58105bd0 0x7fce58105fb0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:56.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.844+0000 7fce5c844700 1 -- 172.21.15.16:0/2406650443 >> 172.21.15.16:0/2406650443 conn(0x7fce58075670 msgr2=0x7fce58075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:56.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.844+0000 7fce5c844700 1 -- 172.21.15.16:0/2406650443 shutdown_connections 2024-04-22T15:45:56.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.844+0000 7fce5c844700 1 -- 172.21.15.16:0/2406650443 wait complete. 2024-04-22T15:45:56.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.845+0000 7fce5c844700 1 Processor -- start 2024-04-22T15:45:56.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.845+0000 7fce5c844700 1 -- start start 2024-04-22T15:45:56.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.846+0000 7fce5c844700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce580ffc20 0x7fce5819c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:56.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.846+0000 7fce5c844700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce58105bd0 0x7fce5819cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:56.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.846+0000 7fce5c844700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5819d260 con 0x7fce580ffc20 2024-04-22T15:45:56.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.846+0000 7fce5c844700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce58196710 con 0x7fce58105bd0 2024-04-22T15:45:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.846+0000 7fce5659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce580ffc20 0x7fce5819c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.846+0000 7fce5659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce580ffc20 0x7fce5819c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39082/0 (socket says 172.21.15.16:39082) 2024-04-22T15:45:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.846+0000 7fce5659c700 1 -- 172.21.15.16:0/1605649420 learned_addr learned my addr 172.21.15.16:0/1605649420 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:45:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.846+0000 7fce55d9b700 1 --2- 172.21.15.16:0/1605649420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce58105bd0 0x7fce5819cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.846+0000 7fce5659c700 1 -- 172.21.15.16:0/1605649420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce58105bd0 msgr2=0x7fce5819cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.846+0000 7fce5659c700 1 --2- 172.21.15.16:0/1605649420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce58105bd0 0x7fce5819cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.846+0000 7fce5659c700 1 -- 172.21.15.16:0/1605649420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce480095f0 con 0x7fce580ffc20 2024-04-22T15:45:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.846+0000 7fce55d9b700 1 --2- 172.21.15.16:0/1605649420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce58105bd0 0x7fce5819cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:45:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.846+0000 7fce5659c700 1 --2- 172.21.15.16:0/1605649420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce580ffc20 0x7fce5819c690 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fce48006010 tx=0x7fce4800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.846+0000 7fce477fe700 1 -- 172.21.15.16:0/1605649420 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce4801d070 con 0x7fce580ffc20 2024-04-22T15:45:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.846+0000 7fce477fe700 1 -- 172.21.15.16:0/1605649420 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fce4800f460 con 0x7fce580ffc20 2024-04-22T15:45:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.846+0000 7fce5c844700 1 -- 172.21.15.16:0/1605649420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce58196990 con 0x7fce580ffc20 2024-04-22T15:45:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.847+0000 7fce477fe700 1 -- 172.21.15.16:0/1605649420 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce48021640 con 0x7fce580ffc20 2024-04-22T15:45:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.847+0000 7fce5c844700 1 -- 172.21.15.16:0/1605649420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce58196e80 con 0x7fce580ffc20 2024-04-22T15:45:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.848+0000 7fce5c844700 1 -- 172.21.15.16:0/1605649420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce5804f2e0 con 0x7fce580ffc20 2024-04-22T15:45:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.848+0000 7fce477fe700 1 -- 172.21.15.16:0/1605649420 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fce48004b70 con 0x7fce580ffc20 2024-04-22T15:45:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.849+0000 7fce477fe700 1 --2- 172.21.15.16:0/1605649420 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fce4006c2d0 0x7fce4006e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:56.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.849+0000 7fce55d9b700 1 --2- 172.21.15.16:0/1605649420 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fce4006c2d0 0x7fce4006e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:56.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.849+0000 7fce477fe700 1 -- 172.21.15.16:0/1605649420 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fce4808c6d0 con 0x7fce580ffc20 2024-04-22T15:45:56.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.849+0000 7fce55d9b700 1 --2- 172.21.15.16:0/1605649420 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fce4006c2d0 0x7fce4006e790 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fce58197b80 tx=0x7fce4c008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:56.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:56.854+0000 7fce477fe700 1 -- 172.21.15.16:0/1605649420 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fce4805ac80 con 0x7fce580ffc20 2024-04-22T15:45:56.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:56 smithi016 ceph-mon[26620]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:57.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:57.053+0000 7fce5c844700 1 -- 172.21.15.16:0/1605649420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fce58197070 con 0x7fce580ffc20 2024-04-22T15:45:57.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:57.053+0000 7fce477fe700 1 -- 172.21.15.16:0/1605649420 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fce4805a810 con 0x7fce580ffc20 2024-04-22T15:45:57.057 INFO:teuthology.orchestra.run.smithi016.stdout:60129542176 2024-04-22T15:45:57.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:57.056+0000 7fce5c844700 1 -- 172.21.15.16:0/1605649420 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fce4006c2d0 msgr2=0x7fce4006e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:57.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:57.056+0000 7fce5c844700 1 --2- 172.21.15.16:0/1605649420 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fce4006c2d0 0x7fce4006e790 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fce58197b80 tx=0x7fce4c008040 comp rx=0 tx=0).stop 2024-04-22T15:45:57.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:57.056+0000 7fce5c844700 1 -- 172.21.15.16:0/1605649420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce580ffc20 msgr2=0x7fce5819c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:57.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:57.056+0000 7fce5c844700 1 --2- 172.21.15.16:0/1605649420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce580ffc20 0x7fce5819c690 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fce48006010 tx=0x7fce4800be20 comp rx=0 tx=0).stop 2024-04-22T15:45:57.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:57.056+0000 7fce5c844700 1 -- 172.21.15.16:0/1605649420 shutdown_connections 2024-04-22T15:45:57.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:57.056+0000 7fce5c844700 1 --2- 172.21.15.16:0/1605649420 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fce4006c2d0 0x7fce4006e790 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:57.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:57.056+0000 7fce5c844700 1 --2- 172.21.15.16:0/1605649420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce580ffc20 0x7fce5819c690 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:57.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:57.056+0000 7fce5c844700 1 --2- 172.21.15.16:0/1605649420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce58105bd0 0x7fce5819cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:57.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:57.056+0000 7fce5c844700 1 -- 172.21.15.16:0/1605649420 >> 172.21.15.16:0/1605649420 conn(0x7fce58075670 msgr2=0x7fce58068620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:57.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:57.056+0000 7fce5c844700 1 -- 172.21.15.16:0/1605649420 shutdown_connections 2024-04-22T15:45:57.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:57.056+0000 7fce5c844700 1 -- 172.21.15.16:0/1605649420 wait complete. 2024-04-22T15:45:57.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:56 smithi027 ceph-mon[34922]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:57.822 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542173 got 60129542176 for osd.1 2024-04-22T15:45:57.822 DEBUG:teuthology.parallel:result is None 2024-04-22T15:45:57.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:57 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/1605649420' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T15:45:58.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:57 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/1605649420' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T15:45:58.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.605+0000 7fea6977a700 1 -- 172.21.15.16:0/1412400673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea64102700 msgr2=0x7fea6410abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:58.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.605+0000 7fea6977a700 1 --2- 172.21.15.16:0/1412400673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea64102700 0x7fea6410abf0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fea58009910 tx=0x7fea58009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:58.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.605+0000 7fea6977a700 1 -- 172.21.15.16:0/1412400673 shutdown_connections 2024-04-22T15:45:58.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.605+0000 7fea6977a700 1 --2- 172.21.15.16:0/1412400673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea64102700 0x7fea6410abf0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:58.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.605+0000 7fea6977a700 1 --2- 172.21.15.16:0/1412400673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea64101de0 0x7fea641021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:58.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.605+0000 7fea6977a700 1 -- 172.21.15.16:0/1412400673 >> 172.21.15.16:0/1412400673 conn(0x7fea640fb610 msgr2=0x7fea640fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:58.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.605+0000 7fea6977a700 1 -- 172.21.15.16:0/1412400673 shutdown_connections 2024-04-22T15:45:58.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.605+0000 7fea6977a700 1 -- 172.21.15.16:0/1412400673 wait complete. 2024-04-22T15:45:58.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.605+0000 7fea6977a700 1 Processor -- start 2024-04-22T15:45:58.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.606+0000 7fea6977a700 1 -- start start 2024-04-22T15:45:58.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.606+0000 7fea6977a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea64101de0 0x7fea640ffa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:58.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.606+0000 7fea6977a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea64102700 0x7fea640fff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:58.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.606+0000 7fea6977a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea641004c0 con 0x7fea64102700 2024-04-22T15:45:58.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.606+0000 7fea6977a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea64100600 con 0x7fea64101de0 2024-04-22T15:45:58.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.606+0000 7fea627fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea64102700 0x7fea640fff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.606+0000 7fea627fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea64102700 0x7fea640fff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39100/0 (socket says 172.21.15.16:39100) 2024-04-22T15:45:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.606+0000 7fea627fc700 1 -- 172.21.15.16:0/695421355 learned_addr learned my addr 172.21.15.16:0/695421355 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:45:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.606+0000 7fea62ffd700 1 --2- 172.21.15.16:0/695421355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea64101de0 0x7fea640ffa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.606+0000 7fea627fc700 1 -- 172.21.15.16:0/695421355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea64101de0 msgr2=0x7fea640ffa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.606+0000 7fea627fc700 1 --2- 172.21.15.16:0/695421355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea64101de0 0x7fea640ffa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.606+0000 7fea627fc700 1 -- 172.21.15.16:0/695421355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea580095f0 con 0x7fea64102700 2024-04-22T15:45:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.606+0000 7fea627fc700 1 --2- 172.21.15.16:0/695421355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea64102700 0x7fea640fff80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fea5800bda0 tx=0x7fea5800bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.607+0000 7fea4bfff700 1 -- 172.21.15.16:0/695421355 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea5801d070 con 0x7fea64102700 2024-04-22T15:45:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.607+0000 7fea4bfff700 1 -- 172.21.15.16:0/695421355 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fea5800f460 con 0x7fea64102700 2024-04-22T15:45:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.607+0000 7fea4bfff700 1 -- 172.21.15.16:0/695421355 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea58017630 con 0x7fea64102700 2024-04-22T15:45:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.607+0000 7fea6977a700 1 -- 172.21.15.16:0/695421355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea64100880 con 0x7fea64102700 2024-04-22T15:45:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.607+0000 7fea6977a700 1 -- 172.21.15.16:0/695421355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea641a2d80 con 0x7fea64102700 2024-04-22T15:45:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.608+0000 7fea6977a700 1 -- 172.21.15.16:0/695421355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea641082f0 con 0x7fea64102700 2024-04-22T15:45:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.608+0000 7fea4bfff700 1 -- 172.21.15.16:0/695421355 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fea5800fab0 con 0x7fea64102700 2024-04-22T15:45:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.609+0000 7fea4bfff700 1 --2- 172.21.15.16:0/695421355 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fea4c06c4e0 0x7fea4c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.609+0000 7fea4bfff700 1 -- 172.21.15.16:0/695421355 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fea5808c730 con 0x7fea64102700 2024-04-22T15:45:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.609+0000 7fea62ffd700 1 --2- 172.21.15.16:0/695421355 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fea4c06c4e0 0x7fea4c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.609+0000 7fea62ffd700 1 --2- 172.21.15.16:0/695421355 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fea4c06c4e0 0x7fea4c06e9a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fea54005950 tx=0x7fea5400b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:58.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.613+0000 7fea4bfff700 1 -- 172.21.15.16:0/695421355 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fea580572d0 con 0x7fea64102700 2024-04-22T15:45:58.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.812+0000 7fea6977a700 1 -- 172.21.15.16:0/695421355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fea6404f2e0 con 0x7fea64102700 2024-04-22T15:45:58.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.813+0000 7fea4bfff700 1 -- 172.21.15.16:0/695421355 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fea5805a8f0 con 0x7fea64102700 2024-04-22T15:45:58.816 INFO:teuthology.orchestra.run.smithi016.stdout:167503724556 2024-04-22T15:45:58.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.815+0000 7fea6977a700 1 -- 172.21.15.16:0/695421355 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fea4c06c4e0 msgr2=0x7fea4c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:58.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.815+0000 7fea6977a700 1 --2- 172.21.15.16:0/695421355 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fea4c06c4e0 0x7fea4c06e9a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fea54005950 tx=0x7fea5400b410 comp rx=0 tx=0).stop 2024-04-22T15:45:58.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.815+0000 7fea6977a700 1 -- 172.21.15.16:0/695421355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea64102700 msgr2=0x7fea640fff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:58.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.815+0000 7fea6977a700 1 --2- 172.21.15.16:0/695421355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea64102700 0x7fea640fff80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fea5800bda0 tx=0x7fea5800bdd0 comp rx=0 tx=0).stop 2024-04-22T15:45:58.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.815+0000 7fea6977a700 1 -- 172.21.15.16:0/695421355 shutdown_connections 2024-04-22T15:45:58.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.815+0000 7fea6977a700 1 --2- 172.21.15.16:0/695421355 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fea4c06c4e0 0x7fea4c06e9a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:58.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.815+0000 7fea6977a700 1 --2- 172.21.15.16:0/695421355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea64101de0 0x7fea640ffa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:58.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.815+0000 7fea6977a700 1 --2- 172.21.15.16:0/695421355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea64102700 0x7fea640fff80 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:58.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.815+0000 7fea6977a700 1 -- 172.21.15.16:0/695421355 >> 172.21.15.16:0/695421355 conn(0x7fea640fb610 msgr2=0x7fea64105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:58.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.815+0000 7fea6977a700 1 -- 172.21.15.16:0/695421355 shutdown_connections 2024-04-22T15:45:58.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:58.815+0000 7fea6977a700 1 -- 172.21.15.16:0/695421355 wait complete. 2024-04-22T15:45:58.850 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:58 smithi016 ceph-mon[26620]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:59.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:58 smithi027 ceph-mon[34922]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:45:59.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.224+0000 7f4e0be65700 1 -- 172.21.15.16:0/1715557272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e04100ea0 msgr2=0x7f4e04101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:59.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.224+0000 7f4e0be65700 1 --2- 172.21.15.16:0/1715557272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e04100ea0 0x7f4e04101280 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4e00009910 tx=0x7f4e00009c20 comp rx=0 tx=0).stop 2024-04-22T15:45:59.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.224+0000 7f4e0be65700 1 -- 172.21.15.16:0/1715557272 shutdown_connections 2024-04-22T15:45:59.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.224+0000 7f4e0be65700 1 --2- 172.21.15.16:0/1715557272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e04101850 0x7f4e041058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:59.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.224+0000 7f4e0be65700 1 --2- 172.21.15.16:0/1715557272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e04100ea0 0x7f4e04101280 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:59.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.224+0000 7f4e0be65700 1 -- 172.21.15.16:0/1715557272 >> 172.21.15.16:0/1715557272 conn(0x7f4e040fc750 msgr2=0x7f4e040feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:59.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.224+0000 7f4e0be65700 1 -- 172.21.15.16:0/1715557272 shutdown_connections 2024-04-22T15:45:59.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.224+0000 7f4e0be65700 1 -- 172.21.15.16:0/1715557272 wait complete. 2024-04-22T15:45:59.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.224+0000 7f4e0be65700 1 Processor -- start 2024-04-22T15:45:59.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.224+0000 7f4e0be65700 1 -- start start 2024-04-22T15:45:59.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.225+0000 7f4e0be65700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e04100ea0 0x7f4e0419c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:59.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.225+0000 7f4e0be65700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e04101850 0x7f4e0419cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:59.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.225+0000 7f4e0be65700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0419d2e0 con 0x7f4e04100ea0 2024-04-22T15:45:59.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.225+0000 7f4e0be65700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e04196790 con 0x7f4e04101850 2024-04-22T15:45:59.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.225+0000 7f4e09c01700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e04100ea0 0x7f4e0419c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:59.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.225+0000 7f4e09c01700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e04100ea0 0x7f4e0419c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39124/0 (socket says 172.21.15.16:39124) 2024-04-22T15:45:59.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.225+0000 7f4e09c01700 1 -- 172.21.15.16:0/3741736030 learned_addr learned my addr 172.21.15.16:0/3741736030 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:45:59.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.225+0000 7f4e09400700 1 --2- 172.21.15.16:0/3741736030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e04101850 0x7f4e0419cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:59.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.225+0000 7f4e09c01700 1 -- 172.21.15.16:0/3741736030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e04101850 msgr2=0x7f4e0419cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:59.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.225+0000 7f4e09c01700 1 --2- 172.21.15.16:0/3741736030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e04101850 0x7f4e0419cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:59.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.225+0000 7f4e09c01700 1 -- 172.21.15.16:0/3741736030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e000095f0 con 0x7f4e04100ea0 2024-04-22T15:45:59.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.225+0000 7f4e09c01700 1 --2- 172.21.15.16:0/3741736030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e04100ea0 0x7f4e0419c710 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4e0000fa00 tx=0x7f4e0000fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:59.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.225+0000 7f4dfaffd700 1 -- 172.21.15.16:0/3741736030 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e0001d070 con 0x7f4e04100ea0 2024-04-22T15:45:59.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.225+0000 7f4e0be65700 1 -- 172.21.15.16:0/3741736030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e04196a10 con 0x7f4e04100ea0 2024-04-22T15:45:59.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.225+0000 7f4dfaffd700 1 -- 172.21.15.16:0/3741736030 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4e0000fe90 con 0x7f4e04100ea0 2024-04-22T15:45:59.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.225+0000 7f4dfaffd700 1 -- 172.21.15.16:0/3741736030 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e00022470 con 0x7f4e04100ea0 2024-04-22T15:45:59.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.225+0000 7f4e0be65700 1 -- 172.21.15.16:0/3741736030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e04196f00 con 0x7f4e04100ea0 2024-04-22T15:45:59.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.227+0000 7f4e0be65700 1 -- 172.21.15.16:0/3741736030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e04109240 con 0x7f4e04100ea0 2024-04-22T15:45:59.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.227+0000 7f4dfaffd700 1 -- 172.21.15.16:0/3741736030 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f4e00003680 con 0x7f4e04100ea0 2024-04-22T15:45:59.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.227+0000 7f4dfaffd700 1 --2- 172.21.15.16:0/3741736030 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4df006c4e0 0x7f4df006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:45:59.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.227+0000 7f4dfaffd700 1 -- 172.21.15.16:0/3741736030 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f4e0008c8a0 con 0x7f4e04100ea0 2024-04-22T15:45:59.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.227+0000 7f4e09400700 1 --2- 172.21.15.16:0/3741736030 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4df006c4e0 0x7f4df006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:45:59.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.228+0000 7f4e09400700 1 --2- 172.21.15.16:0/3741736030 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4df006c4e0 0x7f4df006e9a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4e04197c00 tx=0x7f4df4009260 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:45:59.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.232+0000 7f4dfaffd700 1 -- 172.21.15.16:0/3741736030 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4e0005ae50 con 0x7f4e04100ea0 2024-04-22T15:45:59.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.434+0000 7f4e0be65700 1 -- 172.21.15.16:0/3741736030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f4e0404f2e0 con 0x7f4e04100ea0 2024-04-22T15:45:59.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.434+0000 7f4dfaffd700 1 -- 172.21.15.16:0/3741736030 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f4e0005a9e0 con 0x7f4e04100ea0 2024-04-22T15:45:59.436 INFO:teuthology.orchestra.run.smithi016.stdout:146028888081 2024-04-22T15:45:59.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.436+0000 7f4e0be65700 1 -- 172.21.15.16:0/3741736030 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4df006c4e0 msgr2=0x7f4df006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:59.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.436+0000 7f4e0be65700 1 --2- 172.21.15.16:0/3741736030 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4df006c4e0 0x7f4df006e9a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4e04197c00 tx=0x7f4df4009260 comp rx=0 tx=0).stop 2024-04-22T15:45:59.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.436+0000 7f4e0be65700 1 -- 172.21.15.16:0/3741736030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e04100ea0 msgr2=0x7f4e0419c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:45:59.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.436+0000 7f4e0be65700 1 --2- 172.21.15.16:0/3741736030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e04100ea0 0x7f4e0419c710 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4e0000fa00 tx=0x7f4e0000fa30 comp rx=0 tx=0).stop 2024-04-22T15:45:59.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.436+0000 7f4e0be65700 1 -- 172.21.15.16:0/3741736030 shutdown_connections 2024-04-22T15:45:59.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.436+0000 7f4e0be65700 1 --2- 172.21.15.16:0/3741736030 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4df006c4e0 0x7f4df006e9a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:59.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.436+0000 7f4e0be65700 1 --2- 172.21.15.16:0/3741736030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e04100ea0 0x7f4e0419c710 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:59.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.436+0000 7f4e0be65700 1 --2- 172.21.15.16:0/3741736030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e04101850 0x7f4e0419cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:45:59.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.436+0000 7f4e0be65700 1 -- 172.21.15.16:0/3741736030 >> 172.21.15.16:0/3741736030 conn(0x7f4e040fc750 msgr2=0x7f4e040fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:45:59.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.437+0000 7f4e0be65700 1 -- 172.21.15.16:0/3741736030 shutdown_connections 2024-04-22T15:45:59.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:45:59.437+0000 7f4e0be65700 1 -- 172.21.15.16:0/3741736030 wait complete. 2024-04-22T15:45:59.686 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:59 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/695421355' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T15:45:59.687 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:45:59 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/3741736030' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T15:45:59.706 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724553 got 167503724556 for osd.5 2024-04-22T15:45:59.706 DEBUG:teuthology.parallel:result is None 2024-04-22T15:46:00.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:59 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/695421355' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T15:46:00.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:45:59 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/3741736030' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T15:46:00.541 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888078 got 146028888081 for osd.4 2024-04-22T15:46:00.541 DEBUG:teuthology.parallel:result is None 2024-04-22T15:46:00.542 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-22T15:46:00.542 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T15:46:00.765 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:46:00.873 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:00 smithi016 ceph-mon[26620]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:01.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:00 smithi027 ceph-mon[34922]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:02.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.376+0000 7f63071fe700 1 -- 172.21.15.16:0/3455072542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63001004f0 msgr2=0x7f63001049c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:02.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.376+0000 7f63071fe700 1 --2- 172.21.15.16:0/3455072542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63001004f0 0x7f63001049c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f62ec009910 tx=0x7f62ec009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:02.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.376+0000 7f63071fe700 1 -- 172.21.15.16:0/3455072542 shutdown_connections 2024-04-22T15:46:02.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.376+0000 7f63071fe700 1 --2- 172.21.15.16:0/3455072542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63001004f0 0x7f63001049c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:02.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.376+0000 7f63071fe700 1 --2- 172.21.15.16:0/3455072542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63000ffbd0 0x7f63000fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:02.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.376+0000 7f63071fe700 1 -- 172.21.15.16:0/3455072542 >> 172.21.15.16:0/3455072542 conn(0x7f63000fb650 msgr2=0x7f63000fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:02.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.376+0000 7f63071fe700 1 -- 172.21.15.16:0/3455072542 shutdown_connections 2024-04-22T15:46:02.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.376+0000 7f63071fe700 1 -- 172.21.15.16:0/3455072542 wait complete. 2024-04-22T15:46:02.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.376+0000 7f63071fe700 1 Processor -- start 2024-04-22T15:46:02.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.376+0000 7f63071fe700 1 -- start start 2024-04-22T15:46:02.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.377+0000 7f63071fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63000ffbd0 0x7f630019cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:02.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.377+0000 7f63071fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63001004f0 0x7f630019d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:02.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.377+0000 7f63071fe700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630019dbd0 con 0x7f63000ffbd0 2024-04-22T15:46:02.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.377+0000 7f63071fe700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63001a1960 con 0x7f63001004f0 2024-04-22T15:46:02.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.377+0000 7f6304f9a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63000ffbd0 0x7f630019cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:02.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.377+0000 7f6304f9a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63000ffbd0 0x7f630019cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35136/0 (socket says 172.21.15.16:35136) 2024-04-22T15:46:02.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.377+0000 7f6304f9a700 1 -- 172.21.15.16:0/2706430471 learned_addr learned my addr 172.21.15.16:0/2706430471 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:46:02.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.377+0000 7f62fffff700 1 --2- 172.21.15.16:0/2706430471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63001004f0 0x7f630019d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:02.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.377+0000 7f6304f9a700 1 -- 172.21.15.16:0/2706430471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63001004f0 msgr2=0x7f630019d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:02.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.377+0000 7f6304f9a700 1 --2- 172.21.15.16:0/2706430471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63001004f0 0x7f630019d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:02.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.377+0000 7f6304f9a700 1 -- 172.21.15.16:0/2706430471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ec0095f0 con 0x7f63000ffbd0 2024-04-22T15:46:02.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.377+0000 7f62fffff700 1 --2- 172.21.15.16:0/2706430471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63001004f0 0x7f630019d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:46:02.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.377+0000 7f6304f9a700 1 --2- 172.21.15.16:0/2706430471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63000ffbd0 0x7f630019cfb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f62f400b4b0 tx=0x7f62f400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:02.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.377+0000 7f62fdffb700 1 -- 172.21.15.16:0/2706430471 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62f4011800 con 0x7f63000ffbd0 2024-04-22T15:46:02.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.377+0000 7f62fdffb700 1 -- 172.21.15.16:0/2706430471 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f62f4011e10 con 0x7f63000ffbd0 2024-04-22T15:46:02.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.378+0000 7f63071fe700 1 -- 172.21.15.16:0/2706430471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63001a1c40 con 0x7f63000ffbd0 2024-04-22T15:46:02.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.378+0000 7f62fdffb700 1 -- 172.21.15.16:0/2706430471 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62f400f2e0 con 0x7f63000ffbd0 2024-04-22T15:46:02.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.378+0000 7f63071fe700 1 -- 172.21.15.16:0/2706430471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63001a2190 con 0x7f63000ffbd0 2024-04-22T15:46:02.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.379+0000 7f63071fe700 1 -- 172.21.15.16:0/2706430471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6300102120 con 0x7f63000ffbd0 2024-04-22T15:46:02.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.379+0000 7f62fdffb700 1 -- 172.21.15.16:0/2706430471 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f62f4010380 con 0x7f63000ffbd0 2024-04-22T15:46:02.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.379+0000 7f62fdffb700 1 --2- 172.21.15.16:0/2706430471 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f62f006c4e0 0x7f62f006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:02.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.380+0000 7f62fdffb700 1 -- 172.21.15.16:0/2706430471 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f62f408b050 con 0x7f63000ffbd0 2024-04-22T15:46:02.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.380+0000 7f62fffff700 1 --2- 172.21.15.16:0/2706430471 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f62f006c4e0 0x7f62f006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:02.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.380+0000 7f62fffff700 1 --2- 172.21.15.16:0/2706430471 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f62f006c4e0 0x7f62f006e9a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f630019e5d0 tx=0x7f62ec01a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:02.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.384+0000 7f62fdffb700 1 -- 172.21.15.16:0/2706430471 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f62f4055ca0 con 0x7f63000ffbd0 2024-04-22T15:46:02.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.578+0000 7f63071fe700 1 -- 172.21.15.16:0/2706430471 --> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f63001a2460 con 0x7f62f006c4e0 2024-04-22T15:46:02.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.581+0000 7f62fdffb700 1 -- 172.21.15.16:0/2706430471 <== mgr.14217 v2:172.21.15.16:6800/1057234399 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27657 (secure 0 0 0) 0x7f63001a2460 con 0x7f62f006c4e0 2024-04-22T15:46:02.583 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:46:02.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.583+0000 7f63071fe700 1 -- 172.21.15.16:0/2706430471 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f62f006c4e0 msgr2=0x7f62f006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:02.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.583+0000 7f63071fe700 1 --2- 172.21.15.16:0/2706430471 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f62f006c4e0 0x7f62f006e9a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f630019e5d0 tx=0x7f62ec01a040 comp rx=0 tx=0).stop 2024-04-22T15:46:02.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.583+0000 7f63071fe700 1 -- 172.21.15.16:0/2706430471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63000ffbd0 msgr2=0x7f630019cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:02.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.583+0000 7f63071fe700 1 --2- 172.21.15.16:0/2706430471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63000ffbd0 0x7f630019cfb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f62f400b4b0 tx=0x7f62f400b870 comp rx=0 tx=0).stop 2024-04-22T15:46:02.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.583+0000 7f63071fe700 1 -- 172.21.15.16:0/2706430471 shutdown_connections 2024-04-22T15:46:02.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.583+0000 7f63071fe700 1 --2- 172.21.15.16:0/2706430471 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f62f006c4e0 0x7f62f006e9a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:02.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.583+0000 7f63071fe700 1 --2- 172.21.15.16:0/2706430471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63000ffbd0 0x7f630019cfb0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:02.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.583+0000 7f63071fe700 1 --2- 172.21.15.16:0/2706430471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63001004f0 0x7f630019d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:02.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.583+0000 7f63071fe700 1 -- 172.21.15.16:0/2706430471 >> 172.21.15.16:0/2706430471 conn(0x7f63000fb650 msgr2=0x7f6300111040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:02.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.583+0000 7f63071fe700 1 -- 172.21.15.16:0/2706430471 shutdown_connections 2024-04-22T15:46:02.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:02.584+0000 7f63071fe700 1 -- 172.21.15.16:0/2706430471 wait complete. 2024-04-22T15:46:02.587 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-22T15:46:03.185 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-04-22T15:46:02.423569+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165588,"kb_used_data":3876,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305324,"statfs":{"total":575970213888,"available":575800651776,"internally_reserved":0,"allocated":3969024,"data_stored":2606850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002133"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":185,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T15:45:11.500956+0000","last_change":"2024-04-22T15:44:46.477638+0000","last_active":"2024-04-22T15:45:11.500956+0000","last_peered":"2024-04-22T15:45:11.500956+0000","last_clean":"2024-04-22T15:45:11.500956+0000","last_became_active":"2024-04-22T15:44:46.477249+0000","last_became_peered":"2024-04-22T15:44:46.477249+0000","last_unstale":"2024-04-22T15:45:11.500956+0000","last_undegraded":"2024-04-22T15:45:11.500956+0000","last_fullsized":"2024-04-22T15:45:11.500956+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T15:43:59.149529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T15:43:59.149529+0000","last_clean_scrub_stamp":"2024-04-22T15:43:59.149529+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:45:29.685595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724556,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]}]},{"osd":4,"up_from":34,"seq":146028888082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 15:45:47 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":19.591999999999999,"5min":19.591999999999999,"15min":19.591999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.41699999999999998}]},{"osd":1,"last update":"Mon Apr 22 15:45:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.50600000000000001}]},{"osd":2,"last update":"Mon Apr 22 15:45:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.651,"5min":1.651,"15min":1.651},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":19.608000000000001,"5min":19.608000000000001,"15min":19.608000000000001},"last":0.44700000000000001}]},{"osd":3,"last update":"Mon Apr 22 15:45:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.377},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.42799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":729659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 15:45:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.60599999999999998}]},{"osd":1,"last update":"Mon Apr 22 15:45:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.52400000000000002}]},{"osd":2,"last update":"Mon Apr 22 15:45:20 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.42199999999999999}]},{"osd":4,"last update":"Mon Apr 22 15:45:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.54100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 15:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.27700000000000002}]},{"osd":1,"last update":"Mon Apr 22 15:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.29299999999999998}]},{"osd":3,"last update":"Mon Apr 22 15:45:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.69199999999999995}]},{"osd":4,"last update":"Mon Apr 22 15:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.55500000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":729659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 15:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":0.55700000000000005,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.35399999999999998,"15min":0.35199999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.57399999999999995,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.311}]},{"osd":2,"last update":"Mon Apr 22 15:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.29699999999999999}]},{"osd":3,"last update":"Mon Apr 22 15:45:22 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.497}]},{"osd":4,"last update":"Mon Apr 22 15:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.628}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":729659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 15:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.33600000000000002,"15min":0.32700000000000001},"min":{"1min":0.221,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.316},{"interface":"front","average":{"1min":0.374,"5min":0.33700000000000002,"15min":0.33100000000000002},"min":{"1min":0.23899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52600000000000002,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.433}]},{"osd":2,"last update":"Mon Apr 22 15:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"last":0.374},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.41299999999999998}]},{"osd":3,"last update":"Mon Apr 22 15:45:24 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.63200000000000001}]},{"osd":4,"last update":"Mon Apr 22 15:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.66400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T15:46:03.186 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T15:46:03.401 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:46:03.797 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:03 smithi016 ceph-mon[26620]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:03.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:03 smithi027 ceph-mon[34922]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:04.709 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:04 smithi016 ceph-mon[26620]: from='client.14434 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T15:46:04.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:04 smithi027 ceph-mon[34922]: from='client.14434 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T15:46:04.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.921+0000 7f9e01c6a700 1 -- 172.21.15.16:0/3445265492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dfc102720 msgr2=0x7f9dfc10ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:04.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.921+0000 7f9e01c6a700 1 --2- 172.21.15.16:0/3445265492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dfc102720 0x7f9dfc10ac10 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9df0009910 tx=0x7f9df0009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.921+0000 7f9e01c6a700 1 -- 172.21.15.16:0/3445265492 shutdown_connections 2024-04-22T15:46:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.921+0000 7f9e01c6a700 1 --2- 172.21.15.16:0/3445265492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dfc102720 0x7f9dfc10ac10 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.921+0000 7f9e01c6a700 1 --2- 172.21.15.16:0/3445265492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dfc101e00 0x7f9dfc1021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.921+0000 7f9e01c6a700 1 -- 172.21.15.16:0/3445265492 >> 172.21.15.16:0/3445265492 conn(0x7f9dfc0fb650 msgr2=0x7f9dfc0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.921+0000 7f9e01c6a700 1 -- 172.21.15.16:0/3445265492 shutdown_connections 2024-04-22T15:46:04.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.921+0000 7f9e01c6a700 1 -- 172.21.15.16:0/3445265492 wait complete. 2024-04-22T15:46:04.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.922+0000 7f9e01c6a700 1 Processor -- start 2024-04-22T15:46:04.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.922+0000 7f9e01c6a700 1 -- start start 2024-04-22T15:46:04.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.922+0000 7f9e01c6a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dfc101e00 0x7f9dfc0ffa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:04.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.922+0000 7f9e01c6a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dfc102720 0x7f9dfc0fffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:04.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.922+0000 7f9e01c6a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc1004e0 con 0x7f9dfc102720 2024-04-22T15:46:04.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.922+0000 7f9e01c6a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc100620 con 0x7f9dfc101e00 2024-04-22T15:46:04.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.922+0000 7f9dfaffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dfc102720 0x7f9dfc0fffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:04.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.922+0000 7f9dfaffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dfc102720 0x7f9dfc0fffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35146/0 (socket says 172.21.15.16:35146) 2024-04-22T15:46:04.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.922+0000 7f9dfaffd700 1 -- 172.21.15.16:0/183066750 learned_addr learned my addr 172.21.15.16:0/183066750 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:46:04.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.923+0000 7f9dfb7fe700 1 --2- 172.21.15.16:0/183066750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dfc101e00 0x7f9dfc0ffa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:04.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.923+0000 7f9dfaffd700 1 -- 172.21.15.16:0/183066750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dfc101e00 msgr2=0x7f9dfc0ffa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:04.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.923+0000 7f9dfaffd700 1 --2- 172.21.15.16:0/183066750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dfc101e00 0x7f9dfc0ffa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:04.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.923+0000 7f9dfaffd700 1 -- 172.21.15.16:0/183066750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df00095f0 con 0x7f9dfc102720 2024-04-22T15:46:04.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.923+0000 7f9dfaffd700 1 --2- 172.21.15.16:0/183066750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dfc102720 0x7f9dfc0fffa0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9df000bd70 tx=0x7f9df000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:04.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.923+0000 7f9df8ff9700 1 -- 172.21.15.16:0/183066750 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9df001d070 con 0x7f9dfc102720 2024-04-22T15:46:04.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.923+0000 7f9e01c6a700 1 -- 172.21.15.16:0/183066750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dfc1008a0 con 0x7f9dfc102720 2024-04-22T15:46:04.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.923+0000 7f9df8ff9700 1 -- 172.21.15.16:0/183066750 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9df000f460 con 0x7f9dfc102720 2024-04-22T15:46:04.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.923+0000 7f9df8ff9700 1 -- 172.21.15.16:0/183066750 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9df0017630 con 0x7f9dfc102720 2024-04-22T15:46:04.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.923+0000 7f9e01c6a700 1 -- 172.21.15.16:0/183066750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dfc1a2dc0 con 0x7f9dfc102720 2024-04-22T15:46:04.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.924+0000 7f9e01c6a700 1 -- 172.21.15.16:0/183066750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dfc108310 con 0x7f9dfc102720 2024-04-22T15:46:04.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.925+0000 7f9df8ff9700 1 -- 172.21.15.16:0/183066750 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f9df000f9e0 con 0x7f9dfc102720 2024-04-22T15:46:04.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.925+0000 7f9df8ff9700 1 --2- 172.21.15.16:0/183066750 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f9de406c4e0 0x7f9de406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.925+0000 7f9df8ff9700 1 -- 172.21.15.16:0/183066750 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f9df008c8e0 con 0x7f9dfc102720 2024-04-22T15:46:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.925+0000 7f9dfb7fe700 1 --2- 172.21.15.16:0/183066750 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f9de406c4e0 0x7f9de406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.926+0000 7f9dfb7fe700 1 --2- 172.21.15.16:0/183066750 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f9de406c4e0 0x7f9de406e9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9dec005950 tx=0x7f9dec0058e0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:04.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:04.929+0000 7f9df8ff9700 1 -- 172.21.15.16:0/183066750 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9df0057400 con 0x7f9dfc102720 2024-04-22T15:46:05.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:05.124+0000 7f9e01c6a700 1 -- 172.21.15.16:0/183066750 --> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9dfc0611d0 con 0x7f9de406c4e0 2024-04-22T15:46:05.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:05.126+0000 7f9df8ff9700 1 -- 172.21.15.16:0/183066750 <== mgr.14217 v2:172.21.15.16:6800/1057234399 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27725 (secure 0 0 0) 0x7f9dfc0611d0 con 0x7f9de406c4e0 2024-04-22T15:46:05.129 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:46:05.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:05.129+0000 7f9e01c6a700 1 -- 172.21.15.16:0/183066750 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f9de406c4e0 msgr2=0x7f9de406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:05.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:05.129+0000 7f9e01c6a700 1 --2- 172.21.15.16:0/183066750 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f9de406c4e0 0x7f9de406e9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9dec005950 tx=0x7f9dec0058e0 comp rx=0 tx=0).stop 2024-04-22T15:46:05.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:05.129+0000 7f9e01c6a700 1 -- 172.21.15.16:0/183066750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dfc102720 msgr2=0x7f9dfc0fffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:05.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:05.129+0000 7f9e01c6a700 1 --2- 172.21.15.16:0/183066750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dfc102720 0x7f9dfc0fffa0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9df000bd70 tx=0x7f9df000be50 comp rx=0 tx=0).stop 2024-04-22T15:46:05.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:05.129+0000 7f9e01c6a700 1 -- 172.21.15.16:0/183066750 shutdown_connections 2024-04-22T15:46:05.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:05.129+0000 7f9e01c6a700 1 --2- 172.21.15.16:0/183066750 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f9de406c4e0 0x7f9de406e9a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:05.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:05.129+0000 7f9e01c6a700 1 --2- 172.21.15.16:0/183066750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dfc101e00 0x7f9dfc0ffa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:05.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:05.129+0000 7f9e01c6a700 1 --2- 172.21.15.16:0/183066750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dfc102720 0x7f9dfc0fffa0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:05.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:05.129+0000 7f9e01c6a700 1 -- 172.21.15.16:0/183066750 >> 172.21.15.16:0/183066750 conn(0x7f9dfc0fb650 msgr2=0x7f9dfc105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:05.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:05.129+0000 7f9e01c6a700 1 -- 172.21.15.16:0/183066750 shutdown_connections 2024-04-22T15:46:05.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:05.129+0000 7f9e01c6a700 1 -- 172.21.15.16:0/183066750 wait complete. 2024-04-22T15:46:05.132 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-22T15:46:05.716 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:05 smithi016 ceph-mon[26620]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:05.738 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-04-22T15:46:04.423946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165588,"kb_used_data":3876,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305324,"statfs":{"total":575970213888,"available":575800651776,"internally_reserved":0,"allocated":3969024,"data_stored":2606850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002135"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":185,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T15:45:11.500956+0000","last_change":"2024-04-22T15:44:46.477638+0000","last_active":"2024-04-22T15:45:11.500956+0000","last_peered":"2024-04-22T15:45:11.500956+0000","last_clean":"2024-04-22T15:45:11.500956+0000","last_became_active":"2024-04-22T15:44:46.477249+0000","last_became_peered":"2024-04-22T15:44:46.477249+0000","last_unstale":"2024-04-22T15:45:11.500956+0000","last_undegraded":"2024-04-22T15:45:11.500956+0000","last_fullsized":"2024-04-22T15:45:11.500956+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T15:43:59.149529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T15:43:59.149529+0000","last_clean_scrub_stamp":"2024-04-22T15:43:59.149529+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:45:29.685595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]}]},{"osd":4,"up_from":34,"seq":146028888082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 15:45:47 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":19.591999999999999,"5min":19.591999999999999,"15min":19.591999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.41699999999999998}]},{"osd":1,"last update":"Mon Apr 22 15:45:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.50600000000000001}]},{"osd":2,"last update":"Mon Apr 22 15:45:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.651,"5min":1.651,"15min":1.651},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":19.608000000000001,"5min":19.608000000000001,"15min":19.608000000000001},"last":0.44700000000000001}]},{"osd":3,"last update":"Mon Apr 22 15:45:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.377},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.42799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":729659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 15:45:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.60599999999999998}]},{"osd":1,"last update":"Mon Apr 22 15:45:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.52400000000000002}]},{"osd":2,"last update":"Mon Apr 22 15:45:20 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.42199999999999999}]},{"osd":4,"last update":"Mon Apr 22 15:45:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.54100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 15:46:01 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.35299999999999998,"15min":0.35599999999999998},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.47099999999999997,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.37,"15min":0.374},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.40200000000000002}]},{"osd":1,"last update":"Mon Apr 22 15:46:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.372,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.503,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.40500000000000003,"15min":0.41199999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.52800000000000002,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.41999999999999998}]},{"osd":3,"last update":"Mon Apr 22 15:45:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.52900000000000003}]},{"osd":4,"last update":"Mon Apr 22 15:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.60899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":729659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 15:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":0.55700000000000005,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.35399999999999998,"15min":0.35199999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.57399999999999995,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.311}]},{"osd":2,"last update":"Mon Apr 22 15:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.29699999999999999}]},{"osd":3,"last update":"Mon Apr 22 15:45:22 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.497}]},{"osd":4,"last update":"Mon Apr 22 15:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.628}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":729659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 15:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.33600000000000002,"15min":0.32700000000000001},"min":{"1min":0.221,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.374,"5min":0.33700000000000002,"15min":0.33100000000000002},"min":{"1min":0.23899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52600000000000002,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.30199999999999999}]},{"osd":2,"last update":"Mon Apr 22 15:46:01 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.27700000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.317},{"interface":"front","average":{"1min":0.37,"5min":0.38200000000000001,"15min":0.38400000000000001},"min":{"1min":0.27000000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.32800000000000001}]},{"osd":3,"last update":"Mon Apr 22 15:45:24 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.432},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.40500000000000003}]},{"osd":4,"last update":"Mon Apr 22 15:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.46300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T15:46:05.739 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-22T15:46:05.739 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-22T15:46:05.739 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-22T15:46:05.739 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-22T15:46:05.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:05 smithi027 ceph-mon[34922]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:05.951 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:46:06.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:06 smithi016 ceph-mon[26620]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T15:46:06.738 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:06 smithi027 ceph-mon[34922]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T15:46:07.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.465+0000 7f211c75b700 1 -- 172.21.15.16:0/2158945313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2114101850 msgr2=0x7f21141058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:07.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.465+0000 7f211c75b700 1 --2- 172.21.15.16:0/2158945313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2114101850 0x7f21141058a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2104009910 tx=0x7f2104009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:07.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.465+0000 7f211c75b700 1 -- 172.21.15.16:0/2158945313 shutdown_connections 2024-04-22T15:46:07.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.465+0000 7f211c75b700 1 --2- 172.21.15.16:0/2158945313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2114101850 0x7f21141058a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:07.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.465+0000 7f211c75b700 1 --2- 172.21.15.16:0/2158945313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2114100ea0 0x7f2114101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:07.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.465+0000 7f211c75b700 1 -- 172.21.15.16:0/2158945313 >> 172.21.15.16:0/2158945313 conn(0x7f21140fc750 msgr2=0x7f21140feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:07.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.465+0000 7f211c75b700 1 -- 172.21.15.16:0/2158945313 shutdown_connections 2024-04-22T15:46:07.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.465+0000 7f211c75b700 1 -- 172.21.15.16:0/2158945313 wait complete. 2024-04-22T15:46:07.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.466+0000 7f211c75b700 1 Processor -- start 2024-04-22T15:46:07.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.466+0000 7f211c75b700 1 -- start start 2024-04-22T15:46:07.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.466+0000 7f211c75b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2114100ea0 0x7f211419c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:07.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.466+0000 7f211c75b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2114101850 0x7f211419cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:07.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.466+0000 7f211c75b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211419d2d0 con 0x7f2114101850 2024-04-22T15:46:07.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.466+0000 7f211c75b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2114196780 con 0x7f2114100ea0 2024-04-22T15:46:07.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.466+0000 7f2119cf6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2114101850 0x7f211419cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:07.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.466+0000 7f2119cf6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2114101850 0x7f211419cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35172/0 (socket says 172.21.15.16:35172) 2024-04-22T15:46:07.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.466+0000 7f2119cf6700 1 -- 172.21.15.16:0/2689395456 learned_addr learned my addr 172.21.15.16:0/2689395456 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:46:07.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.467+0000 7f211a4f7700 1 --2- 172.21.15.16:0/2689395456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2114100ea0 0x7f211419c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:07.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.467+0000 7f2119cf6700 1 -- 172.21.15.16:0/2689395456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2114100ea0 msgr2=0x7f211419c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:07.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.467+0000 7f2119cf6700 1 --2- 172.21.15.16:0/2689395456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2114100ea0 0x7f211419c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:07.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.467+0000 7f2119cf6700 1 -- 172.21.15.16:0/2689395456 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21040095f0 con 0x7f2114101850 2024-04-22T15:46:07.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.467+0000 7f2119cf6700 1 --2- 172.21.15.16:0/2689395456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2114101850 0x7f211419cc40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f210400bda0 tx=0x7f210400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:07.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.467+0000 7f210b7fe700 1 -- 172.21.15.16:0/2689395456 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f210401d070 con 0x7f2114101850 2024-04-22T15:46:07.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.467+0000 7f211c75b700 1 -- 172.21.15.16:0/2689395456 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2114196a00 con 0x7f2114101850 2024-04-22T15:46:07.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.467+0000 7f210b7fe700 1 -- 172.21.15.16:0/2689395456 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f210400f460 con 0x7f2114101850 2024-04-22T15:46:07.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.467+0000 7f210b7fe700 1 -- 172.21.15.16:0/2689395456 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2104017630 con 0x7f2114101850 2024-04-22T15:46:07.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.467+0000 7f211c75b700 1 -- 172.21.15.16:0/2689395456 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2114196ef0 con 0x7f2114101850 2024-04-22T15:46:07.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.468+0000 7f211c75b700 1 -- 172.21.15.16:0/2689395456 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21141092a0 con 0x7f2114101850 2024-04-22T15:46:07.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.469+0000 7f210b7fe700 1 -- 172.21.15.16:0/2689395456 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f210400fab0 con 0x7f2114101850 2024-04-22T15:46:07.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.469+0000 7f210b7fe700 1 --2- 172.21.15.16:0/2689395456 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f210006c4e0 0x7f210006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:07.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.469+0000 7f210b7fe700 1 -- 172.21.15.16:0/2689395456 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f210408c890 con 0x7f2114101850 2024-04-22T15:46:07.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.469+0000 7f211a4f7700 1 --2- 172.21.15.16:0/2689395456 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f210006c4e0 0x7f210006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:07.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.470+0000 7f211a4f7700 1 --2- 172.21.15.16:0/2689395456 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f210006c4e0 0x7f210006e9a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2110007be0 tx=0x7f21100073d0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:07.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.474+0000 7f210b7fe700 1 -- 172.21.15.16:0/2689395456 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f210405ae40 con 0x7f2114101850 2024-04-22T15:46:07.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.715+0000 7f211c75b700 1 -- 172.21.15.16:0/2689395456 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f211404f2e0 con 0x7f2114101850 2024-04-22T15:46:07.718 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:07 smithi016 ceph-mon[26620]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:07.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.715+0000 7f210b7fe700 1 -- 172.21.15.16:0/2689395456 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f210405a9d0 con 0x7f2114101850 2024-04-22T15:46:07.719 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:46:07.719 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-22T15:46:07.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.717+0000 7f211c75b700 1 -- 172.21.15.16:0/2689395456 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f210006c4e0 msgr2=0x7f210006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:07.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.717+0000 7f211c75b700 1 --2- 172.21.15.16:0/2689395456 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f210006c4e0 0x7f210006e9a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2110007be0 tx=0x7f21100073d0 comp rx=0 tx=0).stop 2024-04-22T15:46:07.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.717+0000 7f211c75b700 1 -- 172.21.15.16:0/2689395456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2114101850 msgr2=0x7f211419cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:07.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.717+0000 7f211c75b700 1 --2- 172.21.15.16:0/2689395456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2114101850 0x7f211419cc40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f210400bda0 tx=0x7f210400bdd0 comp rx=0 tx=0).stop 2024-04-22T15:46:07.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.717+0000 7f211c75b700 1 -- 172.21.15.16:0/2689395456 shutdown_connections 2024-04-22T15:46:07.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.717+0000 7f211c75b700 1 --2- 172.21.15.16:0/2689395456 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f210006c4e0 0x7f210006e9a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:07.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.717+0000 7f211c75b700 1 --2- 172.21.15.16:0/2689395456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2114100ea0 0x7f211419c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:07.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.717+0000 7f211c75b700 1 --2- 172.21.15.16:0/2689395456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2114101850 0x7f211419cc40 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:07.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.717+0000 7f211c75b700 1 -- 172.21.15.16:0/2689395456 >> 172.21.15.16:0/2689395456 conn(0x7f21140fc750 msgr2=0x7f21140fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:07.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.717+0000 7f211c75b700 1 -- 172.21.15.16:0/2689395456 shutdown_connections 2024-04-22T15:46:07.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:07.717+0000 7f211c75b700 1 -- 172.21.15.16:0/2689395456 wait complete. 2024-04-22T15:46:07.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:07 smithi027 ceph-mon[34922]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:08.318 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-22T15:46:08.318 INFO:tasks.cephadm:Setup complete, yielding 2024-04-22T15:46:08.319 INFO:teuthology.run_tasks:Running task print... 2024-04-22T15:46:08.329 INFO:teuthology.task.print:**** done end installing v18.2.1 cephadm ... 2024-04-22T15:46:08.329 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T15:46:08.366 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-22T15:46:08.366 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-04-22T15:46:08.577 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:46:08.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:08 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/2689395456' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T15:46:08.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:08 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/2689395456' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T15:46:09.595 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:09 smithi016 ceph-mon[26620]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:09.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:09 smithi027 ceph-mon[34922]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:10.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.090+0000 7f7bf83a4700 1 -- 172.21.15.16:0/108321651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf0101fe0 msgr2=0x7f7bf0102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:10.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.090+0000 7f7bf83a4700 1 --2- 172.21.15.16:0/108321651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf0101fe0 0x7f7bf0102440 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7be0009910 tx=0x7f7be0009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:10.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.090+0000 7f7bf83a4700 1 -- 172.21.15.16:0/108321651 shutdown_connections 2024-04-22T15:46:10.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.090+0000 7f7bf83a4700 1 --2- 172.21.15.16:0/108321651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf0101fe0 0x7f7bf0102440 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:10.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.090+0000 7f7bf83a4700 1 --2- 172.21.15.16:0/108321651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bf00fef60 0x7f7bf00ff380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:10.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.090+0000 7f7bf83a4700 1 -- 172.21.15.16:0/108321651 >> 172.21.15.16:0/108321651 conn(0x7f7bf00fa8c0 msgr2=0x7f7bf00fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:10.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.090+0000 7f7bf83a4700 1 -- 172.21.15.16:0/108321651 shutdown_connections 2024-04-22T15:46:10.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.090+0000 7f7bf83a4700 1 -- 172.21.15.16:0/108321651 wait complete. 2024-04-22T15:46:10.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.091+0000 7f7bf83a4700 1 Processor -- start 2024-04-22T15:46:10.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.091+0000 7f7bf83a4700 1 -- start start 2024-04-22T15:46:10.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.091+0000 7f7bf83a4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf00fef60 0x7f7bf0196550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:10.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.091+0000 7f7bf83a4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bf0101fe0 0x7f7bf0196a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:10.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.091+0000 7f7bf83a4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf01970b0 con 0x7f7bf00fef60 2024-04-22T15:46:10.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.091+0000 7f7bf83a4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf01971f0 con 0x7f7bf0101fe0 2024-04-22T15:46:10.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.091+0000 7f7bf6140700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf00fef60 0x7f7bf0196550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:10.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.091+0000 7f7bf6140700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf00fef60 0x7f7bf0196550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35190/0 (socket says 172.21.15.16:35190) 2024-04-22T15:46:10.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.091+0000 7f7bf6140700 1 -- 172.21.15.16:0/438312257 learned_addr learned my addr 172.21.15.16:0/438312257 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:46:10.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.091+0000 7f7bf6140700 1 -- 172.21.15.16:0/438312257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bf0101fe0 msgr2=0x7f7bf0196a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:10.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.091+0000 7f7bf593f700 1 --2- 172.21.15.16:0/438312257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bf0101fe0 0x7f7bf0196a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:10.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.091+0000 7f7bf6140700 1 --2- 172.21.15.16:0/438312257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bf0101fe0 0x7f7bf0196a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:10.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.091+0000 7f7bf6140700 1 -- 172.21.15.16:0/438312257 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be00095f0 con 0x7f7bf00fef60 2024-04-22T15:46:10.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.092+0000 7f7bf6140700 1 --2- 172.21.15.16:0/438312257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf00fef60 0x7f7bf0196550 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7be800b480 tx=0x7f7be800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:10.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.092+0000 7f7be77fe700 1 -- 172.21.15.16:0/438312257 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7be80118c0 con 0x7f7bf00fef60 2024-04-22T15:46:10.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.092+0000 7f7be77fe700 1 -- 172.21.15.16:0/438312257 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7be8010380 con 0x7f7bf00fef60 2024-04-22T15:46:10.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.092+0000 7f7bf83a4700 1 -- 172.21.15.16:0/438312257 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf019bca0 con 0x7f7bf00fef60 2024-04-22T15:46:10.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.092+0000 7f7be77fe700 1 -- 172.21.15.16:0/438312257 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7be800f4f0 con 0x7f7bf00fef60 2024-04-22T15:46:10.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.092+0000 7f7bf83a4700 1 -- 172.21.15.16:0/438312257 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf019c1a0 con 0x7f7bf00fef60 2024-04-22T15:46:10.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.093+0000 7f7bf83a4700 1 -- 172.21.15.16:0/438312257 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf01098a0 con 0x7f7bf00fef60 2024-04-22T15:46:10.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.093+0000 7f7be77fe700 1 -- 172.21.15.16:0/438312257 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f7be80104f0 con 0x7f7bf00fef60 2024-04-22T15:46:10.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.094+0000 7f7be77fe700 1 --2- 172.21.15.16:0/438312257 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f7bdc06c4e0 0x7f7bdc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:10.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.094+0000 7f7be77fe700 1 -- 172.21.15.16:0/438312257 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7be808b080 con 0x7f7bf00fef60 2024-04-22T15:46:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.094+0000 7f7bf593f700 1 --2- 172.21.15.16:0/438312257 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f7bdc06c4e0 0x7f7bdc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.094+0000 7f7bf593f700 1 --2- 172.21.15.16:0/438312257 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f7bdc06c4e0 0x7f7bdc06e9a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7be000b5c0 tx=0x7f7be0005960 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:10.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.098+0000 7f7be77fe700 1 -- 172.21.15.16:0/438312257 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7be80596b0 con 0x7f7bf00fef60 2024-04-22T15:46:10.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.297+0000 7f7bf83a4700 1 -- 172.21.15.16:0/438312257 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f7bf0109a90 con 0x7f7bf00fef60 2024-04-22T15:46:10.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.301+0000 7f7be77fe700 1 -- 172.21.15.16:0/438312257 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f7be8059240 con 0x7f7bf00fef60 2024-04-22T15:46:10.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.303+0000 7f7bf83a4700 1 -- 172.21.15.16:0/438312257 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f7bdc06c4e0 msgr2=0x7f7bdc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:10.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.303+0000 7f7bf83a4700 1 --2- 172.21.15.16:0/438312257 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f7bdc06c4e0 0x7f7bdc06e9a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7be000b5c0 tx=0x7f7be0005960 comp rx=0 tx=0).stop 2024-04-22T15:46:10.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.303+0000 7f7bf83a4700 1 -- 172.21.15.16:0/438312257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf00fef60 msgr2=0x7f7bf0196550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:10.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.303+0000 7f7bf83a4700 1 --2- 172.21.15.16:0/438312257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf00fef60 0x7f7bf0196550 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7be800b480 tx=0x7f7be800b840 comp rx=0 tx=0).stop 2024-04-22T15:46:10.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.303+0000 7f7bf83a4700 1 -- 172.21.15.16:0/438312257 shutdown_connections 2024-04-22T15:46:10.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.303+0000 7f7bf83a4700 1 --2- 172.21.15.16:0/438312257 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f7bdc06c4e0 0x7f7bdc06e9a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:10.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.303+0000 7f7bf83a4700 1 --2- 172.21.15.16:0/438312257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf00fef60 0x7f7bf0196550 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:10.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.303+0000 7f7bf83a4700 1 --2- 172.21.15.16:0/438312257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bf0101fe0 0x7f7bf0196a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:10.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.303+0000 7f7bf83a4700 1 -- 172.21.15.16:0/438312257 >> 172.21.15.16:0/438312257 conn(0x7f7bf00fa8c0 msgr2=0x7f7bf0105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:10.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.303+0000 7f7bf83a4700 1 -- 172.21.15.16:0/438312257 shutdown_connections 2024-04-22T15:46:10.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:10.303+0000 7f7bf83a4700 1 -- 172.21.15.16:0/438312257 wait complete. 2024-04-22T15:46:10.964 INFO:teuthology.run_tasks:Running task print... 2024-04-22T15:46:10.974 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-04-22T15:46:10.974 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T15:46:11.017 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-22T15:46:11.017 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-22T15:46:11.226 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:46:11.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:11 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/438312257' entity='client.admin' 2024-04-22T15:46:11.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:11 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:46:11.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:11 smithi027 ceph-mon[34922]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:11.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:11 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:46:11.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:11 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:46:11.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:11 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:46:11.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:11 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:11.621 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:11 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/438312257' entity='client.admin' 2024-04-22T15:46:11.621 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:11 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:46:11.621 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:11 smithi016 ceph-mon[26620]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:11.621 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:11 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:46:11.622 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:11 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:46:11.622 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:11 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:46:11.622 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:11 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:12.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.570+0000 7f32bb41a700 1 -- 172.21.15.16:0/3168871245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b40ff590 msgr2=0x7f32b40ff9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:12.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.570+0000 7f32bb41a700 1 --2- 172.21.15.16:0/3168871245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b40ff590 0x7f32b40ff9b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f32a400b0a0 tx=0x7f32a400b3b0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.570+0000 7f32bb41a700 1 -- 172.21.15.16:0/3168871245 shutdown_connections 2024-04-22T15:46:12.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.570+0000 7f32bb41a700 1 --2- 172.21.15.16:0/3168871245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32b40ffef0 0x7f32b4100370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.570+0000 7f32bb41a700 1 --2- 172.21.15.16:0/3168871245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b40ff590 0x7f32b40ff9b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.570+0000 7f32bb41a700 1 -- 172.21.15.16:0/3168871245 >> 172.21.15.16:0/3168871245 conn(0x7f32b40fb190 msgr2=0x7f32b40fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:12.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.570+0000 7f32bb41a700 1 -- 172.21.15.16:0/3168871245 shutdown_connections 2024-04-22T15:46:12.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.570+0000 7f32bb41a700 1 -- 172.21.15.16:0/3168871245 wait complete. 2024-04-22T15:46:12.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.571+0000 7f32bb41a700 1 Processor -- start 2024-04-22T15:46:12.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.571+0000 7f32bb41a700 1 -- start start 2024-04-22T15:46:12.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.571+0000 7f32bb41a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32b40ff590 0x7f32b4198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:12.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.571+0000 7f32bb41a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b40ffef0 0x7f32b4198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:12.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.571+0000 7f32bb41a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b41991d0 con 0x7f32b40ffef0 2024-04-22T15:46:12.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.571+0000 7f32bb41a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b4199310 con 0x7f32b40ff590 2024-04-22T15:46:12.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.571+0000 7f32b89b5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b40ffef0 0x7f32b4198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:12.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.571+0000 7f32b89b5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b40ffef0 0x7f32b4198bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57516/0 (socket says 172.21.15.16:57516) 2024-04-22T15:46:12.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.571+0000 7f32b89b5700 1 -- 172.21.15.16:0/1476096091 learned_addr learned my addr 172.21.15.16:0/1476096091 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:46:12.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.571+0000 7f32b91b6700 1 --2- 172.21.15.16:0/1476096091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32b40ff590 0x7f32b4198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:12.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.572+0000 7f32b89b5700 1 -- 172.21.15.16:0/1476096091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32b40ff590 msgr2=0x7f32b4198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:12.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.572+0000 7f32b89b5700 1 --2- 172.21.15.16:0/1476096091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32b40ff590 0x7f32b4198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.572+0000 7f32b89b5700 1 -- 172.21.15.16:0/1476096091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a4009d00 con 0x7f32b40ffef0 2024-04-22T15:46:12.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.572+0000 7f32b89b5700 1 --2- 172.21.15.16:0/1476096091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b40ffef0 0x7f32b4198bb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f32a800c8a0 tx=0x7f32a800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:12.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.572+0000 7f32b27fc700 1 -- 172.21.15.16:0/1476096091 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32a80078e0 con 0x7f32b40ffef0 2024-04-22T15:46:12.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.572+0000 7f32b27fc700 1 -- 172.21.15.16:0/1476096091 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f32a800f450 con 0x7f32b40ffef0 2024-04-22T15:46:12.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.572+0000 7f32bb41a700 1 -- 172.21.15.16:0/1476096091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b419ddc0 con 0x7f32b40ffef0 2024-04-22T15:46:12.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.572+0000 7f32b27fc700 1 -- 172.21.15.16:0/1476096091 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32a800e5e0 con 0x7f32b40ffef0 2024-04-22T15:46:12.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.572+0000 7f32bb41a700 1 -- 172.21.15.16:0/1476096091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b419e2e0 con 0x7f32b40ffef0 2024-04-22T15:46:12.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.573+0000 7f32bb41a700 1 -- 172.21.15.16:0/1476096091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b410b9f0 con 0x7f32b40ffef0 2024-04-22T15:46:12.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.573+0000 7f32b27fc700 1 -- 172.21.15.16:0/1476096091 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f32a800f5c0 con 0x7f32b40ffef0 2024-04-22T15:46:12.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.574+0000 7f32b27fc700 1 --2- 172.21.15.16:0/1476096091 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f329c06c4e0 0x7f329c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:12.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.574+0000 7f32b27fc700 1 -- 172.21.15.16:0/1476096091 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f32a808b380 con 0x7f32b40ffef0 2024-04-22T15:46:12.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.574+0000 7f32b91b6700 1 --2- 172.21.15.16:0/1476096091 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f329c06c4e0 0x7f329c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:12.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.574+0000 7f32b91b6700 1 --2- 172.21.15.16:0/1476096091 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f329c06c4e0 0x7f329c06e9a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f32a4000f80 tx=0x7f32a40060c0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:12.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.578+0000 7f32b27fc700 1 -- 172.21.15.16:0/1476096091 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f32a80599b0 con 0x7f32b40ffef0 2024-04-22T15:46:12.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.792+0000 7f32bb41a700 1 -- 172.21.15.16:0/1476096091 --> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f32b41092e0 con 0x7f329c06c4e0 2024-04-22T15:46:12.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.793+0000 7f32b27fc700 1 -- 172.21.15.16:0/1476096091 <== mgr.14217 v2:172.21.15.16:6800/1057234399 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f32b41092e0 con 0x7f329c06c4e0 2024-04-22T15:46:12.796 INFO:teuthology.orchestra.run.smithi016.stdout:Backend: cephadm 2024-04-22T15:46:12.797 INFO:teuthology.orchestra.run.smithi016.stdout:Available: Yes 2024-04-22T15:46:12.797 INFO:teuthology.orchestra.run.smithi016.stdout:Paused: No 2024-04-22T15:46:12.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.795+0000 7f32bb41a700 1 -- 172.21.15.16:0/1476096091 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f329c06c4e0 msgr2=0x7f329c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:12.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.795+0000 7f32bb41a700 1 --2- 172.21.15.16:0/1476096091 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f329c06c4e0 0x7f329c06e9a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f32a4000f80 tx=0x7f32a40060c0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.795+0000 7f32bb41a700 1 -- 172.21.15.16:0/1476096091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b40ffef0 msgr2=0x7f32b4198bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:12.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.795+0000 7f32bb41a700 1 --2- 172.21.15.16:0/1476096091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b40ffef0 0x7f32b4198bb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f32a800c8a0 tx=0x7f32a800cbb0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.795+0000 7f32bb41a700 1 -- 172.21.15.16:0/1476096091 shutdown_connections 2024-04-22T15:46:12.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.795+0000 7f32bb41a700 1 --2- 172.21.15.16:0/1476096091 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f329c06c4e0 0x7f329c06e9a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.795+0000 7f32bb41a700 1 --2- 172.21.15.16:0/1476096091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32b40ff590 0x7f32b4198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.795+0000 7f32bb41a700 1 --2- 172.21.15.16:0/1476096091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b40ffef0 0x7f32b4198bb0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:12.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.795+0000 7f32bb41a700 1 -- 172.21.15.16:0/1476096091 >> 172.21.15.16:0/1476096091 conn(0x7f32b40fb190 msgr2=0x7f32b4107bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:12.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.796+0000 7f32bb41a700 1 -- 172.21.15.16:0/1476096091 shutdown_connections 2024-04-22T15:46:12.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:12.796+0000 7f32bb41a700 1 -- 172.21.15.16:0/1476096091 wait complete. 2024-04-22T15:46:13.371 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-22T15:46:13.592 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:46:13.657 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:13 smithi016 ceph-mon[26620]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:13.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:13 smithi027 ceph-mon[34922]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:14.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:14 smithi016 ceph-mon[26620]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:14.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:14 smithi027 ceph-mon[34922]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:15.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.193+0000 7f35b2891700 1 -- 172.21.15.16:0/1264660017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35ac100cb0 msgr2=0x7f35ac1010d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:15.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.193+0000 7f35b2891700 1 --2- 172.21.15.16:0/1264660017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35ac100cb0 0x7f35ac1010d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f359c009910 tx=0x7f359c009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:15.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.193+0000 7f35b2891700 1 -- 172.21.15.16:0/1264660017 shutdown_connections 2024-04-22T15:46:15.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.193+0000 7f35b2891700 1 --2- 172.21.15.16:0/1264660017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35ac101eb0 0x7f35ac102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:15.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.193+0000 7f35b2891700 1 --2- 172.21.15.16:0/1264660017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35ac100cb0 0x7f35ac1010d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:15.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.193+0000 7f35b2891700 1 -- 172.21.15.16:0/1264660017 >> 172.21.15.16:0/1264660017 conn(0x7f35ac0fc230 msgr2=0x7f35ac0fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:15.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.193+0000 7f35b2891700 1 -- 172.21.15.16:0/1264660017 shutdown_connections 2024-04-22T15:46:15.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.193+0000 7f35b2891700 1 -- 172.21.15.16:0/1264660017 wait complete. 2024-04-22T15:46:15.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.194+0000 7f35b2891700 1 Processor -- start 2024-04-22T15:46:15.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.194+0000 7f35b2891700 1 -- start start 2024-04-22T15:46:15.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.195+0000 7f35b2891700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35ac100cb0 0x7f35ac196560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:15.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.195+0000 7f35b2891700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35ac101eb0 0x7f35ac196aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:15.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.195+0000 7f35b2891700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac1970c0 con 0x7f35ac101eb0 2024-04-22T15:46:15.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.195+0000 7f35b2891700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac197200 con 0x7f35ac100cb0 2024-04-22T15:46:15.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.195+0000 7f35ab7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35ac101eb0 0x7f35ac196aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:15.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.195+0000 7f35ab7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35ac101eb0 0x7f35ac196aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57542/0 (socket says 172.21.15.16:57542) 2024-04-22T15:46:15.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.195+0000 7f35ab7fe700 1 -- 172.21.15.16:0/2745303083 learned_addr learned my addr 172.21.15.16:0/2745303083 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:46:15.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.195+0000 7f35abfff700 1 --2- 172.21.15.16:0/2745303083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35ac100cb0 0x7f35ac196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:15.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.195+0000 7f35ab7fe700 1 -- 172.21.15.16:0/2745303083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35ac100cb0 msgr2=0x7f35ac196560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:15.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.195+0000 7f35ab7fe700 1 --2- 172.21.15.16:0/2745303083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35ac100cb0 0x7f35ac196560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:15.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.195+0000 7f35ab7fe700 1 -- 172.21.15.16:0/2745303083 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359c0095f0 con 0x7f35ac101eb0 2024-04-22T15:46:15.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.195+0000 7f35ab7fe700 1 --2- 172.21.15.16:0/2745303083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35ac101eb0 0x7f35ac196aa0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f35a000b4b0 tx=0x7f35a000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:15.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.195+0000 7f35a97fa700 1 -- 172.21.15.16:0/2745303083 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35a0011800 con 0x7f35ac101eb0 2024-04-22T15:46:15.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.196+0000 7f35b2891700 1 -- 172.21.15.16:0/2745303083 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35ac19bcb0 con 0x7f35ac101eb0 2024-04-22T15:46:15.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.196+0000 7f35a97fa700 1 -- 172.21.15.16:0/2745303083 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f35a0011e10 con 0x7f35ac101eb0 2024-04-22T15:46:15.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.196+0000 7f35a97fa700 1 -- 172.21.15.16:0/2745303083 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35a000f2e0 con 0x7f35ac101eb0 2024-04-22T15:46:15.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.196+0000 7f35b2891700 1 -- 172.21.15.16:0/2745303083 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35ac19c200 con 0x7f35ac101eb0 2024-04-22T15:46:15.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.197+0000 7f35b2891700 1 -- 172.21.15.16:0/2745303083 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35ac109890 con 0x7f35ac101eb0 2024-04-22T15:46:15.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.197+0000 7f35a97fa700 1 -- 172.21.15.16:0/2745303083 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f35a0010380 con 0x7f35ac101eb0 2024-04-22T15:46:15.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.198+0000 7f35a97fa700 1 --2- 172.21.15.16:0/2745303083 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f359406c4e0 0x7f359406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:15.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.198+0000 7f35a97fa700 1 -- 172.21.15.16:0/2745303083 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f35a008b0e0 con 0x7f35ac101eb0 2024-04-22T15:46:15.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.198+0000 7f35abfff700 1 --2- 172.21.15.16:0/2745303083 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f359406c4e0 0x7f359406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:15.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.198+0000 7f35abfff700 1 --2- 172.21.15.16:0/2745303083 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f359406c4e0 0x7f359406e9a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f359c006010 tx=0x7f359c00b560 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:15.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.203+0000 7f35a97fa700 1 -- 172.21.15.16:0/2745303083 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f35a0059630 con 0x7f35ac101eb0 2024-04-22T15:46:15.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.408+0000 7f35b2891700 1 -- 172.21.15.16:0/2745303083 --> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f35ac106800 con 0x7f359406c4e0 2024-04-22T15:46:15.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.417+0000 7f35a97fa700 1 -- 172.21.15.16:0/2745303083 <== mgr.14217 v2:172.21.15.16:6800/1057234399 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2820 (secure 0 0 0) 0x7f35ac106800 con 0x7f359406c4e0 2024-04-22T15:46:15.420 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:46:15.420 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.smithi016 smithi016 *:9093,9094 running (3m) 2m ago 5m 26.6M - 0.25.0 c8568f914cd2 fab40987f1b0 2024-04-22T15:46:15.421 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi016 smithi016 running (5m) 2m ago 5m 7067k - 18.2.1 5be31c24972a f8b627d3f41e 2024-04-22T15:46:15.421 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi027 smithi027 running (4m) 65s ago 4m 6962k - 18.2.1 5be31c24972a f00fd40b4220 2024-04-22T15:46:15.421 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi016 smithi016 running (5m) 2m ago 5m 6656k - 18.2.1 5be31c24972a 6b93db1fa0d9 2024-04-22T15:46:15.421 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi027 smithi027 running (4m) 65s ago 4m 6665k - 18.2.1 5be31c24972a d3bd439f30a0 2024-04-22T15:46:15.421 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.smithi016 smithi016 *:3000 running (3m) 2m ago 4m 80.5M - 9.4.7 954c08fa6188 a85bdbf5073d 2024-04-22T15:46:15.421 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi016.rnsnnr smithi016 *:9283,8765,8443 running (6m) 2m ago 6m 479M - 18.2.1 5be31c24972a 1cf5a7efea72 2024-04-22T15:46:15.421 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi027.ukdgzs smithi027 *:8443,9283,8765 running (4m) 65s ago 4m 441M - 18.2.1 5be31c24972a c59a9cac2d99 2024-04-22T15:46:15.421 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi016 smithi016 running (6m) 2m ago 6m 46.4M 2048M 18.2.1 5be31c24972a 72af40873cb6 2024-04-22T15:46:15.421 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi027 smithi027 running (4m) 65s ago 4m 40.9M 2048M 18.2.1 5be31c24972a 9f21ce7fa7ef 2024-04-22T15:46:15.421 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi016 smithi016 *:9100 running (5m) 2m ago 5m 15.5M - 1.5.0 0da6a335fe13 1937a0bc1959 2024-04-22T15:46:15.421 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi027 smithi027 *:9100 running (4m) 65s ago 4m 16.0M - 1.5.0 0da6a335fe13 00440c991b3f 2024-04-22T15:46:15.421 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (3m) 2m ago 3m 40.7M 3948M 18.2.1 5be31c24972a 3c6ebc4ea041 2024-04-22T15:46:15.421 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (2m) 2m ago 2m 39.5M 3948M 18.2.1 5be31c24972a 6aa2e6796784 2024-04-22T15:46:15.422 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (2m) 2m ago 2m 38.1M 3948M 18.2.1 5be31c24972a 3d9d310aa9eb 2024-04-22T15:46:15.422 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi027 running (2m) 65s ago 2m 44.4M 4970M 18.2.1 5be31c24972a cdf9b7aaf865 2024-04-22T15:46:15.422 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (96s) 65s ago 95s 38.6M 4970M 18.2.1 5be31c24972a 629397d49c76 2024-04-22T15:46:15.422 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi027 running (71s) 65s ago 70s 13.9M 4970M 18.2.1 5be31c24972a 23adcf675b04 2024-04-22T15:46:15.422 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.smithi016 smithi016 *:9095 running (3m) 2m ago 4m 41.4M - 2.43.0 a07b618ecd1d 2ba3b93ebe19 2024-04-22T15:46:15.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.419+0000 7f35b2891700 1 -- 172.21.15.16:0/2745303083 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f359406c4e0 msgr2=0x7f359406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:15.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.419+0000 7f35b2891700 1 --2- 172.21.15.16:0/2745303083 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f359406c4e0 0x7f359406e9a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f359c006010 tx=0x7f359c00b560 comp rx=0 tx=0).stop 2024-04-22T15:46:15.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.419+0000 7f35b2891700 1 -- 172.21.15.16:0/2745303083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35ac101eb0 msgr2=0x7f35ac196aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:15.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.419+0000 7f35b2891700 1 --2- 172.21.15.16:0/2745303083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35ac101eb0 0x7f35ac196aa0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f35a000b4b0 tx=0x7f35a000b7c0 comp rx=0 tx=0).stop 2024-04-22T15:46:15.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.419+0000 7f35b2891700 1 -- 172.21.15.16:0/2745303083 shutdown_connections 2024-04-22T15:46:15.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.419+0000 7f35b2891700 1 --2- 172.21.15.16:0/2745303083 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f359406c4e0 0x7f359406e9a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:15.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.419+0000 7f35b2891700 1 --2- 172.21.15.16:0/2745303083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35ac100cb0 0x7f35ac196560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:15.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.419+0000 7f35b2891700 1 --2- 172.21.15.16:0/2745303083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35ac101eb0 0x7f35ac196aa0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:15.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.419+0000 7f35b2891700 1 -- 172.21.15.16:0/2745303083 >> 172.21.15.16:0/2745303083 conn(0x7f35ac0fc230 msgr2=0x7f35ac1050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:15.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.419+0000 7f35b2891700 1 -- 172.21.15.16:0/2745303083 shutdown_connections 2024-04-22T15:46:15.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:15.419+0000 7f35b2891700 1 -- 172.21.15.16:0/2745303083 wait complete. 2024-04-22T15:46:15.804 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:15 smithi016 ceph-mon[26620]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:15.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:15 smithi027 ceph-mon[34922]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:16.087 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-22T15:46:16.303 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:46:16.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:16 smithi016 ceph-mon[26620]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:16.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:16 smithi027 ceph-mon[34922]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:17.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:17 smithi016 ceph-mon[26620]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:17.803 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:17 smithi027 ceph-mon[34922]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:18.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.124+0000 7f9531037700 1 -- 172.21.15.16:0/2491382312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f952c101eb0 msgr2=0x7f952c102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:18.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.124+0000 7f9531037700 1 --2- 172.21.15.16:0/2491382312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f952c101eb0 0x7f952c102310 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9520009910 tx=0x7f9520009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:18.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.124+0000 7f9531037700 1 -- 172.21.15.16:0/2491382312 shutdown_connections 2024-04-22T15:46:18.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.124+0000 7f9531037700 1 --2- 172.21.15.16:0/2491382312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f952c101eb0 0x7f952c102310 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:18.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.124+0000 7f9531037700 1 --2- 172.21.15.16:0/2491382312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f952c100cb0 0x7f952c1010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:18.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.124+0000 7f9531037700 1 -- 172.21.15.16:0/2491382312 >> 172.21.15.16:0/2491382312 conn(0x7f952c0fc230 msgr2=0x7f952c0fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:18.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.125+0000 7f9531037700 1 -- 172.21.15.16:0/2491382312 shutdown_connections 2024-04-22T15:46:18.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.125+0000 7f9531037700 1 -- 172.21.15.16:0/2491382312 wait complete. 2024-04-22T15:46:18.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.125+0000 7f9531037700 1 Processor -- start 2024-04-22T15:46:18.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.125+0000 7f9531037700 1 -- start start 2024-04-22T15:46:18.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.126+0000 7f9531037700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f952c100cb0 0x7f952c1964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:18.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.126+0000 7f9531037700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f952c101eb0 0x7f952c1969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:18.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.126+0000 7f9531037700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c197010 con 0x7f952c100cb0 2024-04-22T15:46:18.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.126+0000 7f9531037700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c197150 con 0x7f952c101eb0 2024-04-22T15:46:18.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.126+0000 7f952ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f952c100cb0 0x7f952c1964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:18.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.126+0000 7f952ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f952c100cb0 0x7f952c1964b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57570/0 (socket says 172.21.15.16:57570) 2024-04-22T15:46:18.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.126+0000 7f952ad9d700 1 -- 172.21.15.16:0/506133906 learned_addr learned my addr 172.21.15.16:0/506133906 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:46:18.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.126+0000 7f952a59c700 1 --2- 172.21.15.16:0/506133906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f952c101eb0 0x7f952c1969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:18.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.126+0000 7f952ad9d700 1 -- 172.21.15.16:0/506133906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f952c101eb0 msgr2=0x7f952c1969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:18.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.126+0000 7f952ad9d700 1 --2- 172.21.15.16:0/506133906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f952c101eb0 0x7f952c1969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:18.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.126+0000 7f952ad9d700 1 -- 172.21.15.16:0/506133906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95200095f0 con 0x7f952c100cb0 2024-04-22T15:46:18.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.126+0000 7f952ad9d700 1 --2- 172.21.15.16:0/506133906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f952c100cb0 0x7f952c1964b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f951c00e8f0 tx=0x7f951c00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:18.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.126+0000 7f9513fff700 1 -- 172.21.15.16:0/506133906 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f951c009790 con 0x7f952c100cb0 2024-04-22T15:46:18.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.127+0000 7f9531037700 1 -- 172.21.15.16:0/506133906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f952c19bc00 con 0x7f952c100cb0 2024-04-22T15:46:18.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.127+0000 7f9513fff700 1 -- 172.21.15.16:0/506133906 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f951c00c810 con 0x7f952c100cb0 2024-04-22T15:46:18.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.127+0000 7f9513fff700 1 -- 172.21.15.16:0/506133906 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f951c003d20 con 0x7f952c100cb0 2024-04-22T15:46:18.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.127+0000 7f9531037700 1 -- 172.21.15.16:0/506133906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f952c19c150 con 0x7f952c100cb0 2024-04-22T15:46:18.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.127+0000 7f9531037700 1 -- 172.21.15.16:0/506133906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f952c109810 con 0x7f952c100cb0 2024-04-22T15:46:18.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.128+0000 7f9513fff700 1 -- 172.21.15.16:0/506133906 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f951c0041f0 con 0x7f952c100cb0 2024-04-22T15:46:18.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.128+0000 7f9513fff700 1 --2- 172.21.15.16:0/506133906 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f951406c4e0 0x7f951406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:18.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.129+0000 7f9513fff700 1 -- 172.21.15.16:0/506133906 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f951c014070 con 0x7f952c100cb0 2024-04-22T15:46:18.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.129+0000 7f952a59c700 1 --2- 172.21.15.16:0/506133906 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f951406c4e0 0x7f951406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:18.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.129+0000 7f952a59c700 1 --2- 172.21.15.16:0/506133906 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f951406c4e0 0x7f951406e9a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f952000b6f0 tx=0x7f9520005fb0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:18.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.133+0000 7f9513fff700 1 -- 172.21.15.16:0/506133906 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f951c059d90 con 0x7f952c100cb0 2024-04-22T15:46:18.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.342+0000 7f9531037700 1 -- 172.21.15.16:0/506133906 --> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f952c106800 con 0x7f951406c4e0 2024-04-22T15:46:18.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.347+0000 7f9513fff700 1 -- 172.21.15.16:0/506133906 <== mgr.14217 v2:172.21.15.16:6800/1057234399 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1290 (secure 0 0 0) 0x7f952c106800 con 0x7f951406c4e0 2024-04-22T15:46:18.350 INFO:teuthology.orchestra.run.smithi016.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-22T15:46:18.350 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-04-22T15:46:18.350 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter 2/2 2m ago 6m * 2024-04-22T15:46:18.350 INFO:teuthology.orchestra.run.smithi016.stdout:crash 2/2 2m ago 6m * 2024-04-22T15:46:18.350 INFO:teuthology.orchestra.run.smithi016.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-04-22T15:46:18.350 INFO:teuthology.orchestra.run.smithi016.stdout:mgr 2/2 2m ago 6m count:2 2024-04-22T15:46:18.350 INFO:teuthology.orchestra.run.smithi016.stdout:mon 2/2 2m ago 5m smithi016:172.21.15.16=smithi016;smithi027:172.21.15.27=smithi027;count:2 2024-04-22T15:46:18.351 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-04-22T15:46:18.351 INFO:teuthology.orchestra.run.smithi016.stdout:osd 6 2m ago - 2024-04-22T15:46:18.351 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-04-22T15:46:18.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.349+0000 7f9531037700 1 -- 172.21.15.16:0/506133906 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f951406c4e0 msgr2=0x7f951406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:18.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.349+0000 7f9531037700 1 --2- 172.21.15.16:0/506133906 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f951406c4e0 0x7f951406e9a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f952000b6f0 tx=0x7f9520005fb0 comp rx=0 tx=0).stop 2024-04-22T15:46:18.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.349+0000 7f9531037700 1 -- 172.21.15.16:0/506133906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f952c100cb0 msgr2=0x7f952c1964b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:18.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.349+0000 7f9531037700 1 --2- 172.21.15.16:0/506133906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f952c100cb0 0x7f952c1964b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f951c00e8f0 tx=0x7f951c00ec00 comp rx=0 tx=0).stop 2024-04-22T15:46:18.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.349+0000 7f9531037700 1 -- 172.21.15.16:0/506133906 shutdown_connections 2024-04-22T15:46:18.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.349+0000 7f9531037700 1 --2- 172.21.15.16:0/506133906 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f951406c4e0 0x7f951406e9a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:18.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.349+0000 7f9531037700 1 --2- 172.21.15.16:0/506133906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f952c100cb0 0x7f952c1964b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:18.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.349+0000 7f9531037700 1 --2- 172.21.15.16:0/506133906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f952c101eb0 0x7f952c1969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:18.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.349+0000 7f9531037700 1 -- 172.21.15.16:0/506133906 >> 172.21.15.16:0/506133906 conn(0x7f952c0fc230 msgr2=0x7f952c1050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:18.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.349+0000 7f9531037700 1 -- 172.21.15.16:0/506133906 shutdown_connections 2024-04-22T15:46:18.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:18.349+0000 7f9531037700 1 -- 172.21.15.16:0/506133906 wait complete. 2024-04-22T15:46:19.181 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-22T15:46:19.400 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:46:19.797 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:19 smithi016 ceph-mon[26620]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:19.797 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:19 smithi016 ceph-mon[26620]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:19.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:19 smithi027 ceph-mon[34922]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:19.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:19 smithi027 ceph-mon[34922]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:21.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.103+0000 7f43736d2700 1 -- 172.21.15.16:0/1853814009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f436c1017a0 msgr2=0x7f436c103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:21.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.103+0000 7f43736d2700 1 --2- 172.21.15.16:0/1853814009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f436c1017a0 0x7f436c103b90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f436800b0a0 tx=0x7f436800b3b0 comp rx=0 tx=0).stop 2024-04-22T15:46:21.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.103+0000 7f43736d2700 1 -- 172.21.15.16:0/1853814009 shutdown_connections 2024-04-22T15:46:21.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.103+0000 7f43736d2700 1 --2- 172.21.15.16:0/1853814009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f436c1040d0 0x7f436c1064c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:21.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.103+0000 7f43736d2700 1 --2- 172.21.15.16:0/1853814009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f436c1017a0 0x7f436c103b90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:21.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.103+0000 7f43736d2700 1 -- 172.21.15.16:0/1853814009 >> 172.21.15.16:0/1853814009 conn(0x7f436c0fb190 msgr2=0x7f436c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:21.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.103+0000 7f43736d2700 1 -- 172.21.15.16:0/1853814009 shutdown_connections 2024-04-22T15:46:21.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.103+0000 7f43736d2700 1 -- 172.21.15.16:0/1853814009 wait complete. 2024-04-22T15:46:21.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.104+0000 7f43736d2700 1 Processor -- start 2024-04-22T15:46:21.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.104+0000 7f43736d2700 1 -- start start 2024-04-22T15:46:21.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.104+0000 7f43736d2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f436c1017a0 0x7f436c196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:21.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.104+0000 7f43736d2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f436c1040d0 0x7f436c1969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:21.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.104+0000 7f43736d2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436c196fc0 con 0x7f436c1040d0 2024-04-22T15:46:21.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.104+0000 7f43736d2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436c197100 con 0x7f436c1017a0 2024-04-22T15:46:21.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.104+0000 7f4370c6d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f436c1040d0 0x7f436c1969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:21.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.104+0000 7f4370c6d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f436c1040d0 0x7f436c1969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57596/0 (socket says 172.21.15.16:57596) 2024-04-22T15:46:21.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.104+0000 7f4370c6d700 1 -- 172.21.15.16:0/105013202 learned_addr learned my addr 172.21.15.16:0/105013202 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:46:21.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.104+0000 7f437146e700 1 --2- 172.21.15.16:0/105013202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f436c1017a0 0x7f436c196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:21.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.105+0000 7f4370c6d700 1 -- 172.21.15.16:0/105013202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f436c1017a0 msgr2=0x7f436c196460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:21.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.105+0000 7f4370c6d700 1 --2- 172.21.15.16:0/105013202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f436c1017a0 0x7f436c196460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:21.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.105+0000 7f4370c6d700 1 -- 172.21.15.16:0/105013202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4368009d00 con 0x7f436c1040d0 2024-04-22T15:46:21.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.105+0000 7f4370c6d700 1 --2- 172.21.15.16:0/105013202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f436c1040d0 0x7f436c1969a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f435c00d8d0 tx=0x7f435c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:21.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.105+0000 7f43627fc700 1 -- 172.21.15.16:0/105013202 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f435c0098a0 con 0x7f436c1040d0 2024-04-22T15:46:21.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.105+0000 7f43627fc700 1 -- 172.21.15.16:0/105013202 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f435c010460 con 0x7f436c1040d0 2024-04-22T15:46:21.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.105+0000 7f43736d2700 1 -- 172.21.15.16:0/105013202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f436c19bbb0 con 0x7f436c1040d0 2024-04-22T15:46:21.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.105+0000 7f43627fc700 1 -- 172.21.15.16:0/105013202 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f435c00f5f0 con 0x7f436c1040d0 2024-04-22T15:46:21.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.105+0000 7f43736d2700 1 -- 172.21.15.16:0/105013202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f436c19c100 con 0x7f436c1040d0 2024-04-22T15:46:21.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.106+0000 7f43736d2700 1 -- 172.21.15.16:0/105013202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f436c190610 con 0x7f436c1040d0 2024-04-22T15:46:21.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.107+0000 7f43627fc700 1 -- 172.21.15.16:0/105013202 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f435c009a00 con 0x7f436c1040d0 2024-04-22T15:46:21.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.107+0000 7f43627fc700 1 --2- 172.21.15.16:0/105013202 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f435806c4e0 0x7f435806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:21.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.107+0000 7f43627fc700 1 -- 172.21.15.16:0/105013202 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f435c08b2f0 con 0x7f436c1040d0 2024-04-22T15:46:21.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.107+0000 7f437146e700 1 --2- 172.21.15.16:0/105013202 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f435806c4e0 0x7f435806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:21.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.108+0000 7f437146e700 1 --2- 172.21.15.16:0/105013202 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f435806c4e0 0x7f435806e9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4368009fd0 tx=0x7f43680060c0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:21.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.112+0000 7f43627fc700 1 -- 172.21.15.16:0/105013202 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f435c059920 con 0x7f436c1040d0 2024-04-22T15:46:21.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.316+0000 7f43736d2700 1 -- 172.21.15.16:0/105013202 --> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f436c0611d0 con 0x7f435806c4e0 2024-04-22T15:46:21.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.317+0000 7f43627fc700 1 -- 172.21.15.16:0/105013202 <== mgr.14217 v2:172.21.15.16:6800/1057234399 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+145 (secure 0 0 0) 0x7f436c0611d0 con 0x7f435806c4e0 2024-04-22T15:46:21.321 INFO:teuthology.orchestra.run.smithi016.stdout:HOST ADDR LABELS STATUS 2024-04-22T15:46:21.321 INFO:teuthology.orchestra.run.smithi016.stdout:smithi016 172.21.15.16 2024-04-22T15:46:21.321 INFO:teuthology.orchestra.run.smithi016.stdout:smithi027 172.21.15.27 2024-04-22T15:46:21.321 INFO:teuthology.orchestra.run.smithi016.stdout:2 hosts in cluster 2024-04-22T15:46:21.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.320+0000 7f43736d2700 1 -- 172.21.15.16:0/105013202 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f435806c4e0 msgr2=0x7f435806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:21.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.320+0000 7f43736d2700 1 --2- 172.21.15.16:0/105013202 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f435806c4e0 0x7f435806e9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4368009fd0 tx=0x7f43680060c0 comp rx=0 tx=0).stop 2024-04-22T15:46:21.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.320+0000 7f43736d2700 1 -- 172.21.15.16:0/105013202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f436c1040d0 msgr2=0x7f436c1969a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:21.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.320+0000 7f43736d2700 1 --2- 172.21.15.16:0/105013202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f436c1040d0 0x7f436c1969a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f435c00d8d0 tx=0x7f435c00dbe0 comp rx=0 tx=0).stop 2024-04-22T15:46:21.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.320+0000 7f43736d2700 1 -- 172.21.15.16:0/105013202 shutdown_connections 2024-04-22T15:46:21.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.320+0000 7f43736d2700 1 --2- 172.21.15.16:0/105013202 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f435806c4e0 0x7f435806e9a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:21.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.320+0000 7f43736d2700 1 --2- 172.21.15.16:0/105013202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f436c1017a0 0x7f436c196460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:21.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.320+0000 7f43736d2700 1 --2- 172.21.15.16:0/105013202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f436c1040d0 0x7f436c1969a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:21.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.320+0000 7f43736d2700 1 -- 172.21.15.16:0/105013202 >> 172.21.15.16:0/105013202 conn(0x7f436c0fb190 msgr2=0x7f436c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:21.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.320+0000 7f43736d2700 1 -- 172.21.15.16:0/105013202 shutdown_connections 2024-04-22T15:46:21.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:21.320+0000 7f43736d2700 1 -- 172.21.15.16:0/105013202 wait complete. 2024-04-22T15:46:21.765 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:21 smithi016 ceph-mon[26620]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:21.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:21 smithi027 ceph-mon[34922]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:21.908 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-22T15:46:22.119 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:46:22.800 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:22 smithi016 ceph-mon[26620]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:22.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:22 smithi027 ceph-mon[34922]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:23.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.597+0000 7f6b4bdee700 1 -- 172.21.15.16:0/1853871639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b44101eb0 msgr2=0x7f6b44102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:23.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.597+0000 7f6b4bdee700 1 --2- 172.21.15.16:0/1853871639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b44101eb0 0x7f6b44102310 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6b34009910 tx=0x7f6b34009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:23.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.597+0000 7f6b4bdee700 1 -- 172.21.15.16:0/1853871639 shutdown_connections 2024-04-22T15:46:23.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.597+0000 7f6b4bdee700 1 --2- 172.21.15.16:0/1853871639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b44101eb0 0x7f6b44102310 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:23.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.597+0000 7f6b4bdee700 1 --2- 172.21.15.16:0/1853871639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b44100cb0 0x7f6b441010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:23.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.597+0000 7f6b4bdee700 1 -- 172.21.15.16:0/1853871639 >> 172.21.15.16:0/1853871639 conn(0x7f6b440fc230 msgr2=0x7f6b440fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:23.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.598+0000 7f6b4bdee700 1 -- 172.21.15.16:0/1853871639 shutdown_connections 2024-04-22T15:46:23.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.598+0000 7f6b4bdee700 1 -- 172.21.15.16:0/1853871639 wait complete. 2024-04-22T15:46:23.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.598+0000 7f6b4bdee700 1 Processor -- start 2024-04-22T15:46:23.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.598+0000 7f6b4bdee700 1 -- start start 2024-04-22T15:46:23.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.598+0000 7f6b4bdee700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b44100cb0 0x7f6b44074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:23.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.598+0000 7f6b4bdee700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b44101eb0 0x7f6b44073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:23.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.598+0000 7f6b4bdee700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b440736c0 con 0x7f6b44101eb0 2024-04-22T15:46:23.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.598+0000 7f6b4bdee700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b44073800 con 0x7f6b44100cb0 2024-04-22T15:46:23.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.598+0000 7f6b49389700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b44101eb0 0x7f6b44073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:23.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.598+0000 7f6b49389700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b44101eb0 0x7f6b44073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59798/0 (socket says 172.21.15.16:59798) 2024-04-22T15:46:23.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.598+0000 7f6b49389700 1 -- 172.21.15.16:0/923840705 learned_addr learned my addr 172.21.15.16:0/923840705 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:46:23.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.598+0000 7f6b49b8a700 1 --2- 172.21.15.16:0/923840705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b44100cb0 0x7f6b44074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:23.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.599+0000 7f6b49389700 1 -- 172.21.15.16:0/923840705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b44100cb0 msgr2=0x7f6b44074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:23.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.599+0000 7f6b49389700 1 --2- 172.21.15.16:0/923840705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b44100cb0 0x7f6b44074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:23.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.599+0000 7f6b49389700 1 -- 172.21.15.16:0/923840705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b340095f0 con 0x7f6b44101eb0 2024-04-22T15:46:23.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.599+0000 7f6b49389700 1 --2- 172.21.15.16:0/923840705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b44101eb0 0x7f6b44073180 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6b34006010 tx=0x7f6b3400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:23.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.599+0000 7f6b3affd700 1 -- 172.21.15.16:0/923840705 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b3401d070 con 0x7f6b44101eb0 2024-04-22T15:46:23.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.599+0000 7f6b3affd700 1 -- 172.21.15.16:0/923840705 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6b3400f460 con 0x7f6b44101eb0 2024-04-22T15:46:23.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.599+0000 7f6b4bdee700 1 -- 172.21.15.16:0/923840705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b44073a80 con 0x7f6b44101eb0 2024-04-22T15:46:23.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.599+0000 7f6b3affd700 1 -- 172.21.15.16:0/923840705 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b34021640 con 0x7f6b44101eb0 2024-04-22T15:46:23.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.599+0000 7f6b4bdee700 1 -- 172.21.15.16:0/923840705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b44073f70 con 0x7f6b44101eb0 2024-04-22T15:46:23.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.600+0000 7f6b4bdee700 1 -- 172.21.15.16:0/923840705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b44109810 con 0x7f6b44101eb0 2024-04-22T15:46:23.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.601+0000 7f6b3affd700 1 -- 172.21.15.16:0/923840705 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f6b3400fab0 con 0x7f6b44101eb0 2024-04-22T15:46:23.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.601+0000 7f6b3affd700 1 --2- 172.21.15.16:0/923840705 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f6b3006c4e0 0x7f6b3006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:23.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.601+0000 7f6b3affd700 1 -- 172.21.15.16:0/923840705 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f6b3408c730 con 0x7f6b44101eb0 2024-04-22T15:46:23.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.601+0000 7f6b49b8a700 1 --2- 172.21.15.16:0/923840705 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f6b3006c4e0 0x7f6b3006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:23.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.602+0000 7f6b49b8a700 1 --2- 172.21.15.16:0/923840705 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f6b3006c4e0 0x7f6b3006e9a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6b44101d10 tx=0x7f6b4000b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:23.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.606+0000 7f6b3affd700 1 -- 172.21.15.16:0/923840705 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6b3405ace0 con 0x7f6b44101eb0 2024-04-22T15:46:23.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:23 smithi016 ceph-mon[26620]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:23.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.810+0000 7f6b4bdee700 1 -- 172.21.15.16:0/923840705 --> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b4419e960 con 0x7f6b3006c4e0 2024-04-22T15:46:23.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.811+0000 7f6b3affd700 1 -- 172.21.15.16:0/923840705 <== mgr.14217 v2:172.21.15.16:6800/1057234399 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f6b4419e960 con 0x7f6b3006c4e0 2024-04-22T15:46:23.814 INFO:teuthology.orchestra.run.smithi016.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-22T15:46:23.815 INFO:teuthology.orchestra.run.smithi016.stdout:smithi016 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513002C480DGN 447G No 2m ago Has a FileSystem, LVM detected 2024-04-22T15:46:23.815 INFO:teuthology.orchestra.run.smithi016.stdout:smithi027 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5331001C400BGN 372G No 70s ago Has a FileSystem, LVM detected 2024-04-22T15:46:23.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.813+0000 7f6b4bdee700 1 -- 172.21.15.16:0/923840705 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f6b3006c4e0 msgr2=0x7f6b3006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:23.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.813+0000 7f6b4bdee700 1 --2- 172.21.15.16:0/923840705 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f6b3006c4e0 0x7f6b3006e9a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6b44101d10 tx=0x7f6b4000b410 comp rx=0 tx=0).stop 2024-04-22T15:46:23.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.813+0000 7f6b4bdee700 1 -- 172.21.15.16:0/923840705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b44101eb0 msgr2=0x7f6b44073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:23.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.813+0000 7f6b4bdee700 1 --2- 172.21.15.16:0/923840705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b44101eb0 0x7f6b44073180 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6b34006010 tx=0x7f6b3400bd70 comp rx=0 tx=0).stop 2024-04-22T15:46:23.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.813+0000 7f6b4bdee700 1 -- 172.21.15.16:0/923840705 shutdown_connections 2024-04-22T15:46:23.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.813+0000 7f6b4bdee700 1 --2- 172.21.15.16:0/923840705 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f6b3006c4e0 0x7f6b3006e9a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:23.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.813+0000 7f6b4bdee700 1 --2- 172.21.15.16:0/923840705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b44100cb0 0x7f6b44074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:23.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.813+0000 7f6b4bdee700 1 --2- 172.21.15.16:0/923840705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b44101eb0 0x7f6b44073180 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:23.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.813+0000 7f6b4bdee700 1 -- 172.21.15.16:0/923840705 >> 172.21.15.16:0/923840705 conn(0x7f6b440fc230 msgr2=0x7f6b441050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:23.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.814+0000 7f6b4bdee700 1 -- 172.21.15.16:0/923840705 shutdown_connections 2024-04-22T15:46:23.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:23.814+0000 7f6b4bdee700 1 -- 172.21.15.16:0/923840705 wait complete. 2024-04-22T15:46:23.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:23 smithi027 ceph-mon[34922]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:24.406 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T15:46:24.417 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-22T15:46:24.417 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-04-22T15:46:24.631 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:46:25.589 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:25 smithi016 ceph-mon[26620]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:25.589 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:25 smithi016 ceph-mon[26620]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:25.590 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:25 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:46:25.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:25 smithi027 ceph-mon[34922]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:25.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:25 smithi027 ceph-mon[34922]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:25.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:25 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:46:26.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.047+0000 7f78abd53700 1 -- 172.21.15.16:0/3664040987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78a4101eb0 msgr2=0x7f78a4102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:26.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.047+0000 7f78abd53700 1 --2- 172.21.15.16:0/3664040987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78a4101eb0 0x7f78a4102310 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7894009910 tx=0x7f7894009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:26.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.047+0000 7f78abd53700 1 -- 172.21.15.16:0/3664040987 shutdown_connections 2024-04-22T15:46:26.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.047+0000 7f78abd53700 1 --2- 172.21.15.16:0/3664040987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78a4101eb0 0x7f78a4102310 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:26.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.047+0000 7f78abd53700 1 --2- 172.21.15.16:0/3664040987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78a4100cb0 0x7f78a41010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:26.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.047+0000 7f78abd53700 1 -- 172.21.15.16:0/3664040987 >> 172.21.15.16:0/3664040987 conn(0x7f78a40fc230 msgr2=0x7f78a40fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:26.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.047+0000 7f78abd53700 1 -- 172.21.15.16:0/3664040987 shutdown_connections 2024-04-22T15:46:26.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.047+0000 7f78abd53700 1 -- 172.21.15.16:0/3664040987 wait complete. 2024-04-22T15:46:26.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.047+0000 7f78abd53700 1 Processor -- start 2024-04-22T15:46:26.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.047+0000 7f78abd53700 1 -- start start 2024-04-22T15:46:26.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.047+0000 7f78abd53700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78a4100cb0 0x7f78a4074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:26.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.047+0000 7f78abd53700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78a4101eb0 0x7f78a4073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:26.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.047+0000 7f78abd53700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a40736c0 con 0x7f78a4101eb0 2024-04-22T15:46:26.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.047+0000 7f78abd53700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a4073800 con 0x7f78a4100cb0 2024-04-22T15:46:26.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.048+0000 7f78a92ee700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78a4101eb0 0x7f78a4073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:26.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.048+0000 7f78a92ee700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78a4101eb0 0x7f78a4073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59826/0 (socket says 172.21.15.16:59826) 2024-04-22T15:46:26.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.048+0000 7f78a92ee700 1 -- 172.21.15.16:0/3067159406 learned_addr learned my addr 172.21.15.16:0/3067159406 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:46:26.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.048+0000 7f78a9aef700 1 --2- 172.21.15.16:0/3067159406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78a4100cb0 0x7f78a4074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:26.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.048+0000 7f78a92ee700 1 -- 172.21.15.16:0/3067159406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78a4100cb0 msgr2=0x7f78a4074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:26.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.048+0000 7f78a92ee700 1 --2- 172.21.15.16:0/3067159406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78a4100cb0 0x7f78a4074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:26.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.048+0000 7f78a92ee700 1 -- 172.21.15.16:0/3067159406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78940095f0 con 0x7f78a4101eb0 2024-04-22T15:46:26.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.048+0000 7f78a92ee700 1 --2- 172.21.15.16:0/3067159406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78a4101eb0 0x7f78a4073180 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7894006010 tx=0x7f789400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:26.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.048+0000 7f789affd700 1 -- 172.21.15.16:0/3067159406 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f789401d070 con 0x7f78a4101eb0 2024-04-22T15:46:26.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.048+0000 7f789affd700 1 -- 172.21.15.16:0/3067159406 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f789400f460 con 0x7f78a4101eb0 2024-04-22T15:46:26.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.048+0000 7f78abd53700 1 -- 172.21.15.16:0/3067159406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a4073a80 con 0x7f78a4101eb0 2024-04-22T15:46:26.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.048+0000 7f789affd700 1 -- 172.21.15.16:0/3067159406 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7894021640 con 0x7f78a4101eb0 2024-04-22T15:46:26.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.048+0000 7f78abd53700 1 -- 172.21.15.16:0/3067159406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a4073f70 con 0x7f78a4101eb0 2024-04-22T15:46:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.049+0000 7f78abd53700 1 -- 172.21.15.16:0/3067159406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a41098b0 con 0x7f78a4101eb0 2024-04-22T15:46:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.050+0000 7f789affd700 1 -- 172.21.15.16:0/3067159406 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f789400fab0 con 0x7f78a4101eb0 2024-04-22T15:46:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.050+0000 7f789affd700 1 --2- 172.21.15.16:0/3067159406 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f789006c4e0 0x7f789006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.050+0000 7f789affd700 1 -- 172.21.15.16:0/3067159406 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f789408c8f0 con 0x7f78a4101eb0 2024-04-22T15:46:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.050+0000 7f78a9aef700 1 --2- 172.21.15.16:0/3067159406 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f789006c4e0 0x7f789006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.051+0000 7f78a9aef700 1 --2- 172.21.15.16:0/3067159406 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f789006c4e0 0x7f789006e9a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f78a0006fd0 tx=0x7f78a0008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:26.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.055+0000 7f789affd700 1 -- 172.21.15.16:0/3067159406 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f789405b040 con 0x7f78a4101eb0 2024-04-22T15:46:26.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:26.280+0000 7f78abd53700 1 -- 172.21.15.16:0/3067159406 --> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f78a4106800 con 0x7f789006c4e0 2024-04-22T15:46:27.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:27 smithi027 ceph-mon[34922]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:27.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:27 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-22T15:46:27.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:27 smithi027 ceph-mon[34922]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:27.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:27 smithi016 ceph-mon[26620]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:46:27.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:27 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-22T15:46:27.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:27 smithi016 ceph-mon[26620]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:27.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:27.794+0000 7f789affd700 1 -- 172.21.15.16:0/3067159406 <== mgr.14217 v2:172.21.15.16:6800/1057234399 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f78a4106800 con 0x7f789006c4e0 2024-04-22T15:46:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:27.799+0000 7f78abd53700 1 -- 172.21.15.16:0/3067159406 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f789006c4e0 msgr2=0x7f789006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:27.800+0000 7f78abd53700 1 --2- 172.21.15.16:0/3067159406 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f789006c4e0 0x7f789006e9a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f78a0006fd0 tx=0x7f78a0008040 comp rx=0 tx=0).stop 2024-04-22T15:46:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:27.800+0000 7f78abd53700 1 -- 172.21.15.16:0/3067159406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78a4101eb0 msgr2=0x7f78a4073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:27.800+0000 7f78abd53700 1 --2- 172.21.15.16:0/3067159406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78a4101eb0 0x7f78a4073180 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7894006010 tx=0x7f789400be20 comp rx=0 tx=0).stop 2024-04-22T15:46:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:27.800+0000 7f78abd53700 1 -- 172.21.15.16:0/3067159406 shutdown_connections 2024-04-22T15:46:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:27.800+0000 7f78abd53700 1 --2- 172.21.15.16:0/3067159406 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f789006c4e0 0x7f789006e9a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:27.800+0000 7f78abd53700 1 --2- 172.21.15.16:0/3067159406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78a4100cb0 0x7f78a4074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:27.800+0000 7f78abd53700 1 --2- 172.21.15.16:0/3067159406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78a4101eb0 0x7f78a4073180 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:27.800+0000 7f78abd53700 1 -- 172.21.15.16:0/3067159406 >> 172.21.15.16:0/3067159406 conn(0x7f78a40fc230 msgr2=0x7f78a41050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:27.801+0000 7f78abd53700 1 -- 172.21.15.16:0/3067159406 shutdown_connections 2024-04-22T15:46:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:27.801+0000 7f78abd53700 1 -- 172.21.15.16:0/3067159406 wait complete. 2024-04-22T15:46:28.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:27 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016[26616]: 2024-04-22T15:46:27.773+0000 7f3302311700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T15:46:28.447 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:28 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-22T15:46:28.447 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:28 smithi016 ceph-mon[26620]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T15:46:28.447 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:28 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-22T15:46:28.447 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:28 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-22T15:46:28.447 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:28 smithi016 ceph-mon[26620]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T15:46:28.447 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:28 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-22T15:46:28.447 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:28 smithi016 ceph-mon[26620]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T15:46:28.447 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:28 smithi016 ceph-mon[26620]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T15:46:28.448 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:28 smithi016 ceph-mon[26620]: osdmap e43: 6 total, 6 up, 6 in 2024-04-22T15:46:28.448 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:28 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-22T15:46:28.448 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:28 smithi016 ceph-mon[26620]: fsmap cephfs:0 2024-04-22T15:46:28.448 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:28 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:28.448 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:28 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:46:28.472 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-22T15:46:28.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:28 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-22T15:46:28.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:28 smithi027 ceph-mon[34922]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T15:46:28.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:28 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-22T15:46:28.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:28 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-22T15:46:28.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:28 smithi027 ceph-mon[34922]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T15:46:28.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:28 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-22T15:46:28.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:28 smithi027 ceph-mon[34922]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T15:46:28.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:28 smithi027 ceph-mon[34922]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T15:46:28.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:28 smithi027 ceph-mon[34922]: osdmap e43: 6 total, 6 up, 6 in 2024-04-22T15:46:28.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:28 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-22T15:46:28.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:28 smithi027 ceph-mon[34922]: fsmap cephfs:0 2024-04-22T15:46:28.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:28 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:28.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:28 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:46:28.717 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:46:29.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:29 smithi016 ceph-mon[26620]: Saving service mds.cephfs spec with placement count:4 2024-04-22T15:46:29.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:29 smithi016 ceph-mon[26620]: pgmap v175: 65 pgs: 9 creating+peering, 6 active+clean, 50 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:29.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:29 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:46:29.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:29 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:46:29.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:29 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:29.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:29 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi016.fkkejm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:46:29.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:29 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi016.fkkejm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T15:46:29.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:29 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:46:29.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:29 smithi016 ceph-mon[26620]: osdmap e44: 6 total, 6 up, 6 in 2024-04-22T15:46:29.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:29 smithi027 ceph-mon[34922]: Saving service mds.cephfs spec with placement count:4 2024-04-22T15:46:29.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:29 smithi027 ceph-mon[34922]: pgmap v175: 65 pgs: 9 creating+peering, 6 active+clean, 50 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:29.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:29 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:46:29.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:29 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:46:29.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:29 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:29.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:29 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi016.fkkejm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:46:29.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:29 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi016.fkkejm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T15:46:29.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:29 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:46:29.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:29 smithi027 ceph-mon[34922]: osdmap e44: 6 total, 6 up, 6 in 2024-04-22T15:46:30.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:30 smithi027 ceph-mon[34922]: Deploying daemon mds.cephfs.smithi016.fkkejm on smithi016 2024-04-22T15:46:30.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:30 smithi027 ceph-mon[34922]: osdmap e45: 6 total, 6 up, 6 in 2024-04-22T15:46:30.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:30 smithi016 ceph-mon[26620]: Deploying daemon mds.cephfs.smithi016.fkkejm on smithi016 2024-04-22T15:46:30.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:30 smithi016 ceph-mon[26620]: osdmap e45: 6 total, 6 up, 6 in 2024-04-22T15:46:31.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:31 smithi016 ceph-mon[26620]: pgmap v178: 65 pgs: 14 creating+peering, 22 active+clean, 29 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:31.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:31 smithi027 ceph-mon[34922]: pgmap v178: 65 pgs: 14 creating+peering, 22 active+clean, 29 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:31.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.776+0000 7fc9224b2700 1 -- 172.21.15.16:0/2370362456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc91c101eb0 msgr2=0x7fc91c102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:31.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.776+0000 7fc9224b2700 1 --2- 172.21.15.16:0/2370362456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc91c101eb0 0x7fc91c102310 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc910009910 tx=0x7fc910009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:31.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.776+0000 7fc9224b2700 1 -- 172.21.15.16:0/2370362456 shutdown_connections 2024-04-22T15:46:31.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.776+0000 7fc9224b2700 1 --2- 172.21.15.16:0/2370362456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc91c101eb0 0x7fc91c102310 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:31.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.776+0000 7fc9224b2700 1 --2- 172.21.15.16:0/2370362456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc91c100cb0 0x7fc91c1010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:31.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.776+0000 7fc9224b2700 1 -- 172.21.15.16:0/2370362456 >> 172.21.15.16:0/2370362456 conn(0x7fc91c0fc230 msgr2=0x7fc91c0fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:31.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.776+0000 7fc9224b2700 1 -- 172.21.15.16:0/2370362456 shutdown_connections 2024-04-22T15:46:31.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.776+0000 7fc9224b2700 1 -- 172.21.15.16:0/2370362456 wait complete. 2024-04-22T15:46:31.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.776+0000 7fc9224b2700 1 Processor -- start 2024-04-22T15:46:31.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.776+0000 7fc9224b2700 1 -- start start 2024-04-22T15:46:31.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.777+0000 7fc9224b2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc91c100cb0 0x7fc91c074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:31.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.777+0000 7fc9224b2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc91c101eb0 0x7fc91c073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:31.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.777+0000 7fc9224b2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc91c0736c0 con 0x7fc91c101eb0 2024-04-22T15:46:31.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.777+0000 7fc9224b2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc91c073800 con 0x7fc91c100cb0 2024-04-22T15:46:31.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.777+0000 7fc91b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc91c101eb0 0x7fc91c073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:31.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.777+0000 7fc91b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc91c101eb0 0x7fc91c073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35044/0 (socket says 172.21.15.16:35044) 2024-04-22T15:46:31.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.777+0000 7fc91b7fe700 1 -- 172.21.15.16:0/2192520997 learned_addr learned my addr 172.21.15.16:0/2192520997 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:46:31.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.777+0000 7fc91bfff700 1 --2- 172.21.15.16:0/2192520997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc91c100cb0 0x7fc91c074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:31.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.777+0000 7fc91b7fe700 1 -- 172.21.15.16:0/2192520997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc91c100cb0 msgr2=0x7fc91c074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:31.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.777+0000 7fc91b7fe700 1 --2- 172.21.15.16:0/2192520997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc91c100cb0 0x7fc91c074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:31.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.777+0000 7fc91b7fe700 1 -- 172.21.15.16:0/2192520997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9100095f0 con 0x7fc91c101eb0 2024-04-22T15:46:31.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.778+0000 7fc91b7fe700 1 --2- 172.21.15.16:0/2192520997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc91c101eb0 0x7fc91c073180 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc91000bd70 tx=0x7fc91000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:31.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.778+0000 7fc9197fa700 1 -- 172.21.15.16:0/2192520997 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc91001d070 con 0x7fc91c101eb0 2024-04-22T15:46:31.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.778+0000 7fc9224b2700 1 -- 172.21.15.16:0/2192520997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc91c073a80 con 0x7fc91c101eb0 2024-04-22T15:46:31.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.778+0000 7fc9197fa700 1 -- 172.21.15.16:0/2192520997 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc910005940 con 0x7fc91c101eb0 2024-04-22T15:46:31.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.778+0000 7fc9197fa700 1 -- 172.21.15.16:0/2192520997 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc910021430 con 0x7fc91c101eb0 2024-04-22T15:46:31.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.778+0000 7fc9224b2700 1 -- 172.21.15.16:0/2192520997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc91c073f70 con 0x7fc91c101eb0 2024-04-22T15:46:31.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.779+0000 7fc9224b2700 1 -- 172.21.15.16:0/2192520997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc91c109810 con 0x7fc91c101eb0 2024-04-22T15:46:31.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.780+0000 7fc9197fa700 1 -- 172.21.15.16:0/2192520997 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fc91000f860 con 0x7fc91c101eb0 2024-04-22T15:46:31.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.780+0000 7fc9197fa700 1 --2- 172.21.15.16:0/2192520997 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fc90406c4e0 0x7fc90406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:31.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.780+0000 7fc9197fa700 1 -- 172.21.15.16:0/2192520997 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fc91008cbb0 con 0x7fc91c101eb0 2024-04-22T15:46:31.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.780+0000 7fc91bfff700 1 --2- 172.21.15.16:0/2192520997 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fc90406c4e0 0x7fc90406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:31.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.781+0000 7fc91bfff700 1 --2- 172.21.15.16:0/2192520997 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fc90406c4e0 0x7fc90406e9a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc91c101d10 tx=0x7fc90c00a3b0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:31.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:31.786+0000 7fc9197fa700 1 -- 172.21.15.16:0/2192520997 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc910093080 con 0x7fc91c101eb0 2024-04-22T15:46:32.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:32.078+0000 7fc9224b2700 1 -- 172.21.15.16:0/2192520997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc91c1099d0 con 0x7fc91c101eb0 2024-04-22T15:46:32.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:32.079+0000 7fc9197fa700 1 -- 172.21.15.16:0/2192520997 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1093 (secure 0 0 0) 0x7fc91005ae20 con 0x7fc91c101eb0 2024-04-22T15:46:32.082 INFO:teuthology.orchestra.run.smithi016.stdout:e2 2024-04-22T15:46:32.083 INFO:teuthology.orchestra.run.smithi016.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:46:32.083 INFO:teuthology.orchestra.run.smithi016.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:46:32.083 INFO:teuthology.orchestra.run.smithi016.stdout:legacy client fscid: 1 2024-04-22T15:46:32.083 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:46:32.083 INFO:teuthology.orchestra.run.smithi016.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:46:32.083 INFO:teuthology.orchestra.run.smithi016.stdout:fs_name cephfs 2024-04-22T15:46:32.083 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 2 2024-04-22T15:46:32.083 INFO:teuthology.orchestra.run.smithi016.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-22T15:46:32.083 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-04-22T15:46:27.774571+0000 2024-04-22T15:46:32.083 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-04-22T15:46:27.774667+0000 2024-04-22T15:46:32.083 INFO:teuthology.orchestra.run.smithi016.stdout:tableserver 0 2024-04-22T15:46:32.083 INFO:teuthology.orchestra.run.smithi016.stdout:root 0 2024-04-22T15:46:32.083 INFO:teuthology.orchestra.run.smithi016.stdout:session_timeout 60 2024-04-22T15:46:32.084 INFO:teuthology.orchestra.run.smithi016.stdout:session_autoclose 300 2024-04-22T15:46:32.084 INFO:teuthology.orchestra.run.smithi016.stdout:max_file_size 1099511627776 2024-04-22T15:46:32.084 INFO:teuthology.orchestra.run.smithi016.stdout:required_client_features {} 2024-04-22T15:46:32.084 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure 0 2024-04-22T15:46:32.084 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure_osd_epoch 0 2024-04-22T15:46:32.084 INFO:teuthology.orchestra.run.smithi016.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:46:32.084 INFO:teuthology.orchestra.run.smithi016.stdout:max_mds 1 2024-04-22T15:46:32.084 INFO:teuthology.orchestra.run.smithi016.stdout:in 2024-04-22T15:46:32.084 INFO:teuthology.orchestra.run.smithi016.stdout:up {} 2024-04-22T15:46:32.084 INFO:teuthology.orchestra.run.smithi016.stdout:failed 2024-04-22T15:46:32.084 INFO:teuthology.orchestra.run.smithi016.stdout:damaged 2024-04-22T15:46:32.084 INFO:teuthology.orchestra.run.smithi016.stdout:stopped 2024-04-22T15:46:32.084 INFO:teuthology.orchestra.run.smithi016.stdout:data_pools [3] 2024-04-22T15:46:32.084 INFO:teuthology.orchestra.run.smithi016.stdout:metadata_pool 2 2024-04-22T15:46:32.084 INFO:teuthology.orchestra.run.smithi016.stdout:inline_data disabled 2024-04-22T15:46:32.085 INFO:teuthology.orchestra.run.smithi016.stdout:balancer 2024-04-22T15:46:32.085 INFO:teuthology.orchestra.run.smithi016.stdout:bal_rank_mask -1 2024-04-22T15:46:32.085 INFO:teuthology.orchestra.run.smithi016.stdout:standby_count_wanted 0 2024-04-22T15:46:32.085 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:46:32.085 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:46:32.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:32.082+0000 7fc9224b2700 1 -- 172.21.15.16:0/2192520997 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fc90406c4e0 msgr2=0x7fc90406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:32.082+0000 7fc9224b2700 1 --2- 172.21.15.16:0/2192520997 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fc90406c4e0 0x7fc90406e9a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc91c101d10 tx=0x7fc90c00a3b0 comp rx=0 tx=0).stop 2024-04-22T15:46:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:32.082+0000 7fc9224b2700 1 -- 172.21.15.16:0/2192520997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc91c101eb0 msgr2=0x7fc91c073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:32.082+0000 7fc9224b2700 1 --2- 172.21.15.16:0/2192520997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc91c101eb0 0x7fc91c073180 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc91000bd70 tx=0x7fc91000bda0 comp rx=0 tx=0).stop 2024-04-22T15:46:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:32.082+0000 7fc9224b2700 1 -- 172.21.15.16:0/2192520997 shutdown_connections 2024-04-22T15:46:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:32.082+0000 7fc9224b2700 1 --2- 172.21.15.16:0/2192520997 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fc90406c4e0 0x7fc90406e9a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:32.082+0000 7fc9224b2700 1 --2- 172.21.15.16:0/2192520997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc91c100cb0 0x7fc91c074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:32.083+0000 7fc9224b2700 1 --2- 172.21.15.16:0/2192520997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc91c101eb0 0x7fc91c073180 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:32.083+0000 7fc9224b2700 1 -- 172.21.15.16:0/2192520997 >> 172.21.15.16:0/2192520997 conn(0x7fc91c0fc230 msgr2=0x7fc91c1050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:32.083+0000 7fc9224b2700 1 -- 172.21.15.16:0/2192520997 shutdown_connections 2024-04-22T15:46:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:32.083+0000 7fc9224b2700 1 -- 172.21.15.16:0/2192520997 wait complete. 2024-04-22T15:46:32.087 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 2 2024-04-22T15:46:32.336 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:32 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/2192520997' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:46:32.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:32 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/2192520997' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:46:32.656 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T15:46:32.666 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-22T15:46:32.666 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- bash -c 'ceph fs set cephfs max_mds 1' 2024-04-22T15:46:33.166 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:46:33.522 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:33 smithi016 ceph-mon[26620]: pgmap v179: 65 pgs: 14 creating+peering, 43 active+clean, 8 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:33.522 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:33 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:33.522 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:33 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:33.523 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:33 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:33.523 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:33 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.iobjds", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:46:33.523 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:33 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.iobjds", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T15:46:33.523 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:33 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:46:33.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:33 smithi027 ceph-mon[34922]: pgmap v179: 65 pgs: 14 creating+peering, 43 active+clean, 8 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:46:33.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:33 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:33.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:33 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:33.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:33 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:33.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:33 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.iobjds", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:46:33.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:33 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.iobjds", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T15:46:33.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:33 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:46:34.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:34 smithi027 ceph-mon[34922]: Deploying daemon mds.cephfs.smithi027.iobjds on smithi027 2024-04-22T15:46:34.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:34 smithi027 ceph-mon[34922]: mds.? [v2:172.21.15.16:6826/2331116023,v1:172.21.15.16:6827/2331116023] up:boot 2024-04-22T15:46:34.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:34 smithi027 ceph-mon[34922]: daemon mds.cephfs.smithi016.fkkejm assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-22T15:46:34.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:34 smithi027 ceph-mon[34922]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T15:46:34.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:34 smithi027 ceph-mon[34922]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T15:46:34.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:34 smithi027 ceph-mon[34922]: Cluster is now healthy 2024-04-22T15:46:34.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:34 smithi027 ceph-mon[34922]: fsmap cephfs:0 1 up:standby 2024-04-22T15:46:34.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:34 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.fkkejm"}]: dispatch 2024-04-22T15:46:34.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:34 smithi027 ceph-mon[34922]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:creating} 2024-04-22T15:46:34.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:34 smithi027 ceph-mon[34922]: daemon mds.cephfs.smithi016.fkkejm is now active in filesystem cephfs as rank 0 2024-04-22T15:46:34.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:34 smithi016 ceph-mon[26620]: Deploying daemon mds.cephfs.smithi027.iobjds on smithi027 2024-04-22T15:46:34.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:34 smithi016 ceph-mon[26620]: mds.? [v2:172.21.15.16:6826/2331116023,v1:172.21.15.16:6827/2331116023] up:boot 2024-04-22T15:46:34.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:34 smithi016 ceph-mon[26620]: daemon mds.cephfs.smithi016.fkkejm assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-22T15:46:34.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:34 smithi016 ceph-mon[26620]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T15:46:34.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:34 smithi016 ceph-mon[26620]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T15:46:34.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:34 smithi016 ceph-mon[26620]: Cluster is now healthy 2024-04-22T15:46:34.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:34 smithi016 ceph-mon[26620]: fsmap cephfs:0 1 up:standby 2024-04-22T15:46:34.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:34 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.fkkejm"}]: dispatch 2024-04-22T15:46:34.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:34 smithi016 ceph-mon[26620]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:creating} 2024-04-22T15:46:34.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:34 smithi016 ceph-mon[26620]: daemon mds.cephfs.smithi016.fkkejm is now active in filesystem cephfs as rank 0 2024-04-22T15:46:34.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.673+0000 7fc8dce0b700 1 -- 172.21.15.16:0/3586506090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8d8104070 msgr2=0x7fc8d8106460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:34.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.673+0000 7fc8dce0b700 1 --2- 172.21.15.16:0/3586506090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8d8104070 0x7fc8d8106460 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc009910 tx=0x7fc8cc009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:34.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.674+0000 7fc8dce0b700 1 -- 172.21.15.16:0/3586506090 shutdown_connections 2024-04-22T15:46:34.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.674+0000 7fc8dce0b700 1 --2- 172.21.15.16:0/3586506090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8d8104070 0x7fc8d8106460 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:34.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.674+0000 7fc8dce0b700 1 --2- 172.21.15.16:0/3586506090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8d8101740 0x7fc8d8103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:34.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.674+0000 7fc8dce0b700 1 -- 172.21.15.16:0/3586506090 >> 172.21.15.16:0/3586506090 conn(0x7fc8d80fb150 msgr2=0x7fc8d80fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:34.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.674+0000 7fc8dce0b700 1 -- 172.21.15.16:0/3586506090 shutdown_connections 2024-04-22T15:46:34.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.674+0000 7fc8dce0b700 1 -- 172.21.15.16:0/3586506090 wait complete. 2024-04-22T15:46:34.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.674+0000 7fc8dce0b700 1 Processor -- start 2024-04-22T15:46:34.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.674+0000 7fc8dce0b700 1 -- start start 2024-04-22T15:46:34.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.675+0000 7fc8dce0b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8d8101740 0x7fc8d8196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:34.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.675+0000 7fc8dce0b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8d8104070 0x7fc8d81969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:34.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.675+0000 7fc8dce0b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d8196fc0 con 0x7fc8d8104070 2024-04-22T15:46:34.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.675+0000 7fc8dce0b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d8197100 con 0x7fc8d8101740 2024-04-22T15:46:34.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.675+0000 7fc8d5d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8d8104070 0x7fc8d81969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:34.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.675+0000 7fc8d5d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8d8104070 0x7fc8d81969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35088/0 (socket says 172.21.15.16:35088) 2024-04-22T15:46:34.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.675+0000 7fc8d5d9b700 1 -- 172.21.15.16:0/761968010 learned_addr learned my addr 172.21.15.16:0/761968010 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:46:34.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.675+0000 7fc8d659c700 1 --2- 172.21.15.16:0/761968010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8d8101740 0x7fc8d8196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:34.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.675+0000 7fc8d5d9b700 1 -- 172.21.15.16:0/761968010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8d8101740 msgr2=0x7fc8d8196460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:34.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.675+0000 7fc8d5d9b700 1 --2- 172.21.15.16:0/761968010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8d8101740 0x7fc8d8196460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:34.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.675+0000 7fc8d5d9b700 1 -- 172.21.15.16:0/761968010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8cc0095f0 con 0x7fc8d8104070 2024-04-22T15:46:34.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.675+0000 7fc8d5d9b700 1 --2- 172.21.15.16:0/761968010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8d8104070 0x7fc8d81969a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc000c00 tx=0x7fc8cc00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:34.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.675+0000 7fc8c77fe700 1 -- 172.21.15.16:0/761968010 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8cc01d070 con 0x7fc8d8104070 2024-04-22T15:46:34.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.675+0000 7fc8c77fe700 1 -- 172.21.15.16:0/761968010 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc8cc005dc0 con 0x7fc8d8104070 2024-04-22T15:46:34.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.675+0000 7fc8dce0b700 1 -- 172.21.15.16:0/761968010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d819bb50 con 0x7fc8d8104070 2024-04-22T15:46:34.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.675+0000 7fc8c77fe700 1 -- 172.21.15.16:0/761968010 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8cc0226a0 con 0x7fc8d8104070 2024-04-22T15:46:34.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.676+0000 7fc8dce0b700 1 -- 172.21.15.16:0/761968010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d819c040 con 0x7fc8d8104070 2024-04-22T15:46:34.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.676+0000 7fc8dce0b700 1 -- 172.21.15.16:0/761968010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8d8190620 con 0x7fc8d8104070 2024-04-22T15:46:34.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.677+0000 7fc8c77fe700 1 -- 172.21.15.16:0/761968010 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fc8cc0038f0 con 0x7fc8d8104070 2024-04-22T15:46:34.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.677+0000 7fc8c77fe700 1 --2- 172.21.15.16:0/761968010 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fc8c006c4e0 0x7fc8c006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:34.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.677+0000 7fc8c77fe700 1 -- 172.21.15.16:0/761968010 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fc8cc08cc50 con 0x7fc8d8104070 2024-04-22T15:46:34.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.677+0000 7fc8d659c700 1 --2- 172.21.15.16:0/761968010 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fc8c006c4e0 0x7fc8c006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:34.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.678+0000 7fc8d659c700 1 --2- 172.21.15.16:0/761968010 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fc8c006c4e0 0x7fc8c006e9a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8009c60 tx=0x7fc8c8009260 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:34.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.682+0000 7fc8c77fe700 1 -- 172.21.15.16:0/761968010 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc8cc05af40 con 0x7fc8d8104070 2024-04-22T15:46:34.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:34.935+0000 7fc8dce0b700 1 -- 172.21.15.16:0/761968010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7fc8d80619a0 con 0x7fc8d8104070 2024-04-22T15:46:35.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:35.235+0000 7fc8c77fe700 1 -- 172.21.15.16:0/761968010 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7fc8cc003680 con 0x7fc8d8104070 2024-04-22T15:46:35.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:35.237+0000 7fc8dce0b700 1 -- 172.21.15.16:0/761968010 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fc8c006c4e0 msgr2=0x7fc8c006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:35.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:35.237+0000 7fc8dce0b700 1 --2- 172.21.15.16:0/761968010 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fc8c006c4e0 0x7fc8c006e9a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8009c60 tx=0x7fc8c8009260 comp rx=0 tx=0).stop 2024-04-22T15:46:35.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:35.237+0000 7fc8dce0b700 1 -- 172.21.15.16:0/761968010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8d8104070 msgr2=0x7fc8d81969a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:35.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:35.237+0000 7fc8dce0b700 1 --2- 172.21.15.16:0/761968010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8d8104070 0x7fc8d81969a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc000c00 tx=0x7fc8cc00f690 comp rx=0 tx=0).stop 2024-04-22T15:46:35.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:35.237+0000 7fc8dce0b700 1 -- 172.21.15.16:0/761968010 shutdown_connections 2024-04-22T15:46:35.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:35.237+0000 7fc8dce0b700 1 --2- 172.21.15.16:0/761968010 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fc8c006c4e0 0x7fc8c006e9a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:35.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:35.237+0000 7fc8dce0b700 1 --2- 172.21.15.16:0/761968010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8d8101740 0x7fc8d8196460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:35.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:35.237+0000 7fc8dce0b700 1 --2- 172.21.15.16:0/761968010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8d8104070 0x7fc8d81969a0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:35.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:35.237+0000 7fc8dce0b700 1 -- 172.21.15.16:0/761968010 >> 172.21.15.16:0/761968010 conn(0x7fc8d80fb150 msgr2=0x7fc8d80fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:35.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:35.237+0000 7fc8dce0b700 1 -- 172.21.15.16:0/761968010 shutdown_connections 2024-04-22T15:46:35.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:35.237+0000 7fc8dce0b700 1 -- 172.21.15.16:0/761968010 wait complete. 2024-04-22T15:46:35.554 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:35 smithi016 ceph-mon[26620]: mds.? [v2:172.21.15.16:6826/2331116023,v1:172.21.15.16:6827/2331116023] up:active 2024-04-22T15:46:35.554 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:35 smithi016 ceph-mon[26620]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:active} 2024-04-22T15:46:35.554 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:35 smithi016 ceph-mon[26620]: pgmap v180: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 307 B/s wr, 0 op/s 2024-04-22T15:46:35.554 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:35 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/761968010' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-04-22T15:46:35.562 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:35 smithi027 ceph-mon[34922]: mds.? [v2:172.21.15.16:6826/2331116023,v1:172.21.15.16:6827/2331116023] up:active 2024-04-22T15:46:35.562 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:35 smithi027 ceph-mon[34922]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:active} 2024-04-22T15:46:35.563 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:35 smithi027 ceph-mon[34922]: pgmap v180: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 307 B/s wr, 0 op/s 2024-04-22T15:46:35.563 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:35 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/761968010' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-04-22T15:46:35.904 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T15:46:35.914 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-22T15:46:35.914 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2024-04-22T15:46:36.137 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:46:36.500 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:36 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/761968010' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-04-22T15:46:36.500 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:36 smithi016 ceph-mon[26620]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:active} 2024-04-22T15:46:36.500 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:36 smithi016 ceph-mon[26620]: mds.? [v2:172.21.15.27:6824/4216277609,v1:172.21.15.27:6825/4216277609] up:boot 2024-04-22T15:46:36.500 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:36 smithi016 ceph-mon[26620]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:active} 1 up:standby 2024-04-22T15:46:36.501 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:36 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.iobjds"}]: dispatch 2024-04-22T15:46:36.501 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:36 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:36.501 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:36 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:36.501 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:36 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:36.501 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:36 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi016.pgvphu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:46:36.501 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:36 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi016.pgvphu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T15:46:36.501 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:36 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:46:36.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:36 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/761968010' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-04-22T15:46:36.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:36 smithi027 ceph-mon[34922]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:active} 2024-04-22T15:46:36.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:36 smithi027 ceph-mon[34922]: mds.? [v2:172.21.15.27:6824/4216277609,v1:172.21.15.27:6825/4216277609] up:boot 2024-04-22T15:46:36.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:36 smithi027 ceph-mon[34922]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:active} 1 up:standby 2024-04-22T15:46:36.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:36 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.iobjds"}]: dispatch 2024-04-22T15:46:36.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:36 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:36.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:36 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:36.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:36 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:36.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:36 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi016.pgvphu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:46:36.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:36 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi016.pgvphu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T15:46:36.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:36 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:46:37.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:37 smithi016 ceph-mon[26620]: Deploying daemon mds.cephfs.smithi016.pgvphu on smithi016 2024-04-22T15:46:37.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:37 smithi016 ceph-mon[26620]: pgmap v181: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s wr, 3 op/s 2024-04-22T15:46:37.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:37 smithi027 ceph-mon[34922]: Deploying daemon mds.cephfs.smithi016.pgvphu on smithi016 2024-04-22T15:46:37.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:37 smithi027 ceph-mon[34922]: pgmap v181: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s wr, 3 op/s 2024-04-22T15:46:38.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.451+0000 7ff3bad71700 1 -- 172.21.15.16:0/1910986633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3b40ffba0 msgr2=0x7ff3b40fffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:38.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.451+0000 7ff3bad71700 1 --2- 172.21.15.16:0/1910986633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3b40ffba0 0x7ff3b40fffc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8009910 tx=0x7ff3a8009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:38.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.451+0000 7ff3bad71700 1 -- 172.21.15.16:0/1910986633 shutdown_connections 2024-04-22T15:46:38.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.451+0000 7ff3bad71700 1 --2- 172.21.15.16:0/1910986633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3b4100da0 0x7ff3b4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:38.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.451+0000 7ff3bad71700 1 --2- 172.21.15.16:0/1910986633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3b40ffba0 0x7ff3b40fffc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:38.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.451+0000 7ff3bad71700 1 -- 172.21.15.16:0/1910986633 >> 172.21.15.16:0/1910986633 conn(0x7ff3b40fb120 msgr2=0x7ff3b40fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:38.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.451+0000 7ff3bad71700 1 -- 172.21.15.16:0/1910986633 shutdown_connections 2024-04-22T15:46:38.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.451+0000 7ff3bad71700 1 -- 172.21.15.16:0/1910986633 wait complete. 2024-04-22T15:46:38.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.451+0000 7ff3bad71700 1 Processor -- start 2024-04-22T15:46:38.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.451+0000 7ff3bad71700 1 -- start start 2024-04-22T15:46:38.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.452+0000 7ff3bad71700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3b40ffba0 0x7ff3b41964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:38.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.452+0000 7ff3bad71700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3b4100da0 0x7ff3b41969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:38.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.452+0000 7ff3bad71700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b4197010 con 0x7ff3b4100da0 2024-04-22T15:46:38.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.452+0000 7ff3bad71700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b4197150 con 0x7ff3b40ffba0 2024-04-22T15:46:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.452+0000 7ff3b3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3b4100da0 0x7ff3b41969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.452+0000 7ff3b8b0d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3b40ffba0 0x7ff3b41964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.452+0000 7ff3b3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3b4100da0 0x7ff3b41969f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35092/0 (socket says 172.21.15.16:35092) 2024-04-22T15:46:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.452+0000 7ff3b3fff700 1 -- 172.21.15.16:0/583150803 learned_addr learned my addr 172.21.15.16:0/583150803 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:46:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.452+0000 7ff3b3fff700 1 -- 172.21.15.16:0/583150803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3b40ffba0 msgr2=0x7ff3b41964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.452+0000 7ff3b3fff700 1 --2- 172.21.15.16:0/583150803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3b40ffba0 0x7ff3b41964b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.452+0000 7ff3b3fff700 1 -- 172.21.15.16:0/583150803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a80095f0 con 0x7ff3b4100da0 2024-04-22T15:46:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.452+0000 7ff3b3fff700 1 --2- 172.21.15.16:0/583150803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3b4100da0 0x7ff3b41969f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff3a000d8d0 tx=0x7ff3a000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.452+0000 7ff3b1ffb700 1 -- 172.21.15.16:0/583150803 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3a00098a0 con 0x7ff3b4100da0 2024-04-22T15:46:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.452+0000 7ff3bad71700 1 -- 172.21.15.16:0/583150803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b419bc00 con 0x7ff3b4100da0 2024-04-22T15:46:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.452+0000 7ff3b1ffb700 1 -- 172.21.15.16:0/583150803 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff3a0010460 con 0x7ff3b4100da0 2024-04-22T15:46:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.452+0000 7ff3b1ffb700 1 -- 172.21.15.16:0/583150803 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3a000f5f0 con 0x7ff3b4100da0 2024-04-22T15:46:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.453+0000 7ff3bad71700 1 -- 172.21.15.16:0/583150803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b419c150 con 0x7ff3b4100da0 2024-04-22T15:46:38.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.454+0000 7ff3bad71700 1 -- 172.21.15.16:0/583150803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b4108700 con 0x7ff3b4100da0 2024-04-22T15:46:38.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.454+0000 7ff3b1ffb700 1 -- 172.21.15.16:0/583150803 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7ff3a0009a00 con 0x7ff3b4100da0 2024-04-22T15:46:38.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.454+0000 7ff3b1ffb700 1 --2- 172.21.15.16:0/583150803 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff3a406c4e0 0x7ff3a406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:38.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.454+0000 7ff3b1ffb700 1 -- 172.21.15.16:0/583150803 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7ff3a008b6d0 con 0x7ff3b4100da0 2024-04-22T15:46:38.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.454+0000 7ff3b8b0d700 1 --2- 172.21.15.16:0/583150803 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff3a406c4e0 0x7ff3a406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:38.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.455+0000 7ff3b8b0d700 1 --2- 172.21.15.16:0/583150803 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff3a406c4e0 0x7ff3a406e9a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8006010 tx=0x7ff3a8005ec0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:38.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.459+0000 7ff3b1ffb700 1 -- 172.21.15.16:0/583150803 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff3a0059940 con 0x7ff3b4100da0 2024-04-22T15:46:38.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:38.718+0000 7ff3bad71700 1 -- 172.21.15.16:0/583150803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7ff3b41088f0 con 0x7ff3b4100da0 2024-04-22T15:46:39.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:39.485+0000 7ff3b1ffb700 1 -- 172.21.15.16:0/583150803 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v8) v1 ==== 121+0+0 (secure 0 0 0) 0x7ff3a00594d0 con 0x7ff3b4100da0 2024-04-22T15:46:39.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:39.487+0000 7ff3bad71700 1 -- 172.21.15.16:0/583150803 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff3a406c4e0 msgr2=0x7ff3a406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:39.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:39.488+0000 7ff3bad71700 1 --2- 172.21.15.16:0/583150803 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff3a406c4e0 0x7ff3a406e9a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8006010 tx=0x7ff3a8005ec0 comp rx=0 tx=0).stop 2024-04-22T15:46:39.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:39.488+0000 7ff3bad71700 1 -- 172.21.15.16:0/583150803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3b4100da0 msgr2=0x7ff3b41969f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:39.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:39.488+0000 7ff3bad71700 1 --2- 172.21.15.16:0/583150803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3b4100da0 0x7ff3b41969f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff3a000d8d0 tx=0x7ff3a000dbe0 comp rx=0 tx=0).stop 2024-04-22T15:46:39.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:39.488+0000 7ff3bad71700 1 -- 172.21.15.16:0/583150803 shutdown_connections 2024-04-22T15:46:39.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:39.488+0000 7ff3bad71700 1 --2- 172.21.15.16:0/583150803 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff3a406c4e0 0x7ff3a406e9a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:39.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:39.488+0000 7ff3bad71700 1 --2- 172.21.15.16:0/583150803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3b40ffba0 0x7ff3b41964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:39.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:39.488+0000 7ff3bad71700 1 --2- 172.21.15.16:0/583150803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3b4100da0 0x7ff3b41969f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:39.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:39.488+0000 7ff3bad71700 1 -- 172.21.15.16:0/583150803 >> 172.21.15.16:0/583150803 conn(0x7ff3b40fb120 msgr2=0x7ff3b4068e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:39.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:39.488+0000 7ff3bad71700 1 -- 172.21.15.16:0/583150803 shutdown_connections 2024-04-22T15:46:39.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:39.488+0000 7ff3bad71700 1 -- 172.21.15.16:0/583150803 wait complete. 2024-04-22T15:46:39.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:39 smithi027 ceph-mon[34922]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s wr, 4 op/s 2024-04-22T15:46:39.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:39 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/583150803' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-22T15:46:39.838 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:39 smithi016 ceph-mon[26620]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s wr, 4 op/s 2024-04-22T15:46:39.838 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:39 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/583150803' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-22T15:46:40.155 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T15:46:40.166 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-22T15:46:40.167 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-04-22T15:46:40.384 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:46:40.719 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:40 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/583150803' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-22T15:46:40.719 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:40 smithi027 ceph-mon[34922]: mds.? [v2:172.21.15.16:6828/1006073414,v1:172.21.15.16:6829/1006073414] up:boot 2024-04-22T15:46:40.719 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:40 smithi027 ceph-mon[34922]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:active} 2 up:standby 2024-04-22T15:46:40.719 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:40 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.pgvphu"}]: dispatch 2024-04-22T15:46:40.720 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:40 smithi027 ceph-mon[34922]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:active} 1 up:standby-replay 1 up:standby 2024-04-22T15:46:40.720 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:40 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:40.720 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:40 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:40.720 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:40 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:40.720 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:40 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.dgqyes", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:46:40.720 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:40 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.dgqyes", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T15:46:40.720 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:40 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:46:40.720 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:40 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:46:40.720 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:40 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:40.743 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:40 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/583150803' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-22T15:46:40.743 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:40 smithi016 ceph-mon[26620]: mds.? [v2:172.21.15.16:6828/1006073414,v1:172.21.15.16:6829/1006073414] up:boot 2024-04-22T15:46:40.743 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:40 smithi016 ceph-mon[26620]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:active} 2 up:standby 2024-04-22T15:46:40.743 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.pgvphu"}]: dispatch 2024-04-22T15:46:40.743 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:40 smithi016 ceph-mon[26620]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:active} 1 up:standby-replay 1 up:standby 2024-04-22T15:46:40.743 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:40.743 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:40.743 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:40.743 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.dgqyes", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:46:40.743 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.dgqyes", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T15:46:40.743 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:46:40.743 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:46:40.743 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:41.597 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:41 smithi016 ceph-mon[26620]: Deploying daemon mds.cephfs.smithi027.dgqyes on smithi027 2024-04-22T15:46:41.597 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:41 smithi016 ceph-mon[26620]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 192 B/s rd, 1.3 KiB/s wr, 4 op/s 2024-04-22T15:46:41.747 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:41 smithi027 ceph-mon[34922]: Deploying daemon mds.cephfs.smithi027.dgqyes on smithi027 2024-04-22T15:46:41.748 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:41 smithi027 ceph-mon[34922]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 192 B/s rd, 1.3 KiB/s wr, 4 op/s 2024-04-22T15:46:41.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.798+0000 7f7ea00c5700 1 -- 172.21.15.16:0/479114853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e98100cb0 msgr2=0x7f7e981010d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:41.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.798+0000 7f7ea00c5700 1 --2- 172.21.15.16:0/479114853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e98100cb0 0x7f7e981010d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7e94009910 tx=0x7f7e94009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:41.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.798+0000 7f7ea00c5700 1 -- 172.21.15.16:0/479114853 shutdown_connections 2024-04-22T15:46:41.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.798+0000 7f7ea00c5700 1 --2- 172.21.15.16:0/479114853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e98101eb0 0x7f7e98102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:41.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.798+0000 7f7ea00c5700 1 --2- 172.21.15.16:0/479114853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e98100cb0 0x7f7e981010d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:41.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.798+0000 7f7ea00c5700 1 -- 172.21.15.16:0/479114853 >> 172.21.15.16:0/479114853 conn(0x7f7e980fc230 msgr2=0x7f7e980fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:41.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.798+0000 7f7ea00c5700 1 -- 172.21.15.16:0/479114853 shutdown_connections 2024-04-22T15:46:41.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.798+0000 7f7ea00c5700 1 -- 172.21.15.16:0/479114853 wait complete. 2024-04-22T15:46:41.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.799+0000 7f7ea00c5700 1 Processor -- start 2024-04-22T15:46:41.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.799+0000 7f7ea00c5700 1 -- start start 2024-04-22T15:46:41.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.799+0000 7f7ea00c5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e98100cb0 0x7f7e981964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:41.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.799+0000 7f7ea00c5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e98101eb0 0x7f7e98196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:41.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.799+0000 7f7ea00c5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e98197020 con 0x7f7e98100cb0 2024-04-22T15:46:41.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.799+0000 7f7ea00c5700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e98197160 con 0x7f7e98101eb0 2024-04-22T15:46:41.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.799+0000 7f7e9de61700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e98100cb0 0x7f7e981964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:41.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.799+0000 7f7e9de61700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e98100cb0 0x7f7e981964c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46004/0 (socket says 172.21.15.16:46004) 2024-04-22T15:46:41.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.799+0000 7f7e9de61700 1 -- 172.21.15.16:0/2868735464 learned_addr learned my addr 172.21.15.16:0/2868735464 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:46:41.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.799+0000 7f7e9d660700 1 --2- 172.21.15.16:0/2868735464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e98101eb0 0x7f7e98196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:41.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.799+0000 7f7e9de61700 1 -- 172.21.15.16:0/2868735464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e98101eb0 msgr2=0x7f7e98196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:41.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.799+0000 7f7e9de61700 1 --2- 172.21.15.16:0/2868735464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e98101eb0 0x7f7e98196a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:41.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.800+0000 7f7e9de61700 1 -- 172.21.15.16:0/2868735464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e940095f0 con 0x7f7e98100cb0 2024-04-22T15:46:41.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.800+0000 7f7e9de61700 1 --2- 172.21.15.16:0/2868735464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e98100cb0 0x7f7e981964c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7e94000c00 tx=0x7f7e9400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:41.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.800+0000 7f7e8effd700 1 -- 172.21.15.16:0/2868735464 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e9401d070 con 0x7f7e98100cb0 2024-04-22T15:46:41.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.800+0000 7f7e8effd700 1 -- 172.21.15.16:0/2868735464 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7e94005dc0 con 0x7f7e98100cb0 2024-04-22T15:46:41.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.800+0000 7f7ea00c5700 1 -- 172.21.15.16:0/2868735464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e9819bbb0 con 0x7f7e98100cb0 2024-04-22T15:46:41.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.800+0000 7f7e8effd700 1 -- 172.21.15.16:0/2868735464 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e94022470 con 0x7f7e98100cb0 2024-04-22T15:46:41.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.800+0000 7f7ea00c5700 1 -- 172.21.15.16:0/2868735464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e9819c0a0 con 0x7f7e98100cb0 2024-04-22T15:46:41.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.801+0000 7f7ea00c5700 1 -- 172.21.15.16:0/2868735464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e98109810 con 0x7f7e98100cb0 2024-04-22T15:46:41.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.801+0000 7f7e8effd700 1 -- 172.21.15.16:0/2868735464 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f7e94003810 con 0x7f7e98100cb0 2024-04-22T15:46:41.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.802+0000 7f7e8effd700 1 --2- 172.21.15.16:0/2868735464 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f7e8406c4e0 0x7f7e8406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:41.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.802+0000 7f7e8effd700 1 -- 172.21.15.16:0/2868735464 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f7e9408cc60 con 0x7f7e98100cb0 2024-04-22T15:46:41.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.802+0000 7f7e9d660700 1 --2- 172.21.15.16:0/2868735464 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f7e8406c4e0 0x7f7e8406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:41.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.802+0000 7f7e9d660700 1 --2- 172.21.15.16:0/2868735464 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f7e8406c4e0 0x7f7e8406e9a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7e88009c60 tx=0x7f7e88009260 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:41.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:41.806+0000 7f7e8effd700 1 -- 172.21.15.16:0/2868735464 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7e9405af50 con 0x7f7e98100cb0 2024-04-22T15:46:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:42.051+0000 7f7ea00c5700 1 -- 172.21.15.16:0/2868735464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7e9819c770 con 0x7f7e98100cb0 2024-04-22T15:46:42.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:42.497+0000 7f7e8effd700 1 -- 172.21.15.16:0/2868735464 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v10) v1 ==== 168+0+0 (secure 0 0 0) 0x7f7e9405aae0 con 0x7f7e98100cb0 2024-04-22T15:46:42.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:42.500+0000 7f7ea00c5700 1 -- 172.21.15.16:0/2868735464 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f7e8406c4e0 msgr2=0x7f7e8406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:42.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:42.500+0000 7f7ea00c5700 1 --2- 172.21.15.16:0/2868735464 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f7e8406c4e0 0x7f7e8406e9a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7e88009c60 tx=0x7f7e88009260 comp rx=0 tx=0).stop 2024-04-22T15:46:42.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:42.500+0000 7f7ea00c5700 1 -- 172.21.15.16:0/2868735464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e98100cb0 msgr2=0x7f7e981964c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:42.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:42.500+0000 7f7ea00c5700 1 --2- 172.21.15.16:0/2868735464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e98100cb0 0x7f7e981964c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7e94000c00 tx=0x7f7e9400f690 comp rx=0 tx=0).stop 2024-04-22T15:46:42.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:42.500+0000 7f7ea00c5700 1 -- 172.21.15.16:0/2868735464 shutdown_connections 2024-04-22T15:46:42.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:42.500+0000 7f7ea00c5700 1 --2- 172.21.15.16:0/2868735464 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f7e8406c4e0 0x7f7e8406e9a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:42.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:42.500+0000 7f7ea00c5700 1 --2- 172.21.15.16:0/2868735464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e98100cb0 0x7f7e981964c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:42.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:42.500+0000 7f7ea00c5700 1 --2- 172.21.15.16:0/2868735464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e98101eb0 0x7f7e98196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:42.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:42.500+0000 7f7ea00c5700 1 -- 172.21.15.16:0/2868735464 >> 172.21.15.16:0/2868735464 conn(0x7f7e980fc230 msgr2=0x7f7e981050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:42.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:42.500+0000 7f7ea00c5700 1 -- 172.21.15.16:0/2868735464 shutdown_connections 2024-04-22T15:46:42.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:42.500+0000 7f7ea00c5700 1 -- 172.21.15.16:0/2868735464 wait complete. 2024-04-22T15:46:42.504 INFO:teuthology.orchestra.run.smithi016.stderr:inline data enabled 2024-04-22T15:46:42.754 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:42 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/2868735464' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T15:46:42.754 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:42 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:42.754 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:42 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:42.754 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:42 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:42.754 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:42 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:42.754 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:42 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:42.754 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:42 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:46:42.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:42 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/2868735464' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T15:46:42.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:42 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:42.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:42 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:42.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:42 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:42.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:42 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:42.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:42 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:42.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:42 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:46:43.034 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T15:46:43.044 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-22T15:46:43.044 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-22T15:46:43.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:43 smithi027 ceph-mon[34922]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-22T15:46:43.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:43 smithi027 ceph-mon[34922]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-04-22T15:46:43.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:43 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/2868735464' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T15:46:43.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:43 smithi027 ceph-mon[34922]: mds.? [v2:172.21.15.27:6826/476026355,v1:172.21.15.27:6827/476026355] up:boot 2024-04-22T15:46:43.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:43 smithi027 ceph-mon[34922]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:active} 1 up:standby-replay 2 up:standby 2024-04-22T15:46:43.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:43 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.dgqyes"}]: dispatch 2024-04-22T15:46:43.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:43 smithi016 ceph-mon[26620]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-22T15:46:43.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:43 smithi016 ceph-mon[26620]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-04-22T15:46:43.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:43 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/2868735464' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T15:46:43.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:43 smithi016 ceph-mon[26620]: mds.? [v2:172.21.15.27:6826/476026355,v1:172.21.15.27:6827/476026355] up:boot 2024-04-22T15:46:43.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:43 smithi016 ceph-mon[26620]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:active} 1 up:standby-replay 2 up:standby 2024-04-22T15:46:43.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:43 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.dgqyes"}]: dispatch 2024-04-22T15:46:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:46:45.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.360+0000 7f355aca3700 1 -- 172.21.15.16:0/1457430915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35540fef60 msgr2=0x7f35540ff380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:45.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.360+0000 7f355aca3700 1 --2- 172.21.15.16:0/1457430915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35540fef60 0x7f35540ff380 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3548009910 tx=0x7f3548009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:45.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.360+0000 7f355aca3700 1 -- 172.21.15.16:0/1457430915 shutdown_connections 2024-04-22T15:46:45.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.360+0000 7f355aca3700 1 --2- 172.21.15.16:0/1457430915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3554101f40 0x7f35541023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:45.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.360+0000 7f355aca3700 1 --2- 172.21.15.16:0/1457430915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35540fef60 0x7f35540ff380 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:45.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.360+0000 7f355aca3700 1 -- 172.21.15.16:0/1457430915 >> 172.21.15.16:0/1457430915 conn(0x7f35540fa8c0 msgr2=0x7f35540fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:45.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.360+0000 7f355aca3700 1 -- 172.21.15.16:0/1457430915 shutdown_connections 2024-04-22T15:46:45.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.360+0000 7f355aca3700 1 -- 172.21.15.16:0/1457430915 wait complete. 2024-04-22T15:46:45.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.361+0000 7f355aca3700 1 Processor -- start 2024-04-22T15:46:45.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.361+0000 7f355aca3700 1 -- start start 2024-04-22T15:46:45.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.361+0000 7f355aca3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35540fef60 0x7f35541964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:45.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.361+0000 7f355aca3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3554101f40 0x7f3554196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:45.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.361+0000 7f355aca3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3554197030 con 0x7f35540fef60 2024-04-22T15:46:45.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.361+0000 7f355aca3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3554197170 con 0x7f3554101f40 2024-04-22T15:46:45.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.361+0000 7f3558a3f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35540fef60 0x7f35541964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:45.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.361+0000 7f3558a3f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35540fef60 0x7f35541964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46028/0 (socket says 172.21.15.16:46028) 2024-04-22T15:46:45.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.361+0000 7f3558a3f700 1 -- 172.21.15.16:0/1704927089 learned_addr learned my addr 172.21.15.16:0/1704927089 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:46:45.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.361+0000 7f3553fff700 1 --2- 172.21.15.16:0/1704927089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3554101f40 0x7f3554196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:45.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.361+0000 7f3558a3f700 1 -- 172.21.15.16:0/1704927089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3554101f40 msgr2=0x7f3554196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:45.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.361+0000 7f3558a3f700 1 --2- 172.21.15.16:0/1704927089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3554101f40 0x7f3554196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:45.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.361+0000 7f3558a3f700 1 -- 172.21.15.16:0/1704927089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35480095f0 con 0x7f35540fef60 2024-04-22T15:46:45.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.361+0000 7f3553fff700 1 --2- 172.21.15.16:0/1704927089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3554101f40 0x7f3554196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:46:45.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.362+0000 7f3558a3f700 1 --2- 172.21.15.16:0/1704927089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35540fef60 0x7f35541964d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3548000c00 tx=0x7f354800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:45.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.365+0000 7f3551ffb700 1 -- 172.21.15.16:0/1704927089 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f354801d070 con 0x7f35540fef60 2024-04-22T15:46:45.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.365+0000 7f3551ffb700 1 -- 172.21.15.16:0/1704927089 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3548005dc0 con 0x7f35540fef60 2024-04-22T15:46:45.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.365+0000 7f355aca3700 1 -- 172.21.15.16:0/1704927089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f355419bbc0 con 0x7f35540fef60 2024-04-22T15:46:45.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.365+0000 7f3551ffb700 1 -- 172.21.15.16:0/1704927089 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3548022470 con 0x7f35540fef60 2024-04-22T15:46:45.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.365+0000 7f355aca3700 1 -- 172.21.15.16:0/1704927089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f355419c0b0 con 0x7f35540fef60 2024-04-22T15:46:45.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.366+0000 7f355aca3700 1 -- 172.21.15.16:0/1704927089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3554109800 con 0x7f35540fef60 2024-04-22T15:46:45.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.367+0000 7f3551ffb700 1 -- 172.21.15.16:0/1704927089 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f3548003810 con 0x7f35540fef60 2024-04-22T15:46:45.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.367+0000 7f3551ffb700 1 --2- 172.21.15.16:0/1704927089 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f354406c4e0 0x7f354406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:45.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.367+0000 7f3551ffb700 1 -- 172.21.15.16:0/1704927089 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f354808ce20 con 0x7f35540fef60 2024-04-22T15:46:45.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.367+0000 7f3553fff700 1 --2- 172.21.15.16:0/1704927089 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f354406c4e0 0x7f354406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:45.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.367+0000 7f3553fff700 1 --2- 172.21.15.16:0/1704927089 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f354406c4e0 0x7f354406e9a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3540011490 tx=0x7f3540009260 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:45.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.375+0000 7f3551ffb700 1 -- 172.21.15.16:0/1704927089 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f354805af90 con 0x7f35540fef60 2024-04-22T15:46:45.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:45 smithi016 ceph-mon[26620]: mds.? [v2:172.21.15.16:6828/1006073414,v1:172.21.15.16:6829/1006073414] up:standby 2024-04-22T15:46:45.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:45 smithi016 ceph-mon[26620]: mds.? [v2:172.21.15.27:6824/4216277609,v1:172.21.15.27:6825/4216277609] up:standby-replay 2024-04-22T15:46:45.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:45 smithi016 ceph-mon[26620]: Dropping low affinity active daemon mds.cephfs.smithi016.fkkejm in favor of higher affinity standby. 2024-04-22T15:46:45.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:45 smithi016 ceph-mon[26620]: Replacing daemon mds.cephfs.smithi016.fkkejm as rank 0 with standby daemon mds.cephfs.smithi027.iobjds 2024-04-22T15:46:45.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:45 smithi016 ceph-mon[26620]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T15:46:45.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:45 smithi016 ceph-mon[26620]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:active} 1 up:standby-replay 2 up:standby 2024-04-22T15:46:45.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:45 smithi016 ceph-mon[26620]: osdmap e46: 6 total, 6 up, 6 in 2024-04-22T15:46:45.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:45 smithi016 ceph-mon[26620]: fsmap cephfs:1/1 {0=cephfs.smithi027.iobjds=up:replay} 2 up:standby 2024-04-22T15:46:45.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:45 smithi016 ceph-mon[26620]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-22T15:46:45.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:45 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:45.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:45 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:45.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.628+0000 7f355aca3700 1 -- 172.21.15.16:0/1704927089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f35541099f0 con 0x7f35540fef60 2024-04-22T15:46:45.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.628+0000 7f3551ffb700 1 -- 172.21.15.16:0/1704927089 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 13 v13) v1 ==== 76+0+1833 (secure 0 0 0) 0x7f354805ab20 con 0x7f35540fef60 2024-04-22T15:46:45.632 INFO:teuthology.orchestra.run.smithi016.stdout:e13 2024-04-22T15:46:45.632 INFO:teuthology.orchestra.run.smithi016.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:46:45.632 INFO:teuthology.orchestra.run.smithi016.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:46:45.632 INFO:teuthology.orchestra.run.smithi016.stdout:legacy client fscid: 1 2024-04-22T15:46:45.632 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:46:45.632 INFO:teuthology.orchestra.run.smithi016.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:46:45.632 INFO:teuthology.orchestra.run.smithi016.stdout:fs_name cephfs 2024-04-22T15:46:45.632 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 13 2024-04-22T15:46:45.632 INFO:teuthology.orchestra.run.smithi016.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T15:46:45.632 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-04-22T15:46:27.774571+0000 2024-04-22T15:46:45.632 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-04-22T15:46:45.362388+0000 2024-04-22T15:46:45.632 INFO:teuthology.orchestra.run.smithi016.stdout:tableserver 0 2024-04-22T15:46:45.632 INFO:teuthology.orchestra.run.smithi016.stdout:root 0 2024-04-22T15:46:45.633 INFO:teuthology.orchestra.run.smithi016.stdout:session_timeout 60 2024-04-22T15:46:45.633 INFO:teuthology.orchestra.run.smithi016.stdout:session_autoclose 300 2024-04-22T15:46:45.633 INFO:teuthology.orchestra.run.smithi016.stdout:max_file_size 1099511627776 2024-04-22T15:46:45.633 INFO:teuthology.orchestra.run.smithi016.stdout:required_client_features {} 2024-04-22T15:46:45.633 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure 0 2024-04-22T15:46:45.633 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure_osd_epoch 46 2024-04-22T15:46:45.633 INFO:teuthology.orchestra.run.smithi016.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:46:45.633 INFO:teuthology.orchestra.run.smithi016.stdout:max_mds 1 2024-04-22T15:46:45.633 INFO:teuthology.orchestra.run.smithi016.stdout:in 0 2024-04-22T15:46:45.633 INFO:teuthology.orchestra.run.smithi016.stdout:up {0=24279} 2024-04-22T15:46:45.633 INFO:teuthology.orchestra.run.smithi016.stdout:failed 2024-04-22T15:46:45.633 INFO:teuthology.orchestra.run.smithi016.stdout:damaged 2024-04-22T15:46:45.634 INFO:teuthology.orchestra.run.smithi016.stdout:stopped 2024-04-22T15:46:45.634 INFO:teuthology.orchestra.run.smithi016.stdout:data_pools [3] 2024-04-22T15:46:45.634 INFO:teuthology.orchestra.run.smithi016.stdout:metadata_pool 2 2024-04-22T15:46:45.634 INFO:teuthology.orchestra.run.smithi016.stdout:inline_data enabled 2024-04-22T15:46:45.634 INFO:teuthology.orchestra.run.smithi016.stdout:balancer 2024-04-22T15:46:45.634 INFO:teuthology.orchestra.run.smithi016.stdout:bal_rank_mask -1 2024-04-22T15:46:45.634 INFO:teuthology.orchestra.run.smithi016.stdout:standby_count_wanted 1 2024-04-22T15:46:45.634 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.iobjds{0:24279} state up:reconnect seq 4 join_fscid=1 addr [v2:172.21.15.27:6824/4216277609,v1:172.21.15.27:6825/4216277609] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:46:45.634 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:46:45.634 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:46:45.634 INFO:teuthology.orchestra.run.smithi016.stdout:Standby daemons: 2024-04-22T15:46:45.634 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:46:45.634 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.pgvphu{-1:14494} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.16:6828/1006073414,v1:172.21.15.16:6829/1006073414] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:46:45.635 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.fkkejm{-1:14504} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.16:6826/3484907755,v1:172.21.15.16:6827/3484907755] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:46:45.635 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.dgqyes{-1:24293} state up:standby seq 1 addr [v2:172.21.15.27:6826/476026355,v1:172.21.15.27:6827/476026355] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:46:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.630+0000 7f355aca3700 1 -- 172.21.15.16:0/1704927089 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f354406c4e0 msgr2=0x7f354406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.630+0000 7f355aca3700 1 --2- 172.21.15.16:0/1704927089 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f354406c4e0 0x7f354406e9a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3540011490 tx=0x7f3540009260 comp rx=0 tx=0).stop 2024-04-22T15:46:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.630+0000 7f355aca3700 1 -- 172.21.15.16:0/1704927089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35540fef60 msgr2=0x7f35541964d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.630+0000 7f355aca3700 1 --2- 172.21.15.16:0/1704927089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35540fef60 0x7f35541964d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3548000c00 tx=0x7f354800f690 comp rx=0 tx=0).stop 2024-04-22T15:46:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.630+0000 7f355aca3700 1 -- 172.21.15.16:0/1704927089 shutdown_connections 2024-04-22T15:46:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.630+0000 7f355aca3700 1 --2- 172.21.15.16:0/1704927089 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f354406c4e0 0x7f354406e9a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.630+0000 7f355aca3700 1 --2- 172.21.15.16:0/1704927089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35540fef60 0x7f35541964d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.630+0000 7f355aca3700 1 --2- 172.21.15.16:0/1704927089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3554101f40 0x7f3554196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.630+0000 7f355aca3700 1 -- 172.21.15.16:0/1704927089 >> 172.21.15.16:0/1704927089 conn(0x7f35540fa8c0 msgr2=0x7f3554105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.630+0000 7f355aca3700 1 -- 172.21.15.16:0/1704927089 shutdown_connections 2024-04-22T15:46:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:45.631+0000 7f355aca3700 1 -- 172.21.15.16:0/1704927089 wait complete. 2024-04-22T15:46:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 13 2024-04-22T15:46:45.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:45 smithi027 ceph-mon[34922]: mds.? [v2:172.21.15.16:6828/1006073414,v1:172.21.15.16:6829/1006073414] up:standby 2024-04-22T15:46:45.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:45 smithi027 ceph-mon[34922]: mds.? [v2:172.21.15.27:6824/4216277609,v1:172.21.15.27:6825/4216277609] up:standby-replay 2024-04-22T15:46:45.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:45 smithi027 ceph-mon[34922]: Dropping low affinity active daemon mds.cephfs.smithi016.fkkejm in favor of higher affinity standby. 2024-04-22T15:46:45.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:45 smithi027 ceph-mon[34922]: Replacing daemon mds.cephfs.smithi016.fkkejm as rank 0 with standby daemon mds.cephfs.smithi027.iobjds 2024-04-22T15:46:45.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:45 smithi027 ceph-mon[34922]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T15:46:45.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:45 smithi027 ceph-mon[34922]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:active} 1 up:standby-replay 2 up:standby 2024-04-22T15:46:45.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:45 smithi027 ceph-mon[34922]: osdmap e46: 6 total, 6 up, 6 in 2024-04-22T15:46:45.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:45 smithi027 ceph-mon[34922]: fsmap cephfs:1/1 {0=cephfs.smithi027.iobjds=up:replay} 2 up:standby 2024-04-22T15:46:45.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:45 smithi027 ceph-mon[34922]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-22T15:46:45.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:45 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:45.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:45 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:46.359 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-04-22T15:46:46.648 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:46 smithi016 ceph-mon[26620]: mds.? [v2:172.21.15.16:6826/3484907755,v1:172.21.15.16:6827/3484907755] up:boot 2024-04-22T15:46:46.648 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:46 smithi016 ceph-mon[26620]: mds.? [v2:172.21.15.27:6824/4216277609,v1:172.21.15.27:6825/4216277609] up:reconnect 2024-04-22T15:46:46.648 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:46 smithi016 ceph-mon[26620]: fsmap cephfs:1/1 {0=cephfs.smithi027.iobjds=up:reconnect} 3 up:standby 2024-04-22T15:46:46.648 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:46 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.fkkejm"}]: dispatch 2024-04-22T15:46:46.648 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:46 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:46.648 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:46 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/1704927089' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:46:46.739 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:46:46.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:46 smithi027 ceph-mon[34922]: mds.? [v2:172.21.15.16:6826/3484907755,v1:172.21.15.16:6827/3484907755] up:boot 2024-04-22T15:46:46.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:46 smithi027 ceph-mon[34922]: mds.? [v2:172.21.15.27:6824/4216277609,v1:172.21.15.27:6825/4216277609] up:reconnect 2024-04-22T15:46:46.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:46 smithi027 ceph-mon[34922]: fsmap cephfs:1/1 {0=cephfs.smithi027.iobjds=up:reconnect} 3 up:standby 2024-04-22T15:46:46.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:46 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.fkkejm"}]: dispatch 2024-04-22T15:46:46.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:46 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:46.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:46 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/1704927089' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:46:47.663 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:47 smithi016 ceph-mon[26620]: mds.? [v2:172.21.15.27:6824/4216277609,v1:172.21.15.27:6825/4216277609] up:rejoin 2024-04-22T15:46:47.663 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:47 smithi016 ceph-mon[26620]: mds.? [v2:172.21.15.27:6826/476026355,v1:172.21.15.27:6827/476026355] up:standby 2024-04-22T15:46:47.663 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:47 smithi016 ceph-mon[26620]: fsmap cephfs:1/1 {0=cephfs.smithi027.iobjds=up:rejoin} 3 up:standby 2024-04-22T15:46:47.663 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:47 smithi016 ceph-mon[26620]: daemon mds.cephfs.smithi027.iobjds is now active in filesystem cephfs as rank 0 2024-04-22T15:46:47.664 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:47 smithi016 ceph-mon[26620]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 102 B/s wr, 4 op/s 2024-04-22T15:46:47.664 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:47 smithi016 ceph-mon[26620]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T15:46:47.664 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:47 smithi016 ceph-mon[26620]: mds.? [v2:172.21.15.27:6824/4216277609,v1:172.21.15.27:6825/4216277609] up:active 2024-04-22T15:46:47.664 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:47 smithi016 ceph-mon[26620]: fsmap cephfs:1 {0=cephfs.smithi027.iobjds=up:active} 3 up:standby 2024-04-22T15:46:47.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:47 smithi027 ceph-mon[34922]: mds.? [v2:172.21.15.27:6824/4216277609,v1:172.21.15.27:6825/4216277609] up:rejoin 2024-04-22T15:46:47.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:47 smithi027 ceph-mon[34922]: mds.? [v2:172.21.15.27:6826/476026355,v1:172.21.15.27:6827/476026355] up:standby 2024-04-22T15:46:47.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:47 smithi027 ceph-mon[34922]: fsmap cephfs:1/1 {0=cephfs.smithi027.iobjds=up:rejoin} 3 up:standby 2024-04-22T15:46:47.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:47 smithi027 ceph-mon[34922]: daemon mds.cephfs.smithi027.iobjds is now active in filesystem cephfs as rank 0 2024-04-22T15:46:47.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:47 smithi027 ceph-mon[34922]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 102 B/s wr, 4 op/s 2024-04-22T15:46:47.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:47 smithi027 ceph-mon[34922]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T15:46:47.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:47 smithi027 ceph-mon[34922]: mds.? [v2:172.21.15.27:6824/4216277609,v1:172.21.15.27:6825/4216277609] up:active 2024-04-22T15:46:47.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:47 smithi027 ceph-mon[34922]: fsmap cephfs:1 {0=cephfs.smithi027.iobjds=up:active} 3 up:standby 2024-04-22T15:46:48.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:48 smithi016 ceph-mon[26620]: fsmap cephfs:1 {0=cephfs.smithi027.iobjds=up:active} 1 up:standby-replay 2 up:standby 2024-04-22T15:46:48.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:48 smithi027 ceph-mon[34922]: fsmap cephfs:1 {0=cephfs.smithi027.iobjds=up:active} 1 up:standby-replay 2 up:standby 2024-04-22T15:46:48.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.908+0000 7ff55de3d700 1 -- 172.21.15.16:0/1103787564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff558072b20 msgr2=0x7ff558107d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:48.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.908+0000 7ff55de3d700 1 --2- 172.21.15.16:0/1103787564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff558072b20 0x7ff558107d90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff548009910 tx=0x7ff548009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:48.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.908+0000 7ff55de3d700 1 -- 172.21.15.16:0/1103787564 shutdown_connections 2024-04-22T15:46:48.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.908+0000 7ff55de3d700 1 --2- 172.21.15.16:0/1103787564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff558108360 0x7ff5581087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:48.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.908+0000 7ff55de3d700 1 --2- 172.21.15.16:0/1103787564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff558072b20 0x7ff558107d90 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:48.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.908+0000 7ff55de3d700 1 -- 172.21.15.16:0/1103787564 >> 172.21.15.16:0/1103787564 conn(0x7ff55806da40 msgr2=0x7ff55806fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:48.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.908+0000 7ff55de3d700 1 -- 172.21.15.16:0/1103787564 shutdown_connections 2024-04-22T15:46:48.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.908+0000 7ff55de3d700 1 -- 172.21.15.16:0/1103787564 wait complete. 2024-04-22T15:46:48.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.909+0000 7ff55de3d700 1 Processor -- start 2024-04-22T15:46:48.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.909+0000 7ff55de3d700 1 -- start start 2024-04-22T15:46:48.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.909+0000 7ff55de3d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff558072b20 0x7ff55819eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:48.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.909+0000 7ff55de3d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff558108360 0x7ff55819f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:48.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.909+0000 7ff55de3d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55819f750 con 0x7ff558108360 2024-04-22T15:46:48.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.909+0000 7ff55de3d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55819f8c0 con 0x7ff558072b20 2024-04-22T15:46:48.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.909+0000 7ff556ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff558108360 0x7ff55819f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:48.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.909+0000 7ff556ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff558108360 0x7ff55819f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46048/0 (socket says 172.21.15.16:46048) 2024-04-22T15:46:48.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.909+0000 7ff556ffd700 1 -- 172.21.15.16:0/2524335973 learned_addr learned my addr 172.21.15.16:0/2524335973 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:46:48.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.910+0000 7ff5577fe700 1 --2- 172.21.15.16:0/2524335973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff558072b20 0x7ff55819eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:48.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.910+0000 7ff556ffd700 1 -- 172.21.15.16:0/2524335973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff558072b20 msgr2=0x7ff55819eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:48.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.910+0000 7ff556ffd700 1 --2- 172.21.15.16:0/2524335973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff558072b20 0x7ff55819eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:48.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.910+0000 7ff556ffd700 1 -- 172.21.15.16:0/2524335973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5480095f0 con 0x7ff558108360 2024-04-22T15:46:48.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.910+0000 7ff556ffd700 1 --2- 172.21.15.16:0/2524335973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff558108360 0x7ff55819f1e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff54c00c930 tx=0x7ff54c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:48.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.910+0000 7ff554ff9700 1 -- 172.21.15.16:0/2524335973 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff54c007ad0 con 0x7ff558108360 2024-04-22T15:46:48.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.910+0000 7ff55de3d700 1 -- 172.21.15.16:0/2524335973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5581a4360 con 0x7ff558108360 2024-04-22T15:46:48.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.910+0000 7ff554ff9700 1 -- 172.21.15.16:0/2524335973 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff54c00ce80 con 0x7ff558108360 2024-04-22T15:46:48.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.910+0000 7ff554ff9700 1 -- 172.21.15.16:0/2524335973 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff54c018630 con 0x7ff558108360 2024-04-22T15:46:48.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.910+0000 7ff55de3d700 1 -- 172.21.15.16:0/2524335973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5581a48b0 con 0x7ff558108360 2024-04-22T15:46:48.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.911+0000 7ff55de3d700 1 -- 172.21.15.16:0/2524335973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff558112000 con 0x7ff558108360 2024-04-22T15:46:48.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.912+0000 7ff554ff9700 1 -- 172.21.15.16:0/2524335973 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7ff54c018790 con 0x7ff558108360 2024-04-22T15:46:48.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.912+0000 7ff554ff9700 1 --2- 172.21.15.16:0/2524335973 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff54006c4e0 0x7ff54006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:48.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.912+0000 7ff554ff9700 1 -- 172.21.15.16:0/2524335973 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ff54c08b880 con 0x7ff558108360 2024-04-22T15:46:48.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.912+0000 7ff5577fe700 1 --2- 172.21.15.16:0/2524335973 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff54006c4e0 0x7ff54006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:48.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.913+0000 7ff5577fe700 1 --2- 172.21.15.16:0/2524335973 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff54006c4e0 0x7ff54006e9a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff548006010 tx=0x7ff54800b540 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:48.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:48.917+0000 7ff554ff9700 1 -- 172.21.15.16:0/2524335973 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff54c056000 con 0x7ff558108360 2024-04-22T15:46:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:49.168+0000 7ff55de3d700 1 -- 172.21.15.16:0/2524335973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5581121c0 con 0x7ff558108360 2024-04-22T15:46:49.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:49.169+0000 7ff554ff9700 1 -- 172.21.15.16:0/2524335973 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4744 (secure 0 0 0) 0x7ff54c059620 con 0x7ff558108360 2024-04-22T15:46:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:49.171+0000 7ff55de3d700 1 -- 172.21.15.16:0/2524335973 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff54006c4e0 msgr2=0x7ff54006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:49.171+0000 7ff55de3d700 1 --2- 172.21.15.16:0/2524335973 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff54006c4e0 0x7ff54006e9a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff548006010 tx=0x7ff54800b540 comp rx=0 tx=0).stop 2024-04-22T15:46:49.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:49.171+0000 7ff55de3d700 1 -- 172.21.15.16:0/2524335973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff558108360 msgr2=0x7ff55819f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:49.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:49.171+0000 7ff55de3d700 1 --2- 172.21.15.16:0/2524335973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff558108360 0x7ff55819f1e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff54c00c930 tx=0x7ff54c00cc40 comp rx=0 tx=0).stop 2024-04-22T15:46:49.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:49.171+0000 7ff55de3d700 1 -- 172.21.15.16:0/2524335973 shutdown_connections 2024-04-22T15:46:49.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:49.171+0000 7ff55de3d700 1 --2- 172.21.15.16:0/2524335973 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7ff54006c4e0 0x7ff54006e9a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:49.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:49.171+0000 7ff55de3d700 1 --2- 172.21.15.16:0/2524335973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff558072b20 0x7ff55819eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:49.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:49.171+0000 7ff55de3d700 1 --2- 172.21.15.16:0/2524335973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff558108360 0x7ff55819f1e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:49.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:49.171+0000 7ff55de3d700 1 -- 172.21.15.16:0/2524335973 >> 172.21.15.16:0/2524335973 conn(0x7ff55806da40 msgr2=0x7ff55810d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:49.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:49.171+0000 7ff55de3d700 1 -- 172.21.15.16:0/2524335973 shutdown_connections 2024-04-22T15:46:49.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:49.171+0000 7ff55de3d700 1 -- 172.21.15.16:0/2524335973 wait complete. 2024-04-22T15:46:49.175 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 16 2024-04-22T15:46:49.185 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:46:49.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:49 smithi016 ceph-mon[26620]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 4.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T15:46:49.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:49 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/2524335973' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T15:46:49.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:49 smithi027 ceph-mon[34922]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 4.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T15:46:49.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:49 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/2524335973' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T15:46:50.075 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-04-22T15:46:50.859 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:46:51.624 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:51 smithi016 ceph-mon[26620]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.5 KiB/s rd, 6 op/s 2024-04-22T15:46:51.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:51 smithi027 ceph-mon[34922]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.5 KiB/s rd, 6 op/s 2024-04-22T15:46:52.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.423+0000 7f8761be9700 1 -- 172.21.15.16:0/2752267024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f875c100e10 msgr2=0x7f875c101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:52.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.423+0000 7f8761be9700 1 --2- 172.21.15.16:0/2752267024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f875c100e10 0x7f875c101270 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8750009910 tx=0x7f8750009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:52.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.423+0000 7f8761be9700 1 -- 172.21.15.16:0/2752267024 shutdown_connections 2024-04-22T15:46:52.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.423+0000 7f8761be9700 1 --2- 172.21.15.16:0/2752267024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f875c100e10 0x7f875c101270 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:52.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.423+0000 7f8761be9700 1 --2- 172.21.15.16:0/2752267024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f875c0ffc10 0x7f875c100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:52.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.423+0000 7f8761be9700 1 -- 172.21.15.16:0/2752267024 >> 172.21.15.16:0/2752267024 conn(0x7f875c0fb190 msgr2=0x7f875c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:52.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.423+0000 7f8761be9700 1 -- 172.21.15.16:0/2752267024 shutdown_connections 2024-04-22T15:46:52.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.423+0000 7f8761be9700 1 -- 172.21.15.16:0/2752267024 wait complete. 2024-04-22T15:46:52.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.423+0000 7f8761be9700 1 Processor -- start 2024-04-22T15:46:52.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.424+0000 7f8761be9700 1 -- start start 2024-04-22T15:46:52.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.424+0000 7f8761be9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f875c0ffc10 0x7f875c0690e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:52.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.424+0000 7f8761be9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f875c100e10 0x7f875c19d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:52.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.424+0000 7f8761be9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f875c197070 con 0x7f875c0ffc10 2024-04-22T15:46:52.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.424+0000 7f8761be9700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f875c19db40 con 0x7f875c100e10 2024-04-22T15:46:52.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.424+0000 7f875b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f875c0ffc10 0x7f875c0690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:52.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.424+0000 7f875affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f875c100e10 0x7f875c19d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:52.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.424+0000 7f875b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f875c0ffc10 0x7f875c0690e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57930/0 (socket says 172.21.15.16:57930) 2024-04-22T15:46:52.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.424+0000 7f875b7fe700 1 -- 172.21.15.16:0/267633817 learned_addr learned my addr 172.21.15.16:0/267633817 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:46:52.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.424+0000 7f875affd700 1 -- 172.21.15.16:0/267633817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f875c0ffc10 msgr2=0x7f875c0690e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:52.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.424+0000 7f875affd700 1 --2- 172.21.15.16:0/267633817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f875c0ffc10 0x7f875c0690e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:52.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.424+0000 7f875affd700 1 -- 172.21.15.16:0/267633817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87500095f0 con 0x7f875c100e10 2024-04-22T15:46:52.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.424+0000 7f875affd700 1 --2- 172.21.15.16:0/267633817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f875c100e10 0x7f875c19d600 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8750000c00 tx=0x7f875000f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:52.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.425+0000 7f8758ff9700 1 -- 172.21.15.16:0/267633817 <== mon.1 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f875001d070 con 0x7f875c100e10 2024-04-22T15:46:52.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.425+0000 7f8758ff9700 1 -- 172.21.15.16:0/267633817 <== mon.1 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8750005dc0 con 0x7f875c100e10 2024-04-22T15:46:52.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.425+0000 7f8758ff9700 1 -- 172.21.15.16:0/267633817 <== mon.1 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8750017400 con 0x7f875c100e10 2024-04-22T15:46:52.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.425+0000 7f8761be9700 1 -- 172.21.15.16:0/267633817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f875c19dd90 con 0x7f875c100e10 2024-04-22T15:46:52.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.425+0000 7f8761be9700 1 -- 172.21.15.16:0/267633817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f875c19e280 con 0x7f875c100e10 2024-04-22T15:46:52.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.426+0000 7f8761be9700 1 -- 172.21.15.16:0/267633817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f875c108700 con 0x7f875c100e10 2024-04-22T15:46:52.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.426+0000 7f8758ff9700 1 -- 172.21.15.16:0/267633817 <== mon.1 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f8750003810 con 0x7f875c100e10 2024-04-22T15:46:52.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.427+0000 7f8758ff9700 1 --2- 172.21.15.16:0/267633817 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f874406c4e0 0x7f874406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:52.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.427+0000 7f875b7fe700 1 --2- 172.21.15.16:0/267633817 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f874406c4e0 0x7f874406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:52.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.427+0000 7f8758ff9700 1 -- 172.21.15.16:0/267633817 <== mon.1 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f875008cbe0 con 0x7f875c100e10 2024-04-22T15:46:52.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.427+0000 7f875b7fe700 1 --2- 172.21.15.16:0/267633817 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f874406c4e0 0x7f874406e9a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f874c005950 tx=0x7f874c009260 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:52.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.431+0000 7f8758ff9700 1 -- 172.21.15.16:0/267633817 <== mon.1 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f875005add0 con 0x7f875c100e10 2024-04-22T15:46:52.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.737+0000 7f8761be9700 1 -- 172.21.15.16:0/267633817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f875c19e8e0 con 0x7f875c100e10 2024-04-22T15:46:52.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.738+0000 7f8758ff9700 1 -- 172.21.15.16:0/267633817 <== mon.1 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v16) v1 ==== 78+0+83 (secure 0 0 0) 0x7f875005a960 con 0x7f875c100e10 2024-04-22T15:46:52.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.740+0000 7f8761be9700 1 -- 172.21.15.16:0/267633817 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f874406c4e0 msgr2=0x7f874406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:52.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.740+0000 7f8761be9700 1 --2- 172.21.15.16:0/267633817 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f874406c4e0 0x7f874406e9a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f874c005950 tx=0x7f874c009260 comp rx=0 tx=0).stop 2024-04-22T15:46:52.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.740+0000 7f8761be9700 1 -- 172.21.15.16:0/267633817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f875c100e10 msgr2=0x7f875c19d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:52.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.740+0000 7f8761be9700 1 --2- 172.21.15.16:0/267633817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f875c100e10 0x7f875c19d600 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8750000c00 tx=0x7f875000f690 comp rx=0 tx=0).stop 2024-04-22T15:46:52.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.740+0000 7f8761be9700 1 -- 172.21.15.16:0/267633817 shutdown_connections 2024-04-22T15:46:52.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.740+0000 7f8761be9700 1 --2- 172.21.15.16:0/267633817 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f874406c4e0 0x7f874406e9a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:52.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.740+0000 7f8761be9700 1 --2- 172.21.15.16:0/267633817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f875c0ffc10 0x7f875c0690e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:52.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.740+0000 7f8761be9700 1 --2- 172.21.15.16:0/267633817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f875c100e10 0x7f875c19d600 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:52.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.740+0000 7f8761be9700 1 -- 172.21.15.16:0/267633817 >> 172.21.15.16:0/267633817 conn(0x7f875c0fb190 msgr2=0x7f875c104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:52.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.740+0000 7f8761be9700 1 -- 172.21.15.16:0/267633817 shutdown_connections 2024-04-22T15:46:52.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:52.740+0000 7f8761be9700 1 -- 172.21.15.16:0/267633817 wait complete. 2024-04-22T15:46:52.753 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:46:53.101 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:53 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:53.102 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:53 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:53.102 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:53 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:46:53.102 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:53 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:46:53.102 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:53 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:53.102 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:53 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:46:53.102 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:53 smithi027 ceph-mon[34922]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 9.1 KiB/s rd, 9 op/s 2024-04-22T15:46:53.102 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:53 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/267633817' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-22T15:46:53.323 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:53 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:53.323 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:53 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:53.323 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:53 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:46:53.323 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:53 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:46:53.323 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:53 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:53.323 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:53 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:46:53.323 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:53 smithi016 ceph-mon[26620]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 9.1 KiB/s rd, 9 op/s 2024-04-22T15:46:53.324 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:53 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/267633817' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-22T15:46:53.924 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-04-22T15:46:53.946 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T15:46:54.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.059+0000 7f792a07d640 1 -- 172.21.15.16:0/2421735552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79241066e0 msgr2=0x7f7924106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:54.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.059+0000 7f792a07d640 1 --2- 172.21.15.16:0/2421735552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79241066e0 0x7f7924106b00 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7914009880 tx=0x7f791402bce0 comp rx=0 tx=0).stop 2024-04-22T15:46:54.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.059+0000 7f792a07d640 1 -- 172.21.15.16:0/2421735552 shutdown_connections 2024-04-22T15:46:54.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.059+0000 7f792a07d640 1 --2- 172.21.15.16:0/2421735552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79241078d0 0x7f7924107d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:54.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.059+0000 7f792a07d640 1 --2- 172.21.15.16:0/2421735552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79241066e0 0x7f7924106b00 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:54.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.059+0000 7f792a07d640 1 -- 172.21.15.16:0/2421735552 >> 172.21.15.16:0/2421735552 conn(0x7f79240757f0 msgr2=0x7f7924077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:54.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.059+0000 7f792a07d640 1 -- 172.21.15.16:0/2421735552 shutdown_connections 2024-04-22T15:46:54.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.059+0000 7f792a07d640 1 -- 172.21.15.16:0/2421735552 wait complete. 2024-04-22T15:46:54.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.060+0000 7f792a07d640 1 Processor -- start 2024-04-22T15:46:54.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.060+0000 7f792a07d640 1 -- start start 2024-04-22T15:46:54.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.060+0000 7f792a07d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79241066e0 0x7f792419bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:54.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.060+0000 7f792a07d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79241078d0 0x7f792419c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:54.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.060+0000 7f792a07d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792419ca90 con 0x7f79241066e0 2024-04-22T15:46:54.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.060+0000 7f792a07d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792419cc00 con 0x7f79241078d0 2024-04-22T15:46:54.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.060+0000 7f79237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79241066e0 0x7f792419bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:54.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.060+0000 7f79237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79241066e0 0x7f792419bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57938/0 (socket says 172.21.15.16:57938) 2024-04-22T15:46:54.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.060+0000 7f79237fe640 1 -- 172.21.15.16:0/3808155685 learned_addr learned my addr 172.21.15.16:0/3808155685 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:46:54.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.060+0000 7f7922ffd640 1 --2- 172.21.15.16:0/3808155685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79241078d0 0x7f792419c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:54.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.060+0000 7f79237fe640 1 -- 172.21.15.16:0/3808155685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79241078d0 msgr2=0x7f792419c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:54.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.060+0000 7f79237fe640 1 --2- 172.21.15.16:0/3808155685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79241078d0 0x7f792419c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:54.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.060+0000 7f79237fe640 1 -- 172.21.15.16:0/3808155685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7914009560 con 0x7f79241066e0 2024-04-22T15:46:54.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.061+0000 7f79237fe640 1 --2- 172.21.15.16:0/3808155685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79241066e0 0x7f792419bf80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f791402c220 tx=0x7f791402c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:54.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.061+0000 7f7920ff9640 1 -- 172.21.15.16:0/3808155685 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7914004420 con 0x7f79241066e0 2024-04-22T15:46:54.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.061+0000 7f792a07d640 1 -- 172.21.15.16:0/3808155685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79241a1630 con 0x7f79241066e0 2024-04-22T15:46:54.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.061+0000 7f7920ff9640 1 -- 172.21.15.16:0/3808155685 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f79140054b0 con 0x7f79241066e0 2024-04-22T15:46:54.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.061+0000 7f7920ff9640 1 -- 172.21.15.16:0/3808155685 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7914034440 con 0x7f79241066e0 2024-04-22T15:46:54.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.061+0000 7f792a07d640 1 -- 172.21.15.16:0/3808155685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79241a1b30 con 0x7f79241066e0 2024-04-22T15:46:54.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.062+0000 7f792a07d640 1 -- 172.21.15.16:0/3808155685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f792410f450 con 0x7f79241066e0 2024-04-22T15:46:54.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.063+0000 7f7920ff9640 1 -- 172.21.15.16:0/3808155685 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f7914034640 con 0x7f79241066e0 2024-04-22T15:46:54.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.063+0000 7f7920ff9640 1 --2- 172.21.15.16:0/3808155685 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f790c06c450 0x7f790c06e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:54.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.063+0000 7f7920ff9640 1 -- 172.21.15.16:0/3808155685 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f79140adc00 con 0x7f79241066e0 2024-04-22T15:46:54.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.063+0000 7f7922ffd640 1 --2- 172.21.15.16:0/3808155685 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f790c06c450 0x7f790c06e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:54.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.064+0000 7f7922ffd640 1 --2- 172.21.15.16:0/3808155685 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f790c06c450 0x7f790c06e910 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f792419d490 tx=0x7f7918005f90 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:54.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.068+0000 7f7920ff9640 1 -- 172.21.15.16:0/3808155685 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f79140b3020 con 0x7f79241066e0 2024-04-22T15:46:54.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.255+0000 7f792a07d640 1 -- 172.21.15.16:0/3808155685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79241a21f0 con 0x7f79241066e0 2024-04-22T15:46:54.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.256+0000 7f7920ff9640 1 -- 172.21.15.16:0/3808155685 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+15092 (secure 0 0 0) 0x7f7914078580 con 0x7f79241066e0 2024-04-22T15:46:54.259 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:46:54.259 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":46,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","created":"2024-04-22T15:39:20.727181+0000","modified":"2024-04-22T15:46:44.357561+0000","last_up_change":"2024-04-22T15:45:11.497701+0000","last_in_change":"2024-04-22T15:44:52.054297+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T15:43:58.896563+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-22T15:46:26.715511+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-22T15:46:27.206408+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"67bd69ee-bd8c-4670-927a-f5fbcd92c9bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1210635569},{"type":"v1","addr":"172.21.15.16:6803","nonce":1210635569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1210635569},{"type":"v1","addr":"172.21.15.16:6805","nonce":1210635569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1210635569},{"type":"v1","addr":"172.21.15.16:6809","nonce":1210635569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1210635569},{"type":"v1","addr":"172.21.15.16:6807","nonce":1210635569}]},"public_addr":"172.21.15.16:6803/1210635569","cluster_addr":"172.21.15.16:6805/1210635569","heartbeat_back_addr":"172.21.15.16:6809/1210635569","heartbeat_front_addr":"172.21.15.16:6807/1210635569","state":["exists","up"]},{"osd":1,"uuid":"f51a9e4d-fb0c-4f62-9328-e23eea863213","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3974144580},{"type":"v1","addr":"172.21.15.16:6811","nonce":3974144580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3974144580},{"type":"v1","addr":"172.21.15.16:6813","nonce":3974144580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3974144580},{"type":"v1","addr":"172.21.15.16:6817","nonce":3974144580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3974144580},{"type":"v1","addr":"172.21.15.16:6815","nonce":3974144580}]},"public_addr":"172.21.15.16:6811/3974144580","cluster_addr":"172.21.15.16:6813/3974144580","heartbeat_back_addr":"172.21.15.16:6817/3974144580","heartbeat_front_addr":"172.21.15.16:6815/3974144580","state":["exists","up"]},{"osd":2,"uuid":"e59fdb03-e04e-4afd-a58e-0b58945d3765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2760833880},{"type":"v1","addr":"172.21.15.16:6819","nonce":2760833880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2760833880},{"type":"v1","addr":"172.21.15.16:6821","nonce":2760833880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2760833880},{"type":"v1","addr":"172.21.15.16:6825","nonce":2760833880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2760833880},{"type":"v1","addr":"172.21.15.16:6823","nonce":2760833880}]},"public_addr":"172.21.15.16:6819/2760833880","cluster_addr":"172.21.15.16:6821/2760833880","heartbeat_back_addr":"172.21.15.16:6825/2760833880","heartbeat_front_addr":"172.21.15.16:6823/2760833880","state":["exists","up"]},{"osd":3,"uuid":"fa25b413-d004-4455-8d7b-0ced7f880a7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":286252834},{"type":"v1","addr":"172.21.15.27:6801","nonce":286252834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":286252834},{"type":"v1","addr":"172.21.15.27:6803","nonce":286252834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":286252834},{"type":"v1","addr":"172.21.15.27:6807","nonce":286252834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":286252834},{"type":"v1","addr":"172.21.15.27:6805","nonce":286252834}]},"public_addr":"172.21.15.27:6801/286252834","cluster_addr":"172.21.15.27:6803/286252834","heartbeat_back_addr":"172.21.15.27:6807/286252834","heartbeat_front_addr":"172.21.15.27:6805/286252834","state":["exists","up"]},{"osd":4,"uuid":"6b1eda0d-1627-4c49-93b1-1a78896f660c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":691669816},{"type":"v1","addr":"172.21.15.27:6809","nonce":691669816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":691669816},{"type":"v1","addr":"172.21.15.27:6811","nonce":691669816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":691669816},{"type":"v1","addr":"172.21.15.27:6815","nonce":691669816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":691669816},{"type":"v1","addr":"172.21.15.27:6813","nonce":691669816}]},"public_addr":"172.21.15.27:6809/691669816","cluster_addr":"172.21.15.27:6811/691669816","heartbeat_back_addr":"172.21.15.27:6815/691669816","heartbeat_front_addr":"172.21.15.27:6813/691669816","state":["exists","up"]},{"osd":5,"uuid":"58cebe34-0821-458f-86b5-7d460829d994","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1237795056},{"type":"v1","addr":"172.21.15.27:6817","nonce":1237795056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1237795056},{"type":"v1","addr":"172.21.15.27:6819","nonce":1237795056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1237795056},{"type":"v1","addr":"172.21.15.27:6823","nonce":1237795056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1237795056},{"type":"v1","addr":"172.21.15.27:6821","nonce":1237795056}]},"public_addr":"172.21.15.27:6817/1237795056","cluster_addr":"172.21.15.27:6819/1237795056","heartbeat_back_addr":"172.21.15.27:6823/1237795056","heartbeat_front_addr":"172.21.15.27:6821/1237795056","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:42:58.192450+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:43:27.129522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:43:54.750854+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:44:19.111479+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:44:44.487429+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:45:09.605548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/3431522404":"2024-04-23T15:40:27.107187+0000","172.21.15.16:0/4252231167":"2024-04-23T15:41:40.376957+0000","172.21.15.16:6827/2331116023":"2024-04-23T15:46:44.357516+0000","172.21.15.16:0/2048283809":"2024-04-23T15:39:53.479948+0000","172.21.15.16:0/2570865319":"2024-04-23T15:39:53.479948+0000","172.21.15.16:0/2348469475":"2024-04-23T15:39:53.479948+0000","172.21.15.16:6800/1748504461":"2024-04-23T15:39:53.479948+0000","172.21.15.16:6801/1748504461":"2024-04-23T15:39:53.479948+0000","172.21.15.16:6801/3972982132":"2024-04-23T15:41:40.376957+0000","172.21.15.16:6801/3431522404":"2024-04-23T15:40:27.107187+0000","172.21.15.16:0/955466902":"2024-04-23T15:40:27.107187+0000","172.21.15.16:0/4172787384":"2024-04-23T15:40:27.107187+0000","172.21.15.16:0/4021274304":"2024-04-23T15:41:40.376957+0000","172.21.15.16:0/2002585335":"2024-04-23T15:40:27.107187+0000","172.21.15.16:6800/3972982132":"2024-04-23T15:41:40.376957+0000","172.21.15.16:6826/2331116023":"2024-04-23T15:46:44.357516+0000","172.21.15.16:0/690437556":"2024-04-23T15:41:40.376957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T15:46:54.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.258+0000 7f792a07d640 1 -- 172.21.15.16:0/3808155685 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f790c06c450 msgr2=0x7f790c06e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:54.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.258+0000 7f792a07d640 1 --2- 172.21.15.16:0/3808155685 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f790c06c450 0x7f790c06e910 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f792419d490 tx=0x7f7918005f90 comp rx=0 tx=0).stop 2024-04-22T15:46:54.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.258+0000 7f792a07d640 1 -- 172.21.15.16:0/3808155685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79241066e0 msgr2=0x7f792419bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:54.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.258+0000 7f792a07d640 1 --2- 172.21.15.16:0/3808155685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79241066e0 0x7f792419bf80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f791402c220 tx=0x7f791402c750 comp rx=0 tx=0).stop 2024-04-22T15:46:54.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.258+0000 7f792a07d640 1 -- 172.21.15.16:0/3808155685 shutdown_connections 2024-04-22T15:46:54.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.258+0000 7f792a07d640 1 --2- 172.21.15.16:0/3808155685 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f790c06c450 0x7f790c06e910 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:54.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.258+0000 7f792a07d640 1 --2- 172.21.15.16:0/3808155685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79241078d0 0x7f792419c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:54.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.258+0000 7f792a07d640 1 --2- 172.21.15.16:0/3808155685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79241066e0 0x7f792419bf80 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:54.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.258+0000 7f792a07d640 1 -- 172.21.15.16:0/3808155685 >> 172.21.15.16:0/3808155685 conn(0x7f79240757f0 msgr2=0x7f792410ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:54.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.258+0000 7f792a07d640 1 -- 172.21.15.16:0/3808155685 shutdown_connections 2024-04-22T15:46:54.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.258+0000 7f792a07d640 1 -- 172.21.15.16:0/3808155685 wait complete. 2024-04-22T15:46:54.272 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T15:43:58.896563+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-22T15:46:26.715511+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-22T15:46:27.206408+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T15:46:54.273 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T15:46:54.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.378+0000 7f8653300640 1 -- 172.21.15.16:0/2022620647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f864c100190 msgr2=0x7f864c100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:54.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.378+0000 7f8653300640 1 --2- 172.21.15.16:0/2022620647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f864c100190 0x7f864c100610 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8648009880 tx=0x7f864802bce0 comp rx=0 tx=0).stop 2024-04-22T15:46:54.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.378+0000 7f8653300640 1 -- 172.21.15.16:0/2022620647 shutdown_connections 2024-04-22T15:46:54.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.378+0000 7f8653300640 1 --2- 172.21.15.16:0/2022620647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f864c100190 0x7f864c100610 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:54.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.378+0000 7f8653300640 1 --2- 172.21.15.16:0/2022620647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864c0fefa0 0x7f864c0ff3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:54.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.378+0000 7f8653300640 1 -- 172.21.15.16:0/2022620647 >> 172.21.15.16:0/2022620647 conn(0x7f864c0fa4f0 msgr2=0x7f864c0fc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:54.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.378+0000 7f8653300640 1 -- 172.21.15.16:0/2022620647 shutdown_connections 2024-04-22T15:46:54.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.378+0000 7f8653300640 1 -- 172.21.15.16:0/2022620647 wait complete. 2024-04-22T15:46:54.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.379+0000 7f8653300640 1 Processor -- start 2024-04-22T15:46:54.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.379+0000 7f8653300640 1 -- start start 2024-04-22T15:46:54.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.379+0000 7f8653300640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f864c0fefa0 0x7f864c194880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:54.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.379+0000 7f8653300640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864c100190 0x7f864c194dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:54.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.379+0000 7f8653300640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864c195390 con 0x7f864c0fefa0 2024-04-22T15:46:54.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.379+0000 7f8653300640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864c195500 con 0x7f864c100190 2024-04-22T15:46:54.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.379+0000 7f865109c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f864c0fefa0 0x7f864c194880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:54.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.379+0000 7f865109c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f864c0fefa0 0x7f864c194880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57944/0 (socket says 172.21.15.16:57944) 2024-04-22T15:46:54.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.379+0000 7f865109c640 1 -- 172.21.15.16:0/3099883527 learned_addr learned my addr 172.21.15.16:0/3099883527 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:46:54.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.379+0000 7f865089b640 1 --2- 172.21.15.16:0/3099883527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864c100190 0x7f864c194dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:54.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.379+0000 7f865109c640 1 -- 172.21.15.16:0/3099883527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864c100190 msgr2=0x7f864c194dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:54.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.379+0000 7f865109c640 1 --2- 172.21.15.16:0/3099883527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864c100190 0x7f864c194dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:54.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.379+0000 7f865109c640 1 -- 172.21.15.16:0/3099883527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8648009560 con 0x7f864c0fefa0 2024-04-22T15:46:54.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.380+0000 7f865109c640 1 --2- 172.21.15.16:0/3099883527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f864c0fefa0 0x7f864c194880 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f863c00d8d0 tx=0x7f863c00dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:54.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.380+0000 7f86427fc640 1 -- 172.21.15.16:0/3099883527 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f863c009a30 con 0x7f864c0fefa0 2024-04-22T15:46:54.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.380+0000 7f86427fc640 1 -- 172.21.15.16:0/3099883527 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f863c009b90 con 0x7f864c0fefa0 2024-04-22T15:46:54.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.380+0000 7f8653300640 1 -- 172.21.15.16:0/3099883527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f864c199f90 con 0x7f864c0fefa0 2024-04-22T15:46:54.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.380+0000 7f86427fc640 1 -- 172.21.15.16:0/3099883527 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f863c00f680 con 0x7f864c0fefa0 2024-04-22T15:46:54.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.380+0000 7f8653300640 1 -- 172.21.15.16:0/3099883527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f864c19a460 con 0x7f864c0fefa0 2024-04-22T15:46:54.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.381+0000 7f8653300640 1 -- 172.21.15.16:0/3099883527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f864c107ba0 con 0x7f864c0fefa0 2024-04-22T15:46:54.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.382+0000 7f86427fc640 1 -- 172.21.15.16:0/3099883527 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f863c009d00 con 0x7f864c0fefa0 2024-04-22T15:46:54.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.382+0000 7f86427fc640 1 --2- 172.21.15.16:0/3099883527 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f863806c450 0x7f863806e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:54.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.383+0000 7f86427fc640 1 -- 172.21.15.16:0/3099883527 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f863c08b660 con 0x7f864c0fefa0 2024-04-22T15:46:54.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.383+0000 7f865089b640 1 --2- 172.21.15.16:0/3099883527 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f863806c450 0x7f863806e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:54.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.383+0000 7f865089b640 1 --2- 172.21.15.16:0/3099883527 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f863806c450 0x7f863806e910 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8648006010 tx=0x7f864802c700 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:54.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.388+0000 7f86427fc640 1 -- 172.21.15.16:0/3099883527 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f863c055e90 con 0x7f864c0fefa0 2024-04-22T15:46:54.546 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:54 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/3808155685' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T15:46:54.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.569+0000 7f8653300640 1 -- 172.21.15.16:0/3099883527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f864c19a900 con 0x7f864c0fefa0 2024-04-22T15:46:54.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.569+0000 7f86427fc640 1 -- 172.21.15.16:0/3099883527 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7f863c0594b0 con 0x7f864c0fefa0 2024-04-22T15:46:54.573 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-22T15:46:54.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.571+0000 7f8653300640 1 -- 172.21.15.16:0/3099883527 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f863806c450 msgr2=0x7f863806e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:54.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.571+0000 7f8653300640 1 --2- 172.21.15.16:0/3099883527 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f863806c450 0x7f863806e910 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8648006010 tx=0x7f864802c700 comp rx=0 tx=0).stop 2024-04-22T15:46:54.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.571+0000 7f8653300640 1 -- 172.21.15.16:0/3099883527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f864c0fefa0 msgr2=0x7f864c194880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:54.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.571+0000 7f8653300640 1 --2- 172.21.15.16:0/3099883527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f864c0fefa0 0x7f864c194880 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f863c00d8d0 tx=0x7f863c00dd90 comp rx=0 tx=0).stop 2024-04-22T15:46:54.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.571+0000 7f8653300640 1 -- 172.21.15.16:0/3099883527 shutdown_connections 2024-04-22T15:46:54.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.571+0000 7f8653300640 1 --2- 172.21.15.16:0/3099883527 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f863806c450 0x7f863806e910 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:54.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.571+0000 7f8653300640 1 --2- 172.21.15.16:0/3099883527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864c100190 0x7f864c194dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:54.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.571+0000 7f8653300640 1 --2- 172.21.15.16:0/3099883527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f864c0fefa0 0x7f864c194880 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:54.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.571+0000 7f8653300640 1 -- 172.21.15.16:0/3099883527 >> 172.21.15.16:0/3099883527 conn(0x7f864c0fa4f0 msgr2=0x7f864c1033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:54.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.571+0000 7f8653300640 1 -- 172.21.15.16:0/3099883527 shutdown_connections 2024-04-22T15:46:54.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.571+0000 7f8653300640 1 -- 172.21.15.16:0/3099883527 wait complete. 2024-04-22T15:46:54.585 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-22T15:46:54.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:54 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/3808155685' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T15:46:54.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.687+0000 7fe6ed069640 1 -- 172.21.15.16:0/286068479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6e81000d0 msgr2=0x7fe6e81004f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:54.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.687+0000 7fe6ed069640 1 --2- 172.21.15.16:0/286068479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6e81000d0 0x7fe6e81004f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8009c20 tx=0x7fe6d802c1b0 comp rx=0 tx=0).stop 2024-04-22T15:46:54.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.687+0000 7fe6ed069640 1 -- 172.21.15.16:0/286068479 shutdown_connections 2024-04-22T15:46:54.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.687+0000 7fe6ed069640 1 --2- 172.21.15.16:0/286068479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6e81012c0 0x7fe6e8101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:54.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.687+0000 7fe6ed069640 1 --2- 172.21.15.16:0/286068479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6e81000d0 0x7fe6e81004f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:54.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.687+0000 7fe6ed069640 1 -- 172.21.15.16:0/286068479 >> 172.21.15.16:0/286068479 conn(0x7fe6e80fb680 msgr2=0x7fe6e80fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:54.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.687+0000 7fe6ed069640 1 -- 172.21.15.16:0/286068479 shutdown_connections 2024-04-22T15:46:54.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.687+0000 7fe6ed069640 1 -- 172.21.15.16:0/286068479 wait complete. 2024-04-22T15:46:54.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.688+0000 7fe6ed069640 1 Processor -- start 2024-04-22T15:46:54.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.688+0000 7fe6ed069640 1 -- start start 2024-04-22T15:46:54.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.688+0000 7fe6ed069640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6e81000d0 0x7fe6e8195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:54.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.688+0000 7fe6ed069640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6e81012c0 0x7fe6e8195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:54.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.688+0000 7fe6ed069640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e81964a0 con 0x7fe6e81012c0 2024-04-22T15:46:54.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.688+0000 7fe6ed069640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e8196610 con 0x7fe6e81000d0 2024-04-22T15:46:54.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.689+0000 7fe6e659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6e81012c0 0x7fe6e8195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:54.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.689+0000 7fe6e659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6e81012c0 0x7fe6e8195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57956/0 (socket says 172.21.15.16:57956) 2024-04-22T15:46:54.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.689+0000 7fe6e6d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6e81000d0 0x7fe6e8195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:54.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.689+0000 7fe6e659c640 1 -- 172.21.15.16:0/3149524193 learned_addr learned my addr 172.21.15.16:0/3149524193 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:46:54.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.689+0000 7fe6e659c640 1 -- 172.21.15.16:0/3149524193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6e81000d0 msgr2=0x7fe6e8195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:54.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.689+0000 7fe6e659c640 1 --2- 172.21.15.16:0/3149524193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6e81000d0 0x7fe6e8195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:54.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.689+0000 7fe6e659c640 1 -- 172.21.15.16:0/3149524193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d8009900 con 0x7fe6e81012c0 2024-04-22T15:46:54.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.689+0000 7fe6e6d9d640 1 --2- 172.21.15.16:0/3149524193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6e81000d0 0x7fe6e8195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:46:54.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.689+0000 7fe6e659c640 1 --2- 172.21.15.16:0/3149524193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6e81012c0 0x7fe6e8195ed0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc00e970 tx=0x7fe6dc00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:54.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.689+0000 7fe6cffff640 1 -- 172.21.15.16:0/3149524193 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6dc00cc90 con 0x7fe6e81012c0 2024-04-22T15:46:54.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.689+0000 7fe6ed069640 1 -- 172.21.15.16:0/3149524193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e819b0a0 con 0x7fe6e81012c0 2024-04-22T15:46:54.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.689+0000 7fe6cffff640 1 -- 172.21.15.16:0/3149524193 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe6dc00cdf0 con 0x7fe6e81012c0 2024-04-22T15:46:54.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.689+0000 7fe6cffff640 1 -- 172.21.15.16:0/3149524193 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6dc010430 con 0x7fe6e81012c0 2024-04-22T15:46:54.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.689+0000 7fe6ed069640 1 -- 172.21.15.16:0/3149524193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e819b680 con 0x7fe6e81012c0 2024-04-22T15:46:54.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.690+0000 7fe6ed069640 1 -- 172.21.15.16:0/3149524193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e8108e20 con 0x7fe6e81012c0 2024-04-22T15:46:54.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.691+0000 7fe6cffff640 1 -- 172.21.15.16:0/3149524193 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fe6dc0046c0 con 0x7fe6e81012c0 2024-04-22T15:46:54.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.691+0000 7fe6cffff640 1 --2- 172.21.15.16:0/3149524193 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fe6d006c450 0x7fe6d006e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:54.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.691+0000 7fe6cffff640 1 -- 172.21.15.16:0/3149524193 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fe6dc014070 con 0x7fe6e81012c0 2024-04-22T15:46:54.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.691+0000 7fe6e6d9d640 1 --2- 172.21.15.16:0/3149524193 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fe6d006c450 0x7fe6d006e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:54.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.692+0000 7fe6e6d9d640 1 --2- 172.21.15.16:0/3149524193 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fe6d006c450 0x7fe6d006e910 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8006010 tx=0x7fe6d802e040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:54.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.696+0000 7fe6cffff640 1 -- 172.21.15.16:0/3149524193 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe6dc0564c0 con 0x7fe6e81012c0 2024-04-22T15:46:54.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.899+0000 7fe6ed069640 1 -- 172.21.15.16:0/3149524193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fe6e8108fb0 con 0x7fe6e81012c0 2024-04-22T15:46:54.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.899+0000 7fe6cffff640 1 -- 172.21.15.16:0/3149524193 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7fe6e8108fb0 con 0x7fe6e81012c0 2024-04-22T15:46:54.903 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-22T15:46:54.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.901+0000 7fe6ed069640 1 -- 172.21.15.16:0/3149524193 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fe6d006c450 msgr2=0x7fe6d006e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:54.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.901+0000 7fe6ed069640 1 --2- 172.21.15.16:0/3149524193 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fe6d006c450 0x7fe6d006e910 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8006010 tx=0x7fe6d802e040 comp rx=0 tx=0).stop 2024-04-22T15:46:54.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.901+0000 7fe6ed069640 1 -- 172.21.15.16:0/3149524193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6e81012c0 msgr2=0x7fe6e8195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:54.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.901+0000 7fe6ed069640 1 --2- 172.21.15.16:0/3149524193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6e81012c0 0x7fe6e8195ed0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc00e970 tx=0x7fe6dc00ed70 comp rx=0 tx=0).stop 2024-04-22T15:46:54.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.901+0000 7fe6ed069640 1 -- 172.21.15.16:0/3149524193 shutdown_connections 2024-04-22T15:46:54.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.901+0000 7fe6ed069640 1 --2- 172.21.15.16:0/3149524193 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fe6d006c450 0x7fe6d006e910 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:54.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.901+0000 7fe6ed069640 1 --2- 172.21.15.16:0/3149524193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6e81012c0 0x7fe6e8195ed0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:54.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.901+0000 7fe6ed069640 1 --2- 172.21.15.16:0/3149524193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6e81000d0 0x7fe6e8195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:54.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.901+0000 7fe6ed069640 1 -- 172.21.15.16:0/3149524193 >> 172.21.15.16:0/3149524193 conn(0x7fe6e80fb680 msgr2=0x7fe6e81044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:54.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.901+0000 7fe6ed069640 1 -- 172.21.15.16:0/3149524193 shutdown_connections 2024-04-22T15:46:54.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:54.901+0000 7fe6ed069640 1 -- 172.21.15.16:0/3149524193 wait complete. 2024-04-22T15:46:54.916 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-22T15:46:55.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.022+0000 7f25b1b1d640 1 -- 172.21.15.16:0/1505419164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25ac1034b0 msgr2=0x7f25ac1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:55.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.022+0000 7f25b1b1d640 1 --2- 172.21.15.16:0/1505419164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25ac1034b0 0x7f25ac1058a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f25a0009880 tx=0x7f25a002bce0 comp rx=0 tx=0).stop 2024-04-22T15:46:55.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.022+0000 7f25b1b1d640 1 -- 172.21.15.16:0/1505419164 shutdown_connections 2024-04-22T15:46:55.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.022+0000 7f25b1b1d640 1 --2- 172.21.15.16:0/1505419164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25ac1034b0 0x7f25ac1058a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:55.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.022+0000 7f25b1b1d640 1 --2- 172.21.15.16:0/1505419164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ac100b80 0x7f25ac102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:55.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.022+0000 7f25b1b1d640 1 -- 172.21.15.16:0/1505419164 >> 172.21.15.16:0/1505419164 conn(0x7f25ac0fa4f0 msgr2=0x7f25ac0fc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:55.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.022+0000 7f25b1b1d640 1 -- 172.21.15.16:0/1505419164 shutdown_connections 2024-04-22T15:46:55.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.022+0000 7f25b1b1d640 1 -- 172.21.15.16:0/1505419164 wait complete. 2024-04-22T15:46:55.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.022+0000 7f25b1b1d640 1 Processor -- start 2024-04-22T15:46:55.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.023+0000 7f25b1b1d640 1 -- start start 2024-04-22T15:46:55.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.023+0000 7f25b1b1d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25ac100b80 0x7f25ac195910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:55.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.023+0000 7f25b1b1d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ac1034b0 0x7f25ac195e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:55.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.023+0000 7f25b1b1d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac196420 con 0x7f25ac100b80 2024-04-22T15:46:55.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.023+0000 7f25b1b1d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac196590 con 0x7f25ac1034b0 2024-04-22T15:46:55.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.023+0000 7f25ab7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25ac100b80 0x7f25ac195910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:55.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.023+0000 7f25ab7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25ac100b80 0x7f25ac195910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57980/0 (socket says 172.21.15.16:57980) 2024-04-22T15:46:55.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.023+0000 7f25ab7fe640 1 -- 172.21.15.16:0/2450577392 learned_addr learned my addr 172.21.15.16:0/2450577392 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:46:55.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.023+0000 7f25aaffd640 1 --2- 172.21.15.16:0/2450577392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ac1034b0 0x7f25ac195e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:55.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.023+0000 7f25ab7fe640 1 -- 172.21.15.16:0/2450577392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ac1034b0 msgr2=0x7f25ac195e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:55.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.023+0000 7f25ab7fe640 1 --2- 172.21.15.16:0/2450577392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ac1034b0 0x7f25ac195e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:55.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.023+0000 7f25ab7fe640 1 -- 172.21.15.16:0/2450577392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a0009560 con 0x7f25ac100b80 2024-04-22T15:46:55.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.023+0000 7f25aaffd640 1 --2- 172.21.15.16:0/2450577392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ac1034b0 0x7f25ac195e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:46:55.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.023+0000 7f25ab7fe640 1 --2- 172.21.15.16:0/2450577392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25ac100b80 0x7f25ac195910 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f259c00d6f0 tx=0x7f259c00dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:55.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.024+0000 7f25a8ff9640 1 -- 172.21.15.16:0/2450577392 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f259c009a30 con 0x7f25ac100b80 2024-04-22T15:46:55.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.024+0000 7f25a8ff9640 1 -- 172.21.15.16:0/2450577392 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f259c0054d0 con 0x7f25ac100b80 2024-04-22T15:46:55.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.024+0000 7f25b1b1d640 1 -- 172.21.15.16:0/2450577392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ac19afc0 con 0x7f25ac100b80 2024-04-22T15:46:55.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.024+0000 7f25a8ff9640 1 -- 172.21.15.16:0/2450577392 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f259c010460 con 0x7f25ac100b80 2024-04-22T15:46:55.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.024+0000 7f25b1b1d640 1 -- 172.21.15.16:0/2450577392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ac19b4f0 con 0x7f25ac100b80 2024-04-22T15:46:55.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.025+0000 7f25b1b1d640 1 -- 172.21.15.16:0/2450577392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ac18f9b0 con 0x7f25ac100b80 2024-04-22T15:46:55.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.025+0000 7f25a8ff9640 1 -- 172.21.15.16:0/2450577392 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f259c009b90 con 0x7f25ac100b80 2024-04-22T15:46:55.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.026+0000 7f25a8ff9640 1 --2- 172.21.15.16:0/2450577392 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f259406c450 0x7f259406e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:55.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.026+0000 7f25a8ff9640 1 -- 172.21.15.16:0/2450577392 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f259c08b4c0 con 0x7f25ac100b80 2024-04-22T15:46:55.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.026+0000 7f25aaffd640 1 --2- 172.21.15.16:0/2450577392 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f259406c450 0x7f259406e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:55.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.026+0000 7f25aaffd640 1 --2- 172.21.15.16:0/2450577392 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f259406c450 0x7f259406e910 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f25a0006010 tx=0x7f25a0005a20 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:55.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.030+0000 7f25a8ff9640 1 -- 172.21.15.16:0/2450577392 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f259c055ba0 con 0x7f25ac100b80 2024-04-22T15:46:55.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.212+0000 7f25b1b1d640 1 -- 172.21.15.16:0/2450577392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f25ac02c2e0 con 0x7f25ac100b80 2024-04-22T15:46:55.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.212+0000 7f25a8ff9640 1 -- 172.21.15.16:0/2450577392 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f259c0591c0 con 0x7f25ac100b80 2024-04-22T15:46:55.215 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-22T15:46:55.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.214+0000 7f25b1b1d640 1 -- 172.21.15.16:0/2450577392 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f259406c450 msgr2=0x7f259406e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:55.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.214+0000 7f25b1b1d640 1 --2- 172.21.15.16:0/2450577392 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f259406c450 0x7f259406e910 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f25a0006010 tx=0x7f25a0005a20 comp rx=0 tx=0).stop 2024-04-22T15:46:55.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.214+0000 7f25b1b1d640 1 -- 172.21.15.16:0/2450577392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25ac100b80 msgr2=0x7f25ac195910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:55.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.214+0000 7f25b1b1d640 1 --2- 172.21.15.16:0/2450577392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25ac100b80 0x7f25ac195910 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f259c00d6f0 tx=0x7f259c00dbb0 comp rx=0 tx=0).stop 2024-04-22T15:46:55.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.214+0000 7f25b1b1d640 1 -- 172.21.15.16:0/2450577392 shutdown_connections 2024-04-22T15:46:55.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.214+0000 7f25b1b1d640 1 --2- 172.21.15.16:0/2450577392 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f259406c450 0x7f259406e910 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:55.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.214+0000 7f25b1b1d640 1 --2- 172.21.15.16:0/2450577392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ac1034b0 0x7f25ac195e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:55.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.214+0000 7f25b1b1d640 1 --2- 172.21.15.16:0/2450577392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25ac100b80 0x7f25ac195910 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:55.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.214+0000 7f25b1b1d640 1 -- 172.21.15.16:0/2450577392 >> 172.21.15.16:0/2450577392 conn(0x7f25ac0fa4f0 msgr2=0x7f25ac0fc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:55.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.215+0000 7f25b1b1d640 1 -- 172.21.15.16:0/2450577392 shutdown_connections 2024-04-22T15:46:55.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.215+0000 7f25b1b1d640 1 -- 172.21.15.16:0/2450577392 wait complete. 2024-04-22T15:46:55.229 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T15:46:55.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.328+0000 7f4b0b186640 1 -- 172.21.15.16:0/2657697539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b04100ba0 msgr2=0x7f4b04102f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:55.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.328+0000 7f4b0b186640 1 --2- 172.21.15.16:0/2657697539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b04100ba0 0x7f4b04102f90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4af8009880 tx=0x7f4af802bce0 comp rx=0 tx=0).stop 2024-04-22T15:46:55.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.328+0000 7f4b0b186640 1 -- 172.21.15.16:0/2657697539 shutdown_connections 2024-04-22T15:46:55.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.328+0000 7f4b0b186640 1 --2- 172.21.15.16:0/2657697539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b041034d0 0x7f4b041058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:55.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.328+0000 7f4b0b186640 1 --2- 172.21.15.16:0/2657697539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b04100ba0 0x7f4b04102f90 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:55.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.328+0000 7f4b0b186640 1 -- 172.21.15.16:0/2657697539 >> 172.21.15.16:0/2657697539 conn(0x7f4b040fa510 msgr2=0x7f4b040fc970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:55.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.328+0000 7f4b0b186640 1 -- 172.21.15.16:0/2657697539 shutdown_connections 2024-04-22T15:46:55.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.328+0000 7f4b0b186640 1 -- 172.21.15.16:0/2657697539 wait complete. 2024-04-22T15:46:55.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.329+0000 7f4b0b186640 1 Processor -- start 2024-04-22T15:46:55.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.329+0000 7f4b0b186640 1 -- start start 2024-04-22T15:46:55.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.329+0000 7f4b0b186640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b04100ba0 0x7f4b04195900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:55.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.329+0000 7f4b0b186640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b041034d0 0x7f4b04195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:55.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.329+0000 7f4b0b186640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b04196410 con 0x7f4b041034d0 2024-04-22T15:46:55.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.329+0000 7f4b0b186640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b04196580 con 0x7f4b04100ba0 2024-04-22T15:46:55.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.329+0000 7f4b03fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b041034d0 0x7f4b04195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:55.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.329+0000 7f4b03fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b041034d0 0x7f4b04195e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57990/0 (socket says 172.21.15.16:57990) 2024-04-22T15:46:55.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.329+0000 7f4b03fff640 1 -- 172.21.15.16:0/2117904040 learned_addr learned my addr 172.21.15.16:0/2117904040 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:46:55.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.329+0000 7f4b08f22640 1 --2- 172.21.15.16:0/2117904040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b04100ba0 0x7f4b04195900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:55.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.329+0000 7f4b03fff640 1 -- 172.21.15.16:0/2117904040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b04100ba0 msgr2=0x7f4b04195900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:55.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.329+0000 7f4b03fff640 1 --2- 172.21.15.16:0/2117904040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b04100ba0 0x7f4b04195900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:55.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.329+0000 7f4b03fff640 1 -- 172.21.15.16:0/2117904040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af8009560 con 0x7f4b041034d0 2024-04-22T15:46:55.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.330+0000 7f4b03fff640 1 --2- 172.21.15.16:0/2117904040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b041034d0 0x7f4b04195e40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4af0005fd0 tx=0x7f4af000dd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:55.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.330+0000 7f4b01ffb640 1 -- 172.21.15.16:0/2117904040 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4af0009c00 con 0x7f4b041034d0 2024-04-22T15:46:55.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.330+0000 7f4b01ffb640 1 -- 172.21.15.16:0/2117904040 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4af000de90 con 0x7f4b041034d0 2024-04-22T15:46:55.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.330+0000 7f4b0b186640 1 -- 172.21.15.16:0/2117904040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b0419b010 con 0x7f4b041034d0 2024-04-22T15:46:55.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.330+0000 7f4b01ffb640 1 -- 172.21.15.16:0/2117904040 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4af0003e70 con 0x7f4b041034d0 2024-04-22T15:46:55.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.330+0000 7f4b0b186640 1 -- 172.21.15.16:0/2117904040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b0419b540 con 0x7f4b041034d0 2024-04-22T15:46:55.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.331+0000 7f4b0b186640 1 -- 172.21.15.16:0/2117904040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b0418f9d0 con 0x7f4b041034d0 2024-04-22T15:46:55.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.331+0000 7f4b01ffb640 1 -- 172.21.15.16:0/2117904040 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f4af0004000 con 0x7f4b041034d0 2024-04-22T15:46:55.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.332+0000 7f4b01ffb640 1 --2- 172.21.15.16:0/2117904040 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4af406c450 0x7f4af406e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:55.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.332+0000 7f4b08f22640 1 --2- 172.21.15.16:0/2117904040 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4af406c450 0x7f4af406e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:55.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.332+0000 7f4b01ffb640 1 -- 172.21.15.16:0/2117904040 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f4af008b500 con 0x7f4b041034d0 2024-04-22T15:46:55.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.332+0000 7f4b08f22640 1 --2- 172.21.15.16:0/2117904040 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4af406c450 0x7f4af406e910 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4af8006010 tx=0x7f4af80058d0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:55.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.336+0000 7f4b01ffb640 1 -- 172.21.15.16:0/2117904040 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4af0055d00 con 0x7f4b041034d0 2024-04-22T15:46:55.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.564+0000 7f4b0b186640 1 -- 172.21.15.16:0/2117904040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b04108c50 con 0x7f4b041034d0 2024-04-22T15:46:55.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.564+0000 7f4b01ffb640 1 -- 172.21.15.16:0/2117904040 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4744 (secure 0 0 0) 0x7f4af0059320 con 0x7f4b041034d0 2024-04-22T15:46:55.568 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:46:55.568 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14504,"name":"cephfs.smithi016.fkkejm","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6827/3484907755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3484907755},{"type":"v1","addr":"172.21.15.16:6827","nonce":3484907755}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24293,"name":"cephfs.smithi027.dgqyes","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.27:6827/476026355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":476026355},{"type":"v1","addr":"172.21.15.27:6827","nonce":476026355}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":16,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:46:27.774571+0000","modified":"2024-04-22T15:46:47.381709+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14494":{"gid":14494,"name":"cephfs.smithi016.pgvphu","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.16:6829/1006073414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1006073414},{"type":"v1","addr":"172.21.15.16:6829","nonce":1006073414}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24279":{"gid":24279,"name":"cephfs.smithi027.iobjds","rank":0,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6825/4216277609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4216277609},{"type":"v1","addr":"172.21.15.27:6825","nonce":4216277609}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-22T15:46:55.569 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 16 2024-04-22T15:46:55.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.567+0000 7f4b0b186640 1 -- 172.21.15.16:0/2117904040 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4af406c450 msgr2=0x7f4af406e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:55.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.567+0000 7f4b0b186640 1 --2- 172.21.15.16:0/2117904040 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4af406c450 0x7f4af406e910 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4af8006010 tx=0x7f4af80058d0 comp rx=0 tx=0).stop 2024-04-22T15:46:55.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.567+0000 7f4b0b186640 1 -- 172.21.15.16:0/2117904040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b041034d0 msgr2=0x7f4b04195e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:55.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.567+0000 7f4b0b186640 1 --2- 172.21.15.16:0/2117904040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b041034d0 0x7f4b04195e40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4af0005fd0 tx=0x7f4af000dd60 comp rx=0 tx=0).stop 2024-04-22T15:46:55.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.567+0000 7f4b0b186640 1 -- 172.21.15.16:0/2117904040 shutdown_connections 2024-04-22T15:46:55.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.567+0000 7f4b0b186640 1 --2- 172.21.15.16:0/2117904040 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4af406c450 0x7f4af406e910 secure :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4af8006010 tx=0x7f4af80058d0 comp rx=0 tx=0).stop 2024-04-22T15:46:55.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.567+0000 7f4b0b186640 1 --2- 172.21.15.16:0/2117904040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b041034d0 0x7f4b04195e40 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:55.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.567+0000 7f4b0b186640 1 --2- 172.21.15.16:0/2117904040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b04100ba0 0x7f4b04195900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:55.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.567+0000 7f4b0b186640 1 -- 172.21.15.16:0/2117904040 >> 172.21.15.16:0/2117904040 conn(0x7f4b040fa510 msgr2=0x7f4b040fc970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:55.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.567+0000 7f4b0b186640 1 -- 172.21.15.16:0/2117904040 shutdown_connections 2024-04-22T15:46:55.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:55.567+0000 7f4b0b186640 1 -- 172.21.15.16:0/2117904040 wait complete. 2024-04-22T15:46:55.580 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:55 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:55.580 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:55 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:55.580 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:55 smithi016 ceph-mon[26620]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.9 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-22T15:46:55.580 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:55 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/3099883527' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T15:46:55.580 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:55 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/3149524193' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-22T15:46:55.580 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:55 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/2450577392' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-22T15:46:55.581 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 16, 'max_mds': 1, 'flags': 50} 2024-04-22T15:46:55.582 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-22T15:46:55.607 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-22T15:46:55.607 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2024-04-22T15:46:55.607 INFO:tasks.ceph_fuse:client.0 config is {} 2024-04-22T15:46:55.607 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-22T15:46:55.608 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T15:46:55.608 INFO:tasks.ceph_fuse:client.1 config is {} 2024-04-22T15:46:55.608 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-22T15:46:55.608 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-22T15:46:55.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:46:55.608 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-04-22T15:46:55.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:46:55.625 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2024-04-22T15:46:55.691 INFO:teuthology.orchestra.run.smithi016.stderr:Cannot find device "ceph-brx" 2024-04-22T15:46:55.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T15:46:55.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:46:55.693 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-04-22T15:46:55.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:46:55.714 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link delete ceph-brx 2024-04-22T15:46:55.780 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:55 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:55.780 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:55 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:55.780 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:55 smithi027 ceph-mon[34922]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.9 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-22T15:46:55.780 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:55 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/3099883527' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T15:46:55.781 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:55 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/3149524193' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-22T15:46:55.781 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:55 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/2450577392' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-22T15:46:55.787 INFO:teuthology.orchestra.run.smithi027.stderr:Cannot find device "ceph-brx" 2024-04-22T15:46:55.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T15:46:55.789 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-22T15:46:55.789 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-22T15:46:56.775 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:56 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:56.775 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:56 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:46:56.775 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:56 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/2117904040' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T15:46:56.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:56 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:56.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:56 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:46:56.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:56 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/2117904040' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T15:46:57.175 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:46:57.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:57 smithi016 ceph-mon[26620]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 7.9 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-22T15:46:57.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:57 smithi027 ceph-mon[34922]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 7.9 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-22T15:46:58.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.738+0000 7f4cb50e2700 1 -- 172.21.15.16:0/972327973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb0105bd0 msgr2=0x7f4cb0105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:58.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.738+0000 7f4cb50e2700 1 --2- 172.21.15.16:0/972327973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb0105bd0 0x7f4cb0105fb0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0009910 tx=0x7f4ca0009c20 comp rx=0 tx=0).stop 2024-04-22T15:46:58.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.738+0000 7f4cb50e2700 1 -- 172.21.15.16:0/972327973 shutdown_connections 2024-04-22T15:46:58.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.738+0000 7f4cb50e2700 1 --2- 172.21.15.16:0/972327973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4cb00ffc20 0x7f4cb0100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:58.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.738+0000 7f4cb50e2700 1 --2- 172.21.15.16:0/972327973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb0105bd0 0x7f4cb0105fb0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:58.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.738+0000 7f4cb50e2700 1 -- 172.21.15.16:0/972327973 >> 172.21.15.16:0/972327973 conn(0x7f4cb0075670 msgr2=0x7f4cb0075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:58.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.738+0000 7f4cb50e2700 1 -- 172.21.15.16:0/972327973 shutdown_connections 2024-04-22T15:46:58.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.738+0000 7f4cb50e2700 1 -- 172.21.15.16:0/972327973 wait complete. 2024-04-22T15:46:58.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.739+0000 7f4cb50e2700 1 Processor -- start 2024-04-22T15:46:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.739+0000 7f4cb50e2700 1 -- start start 2024-04-22T15:46:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.739+0000 7f4cb50e2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb00ffc20 0x7f4cb0101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.739+0000 7f4cb50e2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4cb0105bd0 0x7f4cb0101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:58.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.739+0000 7f4cb50e2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb01022c0 con 0x7f4cb00ffc20 2024-04-22T15:46:58.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.739+0000 7f4cb50e2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb0102400 con 0x7f4cb0105bd0 2024-04-22T15:46:58.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.739+0000 7f4caed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb00ffc20 0x7f4cb0101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:58.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.739+0000 7f4caed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb00ffc20 0x7f4cb0101840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58008/0 (socket says 172.21.15.16:58008) 2024-04-22T15:46:58.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.739+0000 7f4caed9d700 1 -- 172.21.15.16:0/1159387677 learned_addr learned my addr 172.21.15.16:0/1159387677 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:46:58.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.740+0000 7f4cae59c700 1 --2- 172.21.15.16:0/1159387677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4cb0105bd0 0x7f4cb0101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:58.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.740+0000 7f4caed9d700 1 -- 172.21.15.16:0/1159387677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4cb0105bd0 msgr2=0x7f4cb0101d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:58.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.740+0000 7f4caed9d700 1 --2- 172.21.15.16:0/1159387677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4cb0105bd0 0x7f4cb0101d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:58.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.740+0000 7f4caed9d700 1 -- 172.21.15.16:0/1159387677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca00095f0 con 0x7f4cb00ffc20 2024-04-22T15:46:58.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.740+0000 7f4caed9d700 1 --2- 172.21.15.16:0/1159387677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb00ffc20 0x7f4cb0101840 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0006010 tx=0x7f4ca000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:58.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.740+0000 7f4c97fff700 1 -- 172.21.15.16:0/1159387677 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ca001d070 con 0x7f4cb00ffc20 2024-04-22T15:46:58.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.740+0000 7f4cb50e2700 1 -- 172.21.15.16:0/1159387677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb0102680 con 0x7f4cb00ffc20 2024-04-22T15:46:58.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.740+0000 7f4c97fff700 1 -- 172.21.15.16:0/1159387677 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4ca000f460 con 0x7f4cb00ffc20 2024-04-22T15:46:58.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.740+0000 7f4c97fff700 1 -- 172.21.15.16:0/1159387677 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ca0017630 con 0x7f4cb00ffc20 2024-04-22T15:46:58.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.740+0000 7f4cb50e2700 1 -- 172.21.15.16:0/1159387677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb01a2d50 con 0x7f4cb00ffc20 2024-04-22T15:46:58.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.741+0000 7f4cb50e2700 1 -- 172.21.15.16:0/1159387677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb004f2e0 con 0x7f4cb00ffc20 2024-04-22T15:46:58.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.742+0000 7f4c97fff700 1 -- 172.21.15.16:0/1159387677 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f4ca000f5d0 con 0x7f4cb00ffc20 2024-04-22T15:46:58.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.742+0000 7f4c97fff700 1 --2- 172.21.15.16:0/1159387677 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4c9806c4e0 0x7f4c9806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:46:58.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.742+0000 7f4c97fff700 1 -- 172.21.15.16:0/1159387677 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f4ca008ced0 con 0x7f4cb00ffc20 2024-04-22T15:46:58.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.742+0000 7f4cae59c700 1 --2- 172.21.15.16:0/1159387677 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4c9806c4e0 0x7f4c9806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:46:58.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.743+0000 7f4cae59c700 1 --2- 172.21.15.16:0/1159387677 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4c9806c4e0 0x7f4c9806e9a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4005fd0 tx=0x7f4ca400a560 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:46:58.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.747+0000 7f4c97fff700 1 -- 172.21.15.16:0/1159387677 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4ca0057650 con 0x7f4cb00ffc20 2024-04-22T15:46:58.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.992+0000 7f4cb50e2700 1 -- 172.21.15.16:0/1159387677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f4cb0102d70 con 0x7f4cb00ffc20 2024-04-22T15:46:58.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.992+0000 7f4c97fff700 1 -- 172.21.15.16:0/1159387677 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+83 (secure 0 0 0) 0x7f4ca005ac70 con 0x7f4cb00ffc20 2024-04-22T15:46:58.995 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-22T15:46:58.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.994+0000 7f4cb50e2700 1 -- 172.21.15.16:0/1159387677 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4c9806c4e0 msgr2=0x7f4c9806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:58.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.994+0000 7f4cb50e2700 1 --2- 172.21.15.16:0/1159387677 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4c9806c4e0 0x7f4c9806e9a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4005fd0 tx=0x7f4ca400a560 comp rx=0 tx=0).stop 2024-04-22T15:46:58.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.994+0000 7f4cb50e2700 1 -- 172.21.15.16:0/1159387677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb00ffc20 msgr2=0x7f4cb0101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:46:58.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.994+0000 7f4cb50e2700 1 --2- 172.21.15.16:0/1159387677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb00ffc20 0x7f4cb0101840 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0006010 tx=0x7f4ca000be20 comp rx=0 tx=0).stop 2024-04-22T15:46:58.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.994+0000 7f4cb50e2700 1 -- 172.21.15.16:0/1159387677 shutdown_connections 2024-04-22T15:46:58.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.994+0000 7f4cb50e2700 1 --2- 172.21.15.16:0/1159387677 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4c9806c4e0 0x7f4c9806e9a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:58.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.994+0000 7f4cb50e2700 1 --2- 172.21.15.16:0/1159387677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb00ffc20 0x7f4cb0101840 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:58.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.994+0000 7f4cb50e2700 1 --2- 172.21.15.16:0/1159387677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4cb0105bd0 0x7f4cb0101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:46:58.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.994+0000 7f4cb50e2700 1 -- 172.21.15.16:0/1159387677 >> 172.21.15.16:0/1159387677 conn(0x7f4cb0075670 msgr2=0x7f4cb00686c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:46:58.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.995+0000 7f4cb50e2700 1 -- 172.21.15.16:0/1159387677 shutdown_connections 2024-04-22T15:46:58.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:58.995+0000 7f4cb50e2700 1 -- 172.21.15.16:0/1159387677 wait complete. 2024-04-22T15:46:59.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:58 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:59.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:58 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:59.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:58 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:46:59.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:58 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:46:59.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:58 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:59.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:58 smithi016 ceph-mon[26620]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 7.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-22T15:46:59.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:58 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:59.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:58 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:59.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:58 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:46:59.339 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:58 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:46:59.339 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:58 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:46:59.339 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:58 smithi027 ceph-mon[34922]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 7.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-22T15:46:59.627 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T15:46:59.627 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T15:46:59.627 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-04-22T15:46:59.628 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-22T15:46:59.628 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T15:46:59.628 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T15:46:59.628 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T15:46:59.628 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T15:46:59.628 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-22T15:46:59.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:46:59.628 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-04-22T15:46:59.644 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T15:46:59.645 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T15:46:59.645 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T15:46:59.645 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-22T15:46:59.645 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-04-22T15:46:59.645 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-22T15:46:59.645 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T15:46:59.645 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-04-22T15:46:59.645 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-04-22T15:46:59.645 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T15:46:59.645 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-04-22T15:46:59.645 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-04-22T15:46:59.645 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T15:46:59.645 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-04-22T15:46:59.646 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-04-22T15:46:59.646 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-22T15:46:59.646 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42157sec preferred_lft 42157sec 2024-04-22T15:46:59.646 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link noprefixroute 2024-04-22T15:46:59.646 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-22T15:46:59.646 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T15:46:59.646 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-04-22T15:46:59.646 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-04-22T15:46:59.647 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-22T15:46:59.647 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T15:46:59.647 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-22T15:46:59.647 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add name ceph-brx type bridge 2024-04-22T15:46:59.647 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr flush dev ceph-brx 2024-04-22T15:46:59.647 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set ceph-brx up 2024-04-22T15:46:59.647 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-22T15:46:59.647 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-22T15:46:59.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T15:46:59.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:46:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T15:46:59.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:46:59.837 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T15:46:59.861 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-04-22T15:46:59.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:46:59.863 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-04-22T15:46:59.930 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-04-22T15:46:59.931 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T15:46:59.931 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-22T15:46:59.931 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-22T15:46:59.931 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T15:46:59.932 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T15:46:59.932 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-22T15:46:59.932 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-22T15:46:59.933 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-22T15:46:59.933 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-22T15:46:59.933 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-22T15:47:00.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T15:47:00.121 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:46:59 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/1159387677' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T15:47:00.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T15:47:00.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:47:00.217 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-04-22T15:47:00.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:47:00.233 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list-id 2024-04-22T15:47:00.288 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T15:47:00.288 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-22T15:47:00.288 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T15:47:00.289 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-22T15:47:00.289 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-22T15:47:00.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:46:59 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/1159387677' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T15:47:00.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T15:47:00.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T15:47:00.406 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-22T15:47:00.407 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T15:47:00.407 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-22T15:47:00.407 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-22T15:47:00.407 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-22T15:47:00.407 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-22T15:47:00.407 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-22T15:47:00.407 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-22T15:47:00.407 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-22T15:47:00.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T15:47:00.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T15:47:00.536 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T15:47:00.536 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-22T15:47:00.536 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-04-22T15:47:00.536 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set dev brx.0 master ceph-brx 2024-04-22T15:47:00.536 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-22T15:47:00.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T15:47:00.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T15:47:00.613 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-22T15:47:00.613 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T15:47:00.613 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-22T15:47:00.628 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-22T15:47:00.628 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T15:47:00.628 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-22T15:47:00.683 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-04-22T15:47:00.749 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T15:47:00.806 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-04-22T15:47:00.806 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-04-22T15:47:00.806 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-04-22T15:47:00.807 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-04-22T15:47:00.807 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-04-22T15:47:00.807 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-04-22T15:47:00.807 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-04-22T15:47:00.807 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-04-22T15:47:00.807 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-04-22T15:47:00.807 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-04-22T15:47:00.807 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-04-22T15:47:00.807 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2024-04-22T15:47:00.807 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T15:47:00.807 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-04-22T15:47:00.807 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-04-22T15:47:00.807 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-04-22T15:47:00.807 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-04-22T15:47:00.807 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T15:47:00.808 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-04-22T15:47:00.808 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-04-22T15:47:00.808 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysctl.service 2024-04-22T15:47:00.808 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-22T15:47:00.808 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-22T15:47:00.808 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2024-04-22T15:47:00.808 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-04-22T15:47:00.808 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-04-22T15:47:00.808 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay 2024-04-22T15:47:00.808 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/c930765c70364866270a212fb7e20ff9a8028eff44af52fa356d79da2e21b6f6/merged 2024-04-22T15:47:00.808 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/3fb28387bfcb5fb88db4b77aa5de487ed94bb2afbea32b6245379506ab29dd53/merged 2024-04-22T15:47:00.808 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/0 2024-04-22T15:47:00.808 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T15:47:00.808 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/fdd622ba54ef5a6683784144149edabc97402b30be06fc1b47dbcf71d432d5ec/merged 2024-04-22T15:47:00.809 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/f59f49ca95d49dc42fe71d798d153d995f34a9723c083887652a676652714010/merged 2024-04-22T15:47:00.809 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/92113f59a8f1ae957d05fc674c3744c760495afef751966c75a5cfb358c1b00b/merged 2024-04-22T15:47:00.809 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/34fc0312ad0ab6a5e413122c4c519b59722321d4f04b273a1c8dfa39aeb4e4f1/merged 2024-04-22T15:47:00.809 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/033e86d5d407dbe8f8f57906ae6361b33f40cafc5dfc3ec05ee3b3ddbbebc528/merged 2024-04-22T15:47:00.809 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/b721a2da3e08d773ec167d11944514b29505a895c747963a1f7780cd7b2b73e0/merged 2024-04-22T15:47:00.809 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/52509b4ed8cede37320a920e89e0c73b671a3ab0fd4d6a72999a242aafead7b1/merged 2024-04-22T15:47:00.809 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/4cb52702590f1e3eafe2e0140bf1c856bacd0ce3d986195634df3394e4882a73/merged 2024-04-22T15:47:00.809 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/67cdf529c80966f9ee89e1d4b86d38d6ec99c8a1eec13f0901c8233a8e002662/merged 2024-04-22T15:47:00.809 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/5c61ee3a3f9a7bafc4db1f09c04f44c1bb50b55adb02cd3a794deaee1b16eed8/merged 2024-04-22T15:47:00.809 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/b5356c1367174a0ce73704efd3fbeea6ee90bcd1f224dab0733d55befb77d675/merged 2024-04-22T15:47:00.809 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-04-22T15:47:00.809 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T15:47:00.809 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T15:47:00.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:47:00.810 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-04-22T15:47:00.862 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-22T15:47:00.862 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-22T15:47:00.904 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-04-22T15:47:00.928 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T15:47:00.985 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-04-22T15:47:00.985 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-04-22T15:47:00.985 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-04-22T15:47:00.985 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-04-22T15:47:00.985 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-04-22T15:47:00.985 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-04-22T15:47:00.985 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-04-22T15:47:00.985 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-04-22T15:47:00.985 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-04-22T15:47:00.985 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-04-22T15:47:00.985 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-04-22T15:47:00.986 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2024-04-22T15:47:00.986 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T15:47:00.986 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-04-22T15:47:00.986 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-04-22T15:47:00.986 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-04-22T15:47:00.986 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-04-22T15:47:00.986 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T15:47:00.986 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-04-22T15:47:00.986 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-04-22T15:47:00.986 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysctl.service 2024-04-22T15:47:00.986 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-22T15:47:00.986 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-22T15:47:00.986 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2024-04-22T15:47:00.987 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-04-22T15:47:00.987 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-04-22T15:47:00.987 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay 2024-04-22T15:47:00.987 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/c930765c70364866270a212fb7e20ff9a8028eff44af52fa356d79da2e21b6f6/merged 2024-04-22T15:47:00.987 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/3fb28387bfcb5fb88db4b77aa5de487ed94bb2afbea32b6245379506ab29dd53/merged 2024-04-22T15:47:00.987 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/0 2024-04-22T15:47:00.987 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T15:47:00.987 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/fdd622ba54ef5a6683784144149edabc97402b30be06fc1b47dbcf71d432d5ec/merged 2024-04-22T15:47:00.987 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/f59f49ca95d49dc42fe71d798d153d995f34a9723c083887652a676652714010/merged 2024-04-22T15:47:00.987 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/92113f59a8f1ae957d05fc674c3744c760495afef751966c75a5cfb358c1b00b/merged 2024-04-22T15:47:00.987 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/34fc0312ad0ab6a5e413122c4c519b59722321d4f04b273a1c8dfa39aeb4e4f1/merged 2024-04-22T15:47:00.987 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/033e86d5d407dbe8f8f57906ae6361b33f40cafc5dfc3ec05ee3b3ddbbebc528/merged 2024-04-22T15:47:00.987 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/b721a2da3e08d773ec167d11944514b29505a895c747963a1f7780cd7b2b73e0/merged 2024-04-22T15:47:00.988 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/52509b4ed8cede37320a920e89e0c73b671a3ab0fd4d6a72999a242aafead7b1/merged 2024-04-22T15:47:00.988 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/4cb52702590f1e3eafe2e0140bf1c856bacd0ce3d986195634df3394e4882a73/merged 2024-04-22T15:47:00.988 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/67cdf529c80966f9ee89e1d4b86d38d6ec99c8a1eec13f0901c8233a8e002662/merged 2024-04-22T15:47:00.988 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/5c61ee3a3f9a7bafc4db1f09c04f44c1bb50b55adb02cd3a794deaee1b16eed8/merged 2024-04-22T15:47:00.988 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/b5356c1367174a0ce73704efd3fbeea6ee90bcd1f224dab0733d55befb77d675/merged 2024-04-22T15:47:00.988 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-04-22T15:47:00.988 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T15:47:00.988 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T15:47:00.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:47:00.988 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-04-22T15:47:00.998 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2024-04-22T15:47:00.994+0000 7f4a77841480 -1 init, newargv = 0x562ab1d80ee0 newargc=15 2024-04-22T15:47:00.998 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[75136]: starting ceph client 2024-04-22T15:47:01.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[75136]: starting fuse 2024-04-22T15:47:01.014 INFO:teuthology.orchestra.run.smithi016.stdout:147 2024-04-22T15:47:01.014 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [147] 2024-04-22T15:47:01.014 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-04-22T15:47:01.015 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-04-22T15:47:01.015 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-04-22T15:47:01.015 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-04-22T15:47:01.015 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-04-22T15:47:01.015 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-22T15:47:01.015 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-04-22T15:47:01.015 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-22T15:47:01.015 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-04-22T15:47:01.016 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-22T15:47:01.016 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-22T15:47:01.016 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-04-22T15:47:01.016 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-04-22T15:47:01.016 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-04-22T15:47:01.016 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-04-22T15:47:01.016 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-22T15:47:01.017 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-04-22T15:47:01.017 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-22T15:47:01.017 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-22T15:47:01.017 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-22T15:47:01.017 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-04-22T15:47:01.017 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-04-22T15:47:01.017 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-04-22T15:47:01.017 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-22T15:47:01.018 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-22T15:47:01.018 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2024-04-22T15:47:01.018 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-04-22T15:47:01.075 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:00 smithi016 ceph-mon[26620]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T15:47:01.133 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.75136.asok 2024-04-22T15:47:01.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T15:47:01.144 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.75136.asok 2024-04-22T15:47:01.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:47:01.145 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.75136.asok status 2024-04-22T15:47:01.281 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:47:01.281 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-04-22T15:47:01.281 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "7fe91d5d5842e04be3b4f514d6dd990c54b29c76", 2024-04-22T15:47:01.281 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-04-22T15:47:01.281 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2024-04-22T15:47:01.281 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-04-22T15:47:01.281 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-22T15:47:01.281 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "75136", 2024-04-22T15:47:01.281 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2024-04-22T15:47:01.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:47:01.282 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-04-22T15:47:01.282 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-04-22T15:47:01.282 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 14544, 2024-04-22T15:47:01.282 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-04-22T15:47:01.282 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-04-22T15:47:01.282 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-04-22T15:47:01.282 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 14544 2024-04-22T15:47:01.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:47:01.282 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-04-22T15:47:01.282 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-04-22T15:47:01.282 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-04-22T15:47:01.282 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3830921175 2024-04-22T15:47:01.282 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-22T15:47:01.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:47:01.283 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-04-22T15:47:01.283 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-04-22T15:47:01.283 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-04-22T15:47:01.283 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3830921175 2024-04-22T15:47:01.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:47:01.283 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.14544 192.168.0.1:0/3830921175", 2024-04-22T15:47:01.283 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/3830921175", 2024-04-22T15:47:01.283 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2024-04-22T15:47:01.283 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 16, 2024-04-22T15:47:01.283 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 46, 2024-04-22T15:47:01.283 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-04-22T15:47:01.283 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-04-22T15:47:01.283 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-04-22T15:47:01.283 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:47:01.289 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-22T15:47:01.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:00 smithi027 ceph-mon[34922]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T15:47:01.500 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:47:03.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.202+0000 7fc8c94ef700 1 -- 172.21.15.16:0/940979890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c4101e00 msgr2=0x7fc8c41021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:03.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.202+0000 7fc8c94ef700 1 --2- 172.21.15.16:0/940979890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c4101e00 0x7fc8c41021e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4009910 tx=0x7fc8b4009c20 comp rx=0 tx=0).stop 2024-04-22T15:47:03.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.202+0000 7fc8c94ef700 1 -- 172.21.15.16:0/940979890 shutdown_connections 2024-04-22T15:47:03.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.202+0000 7fc8c94ef700 1 --2- 172.21.15.16:0/940979890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8c4102720 0x7fc8c410ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:03.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.202+0000 7fc8c94ef700 1 --2- 172.21.15.16:0/940979890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c4101e00 0x7fc8c41021e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:03.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.202+0000 7fc8c94ef700 1 -- 172.21.15.16:0/940979890 >> 172.21.15.16:0/940979890 conn(0x7fc8c40fb650 msgr2=0x7fc8c40fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:03.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.202+0000 7fc8c94ef700 1 -- 172.21.15.16:0/940979890 shutdown_connections 2024-04-22T15:47:03.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.202+0000 7fc8c94ef700 1 -- 172.21.15.16:0/940979890 wait complete. 2024-04-22T15:47:03.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.203+0000 7fc8c94ef700 1 Processor -- start 2024-04-22T15:47:03.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.203+0000 7fc8c94ef700 1 -- start start 2024-04-22T15:47:03.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.203+0000 7fc8c94ef700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c4101e00 0x7fc8c419c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:03.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.203+0000 7fc8c94ef700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8c4102720 0x7fc8c419cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:03.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.203+0000 7fc8c94ef700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c419d260 con 0x7fc8c4101e00 2024-04-22T15:47:03.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.203+0000 7fc8c94ef700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c4196710 con 0x7fc8c4102720 2024-04-22T15:47:03.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.203+0000 7fc8c2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c4101e00 0x7fc8c419c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:03.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.203+0000 7fc8c2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c4101e00 0x7fc8c419c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37660/0 (socket says 172.21.15.16:37660) 2024-04-22T15:47:03.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.203+0000 7fc8c27fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8c4102720 0x7fc8c419cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:03.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.203+0000 7fc8c2ffd700 1 -- 172.21.15.16:0/3068464214 learned_addr learned my addr 172.21.15.16:0/3068464214 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:47:03.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.204+0000 7fc8c2ffd700 1 -- 172.21.15.16:0/3068464214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8c4102720 msgr2=0x7fc8c419cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:03.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.204+0000 7fc8c2ffd700 1 --2- 172.21.15.16:0/3068464214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8c4102720 0x7fc8c419cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:03.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.204+0000 7fc8c2ffd700 1 -- 172.21.15.16:0/3068464214 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b40095f0 con 0x7fc8c4101e00 2024-04-22T15:47:03.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.204+0000 7fc8c2ffd700 1 --2- 172.21.15.16:0/3068464214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c4101e00 0x7fc8c419c690 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4006010 tx=0x7fc8b400fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:03.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.204+0000 7fc8abfff700 1 -- 172.21.15.16:0/3068464214 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8b401d070 con 0x7fc8c4101e00 2024-04-22T15:47:03.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.204+0000 7fc8abfff700 1 -- 172.21.15.16:0/3068464214 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc8b400fe60 con 0x7fc8c4101e00 2024-04-22T15:47:03.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.204+0000 7fc8c94ef700 1 -- 172.21.15.16:0/3068464214 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8c4196990 con 0x7fc8c4101e00 2024-04-22T15:47:03.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.204+0000 7fc8abfff700 1 -- 172.21.15.16:0/3068464214 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8b4017400 con 0x7fc8c4101e00 2024-04-22T15:47:03.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.204+0000 7fc8c94ef700 1 -- 172.21.15.16:0/3068464214 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8c4196e80 con 0x7fc8c4101e00 2024-04-22T15:47:03.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.205+0000 7fc8c94ef700 1 -- 172.21.15.16:0/3068464214 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8c4108310 con 0x7fc8c4101e00 2024-04-22T15:47:03.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.206+0000 7fc8abfff700 1 -- 172.21.15.16:0/3068464214 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fc8b4003680 con 0x7fc8c4101e00 2024-04-22T15:47:03.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.206+0000 7fc8abfff700 1 --2- 172.21.15.16:0/3068464214 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fc8ac06c4e0 0x7fc8ac06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:03.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.206+0000 7fc8abfff700 1 -- 172.21.15.16:0/3068464214 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fc8b408cc90 con 0x7fc8c4101e00 2024-04-22T15:47:03.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.206+0000 7fc8c27fc700 1 --2- 172.21.15.16:0/3068464214 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fc8ac06c4e0 0x7fc8ac06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:03.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.207+0000 7fc8c27fc700 1 --2- 172.21.15.16:0/3068464214 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fc8ac06c4e0 0x7fc8ac06e9a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4197b80 tx=0x7fc8b8009260 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.211+0000 7fc8abfff700 1 -- 172.21.15.16:0/3068464214 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc8b405af20 con 0x7fc8c4101e00 2024-04-22T15:47:03.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.451+0000 7fc8c94ef700 1 -- 172.21.15.16:0/3068464214 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc8c4197cc0 con 0x7fc8c4101e00 2024-04-22T15:47:03.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.452+0000 7fc8abfff700 1 -- 172.21.15.16:0/3068464214 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+83 (secure 0 0 0) 0x7fc8b405aab0 con 0x7fc8c4101e00 2024-04-22T15:47:03.454 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-22T15:47:03.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.454+0000 7fc8c94ef700 1 -- 172.21.15.16:0/3068464214 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fc8ac06c4e0 msgr2=0x7fc8ac06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:03.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.454+0000 7fc8c94ef700 1 --2- 172.21.15.16:0/3068464214 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fc8ac06c4e0 0x7fc8ac06e9a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4197b80 tx=0x7fc8b8009260 comp rx=0 tx=0).stop 2024-04-22T15:47:03.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.454+0000 7fc8c94ef700 1 -- 172.21.15.16:0/3068464214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c4101e00 msgr2=0x7fc8c419c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:03.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.454+0000 7fc8c94ef700 1 --2- 172.21.15.16:0/3068464214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c4101e00 0x7fc8c419c690 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4006010 tx=0x7fc8b400fa00 comp rx=0 tx=0).stop 2024-04-22T15:47:03.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.454+0000 7fc8c94ef700 1 -- 172.21.15.16:0/3068464214 shutdown_connections 2024-04-22T15:47:03.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.454+0000 7fc8c94ef700 1 --2- 172.21.15.16:0/3068464214 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fc8ac06c4e0 0x7fc8ac06e9a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:03.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.454+0000 7fc8c94ef700 1 --2- 172.21.15.16:0/3068464214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c4101e00 0x7fc8c419c690 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:03.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.454+0000 7fc8c94ef700 1 --2- 172.21.15.16:0/3068464214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8c4102720 0x7fc8c419cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:03.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.454+0000 7fc8c94ef700 1 -- 172.21.15.16:0/3068464214 >> 172.21.15.16:0/3068464214 conn(0x7fc8c40fb650 msgr2=0x7fc8c4105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:03.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.454+0000 7fc8c94ef700 1 -- 172.21.15.16:0/3068464214 shutdown_connections 2024-04-22T15:47:03.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:03.454+0000 7fc8c94ef700 1 -- 172.21.15.16:0/3068464214 wait complete. 2024-04-22T15:47:03.763 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:03 smithi016 ceph-mon[26620]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.6 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-22T15:47:03.763 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:03 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/3068464214' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T15:47:03.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:03 smithi027 ceph-mon[34922]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.6 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-22T15:47:03.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:03 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/3068464214' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T15:47:04.079 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T15:47:04.079 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T15:47:04.079 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-04-22T15:47:04.079 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-22T15:47:04.079 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-22T15:47:04.079 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T15:47:04.079 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T15:47:04.079 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T15:47:04.079 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-22T15:47:04.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:47:04.080 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-04-22T15:47:04.099 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T15:47:04.099 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T15:47:04.099 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T15:47:04.099 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-04-22T15:47:04.099 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-04-22T15:47:04.099 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-04-22T15:47:04.099 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T15:47:04.100 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-04-22T15:47:04.100 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-04-22T15:47:04.100 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T15:47:04.100 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-04-22T15:47:04.100 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-04-22T15:47:04.100 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T15:47:04.100 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-04-22T15:47:04.100 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-04-22T15:47:04.100 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-22T15:47:04.100 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42248sec preferred_lft 42248sec 2024-04-22T15:47:04.100 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-04-22T15:47:04.101 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-04-22T15:47:04.101 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T15:47:04.101 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-04-22T15:47:04.101 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-04-22T15:47:04.101 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-22T15:47:04.101 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T15:47:04.101 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-22T15:47:04.101 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add name ceph-brx type bridge 2024-04-22T15:47:04.101 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr flush dev ceph-brx 2024-04-22T15:47:04.101 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set ceph-brx up 2024-04-22T15:47:04.101 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-22T15:47:04.102 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-22T15:47:04.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:47:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T15:47:04.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:47:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T15:47:04.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:47:04.390 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T15:47:04.416 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-04-22T15:47:04.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:47:04.418 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-04-22T15:47:04.489 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-04-22T15:47:04.489 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T15:47:04.489 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-22T15:47:04.489 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-22T15:47:04.489 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T15:47:04.490 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T15:47:04.490 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-22T15:47:04.490 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-22T15:47:04.490 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-22T15:47:04.490 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-22T15:47:04.490 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-22T15:47:04.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:47:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T15:47:04.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:47:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T15:47:04.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:47:04.775 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-04-22T15:47:04.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:47:04.791 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list-id 2024-04-22T15:47:04.846 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T15:47:04.846 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-22T15:47:04.846 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T15:47:04.846 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-22T15:47:04.846 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-22T15:47:04.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:47:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T15:47:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:47:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T15:47:04.963 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-22T15:47:04.963 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T15:47:04.963 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-22T15:47:04.964 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-22T15:47:04.964 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-22T15:47:04.964 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-22T15:47:04.964 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-22T15:47:04.964 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-22T15:47:04.964 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-22T15:47:05.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:47:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T15:47:05.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:47:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T15:47:05.094 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T15:47:05.094 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-22T15:47:05.094 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-04-22T15:47:05.094 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set dev brx.0 master ceph-brx 2024-04-22T15:47:05.094 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-22T15:47:05.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:47:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T15:47:05.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:47:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T15:47:05.171 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-22T15:47:05.171 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T15:47:05.171 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-22T15:47:05.195 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-22T15:47:05.196 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T15:47:05.196 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-22T15:47:05.251 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-04-22T15:47:05.316 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T15:47:05.373 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-04-22T15:47:05.373 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-04-22T15:47:05.373 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-04-22T15:47:05.373 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-04-22T15:47:05.373 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-04-22T15:47:05.373 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-04-22T15:47:05.373 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-04-22T15:47:05.374 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-04-22T15:47:05.374 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-04-22T15:47:05.374 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-04-22T15:47:05.374 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-04-22T15:47:05.374 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-04-22T15:47:05.374 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T15:47:05.374 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-04-22T15:47:05.374 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-04-22T15:47:05.374 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-04-22T15:47:05.374 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-04-22T15:47:05.374 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T15:47:05.374 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-04-22T15:47:05.374 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-04-22T15:47:05.375 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-04-22T15:47:05.375 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-22T15:47:05.375 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-22T15:47:05.375 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-04-22T15:47:05.375 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-04-22T15:47:05.375 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-04-22T15:47:05.375 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/0 2024-04-22T15:47:05.375 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T15:47:05.375 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay 2024-04-22T15:47:05.375 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/943288f6a9821d38f27eeaa41c386ef6819540bda883fdf858d5a839f0b957bf/merged 2024-04-22T15:47:05.375 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/fa4185008d7af4ad028340c9a8acf1d33049806e4fad0d968996493791af6572/merged 2024-04-22T15:47:05.375 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/d2fe763da4b8c21c483a5d68bb65b520136a34884b96fb3727ffc508eeaaa5cb/merged 2024-04-22T15:47:05.375 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/1db6d7e20c0a94981ed047612049b8a901bedd8f1ce908006834765a5f443584/merged 2024-04-22T15:47:05.375 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/18c30da815834afd4750fc872501dba03ad3408c817ef5f87fb38336e8b2183a/merged 2024-04-22T15:47:05.376 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/7275db65f79625d2dfba474bc0dca70bd8cf161e0a78ec03c97829afdf8c5d7b/merged 2024-04-22T15:47:05.376 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/bd7dd967212c3e243d7abd7dcfd61e30e6c67f667ee1eba62d757fa3c229998b/merged 2024-04-22T15:47:05.376 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/0e748dd01f021abdf184daae0467fa9c6c9a0ad31e0ecebcd6b89876c3bdcfc0/merged 2024-04-22T15:47:05.376 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/5f17fc8323542e7d26a622d7be755ba92b20f967aa8c25cea9848d2b49ddc6c9/merged 2024-04-22T15:47:05.376 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/b3eab49061f94032ed2dbb01bb561f217c3fa22dca137bfa980c57dd18f264ec/merged 2024-04-22T15:47:05.376 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-04-22T15:47:05.376 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T15:47:05.376 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T15:47:05.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:47:05.376 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-04-22T15:47:05.429 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-22T15:47:05.429 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-22T15:47:05.471 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-04-22T15:47:05.496 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T15:47:05.552 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-04-22T15:47:05.552 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-04-22T15:47:05.552 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-04-22T15:47:05.552 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-04-22T15:47:05.552 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-04-22T15:47:05.552 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-04-22T15:47:05.552 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-04-22T15:47:05.552 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-04-22T15:47:05.552 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-04-22T15:47:05.552 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-04-22T15:47:05.553 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-04-22T15:47:05.553 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-04-22T15:47:05.553 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T15:47:05.553 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-04-22T15:47:05.553 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-04-22T15:47:05.553 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-04-22T15:47:05.553 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-04-22T15:47:05.553 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T15:47:05.553 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-04-22T15:47:05.553 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-04-22T15:47:05.553 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-04-22T15:47:05.553 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-22T15:47:05.553 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-22T15:47:05.553 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-04-22T15:47:05.554 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-04-22T15:47:05.554 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-04-22T15:47:05.554 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/0 2024-04-22T15:47:05.554 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T15:47:05.554 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay 2024-04-22T15:47:05.554 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/943288f6a9821d38f27eeaa41c386ef6819540bda883fdf858d5a839f0b957bf/merged 2024-04-22T15:47:05.554 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/fa4185008d7af4ad028340c9a8acf1d33049806e4fad0d968996493791af6572/merged 2024-04-22T15:47:05.554 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/d2fe763da4b8c21c483a5d68bb65b520136a34884b96fb3727ffc508eeaaa5cb/merged 2024-04-22T15:47:05.554 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/1db6d7e20c0a94981ed047612049b8a901bedd8f1ce908006834765a5f443584/merged 2024-04-22T15:47:05.554 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/18c30da815834afd4750fc872501dba03ad3408c817ef5f87fb38336e8b2183a/merged 2024-04-22T15:47:05.554 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/7275db65f79625d2dfba474bc0dca70bd8cf161e0a78ec03c97829afdf8c5d7b/merged 2024-04-22T15:47:05.554 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/bd7dd967212c3e243d7abd7dcfd61e30e6c67f667ee1eba62d757fa3c229998b/merged 2024-04-22T15:47:05.554 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/0e748dd01f021abdf184daae0467fa9c6c9a0ad31e0ecebcd6b89876c3bdcfc0/merged 2024-04-22T15:47:05.554 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/5f17fc8323542e7d26a622d7be755ba92b20f967aa8c25cea9848d2b49ddc6c9/merged 2024-04-22T15:47:05.555 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/b3eab49061f94032ed2dbb01bb561f217c3fa22dca137bfa980c57dd18f264ec/merged 2024-04-22T15:47:05.555 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-04-22T15:47:05.555 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T15:47:05.555 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T15:47:05.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:47:05.555 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-04-22T15:47:05.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:05 smithi027 ceph-mon[34922]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 170 B/s wr, 1 op/s 2024-04-22T15:47:05.591 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi027.stderr:2024-04-22T15:47:05.589+0000 7feffd636480 -1 init, newargv = 0x55c16308c3f0 newargc=15 2024-04-22T15:47:05.591 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi027.stderr:ceph-fuse[61322]: starting ceph client 2024-04-22T15:47:05.596 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi027.stderr:ceph-fuse[61322]: starting fuse 2024-04-22T15:47:05.603 INFO:teuthology.orchestra.run.smithi027.stdout:123 2024-04-22T15:47:05.603 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [123] 2024-04-22T15:47:05.603 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-04-22T15:47:05.603 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-04-22T15:47:05.603 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-04-22T15:47:05.603 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-04-22T15:47:05.604 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-04-22T15:47:05.604 DEBUG:teuthology.orchestra.run.smithi027:> 2024-04-22T15:47:05.604 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-04-22T15:47:05.604 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-22T15:47:05.604 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-04-22T15:47:05.604 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-22T15:47:05.604 DEBUG:teuthology.orchestra.run.smithi027:> 2024-04-22T15:47:05.604 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-04-22T15:47:05.604 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-04-22T15:47:05.604 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-04-22T15:47:05.604 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-04-22T15:47:05.604 DEBUG:teuthology.orchestra.run.smithi027:> 2024-04-22T15:47:05.604 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-04-22T15:47:05.604 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-22T15:47:05.604 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-22T15:47:05.604 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-22T15:47:05.605 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-04-22T15:47:05.605 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-04-22T15:47:05.605 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-04-22T15:47:05.605 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-22T15:47:05.605 DEBUG:teuthology.orchestra.run.smithi027:> 2024-04-22T15:47:05.605 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.1")) 2024-04-22T15:47:05.605 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-04-22T15:47:05.725 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.1.61322.asok 2024-04-22T15:47:05.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-22T15:47:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T15:47:05.735 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.61322.asok 2024-04-22T15:47:05.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:47:05.736 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.61322.asok status 2024-04-22T15:47:05.874 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-22T15:47:05.874 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-04-22T15:47:05.874 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "7fe91d5d5842e04be3b4f514d6dd990c54b29c76", 2024-04-22T15:47:05.875 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-04-22T15:47:05.875 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "1", 2024-04-22T15:47:05.875 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-04-22T15:47:05.875 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-22T15:47:05.875 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "61322", 2024-04-22T15:47:05.875 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-04-22T15:47:05.875 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-22T15:47:05.875 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-04-22T15:47:05.875 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-04-22T15:47:05.875 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24341, 2024-04-22T15:47:05.875 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-04-22T15:47:05.876 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-04-22T15:47:05.876 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-04-22T15:47:05.876 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 24341 2024-04-22T15:47:05.876 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-22T15:47:05.876 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-04-22T15:47:05.876 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-04-22T15:47:05.876 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-04-22T15:47:05.876 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2832093308 2024-04-22T15:47:05.876 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-22T15:47:05.876 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-22T15:47:05.876 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-04-22T15:47:05.877 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-04-22T15:47:05.877 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-04-22T15:47:05.877 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2832093308 2024-04-22T15:47:05.877 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-22T15:47:05.877 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.24341 192.168.0.1:0/2832093308", 2024-04-22T15:47:05.877 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.1:0/2832093308", 2024-04-22T15:47:05.877 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-04-22T15:47:05.877 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 16, 2024-04-22T15:47:05.877 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 46, 2024-04-22T15:47:05.877 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-04-22T15:47:05.877 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-04-22T15:47:05.878 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-04-22T15:47:05.878 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-22T15:47:05.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:47:05.883 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-04-22T15:47:05.883 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-22T15:47:05.905 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-04-22T15:47:05.905 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-22T15:47:05.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:47:05.906 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-22T15:47:05.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:05 smithi016 ceph-mon[26620]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 170 B/s wr, 1 op/s 2024-04-22T15:47:05.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:47:05.949 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-04-22T15:47:05.949 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-22T15:47:05.965 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-04-22T15:47:05.965 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-22T15:47:05.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T15:47:05.965 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-22T15:47:06.033 INFO:teuthology.run_tasks:Running task print... 2024-04-22T15:47:06.044 INFO:teuthology.task.print:**** done client 2024-04-22T15:47:06.044 INFO:teuthology.run_tasks:Running task parallel... 2024-04-22T15:47:06.068 INFO:teuthology.task.parallel:starting parallel... 2024-04-22T15:47:06.068 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-22T15:47:06.069 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-22T15:47:06.069 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-22T15:47:06.069 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-22T15:47:06.070 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-22T15:47:06.070 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-04-22T15:47:06.073 INFO:tasks.workunit:Pulling workunits from ref 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:47:06.074 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-22T15:47:06.074 INFO:tasks.workunit:timeout=3h 2024-04-22T15:47:06.074 INFO:tasks.workunit:cleanup=True 2024-04-22T15:47:06.074 DEBUG:teuthology.orchestra.run.smithi016:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-22T15:47:06.094 INFO:teuthology.orchestra.run.smithi016.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-22T15:47:06.094 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-22T15:47:06.095 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 93h/147d Inode: 1 Links: 2 2024-04-22T15:47:06.095 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T15:47:06.095 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:fusefs_t:s0 2024-04-22T15:47:06.095 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-22T15:47:06.095 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-22 15:46:33.230342338 +0000 2024-04-22T15:47:06.095 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-22 15:47:05.944447619 +0000 2024-04-22T15:47:06.095 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2024-04-22T15:47:06.095 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-22T15:47:06.095 DEBUG:teuthology.orchestra.run.smithi016:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-22T15:47:06.166 DEBUG:teuthology.orchestra.run.smithi027:> stat -- /home/ubuntu/cephtest/mnt.1 2024-04-22T15:47:06.182 INFO:teuthology.orchestra.run.smithi027.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-04-22T15:47:06.182 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-22T15:47:06.182 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 7bh/123d Inode: 1 Links: 3 2024-04-22T15:47:06.182 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T15:47:06.183 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:fusefs_t:s0 2024-04-22T15:47:06.183 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-22T15:47:06.183 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-22 15:47:06.159060201 +0000 2024-04-22T15:47:06.183 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-22 15:47:06.159060201 +0000 2024-04-22T15:47:06.183 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2024-04-22T15:47:06.183 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-04-22T15:47:06.183 DEBUG:teuthology.orchestra.run.smithi027:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-04-22T15:47:06.252 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:47:06.253 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:47:06.270 INFO:tasks.workunit.client.0.smithi016.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-22T15:47:06.303 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:47:06.308 INFO:tasks.workunit.client.1.smithi027.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-04-22T15:47:06.709 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:06 smithi016 ceph-mon[26620]: pgmap v197: 65 pgs: 65 active+clean; 583 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-22T15:47:06.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:06 smithi027 ceph-mon[34922]: pgmap v197: 65 pgs: 65 active+clean; 583 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-22T15:47:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.855+0000 7fbc226f2700 1 -- 172.21.15.16:0/3767633849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc1c0ffbc0 msgr2=0x7fbc1c0fffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.855+0000 7fbc226f2700 1 --2- 172.21.15.16:0/3767633849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc1c0ffbc0 0x7fbc1c0fffe0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c009910 tx=0x7fbc0c009c20 comp rx=0 tx=0).stop 2024-04-22T15:47:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.855+0000 7fbc226f2700 1 -- 172.21.15.16:0/3767633849 shutdown_connections 2024-04-22T15:47:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.855+0000 7fbc226f2700 1 --2- 172.21.15.16:0/3767633849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc1c100dc0 0x7fbc1c101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.855+0000 7fbc226f2700 1 --2- 172.21.15.16:0/3767633849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc1c0ffbc0 0x7fbc1c0fffe0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.855+0000 7fbc226f2700 1 -- 172.21.15.16:0/3767633849 >> 172.21.15.16:0/3767633849 conn(0x7fbc1c0fb160 msgr2=0x7fbc1c0fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.855+0000 7fbc226f2700 1 -- 172.21.15.16:0/3767633849 shutdown_connections 2024-04-22T15:47:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.855+0000 7fbc226f2700 1 -- 172.21.15.16:0/3767633849 wait complete. 2024-04-22T15:47:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.855+0000 7fbc226f2700 1 Processor -- start 2024-04-22T15:47:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.855+0000 7fbc226f2700 1 -- start start 2024-04-22T15:47:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.856+0000 7fbc226f2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc1c0ffbc0 0x7fbc1c198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.856+0000 7fbc226f2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc1c100dc0 0x7fbc1c198f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.856+0000 7fbc226f2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c199560 con 0x7fbc1c0ffbc0 2024-04-22T15:47:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.856+0000 7fbc226f2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c1996a0 con 0x7fbc1c100dc0 2024-04-22T15:47:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.856+0000 7fbc1bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc1c0ffbc0 0x7fbc1c198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.856+0000 7fbc1bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc1c0ffbc0 0x7fbc1c198a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37664/0 (socket says 172.21.15.16:37664) 2024-04-22T15:47:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.856+0000 7fbc1bfff700 1 -- 172.21.15.16:0/2260180024 learned_addr learned my addr 172.21.15.16:0/2260180024 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:47:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.856+0000 7fbc1b7fe700 1 --2- 172.21.15.16:0/2260180024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc1c100dc0 0x7fbc1c198f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.856+0000 7fbc1bfff700 1 -- 172.21.15.16:0/2260180024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc1c100dc0 msgr2=0x7fbc1c198f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.856+0000 7fbc1bfff700 1 --2- 172.21.15.16:0/2260180024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc1c100dc0 0x7fbc1c198f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.856+0000 7fbc1bfff700 1 -- 172.21.15.16:0/2260180024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0c0095f0 con 0x7fbc1c0ffbc0 2024-04-22T15:47:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.856+0000 7fbc1bfff700 1 --2- 172.21.15.16:0/2260180024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc1c0ffbc0 0x7fbc1c198a00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c00bda0 tx=0x7fbc0c00bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.856+0000 7fbc197fa700 1 -- 172.21.15.16:0/2260180024 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc0c01d070 con 0x7fbc1c0ffbc0 2024-04-22T15:47:07.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.856+0000 7fbc226f2700 1 -- 172.21.15.16:0/2260180024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc1c19e0f0 con 0x7fbc1c0ffbc0 2024-04-22T15:47:07.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.856+0000 7fbc197fa700 1 -- 172.21.15.16:0/2260180024 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbc0c022470 con 0x7fbc1c0ffbc0 2024-04-22T15:47:07.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.856+0000 7fbc197fa700 1 -- 172.21.15.16:0/2260180024 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc0c00f670 con 0x7fbc1c0ffbc0 2024-04-22T15:47:07.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.857+0000 7fbc226f2700 1 -- 172.21.15.16:0/2260180024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc1c19e5e0 con 0x7fbc1c0ffbc0 2024-04-22T15:47:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.857+0000 7fbc226f2700 1 -- 172.21.15.16:0/2260180024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc1c068af0 con 0x7fbc1c0ffbc0 2024-04-22T15:47:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.858+0000 7fbc197fa700 1 -- 172.21.15.16:0/2260180024 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fbc0c004af0 con 0x7fbc1c0ffbc0 2024-04-22T15:47:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.858+0000 7fbc197fa700 1 --2- 172.21.15.16:0/2260180024 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fbc0406c4e0 0x7fbc0406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.858+0000 7fbc197fa700 1 -- 172.21.15.16:0/2260180024 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fbc0c08cef0 con 0x7fbc1c0ffbc0 2024-04-22T15:47:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.858+0000 7fbc1b7fe700 1 --2- 172.21.15.16:0/2260180024 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fbc0406c4e0 0x7fbc0406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.859+0000 7fbc1b7fe700 1 --2- 172.21.15.16:0/2260180024 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fbc0406c4e0 0x7fbc0406e9a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbc10005950 tx=0x7fbc1000a300 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:07.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:07.863+0000 7fbc197fa700 1 -- 172.21.15.16:0/2260180024 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fbc0c05b110 con 0x7fbc1c0ffbc0 2024-04-22T15:47:08.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:08.069+0000 7fbc226f2700 1 -- 172.21.15.16:0/2260180024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7fbc1c19e800 con 0x7fbc1c0ffbc0 2024-04-22T15:47:08.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:08.070+0000 7fbc197fa700 1 -- 172.21.15.16:0/2260180024 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7fbc0c05aca0 con 0x7fbc1c0ffbc0 2024-04-22T15:47:08.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:08.073+0000 7fbc226f2700 1 -- 172.21.15.16:0/2260180024 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fbc0406c4e0 msgr2=0x7fbc0406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:08.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:08.073+0000 7fbc226f2700 1 --2- 172.21.15.16:0/2260180024 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fbc0406c4e0 0x7fbc0406e9a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbc10005950 tx=0x7fbc1000a300 comp rx=0 tx=0).stop 2024-04-22T15:47:08.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:08.073+0000 7fbc226f2700 1 -- 172.21.15.16:0/2260180024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc1c0ffbc0 msgr2=0x7fbc1c198a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:08.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:08.073+0000 7fbc226f2700 1 --2- 172.21.15.16:0/2260180024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc1c0ffbc0 0x7fbc1c198a00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c00bda0 tx=0x7fbc0c00bdd0 comp rx=0 tx=0).stop 2024-04-22T15:47:08.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:08.073+0000 7fbc226f2700 1 -- 172.21.15.16:0/2260180024 shutdown_connections 2024-04-22T15:47:08.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:08.073+0000 7fbc226f2700 1 --2- 172.21.15.16:0/2260180024 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fbc0406c4e0 0x7fbc0406e9a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:08.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:08.073+0000 7fbc226f2700 1 --2- 172.21.15.16:0/2260180024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc1c0ffbc0 0x7fbc1c198a00 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:08.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:08.073+0000 7fbc226f2700 1 --2- 172.21.15.16:0/2260180024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc1c100dc0 0x7fbc1c198f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:08.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:08.073+0000 7fbc226f2700 1 -- 172.21.15.16:0/2260180024 >> 172.21.15.16:0/2260180024 conn(0x7fbc1c0fb160 msgr2=0x7fbc1c068eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:08.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:08.073+0000 7fbc226f2700 1 -- 172.21.15.16:0/2260180024 shutdown_connections 2024-04-22T15:47:08.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:08.073+0000 7fbc226f2700 1 -- 172.21.15.16:0/2260180024 wait complete. 2024-04-22T15:47:08.685 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-22T15:47:08.942 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:47:09.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:09 smithi016 ceph-mon[26620]: pgmap v198: 65 pgs: 65 active+clean; 583 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 511 B/s wr, 2 op/s 2024-04-22T15:47:09.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:09 smithi027 ceph-mon[34922]: pgmap v198: 65 pgs: 65 active+clean; 583 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 511 B/s wr, 2 op/s 2024-04-22T15:47:10.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.482+0000 7f618ba38700 1 -- 172.21.15.16:0/1512303178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61840fef60 msgr2=0x7f61840ff380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:10.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.482+0000 7f618ba38700 1 --2- 172.21.15.16:0/1512303178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61840fef60 0x7f61840ff380 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f617c00b0a0 tx=0x7f617c00b3b0 comp rx=0 tx=0).stop 2024-04-22T15:47:10.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.482+0000 7f618ba38700 1 -- 172.21.15.16:0/1512303178 shutdown_connections 2024-04-22T15:47:10.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.482+0000 7f618ba38700 1 --2- 172.21.15.16:0/1512303178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6184101f40 0x7f61841023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:10.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.482+0000 7f618ba38700 1 --2- 172.21.15.16:0/1512303178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61840fef60 0x7f61840ff380 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:10.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.482+0000 7f618ba38700 1 -- 172.21.15.16:0/1512303178 >> 172.21.15.16:0/1512303178 conn(0x7f61840fa8c0 msgr2=0x7f61840fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:10.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.482+0000 7f618ba38700 1 -- 172.21.15.16:0/1512303178 shutdown_connections 2024-04-22T15:47:10.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.482+0000 7f618ba38700 1 -- 172.21.15.16:0/1512303178 wait complete. 2024-04-22T15:47:10.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.483+0000 7f618ba38700 1 Processor -- start 2024-04-22T15:47:10.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.483+0000 7f618ba38700 1 -- start start 2024-04-22T15:47:10.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.483+0000 7f618ba38700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61840fef60 0x7f6184196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:10.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.483+0000 7f618ba38700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6184101f40 0x7f6184196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:10.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.483+0000 7f618ba38700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6184197070 con 0x7f6184101f40 2024-04-22T15:47:10.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.483+0000 7f618ba38700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61841971b0 con 0x7f61840fef60 2024-04-22T15:47:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.483+0000 7f6188fd3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6184101f40 0x7f6184196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.483+0000 7f6188fd3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6184101f40 0x7f6184196a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37684/0 (socket says 172.21.15.16:37684) 2024-04-22T15:47:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.483+0000 7f6188fd3700 1 -- 172.21.15.16:0/3201443236 learned_addr learned my addr 172.21.15.16:0/3201443236 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:47:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.483+0000 7f61897d4700 1 --2- 172.21.15.16:0/3201443236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61840fef60 0x7f6184196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.484+0000 7f6188fd3700 1 -- 172.21.15.16:0/3201443236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61840fef60 msgr2=0x7f6184196510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.484+0000 7f6188fd3700 1 --2- 172.21.15.16:0/3201443236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61840fef60 0x7f6184196510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.484+0000 7f6188fd3700 1 -- 172.21.15.16:0/3201443236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617c009d00 con 0x7f6184101f40 2024-04-22T15:47:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.484+0000 7f61897d4700 1 --2- 172.21.15.16:0/3201443236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61840fef60 0x7f6184196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:47:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.484+0000 7f6188fd3700 1 --2- 172.21.15.16:0/3201443236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6184101f40 0x7f6184196a50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f617800d8d0 tx=0x7f617800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.484+0000 7f61827fc700 1 -- 172.21.15.16:0/3201443236 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6178009940 con 0x7f6184101f40 2024-04-22T15:47:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.484+0000 7f618ba38700 1 -- 172.21.15.16:0/3201443236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f618419bc60 con 0x7f6184101f40 2024-04-22T15:47:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.484+0000 7f61827fc700 1 -- 172.21.15.16:0/3201443236 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6178010460 con 0x7f6184101f40 2024-04-22T15:47:10.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.484+0000 7f61827fc700 1 -- 172.21.15.16:0/3201443236 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f617800f5f0 con 0x7f6184101f40 2024-04-22T15:47:10.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.484+0000 7f618ba38700 1 -- 172.21.15.16:0/3201443236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f618419c1b0 con 0x7f6184101f40 2024-04-22T15:47:10.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.485+0000 7f618ba38700 1 -- 172.21.15.16:0/3201443236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6184109800 con 0x7f6184101f40 2024-04-22T15:47:10.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.486+0000 7f61827fc700 1 -- 172.21.15.16:0/3201443236 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f61780105d0 con 0x7f6184101f40 2024-04-22T15:47:10.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.486+0000 7f61827fc700 1 --2- 172.21.15.16:0/3201443236 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f616c06c4e0 0x7f616c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:10.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.486+0000 7f61827fc700 1 -- 172.21.15.16:0/3201443236 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f617808b910 con 0x7f6184101f40 2024-04-22T15:47:10.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.486+0000 7f61897d4700 1 --2- 172.21.15.16:0/3201443236 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f616c06c4e0 0x7f616c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:10.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.487+0000 7f61897d4700 1 --2- 172.21.15.16:0/3201443236 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f616c06c4e0 0x7f616c06e9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f617c000f80 tx=0x7f617c006210 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:10.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.491+0000 7f61827fc700 1 -- 172.21.15.16:0/3201443236 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6178059a80 con 0x7f6184101f40 2024-04-22T15:47:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.690+0000 7f618ba38700 1 -- 172.21.15.16:0/3201443236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f618419c880 con 0x7f6184101f40 2024-04-22T15:47:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.690+0000 7f61827fc700 1 -- 172.21.15.16:0/3201443236 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f6178059610 con 0x7f6184101f40 2024-04-22T15:47:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.692+0000 7f618ba38700 1 -- 172.21.15.16:0/3201443236 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f616c06c4e0 msgr2=0x7f616c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.692+0000 7f618ba38700 1 --2- 172.21.15.16:0/3201443236 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f616c06c4e0 0x7f616c06e9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f617c000f80 tx=0x7f617c006210 comp rx=0 tx=0).stop 2024-04-22T15:47:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.692+0000 7f618ba38700 1 -- 172.21.15.16:0/3201443236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6184101f40 msgr2=0x7f6184196a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.692+0000 7f618ba38700 1 --2- 172.21.15.16:0/3201443236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6184101f40 0x7f6184196a50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f617800d8d0 tx=0x7f617800dc90 comp rx=0 tx=0).stop 2024-04-22T15:47:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.692+0000 7f618ba38700 1 -- 172.21.15.16:0/3201443236 shutdown_connections 2024-04-22T15:47:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.692+0000 7f618ba38700 1 --2- 172.21.15.16:0/3201443236 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f616c06c4e0 0x7f616c06e9a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.692+0000 7f618ba38700 1 --2- 172.21.15.16:0/3201443236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61840fef60 0x7f6184196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.692+0000 7f618ba38700 1 --2- 172.21.15.16:0/3201443236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6184101f40 0x7f6184196a50 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.692+0000 7f618ba38700 1 -- 172.21.15.16:0/3201443236 >> 172.21.15.16:0/3201443236 conn(0x7f61840fa8c0 msgr2=0x7f6184105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.692+0000 7f618ba38700 1 -- 172.21.15.16:0/3201443236 shutdown_connections 2024-04-22T15:47:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:10.692+0000 7f618ba38700 1 -- 172.21.15.16:0/3201443236 wait complete. 2024-04-22T15:47:10.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:10 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:47:10.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:10 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:47:11.321 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-22T15:47:11.540 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:47:11.653 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:11 smithi016 ceph-mon[26620]: pgmap v199: 65 pgs: 65 active+clean; 583 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 511 B/s wr, 2 op/s 2024-04-22T15:47:11.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:11 smithi027 ceph-mon[34922]: pgmap v199: 65 pgs: 65 active+clean; 583 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 511 B/s wr, 2 op/s 2024-04-22T15:47:12.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.944+0000 7f32f502c700 1 -- 172.21.15.16:0/375979077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32f0100dc0 msgr2=0x7f32f0101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:12.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.944+0000 7f32f502c700 1 --2- 172.21.15.16:0/375979077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32f0100dc0 0x7f32f0101220 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f32e4009910 tx=0x7f32e4009c20 comp rx=0 tx=0).stop 2024-04-22T15:47:12.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.944+0000 7f32f502c700 1 -- 172.21.15.16:0/375979077 shutdown_connections 2024-04-22T15:47:12.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.944+0000 7f32f502c700 1 --2- 172.21.15.16:0/375979077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32f0100dc0 0x7f32f0101220 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:12.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.944+0000 7f32f502c700 1 --2- 172.21.15.16:0/375979077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f00ffbc0 0x7f32f00fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:12.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.944+0000 7f32f502c700 1 -- 172.21.15.16:0/375979077 >> 172.21.15.16:0/375979077 conn(0x7f32f00fb160 msgr2=0x7f32f00fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:12.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.944+0000 7f32f502c700 1 -- 172.21.15.16:0/375979077 shutdown_connections 2024-04-22T15:47:12.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.944+0000 7f32f502c700 1 -- 172.21.15.16:0/375979077 wait complete. 2024-04-22T15:47:12.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.944+0000 7f32f502c700 1 Processor -- start 2024-04-22T15:47:12.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.945+0000 7f32f502c700 1 -- start start 2024-04-22T15:47:12.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.945+0000 7f32f502c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32f00ffbc0 0x7f32f0103b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:12.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.945+0000 7f32f502c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f0100dc0 0x7f32f01021a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:12.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.945+0000 7f32f502c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f01026e0 con 0x7f32f00ffbc0 2024-04-22T15:47:12.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.945+0000 7f32f502c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f0102820 con 0x7f32f0100dc0 2024-04-22T15:47:12.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.945+0000 7f32eed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32f00ffbc0 0x7f32f0103b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:12.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.945+0000 7f32eed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32f00ffbc0 0x7f32f0103b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41154/0 (socket says 172.21.15.16:41154) 2024-04-22T15:47:12.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.945+0000 7f32eed9d700 1 -- 172.21.15.16:0/1701466380 learned_addr learned my addr 172.21.15.16:0/1701466380 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:47:12.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.945+0000 7f32ee59c700 1 --2- 172.21.15.16:0/1701466380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f0100dc0 0x7f32f01021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:12.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.945+0000 7f32eed9d700 1 -- 172.21.15.16:0/1701466380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f0100dc0 msgr2=0x7f32f01021a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:12.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.945+0000 7f32eed9d700 1 --2- 172.21.15.16:0/1701466380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f0100dc0 0x7f32f01021a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:12.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.945+0000 7f32eed9d700 1 -- 172.21.15.16:0/1701466380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e40095f0 con 0x7f32f00ffbc0 2024-04-22T15:47:12.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.946+0000 7f32eed9d700 1 --2- 172.21.15.16:0/1701466380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32f00ffbc0 0x7f32f0103b50 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f32e000b480 tx=0x7f32e000b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:12.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.946+0000 7f32d7fff700 1 -- 172.21.15.16:0/1701466380 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32e0011800 con 0x7f32f00ffbc0 2024-04-22T15:47:12.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.946+0000 7f32d7fff700 1 -- 172.21.15.16:0/1701466380 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f32e0011e10 con 0x7f32f00ffbc0 2024-04-22T15:47:12.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.946+0000 7f32f502c700 1 -- 172.21.15.16:0/1701466380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f0102b00 con 0x7f32f00ffbc0 2024-04-22T15:47:12.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.946+0000 7f32d7fff700 1 -- 172.21.15.16:0/1701466380 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32e0010ea0 con 0x7f32f00ffbc0 2024-04-22T15:47:12.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.946+0000 7f32f502c700 1 -- 172.21.15.16:0/1701466380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f019e8d0 con 0x7f32f00ffbc0 2024-04-22T15:47:12.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.947+0000 7f32f502c700 1 -- 172.21.15.16:0/1701466380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f0068af0 con 0x7f32f00ffbc0 2024-04-22T15:47:12.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.947+0000 7f32d7fff700 1 -- 172.21.15.16:0/1701466380 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f32e001a400 con 0x7f32f00ffbc0 2024-04-22T15:47:12.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.948+0000 7f32d7fff700 1 --2- 172.21.15.16:0/1701466380 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f32d806c4e0 0x7f32d806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:12.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.948+0000 7f32d7fff700 1 -- 172.21.15.16:0/1701466380 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f32e008b430 con 0x7f32f00ffbc0 2024-04-22T15:47:12.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.948+0000 7f32ee59c700 1 --2- 172.21.15.16:0/1701466380 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f32d806c4e0 0x7f32d806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:12.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.948+0000 7f32ee59c700 1 --2- 172.21.15.16:0/1701466380 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f32d806c4e0 0x7f32d806e9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f32e4006010 tx=0x7f32e400b540 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:12.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:12.952+0000 7f32d7fff700 1 -- 172.21.15.16:0/1701466380 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f32e0059790 con 0x7f32f00ffbc0 2024-04-22T15:47:13.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:13.150+0000 7f32f502c700 1 -- 172.21.15.16:0/1701466380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f32f019ea60 con 0x7f32f00ffbc0 2024-04-22T15:47:13.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:13.151+0000 7f32d7fff700 1 -- 172.21.15.16:0/1701466380 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7f32e0059320 con 0x7f32f00ffbc0 2024-04-22T15:47:13.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:13.153+0000 7f32f502c700 1 -- 172.21.15.16:0/1701466380 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f32d806c4e0 msgr2=0x7f32d806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:13.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:13.153+0000 7f32f502c700 1 --2- 172.21.15.16:0/1701466380 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f32d806c4e0 0x7f32d806e9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f32e4006010 tx=0x7f32e400b540 comp rx=0 tx=0).stop 2024-04-22T15:47:13.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:13.153+0000 7f32f502c700 1 -- 172.21.15.16:0/1701466380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32f00ffbc0 msgr2=0x7f32f0103b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:13.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:13.153+0000 7f32f502c700 1 --2- 172.21.15.16:0/1701466380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32f00ffbc0 0x7f32f0103b50 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f32e000b480 tx=0x7f32e000b790 comp rx=0 tx=0).stop 2024-04-22T15:47:13.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:13.153+0000 7f32f502c700 1 -- 172.21.15.16:0/1701466380 shutdown_connections 2024-04-22T15:47:13.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:13.154+0000 7f32f502c700 1 --2- 172.21.15.16:0/1701466380 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f32d806c4e0 0x7f32d806e9a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:13.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:13.154+0000 7f32f502c700 1 --2- 172.21.15.16:0/1701466380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32f00ffbc0 0x7f32f0103b50 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:13.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:13.154+0000 7f32f502c700 1 --2- 172.21.15.16:0/1701466380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f0100dc0 0x7f32f01021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:13.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:13.154+0000 7f32f502c700 1 -- 172.21.15.16:0/1701466380 >> 172.21.15.16:0/1701466380 conn(0x7f32f00fb160 msgr2=0x7f32f0068e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:13.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:13.154+0000 7f32f502c700 1 -- 172.21.15.16:0/1701466380 shutdown_connections 2024-04-22T15:47:13.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:13.154+0000 7f32f502c700 1 -- 172.21.15.16:0/1701466380 wait complete. 2024-04-22T15:47:13.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:13 smithi027 ceph-mon[34922]: pgmap v200: 65 pgs: 65 active+clean; 585 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 682 B/s wr, 3 op/s 2024-04-22T15:47:13.766 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:13 smithi016 ceph-mon[26620]: pgmap v200: 65 pgs: 65 active+clean; 585 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 682 B/s wr, 3 op/s 2024-04-22T15:47:13.793 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-04-22T15:47:14.013 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:47:15.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.418+0000 7fa81778a700 1 -- 172.21.15.16:0/3664659686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa810100dc0 msgr2=0x7fa810101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:15.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.418+0000 7fa81778a700 1 --2- 172.21.15.16:0/3664659686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa810100dc0 0x7fa810101220 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa800009910 tx=0x7fa800009c20 comp rx=0 tx=0).stop 2024-04-22T15:47:15.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.419+0000 7fa81778a700 1 -- 172.21.15.16:0/3664659686 shutdown_connections 2024-04-22T15:47:15.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.419+0000 7fa81778a700 1 --2- 172.21.15.16:0/3664659686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa810100dc0 0x7fa810101220 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:15.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.419+0000 7fa81778a700 1 --2- 172.21.15.16:0/3664659686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8100ffbc0 0x7fa8100fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:15.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.419+0000 7fa81778a700 1 -- 172.21.15.16:0/3664659686 >> 172.21.15.16:0/3664659686 conn(0x7fa8100fb160 msgr2=0x7fa8100fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:15.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.419+0000 7fa81778a700 1 -- 172.21.15.16:0/3664659686 shutdown_connections 2024-04-22T15:47:15.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.419+0000 7fa81778a700 1 -- 172.21.15.16:0/3664659686 wait complete. 2024-04-22T15:47:15.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.419+0000 7fa81778a700 1 Processor -- start 2024-04-22T15:47:15.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.419+0000 7fa81778a700 1 -- start start 2024-04-22T15:47:15.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.419+0000 7fa81778a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8100ffbc0 0x7fa810198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:15.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.419+0000 7fa81778a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa810100dc0 0x7fa810198f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:15.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.420+0000 7fa81778a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8101995b0 con 0x7fa810100dc0 2024-04-22T15:47:15.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.420+0000 7fa81778a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8101996f0 con 0x7fa8100ffbc0 2024-04-22T15:47:15.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.420+0000 7fa814d25700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa810100dc0 0x7fa810198f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:15.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.420+0000 7fa814d25700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa810100dc0 0x7fa810198f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41166/0 (socket says 172.21.15.16:41166) 2024-04-22T15:47:15.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.420+0000 7fa814d25700 1 -- 172.21.15.16:0/30329073 learned_addr learned my addr 172.21.15.16:0/30329073 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:47:15.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.420+0000 7fa815526700 1 --2- 172.21.15.16:0/30329073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8100ffbc0 0x7fa810198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:15.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.420+0000 7fa814d25700 1 -- 172.21.15.16:0/30329073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8100ffbc0 msgr2=0x7fa810198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:15.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.420+0000 7fa814d25700 1 --2- 172.21.15.16:0/30329073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8100ffbc0 0x7fa810198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:15.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.420+0000 7fa814d25700 1 -- 172.21.15.16:0/30329073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8000095f0 con 0x7fa810100dc0 2024-04-22T15:47:15.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.420+0000 7fa814d25700 1 --2- 172.21.15.16:0/30329073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa810100dc0 0x7fa810198f90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa80000bd70 tx=0x7fa80000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:15.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.420+0000 7fa8067fc700 1 -- 172.21.15.16:0/30329073 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa80001d070 con 0x7fa810100dc0 2024-04-22T15:47:15.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.421+0000 7fa81778a700 1 -- 172.21.15.16:0/30329073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa81019e140 con 0x7fa810100dc0 2024-04-22T15:47:15.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.421+0000 7fa8067fc700 1 -- 172.21.15.16:0/30329073 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa800005940 con 0x7fa810100dc0 2024-04-22T15:47:15.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.421+0000 7fa81778a700 1 -- 172.21.15.16:0/30329073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81019e630 con 0x7fa810100dc0 2024-04-22T15:47:15.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.421+0000 7fa8067fc700 1 -- 172.21.15.16:0/30329073 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa800021430 con 0x7fa810100dc0 2024-04-22T15:47:15.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.422+0000 7fa81778a700 1 -- 172.21.15.16:0/30329073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7f4005320 con 0x7fa810100dc0 2024-04-22T15:47:15.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.422+0000 7fa8067fc700 1 -- 172.21.15.16:0/30329073 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fa800003a60 con 0x7fa810100dc0 2024-04-22T15:47:15.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.423+0000 7fa8067fc700 1 --2- 172.21.15.16:0/30329073 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fa7fc06c4e0 0x7fa7fc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:15.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.423+0000 7fa8067fc700 1 -- 172.21.15.16:0/30329073 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa80008cc10 con 0x7fa810100dc0 2024-04-22T15:47:15.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.423+0000 7fa815526700 1 --2- 172.21.15.16:0/30329073 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fa7fc06c4e0 0x7fa7fc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:15.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.423+0000 7fa815526700 1 --2- 172.21.15.16:0/30329073 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fa7fc06c4e0 0x7fa7fc06e9a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa810100c20 tx=0x7fa80c009190 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:15.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.428+0000 7fa8067fc700 1 -- 172.21.15.16:0/30329073 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa80005ae20 con 0x7fa810100dc0 2024-04-22T15:47:15.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:15.647+0000 7fa81778a700 1 -- 172.21.15.16:0/30329073 --> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7f4000ca0 con 0x7fa7fc06c4e0 2024-04-22T15:47:15.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:15 smithi027 ceph-mon[34922]: pgmap v201: 65 pgs: 65 active+clean; 585 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 597 B/s wr, 2 op/s 2024-04-22T15:47:15.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:15 smithi016 ceph-mon[26620]: pgmap v201: 65 pgs: 65 active+clean; 585 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 597 B/s wr, 2 op/s 2024-04-22T15:47:16.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:16 smithi016 ceph-mon[26620]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:16.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:16 smithi016 ceph-mon[26620]: pgmap v202: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 938 B/s wr, 3 op/s 2024-04-22T15:47:17.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:16 smithi027 ceph-mon[34922]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:17.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:16 smithi027 ceph-mon[34922]: pgmap v202: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 938 B/s wr, 3 op/s 2024-04-22T15:47:19.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:19 smithi027 ceph-mon[34922]: pgmap v203: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 597 B/s wr, 2 op/s 2024-04-22T15:47:19.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:19 smithi016 ceph-mon[26620]: pgmap v203: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 597 B/s wr, 2 op/s 2024-04-22T15:47:21.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:21 smithi027 ceph-mon[34922]: pgmap v204: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-22T15:47:21.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:21 smithi016 ceph-mon[26620]: pgmap v204: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-22T15:47:23.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:23 smithi027 ceph-mon[34922]: pgmap v205: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 682 B/s wr, 3 op/s 2024-04-22T15:47:23.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:23 smithi016 ceph-mon[26620]: pgmap v205: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 682 B/s wr, 3 op/s 2024-04-22T15:47:25.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:25 smithi016 ceph-mon[26620]: pgmap v206: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-22T15:47:25.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:25 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:47:25.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:25 smithi027 ceph-mon[34922]: pgmap v206: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-22T15:47:25.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:25 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:47:26.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:26 smithi016 ceph-mon[26620]: pgmap v207: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 426 B/s wr, 2 op/s 2024-04-22T15:47:27.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:26 smithi027 ceph-mon[34922]: pgmap v207: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 426 B/s wr, 2 op/s 2024-04-22T15:47:29.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:29 smithi027 ceph-mon[34922]: pgmap v208: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-04-22T15:47:29.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:29 smithi016 ceph-mon[26620]: pgmap v208: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-04-22T15:47:31.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:31 smithi027 ceph-mon[34922]: pgmap v209: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-04-22T15:47:31.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:31 smithi016 ceph-mon[26620]: pgmap v209: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-04-22T15:47:33.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:33 smithi027 ceph-mon[34922]: pgmap v210: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-22T15:47:33.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:33 smithi016 ceph-mon[26620]: pgmap v210: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-22T15:47:35.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:35 smithi016 ceph-mon[26620]: pgmap v211: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:47:35.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:35 smithi027 ceph-mon[34922]: pgmap v211: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:47:36.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:36 smithi016 ceph-mon[26620]: pgmap v212: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:47:37.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:36 smithi027 ceph-mon[34922]: pgmap v212: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:47:39.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:39 smithi027 ceph-mon[34922]: pgmap v213: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:47:39.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:39 smithi016 ceph-mon[26620]: pgmap v213: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:47:40.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:40 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:47:40.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:47:41.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:41 smithi027 ceph-mon[34922]: pgmap v214: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:47:41.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:41 smithi016 ceph-mon[26620]: pgmap v214: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:47:43.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:43 smithi027 ceph-mon[34922]: pgmap v215: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:47:43.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:43 smithi016 ceph-mon[26620]: pgmap v215: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:47:45.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:45 smithi027 ceph-mon[34922]: pgmap v216: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:47:45.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:45 smithi016 ceph-mon[26620]: pgmap v216: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:47:46.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:46 smithi016 ceph-mon[26620]: pgmap v217: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:47:47.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:46 smithi027 ceph-mon[34922]: pgmap v217: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:47:48.615 INFO:teuthology.orchestra.run.smithi016.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:47:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:48.612+0000 7fa8067fc700 1 -- 172.21.15.16:0/30329073 <== mgr.14217 v2:172.21.15.16:6800/1057234399 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7fa7f4000ca0 con 0x7fa7fc06c4e0 2024-04-22T15:47:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:48.614+0000 7fa81778a700 1 -- 172.21.15.16:0/30329073 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fa7fc06c4e0 msgr2=0x7fa7fc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:48.614+0000 7fa81778a700 1 --2- 172.21.15.16:0/30329073 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fa7fc06c4e0 0x7fa7fc06e9a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa810100c20 tx=0x7fa80c009190 comp rx=0 tx=0).stop 2024-04-22T15:47:48.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:48.614+0000 7fa81778a700 1 -- 172.21.15.16:0/30329073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa810100dc0 msgr2=0x7fa810198f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:48.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:48.614+0000 7fa81778a700 1 --2- 172.21.15.16:0/30329073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa810100dc0 0x7fa810198f90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa80000bd70 tx=0x7fa80000bda0 comp rx=0 tx=0).stop 2024-04-22T15:47:48.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:48.614+0000 7fa81778a700 1 -- 172.21.15.16:0/30329073 shutdown_connections 2024-04-22T15:47:48.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:48.614+0000 7fa81778a700 1 --2- 172.21.15.16:0/30329073 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fa7fc06c4e0 0x7fa7fc06e9a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:48.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:48.614+0000 7fa81778a700 1 --2- 172.21.15.16:0/30329073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8100ffbc0 0x7fa810198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:48.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:48.614+0000 7fa81778a700 1 --2- 172.21.15.16:0/30329073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa810100dc0 0x7fa810198f90 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:48.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:48.614+0000 7fa81778a700 1 -- 172.21.15.16:0/30329073 >> 172.21.15.16:0/30329073 conn(0x7fa8100fb160 msgr2=0x7fa810068e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:48.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:48.615+0000 7fa81778a700 1 -- 172.21.15.16:0/30329073 shutdown_connections 2024-04-22T15:47:48.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:48.615+0000 7fa81778a700 1 -- 172.21.15.16:0/30329073 wait complete. 2024-04-22T15:47:49.261 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-22T15:47:49.498 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:47:49.764 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:49 smithi016 ceph-mon[26620]: pgmap v218: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:47:49.764 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:49 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:47:49.764 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:49 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:47:49.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:49 smithi027 ceph-mon[34922]: pgmap v218: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:47:49.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:49 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:47:49.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:49 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:47:50.570 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:50 smithi016 ceph-mon[26620]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:47:50.571 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:50 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:47:50.571 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:50 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:47:50.571 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:50 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:47:50.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:50 smithi027 ceph-mon[34922]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:47:50.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:50 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:47:50.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:50 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:47:50.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:50 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:47:51.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:51 smithi027 ceph-mon[34922]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:47:51.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:51 smithi027 ceph-mon[34922]: pgmap v219: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:47:51.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:51 smithi016 ceph-mon[26620]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:47:51.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:51 smithi016 ceph-mon[26620]: pgmap v219: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:47:52.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.431+0000 7f73fae42700 1 -- 172.21.15.16:0/4265662988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73f40ffc10 msgr2=0x7f73f4100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:52.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.431+0000 7f73fae42700 1 --2- 172.21.15.16:0/4265662988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73f40ffc10 0x7f73f4100030 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f73e8009910 tx=0x7f73e8009c20 comp rx=0 tx=0).stop 2024-04-22T15:47:52.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.431+0000 7f73fae42700 1 -- 172.21.15.16:0/4265662988 shutdown_connections 2024-04-22T15:47:52.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.431+0000 7f73fae42700 1 --2- 172.21.15.16:0/4265662988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73f4100e10 0x7f73f4101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:52.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.431+0000 7f73fae42700 1 --2- 172.21.15.16:0/4265662988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73f40ffc10 0x7f73f4100030 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:52.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.431+0000 7f73fae42700 1 -- 172.21.15.16:0/4265662988 >> 172.21.15.16:0/4265662988 conn(0x7f73f40fb190 msgr2=0x7f73f40fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:52.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.431+0000 7f73fae42700 1 -- 172.21.15.16:0/4265662988 shutdown_connections 2024-04-22T15:47:52.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.431+0000 7f73fae42700 1 -- 172.21.15.16:0/4265662988 wait complete. 2024-04-22T15:47:52.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.432+0000 7f73fae42700 1 Processor -- start 2024-04-22T15:47:52.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.432+0000 7f73fae42700 1 -- start start 2024-04-22T15:47:52.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.432+0000 7f73fae42700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73f40ffc10 0x7f73f41964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:52.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.432+0000 7f73fae42700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73f4100e10 0x7f73f4196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:52.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.432+0000 7f73fae42700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f4197030 con 0x7f73f40ffc10 2024-04-22T15:47:52.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.432+0000 7f73fae42700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f4197170 con 0x7f73f4100e10 2024-04-22T15:47:52.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.432+0000 7f73f3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73f4100e10 0x7f73f4196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:52.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.432+0000 7f73f8bde700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73f40ffc10 0x7f73f41964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:52.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.432+0000 7f73f3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73f4100e10 0x7f73f4196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:48264/0 (socket says 172.21.15.16:48264) 2024-04-22T15:47:52.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.432+0000 7f73f3fff700 1 -- 172.21.15.16:0/287614422 learned_addr learned my addr 172.21.15.16:0/287614422 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:47:52.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.432+0000 7f73f3fff700 1 -- 172.21.15.16:0/287614422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73f40ffc10 msgr2=0x7f73f41964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:52.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.432+0000 7f73f3fff700 1 --2- 172.21.15.16:0/287614422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73f40ffc10 0x7f73f41964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:52.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.432+0000 7f73f3fff700 1 -- 172.21.15.16:0/287614422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e80095f0 con 0x7f73f4100e10 2024-04-22T15:47:52.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.433+0000 7f73f3fff700 1 --2- 172.21.15.16:0/287614422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73f4100e10 0x7f73f4196a10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f73e000e920 tx=0x7f73e000ec30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:52.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.433+0000 7f73f1ffb700 1 -- 172.21.15.16:0/287614422 <== mon.1 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73e000c9d0 con 0x7f73f4100e10 2024-04-22T15:47:52.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.433+0000 7f73f1ffb700 1 -- 172.21.15.16:0/287614422 <== mon.1 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f73e0010430 con 0x7f73f4100e10 2024-04-22T15:47:52.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.433+0000 7f73fae42700 1 -- 172.21.15.16:0/287614422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f419bc20 con 0x7f73f4100e10 2024-04-22T15:47:52.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.433+0000 7f73f1ffb700 1 -- 172.21.15.16:0/287614422 <== mon.1 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73e0018650 con 0x7f73f4100e10 2024-04-22T15:47:52.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.433+0000 7f73fae42700 1 -- 172.21.15.16:0/287614422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f419c170 con 0x7f73f4100e10 2024-04-22T15:47:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.434+0000 7f73fae42700 1 -- 172.21.15.16:0/287614422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73f4108700 con 0x7f73f4100e10 2024-04-22T15:47:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.435+0000 7f73f1ffb700 1 -- 172.21.15.16:0/287614422 <== mon.1 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f73e0018910 con 0x7f73f4100e10 2024-04-22T15:47:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.435+0000 7f73f1ffb700 1 --2- 172.21.15.16:0/287614422 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f73e406c4e0 0x7f73e406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.435+0000 7f73f1ffb700 1 -- 172.21.15.16:0/287614422 <== mon.1 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f73e0014070 con 0x7f73f4100e10 2024-04-22T15:47:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.435+0000 7f73f8bde700 1 --2- 172.21.15.16:0/287614422 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f73e406c4e0 0x7f73e406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.436+0000 7f73f8bde700 1 --2- 172.21.15.16:0/287614422 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f73e406c4e0 0x7f73e406e9a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f73e800b6f0 tx=0x7f73e8005940 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:52.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.440+0000 7f73f1ffb700 1 -- 172.21.15.16:0/287614422 <== mon.1 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f73e00520e0 con 0x7f73f4100e10 2024-04-22T15:47:52.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.677+0000 7f73fae42700 1 -- 172.21.15.16:0/287614422 --> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f73f4068c90 con 0x7f73e406c4e0 2024-04-22T15:47:52.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.678+0000 7f73f1ffb700 1 -- 172.21.15.16:0/287614422 <== mgr.14217 v2:172.21.15.16:6800/1057234399 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f73f4068c90 con 0x7f73e406c4e0 2024-04-22T15:47:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.680+0000 7f73fae42700 1 -- 172.21.15.16:0/287614422 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f73e406c4e0 msgr2=0x7f73e406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.680+0000 7f73fae42700 1 --2- 172.21.15.16:0/287614422 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f73e406c4e0 0x7f73e406e9a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f73e800b6f0 tx=0x7f73e8005940 comp rx=0 tx=0).stop 2024-04-22T15:47:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.680+0000 7f73fae42700 1 -- 172.21.15.16:0/287614422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73f4100e10 msgr2=0x7f73f4196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.680+0000 7f73fae42700 1 --2- 172.21.15.16:0/287614422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73f4100e10 0x7f73f4196a10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f73e000e920 tx=0x7f73e000ec30 comp rx=0 tx=0).stop 2024-04-22T15:47:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.680+0000 7f73fae42700 1 -- 172.21.15.16:0/287614422 shutdown_connections 2024-04-22T15:47:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.680+0000 7f73fae42700 1 --2- 172.21.15.16:0/287614422 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f73e406c4e0 0x7f73e406e9a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.680+0000 7f73fae42700 1 --2- 172.21.15.16:0/287614422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73f40ffc10 0x7f73f41964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.680+0000 7f73fae42700 1 --2- 172.21.15.16:0/287614422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73f4100e10 0x7f73f4196a10 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.680+0000 7f73fae42700 1 -- 172.21.15.16:0/287614422 >> 172.21.15.16:0/287614422 conn(0x7f73f40fb190 msgr2=0x7f73f4068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.680+0000 7f73fae42700 1 -- 172.21.15.16:0/287614422 shutdown_connections 2024-04-22T15:47:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.680+0000 7f73fae42700 1 -- 172.21.15.16:0/287614422 wait complete. 2024-04-22T15:47:52.695 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:47:52.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.792+0000 7f887f7a0700 1 -- 172.21.15.16:0/213700594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88780fef60 msgr2=0x7f88780ff380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:52.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.792+0000 7f887f7a0700 1 --2- 172.21.15.16:0/213700594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88780fef60 0x7f88780ff380 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f886800b0a0 tx=0x7f886800b3b0 comp rx=0 tx=0).stop 2024-04-22T15:47:52.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.792+0000 7f887f7a0700 1 -- 172.21.15.16:0/213700594 shutdown_connections 2024-04-22T15:47:52.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.792+0000 7f887f7a0700 1 --2- 172.21.15.16:0/213700594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8878101f40 0x7f88781023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:52.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.792+0000 7f887f7a0700 1 --2- 172.21.15.16:0/213700594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88780fef60 0x7f88780ff380 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:52.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.792+0000 7f887f7a0700 1 -- 172.21.15.16:0/213700594 >> 172.21.15.16:0/213700594 conn(0x7f88780fa8c0 msgr2=0x7f88780fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:52.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.792+0000 7f887f7a0700 1 -- 172.21.15.16:0/213700594 shutdown_connections 2024-04-22T15:47:52.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.792+0000 7f887f7a0700 1 -- 172.21.15.16:0/213700594 wait complete. 2024-04-22T15:47:52.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.793+0000 7f887f7a0700 1 Processor -- start 2024-04-22T15:47:52.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.793+0000 7f887f7a0700 1 -- start start 2024-04-22T15:47:52.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.793+0000 7f887f7a0700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88780fef60 0x7f88781018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:52.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.793+0000 7f887f7a0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8878101f40 0x7f88780ffef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:52.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.793+0000 7f887f7a0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8878100430 con 0x7f8878101f40 2024-04-22T15:47:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.793+0000 7f887f7a0700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8878100570 con 0x7f88780fef60 2024-04-22T15:47:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.793+0000 7f887cd3b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8878101f40 0x7f88780ffef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.793+0000 7f887cd3b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8878101f40 0x7f88780ffef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35518/0 (socket says 172.21.15.16:35518) 2024-04-22T15:47:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.793+0000 7f887cd3b700 1 -- 172.21.15.16:0/3064533612 learned_addr learned my addr 172.21.15.16:0/3064533612 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:47:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.793+0000 7f887d53c700 1 --2- 172.21.15.16:0/3064533612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88780fef60 0x7f88781018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.794+0000 7f887cd3b700 1 -- 172.21.15.16:0/3064533612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88780fef60 msgr2=0x7f88781018a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.794+0000 7f887cd3b700 1 --2- 172.21.15.16:0/3064533612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88780fef60 0x7f88781018a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.794+0000 7f887cd3b700 1 -- 172.21.15.16:0/3064533612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8868009d00 con 0x7f8878101f40 2024-04-22T15:47:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.794+0000 7f887cd3b700 1 --2- 172.21.15.16:0/3064533612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8878101f40 0x7f88780ffef0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f886c00d900 tx=0x7f886c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.794+0000 7f88767fc700 1 -- 172.21.15.16:0/3064533612 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f886c0041f0 con 0x7f8878101f40 2024-04-22T15:47:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.794+0000 7f887f7a0700 1 -- 172.21.15.16:0/3064533612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8878100850 con 0x7f8878101f40 2024-04-22T15:47:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.794+0000 7f88767fc700 1 -- 172.21.15.16:0/3064533612 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f886c004ce0 con 0x7f8878101f40 2024-04-22T15:47:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.794+0000 7f88767fc700 1 -- 172.21.15.16:0/3064533612 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f886c00b750 con 0x7f8878101f40 2024-04-22T15:47:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.794+0000 7f887f7a0700 1 -- 172.21.15.16:0/3064533612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f887819e530 con 0x7f8878101f40 2024-04-22T15:47:52.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.795+0000 7f887f7a0700 1 -- 172.21.15.16:0/3064533612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8878109800 con 0x7f8878101f40 2024-04-22T15:47:52.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.796+0000 7f88767fc700 1 -- 172.21.15.16:0/3064533612 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f886c004e50 con 0x7f8878101f40 2024-04-22T15:47:52.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.797+0000 7f88767fc700 1 --2- 172.21.15.16:0/3064533612 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f886006c4e0 0x7f886006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:52.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.797+0000 7f88767fc700 1 -- 172.21.15.16:0/3064533612 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f886c08b740 con 0x7f8878101f40 2024-04-22T15:47:52.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.797+0000 7f887d53c700 1 --2- 172.21.15.16:0/3064533612 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f886006c4e0 0x7f886006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:52.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.797+0000 7f887d53c700 1 --2- 172.21.15.16:0/3064533612 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f886006c4e0 0x7f886006e9a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f886800b830 tx=0x7f88680060c0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:52.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:52.802+0000 7f88767fc700 1 -- 172.21.15.16:0/3064533612 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f886c059950 con 0x7f8878101f40 2024-04-22T15:47:53.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.032+0000 7f887f7a0700 1 -- 172.21.15.16:0/3064533612 --> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8878106890 con 0x7f886006c4e0 2024-04-22T15:47:53.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.033+0000 7f88767fc700 1 -- 172.21.15.16:0/3064533612 <== mgr.14217 v2:172.21.15.16:6800/1057234399 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f8878106890 con 0x7f886006c4e0 2024-04-22T15:47:53.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.035+0000 7f887f7a0700 1 -- 172.21.15.16:0/3064533612 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f886006c4e0 msgr2=0x7f886006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:53.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.035+0000 7f887f7a0700 1 --2- 172.21.15.16:0/3064533612 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f886006c4e0 0x7f886006e9a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f886800b830 tx=0x7f88680060c0 comp rx=0 tx=0).stop 2024-04-22T15:47:53.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.035+0000 7f887f7a0700 1 -- 172.21.15.16:0/3064533612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8878101f40 msgr2=0x7f88780ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:53.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.035+0000 7f887f7a0700 1 --2- 172.21.15.16:0/3064533612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8878101f40 0x7f88780ffef0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f886c00d900 tx=0x7f886c00dc10 comp rx=0 tx=0).stop 2024-04-22T15:47:53.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.035+0000 7f887f7a0700 1 -- 172.21.15.16:0/3064533612 shutdown_connections 2024-04-22T15:47:53.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.035+0000 7f887f7a0700 1 --2- 172.21.15.16:0/3064533612 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f886006c4e0 0x7f886006e9a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:53.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.035+0000 7f887f7a0700 1 --2- 172.21.15.16:0/3064533612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88780fef60 0x7f88781018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:53.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.035+0000 7f887f7a0700 1 --2- 172.21.15.16:0/3064533612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8878101f40 0x7f88780ffef0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:53.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.035+0000 7f887f7a0700 1 -- 172.21.15.16:0/3064533612 >> 172.21.15.16:0/3064533612 conn(0x7f88780fa8c0 msgr2=0x7f8878105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:53.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.035+0000 7f887f7a0700 1 -- 172.21.15.16:0/3064533612 shutdown_connections 2024-04-22T15:47:53.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.035+0000 7f887f7a0700 1 -- 172.21.15.16:0/3064533612 wait complete. 2024-04-22T15:47:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.160+0000 7f52f70b5700 1 -- 172.21.15.16:0/812635176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52f00ffc10 msgr2=0x7f52f0100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.160+0000 7f52f70b5700 1 --2- 172.21.15.16:0/812635176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52f00ffc10 0x7f52f0100030 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f52e4009910 tx=0x7f52e4009c20 comp rx=0 tx=0).stop 2024-04-22T15:47:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.161+0000 7f52f70b5700 1 -- 172.21.15.16:0/812635176 shutdown_connections 2024-04-22T15:47:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.161+0000 7f52f70b5700 1 --2- 172.21.15.16:0/812635176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52f0100e10 0x7f52f0101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.161+0000 7f52f70b5700 1 --2- 172.21.15.16:0/812635176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52f00ffc10 0x7f52f0100030 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.161+0000 7f52f70b5700 1 -- 172.21.15.16:0/812635176 >> 172.21.15.16:0/812635176 conn(0x7f52f00fb190 msgr2=0x7f52f00fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.161+0000 7f52f70b5700 1 -- 172.21.15.16:0/812635176 shutdown_connections 2024-04-22T15:47:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.161+0000 7f52f70b5700 1 -- 172.21.15.16:0/812635176 wait complete. 2024-04-22T15:47:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.161+0000 7f52f70b5700 1 Processor -- start 2024-04-22T15:47:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.161+0000 7f52f70b5700 1 -- start start 2024-04-22T15:47:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.161+0000 7f52f70b5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52f00ffc10 0x7f52f0103ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.161+0000 7f52f70b5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52f0100e10 0x7f52f01021f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.161+0000 7f52f70b5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f0102730 con 0x7f52f0100e10 2024-04-22T15:47:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.161+0000 7f52f70b5700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f0102870 con 0x7f52f00ffc10 2024-04-22T15:47:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.162+0000 7f52f4e51700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52f00ffc10 0x7f52f0103ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.162+0000 7f52f4e51700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52f00ffc10 0x7f52f0103ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:48296/0 (socket says 172.21.15.16:48296) 2024-04-22T15:47:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.162+0000 7f52f4e51700 1 -- 172.21.15.16:0/3649133674 learned_addr learned my addr 172.21.15.16:0/3649133674 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:47:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.162+0000 7f52effff700 1 --2- 172.21.15.16:0/3649133674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52f0100e10 0x7f52f01021f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.162+0000 7f52f4e51700 1 -- 172.21.15.16:0/3649133674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52f0100e10 msgr2=0x7f52f01021f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.162+0000 7f52f4e51700 1 --2- 172.21.15.16:0/3649133674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52f0100e10 0x7f52f01021f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.162+0000 7f52f4e51700 1 -- 172.21.15.16:0/3649133674 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e40095f0 con 0x7f52f00ffc10 2024-04-22T15:47:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.162+0000 7f52effff700 1 --2- 172.21.15.16:0/3649133674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52f0100e10 0x7f52f01021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:47:53.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.162+0000 7f52f4e51700 1 --2- 172.21.15.16:0/3649133674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52f00ffc10 0x7f52f0103ba0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f52e4000c00 tx=0x7f52e400f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:53.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.162+0000 7f52edffb700 1 -- 172.21.15.16:0/3649133674 <== mon.1 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52e401d070 con 0x7f52f00ffc10 2024-04-22T15:47:53.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.162+0000 7f52edffb700 1 -- 172.21.15.16:0/3649133674 <== mon.1 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f52e4005dc0 con 0x7f52f00ffc10 2024-04-22T15:47:53.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.162+0000 7f52edffb700 1 -- 172.21.15.16:0/3649133674 <== mon.1 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52e4022470 con 0x7f52f00ffc10 2024-04-22T15:47:53.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.162+0000 7f52f70b5700 1 -- 172.21.15.16:0/3649133674 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f0102af0 con 0x7f52f00ffc10 2024-04-22T15:47:53.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.163+0000 7f52f70b5700 1 -- 172.21.15.16:0/3649133674 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f0102f90 con 0x7f52f00ffc10 2024-04-22T15:47:53.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.164+0000 7f52f70b5700 1 -- 172.21.15.16:0/3649133674 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f0108700 con 0x7f52f00ffc10 2024-04-22T15:47:53.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.165+0000 7f52edffb700 1 -- 172.21.15.16:0/3649133674 <== mon.1 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f52e4003810 con 0x7f52f00ffc10 2024-04-22T15:47:53.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.165+0000 7f52edffb700 1 --2- 172.21.15.16:0/3649133674 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f52e006c4e0 0x7f52e006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:53.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.165+0000 7f52edffb700 1 -- 172.21.15.16:0/3649133674 <== mon.1 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f52e408cde0 con 0x7f52f00ffc10 2024-04-22T15:47:53.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.165+0000 7f52effff700 1 --2- 172.21.15.16:0/3649133674 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f52e006c4e0 0x7f52e006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:53.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.166+0000 7f52effff700 1 --2- 172.21.15.16:0/3649133674 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f52e006c4e0 0x7f52e006e9a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f52dc011490 tx=0x7f52dc009260 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:53.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.170+0000 7f52edffb700 1 -- 172.21.15.16:0/3649133674 <== mon.1 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f52e405af90 con 0x7f52f00ffc10 2024-04-22T15:47:53.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:53 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:47:53.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:53 smithi027 ceph-mon[34922]: Upgrade: Target is version 19.0.0-3223-g5883d494 (unknown) 2024-04-22T15:47:53.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:53 smithi027 ceph-mon[34922]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, digests ['quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb'] 2024-04-22T15:47:53.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:53 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:47:53.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:53 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:47:53.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:53 smithi027 ceph-mon[34922]: Upgrade: Need to upgrade myself (mgr.smithi016.rnsnnr) 2024-04-22T15:47:53.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:53 smithi027 ceph-mon[34922]: pgmap v220: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:47:53.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:53 smithi027 ceph-mon[34922]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb on smithi027 2024-04-22T15:47:53.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.401+0000 7f52f70b5700 1 -- 172.21.15.16:0/3649133674 --> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f52f019e950 con 0x7f52e006c4e0 2024-04-22T15:47:53.405 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:53 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:47:53.405 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:53 smithi016 ceph-mon[26620]: Upgrade: Target is version 19.0.0-3223-g5883d494 (unknown) 2024-04-22T15:47:53.405 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:53 smithi016 ceph-mon[26620]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, digests ['quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb'] 2024-04-22T15:47:53.405 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:53 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:47:53.405 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:53 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:47:53.405 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:53 smithi016 ceph-mon[26620]: Upgrade: Need to upgrade myself (mgr.smithi016.rnsnnr) 2024-04-22T15:47:53.405 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:53 smithi016 ceph-mon[26620]: pgmap v220: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:47:53.405 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:53 smithi016 ceph-mon[26620]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb on smithi027 2024-04-22T15:47:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.413+0000 7f52edffb700 1 -- 172.21.15.16:0/3649133674 <== mgr.14217 v2:172.21.15.16:6800/1057234399 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3480 (secure 0 0 0) 0x7f52f019e950 con 0x7f52e006c4e0 2024-04-22T15:47:53.419 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:47:53.419 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.smithi016 smithi016 *:9093,9094 running (5m) 55s ago 6m 24.9M - 0.25.0 c8568f914cd2 fab40987f1b0 2024-04-22T15:47:53.419 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi016 smithi016 running (7m) 55s ago 7m 7461k - 18.2.1 5be31c24972a f8b627d3f41e 2024-04-22T15:47:53.419 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi027 smithi027 running (6m) 59s ago 6m 7319k - 18.2.1 5be31c24972a f00fd40b4220 2024-04-22T15:47:53.419 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi016 smithi016 running (7m) 55s ago 7m 6656k - 18.2.1 5be31c24972a 6b93db1fa0d9 2024-04-22T15:47:53.419 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi027 smithi027 running (5m) 59s ago 5m 6665k - 18.2.1 5be31c24972a d3bd439f30a0 2024-04-22T15:47:53.419 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.smithi016 smithi016 *:3000 running (5m) 55s ago 6m 86.2M - 9.4.7 954c08fa6188 a85bdbf5073d 2024-04-22T15:47:53.419 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.fkkejm smithi016 running (80s) 55s ago 80s 11.4M - 18.2.1 5be31c24972a 4153ac94cea0 2024-04-22T15:47:53.419 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.pgvphu smithi016 running (74s) 55s ago 73s 14.4M - 18.2.1 5be31c24972a 93dc072068a2 2024-04-22T15:47:53.420 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.dgqyes smithi027 running (71s) 59s ago 71s 14.6M - 18.2.1 5be31c24972a 41a9220430fd 2024-04-22T15:47:53.420 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.iobjds smithi027 running (78s) 59s ago 77s 15.2M - 18.2.1 5be31c24972a c1c8d6770f02 2024-04-22T15:47:53.420 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi016.rnsnnr smithi016 *:9283,8765,8443 running (8m) 55s ago 8m 488M - 18.2.1 5be31c24972a 1cf5a7efea72 2024-04-22T15:47:53.420 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi027.ukdgzs smithi027 *:8443,9283,8765 running (5m) 59s ago 5m 442M - 18.2.1 5be31c24972a c59a9cac2d99 2024-04-22T15:47:53.420 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi016 smithi016 running (8m) 55s ago 8m 47.9M 2048M 18.2.1 5be31c24972a 72af40873cb6 2024-04-22T15:47:53.420 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi027 smithi027 running (5m) 59s ago 5m 36.7M 2048M 18.2.1 5be31c24972a 9f21ce7fa7ef 2024-04-22T15:47:53.420 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi016 smithi016 *:9100 running (6m) 55s ago 6m 15.7M - 1.5.0 0da6a335fe13 1937a0bc1959 2024-04-22T15:47:53.420 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi027 smithi027 *:9100 running (5m) 59s ago 5m 15.7M - 1.5.0 0da6a335fe13 00440c991b3f 2024-04-22T15:47:53.420 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (5m) 55s ago 5m 48.4M 3948M 18.2.1 5be31c24972a 3c6ebc4ea041 2024-04-22T15:47:53.420 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (4m) 55s ago 4m 47.8M 3948M 18.2.1 5be31c24972a 6aa2e6796784 2024-04-22T15:47:53.420 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (4m) 55s ago 4m 45.9M 3948M 18.2.1 5be31c24972a 3d9d310aa9eb 2024-04-22T15:47:53.420 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi027 running (3m) 59s ago 3m 49.1M 4970M 18.2.1 5be31c24972a cdf9b7aaf865 2024-04-22T15:47:53.420 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (3m) 59s ago 3m 43.5M 4970M 18.2.1 5be31c24972a 629397d49c76 2024-04-22T15:47:53.420 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi027 running (2m) 59s ago 2m 46.6M 4970M 18.2.1 5be31c24972a 23adcf675b04 2024-04-22T15:47:53.421 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.smithi016 smithi016 *:9095 running (5m) 55s ago 6m 59.2M - 2.43.0 a07b618ecd1d 2ba3b93ebe19 2024-04-22T15:47:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.416+0000 7f52f70b5700 1 -- 172.21.15.16:0/3649133674 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f52e006c4e0 msgr2=0x7f52e006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.416+0000 7f52f70b5700 1 --2- 172.21.15.16:0/3649133674 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f52e006c4e0 0x7f52e006e9a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f52dc011490 tx=0x7f52dc009260 comp rx=0 tx=0).stop 2024-04-22T15:47:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.416+0000 7f52f70b5700 1 -- 172.21.15.16:0/3649133674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52f00ffc10 msgr2=0x7f52f0103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.416+0000 7f52f70b5700 1 --2- 172.21.15.16:0/3649133674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52f00ffc10 0x7f52f0103ba0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f52e4000c00 tx=0x7f52e400f690 comp rx=0 tx=0).stop 2024-04-22T15:47:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.416+0000 7f52f70b5700 1 -- 172.21.15.16:0/3649133674 shutdown_connections 2024-04-22T15:47:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.416+0000 7f52f70b5700 1 --2- 172.21.15.16:0/3649133674 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f52e006c4e0 0x7f52e006e9a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:53.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.416+0000 7f52f70b5700 1 --2- 172.21.15.16:0/3649133674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52f00ffc10 0x7f52f0103ba0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:53.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.416+0000 7f52f70b5700 1 --2- 172.21.15.16:0/3649133674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52f0100e10 0x7f52f01021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:53.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.416+0000 7f52f70b5700 1 -- 172.21.15.16:0/3649133674 >> 172.21.15.16:0/3649133674 conn(0x7f52f00fb190 msgr2=0x7f52f0068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:53.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.416+0000 7f52f70b5700 1 -- 172.21.15.16:0/3649133674 shutdown_connections 2024-04-22T15:47:53.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.416+0000 7f52f70b5700 1 -- 172.21.15.16:0/3649133674 wait complete. 2024-04-22T15:47:53.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.524+0000 7f3719a1b700 1 -- 172.21.15.16:0/133158213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3714100e10 msgr2=0x7f3714101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:53.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.524+0000 7f3719a1b700 1 --2- 172.21.15.16:0/133158213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3714100e10 0x7f3714101270 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f3708009910 tx=0x7f3708009c20 comp rx=0 tx=0).stop 2024-04-22T15:47:53.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.524+0000 7f3719a1b700 1 -- 172.21.15.16:0/133158213 shutdown_connections 2024-04-22T15:47:53.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.524+0000 7f3719a1b700 1 --2- 172.21.15.16:0/133158213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3714100e10 0x7f3714101270 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:53.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.524+0000 7f3719a1b700 1 --2- 172.21.15.16:0/133158213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37140ffc10 0x7f3714100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:53.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.524+0000 7f3719a1b700 1 -- 172.21.15.16:0/133158213 >> 172.21.15.16:0/133158213 conn(0x7f37140fb190 msgr2=0x7f37140fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:53.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.524+0000 7f3719a1b700 1 -- 172.21.15.16:0/133158213 shutdown_connections 2024-04-22T15:47:53.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.524+0000 7f3719a1b700 1 -- 172.21.15.16:0/133158213 wait complete. 2024-04-22T15:47:53.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.525+0000 7f3719a1b700 1 Processor -- start 2024-04-22T15:47:53.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.525+0000 7f3719a1b700 1 -- start start 2024-04-22T15:47:53.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.525+0000 7f3719a1b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37140ffc10 0x7f37141953a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:53.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.525+0000 7f3719a1b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3714100e10 0x7f37141958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:53.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.525+0000 7f3719a1b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3714195f00 con 0x7f37140ffc10 2024-04-22T15:47:53.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.525+0000 7f3719a1b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3714196040 con 0x7f3714100e10 2024-04-22T15:47:53.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.525+0000 7f3712ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37140ffc10 0x7f37141953a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:53.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.525+0000 7f37127fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3714100e10 0x7f37141958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:53.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.526+0000 7f3712ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37140ffc10 0x7f37141953a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35556/0 (socket says 172.21.15.16:35556) 2024-04-22T15:47:53.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.526+0000 7f3712ffd700 1 -- 172.21.15.16:0/1987644765 learned_addr learned my addr 172.21.15.16:0/1987644765 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:47:53.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.526+0000 7f37127fc700 1 -- 172.21.15.16:0/1987644765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37140ffc10 msgr2=0x7f37141953a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:53.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.526+0000 7f37127fc700 1 --2- 172.21.15.16:0/1987644765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37140ffc10 0x7f37141953a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:53.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.526+0000 7f37127fc700 1 -- 172.21.15.16:0/1987644765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37080095f0 con 0x7f3714100e10 2024-04-22T15:47:53.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.526+0000 7f37127fc700 1 --2- 172.21.15.16:0/1987644765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3714100e10 0x7f37141958e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3708000c00 tx=0x7f370801a850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:53.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.526+0000 7f3718a19700 1 -- 172.21.15.16:0/1987644765 <== mon.1 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f370801f070 con 0x7f3714100e10 2024-04-22T15:47:53.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.526+0000 7f3719a1b700 1 -- 172.21.15.16:0/1987644765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3714068580 con 0x7f3714100e10 2024-04-22T15:47:53.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.526+0000 7f3718a19700 1 -- 172.21.15.16:0/1987644765 <== mon.1 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3708005dc0 con 0x7f3714100e10 2024-04-22T15:47:53.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.526+0000 7f3718a19700 1 -- 172.21.15.16:0/1987644765 <== mon.1 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3708026470 con 0x7f3714100e10 2024-04-22T15:47:53.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.526+0000 7f3719a1b700 1 -- 172.21.15.16:0/1987644765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3714068a70 con 0x7f3714100e10 2024-04-22T15:47:53.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.527+0000 7f3719a1b700 1 -- 172.21.15.16:0/1987644765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3714108700 con 0x7f3714100e10 2024-04-22T15:47:53.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.528+0000 7f3718a19700 1 -- 172.21.15.16:0/1987644765 <== mon.1 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f3708023070 con 0x7f3714100e10 2024-04-22T15:47:53.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.529+0000 7f3718a19700 1 --2- 172.21.15.16:0/1987644765 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f36fc06c4e0 0x7f36fc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:53.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.529+0000 7f3718a19700 1 -- 172.21.15.16:0/1987644765 <== mon.1 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f370808ee30 con 0x7f3714100e10 2024-04-22T15:47:53.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.529+0000 7f3712ffd700 1 --2- 172.21.15.16:0/1987644765 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f36fc06c4e0 0x7f36fc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:53.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.529+0000 7f3712ffd700 1 --2- 172.21.15.16:0/1987644765 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f36fc06c4e0 0x7f36fc06e9a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3704005950 tx=0x7f3704009260 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:53.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.534+0000 7f3718a19700 1 -- 172.21.15.16:0/1987644765 <== mon.1 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f370805cfa0 con 0x7f3714100e10 2024-04-22T15:47:53.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.895+0000 7f3719a1b700 1 -- 172.21.15.16:0/1987644765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f37141088f0 con 0x7f3714100e10 2024-04-22T15:47:53.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.895+0000 7f3718a19700 1 -- 172.21.15.16:0/1987644765 <== mon.1 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7f370805cb30 con 0x7f3714100e10 2024-04-22T15:47:53.899 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:47:53.899 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-22T15:47:53.899 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T15:47:53.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:47:53.899 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-22T15:47:53.899 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T15:47:53.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:47:53.899 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-22T15:47:53.899 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-22T15:47:53.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:47:53.899 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-22T15:47:53.899 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T15:47:53.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:47:53.900 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-22T15:47:53.900 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 14 2024-04-22T15:47:53.900 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-22T15:47:53.900 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:47:53.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.898+0000 7f3719a1b700 1 -- 172.21.15.16:0/1987644765 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f36fc06c4e0 msgr2=0x7f36fc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:53.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.898+0000 7f3719a1b700 1 --2- 172.21.15.16:0/1987644765 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f36fc06c4e0 0x7f36fc06e9a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3704005950 tx=0x7f3704009260 comp rx=0 tx=0).stop 2024-04-22T15:47:53.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.898+0000 7f3719a1b700 1 -- 172.21.15.16:0/1987644765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3714100e10 msgr2=0x7f37141958e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:53.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.898+0000 7f3719a1b700 1 --2- 172.21.15.16:0/1987644765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3714100e10 0x7f37141958e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3708000c00 tx=0x7f370801a850 comp rx=0 tx=0).stop 2024-04-22T15:47:53.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.898+0000 7f3719a1b700 1 -- 172.21.15.16:0/1987644765 shutdown_connections 2024-04-22T15:47:53.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.898+0000 7f3719a1b700 1 --2- 172.21.15.16:0/1987644765 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f36fc06c4e0 0x7f36fc06e9a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:53.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.898+0000 7f3719a1b700 1 --2- 172.21.15.16:0/1987644765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37140ffc10 0x7f37141953a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:53.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.898+0000 7f3719a1b700 1 --2- 172.21.15.16:0/1987644765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3714100e10 0x7f37141958e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:53.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.898+0000 7f3719a1b700 1 -- 172.21.15.16:0/1987644765 >> 172.21.15.16:0/1987644765 conn(0x7f37140fb190 msgr2=0x7f3714104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:53.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.898+0000 7f3719a1b700 1 -- 172.21.15.16:0/1987644765 shutdown_connections 2024-04-22T15:47:53.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:53.898+0000 7f3719a1b700 1 -- 172.21.15.16:0/1987644765 wait complete. 2024-04-22T15:47:54.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.011+0000 7fc83d299700 1 -- 172.21.15.16:0/3281398081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8380ffc10 msgr2=0x7fc838100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:54.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.011+0000 7fc83d299700 1 --2- 172.21.15.16:0/3281398081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8380ffc10 0x7fc838100030 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc828009910 tx=0x7fc828009c20 comp rx=0 tx=0).stop 2024-04-22T15:47:54.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.011+0000 7fc83d299700 1 -- 172.21.15.16:0/3281398081 shutdown_connections 2024-04-22T15:47:54.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.011+0000 7fc83d299700 1 --2- 172.21.15.16:0/3281398081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc838100e10 0x7fc838101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:54.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.011+0000 7fc83d299700 1 --2- 172.21.15.16:0/3281398081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8380ffc10 0x7fc838100030 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:54.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.011+0000 7fc83d299700 1 -- 172.21.15.16:0/3281398081 >> 172.21.15.16:0/3281398081 conn(0x7fc8380fb190 msgr2=0x7fc8380fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:54.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.011+0000 7fc83d299700 1 -- 172.21.15.16:0/3281398081 shutdown_connections 2024-04-22T15:47:54.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.011+0000 7fc83d299700 1 -- 172.21.15.16:0/3281398081 wait complete. 2024-04-22T15:47:54.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.012+0000 7fc83d299700 1 Processor -- start 2024-04-22T15:47:54.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.012+0000 7fc83d299700 1 -- start start 2024-04-22T15:47:54.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.012+0000 7fc83d299700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8380ffc10 0x7fc8381953d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:54.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.012+0000 7fc83d299700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc838100e10 0x7fc838195910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:54.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.012+0000 7fc83d299700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc838195f30 con 0x7fc838100e10 2024-04-22T15:47:54.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.012+0000 7fc83d299700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc838196070 con 0x7fc8380ffc10 2024-04-22T15:47:54.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.012+0000 7fc8367fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc838100e10 0x7fc838195910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:54.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.012+0000 7fc836ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8380ffc10 0x7fc8381953d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:54.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.012+0000 7fc8367fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc838100e10 0x7fc838195910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35570/0 (socket says 172.21.15.16:35570) 2024-04-22T15:47:54.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.012+0000 7fc8367fc700 1 -- 172.21.15.16:0/2050699761 learned_addr learned my addr 172.21.15.16:0/2050699761 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:47:54.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.013+0000 7fc8367fc700 1 -- 172.21.15.16:0/2050699761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8380ffc10 msgr2=0x7fc8381953d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:54.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.013+0000 7fc8367fc700 1 --2- 172.21.15.16:0/2050699761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8380ffc10 0x7fc8381953d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:54.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.013+0000 7fc8367fc700 1 -- 172.21.15.16:0/2050699761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8280095f0 con 0x7fc838100e10 2024-04-22T15:47:54.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.013+0000 7fc836ffd700 1 --2- 172.21.15.16:0/2050699761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8380ffc10 0x7fc8381953d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:47:54.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.013+0000 7fc8367fc700 1 --2- 172.21.15.16:0/2050699761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc838100e10 0x7fc838195910 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc82c007910 tx=0x7fc82c007c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:54.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.013+0000 7fc81ffff700 1 -- 172.21.15.16:0/2050699761 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc82c012070 con 0x7fc838100e10 2024-04-22T15:47:54.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.013+0000 7fc81ffff700 1 -- 172.21.15.16:0/2050699761 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc82c007e60 con 0x7fc838100e10 2024-04-22T15:47:54.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.013+0000 7fc81ffff700 1 -- 172.21.15.16:0/2050699761 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc82c0054c0 con 0x7fc838100e10 2024-04-22T15:47:54.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.013+0000 7fc83d299700 1 -- 172.21.15.16:0/2050699761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8380685e0 con 0x7fc838100e10 2024-04-22T15:47:54.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.013+0000 7fc83d299700 1 -- 172.21.15.16:0/2050699761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc838068b30 con 0x7fc838100e10 2024-04-22T15:47:54.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.014+0000 7fc83d299700 1 -- 172.21.15.16:0/2050699761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc838108700 con 0x7fc838100e10 2024-04-22T15:47:54.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.015+0000 7fc81ffff700 1 -- 172.21.15.16:0/2050699761 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fc82c016070 con 0x7fc838100e10 2024-04-22T15:47:54.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.015+0000 7fc81ffff700 1 --2- 172.21.15.16:0/2050699761 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fc82006c4e0 0x7fc82006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:47:54.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.015+0000 7fc81ffff700 1 -- 172.21.15.16:0/2050699761 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fc82c08ca20 con 0x7fc838100e10 2024-04-22T15:47:54.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.015+0000 7fc836ffd700 1 --2- 172.21.15.16:0/2050699761 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fc82006c4e0 0x7fc82006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:47:54.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.016+0000 7fc836ffd700 1 --2- 172.21.15.16:0/2050699761 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fc82006c4e0 0x7fc82006e9a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc828006010 tx=0x7fc82800b560 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:47:54.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.021+0000 7fc81ffff700 1 -- 172.21.15.16:0/2050699761 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc82c05a9a0 con 0x7fc838100e10 2024-04-22T15:47:54.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:54 smithi016 ceph-mon[26620]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:54.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:54 smithi016 ceph-mon[26620]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:54.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:54 smithi016 ceph-mon[26620]: from='client.24363 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:54.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:54 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/1987644765' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:47:54.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.252+0000 7fc83d299700 1 -- 172.21.15.16:0/2050699761 --> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc838105760 con 0x7fc82006c4e0 2024-04-22T15:47:54.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.253+0000 7fc81ffff700 1 -- 172.21.15.16:0/2050699761 <== mgr.14217 v2:172.21.15.16:6800/1057234399 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fc838105760 con 0x7fc82006c4e0 2024-04-22T15:47:54.256 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:47:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:47:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-22T15:47:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T15:47:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [], 2024-04-22T15:47:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "0/2 daemons upgraded", 2024-04-22T15:47:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb image on host smithi027", 2024-04-22T15:47:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-22T15:47:54.257 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:47:54.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.255+0000 7fc83d299700 1 -- 172.21.15.16:0/2050699761 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fc82006c4e0 msgr2=0x7fc82006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:54.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.255+0000 7fc83d299700 1 --2- 172.21.15.16:0/2050699761 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fc82006c4e0 0x7fc82006e9a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc828006010 tx=0x7fc82800b560 comp rx=0 tx=0).stop 2024-04-22T15:47:54.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.255+0000 7fc83d299700 1 -- 172.21.15.16:0/2050699761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc838100e10 msgr2=0x7fc838195910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:47:54.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.255+0000 7fc83d299700 1 --2- 172.21.15.16:0/2050699761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc838100e10 0x7fc838195910 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc82c007910 tx=0x7fc82c007c20 comp rx=0 tx=0).stop 2024-04-22T15:47:54.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.255+0000 7fc83d299700 1 -- 172.21.15.16:0/2050699761 shutdown_connections 2024-04-22T15:47:54.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.255+0000 7fc83d299700 1 --2- 172.21.15.16:0/2050699761 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7fc82006c4e0 0x7fc82006e9a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:54.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.255+0000 7fc83d299700 1 --2- 172.21.15.16:0/2050699761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8380ffc10 0x7fc8381953d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:54.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.255+0000 7fc83d299700 1 --2- 172.21.15.16:0/2050699761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc838100e10 0x7fc838195910 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:47:54.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.255+0000 7fc83d299700 1 -- 172.21.15.16:0/2050699761 >> 172.21.15.16:0/2050699761 conn(0x7fc8380fb190 msgr2=0x7fc838104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:47:54.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.255+0000 7fc83d299700 1 -- 172.21.15.16:0/2050699761 shutdown_connections 2024-04-22T15:47:54.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:47:54.255+0000 7fc83d299700 1 -- 172.21.15.16:0/2050699761 wait complete. 2024-04-22T15:47:54.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:54 smithi027 ceph-mon[34922]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:54.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:54 smithi027 ceph-mon[34922]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:54.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:54 smithi027 ceph-mon[34922]: from='client.24363 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:54.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:54 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/1987644765' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:47:55.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:55 smithi027 ceph-mon[34922]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:55.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:55 smithi027 ceph-mon[34922]: pgmap v221: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:47:55.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:55 smithi016 ceph-mon[26620]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:47:55.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:55 smithi016 ceph-mon[26620]: pgmap v221: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:47:56.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:56 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:47:56.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:56 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:47:57.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:57 smithi027 ceph-mon[34922]: pgmap v222: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:47:57.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:57 smithi016 ceph-mon[26620]: pgmap v222: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:47:59.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:47:59 smithi027 ceph-mon[34922]: pgmap v223: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:47:59.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:47:59 smithi016 ceph-mon[26620]: pgmap v223: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:48:01.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:01 smithi027 ceph-mon[34922]: pgmap v224: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:48:01.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:01 smithi016 ceph-mon[26620]: pgmap v224: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:48:03.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:03 smithi016 ceph-mon[26620]: pgmap v225: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:48:03.820 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:03 smithi027 ceph-mon[34922]: pgmap v225: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:48:05.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:05 smithi027 ceph-mon[34922]: pgmap v226: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:48:05.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:05 smithi016 ceph-mon[26620]: pgmap v226: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:48:07.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:06 smithi027 ceph-mon[34922]: pgmap v227: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:48:07.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:06 smithi016 ceph-mon[26620]: pgmap v227: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:48:09.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:09 smithi027 ceph-mon[34922]: pgmap v228: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:48:09.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:09 smithi016 ceph-mon[26620]: pgmap v228: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:48:10.815 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:10 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:48:10.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:10 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:48:11.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:11 smithi027 ceph-mon[34922]: pgmap v229: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:48:11.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:11 smithi016 ceph-mon[26620]: pgmap v229: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:48:13.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:13 smithi027 ceph-mon[34922]: pgmap v230: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:48:13.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:13 smithi016 ceph-mon[26620]: pgmap v230: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:48:15.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:15 smithi027 ceph-mon[34922]: pgmap v231: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:48:15.899 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:15 smithi016 ceph-mon[26620]: pgmap v231: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:48:17.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:16 smithi027 ceph-mon[34922]: pgmap v232: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:48:17.153 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:16 smithi016 ceph-mon[26620]: pgmap v232: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:48:19.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:19 smithi027 ceph-mon[34922]: pgmap v233: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:48:19.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:19 smithi016 ceph-mon[26620]: pgmap v233: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:48:21.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:21 smithi027 ceph-mon[34922]: pgmap v234: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:48:21.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:21 smithi016 ceph-mon[26620]: pgmap v234: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:48:23.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:23 smithi027 ceph-mon[34922]: pgmap v235: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:48:23.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:23 smithi016 ceph-mon[26620]: pgmap v235: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:48:24.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.423+0000 7f4d287a6700 1 -- 172.21.15.16:0/3127636439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d200fff50 msgr2=0x7f4d201003d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.423+0000 7f4d287a6700 1 --2- 172.21.15.16:0/3127636439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d200fff50 0x7f4d201003d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4d10009910 tx=0x7f4d10009c20 comp rx=0 tx=0).stop 2024-04-22T15:48:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.423+0000 7f4d287a6700 1 -- 172.21.15.16:0/3127636439 shutdown_connections 2024-04-22T15:48:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.423+0000 7f4d287a6700 1 --2- 172.21.15.16:0/3127636439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d200fff50 0x7f4d201003d0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.423+0000 7f4d287a6700 1 --2- 172.21.15.16:0/3127636439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d200ff5f0 0x7f4d200ffa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.423+0000 7f4d287a6700 1 -- 172.21.15.16:0/3127636439 >> 172.21.15.16:0/3127636439 conn(0x7f4d200fb190 msgr2=0x7f4d200fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.423+0000 7f4d287a6700 1 -- 172.21.15.16:0/3127636439 shutdown_connections 2024-04-22T15:48:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.423+0000 7f4d287a6700 1 -- 172.21.15.16:0/3127636439 wait complete. 2024-04-22T15:48:24.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.424+0000 7f4d287a6700 1 Processor -- start 2024-04-22T15:48:24.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.424+0000 7f4d287a6700 1 -- start start 2024-04-22T15:48:24.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.424+0000 7f4d287a6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d200ff5f0 0x7f4d20198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:24.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.424+0000 7f4d287a6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d200fff50 0x7f4d20198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:24.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.424+0000 7f4d287a6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d201991d0 con 0x7f4d200ff5f0 2024-04-22T15:48:24.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.424+0000 7f4d287a6700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d20199310 con 0x7f4d200fff50 2024-04-22T15:48:24.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.424+0000 7f4d26542700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d200ff5f0 0x7f4d20198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:24.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.425+0000 7f4d26542700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d200ff5f0 0x7f4d20198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33830/0 (socket says 172.21.15.16:33830) 2024-04-22T15:48:24.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.425+0000 7f4d26542700 1 -- 172.21.15.16:0/2431504523 learned_addr learned my addr 172.21.15.16:0/2431504523 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:48:24.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.425+0000 7f4d25d41700 1 --2- 172.21.15.16:0/2431504523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d200fff50 0x7f4d20198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:24.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.425+0000 7f4d26542700 1 -- 172.21.15.16:0/2431504523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d200fff50 msgr2=0x7f4d20198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:24.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.425+0000 7f4d26542700 1 --2- 172.21.15.16:0/2431504523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d200fff50 0x7f4d20198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:24.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.425+0000 7f4d26542700 1 -- 172.21.15.16:0/2431504523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d100095f0 con 0x7f4d200ff5f0 2024-04-22T15:48:24.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.425+0000 7f4d26542700 1 --2- 172.21.15.16:0/2431504523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d200ff5f0 0x7f4d20198670 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c00d8d0 tx=0x7f4d1c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:24.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.425+0000 7f4d177fe700 1 -- 172.21.15.16:0/2431504523 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d1c0098a0 con 0x7f4d200ff5f0 2024-04-22T15:48:24.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.425+0000 7f4d177fe700 1 -- 172.21.15.16:0/2431504523 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4d1c010460 con 0x7f4d200ff5f0 2024-04-22T15:48:24.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.425+0000 7f4d287a6700 1 -- 172.21.15.16:0/2431504523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d2019ddc0 con 0x7f4d200ff5f0 2024-04-22T15:48:24.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.426+0000 7f4d287a6700 1 -- 172.21.15.16:0/2431504523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d2019e240 con 0x7f4d200ff5f0 2024-04-22T15:48:24.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.426+0000 7f4d177fe700 1 -- 172.21.15.16:0/2431504523 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d1c00f5f0 con 0x7f4d200ff5f0 2024-04-22T15:48:24.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.427+0000 7f4d287a6700 1 -- 172.21.15.16:0/2431504523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d2010b9f0 con 0x7f4d200ff5f0 2024-04-22T15:48:24.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.428+0000 7f4d177fe700 1 -- 172.21.15.16:0/2431504523 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f4d1c009a00 con 0x7f4d200ff5f0 2024-04-22T15:48:24.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.429+0000 7f4d177fe700 1 --2- 172.21.15.16:0/2431504523 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4d0c06c4e0 0x7f4d0c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:24.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.429+0000 7f4d177fe700 1 -- 172.21.15.16:0/2431504523 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f4d1c08b740 con 0x7f4d200ff5f0 2024-04-22T15:48:24.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.429+0000 7f4d25d41700 1 --2- 172.21.15.16:0/2431504523 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4d0c06c4e0 0x7f4d0c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:24.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.430+0000 7f4d25d41700 1 --2- 172.21.15.16:0/2431504523 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4d0c06c4e0 0x7f4d0c06e9a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4d1000b5c0 tx=0x7f4d10005ee0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:24.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.434+0000 7f4d177fe700 1 -- 172.21.15.16:0/2431504523 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4d1c059950 con 0x7f4d200ff5f0 2024-04-22T15:48:24.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.874+0000 7f4d287a6700 1 -- 172.21.15.16:0/2431504523 --> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d20109340 con 0x7f4d0c06c4e0 2024-04-22T15:48:24.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.876+0000 7f4d177fe700 1 -- 172.21.15.16:0/2431504523 <== mgr.14217 v2:172.21.15.16:6800/1057234399 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f4d20109340 con 0x7f4d0c06c4e0 2024-04-22T15:48:24.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.878+0000 7f4d287a6700 1 -- 172.21.15.16:0/2431504523 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4d0c06c4e0 msgr2=0x7f4d0c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:24.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.878+0000 7f4d287a6700 1 --2- 172.21.15.16:0/2431504523 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4d0c06c4e0 0x7f4d0c06e9a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4d1000b5c0 tx=0x7f4d10005ee0 comp rx=0 tx=0).stop 2024-04-22T15:48:24.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.878+0000 7f4d287a6700 1 -- 172.21.15.16:0/2431504523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d200ff5f0 msgr2=0x7f4d20198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:24.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.878+0000 7f4d287a6700 1 --2- 172.21.15.16:0/2431504523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d200ff5f0 0x7f4d20198670 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c00d8d0 tx=0x7f4d1c00dbe0 comp rx=0 tx=0).stop 2024-04-22T15:48:24.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.879+0000 7f4d287a6700 1 -- 172.21.15.16:0/2431504523 shutdown_connections 2024-04-22T15:48:24.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.879+0000 7f4d287a6700 1 --2- 172.21.15.16:0/2431504523 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f4d0c06c4e0 0x7f4d0c06e9a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:24.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.879+0000 7f4d287a6700 1 --2- 172.21.15.16:0/2431504523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d200ff5f0 0x7f4d20198670 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:24.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.879+0000 7f4d287a6700 1 --2- 172.21.15.16:0/2431504523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d200fff50 0x7f4d20198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:24.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.879+0000 7f4d287a6700 1 -- 172.21.15.16:0/2431504523 >> 172.21.15.16:0/2431504523 conn(0x7f4d200fb190 msgr2=0x7f4d20107c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:24.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.879+0000 7f4d287a6700 1 -- 172.21.15.16:0/2431504523 shutdown_connections 2024-04-22T15:48:24.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:24.879+0000 7f4d287a6700 1 -- 172.21.15.16:0/2431504523 wait complete. 2024-04-22T15:48:24.897 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:48:25.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.047+0000 7f8110952700 1 -- 172.21.15.16:0/3125316258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f810c100e10 msgr2=0x7f810c101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:25.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.047+0000 7f8110952700 1 --2- 172.21.15.16:0/3125316258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f810c100e10 0x7f810c101270 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8100009910 tx=0x7f8100009c20 comp rx=0 tx=0).stop 2024-04-22T15:48:25.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.047+0000 7f8110952700 1 -- 172.21.15.16:0/3125316258 shutdown_connections 2024-04-22T15:48:25.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.047+0000 7f8110952700 1 --2- 172.21.15.16:0/3125316258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f810c100e10 0x7f810c101270 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:25.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.047+0000 7f8110952700 1 --2- 172.21.15.16:0/3125316258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f810c0ffc10 0x7f810c100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:25.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.047+0000 7f8110952700 1 -- 172.21.15.16:0/3125316258 >> 172.21.15.16:0/3125316258 conn(0x7f810c0fb190 msgr2=0x7f810c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:25.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.048+0000 7f8110952700 1 -- 172.21.15.16:0/3125316258 shutdown_connections 2024-04-22T15:48:25.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.048+0000 7f8110952700 1 -- 172.21.15.16:0/3125316258 wait complete. 2024-04-22T15:48:25.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.048+0000 7f8110952700 1 Processor -- start 2024-04-22T15:48:25.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.048+0000 7f8110952700 1 -- start start 2024-04-22T15:48:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.049+0000 7f8110952700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f810c0ffc10 0x7f810c103ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.049+0000 7f8110952700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f810c100e10 0x7f810c1021f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.049+0000 7f8110952700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810c19a010 con 0x7f810c0ffc10 2024-04-22T15:48:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.049+0000 7f8110952700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810c19a150 con 0x7f810c100e10 2024-04-22T15:48:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.049+0000 7f8109d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f810c100e10 0x7f810c1021f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.049+0000 7f810a59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f810c0ffc10 0x7f810c103ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.049+0000 7f8109d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f810c100e10 0x7f810c1021f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:52866/0 (socket says 172.21.15.16:52866) 2024-04-22T15:48:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.049+0000 7f8109d9b700 1 -- 172.21.15.16:0/3191860649 learned_addr learned my addr 172.21.15.16:0/3191860649 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:48:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.049+0000 7f8109d9b700 1 -- 172.21.15.16:0/3191860649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f810c0ffc10 msgr2=0x7f810c103ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.049+0000 7f8109d9b700 1 --2- 172.21.15.16:0/3191860649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f810c0ffc10 0x7f810c103ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.049+0000 7f8109d9b700 1 -- 172.21.15.16:0/3191860649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81000095f0 con 0x7f810c100e10 2024-04-22T15:48:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.050+0000 7f8109d9b700 1 --2- 172.21.15.16:0/3191860649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f810c100e10 0x7f810c1021f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f810000b6f0 tx=0x7f810000ff80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:25.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.050+0000 7f80fb7fe700 1 -- 172.21.15.16:0/3191860649 <== mon.1 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f810001d070 con 0x7f810c100e10 2024-04-22T15:48:25.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.050+0000 7f80fb7fe700 1 -- 172.21.15.16:0/3191860649 <== mon.1 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f81000052a0 con 0x7f810c100e10 2024-04-22T15:48:25.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.050+0000 7f8110952700 1 -- 172.21.15.16:0/3191860649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f810c102730 con 0x7f810c100e10 2024-04-22T15:48:25.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.051+0000 7f80fb7fe700 1 -- 172.21.15.16:0/3191860649 <== mon.1 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81000179a0 con 0x7f810c100e10 2024-04-22T15:48:25.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.051+0000 7f8110952700 1 -- 172.21.15.16:0/3191860649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f810c102b90 con 0x7f810c100e10 2024-04-22T15:48:25.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.052+0000 7f80fb7fe700 1 -- 172.21.15.16:0/3191860649 <== mon.1 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f810000b7c0 con 0x7f810c100e10 2024-04-22T15:48:25.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.052+0000 7f8110952700 1 -- 172.21.15.16:0/3191860649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f810c108700 con 0x7f810c100e10 2024-04-22T15:48:25.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.053+0000 7f80fb7fe700 1 --2- 172.21.15.16:0/3191860649 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f80f406c320 0x7f80f406e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:25.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.053+0000 7f80fb7fe700 1 -- 172.21.15.16:0/3191860649 <== mon.1 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f810008c1b0 con 0x7f810c100e10 2024-04-22T15:48:25.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.053+0000 7f810a59c700 1 --2- 172.21.15.16:0/3191860649 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f80f406c320 0x7f80f406e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:25.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.053+0000 7f810a59c700 1 --2- 172.21.15.16:0/3191860649 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f80f406c320 0x7f80f406e7e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f810c100c70 tx=0x7f80fc009260 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:25.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.062+0000 7f80fb7fe700 1 -- 172.21.15.16:0/3191860649 <== mon.1 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f810005a3a0 con 0x7f810c100e10 2024-04-22T15:48:25.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.506+0000 7f8110952700 1 -- 172.21.15.16:0/3191860649 --> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f810c1a2c10 con 0x7f80f406c320 2024-04-22T15:48:25.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.705+0000 7f80fb7fe700 1 -- 172.21.15.16:0/3191860649 <== mgr.14217 v2:172.21.15.16:6800/1057234399 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f810c1a2c10 con 0x7f80f406c320 2024-04-22T15:48:25.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.708+0000 7f8110952700 1 -- 172.21.15.16:0/3191860649 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f80f406c320 msgr2=0x7f80f406e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:25.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.708+0000 7f8110952700 1 --2- 172.21.15.16:0/3191860649 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f80f406c320 0x7f80f406e7e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f810c100c70 tx=0x7f80fc009260 comp rx=0 tx=0).stop 2024-04-22T15:48:25.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.708+0000 7f8110952700 1 -- 172.21.15.16:0/3191860649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f810c100e10 msgr2=0x7f810c1021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:25.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.708+0000 7f8110952700 1 --2- 172.21.15.16:0/3191860649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f810c100e10 0x7f810c1021f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f810000b6f0 tx=0x7f810000ff80 comp rx=0 tx=0).stop 2024-04-22T15:48:25.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.708+0000 7f8110952700 1 -- 172.21.15.16:0/3191860649 shutdown_connections 2024-04-22T15:48:25.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.708+0000 7f8110952700 1 --2- 172.21.15.16:0/3191860649 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f80f406c320 0x7f80f406e7e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:25.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.708+0000 7f8110952700 1 --2- 172.21.15.16:0/3191860649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f810c0ffc10 0x7f810c103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:25.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.708+0000 7f8110952700 1 --2- 172.21.15.16:0/3191860649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f810c100e10 0x7f810c1021f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:25.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.708+0000 7f8110952700 1 -- 172.21.15.16:0/3191860649 >> 172.21.15.16:0/3191860649 conn(0x7f810c0fb190 msgr2=0x7f810c068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:25.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.708+0000 7f8110952700 1 -- 172.21.15.16:0/3191860649 shutdown_connections 2024-04-22T15:48:25.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.708+0000 7f8110952700 1 -- 172.21.15.16:0/3191860649 wait complete. 2024-04-22T15:48:25.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.822+0000 7f5c2f915700 1 -- 172.21.15.16:0/77584334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c281040d0 msgr2=0x7f5c281064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:25.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.822+0000 7f5c2f915700 1 --2- 172.21.15.16:0/77584334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c281040d0 0x7f5c281064c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f5c18009910 tx=0x7f5c18009c20 comp rx=0 tx=0).stop 2024-04-22T15:48:25.825 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:25 smithi016 ceph-mon[26620]: pgmap v236: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:48:25.825 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:25 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:48:25.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.822+0000 7f5c2f915700 1 -- 172.21.15.16:0/77584334 shutdown_connections 2024-04-22T15:48:25.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.822+0000 7f5c2f915700 1 --2- 172.21.15.16:0/77584334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c281040d0 0x7f5c281064c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:25.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.822+0000 7f5c2f915700 1 --2- 172.21.15.16:0/77584334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c281017a0 0x7f5c28103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:25.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.822+0000 7f5c2f915700 1 -- 172.21.15.16:0/77584334 >> 172.21.15.16:0/77584334 conn(0x7f5c280fb190 msgr2=0x7f5c280fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:25.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.822+0000 7f5c2f915700 1 -- 172.21.15.16:0/77584334 shutdown_connections 2024-04-22T15:48:25.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.823+0000 7f5c2f915700 1 -- 172.21.15.16:0/77584334 wait complete. 2024-04-22T15:48:25.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.823+0000 7f5c2f915700 1 Processor -- start 2024-04-22T15:48:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.823+0000 7f5c2f915700 1 -- start start 2024-04-22T15:48:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.824+0000 7f5c2f915700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c281017a0 0x7f5c281962b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.824+0000 7f5c2f915700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c281040d0 0x7f5c281967f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.824+0000 7f5c2f915700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c28196e10 con 0x7f5c281040d0 2024-04-22T15:48:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.824+0000 7f5c2f915700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c28196f50 con 0x7f5c281017a0 2024-04-22T15:48:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.824+0000 7f5c2d6b1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c281017a0 0x7f5c281962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.824+0000 7f5c2d6b1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c281017a0 0x7f5c281962b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:52878/0 (socket says 172.21.15.16:52878) 2024-04-22T15:48:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.824+0000 7f5c2d6b1700 1 -- 172.21.15.16:0/1908917912 learned_addr learned my addr 172.21.15.16:0/1908917912 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:48:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.824+0000 7f5c2d6b1700 1 -- 172.21.15.16:0/1908917912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c281040d0 msgr2=0x7f5c281967f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.824+0000 7f5c2d6b1700 1 --2- 172.21.15.16:0/1908917912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c281040d0 0x7f5c281967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.824+0000 7f5c2d6b1700 1 -- 172.21.15.16:0/1908917912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c180095f0 con 0x7f5c281017a0 2024-04-22T15:48:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.825+0000 7f5c2d6b1700 1 --2- 172.21.15.16:0/1908917912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c281017a0 0x7f5c281962b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5c2400e920 tx=0x7f5c2400ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:25.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.825+0000 7f5c1e7fc700 1 -- 172.21.15.16:0/1908917912 <== mon.1 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c2400cba0 con 0x7f5c281017a0 2024-04-22T15:48:25.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.825+0000 7f5c1e7fc700 1 -- 172.21.15.16:0/1908917912 <== mon.1 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5c2400cd00 con 0x7f5c281017a0 2024-04-22T15:48:25.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.825+0000 7f5c2f915700 1 -- 172.21.15.16:0/1908917912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c2819ba00 con 0x7f5c281017a0 2024-04-22T15:48:25.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.825+0000 7f5c2f915700 1 -- 172.21.15.16:0/1908917912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c2819bf50 con 0x7f5c281017a0 2024-04-22T15:48:25.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.825+0000 7f5c1e7fc700 1 -- 172.21.15.16:0/1908917912 <== mon.1 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c24018770 con 0x7f5c281017a0 2024-04-22T15:48:25.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.826+0000 7f5c2f915700 1 -- 172.21.15.16:0/1908917912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c0c005320 con 0x7f5c281017a0 2024-04-22T15:48:25.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.827+0000 7f5c1e7fc700 1 -- 172.21.15.16:0/1908917912 <== mon.1 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f5c24010430 con 0x7f5c281017a0 2024-04-22T15:48:25.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.828+0000 7f5c1e7fc700 1 --2- 172.21.15.16:0/1908917912 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f5c1406c530 0x7f5c1406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:25.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.828+0000 7f5c1e7fc700 1 -- 172.21.15.16:0/1908917912 <== mon.1 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f5c24014070 con 0x7f5c281017a0 2024-04-22T15:48:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.828+0000 7f5c2ceb0700 1 --2- 172.21.15.16:0/1908917912 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f5c1406c530 0x7f5c1406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.829+0000 7f5c2ceb0700 1 --2- 172.21.15.16:0/1908917912 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f5c1406c530 0x7f5c1406e9f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5c180098e0 tx=0x7f5c18009f70 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:25.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:25 smithi027 ceph-mon[34922]: pgmap v236: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:48:25.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:25 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:48:25.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:25.837+0000 7f5c1e7fc700 1 -- 172.21.15.16:0/1908917912 <== mon.1 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5c240520e0 con 0x7f5c281017a0 2024-04-22T15:48:26.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.078+0000 7f5c2f915700 1 -- 172.21.15.16:0/1908917912 --> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c0c000bf0 con 0x7f5c1406c530 2024-04-22T15:48:26.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.094+0000 7f5c1e7fc700 1 -- 172.21.15.16:0/1908917912 <== mgr.14217 v2:172.21.15.16:6800/1057234399 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3528 (secure 0 0 0) 0x7f5c0c000bf0 con 0x7f5c1406c530 2024-04-22T15:48:26.097 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:48:26.097 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.smithi016 smithi016 *:9093,9094 running (5m) 88s ago 7m 24.9M - 0.25.0 c8568f914cd2 fab40987f1b0 2024-04-22T15:48:26.097 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi016 smithi016 running (7m) 88s ago 7m 7461k - 18.2.1 5be31c24972a f8b627d3f41e 2024-04-22T15:48:26.097 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi027 smithi027 running (6m) 91s ago 6m 7319k - 18.2.1 5be31c24972a f00fd40b4220 2024-04-22T15:48:26.098 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi016 smithi016 running (7m) 88s ago 7m 6656k - 18.2.1 5be31c24972a 6b93db1fa0d9 2024-04-22T15:48:26.098 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi027 smithi027 running (6m) 91s ago 6m 6665k - 18.2.1 5be31c24972a d3bd439f30a0 2024-04-22T15:48:26.098 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.smithi016 smithi016 *:3000 running (5m) 88s ago 7m 86.2M - 9.4.7 954c08fa6188 a85bdbf5073d 2024-04-22T15:48:26.098 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.fkkejm smithi016 running (113s) 88s ago 112s 11.4M - 18.2.1 5be31c24972a 4153ac94cea0 2024-04-22T15:48:26.098 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.pgvphu smithi016 running (106s) 88s ago 106s 14.4M - 18.2.1 5be31c24972a 93dc072068a2 2024-04-22T15:48:26.098 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.dgqyes smithi027 running (104s) 91s ago 103s 14.6M - 18.2.1 5be31c24972a 41a9220430fd 2024-04-22T15:48:26.098 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.iobjds smithi027 running (110s) 91s ago 110s 15.2M - 18.2.1 5be31c24972a c1c8d6770f02 2024-04-22T15:48:26.098 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi016.rnsnnr smithi016 *:9283,8765,8443 running (8m) 88s ago 8m 488M - 18.2.1 5be31c24972a 1cf5a7efea72 2024-04-22T15:48:26.098 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi027.ukdgzs smithi027 *:8443,9283,8765 running (6m) 91s ago 6m 442M - 18.2.1 5be31c24972a c59a9cac2d99 2024-04-22T15:48:26.098 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi016 smithi016 running (9m) 88s ago 9m 47.9M 2048M 18.2.1 5be31c24972a 72af40873cb6 2024-04-22T15:48:26.098 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi027 smithi027 running (6m) 91s ago 6m 36.7M 2048M 18.2.1 5be31c24972a 9f21ce7fa7ef 2024-04-22T15:48:26.098 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi016 smithi016 *:9100 running (7m) 88s ago 7m 15.7M - 1.5.0 0da6a335fe13 1937a0bc1959 2024-04-22T15:48:26.099 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi027 smithi027 *:9100 running (6m) 91s ago 6m 15.7M - 1.5.0 0da6a335fe13 00440c991b3f 2024-04-22T15:48:26.099 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (5m) 88s ago 5m 48.4M 3948M 18.2.1 5be31c24972a 3c6ebc4ea041 2024-04-22T15:48:26.099 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (5m) 88s ago 5m 47.8M 3948M 18.2.1 5be31c24972a 6aa2e6796784 2024-04-22T15:48:26.099 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (4m) 88s ago 4m 45.9M 3948M 18.2.1 5be31c24972a 3d9d310aa9eb 2024-04-22T15:48:26.099 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi027 running (4m) 91s ago 4m 49.1M 4970M 18.2.1 5be31c24972a cdf9b7aaf865 2024-04-22T15:48:26.099 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (3m) 91s ago 3m 43.5M 4970M 18.2.1 5be31c24972a 629397d49c76 2024-04-22T15:48:26.099 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi027 running (3m) 91s ago 3m 46.6M 4970M 18.2.1 5be31c24972a 23adcf675b04 2024-04-22T15:48:26.099 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.smithi016 smithi016 *:9095 running (5m) 88s ago 6m 59.2M - 2.43.0 a07b618ecd1d 2ba3b93ebe19 2024-04-22T15:48:26.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.096+0000 7f5c2f915700 1 -- 172.21.15.16:0/1908917912 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f5c1406c530 msgr2=0x7f5c1406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:26.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.096+0000 7f5c2f915700 1 --2- 172.21.15.16:0/1908917912 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f5c1406c530 0x7f5c1406e9f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5c180098e0 tx=0x7f5c18009f70 comp rx=0 tx=0).stop 2024-04-22T15:48:26.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.096+0000 7f5c2f915700 1 -- 172.21.15.16:0/1908917912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c281017a0 msgr2=0x7f5c281962b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:26.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.096+0000 7f5c2f915700 1 --2- 172.21.15.16:0/1908917912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c281017a0 0x7f5c281962b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5c2400e920 tx=0x7f5c2400ece0 comp rx=0 tx=0).stop 2024-04-22T15:48:26.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.097+0000 7f5c2f915700 1 -- 172.21.15.16:0/1908917912 shutdown_connections 2024-04-22T15:48:26.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.097+0000 7f5c2f915700 1 --2- 172.21.15.16:0/1908917912 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f5c1406c530 0x7f5c1406e9f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:26.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.097+0000 7f5c2f915700 1 --2- 172.21.15.16:0/1908917912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c281017a0 0x7f5c281962b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:26.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.097+0000 7f5c2f915700 1 --2- 172.21.15.16:0/1908917912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c281040d0 0x7f5c281967f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:26.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.097+0000 7f5c2f915700 1 -- 172.21.15.16:0/1908917912 >> 172.21.15.16:0/1908917912 conn(0x7f5c280fb190 msgr2=0x7f5c280fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:26.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.097+0000 7f5c2f915700 1 -- 172.21.15.16:0/1908917912 shutdown_connections 2024-04-22T15:48:26.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.097+0000 7f5c2f915700 1 -- 172.21.15.16:0/1908917912 wait complete. 2024-04-22T15:48:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.218+0000 7f21a9ea8700 1 -- 172.21.15.16:0/2818410004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a40fff50 msgr2=0x7f21a41003d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.218+0000 7f21a9ea8700 1 --2- 172.21.15.16:0/2818410004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a40fff50 0x7f21a41003d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2198009910 tx=0x7f2198009c20 comp rx=0 tx=0).stop 2024-04-22T15:48:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.218+0000 7f21a9ea8700 1 -- 172.21.15.16:0/2818410004 shutdown_connections 2024-04-22T15:48:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.218+0000 7f21a9ea8700 1 --2- 172.21.15.16:0/2818410004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a40fff50 0x7f21a41003d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.218+0000 7f21a9ea8700 1 --2- 172.21.15.16:0/2818410004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21a40ff5f0 0x7f21a40ffa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.218+0000 7f21a9ea8700 1 -- 172.21.15.16:0/2818410004 >> 172.21.15.16:0/2818410004 conn(0x7f21a40fb190 msgr2=0x7f21a40fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.219+0000 7f21a9ea8700 1 -- 172.21.15.16:0/2818410004 shutdown_connections 2024-04-22T15:48:26.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.219+0000 7f21a9ea8700 1 -- 172.21.15.16:0/2818410004 wait complete. 2024-04-22T15:48:26.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.219+0000 7f21a9ea8700 1 Processor -- start 2024-04-22T15:48:26.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.220+0000 7f21a9ea8700 1 -- start start 2024-04-22T15:48:26.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.220+0000 7f21a9ea8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a40ff5f0 0x7f21a4198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:26.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.220+0000 7f21a9ea8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21a40fff50 0x7f21a4198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:26.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.220+0000 7f21a9ea8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a41991d0 con 0x7f21a40ff5f0 2024-04-22T15:48:26.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.220+0000 7f21a9ea8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a4199310 con 0x7f21a40fff50 2024-04-22T15:48:26.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.220+0000 7f21a2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21a40fff50 0x7f21a4198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:26.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.220+0000 7f21a2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21a40fff50 0x7f21a4198bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:52894/0 (socket says 172.21.15.16:52894) 2024-04-22T15:48:26.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.220+0000 7f21a2ffd700 1 -- 172.21.15.16:0/3375093496 learned_addr learned my addr 172.21.15.16:0/3375093496 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:48:26.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.220+0000 7f21a37fe700 1 --2- 172.21.15.16:0/3375093496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a40ff5f0 0x7f21a4198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:26.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.221+0000 7f21a2ffd700 1 -- 172.21.15.16:0/3375093496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a40ff5f0 msgr2=0x7f21a4198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:26.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.221+0000 7f21a2ffd700 1 --2- 172.21.15.16:0/3375093496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a40ff5f0 0x7f21a4198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:26.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.221+0000 7f21a2ffd700 1 -- 172.21.15.16:0/3375093496 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21980095f0 con 0x7f21a40fff50 2024-04-22T15:48:26.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.221+0000 7f21a2ffd700 1 --2- 172.21.15.16:0/3375093496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21a40fff50 0x7f21a4198bb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f219800bd70 tx=0x7f219800bda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:26.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.221+0000 7f21a0ff9700 1 -- 172.21.15.16:0/3375093496 <== mon.1 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f219801d070 con 0x7f21a40fff50 2024-04-22T15:48:26.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.221+0000 7f21a9ea8700 1 -- 172.21.15.16:0/3375093496 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a419dd60 con 0x7f21a40fff50 2024-04-22T15:48:26.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.221+0000 7f21a0ff9700 1 -- 172.21.15.16:0/3375093496 <== mon.1 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2198005dc0 con 0x7f21a40fff50 2024-04-22T15:48:26.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.221+0000 7f21a0ff9700 1 -- 172.21.15.16:0/3375093496 <== mon.1 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2198021410 con 0x7f21a40fff50 2024-04-22T15:48:26.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.221+0000 7f21a9ea8700 1 -- 172.21.15.16:0/3375093496 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a419e200 con 0x7f21a40fff50 2024-04-22T15:48:26.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.222+0000 7f21a9ea8700 1 -- 172.21.15.16:0/3375093496 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a410b9f0 con 0x7f21a40fff50 2024-04-22T15:48:26.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.223+0000 7f21a0ff9700 1 -- 172.21.15.16:0/3375093496 <== mon.1 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f219800f7d0 con 0x7f21a40fff50 2024-04-22T15:48:26.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.223+0000 7f21a0ff9700 1 --2- 172.21.15.16:0/3375093496 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f218c06c4e0 0x7f218c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:26.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.223+0000 7f21a0ff9700 1 -- 172.21.15.16:0/3375093496 <== mon.1 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f219808cbd0 con 0x7f21a40fff50 2024-04-22T15:48:26.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.223+0000 7f21a37fe700 1 --2- 172.21.15.16:0/3375093496 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f218c06c4e0 0x7f218c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:26.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.224+0000 7f21a37fe700 1 --2- 172.21.15.16:0/3375093496 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f218c06c4e0 0x7f218c06e9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2194007be0 tx=0x7f21940073d0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:26.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.232+0000 7f21a0ff9700 1 -- 172.21.15.16:0/3375093496 <== mon.1 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f219805ad80 con 0x7f21a40fff50 2024-04-22T15:48:26.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.519+0000 7f21a9ea8700 1 -- 172.21.15.16:0/3375093496 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f21a410bbe0 con 0x7f21a40fff50 2024-04-22T15:48:26.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.519+0000 7f21a0ff9700 1 -- 172.21.15.16:0/3375093496 <== mon.1 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7f219800f460 con 0x7f21a40fff50 2024-04-22T15:48:26.523 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:48:26.523 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-22T15:48:26.523 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T15:48:26.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:48:26.523 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-22T15:48:26.523 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T15:48:26.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:48:26.523 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-22T15:48:26.524 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-22T15:48:26.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:48:26.524 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-22T15:48:26.524 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T15:48:26.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:48:26.524 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-22T15:48:26.524 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 14 2024-04-22T15:48:26.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-22T15:48:26.524 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:48:26.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.522+0000 7f21a9ea8700 1 -- 172.21.15.16:0/3375093496 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f218c06c4e0 msgr2=0x7f218c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:26.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.522+0000 7f21a9ea8700 1 --2- 172.21.15.16:0/3375093496 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f218c06c4e0 0x7f218c06e9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2194007be0 tx=0x7f21940073d0 comp rx=0 tx=0).stop 2024-04-22T15:48:26.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.522+0000 7f21a9ea8700 1 -- 172.21.15.16:0/3375093496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21a40fff50 msgr2=0x7f21a4198bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:26.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.522+0000 7f21a9ea8700 1 --2- 172.21.15.16:0/3375093496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21a40fff50 0x7f21a4198bb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f219800bd70 tx=0x7f219800bda0 comp rx=0 tx=0).stop 2024-04-22T15:48:26.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.522+0000 7f21a9ea8700 1 -- 172.21.15.16:0/3375093496 shutdown_connections 2024-04-22T15:48:26.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.522+0000 7f21a9ea8700 1 --2- 172.21.15.16:0/3375093496 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f218c06c4e0 0x7f218c06e9a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:26.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.522+0000 7f21a9ea8700 1 --2- 172.21.15.16:0/3375093496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a40ff5f0 0x7f21a4198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:26.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.522+0000 7f21a9ea8700 1 --2- 172.21.15.16:0/3375093496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21a40fff50 0x7f21a4198bb0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:26.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.522+0000 7f21a9ea8700 1 -- 172.21.15.16:0/3375093496 >> 172.21.15.16:0/3375093496 conn(0x7f21a40fb190 msgr2=0x7f21a4107c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:26.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.522+0000 7f21a9ea8700 1 -- 172.21.15.16:0/3375093496 shutdown_connections 2024-04-22T15:48:26.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.522+0000 7f21a9ea8700 1 -- 172.21.15.16:0/3375093496 wait complete. 2024-04-22T15:48:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.633+0000 7f498cade700 1 -- 172.21.15.16:0/1592977196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49881017a0 msgr2=0x7f4988103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.633+0000 7f498cade700 1 --2- 172.21.15.16:0/1592977196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49881017a0 0x7f4988103b90 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4978009910 tx=0x7f4978009c20 comp rx=0 tx=0).stop 2024-04-22T15:48:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.633+0000 7f498cade700 1 -- 172.21.15.16:0/1592977196 shutdown_connections 2024-04-22T15:48:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.633+0000 7f498cade700 1 --2- 172.21.15.16:0/1592977196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49881040d0 0x7f49881064c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.633+0000 7f498cade700 1 --2- 172.21.15.16:0/1592977196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49881017a0 0x7f4988103b90 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.633+0000 7f498cade700 1 -- 172.21.15.16:0/1592977196 >> 172.21.15.16:0/1592977196 conn(0x7f49880fb190 msgr2=0x7f49880fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.634+0000 7f498cade700 1 -- 172.21.15.16:0/1592977196 shutdown_connections 2024-04-22T15:48:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.634+0000 7f498cade700 1 -- 172.21.15.16:0/1592977196 wait complete. 2024-04-22T15:48:26.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.634+0000 7f498cade700 1 Processor -- start 2024-04-22T15:48:26.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.634+0000 7f498cade700 1 -- start start 2024-04-22T15:48:26.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.635+0000 7f498cade700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49881017a0 0x7f4988196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:26.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.635+0000 7f498cade700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49881040d0 0x7f49881969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:26.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.635+0000 7f498cade700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4988196fc0 con 0x7f49881017a0 2024-04-22T15:48:26.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.635+0000 7f498cade700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4988197100 con 0x7f49881040d0 2024-04-22T15:48:26.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.635+0000 7f4985d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49881040d0 0x7f49881969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:26.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.635+0000 7f4985d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49881040d0 0x7f49881969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:52908/0 (socket says 172.21.15.16:52908) 2024-04-22T15:48:26.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.635+0000 7f4985d9b700 1 -- 172.21.15.16:0/1931207711 learned_addr learned my addr 172.21.15.16:0/1931207711 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:48:26.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.635+0000 7f498659c700 1 --2- 172.21.15.16:0/1931207711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49881017a0 0x7f4988196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:26.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.635+0000 7f4985d9b700 1 -- 172.21.15.16:0/1931207711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49881017a0 msgr2=0x7f4988196460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:26.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.635+0000 7f4985d9b700 1 --2- 172.21.15.16:0/1931207711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49881017a0 0x7f4988196460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:26.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.635+0000 7f4985d9b700 1 -- 172.21.15.16:0/1931207711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49780095f0 con 0x7f49881040d0 2024-04-22T15:48:26.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.635+0000 7f498659c700 1 --2- 172.21.15.16:0/1931207711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49881017a0 0x7f4988196460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:48:26.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.636+0000 7f4985d9b700 1 --2- 172.21.15.16:0/1931207711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49881040d0 0x7f49881969a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f497c00d900 tx=0x7f497c00dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:26.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.636+0000 7f49777fe700 1 -- 172.21.15.16:0/1931207711 <== mon.1 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f497c0041f0 con 0x7f49881040d0 2024-04-22T15:48:26.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.636+0000 7f49777fe700 1 -- 172.21.15.16:0/1931207711 <== mon.1 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f497c004ce0 con 0x7f49881040d0 2024-04-22T15:48:26.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.636+0000 7f498cade700 1 -- 172.21.15.16:0/1931207711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49880ff5f0 con 0x7f49881040d0 2024-04-22T15:48:26.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.636+0000 7f49777fe700 1 -- 172.21.15.16:0/1931207711 <== mon.1 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f497c00b750 con 0x7f49881040d0 2024-04-22T15:48:26.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.636+0000 7f498cade700 1 -- 172.21.15.16:0/1931207711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49880ffb40 con 0x7f49881040d0 2024-04-22T15:48:26.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.637+0000 7f498cade700 1 -- 172.21.15.16:0/1931207711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4988190610 con 0x7f49881040d0 2024-04-22T15:48:26.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.638+0000 7f49777fe700 1 -- 172.21.15.16:0/1931207711 <== mon.1 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f497c004350 con 0x7f49881040d0 2024-04-22T15:48:26.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.639+0000 7f49777fe700 1 --2- 172.21.15.16:0/1931207711 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f497006c4e0 0x7f497006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:26.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.639+0000 7f49777fe700 1 -- 172.21.15.16:0/1931207711 <== mon.1 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f497c08b9b0 con 0x7f49881040d0 2024-04-22T15:48:26.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.639+0000 7f498659c700 1 --2- 172.21.15.16:0/1931207711 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f497006c4e0 0x7f497006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:26.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.640+0000 7f498659c700 1 --2- 172.21.15.16:0/1931207711 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f497006c4e0 0x7f497006e9a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4978005b40 tx=0x7f4978005ab0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:26.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.647+0000 7f49777fe700 1 -- 172.21.15.16:0/1931207711 <== mon.1 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f497c059bc0 con 0x7f49881040d0 2024-04-22T15:48:26.779 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:26 smithi027 ceph-mon[34922]: from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:26.779 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:26 smithi027 ceph-mon[34922]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:26.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.865+0000 7f498cade700 1 -- 172.21.15.16:0/1931207711 --> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4988100f50 con 0x7f497006c4e0 2024-04-22T15:48:26.869 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:26 smithi016 ceph-mon[26620]: from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:26.869 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:26 smithi016 ceph-mon[26620]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:26.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.867+0000 7f49777fe700 1 -- 172.21.15.16:0/1931207711 <== mgr.14217 v2:172.21.15.16:6800/1057234399 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f4988100f50 con 0x7f497006c4e0 2024-04-22T15:48:26.870 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:48:26.870 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:48:26.870 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-22T15:48:26.870 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T15:48:26.870 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [], 2024-04-22T15:48:26.870 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "0/2 daemons upgraded", 2024-04-22T15:48:26.871 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb image on host smithi027", 2024-04-22T15:48:26.871 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-22T15:48:26.871 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:48:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.870+0000 7f498cade700 1 -- 172.21.15.16:0/1931207711 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f497006c4e0 msgr2=0x7f497006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.870+0000 7f498cade700 1 --2- 172.21.15.16:0/1931207711 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f497006c4e0 0x7f497006e9a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4978005b40 tx=0x7f4978005ab0 comp rx=0 tx=0).stop 2024-04-22T15:48:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.870+0000 7f498cade700 1 -- 172.21.15.16:0/1931207711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49881040d0 msgr2=0x7f49881969a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.870+0000 7f498cade700 1 --2- 172.21.15.16:0/1931207711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49881040d0 0x7f49881969a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f497c00d900 tx=0x7f497c00dcc0 comp rx=0 tx=0).stop 2024-04-22T15:48:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.870+0000 7f498cade700 1 -- 172.21.15.16:0/1931207711 shutdown_connections 2024-04-22T15:48:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.870+0000 7f498cade700 1 --2- 172.21.15.16:0/1931207711 >> [v2:172.21.15.16:6800/1057234399,v1:172.21.15.16:6801/1057234399] conn(0x7f497006c4e0 0x7f497006e9a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.870+0000 7f498cade700 1 --2- 172.21.15.16:0/1931207711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49881017a0 0x7f4988196460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.870+0000 7f498cade700 1 --2- 172.21.15.16:0/1931207711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49881040d0 0x7f49881969a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.870+0000 7f498cade700 1 -- 172.21.15.16:0/1931207711 >> 172.21.15.16:0/1931207711 conn(0x7f49880fb190 msgr2=0x7f49880fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.871+0000 7f498cade700 1 -- 172.21.15.16:0/1931207711 shutdown_connections 2024-04-22T15:48:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:26.871+0000 7f498cade700 1 -- 172.21.15.16:0/1931207711 wait complete. 2024-04-22T15:48:27.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:27 smithi027 ceph-mon[34922]: from='client.24383 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:27.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:27 smithi027 ceph-mon[34922]: pgmap v237: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:48:27.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:27 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/3375093496' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:48:27.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:27 smithi016 ceph-mon[26620]: from='client.24383 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:27.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:27 smithi016 ceph-mon[26620]: pgmap v237: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:48:27.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:27 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/3375093496' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:48:28.719 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:28 smithi027 ceph-mon[34922]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:28.719 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:28 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:48:28.719 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:28 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi027.ukdgzs", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:48:28.719 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:28 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:48:28.720 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:28 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:48:28.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:28 smithi016 ceph-mon[26620]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:28.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:28 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:48:28.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:28 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi027.ukdgzs", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:48:28.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:28 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:48:28.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:28 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:48:29.749 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:29 smithi027 ceph-mon[34922]: Upgrade: Updating mgr.smithi027.ukdgzs 2024-04-22T15:48:29.749 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:29 smithi027 ceph-mon[34922]: Deploying daemon mgr.smithi027.ukdgzs on smithi027 2024-04-22T15:48:29.750 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:29 smithi027 ceph-mon[34922]: pgmap v238: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:48:29.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:29 smithi016 ceph-mon[26620]: Upgrade: Updating mgr.smithi027.ukdgzs 2024-04-22T15:48:29.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:29 smithi016 ceph-mon[26620]: Deploying daemon mgr.smithi027.ukdgzs on smithi027 2024-04-22T15:48:29.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:29 smithi016 ceph-mon[26620]: pgmap v238: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:48:30.053 INFO:tasks.workunit.client.0.smithi016.stderr:Updating files: 92% (11355/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-04-22T15:48:30.368 INFO:tasks.workunit.client.0.smithi016.stderr:Note: switching to '5883d494d3973b30b3f03e4a3f0e1e06e3c131aa'. 2024-04-22T15:48:30.368 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-22T15:48:30.368 INFO:tasks.workunit.client.0.smithi016.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-22T15:48:30.368 INFO:tasks.workunit.client.0.smithi016.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-22T15:48:30.368 INFO:tasks.workunit.client.0.smithi016.stderr:state without impacting any branches by switching back to a branch. 2024-04-22T15:48:30.368 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-22T15:48:30.368 INFO:tasks.workunit.client.0.smithi016.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-22T15:48:30.369 INFO:tasks.workunit.client.0.smithi016.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-22T15:48:30.369 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-22T15:48:30.369 INFO:tasks.workunit.client.0.smithi016.stderr: git switch -c 2024-04-22T15:48:30.369 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-22T15:48:30.369 INFO:tasks.workunit.client.0.smithi016.stderr:Or undo this operation with: 2024-04-22T15:48:30.369 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-22T15:48:30.369 INFO:tasks.workunit.client.0.smithi016.stderr: git switch - 2024-04-22T15:48:30.369 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-22T15:48:30.369 INFO:tasks.workunit.client.0.smithi016.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-22T15:48:30.369 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-22T15:48:30.369 INFO:tasks.workunit.client.0.smithi016.stderr:HEAD is now at 5883d494d39 Merge branch 'cheroot-upgrade' of https://github.com/rhcs-dashboard/ceph into wip-adk-testing-2024-04-22-0809 2024-04-22T15:48:30.375 DEBUG:teuthology.orchestra.run.smithi016:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-22T15:48:30.441 INFO:tasks.workunit.client.0.smithi016.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-22T15:48:30.443 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-22T15:48:30.443 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-22T15:48:31.147 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-22T15:48:31.203 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-22T15:48:31.243 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-22T15:48:31.245 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-22T15:48:31.245 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-22T15:48:31.284 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-22T15:48:31.287 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-22T15:48:31.288 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-22T15:48:31.359 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-04-22T15:48:31.361 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-22T15:48:31.361 DEBUG:teuthology.orchestra.run.smithi016:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-04-22T15:48:31.425 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir -p fsstress 2024-04-22T15:48:31.427 INFO:tasks.workunit.client.0.smithi016.stderr:+ pushd fsstress 2024-04-22T15:48:31.428 INFO:tasks.workunit.client.0.smithi016.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-22T15:48:31.429 INFO:tasks.workunit.client.0.smithi016.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-22T15:48:31.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:31 smithi027 ceph-mon[34922]: pgmap v239: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:48:31.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:31 smithi016 ceph-mon[26620]: pgmap v239: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:48:31.977 INFO:tasks.workunit.client.0.smithi016.stderr:+ tar xzf ltp-full.tgz 2024-04-22T15:48:32.667 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:32 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:48:32.667 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:32 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:48:32.667 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:32 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:48:32.667 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:32 smithi027 ceph-mon[34922]: pgmap v240: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:48:32.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:32 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:48:32.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:32 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:48:32.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:32 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:48:32.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:32 smithi016 ceph-mon[26620]: pgmap v240: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:48:35.515 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:35 smithi027 ceph-mon[34922]: pgmap v241: 65 pgs: 65 active+clean; 5.1 MiB data, 174 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 390 KiB/s wr, 6 op/s 2024-04-22T15:48:35.515 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:35 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:48:35.515 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:35 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:48:35.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:35 smithi016 ceph-mon[26620]: pgmap v241: 65 pgs: 65 active+clean; 5.1 MiB data, 174 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 390 KiB/s wr, 6 op/s 2024-04-22T15:48:35.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:35 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:48:35.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:35 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:48:36.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:36 smithi027 ceph-mon[34922]: pgmap v242: 65 pgs: 65 active+clean; 14 MiB data, 192 MiB used, 536 GiB / 536 GiB avail; 168 KiB/s rd, 1.2 MiB/s wr, 22 op/s 2024-04-22T15:48:36.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:36 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:48:36.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:36 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:48:37.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:36 smithi016 ceph-mon[26620]: pgmap v242: 65 pgs: 65 active+clean; 14 MiB data, 192 MiB used, 536 GiB / 536 GiB avail; 168 KiB/s rd, 1.2 MiB/s wr, 22 op/s 2024-04-22T15:48:37.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:36 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:48:37.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:36 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:48:39.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:39 smithi027 ceph-mon[34922]: pgmap v243: 65 pgs: 65 active+clean; 15 MiB data, 206 MiB used, 536 GiB / 536 GiB avail; 167 KiB/s rd, 1.2 MiB/s wr, 40 op/s 2024-04-22T15:48:39.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:39 smithi016 ceph-mon[26620]: pgmap v243: 65 pgs: 65 active+clean; 15 MiB data, 206 MiB used, 536 GiB / 536 GiB avail; 167 KiB/s rd, 1.2 MiB/s wr, 40 op/s 2024-04-22T15:48:40.006 INFO:tasks.workunit.client.1.smithi027.stderr:Updating files: 79% (9779/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-04-22T15:48:40.382 INFO:tasks.workunit.client.1.smithi027.stderr:Note: switching to '5883d494d3973b30b3f03e4a3f0e1e06e3c131aa'. 2024-04-22T15:48:40.382 INFO:tasks.workunit.client.1.smithi027.stderr: 2024-04-22T15:48:40.382 INFO:tasks.workunit.client.1.smithi027.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-22T15:48:40.382 INFO:tasks.workunit.client.1.smithi027.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-22T15:48:40.382 INFO:tasks.workunit.client.1.smithi027.stderr:state without impacting any branches by switching back to a branch. 2024-04-22T15:48:40.382 INFO:tasks.workunit.client.1.smithi027.stderr: 2024-04-22T15:48:40.382 INFO:tasks.workunit.client.1.smithi027.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-22T15:48:40.382 INFO:tasks.workunit.client.1.smithi027.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-22T15:48:40.382 INFO:tasks.workunit.client.1.smithi027.stderr: 2024-04-22T15:48:40.383 INFO:tasks.workunit.client.1.smithi027.stderr: git switch -c 2024-04-22T15:48:40.383 INFO:tasks.workunit.client.1.smithi027.stderr: 2024-04-22T15:48:40.383 INFO:tasks.workunit.client.1.smithi027.stderr:Or undo this operation with: 2024-04-22T15:48:40.383 INFO:tasks.workunit.client.1.smithi027.stderr: 2024-04-22T15:48:40.383 INFO:tasks.workunit.client.1.smithi027.stderr: git switch - 2024-04-22T15:48:40.383 INFO:tasks.workunit.client.1.smithi027.stderr: 2024-04-22T15:48:40.383 INFO:tasks.workunit.client.1.smithi027.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-22T15:48:40.383 INFO:tasks.workunit.client.1.smithi027.stderr: 2024-04-22T15:48:40.383 INFO:tasks.workunit.client.1.smithi027.stderr:HEAD is now at 5883d494d39 Merge branch 'cheroot-upgrade' of https://github.com/rhcs-dashboard/ceph into wip-adk-testing-2024-04-22-0809 2024-04-22T15:48:40.388 DEBUG:teuthology.orchestra.run.smithi027:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-04-22T15:48:40.473 INFO:tasks.workunit.client.1.smithi027.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-22T15:48:40.475 INFO:tasks.workunit.client.1.smithi027.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-22T15:48:40.476 INFO:tasks.workunit.client.1.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-22T15:48:40.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:40 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:48:40.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:40 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:48:41.459 INFO:tasks.workunit.client.1.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-22T15:48:41.532 INFO:tasks.workunit.client.1.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-22T15:48:41.590 INFO:tasks.workunit.client.1.smithi027.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-22T15:48:41.593 INFO:tasks.workunit.client.1.smithi027.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-22T15:48:41.593 INFO:tasks.workunit.client.1.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-22T15:48:41.641 INFO:tasks.workunit.client.1.smithi027.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-22T15:48:41.644 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-22T15:48:41.644 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-04-22T15:48:41.661 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-04-22T15:48:41.663 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-22T15:48:41.663 DEBUG:teuthology.orchestra.run.smithi027:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-04-22T15:48:41.730 INFO:tasks.workunit.client.1.smithi027.stderr:+ mkdir -p fsstress 2024-04-22T15:48:41.733 INFO:tasks.workunit.client.1.smithi027.stderr:+ pushd fsstress 2024-04-22T15:48:41.734 INFO:tasks.workunit.client.1.smithi027.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-22T15:48:41.735 INFO:tasks.workunit.client.1.smithi027.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-22T15:48:41.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: pgmap v244: 65 pgs: 65 active+clean; 16 MiB data, 209 MiB used, 536 GiB / 536 GiB avail; 167 KiB/s rd, 1.3 MiB/s wr, 54 op/s 2024-04-22T15:48:41.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: Standby manager daemon smithi027.ukdgzs restarted 2024-04-22T15:48:41.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: Standby manager daemon smithi027.ukdgzs started 2024-04-22T15:48:41.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.ukdgzs/crt"}]: dispatch 2024-04-22T15:48:41.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:48:41.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.ukdgzs/key"}]: dispatch 2024-04-22T15:48:41.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:48:41.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:48:41.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:48:41.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:48:41.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:48:41.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:48:41.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:48:41.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr fail", "who": "smithi016.rnsnnr"}]: dispatch 2024-04-22T15:48:41.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: osdmap e47: 6 total, 6 up, 6 in 2024-04-22T15:48:41.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "mgr fail", "who": "smithi016.rnsnnr"}]': finished 2024-04-22T15:48:41.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: mgrmap e20: smithi027.ukdgzs(active, starting, since 0.00791156s) 2024-04-22T15:48:41.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-04-22T15:48:41.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:48:41.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.pgvphu"}]: dispatch 2024-04-22T15:48:41.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.fkkejm"}]: dispatch 2024-04-22T15:48:41.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.iobjds"}]: dispatch 2024-04-22T15:48:41.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.dgqyes"}]: dispatch 2024-04-22T15:48:41.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mgr metadata", "who": "smithi027.ukdgzs", "id": "smithi027.ukdgzs"}]: dispatch 2024-04-22T15:48:41.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:48:41.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:48:41.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:48:41.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:48:41.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:48:41.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:48:41.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T15:48:41.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T15:48:41.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T15:48:41.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: Manager daemon smithi027.ukdgzs is now available 2024-04-22T15:48:41.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:48:41.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:48:41.840 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.ukdgzs/mirror_snapshot_schedule"}]: dispatch 2024-04-22T15:48:41.840 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.ukdgzs/mirror_snapshot_schedule"}]: dispatch 2024-04-22T15:48:41.840 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.ukdgzs/trash_purge_schedule"}]: dispatch 2024-04-22T15:48:41.840 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:41 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.ukdgzs/trash_purge_schedule"}]: dispatch 2024-04-22T15:48:41.857 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: pgmap v244: 65 pgs: 65 active+clean; 16 MiB data, 209 MiB used, 536 GiB / 536 GiB avail; 167 KiB/s rd, 1.3 MiB/s wr, 54 op/s 2024-04-22T15:48:41.857 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: Standby manager daemon smithi027.ukdgzs restarted 2024-04-22T15:48:41.857 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: Standby manager daemon smithi027.ukdgzs started 2024-04-22T15:48:41.857 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.? 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.ukdgzs/crt"}]: dispatch 2024-04-22T15:48:41.857 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.? 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:48:41.857 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.? 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.ukdgzs/key"}]: dispatch 2024-04-22T15:48:41.857 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.? 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:48:41.857 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:48:41.857 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:48:41.858 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:48:41.858 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:48:41.858 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:48:41.858 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:48:41.858 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr fail", "who": "smithi016.rnsnnr"}]: dispatch 2024-04-22T15:48:41.858 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: osdmap e47: 6 total, 6 up, 6 in 2024-04-22T15:48:41.858 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.14217 172.21.15.16:0/2657560234' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "mgr fail", "who": "smithi016.rnsnnr"}]': finished 2024-04-22T15:48:41.858 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: mgrmap e20: smithi027.ukdgzs(active, starting, since 0.00791156s) 2024-04-22T15:48:41.858 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-04-22T15:48:41.858 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:48:41.858 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.pgvphu"}]: dispatch 2024-04-22T15:48:41.858 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.fkkejm"}]: dispatch 2024-04-22T15:48:41.858 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.iobjds"}]: dispatch 2024-04-22T15:48:41.859 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.dgqyes"}]: dispatch 2024-04-22T15:48:41.859 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mgr metadata", "who": "smithi027.ukdgzs", "id": "smithi027.ukdgzs"}]: dispatch 2024-04-22T15:48:41.859 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:48:41.859 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:48:41.859 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:48:41.859 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:48:41.859 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:48:41.859 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:48:41.859 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T15:48:41.859 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T15:48:41.859 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T15:48:41.859 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: Manager daemon smithi027.ukdgzs is now available 2024-04-22T15:48:41.859 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:48:41.859 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:48:41.860 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.ukdgzs/mirror_snapshot_schedule"}]: dispatch 2024-04-22T15:48:41.860 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.ukdgzs/mirror_snapshot_schedule"}]: dispatch 2024-04-22T15:48:41.860 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.ukdgzs/trash_purge_schedule"}]: dispatch 2024-04-22T15:48:41.860 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:41 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.ukdgzs/trash_purge_schedule"}]: dispatch 2024-04-22T15:48:42.836 INFO:tasks.workunit.client.1.smithi027.stderr:+ tar xzf ltp-full.tgz 2024-04-22T15:48:43.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:43 smithi016 ceph-mon[26620]: Deploying cephadm binary to smithi027 2024-04-22T15:48:43.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:43 smithi016 ceph-mon[26620]: Deploying cephadm binary to smithi016 2024-04-22T15:48:43.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:43 smithi016 ceph-mon[26620]: mgrmap e21: smithi027.ukdgzs(active, since 1.00909s) 2024-04-22T15:48:43.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:43 smithi016 ceph-mon[26620]: pgmap v3: 65 pgs: 65 active+clean; 26 MiB data, 240 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:48:43.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:43 smithi027 ceph-mon[34922]: Deploying cephadm binary to smithi027 2024-04-22T15:48:43.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:43 smithi027 ceph-mon[34922]: Deploying cephadm binary to smithi016 2024-04-22T15:48:43.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:43 smithi027 ceph-mon[34922]: mgrmap e21: smithi027.ukdgzs(active, since 1.00909s) 2024-04-22T15:48:43.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:43 smithi027 ceph-mon[34922]: pgmap v3: 65 pgs: 65 active+clean; 26 MiB data, 240 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:48:44.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:44 smithi016 ceph-mon[26620]: [22/Apr/2024:15:48:43] ENGINE Bus STARTING 2024-04-22T15:48:44.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:44 smithi016 ceph-mon[26620]: mgrmap e22: smithi027.ukdgzs(active, since 2s) 2024-04-22T15:48:44.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:44 smithi016 ceph-mon[26620]: pgmap v4: 65 pgs: 65 active+clean; 26 MiB data, 240 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:48:44.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:44 smithi027 ceph-mon[34922]: [22/Apr/2024:15:48:43] ENGINE Bus STARTING 2024-04-22T15:48:44.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:44 smithi027 ceph-mon[34922]: mgrmap e22: smithi027.ukdgzs(active, since 2s) 2024-04-22T15:48:44.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:44 smithi027 ceph-mon[34922]: pgmap v4: 65 pgs: 65 active+clean; 26 MiB data, 240 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:48:45.579 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:45 smithi016 ceph-mon[26620]: [22/Apr/2024:15:48:43] ENGINE Serving on https://172.21.15.27:7150 2024-04-22T15:48:45.579 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:45 smithi016 ceph-mon[26620]: [22/Apr/2024:15:48:43] ENGINE Serving on http://172.21.15.27:8765 2024-04-22T15:48:45.579 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:45 smithi016 ceph-mon[26620]: [22/Apr/2024:15:48:43] ENGINE Bus STARTED 2024-04-22T15:48:45.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:45 smithi027 ceph-mon[34922]: [22/Apr/2024:15:48:43] ENGINE Serving on https://172.21.15.27:7150 2024-04-22T15:48:45.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:45 smithi027 ceph-mon[34922]: [22/Apr/2024:15:48:43] ENGINE Serving on http://172.21.15.27:8765 2024-04-22T15:48:45.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:45 smithi027 ceph-mon[34922]: [22/Apr/2024:15:48:43] ENGINE Bus STARTED 2024-04-22T15:48:46.574 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:46 smithi016 ceph-mon[26620]: pgmap v5: 65 pgs: 65 active+clean; 26 MiB data, 240 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:48:46.574 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:46 smithi016 ceph-mon[26620]: mgrmap e23: smithi027.ukdgzs(active, since 4s) 2024-04-22T15:48:46.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:46 smithi027 ceph-mon[34922]: pgmap v5: 65 pgs: 65 active+clean; 26 MiB data, 240 MiB used, 536 GiB / 536 GiB avail 2024-04-22T15:48:46.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:46 smithi027 ceph-mon[34922]: mgrmap e23: smithi027.ukdgzs(active, since 4s) 2024-04-22T15:48:48.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:48 smithi016 ceph-mon[26620]: pgmap v6: 65 pgs: 65 active+clean; 35 MiB data, 296 MiB used, 536 GiB / 536 GiB avail; 821 KiB/s rd, 1.7 MiB/s wr, 87 op/s 2024-04-22T15:48:48.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:48 smithi027 ceph-mon[34922]: pgmap v6: 65 pgs: 65 active+clean; 35 MiB data, 296 MiB used, 536 GiB / 536 GiB avail; 821 KiB/s rd, 1.7 MiB/s wr, 87 op/s 2024-04-22T15:48:49.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:49 smithi016 ceph-mon[26620]: Standby manager daemon smithi016.rnsnnr started 2024-04-22T15:48:49.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:49 smithi016 ceph-mon[26620]: from='mgr.? 172.21.15.16:0/2207896215' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi016.rnsnnr/crt"}]: dispatch 2024-04-22T15:48:49.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:49 smithi016 ceph-mon[26620]: from='mgr.? 172.21.15.16:0/2207896215' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:48:49.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:49 smithi016 ceph-mon[26620]: from='mgr.? 172.21.15.16:0/2207896215' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi016.rnsnnr/key"}]: dispatch 2024-04-22T15:48:49.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:49 smithi016 ceph-mon[26620]: from='mgr.? 172.21.15.16:0/2207896215' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:48:49.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:49 smithi027 ceph-mon[34922]: Standby manager daemon smithi016.rnsnnr started 2024-04-22T15:48:49.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:49 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.16:0/2207896215' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi016.rnsnnr/crt"}]: dispatch 2024-04-22T15:48:49.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:49 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.16:0/2207896215' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:48:49.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:49 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.16:0/2207896215' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi016.rnsnnr/key"}]: dispatch 2024-04-22T15:48:49.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:49 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.16:0/2207896215' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:48:50.641 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:50 smithi016 ceph-mon[26620]: pgmap v7: 65 pgs: 65 active+clean; 59 MiB data, 361 MiB used, 536 GiB / 536 GiB avail; 1.6 MiB/s rd, 4.8 MiB/s wr, 182 op/s 2024-04-22T15:48:50.642 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:50 smithi016 ceph-mon[26620]: mgrmap e24: smithi027.ukdgzs(active, since 8s), standbys: smithi016.rnsnnr 2024-04-22T15:48:50.642 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:50 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mgr metadata", "who": "smithi016.rnsnnr", "id": "smithi016.rnsnnr"}]: dispatch 2024-04-22T15:48:50.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:50 smithi027 ceph-mon[34922]: pgmap v7: 65 pgs: 65 active+clean; 59 MiB data, 361 MiB used, 536 GiB / 536 GiB avail; 1.6 MiB/s rd, 4.8 MiB/s wr, 182 op/s 2024-04-22T15:48:50.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:50 smithi027 ceph-mon[34922]: mgrmap e24: smithi027.ukdgzs(active, since 8s), standbys: smithi016.rnsnnr 2024-04-22T15:48:50.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:50 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mgr metadata", "who": "smithi016.rnsnnr", "id": "smithi016.rnsnnr"}]: dispatch 2024-04-22T15:48:52.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:51 smithi027 ceph-mon[34922]: pgmap v8: 65 pgs: 65 active+clean; 59 MiB data, 361 MiB used, 536 GiB / 536 GiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 141 op/s 2024-04-22T15:48:52.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:51 smithi016 ceph-mon[26620]: pgmap v8: 65 pgs: 65 active+clean; 59 MiB data, 361 MiB used, 536 GiB / 536 GiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 141 op/s 2024-04-22T15:48:54.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:54 smithi016 ceph-mon[26620]: pgmap v9: 65 pgs: 65 active+clean; 72 MiB data, 406 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 4.3 MiB/s wr, 165 op/s 2024-04-22T15:48:54.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:54 smithi027 ceph-mon[34922]: pgmap v9: 65 pgs: 65 active+clean; 72 MiB data, 406 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 4.3 MiB/s wr, 165 op/s 2024-04-22T15:48:56.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:56 smithi016 ceph-mon[26620]: pgmap v10: 65 pgs: 65 active+clean; 72 MiB data, 406 MiB used, 536 GiB / 536 GiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 151 op/s 2024-04-22T15:48:56.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:56 smithi027 ceph-mon[34922]: pgmap v10: 65 pgs: 65 active+clean; 72 MiB data, 406 MiB used, 536 GiB / 536 GiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 151 op/s 2024-04-22T15:48:56.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.991+0000 7f2b00746700 1 -- 172.21.15.16:0/2035526241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af8100cb0 msgr2=0x7f2af81010d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:56.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.991+0000 7f2b00746700 1 --2- 172.21.15.16:0/2035526241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af8100cb0 0x7f2af81010d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2af4009910 tx=0x7f2af4009c20 comp rx=0 tx=0).stop 2024-04-22T15:48:56.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.991+0000 7f2b00746700 1 -- 172.21.15.16:0/2035526241 shutdown_connections 2024-04-22T15:48:56.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.991+0000 7f2b00746700 1 --2- 172.21.15.16:0/2035526241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2af8101eb0 0x7f2af8102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:56.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.991+0000 7f2b00746700 1 --2- 172.21.15.16:0/2035526241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af8100cb0 0x7f2af81010d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:56.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.991+0000 7f2b00746700 1 -- 172.21.15.16:0/2035526241 >> 172.21.15.16:0/2035526241 conn(0x7f2af80fc230 msgr2=0x7f2af80fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:56.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.991+0000 7f2b00746700 1 -- 172.21.15.16:0/2035526241 shutdown_connections 2024-04-22T15:48:56.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.991+0000 7f2b00746700 1 -- 172.21.15.16:0/2035526241 wait complete. 2024-04-22T15:48:56.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.992+0000 7f2b00746700 1 Processor -- start 2024-04-22T15:48:56.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.992+0000 7f2b00746700 1 -- start start 2024-04-22T15:48:56.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.992+0000 7f2b00746700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2af8100cb0 0x7f2af81964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:56.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.992+0000 7f2b00746700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af8101eb0 0x7f2af8196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:56.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.992+0000 7f2b00746700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af8197050 con 0x7f2af8101eb0 2024-04-22T15:48:56.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.992+0000 7f2b00746700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af8197190 con 0x7f2af8100cb0 2024-04-22T15:48:56.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.992+0000 7f2afdce1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af8101eb0 0x7f2af8196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:56.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.992+0000 7f2afdce1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af8101eb0 0x7f2af8196a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50872/0 (socket says 172.21.15.16:50872) 2024-04-22T15:48:56.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.992+0000 7f2afdce1700 1 -- 172.21.15.16:0/3984852001 learned_addr learned my addr 172.21.15.16:0/3984852001 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:48:56.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.992+0000 7f2afe4e2700 1 --2- 172.21.15.16:0/3984852001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2af8100cb0 0x7f2af81964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:56.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.992+0000 7f2afdce1700 1 -- 172.21.15.16:0/3984852001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2af8100cb0 msgr2=0x7f2af81964f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:56.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.992+0000 7f2afdce1700 1 --2- 172.21.15.16:0/3984852001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2af8100cb0 0x7f2af81964f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:56.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.992+0000 7f2afdce1700 1 -- 172.21.15.16:0/3984852001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af40095f0 con 0x7f2af8101eb0 2024-04-22T15:48:56.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.993+0000 7f2afdce1700 1 --2- 172.21.15.16:0/3984852001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af8101eb0 0x7f2af8196a30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2ae800e920 tx=0x7f2ae800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:56.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.993+0000 7f2aef7fe700 1 -- 172.21.15.16:0/3984852001 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ae800c9d0 con 0x7f2af8101eb0 2024-04-22T15:48:56.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.993+0000 7f2aef7fe700 1 -- 172.21.15.16:0/3984852001 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2ae8010430 con 0x7f2af8101eb0 2024-04-22T15:48:56.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.993+0000 7f2b00746700 1 -- 172.21.15.16:0/3984852001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2af819bc40 con 0x7f2af8101eb0 2024-04-22T15:48:56.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.993+0000 7f2aef7fe700 1 -- 172.21.15.16:0/3984852001 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ae8018650 con 0x7f2af8101eb0 2024-04-22T15:48:56.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.993+0000 7f2b00746700 1 -- 172.21.15.16:0/3984852001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2af819c190 con 0x7f2af8101eb0 2024-04-22T15:48:56.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.994+0000 7f2b00746700 1 -- 172.21.15.16:0/3984852001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2af8109810 con 0x7f2af8101eb0 2024-04-22T15:48:56.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.994+0000 7f2aef7fe700 1 -- 172.21.15.16:0/3984852001 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) v1 ==== 91557+0+0 (secure 0 0 0) 0x7f2ae80105a0 con 0x7f2af8101eb0 2024-04-22T15:48:56.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.995+0000 7f2aef7fe700 1 --2- 172.21.15.16:0/3984852001 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f2ae406d850 0x7f2ae406fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:56.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.995+0000 7f2aef7fe700 1 -- 172.21.15.16:0/3984852001 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f2ae8014070 con 0x7f2af8101eb0 2024-04-22T15:48:56.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.995+0000 7f2afe4e2700 1 --2- 172.21.15.16:0/3984852001 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f2ae406d850 0x7f2ae406fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:56.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:56.996+0000 7f2afe4e2700 1 --2- 172.21.15.16:0/3984852001 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f2ae406d850 0x7f2ae406fd10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2af400b6f0 tx=0x7f2af4005940 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:57.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.000+0000 7f2aef7fe700 1 -- 172.21.15.16:0/3984852001 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f2ae8053e20 con 0x7f2af8101eb0 2024-04-22T15:48:57.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.292+0000 7f2b00746700 1 -- 172.21.15.16:0/3984852001 --> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2af8106800 con 0x7f2ae406d850 2024-04-22T15:48:57.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.293+0000 7f2aef7fe700 1 -- 172.21.15.16:0/3984852001 <== mgr.24395 v2:172.21.15.27:6828/2676414545 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f2af8106800 con 0x7f2ae406d850 2024-04-22T15:48:57.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.297+0000 7f2b00746700 1 -- 172.21.15.16:0/3984852001 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f2ae406d850 msgr2=0x7f2ae406fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:57.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.297+0000 7f2b00746700 1 --2- 172.21.15.16:0/3984852001 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f2ae406d850 0x7f2ae406fd10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2af400b6f0 tx=0x7f2af4005940 comp rx=0 tx=0).stop 2024-04-22T15:48:57.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.297+0000 7f2b00746700 1 -- 172.21.15.16:0/3984852001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af8101eb0 msgr2=0x7f2af8196a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:57.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.297+0000 7f2b00746700 1 --2- 172.21.15.16:0/3984852001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af8101eb0 0x7f2af8196a30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2ae800e920 tx=0x7f2ae800ec30 comp rx=0 tx=0).stop 2024-04-22T15:48:57.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.297+0000 7f2b00746700 1 -- 172.21.15.16:0/3984852001 shutdown_connections 2024-04-22T15:48:57.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.297+0000 7f2b00746700 1 --2- 172.21.15.16:0/3984852001 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f2ae406d850 0x7f2ae406fd10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:57.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.297+0000 7f2b00746700 1 --2- 172.21.15.16:0/3984852001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2af8100cb0 0x7f2af81964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:57.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.297+0000 7f2b00746700 1 --2- 172.21.15.16:0/3984852001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af8101eb0 0x7f2af8196a30 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:57.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.297+0000 7f2b00746700 1 -- 172.21.15.16:0/3984852001 >> 172.21.15.16:0/3984852001 conn(0x7f2af80fc230 msgr2=0x7f2af81050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:57.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.297+0000 7f2b00746700 1 -- 172.21.15.16:0/3984852001 shutdown_connections 2024-04-22T15:48:57.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.297+0000 7f2b00746700 1 -- 172.21.15.16:0/3984852001 wait complete. 2024-04-22T15:48:57.313 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:48:57.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.410+0000 7f5709a1a700 1 -- 172.21.15.16:0/2482482128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57040fef60 msgr2=0x7f57040ff380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:57.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.410+0000 7f5709a1a700 1 --2- 172.21.15.16:0/2482482128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57040fef60 0x7f57040ff380 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f56f4009910 tx=0x7f56f4009c20 comp rx=0 tx=0).stop 2024-04-22T15:48:57.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.411+0000 7f5709a1a700 1 -- 172.21.15.16:0/2482482128 shutdown_connections 2024-04-22T15:48:57.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.411+0000 7f5709a1a700 1 --2- 172.21.15.16:0/2482482128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5704101f40 0x7f57041023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:57.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.411+0000 7f5709a1a700 1 --2- 172.21.15.16:0/2482482128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57040fef60 0x7f57040ff380 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:57.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.411+0000 7f5709a1a700 1 -- 172.21.15.16:0/2482482128 >> 172.21.15.16:0/2482482128 conn(0x7f57040fa8c0 msgr2=0x7f57040fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:57.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.411+0000 7f5709a1a700 1 -- 172.21.15.16:0/2482482128 shutdown_connections 2024-04-22T15:48:57.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.411+0000 7f5709a1a700 1 -- 172.21.15.16:0/2482482128 wait complete. 2024-04-22T15:48:57.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.411+0000 7f5709a1a700 1 Processor -- start 2024-04-22T15:48:57.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.411+0000 7f5709a1a700 1 -- start start 2024-04-22T15:48:57.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.412+0000 7f5709a1a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57040fef60 0x7f57041964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:57.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.412+0000 7f5709a1a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5704101f40 0x7f57041969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:57.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.412+0000 7f5709a1a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5704197010 con 0x7f57040fef60 2024-04-22T15:48:57.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.412+0000 7f5709a1a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5704197150 con 0x7f5704101f40 2024-04-22T15:48:57.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.412+0000 7f5702ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57040fef60 0x7f57041964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:57.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.412+0000 7f5702ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57040fef60 0x7f57041964b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50896/0 (socket says 172.21.15.16:50896) 2024-04-22T15:48:57.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.412+0000 7f5702ffd700 1 -- 172.21.15.16:0/3564052078 learned_addr learned my addr 172.21.15.16:0/3564052078 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:48:57.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.412+0000 7f57027fc700 1 --2- 172.21.15.16:0/3564052078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5704101f40 0x7f57041969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:57.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.412+0000 7f5702ffd700 1 -- 172.21.15.16:0/3564052078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5704101f40 msgr2=0x7f57041969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:57.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.412+0000 7f5702ffd700 1 --2- 172.21.15.16:0/3564052078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5704101f40 0x7f57041969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:57.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.412+0000 7f5702ffd700 1 -- 172.21.15.16:0/3564052078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f40095f0 con 0x7f57040fef60 2024-04-22T15:48:57.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.412+0000 7f5702ffd700 1 --2- 172.21.15.16:0/3564052078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57040fef60 0x7f57041964b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f56f4000c00 tx=0x7f56f400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:57.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.414+0000 7f5708a18700 1 -- 172.21.15.16:0/3564052078 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56f401d070 con 0x7f57040fef60 2024-04-22T15:48:57.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.414+0000 7f5708a18700 1 -- 172.21.15.16:0/3564052078 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f56f4005dc0 con 0x7f57040fef60 2024-04-22T15:48:57.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.414+0000 7f5709a1a700 1 -- 172.21.15.16:0/3564052078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f570419bba0 con 0x7f57040fef60 2024-04-22T15:48:57.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.414+0000 7f5708a18700 1 -- 172.21.15.16:0/3564052078 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56f4022470 con 0x7f57040fef60 2024-04-22T15:48:57.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.414+0000 7f5709a1a700 1 -- 172.21.15.16:0/3564052078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f570419c090 con 0x7f57040fef60 2024-04-22T15:48:57.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.415+0000 7f5709a1a700 1 -- 172.21.15.16:0/3564052078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5704109800 con 0x7f57040fef60 2024-04-22T15:48:57.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.415+0000 7f5708a18700 1 -- 172.21.15.16:0/3564052078 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) v1 ==== 91557+0+0 (secure 0 0 0) 0x7f56f400fa70 con 0x7f57040fef60 2024-04-22T15:48:57.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.416+0000 7f5708a18700 1 --2- 172.21.15.16:0/3564052078 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f56ec06d850 0x7f56ec06fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:57.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.416+0000 7f5708a18700 1 -- 172.21.15.16:0/3564052078 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f56f408e840 con 0x7f57040fef60 2024-04-22T15:48:57.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.416+0000 7f57027fc700 1 --2- 172.21.15.16:0/3564052078 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f56ec06d850 0x7f56ec06fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:57.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.416+0000 7f57027fc700 1 --2- 172.21.15.16:0/3564052078 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f56ec06d850 0x7f56ec06fd10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f56f8009c60 tx=0x7f56f8009260 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:57.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.421+0000 7f5708a18700 1 -- 172.21.15.16:0/3564052078 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f56f405b530 con 0x7f57040fef60 2024-04-22T15:48:57.600 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:57 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:48:57.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.690+0000 7f5709a1a700 1 -- 172.21.15.16:0/3564052078 --> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5704106890 con 0x7f56ec06d850 2024-04-22T15:48:57.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.692+0000 7f5708a18700 1 -- 172.21.15.16:0/3564052078 <== mgr.24395 v2:172.21.15.27:6828/2676414545 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f5704106890 con 0x7f56ec06d850 2024-04-22T15:48:57.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.694+0000 7f5709a1a700 1 -- 172.21.15.16:0/3564052078 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f56ec06d850 msgr2=0x7f56ec06fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:57.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.694+0000 7f5709a1a700 1 --2- 172.21.15.16:0/3564052078 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f56ec06d850 0x7f56ec06fd10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f56f8009c60 tx=0x7f56f8009260 comp rx=0 tx=0).stop 2024-04-22T15:48:57.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.694+0000 7f5709a1a700 1 -- 172.21.15.16:0/3564052078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57040fef60 msgr2=0x7f57041964b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:57.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.694+0000 7f5709a1a700 1 --2- 172.21.15.16:0/3564052078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57040fef60 0x7f57041964b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f56f4000c00 tx=0x7f56f400f690 comp rx=0 tx=0).stop 2024-04-22T15:48:57.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.694+0000 7f5709a1a700 1 -- 172.21.15.16:0/3564052078 shutdown_connections 2024-04-22T15:48:57.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.694+0000 7f5709a1a700 1 --2- 172.21.15.16:0/3564052078 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f56ec06d850 0x7f56ec06fd10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:57.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.694+0000 7f5709a1a700 1 --2- 172.21.15.16:0/3564052078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57040fef60 0x7f57041964b0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:57.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.694+0000 7f5709a1a700 1 --2- 172.21.15.16:0/3564052078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5704101f40 0x7f57041969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:57.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.694+0000 7f5709a1a700 1 -- 172.21.15.16:0/3564052078 >> 172.21.15.16:0/3564052078 conn(0x7f57040fa8c0 msgr2=0x7f5704105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:57.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.694+0000 7f5709a1a700 1 -- 172.21.15.16:0/3564052078 shutdown_connections 2024-04-22T15:48:57.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.695+0000 7f5709a1a700 1 -- 172.21.15.16:0/3564052078 wait complete. 2024-04-22T15:48:57.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.832+0000 7fd4782d3700 1 -- 172.21.15.16:0/616778405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd470073770 msgr2=0x7fd470073bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:57.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.832+0000 7fd4782d3700 1 --2- 172.21.15.16:0/616778405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd470073770 0x7fd470073bf0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd460009910 tx=0x7fd460009c20 comp rx=0 tx=0).stop 2024-04-22T15:48:57.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.833+0000 7fd4782d3700 1 -- 172.21.15.16:0/616778405 shutdown_connections 2024-04-22T15:48:57.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.833+0000 7fd4782d3700 1 --2- 172.21.15.16:0/616778405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd470073770 0x7fd470073bf0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:57.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.833+0000 7fd4782d3700 1 --2- 172.21.15.16:0/616778405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd470074d40 0x7fd4700731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:57.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.833+0000 7fd4782d3700 1 -- 172.21.15.16:0/616778405 >> 172.21.15.16:0/616778405 conn(0x7fd4700fc230 msgr2=0x7fd4700fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:57.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.833+0000 7fd4782d3700 1 -- 172.21.15.16:0/616778405 shutdown_connections 2024-04-22T15:48:57.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.833+0000 7fd4782d3700 1 -- 172.21.15.16:0/616778405 wait complete. 2024-04-22T15:48:57.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.833+0000 7fd4782d3700 1 Processor -- start 2024-04-22T15:48:57.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.833+0000 7fd4782d3700 1 -- start start 2024-04-22T15:48:57.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.833+0000 7fd4782d3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd470073770 0x7fd470198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:57.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.833+0000 7fd4782d3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd470074d40 0x7fd470198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:57.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.833+0000 7fd4782d3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4701991d0 con 0x7fd470074d40 2024-04-22T15:48:57.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.833+0000 7fd4782d3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd470199310 con 0x7fd470073770 2024-04-22T15:48:57.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.834+0000 7fd47606f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd470073770 0x7fd470198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:57.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.834+0000 7fd47586e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd470074d40 0x7fd470198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:57.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.834+0000 7fd47606f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd470073770 0x7fd470198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:40814/0 (socket says 172.21.15.16:40814) 2024-04-22T15:48:57.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.834+0000 7fd47606f700 1 -- 172.21.15.16:0/4030634547 learned_addr learned my addr 172.21.15.16:0/4030634547 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:48:57.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.834+0000 7fd47586e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd470074d40 0x7fd470198bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50902/0 (socket says 172.21.15.16:50902) 2024-04-22T15:48:57.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:57 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:48:57.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.834+0000 7fd47586e700 1 -- 172.21.15.16:0/4030634547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd470073770 msgr2=0x7fd470198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:57.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.834+0000 7fd47586e700 1 --2- 172.21.15.16:0/4030634547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd470073770 0x7fd470198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:57.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.834+0000 7fd47586e700 1 -- 172.21.15.16:0/4030634547 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4600095f0 con 0x7fd470074d40 2024-04-22T15:48:57.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.834+0000 7fd47586e700 1 --2- 172.21.15.16:0/4030634547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd470074d40 0x7fd470198bb0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd460005b40 tx=0x7fd46000fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:57.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.834+0000 7fd4677fe700 1 -- 172.21.15.16:0/4030634547 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd46001d070 con 0x7fd470074d40 2024-04-22T15:48:57.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.834+0000 7fd4677fe700 1 -- 172.21.15.16:0/4030634547 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd460005dc0 con 0x7fd470074d40 2024-04-22T15:48:57.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.834+0000 7fd4677fe700 1 -- 172.21.15.16:0/4030634547 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd460022470 con 0x7fd470074d40 2024-04-22T15:48:57.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.834+0000 7fd4782d3700 1 -- 172.21.15.16:0/4030634547 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd47019dd60 con 0x7fd470074d40 2024-04-22T15:48:57.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.834+0000 7fd4782d3700 1 -- 172.21.15.16:0/4030634547 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd47019e250 con 0x7fd470074d40 2024-04-22T15:48:57.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.835+0000 7fd4782d3700 1 -- 172.21.15.16:0/4030634547 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd47010ba00 con 0x7fd470074d40 2024-04-22T15:48:57.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.836+0000 7fd4677fe700 1 -- 172.21.15.16:0/4030634547 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) v1 ==== 91557+0+0 (secure 0 0 0) 0x7fd460005160 con 0x7fd470074d40 2024-04-22T15:48:57.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.836+0000 7fd4677fe700 1 --2- 172.21.15.16:0/4030634547 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7fd45c06d840 0x7fd45c06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:57.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.836+0000 7fd4677fe700 1 -- 172.21.15.16:0/4030634547 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fd46008e700 con 0x7fd470074d40 2024-04-22T15:48:57.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.837+0000 7fd47606f700 1 --2- 172.21.15.16:0/4030634547 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7fd45c06d840 0x7fd45c06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:57.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.837+0000 7fd47606f700 1 --2- 172.21.15.16:0/4030634547 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7fd45c06d840 0x7fd45c06fd00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd46c009d20 tx=0x7fd46c009260 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:57.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:57.841+0000 7fd4677fe700 1 -- 172.21.15.16:0/4030634547 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7fd46005b3d0 con 0x7fd470074d40 2024-04-22T15:48:58.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.085+0000 7fd4782d3700 1 -- 172.21.15.16:0/4030634547 --> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd470108230 con 0x7fd45c06d840 2024-04-22T15:48:58.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.100+0000 7fd4677fe700 1 -- 172.21.15.16:0/4030634547 <== mgr.24395 v2:172.21.15.27:6828/2676414545 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fd470108230 con 0x7fd45c06d840 2024-04-22T15:48:58.103 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:48:58.103 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.smithi016 smithi016 *:9093,9094 running (6m) 2m ago 7m 24.9M - 0.25.0 c8568f914cd2 fab40987f1b0 2024-04-22T15:48:58.103 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi016 smithi016 running (8m) 2m ago 8m 7461k - 18.2.1 5be31c24972a f8b627d3f41e 2024-04-22T15:48:58.103 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi027 smithi027 running (7m) 22s ago 7m 7583k - 18.2.1 5be31c24972a f00fd40b4220 2024-04-22T15:48:58.103 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi016 smithi016 running (8m) 2m ago 8m 6656k - 18.2.1 5be31c24972a 6b93db1fa0d9 2024-04-22T15:48:58.103 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi027 smithi027 running (7m) 22s ago 7m 6665k - 18.2.1 5be31c24972a d3bd439f30a0 2024-04-22T15:48:58.103 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.smithi016 smithi016 *:3000 running (6m) 2m ago 7m 86.2M - 9.4.7 954c08fa6188 a85bdbf5073d 2024-04-22T15:48:58.104 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.fkkejm smithi016 running (2m) 2m ago 2m 11.4M - 18.2.1 5be31c24972a 4153ac94cea0 2024-04-22T15:48:58.104 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.pgvphu smithi016 running (2m) 2m ago 2m 14.4M - 18.2.1 5be31c24972a 93dc072068a2 2024-04-22T15:48:58.104 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.dgqyes smithi027 running (2m) 22s ago 2m 15.4M - 18.2.1 5be31c24972a 41a9220430fd 2024-04-22T15:48:58.104 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.iobjds smithi027 running (2m) 22s ago 2m 24.7M - 18.2.1 5be31c24972a c1c8d6770f02 2024-04-22T15:48:58.104 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi016.rnsnnr smithi016 *:9283,8765,8443 running (9m) 2m ago 9m 488M - 18.2.1 5be31c24972a 1cf5a7efea72 2024-04-22T15:48:58.104 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi027.ukdgzs smithi027 *:8443,9283,8765 running (27s) 22s ago 6m 42.1M - 19.0.0-3223-g5883d494 3d983c26e278 8dfb23a29706 2024-04-22T15:48:58.104 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi016 smithi016 running (9m) 2m ago 9m 47.9M 2048M 18.2.1 5be31c24972a 72af40873cb6 2024-04-22T15:48:58.104 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi027 smithi027 running (6m) 22s ago 6m 35.7M 2048M 18.2.1 5be31c24972a 9f21ce7fa7ef 2024-04-22T15:48:58.104 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi016 smithi016 *:9100 running (8m) 2m ago 8m 15.7M - 1.5.0 0da6a335fe13 1937a0bc1959 2024-04-22T15:48:58.104 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi027 smithi027 *:9100 running (6m) 22s ago 6m 15.7M - 1.5.0 0da6a335fe13 00440c991b3f 2024-04-22T15:48:58.104 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (6m) 2m ago 6m 48.4M 3948M 18.2.1 5be31c24972a 3c6ebc4ea041 2024-04-22T15:48:58.104 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (5m) 2m ago 5m 47.8M 3948M 18.2.1 5be31c24972a 6aa2e6796784 2024-04-22T15:48:58.104 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (5m) 2m ago 5m 45.9M 3948M 18.2.1 5be31c24972a 3d9d310aa9eb 2024-04-22T15:48:58.104 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi027 running (4m) 22s ago 4m 58.2M 4970M 18.2.1 5be31c24972a cdf9b7aaf865 2024-04-22T15:48:58.105 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (4m) 22s ago 4m 44.8M 4970M 18.2.1 5be31c24972a 629397d49c76 2024-04-22T15:48:58.105 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi027 running (3m) 22s ago 3m 47.4M 4970M 18.2.1 5be31c24972a 23adcf675b04 2024-04-22T15:48:58.105 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.smithi016 smithi016 *:9095 running (6m) 2m ago 7m 59.2M - 2.43.0 a07b618ecd1d 2ba3b93ebe19 2024-04-22T15:48:58.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.103+0000 7fd4782d3700 1 -- 172.21.15.16:0/4030634547 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7fd45c06d840 msgr2=0x7fd45c06fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:58.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.103+0000 7fd4782d3700 1 --2- 172.21.15.16:0/4030634547 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7fd45c06d840 0x7fd45c06fd00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd46c009d20 tx=0x7fd46c009260 comp rx=0 tx=0).stop 2024-04-22T15:48:58.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.103+0000 7fd4782d3700 1 -- 172.21.15.16:0/4030634547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd470074d40 msgr2=0x7fd470198bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:58.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.103+0000 7fd4782d3700 1 --2- 172.21.15.16:0/4030634547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd470074d40 0x7fd470198bb0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd460005b40 tx=0x7fd46000fa00 comp rx=0 tx=0).stop 2024-04-22T15:48:58.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.103+0000 7fd4782d3700 1 -- 172.21.15.16:0/4030634547 shutdown_connections 2024-04-22T15:48:58.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.103+0000 7fd4782d3700 1 --2- 172.21.15.16:0/4030634547 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7fd45c06d840 0x7fd45c06fd00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:58.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.103+0000 7fd4782d3700 1 --2- 172.21.15.16:0/4030634547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd470073770 0x7fd470198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:58.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.103+0000 7fd4782d3700 1 --2- 172.21.15.16:0/4030634547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd470074d40 0x7fd470198bb0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:58.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.103+0000 7fd4782d3700 1 -- 172.21.15.16:0/4030634547 >> 172.21.15.16:0/4030634547 conn(0x7fd4700fc230 msgr2=0x7fd470106b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:58.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.103+0000 7fd4782d3700 1 -- 172.21.15.16:0/4030634547 shutdown_connections 2024-04-22T15:48:58.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.103+0000 7fd4782d3700 1 -- 172.21.15.16:0/4030634547 wait complete. 2024-04-22T15:48:58.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.224+0000 7f28e8452700 1 -- 172.21.15.16:0/2360808009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28e0104190 msgr2=0x7f28e0106580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:58.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.224+0000 7f28e8452700 1 --2- 172.21.15.16:0/2360808009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28e0104190 0x7f28e0106580 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f28d0009910 tx=0x7f28d0009c20 comp rx=0 tx=0).stop 2024-04-22T15:48:58.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.224+0000 7f28e8452700 1 -- 172.21.15.16:0/2360808009 shutdown_connections 2024-04-22T15:48:58.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.224+0000 7f28e8452700 1 --2- 172.21.15.16:0/2360808009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28e0104190 0x7f28e0106580 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:58.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.224+0000 7f28e8452700 1 --2- 172.21.15.16:0/2360808009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28e0101860 0x7f28e0103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:58.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.224+0000 7f28e8452700 1 -- 172.21.15.16:0/2360808009 >> 172.21.15.16:0/2360808009 conn(0x7f28e00fb190 msgr2=0x7f28e00fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:58.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.225+0000 7f28e8452700 1 -- 172.21.15.16:0/2360808009 shutdown_connections 2024-04-22T15:48:58.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.225+0000 7f28e8452700 1 -- 172.21.15.16:0/2360808009 wait complete. 2024-04-22T15:48:58.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.225+0000 7f28e8452700 1 Processor -- start 2024-04-22T15:48:58.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.225+0000 7f28e8452700 1 -- start start 2024-04-22T15:48:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.225+0000 7f28e8452700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28e0101860 0x7f28e0196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.226+0000 7f28e8452700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28e0104190 0x7f28e01969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.226+0000 7f28e8452700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e0196fc0 con 0x7f28e0101860 2024-04-22T15:48:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.226+0000 7f28e8452700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e0197100 con 0x7f28e0104190 2024-04-22T15:48:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.226+0000 7f28e61ee700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28e0101860 0x7f28e0196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.226+0000 7f28e61ee700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28e0101860 0x7f28e0196460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50920/0 (socket says 172.21.15.16:50920) 2024-04-22T15:48:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.226+0000 7f28e61ee700 1 -- 172.21.15.16:0/30588634 learned_addr learned my addr 172.21.15.16:0/30588634 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:48:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.226+0000 7f28e59ed700 1 --2- 172.21.15.16:0/30588634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28e0104190 0x7f28e01969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.226+0000 7f28e61ee700 1 -- 172.21.15.16:0/30588634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28e0104190 msgr2=0x7f28e01969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.226+0000 7f28e61ee700 1 --2- 172.21.15.16:0/30588634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28e0104190 0x7f28e01969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.226+0000 7f28e61ee700 1 -- 172.21.15.16:0/30588634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d00095f0 con 0x7f28e0101860 2024-04-22T15:48:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.226+0000 7f28e61ee700 1 --2- 172.21.15.16:0/30588634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28e0101860 0x7f28e0196460 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f28dc00e920 tx=0x7f28dc00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.227+0000 7f28d77fe700 1 -- 172.21.15.16:0/30588634 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28dc00c9d0 con 0x7f28e0101860 2024-04-22T15:48:58.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.227+0000 7f28e8452700 1 -- 172.21.15.16:0/30588634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28e019bbb0 con 0x7f28e0101860 2024-04-22T15:48:58.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.227+0000 7f28d77fe700 1 -- 172.21.15.16:0/30588634 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f28dc010430 con 0x7f28e0101860 2024-04-22T15:48:58.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.227+0000 7f28e8452700 1 -- 172.21.15.16:0/30588634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28e019c100 con 0x7f28e0101860 2024-04-22T15:48:58.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.227+0000 7f28d77fe700 1 -- 172.21.15.16:0/30588634 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28dc018650 con 0x7f28e0101860 2024-04-22T15:48:58.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.228+0000 7f28e8452700 1 -- 172.21.15.16:0/30588634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28e0190610 con 0x7f28e0101860 2024-04-22T15:48:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.228+0000 7f28d77fe700 1 -- 172.21.15.16:0/30588634 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) v1 ==== 91557+0+0 (secure 0 0 0) 0x7f28dc0105a0 con 0x7f28e0101860 2024-04-22T15:48:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.229+0000 7f28d77fe700 1 --2- 172.21.15.16:0/30588634 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f28cc06d840 0x7f28cc06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.229+0000 7f28d77fe700 1 -- 172.21.15.16:0/30588634 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f28dc014070 con 0x7f28e0101860 2024-04-22T15:48:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.229+0000 7f28e59ed700 1 --2- 172.21.15.16:0/30588634 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f28cc06d840 0x7f28cc06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.230+0000 7f28e59ed700 1 --2- 172.21.15.16:0/30588634 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f28cc06d840 0x7f28cc06fd00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f28d0009fd0 tx=0x7f28d0005fd0 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.235+0000 7f28d77fe700 1 -- 172.21.15.16:0/30588634 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f28dc053e20 con 0x7f28e0101860 2024-04-22T15:48:58.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:58 smithi016 ceph-mon[26620]: from='client.14636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:58.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:58 smithi016 ceph-mon[26620]: pgmap v11: 65 pgs: 65 active+clean; 78 MiB data, 431 MiB used, 536 GiB / 536 GiB avail; 2.1 MiB/s rd, 4.4 MiB/s wr, 175 op/s 2024-04-22T15:48:58.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.596+0000 7f28e8452700 1 -- 172.21.15.16:0/30588634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f28e002cee0 con 0x7f28e0101860 2024-04-22T15:48:58.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.597+0000 7f28d77fe700 1 -- 172.21.15.16:0/30588634 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+766 (secure 0 0 0) 0x7f28dc05a310 con 0x7f28e0101860 2024-04-22T15:48:58.600 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:48:58.600 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-22T15:48:58.600 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T15:48:58.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:48:58.601 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-22T15:48:58.601 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T15:48:58.601 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 1 2024-04-22T15:48:58.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:48:58.601 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-22T15:48:58.601 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-22T15:48:58.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:48:58.601 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-22T15:48:58.601 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T15:48:58.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:48:58.601 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-22T15:48:58.601 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 13, 2024-04-22T15:48:58.601 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 1 2024-04-22T15:48:58.602 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-22T15:48:58.602 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:48:58.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.599+0000 7f28e8452700 1 -- 172.21.15.16:0/30588634 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f28cc06d840 msgr2=0x7f28cc06fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:58.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.599+0000 7f28e8452700 1 --2- 172.21.15.16:0/30588634 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f28cc06d840 0x7f28cc06fd00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f28d0009fd0 tx=0x7f28d0005fd0 comp rx=0 tx=0).stop 2024-04-22T15:48:58.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.599+0000 7f28e8452700 1 -- 172.21.15.16:0/30588634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28e0101860 msgr2=0x7f28e0196460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:58.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.599+0000 7f28e8452700 1 --2- 172.21.15.16:0/30588634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28e0101860 0x7f28e0196460 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f28dc00e920 tx=0x7f28dc00ec30 comp rx=0 tx=0).stop 2024-04-22T15:48:58.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.599+0000 7f28e8452700 1 -- 172.21.15.16:0/30588634 shutdown_connections 2024-04-22T15:48:58.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.599+0000 7f28e8452700 1 --2- 172.21.15.16:0/30588634 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f28cc06d840 0x7f28cc06fd00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:58.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.599+0000 7f28e8452700 1 --2- 172.21.15.16:0/30588634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28e0101860 0x7f28e0196460 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:58.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.599+0000 7f28e8452700 1 --2- 172.21.15.16:0/30588634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28e0104190 0x7f28e01969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:58.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.599+0000 7f28e8452700 1 -- 172.21.15.16:0/30588634 >> 172.21.15.16:0/30588634 conn(0x7f28e00fb190 msgr2=0x7f28e00fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:58.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.599+0000 7f28e8452700 1 -- 172.21.15.16:0/30588634 shutdown_connections 2024-04-22T15:48:58.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.599+0000 7f28e8452700 1 -- 172.21.15.16:0/30588634 wait complete. 2024-04-22T15:48:58.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.728+0000 7f86b6dfb700 1 -- 172.21.15.16:0/3977038380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b0100e10 msgr2=0x7f86b0101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:58.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.728+0000 7f86b6dfb700 1 --2- 172.21.15.16:0/3977038380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b0100e10 0x7f86b0101270 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f869c009910 tx=0x7f869c009c20 comp rx=0 tx=0).stop 2024-04-22T15:48:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.729+0000 7f86b6dfb700 1 -- 172.21.15.16:0/3977038380 shutdown_connections 2024-04-22T15:48:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.729+0000 7f86b6dfb700 1 --2- 172.21.15.16:0/3977038380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b0100e10 0x7f86b0101270 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.729+0000 7f86b6dfb700 1 --2- 172.21.15.16:0/3977038380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86b00ffc10 0x7f86b0100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.729+0000 7f86b6dfb700 1 -- 172.21.15.16:0/3977038380 >> 172.21.15.16:0/3977038380 conn(0x7f86b00fb190 msgr2=0x7f86b00fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.729+0000 7f86b6dfb700 1 -- 172.21.15.16:0/3977038380 shutdown_connections 2024-04-22T15:48:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.729+0000 7f86b6dfb700 1 -- 172.21.15.16:0/3977038380 wait complete. 2024-04-22T15:48:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.729+0000 7f86b6dfb700 1 Processor -- start 2024-04-22T15:48:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.729+0000 7f86b6dfb700 1 -- start start 2024-04-22T15:48:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.729+0000 7f86b6dfb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86b00ffc10 0x7f86b01964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.729+0000 7f86b6dfb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b0100e10 0x7f86b01969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.729+0000 7f86b6dfb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b0197000 con 0x7f86b0100e10 2024-04-22T15:48:58.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.729+0000 7f86b6dfb700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b0197140 con 0x7f86b00ffc10 2024-04-22T15:48:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.730+0000 7f86affff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b0100e10 0x7f86b01969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.730+0000 7f86affff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b0100e10 0x7f86b01969e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50944/0 (socket says 172.21.15.16:50944) 2024-04-22T15:48:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.730+0000 7f86affff700 1 -- 172.21.15.16:0/981551250 learned_addr learned my addr 172.21.15.16:0/981551250 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:48:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.730+0000 7f86b4b97700 1 --2- 172.21.15.16:0/981551250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86b00ffc10 0x7f86b01964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.730+0000 7f86affff700 1 -- 172.21.15.16:0/981551250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86b00ffc10 msgr2=0x7f86b01964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.730+0000 7f86affff700 1 --2- 172.21.15.16:0/981551250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86b00ffc10 0x7f86b01964a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.730+0000 7f86affff700 1 -- 172.21.15.16:0/981551250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869c0095f0 con 0x7f86b0100e10 2024-04-22T15:48:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.730+0000 7f86affff700 1 --2- 172.21.15.16:0/981551250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b0100e10 0x7f86b01969e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f869c00fa00 tx=0x7f869c00fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.730+0000 7f86adffb700 1 -- 172.21.15.16:0/981551250 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f869c01d070 con 0x7f86b0100e10 2024-04-22T15:48:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.730+0000 7f86b6dfb700 1 -- 172.21.15.16:0/981551250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86b019bb90 con 0x7f86b0100e10 2024-04-22T15:48:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.730+0000 7f86adffb700 1 -- 172.21.15.16:0/981551250 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f869c005640 con 0x7f86b0100e10 2024-04-22T15:48:58.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.730+0000 7f86adffb700 1 -- 172.21.15.16:0/981551250 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f869c017630 con 0x7f86b0100e10 2024-04-22T15:48:58.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.731+0000 7f86b6dfb700 1 -- 172.21.15.16:0/981551250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86b019c080 con 0x7f86b0100e10 2024-04-22T15:48:58.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.732+0000 7f86b6dfb700 1 -- 172.21.15.16:0/981551250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86b0109800 con 0x7f86b0100e10 2024-04-22T15:48:58.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.732+0000 7f86adffb700 1 -- 172.21.15.16:0/981551250 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) v1 ==== 91557+0+0 (secure 0 0 0) 0x7f869c003680 con 0x7f86b0100e10 2024-04-22T15:48:58.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.733+0000 7f86adffb700 1 --2- 172.21.15.16:0/981551250 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f86a006d850 0x7f86a006fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:48:58.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.733+0000 7f86adffb700 1 -- 172.21.15.16:0/981551250 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f869c08e740 con 0x7f86b0100e10 2024-04-22T15:48:58.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.733+0000 7f86b4b97700 1 --2- 172.21.15.16:0/981551250 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f86a006d850 0x7f86a006fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:48:58.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.734+0000 7f86b4b97700 1 --2- 172.21.15.16:0/981551250 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f86a006d850 0x7f86a006fd10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f86b0100c70 tx=0x7f86a4009260 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:48:58.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.738+0000 7f86adffb700 1 -- 172.21.15.16:0/981551250 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f869c05b430 con 0x7f86b0100e10 2024-04-22T15:48:58.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:58 smithi027 ceph-mon[34922]: from='client.14636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:58.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:58 smithi027 ceph-mon[34922]: pgmap v11: 65 pgs: 65 active+clean; 78 MiB data, 431 MiB used, 536 GiB / 536 GiB avail; 2.1 MiB/s rd, 4.4 MiB/s wr, 175 op/s 2024-04-22T15:48:58.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.994+0000 7f86b6dfb700 1 -- 172.21.15.16:0/981551250 --> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f86b0105760 con 0x7f86a006d850 2024-04-22T15:48:58.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.996+0000 7f86adffb700 1 -- 172.21.15.16:0/981551250 <== mgr.24395 v2:172.21.15.27:6828/2676414545 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f86b0105760 con 0x7f86a006d850 2024-04-22T15:48:58.999 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:48:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:48:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-22T15:48:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T15:48:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [], 2024-04-22T15:48:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "1/2 daemons upgraded", 2024-04-22T15:48:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "", 2024-04-22T15:48:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-22T15:48:58.999 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:48:59.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.998+0000 7f86b6dfb700 1 -- 172.21.15.16:0/981551250 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f86a006d850 msgr2=0x7f86a006fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:59.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.998+0000 7f86b6dfb700 1 --2- 172.21.15.16:0/981551250 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f86a006d850 0x7f86a006fd10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f86b0100c70 tx=0x7f86a4009260 comp rx=0 tx=0).stop 2024-04-22T15:48:59.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.998+0000 7f86b6dfb700 1 -- 172.21.15.16:0/981551250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b0100e10 msgr2=0x7f86b01969e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:48:59.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.998+0000 7f86b6dfb700 1 --2- 172.21.15.16:0/981551250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b0100e10 0x7f86b01969e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f869c00fa00 tx=0x7f869c00fa30 comp rx=0 tx=0).stop 2024-04-22T15:48:59.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.998+0000 7f86b6dfb700 1 -- 172.21.15.16:0/981551250 shutdown_connections 2024-04-22T15:48:59.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.998+0000 7f86b6dfb700 1 --2- 172.21.15.16:0/981551250 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f86a006d850 0x7f86a006fd10 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:59.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.998+0000 7f86b6dfb700 1 --2- 172.21.15.16:0/981551250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86b00ffc10 0x7f86b01964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:59.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.998+0000 7f86b6dfb700 1 --2- 172.21.15.16:0/981551250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b0100e10 0x7f86b01969e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:48:59.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.998+0000 7f86b6dfb700 1 -- 172.21.15.16:0/981551250 >> 172.21.15.16:0/981551250 conn(0x7f86b00fb190 msgr2=0x7f86b0104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:48:59.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.998+0000 7f86b6dfb700 1 -- 172.21.15.16:0/981551250 shutdown_connections 2024-04-22T15:48:59.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:48:58.998+0000 7f86b6dfb700 1 -- 172.21.15.16:0/981551250 wait complete. 2024-04-22T15:48:59.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:59 smithi027 ceph-mon[34922]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:59.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:59 smithi027 ceph-mon[34922]: from='client.14644 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:59.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:48:59 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/30588634' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:48:59.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:59 smithi016 ceph-mon[26620]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:59.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:59 smithi016 ceph-mon[26620]: from='client.14644 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:48:59.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:48:59 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/30588634' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:49:00.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:00 smithi027 ceph-mon[34922]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:49:00.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:00 smithi027 ceph-mon[34922]: pgmap v12: 65 pgs: 65 active+clean; 88 MiB data, 457 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 4.5 MiB/s wr, 173 op/s 2024-04-22T15:49:00.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:00 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:00.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:00 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:00.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:00 smithi016 ceph-mon[26620]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:49:00.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:00 smithi016 ceph-mon[26620]: pgmap v12: 65 pgs: 65 active+clean; 88 MiB data, 457 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 4.5 MiB/s wr, 173 op/s 2024-04-22T15:49:00.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:00 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:00.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:00 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:02.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:01 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:02.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:01 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:02.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:01 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:49:02.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:01 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:49:02.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:01 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:49:02.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:01 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:49:02.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:01 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:49:02.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:01 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:49:02.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:01 smithi027 ceph-mon[34922]: Adjusting osd_memory_target on smithi016 to 1217M 2024-04-22T15:49:02.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:01 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:02.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:01 smithi027 ceph-mon[34922]: pgmap v13: 65 pgs: 65 active+clean; 88 MiB data, 457 MiB used, 536 GiB / 536 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 104 op/s 2024-04-22T15:49:02.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:01 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:02.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:01 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:02.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:01 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:49:02.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:01 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:49:02.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:01 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:49:02.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:01 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:49:02.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:01 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:49:02.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:01 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:49:02.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:01 smithi016 ceph-mon[26620]: Adjusting osd_memory_target on smithi016 to 1217M 2024-04-22T15:49:02.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:01 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:02.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:01 smithi016 ceph-mon[26620]: pgmap v13: 65 pgs: 65 active+clean; 88 MiB data, 457 MiB used, 536 GiB / 536 GiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 104 op/s 2024-04-22T15:49:04.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:04 smithi016 ceph-mon[26620]: pgmap v14: 65 pgs: 65 active+clean; 109 MiB data, 533 MiB used, 536 GiB / 536 GiB avail; 2.7 MiB/s rd, 4.2 MiB/s wr, 205 op/s 2024-04-22T15:49:04.711 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:04 smithi027 ceph-mon[34922]: pgmap v14: 65 pgs: 65 active+clean; 109 MiB data, 533 MiB used, 536 GiB / 536 GiB avail; 2.7 MiB/s rd, 4.2 MiB/s wr, 205 op/s 2024-04-22T15:49:06.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:06 smithi016 ceph-mon[26620]: pgmap v15: 65 pgs: 65 active+clean; 109 MiB data, 533 MiB used, 536 GiB / 536 GiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 159 op/s 2024-04-22T15:49:06.737 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:06 smithi027 ceph-mon[34922]: pgmap v15: 65 pgs: 65 active+clean; 109 MiB data, 533 MiB used, 536 GiB / 536 GiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 159 op/s 2024-04-22T15:49:08.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:08 smithi016 ceph-mon[26620]: pgmap v16: 65 pgs: 65 active+clean; 111 MiB data, 564 MiB used, 536 GiB / 536 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 190 op/s 2024-04-22T15:49:08.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:08 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:08.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:08 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:08.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:08 smithi027 ceph-mon[34922]: pgmap v16: 65 pgs: 65 active+clean; 111 MiB data, 564 MiB used, 536 GiB / 536 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 190 op/s 2024-04-22T15:49:08.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:08 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:08.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:08 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:10.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:10 smithi027 ceph-mon[34922]: pgmap v17: 65 pgs: 65 active+clean; 129 MiB data, 603 MiB used, 536 GiB / 536 GiB avail; 2.5 MiB/s rd, 4.3 MiB/s wr, 222 op/s 2024-04-22T15:49:10.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:10 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:10.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:10 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:10.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:10 smithi016 ceph-mon[26620]: pgmap v17: 65 pgs: 65 active+clean; 129 MiB data, 603 MiB used, 536 GiB / 536 GiB avail; 2.5 MiB/s rd, 4.3 MiB/s wr, 222 op/s 2024-04-22T15:49:10.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:10 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:10.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:10 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:11.908 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:11 smithi027 ceph-mon[34922]: pgmap v18: 65 pgs: 65 active+clean; 129 MiB data, 603 MiB used, 536 GiB / 536 GiB avail; 1.8 MiB/s rd, 3.5 MiB/s wr, 187 op/s 2024-04-22T15:49:11.909 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:11 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:49:11.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:11 smithi016 ceph-mon[26620]: pgmap v18: 65 pgs: 65 active+clean; 129 MiB data, 603 MiB used, 536 GiB / 536 GiB avail; 1.8 MiB/s rd, 3.5 MiB/s wr, 187 op/s 2024-04-22T15:49:11.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:11 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:49:14.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:14 smithi027 ceph-mon[34922]: pgmap v19: 65 pgs: 65 active+clean; 142 MiB data, 662 MiB used, 536 GiB / 536 GiB avail; 2.6 MiB/s rd, 4.6 MiB/s wr, 294 op/s 2024-04-22T15:49:14.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:14 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:14.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:14 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:14.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:14 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:49:14.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:14 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:49:14.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:14 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:49:14.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:14 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:49:14.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:14 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:49:14.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:14 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:49:14.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:14 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:14.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:14 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:49:14.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:14 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:49:14.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:14 smithi016 ceph-mon[26620]: pgmap v19: 65 pgs: 65 active+clean; 142 MiB data, 662 MiB used, 536 GiB / 536 GiB avail; 2.6 MiB/s rd, 4.6 MiB/s wr, 294 op/s 2024-04-22T15:49:14.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:14 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:14.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:14 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:14.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:14 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:49:14.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:14 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:49:14.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:14 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:49:14.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:14 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:49:14.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:14 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:49:14.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:14 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T15:49:14.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:14 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:14.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:14 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:49:14.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:14 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:49:15.676 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:15 smithi016 ceph-mon[26620]: Adjusting osd_memory_target on smithi027 to 2240M 2024-04-22T15:49:15.676 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:15 smithi016 ceph-mon[26620]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-22T15:49:15.676 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:15 smithi016 ceph-mon[26620]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-22T15:49:15.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:15 smithi027 ceph-mon[34922]: Adjusting osd_memory_target on smithi027 to 2240M 2024-04-22T15:49:15.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:15 smithi027 ceph-mon[34922]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-22T15:49:15.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:15 smithi027 ceph-mon[34922]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-22T15:49:16.703 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:16 smithi016 ceph-mon[26620]: Updating smithi016:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:49:16.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:16 smithi016 ceph-mon[26620]: Updating smithi027:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:49:16.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:16 smithi016 ceph-mon[26620]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:49:16.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:16 smithi016 ceph-mon[26620]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:49:16.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:16 smithi016 ceph-mon[26620]: Updating smithi016:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:49:16.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:16 smithi016 ceph-mon[26620]: Updating smithi027:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:49:16.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:16 smithi016 ceph-mon[26620]: pgmap v20: 65 pgs: 65 active+clean; 142 MiB data, 662 MiB used, 536 GiB / 536 GiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 193 op/s 2024-04-22T15:49:16.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:16 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:16.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:16 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:16.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:16 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:16.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:16 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:16.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:16 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:16.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:16 smithi027 ceph-mon[34922]: Updating smithi016:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:49:16.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:16 smithi027 ceph-mon[34922]: Updating smithi027:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:49:16.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:16 smithi027 ceph-mon[34922]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:49:16.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:16 smithi027 ceph-mon[34922]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:49:16.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:16 smithi027 ceph-mon[34922]: Updating smithi016:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:49:16.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:16 smithi027 ceph-mon[34922]: Updating smithi027:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:49:16.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:16 smithi027 ceph-mon[34922]: pgmap v20: 65 pgs: 65 active+clean; 142 MiB data, 662 MiB used, 536 GiB / 536 GiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 193 op/s 2024-04-22T15:49:16.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:16 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:16.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:16 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:16.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:16 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:16.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:16 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:16.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:16 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:17.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:17 smithi016 ceph-mon[26620]: Reconfiguring prometheus.smithi016 (dependencies changed)... 2024-04-22T15:49:17.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:17 smithi016 ceph-mon[26620]: Reconfiguring daemon prometheus.smithi016 on smithi016 2024-04-22T15:49:17.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:17 smithi027 ceph-mon[34922]: Reconfiguring prometheus.smithi016 (dependencies changed)... 2024-04-22T15:49:17.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:17 smithi027 ceph-mon[34922]: Reconfiguring daemon prometheus.smithi016 on smithi016 2024-04-22T15:49:18.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:18 smithi016 ceph-mon[26620]: pgmap v21: 65 pgs: 65 active+clean; 151 MiB data, 696 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 3.6 MiB/s wr, 247 op/s 2024-04-22T15:49:18.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:18 smithi027 ceph-mon[34922]: pgmap v21: 65 pgs: 65 active+clean; 151 MiB data, 696 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 3.6 MiB/s wr, 247 op/s 2024-04-22T15:49:19.724 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:19 smithi016 ceph-mon[26620]: pgmap v22: 65 pgs: 65 active+clean; 159 MiB data, 729 MiB used, 536 GiB / 536 GiB avail; 2.7 MiB/s rd, 4.1 MiB/s wr, 300 op/s 2024-04-22T15:49:19.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:19 smithi027 ceph-mon[34922]: pgmap v22: 65 pgs: 65 active+clean; 159 MiB data, 729 MiB used, 536 GiB / 536 GiB avail; 2.7 MiB/s rd, 4.1 MiB/s wr, 300 op/s 2024-04-22T15:49:21.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:21 smithi016 ceph-mon[26620]: pgmap v23: 65 pgs: 65 active+clean; 159 MiB data, 729 MiB used, 536 GiB / 536 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 244 op/s 2024-04-22T15:49:22.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:21 smithi027 ceph-mon[34922]: pgmap v23: 65 pgs: 65 active+clean; 159 MiB data, 729 MiB used, 536 GiB / 536 GiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 244 op/s 2024-04-22T15:49:22.513 INFO:tasks.workunit.client.0.smithi016.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-22T15:49:22.520 INFO:tasks.workunit.client.0.smithi016.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-22T15:49:22.521 INFO:tasks.workunit.client.0.smithi016.stderr:+ make 2024-04-22T15:49:22.750 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:22 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:22.750 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:22 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:22.750 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:22 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:49:22.750 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:22 smithi016 ceph-mon[26620]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:49:23.010 INFO:tasks.workunit.client.0.smithi016.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-22T15:49:23.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:22 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:23.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:22 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:23.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:22 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:49:23.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:22 smithi027 ceph-mon[34922]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:49:23.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:22 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi016.front.sepia.ceph.com:9095"}]: dispatch 2024-04-22T15:49:23.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:22 smithi027 ceph-mon[34922]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi016.front.sepia.ceph.com:9095"}]: dispatch 2024-04-22T15:49:23.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:22 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:23.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:22 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:49:23.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:22 smithi027 ceph-mon[34922]: Upgrade: Need to upgrade myself (mgr.smithi027.ukdgzs) 2024-04-22T15:49:23.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:22 smithi027 ceph-mon[34922]: Upgrade: Need to upgrade myself (mgr.smithi027.ukdgzs) 2024-04-22T15:49:23.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:22 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi016.front.sepia.ceph.com:9095"}]: dispatch 2024-04-22T15:49:23.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:22 smithi016 ceph-mon[26620]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi016.front.sepia.ceph.com:9095"}]: dispatch 2024-04-22T15:49:23.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:22 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:23.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:22 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:49:23.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:22 smithi016 ceph-mon[26620]: Upgrade: Need to upgrade myself (mgr.smithi027.ukdgzs) 2024-04-22T15:49:23.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:22 smithi016 ceph-mon[26620]: Upgrade: Need to upgrade myself (mgr.smithi027.ukdgzs) 2024-04-22T15:49:23.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:23 smithi027 ceph-mon[34922]: pgmap v24: 65 pgs: 65 active+clean; 188 MiB data, 781 MiB used, 536 GiB / 536 GiB avail; 2.5 MiB/s rd, 5.0 MiB/s wr, 386 op/s 2024-04-22T15:49:23.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:23 smithi016 ceph-mon[26620]: pgmap v24: 65 pgs: 65 active+clean; 188 MiB data, 781 MiB used, 536 GiB / 536 GiB avail; 2.5 MiB/s rd, 5.0 MiB/s wr, 386 op/s 2024-04-22T15:49:25.101 INFO:tasks.workunit.client.0.smithi016.stderr:++ readlink -f fsstress 2024-04-22T15:49:25.102 INFO:tasks.workunit.client.0.smithi016.stderr:+ BIN=/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-04-22T15:49:25.103 INFO:tasks.workunit.client.0.smithi016.stderr:+ popd 2024-04-22T15:49:25.103 INFO:tasks.workunit.client.0.smithi016.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-22T15:49:25.104 INFO:tasks.workunit.client.0.smithi016.stderr:+ popd 2024-04-22T15:49:25.104 INFO:tasks.workunit.client.0.smithi016.stdout:~/cephtest/mnt.0/client.0/tmp 2024-04-22T15:49:25.104 INFO:tasks.workunit.client.0.smithi016.stderr:++ mktemp -d -p . 2024-04-22T15:49:25.107 INFO:tasks.workunit.client.0.smithi016.stderr:+ T=./tmp.8wrsDzmH6d 2024-04-22T15:49:25.107 INFO:tasks.workunit.client.0.smithi016.stderr:+ /home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.8wrsDzmH6d -l 1 -n 1000 -p 10 -v 2024-04-22T15:49:25.112 INFO:tasks.workunit.client.0.smithi016.stdout:seed = 1713845682 2024-04-22T15:49:25.118 INFO:tasks.workunit.client.0.smithi016.stdout:0/0: dwrite - no filename 2024-04-22T15:49:25.120 INFO:tasks.workunit.client.0.smithi016.stdout:1/0: dwrite - no filename 2024-04-22T15:49:25.120 INFO:tasks.workunit.client.0.smithi016.stdout:0/1: mknod c0 0 2024-04-22T15:49:25.120 INFO:tasks.workunit.client.0.smithi016.stdout:0/2: dread - no filename 2024-04-22T15:49:25.123 INFO:tasks.workunit.client.0.smithi016.stdout:1/1: symlink l0 0 2024-04-22T15:49:25.123 INFO:tasks.workunit.client.0.smithi016.stdout:3/0: write - no filename 2024-04-22T15:49:25.123 INFO:tasks.workunit.client.0.smithi016.stdout:3/1: rename - no filename 2024-04-22T15:49:25.126 INFO:tasks.workunit.client.0.smithi016.stdout:1/2: rename l0 to l1 0 2024-04-22T15:49:25.127 INFO:tasks.workunit.client.0.smithi016.stdout:4/0: chown . 452990003 1 2024-04-22T15:49:25.127 INFO:tasks.workunit.client.0.smithi016.stdout:4/1: link - no file 2024-04-22T15:49:25.127 INFO:tasks.workunit.client.0.smithi016.stdout:4/2: rmdir - no directory 2024-04-22T15:49:25.127 INFO:tasks.workunit.client.0.smithi016.stdout:4/3: rename - no filename 2024-04-22T15:49:25.127 INFO:tasks.workunit.client.0.smithi016.stdout:4/4: dread - no filename 2024-04-22T15:49:25.127 INFO:tasks.workunit.client.0.smithi016.stdout:4/5: dread - no filename 2024-04-22T15:49:25.127 INFO:tasks.workunit.client.0.smithi016.stdout:4/6: dwrite - no filename 2024-04-22T15:49:25.127 INFO:tasks.workunit.client.0.smithi016.stdout:4/7: rmdir - no directory 2024-04-22T15:49:25.127 INFO:tasks.workunit.client.0.smithi016.stdout:4/8: write - no filename 2024-04-22T15:49:25.127 INFO:tasks.workunit.client.0.smithi016.stdout:4/9: write - no filename 2024-04-22T15:49:25.129 INFO:tasks.workunit.client.0.smithi016.stdout:2/0: creat f0 x:0 0 0 2024-04-22T15:49:25.131 INFO:tasks.workunit.client.0.smithi016.stdout:5/0: chown . 83 1 2024-04-22T15:49:25.132 INFO:tasks.workunit.client.0.smithi016.stdout:2/1: mknod c1 0 2024-04-22T15:49:25.134 INFO:tasks.workunit.client.0.smithi016.stdout:6/0: rename - no filename 2024-04-22T15:49:25.134 INFO:tasks.workunit.client.0.smithi016.stdout:6/1: dwrite - no filename 2024-04-22T15:49:25.135 INFO:tasks.workunit.client.0.smithi016.stdout:5/1: creat f0 x:0 0 0 2024-04-22T15:49:25.137 INFO:tasks.workunit.client.0.smithi016.stdout:7/0: write - no filename 2024-04-22T15:49:25.137 INFO:tasks.workunit.client.0.smithi016.stdout:6/2: symlink l0 0 2024-04-22T15:49:25.138 INFO:tasks.workunit.client.0.smithi016.stdout:6/3: chown l0 305733 1 2024-04-22T15:49:25.139 INFO:tasks.workunit.client.0.smithi016.stdout:6/4: write - no filename 2024-04-22T15:49:25.139 INFO:tasks.workunit.client.0.smithi016.stdout:6/5: stat l0 0 2024-04-22T15:49:25.139 INFO:tasks.workunit.client.0.smithi016.stdout:6/6: stat l0 0 2024-04-22T15:49:25.139 INFO:tasks.workunit.client.0.smithi016.stdout:6/7: dread - no filename 2024-04-22T15:49:25.140 INFO:tasks.workunit.client.0.smithi016.stdout:2/2: dwrite f0 [0,4194304] 0 2024-04-22T15:49:25.140 INFO:tasks.workunit.client.0.smithi016.stdout:6/8: chown l0 7055 1 2024-04-22T15:49:25.140 INFO:tasks.workunit.client.0.smithi016.stdout:8/0: link - no file 2024-04-22T15:49:25.140 INFO:tasks.workunit.client.0.smithi016.stdout:8/1: dwrite - no filename 2024-04-22T15:49:25.140 INFO:tasks.workunit.client.0.smithi016.stdout:8/2: dwrite - no filename 2024-04-22T15:49:25.141 INFO:tasks.workunit.client.0.smithi016.stdout:8/3: truncate - no filename 2024-04-22T15:49:25.141 INFO:tasks.workunit.client.0.smithi016.stdout:8/4: dread - no filename 2024-04-22T15:49:25.141 INFO:tasks.workunit.client.0.smithi016.stdout:8/5: truncate - no filename 2024-04-22T15:49:25.141 INFO:tasks.workunit.client.0.smithi016.stdout:8/6: chown . 0 1 2024-04-22T15:49:25.141 INFO:tasks.workunit.client.0.smithi016.stdout:8/7: dread - no filename 2024-04-22T15:49:25.141 INFO:tasks.workunit.client.0.smithi016.stdout:8/8: write - no filename 2024-04-22T15:49:25.141 INFO:tasks.workunit.client.0.smithi016.stdout:8/9: unlink - no file 2024-04-22T15:49:25.141 INFO:tasks.workunit.client.0.smithi016.stdout:8/10: fdatasync - no filename 2024-04-22T15:49:25.141 INFO:tasks.workunit.client.0.smithi016.stdout:8/11: readlink - no filename 2024-04-22T15:49:25.142 INFO:tasks.workunit.client.0.smithi016.stdout:8/12: fdatasync - no filename 2024-04-22T15:49:25.142 INFO:tasks.workunit.client.0.smithi016.stdout:6/9: chown l0 12430144 1 2024-04-22T15:49:25.142 INFO:tasks.workunit.client.0.smithi016.stdout:6/10: read - no filename 2024-04-22T15:49:25.142 INFO:tasks.workunit.client.0.smithi016.stdout:6/11: write - no filename 2024-04-22T15:49:25.142 INFO:tasks.workunit.client.0.smithi016.stdout:9/0: dwrite - no filename 2024-04-22T15:49:25.143 INFO:tasks.workunit.client.0.smithi016.stdout:9/1: chown . 7381833 1 2024-04-22T15:49:25.143 INFO:tasks.workunit.client.0.smithi016.stdout:9/2: dwrite - no filename 2024-04-22T15:49:25.143 INFO:tasks.workunit.client.0.smithi016.stdout:9/3: dread - no filename 2024-04-22T15:49:25.143 INFO:tasks.workunit.client.0.smithi016.stdout:9/4: unlink - no file 2024-04-22T15:49:25.143 INFO:tasks.workunit.client.0.smithi016.stdout:7/1: getdents . 0 2024-04-22T15:49:25.143 INFO:tasks.workunit.client.0.smithi016.stdout:7/2: readlink - no filename 2024-04-22T15:49:25.146 INFO:tasks.workunit.client.0.smithi016.stdout:6/12: rename l0 to l1 0 2024-04-22T15:49:25.146 INFO:tasks.workunit.client.0.smithi016.stdout:6/13: fdatasync - no filename 2024-04-22T15:49:25.153 INFO:tasks.workunit.client.0.smithi016.stdout:8/13: mknod c0 0 2024-04-22T15:49:25.153 INFO:tasks.workunit.client.0.smithi016.stdout:8/14: fdatasync - no filename 2024-04-22T15:49:25.154 INFO:tasks.workunit.client.0.smithi016.stdout:6/14: creat f2 x:0 0 0 2024-04-22T15:49:25.155 INFO:tasks.workunit.client.0.smithi016.stdout:6/15: dread - f2 zero size 2024-04-22T15:49:25.155 INFO:tasks.workunit.client.0.smithi016.stdout:6/16: chown f2 35002 1 2024-04-22T15:49:25.155 INFO:tasks.workunit.client.0.smithi016.stdout:9/5: symlink l0 0 2024-04-22T15:49:25.155 INFO:tasks.workunit.client.0.smithi016.stdout:9/6: truncate - no filename 2024-04-22T15:49:25.158 INFO:tasks.workunit.client.0.smithi016.stdout:7/3: creat f0 x:0 0 0 2024-04-22T15:49:25.161 INFO:tasks.workunit.client.0.smithi016.stdout:9/7: symlink l1 0 2024-04-22T15:49:25.161 INFO:tasks.workunit.client.0.smithi016.stdout:9/8: readlink l1 0 2024-04-22T15:49:25.161 INFO:tasks.workunit.client.0.smithi016.stdout:9/9: truncate - no filename 2024-04-22T15:49:25.161 INFO:tasks.workunit.client.0.smithi016.stdout:9/10: dwrite - no filename 2024-04-22T15:49:25.162 INFO:tasks.workunit.client.0.smithi016.stdout:9/11: chown l0 960218958 1 2024-04-22T15:49:25.163 INFO:tasks.workunit.client.0.smithi016.stdout:7/4: mknod c1 0 2024-04-22T15:49:25.166 INFO:tasks.workunit.client.0.smithi016.stdout:9/12: creat f2 x:0 0 0 2024-04-22T15:49:25.167 INFO:tasks.workunit.client.0.smithi016.stdout:9/13: chown f2 308728947 1 2024-04-22T15:49:25.170 INFO:tasks.workunit.client.0.smithi016.stdout:7/5: dwrite f0 [0,4194304] 0 2024-04-22T15:49:25.170 INFO:tasks.workunit.client.0.smithi016.stdout:7/6: chown f0 8273936 1 2024-04-22T15:49:25.173 INFO:tasks.workunit.client.0.smithi016.stdout:7/7: creat f2 x:0 0 0 2024-04-22T15:49:25.173 INFO:tasks.workunit.client.0.smithi016.stdout:7/8: truncate f2 494515 0 2024-04-22T15:49:25.396 INFO:tasks.workunit.client.0.smithi016.stdout:0/3: getdents . 0 2024-04-22T15:49:25.397 INFO:tasks.workunit.client.0.smithi016.stdout:0/4: write - no filename 2024-04-22T15:49:25.397 INFO:tasks.workunit.client.0.smithi016.stdout:0/5: dwrite - no filename 2024-04-22T15:49:25.401 INFO:tasks.workunit.client.0.smithi016.stdout:1/3: unlink l1 0 2024-04-22T15:49:25.401 INFO:tasks.workunit.client.0.smithi016.stdout:1/4: dwrite - no filename 2024-04-22T15:49:25.401 INFO:tasks.workunit.client.0.smithi016.stdout:1/5: unlink - no file 2024-04-22T15:49:25.403 INFO:tasks.workunit.client.0.smithi016.stdout:0/6: unlink c0 0 2024-04-22T15:49:25.403 INFO:tasks.workunit.client.0.smithi016.stdout:0/7: dwrite - no filename 2024-04-22T15:49:25.403 INFO:tasks.workunit.client.0.smithi016.stdout:0/8: dread - no filename 2024-04-22T15:49:25.403 INFO:tasks.workunit.client.0.smithi016.stdout:0/9: dwrite - no filename 2024-04-22T15:49:25.404 INFO:tasks.workunit.client.0.smithi016.stdout:5/2: fsync f0 0 2024-04-22T15:49:25.405 INFO:tasks.workunit.client.0.smithi016.stdout:1/6: creat f2 x:0 0 0 2024-04-22T15:49:25.406 INFO:tasks.workunit.client.0.smithi016.stdout:0/10: mknod c1 0 2024-04-22T15:49:25.410 INFO:tasks.workunit.client.0.smithi016.stdout:0/11: mknod c2 0 2024-04-22T15:49:25.413 INFO:tasks.workunit.client.0.smithi016.stdout:6/17: chown l1 1 1 2024-04-22T15:49:25.413 INFO:tasks.workunit.client.0.smithi016.stdout:2/3: truncate f0 483740 0 2024-04-22T15:49:25.416 INFO:tasks.workunit.client.0.smithi016.stdout:8/15: getdents . 0 2024-04-22T15:49:25.416 INFO:tasks.workunit.client.0.smithi016.stdout:5/3: dwrite f0 [0,4194304] 0 2024-04-22T15:49:25.416 INFO:tasks.workunit.client.0.smithi016.stdout:1/7: dwrite f2 [0,4194304] 0 2024-04-22T15:49:25.417 INFO:tasks.workunit.client.0.smithi016.stdout:1/8: readlink - no filename 2024-04-22T15:49:25.419 INFO:tasks.workunit.client.0.smithi016.stdout:0/12: symlink l3 0 2024-04-22T15:49:25.420 INFO:tasks.workunit.client.0.smithi016.stdout:6/18: mkdir d3 0 2024-04-22T15:49:25.423 INFO:tasks.workunit.client.0.smithi016.stdout:6/19: write f2 [796862,84298] 0 2024-04-22T15:49:25.423 INFO:tasks.workunit.client.0.smithi016.stdout:2/4: mkdir d2 0 2024-04-22T15:49:25.425 INFO:tasks.workunit.client.0.smithi016.stdout:8/16: creat f1 x:0 0 0 2024-04-22T15:49:25.425 INFO:tasks.workunit.client.0.smithi016.stdout:8/17: rmdir - no directory 2024-04-22T15:49:25.425 INFO:tasks.workunit.client.0.smithi016.stdout:8/18: rmdir - no directory 2024-04-22T15:49:25.425 INFO:tasks.workunit.client.0.smithi016.stdout:8/19: readlink - no filename 2024-04-22T15:49:25.425 INFO:tasks.workunit.client.0.smithi016.stdout:9/14: fsync f2 0 2024-04-22T15:49:25.426 INFO:tasks.workunit.client.0.smithi016.stdout:6/20: write f2 [1866635,43322] 0 2024-04-22T15:49:25.427 INFO:tasks.workunit.client.0.smithi016.stdout:7/9: write f0 [4189267,77391] 0 2024-04-22T15:49:25.428 INFO:tasks.workunit.client.0.smithi016.stdout:9/15: fdatasync f2 0 2024-04-22T15:49:25.429 INFO:tasks.workunit.client.0.smithi016.stdout:5/4: dwrite f0 [0,4194304] 0 2024-04-22T15:49:25.429 INFO:tasks.workunit.client.0.smithi016.stdout:7/10: truncate f2 884442 0 2024-04-22T15:49:25.430 INFO:tasks.workunit.client.0.smithi016.stdout:9/16: write f2 [923825,121211] 0 2024-04-22T15:49:25.431 INFO:tasks.workunit.client.0.smithi016.stdout:0/13: unlink c1 0 2024-04-22T15:49:25.439 INFO:tasks.workunit.client.0.smithi016.stdout:8/20: mkdir d2 0 2024-04-22T15:49:25.439 INFO:tasks.workunit.client.0.smithi016.stdout:8/21: dread - f1 zero size 2024-04-22T15:49:25.445 INFO:tasks.workunit.client.0.smithi016.stdout:7/11: mkdir d3 0 2024-04-22T15:49:25.446 INFO:tasks.workunit.client.0.smithi016.stdout:7/12: dread f2 [0,4194304] 0 2024-04-22T15:49:25.446 INFO:tasks.workunit.client.0.smithi016.stdout:5/5: creat f1 x:0 0 0 2024-04-22T15:49:25.446 INFO:tasks.workunit.client.0.smithi016.stdout:5/6: dread - f1 zero size 2024-04-22T15:49:25.448 INFO:tasks.workunit.client.0.smithi016.stdout:5/7: write f0 [2390528,98460] 0 2024-04-22T15:49:25.448 INFO:tasks.workunit.client.0.smithi016.stdout:9/17: symlink l3 0 2024-04-22T15:49:25.450 INFO:tasks.workunit.client.0.smithi016.stdout:0/14: creat f4 x:0 0 0 2024-04-22T15:49:25.451 INFO:tasks.workunit.client.0.smithi016.stdout:2/5: symlink d2/l3 0 2024-04-22T15:49:25.452 INFO:tasks.workunit.client.0.smithi016.stdout:7/13: dwrite f0 [0,4194304] 0 2024-04-22T15:49:25.456 INFO:tasks.workunit.client.0.smithi016.stdout:9/18: dwrite f2 [0,4194304] 0 2024-04-22T15:49:25.467 INFO:tasks.workunit.client.0.smithi016.stdout:5/8: creat f2 x:0 0 0 2024-04-22T15:49:25.468 INFO:tasks.workunit.client.0.smithi016.stdout:5/9: dread - f2 zero size 2024-04-22T15:49:25.471 INFO:tasks.workunit.client.0.smithi016.stdout:0/15: mknod c5 0 2024-04-22T15:49:25.471 INFO:tasks.workunit.client.0.smithi016.stdout:0/16: chown l3 13 1 2024-04-22T15:49:25.472 INFO:tasks.workunit.client.0.smithi016.stdout:0/17: truncate f4 936818 0 2024-04-22T15:49:25.475 INFO:tasks.workunit.client.0.smithi016.stdout:8/22: mknod d2/c3 0 2024-04-22T15:49:25.479 INFO:tasks.workunit.client.0.smithi016.stdout:9/19: symlink l4 0 2024-04-22T15:49:25.480 INFO:tasks.workunit.client.0.smithi016.stdout:6/21: link l1 d3/l4 0 2024-04-22T15:49:25.484 INFO:tasks.workunit.client.0.smithi016.stdout:5/10: rename f1 to f3 0 2024-04-22T15:49:25.484 INFO:tasks.workunit.client.0.smithi016.stdout:5/11: read - f3 zero size 2024-04-22T15:49:25.486 INFO:tasks.workunit.client.0.smithi016.stdout:7/14: mkdir d3/d4 0 2024-04-22T15:49:25.488 INFO:tasks.workunit.client.0.smithi016.stdout:9/20: symlink l5 0 2024-04-22T15:49:25.488 INFO:tasks.workunit.client.0.smithi016.stdout:9/21: fsync f2 0 2024-04-22T15:49:25.488 INFO:tasks.workunit.client.0.smithi016.stdout:9/22: fdatasync f2 0 2024-04-22T15:49:25.491 INFO:tasks.workunit.client.0.smithi016.stdout:9/23: dread f2 [0,4194304] 0 2024-04-22T15:49:25.492 INFO:tasks.workunit.client.0.smithi016.stdout:5/12: creat f4 x:0 0 0 2024-04-22T15:49:25.492 INFO:tasks.workunit.client.0.smithi016.stdout:5/13: write f0 [3286480,30793] 0 2024-04-22T15:49:25.493 INFO:tasks.workunit.client.0.smithi016.stdout:5/14: write f2 [833975,80168] 0 2024-04-22T15:49:25.496 INFO:tasks.workunit.client.0.smithi016.stdout:9/24: dread f2 [0,4194304] 0 2024-04-22T15:49:25.500 INFO:tasks.workunit.client.0.smithi016.stdout:6/22: mknod d3/c5 0 2024-04-22T15:49:25.503 INFO:tasks.workunit.client.0.smithi016.stdout:5/15: creat f5 x:0 0 0 2024-04-22T15:49:25.504 INFO:tasks.workunit.client.0.smithi016.stdout:5/16: write f2 [1240298,34980] 0 2024-04-22T15:49:25.507 INFO:tasks.workunit.client.0.smithi016.stdout:7/15: getdents d3/d4 0 2024-04-22T15:49:25.509 INFO:tasks.workunit.client.0.smithi016.stdout:6/23: creat d3/f6 x:0 0 0 2024-04-22T15:49:25.509 INFO:tasks.workunit.client.0.smithi016.stdout:6/24: chown f2 48222890 1 2024-04-22T15:49:25.510 INFO:tasks.workunit.client.0.smithi016.stdout:5/17: dwrite f2 [0,4194304] 0 2024-04-22T15:49:25.511 INFO:tasks.workunit.client.0.smithi016.stdout:5/18: dread - f5 zero size 2024-04-22T15:49:25.511 INFO:tasks.workunit.client.0.smithi016.stdout:5/19: chown f0 134289 1 2024-04-22T15:49:25.516 INFO:tasks.workunit.client.0.smithi016.stdout:1/9: getdents . 0 2024-04-22T15:49:25.517 INFO:tasks.workunit.client.0.smithi016.stdout:1/10: chown f2 65100 1 2024-04-22T15:49:25.518 INFO:tasks.workunit.client.0.smithi016.stdout:5/20: dwrite f0 [0,4194304] 0 2024-04-22T15:49:25.519 INFO:tasks.workunit.client.0.smithi016.stdout:7/16: write f2 [1282874,98869] 0 2024-04-22T15:49:25.531 INFO:tasks.workunit.client.0.smithi016.stdout:7/17: mknod d3/c5 0 2024-04-22T15:49:25.532 INFO:tasks.workunit.client.0.smithi016.stdout:2/6: dwrite f0 [0,4194304] 0 2024-04-22T15:49:25.543 INFO:tasks.workunit.client.0.smithi016.stdout:7/18: dwrite f2 [0,4194304] 0 2024-04-22T15:49:25.543 INFO:tasks.workunit.client.0.smithi016.stdout:2/7: dwrite f0 [4194304,4194304] 0 2024-04-22T15:49:25.547 INFO:tasks.workunit.client.0.smithi016.stdout:7/19: unlink c1 0 2024-04-22T15:49:25.549 INFO:tasks.workunit.client.0.smithi016.stdout:2/8: creat d2/f4 x:0 0 0 2024-04-22T15:49:25.550 INFO:tasks.workunit.client.0.smithi016.stdout:2/9: stat d2 0 2024-04-22T15:49:25.553 INFO:tasks.workunit.client.0.smithi016.stdout:2/10: creat d2/f5 x:0 0 0 2024-04-22T15:49:25.555 INFO:tasks.workunit.client.0.smithi016.stdout:7/20: creat d3/d4/f6 x:0 0 0 2024-04-22T15:49:25.555 INFO:tasks.workunit.client.0.smithi016.stdout:7/21: chown d3 45264 1 2024-04-22T15:49:25.556 INFO:tasks.workunit.client.0.smithi016.stdout:2/11: creat d2/f6 x:0 0 0 2024-04-22T15:49:25.560 INFO:tasks.workunit.client.0.smithi016.stdout:2/12: creat d2/f7 x:0 0 0 2024-04-22T15:49:25.569 INFO:tasks.workunit.client.0.smithi016.stdout:2/13: link d2/f5 d2/f8 0 2024-04-22T15:49:25.571 INFO:tasks.workunit.client.0.smithi016.stdout:5/21: fsync f2 0 2024-04-22T15:49:25.573 INFO:tasks.workunit.client.0.smithi016.stdout:2/14: mkdir d2/d9 0 2024-04-22T15:49:25.574 INFO:tasks.workunit.client.0.smithi016.stdout:2/15: read - d2/f6 zero size 2024-04-22T15:49:25.574 INFO:tasks.workunit.client.0.smithi016.stdout:2/16: dread - d2/f6 zero size 2024-04-22T15:49:25.584 INFO:tasks.workunit.client.0.smithi016.stdout:2/17: dwrite d2/f5 [0,4194304] 0 2024-04-22T15:49:25.588 INFO:tasks.workunit.client.0.smithi016.stdout:2/18: dread f0 [0,4194304] 0 2024-04-22T15:49:25.595 INFO:tasks.workunit.client.0.smithi016.stdout:2/19: link d2/f8 d2/d9/fa 0 2024-04-22T15:49:25.599 INFO:tasks.workunit.client.0.smithi016.stdout:0/18: truncate f4 886518 0 2024-04-22T15:49:25.602 INFO:tasks.workunit.client.0.smithi016.stdout:9/25: truncate f2 2279634 0 2024-04-22T15:49:25.663 INFO:tasks.workunit.client.0.smithi016.stdout:6/25: fsync d3/f6 0 2024-04-22T15:49:25.667 INFO:tasks.workunit.client.0.smithi016.stdout:6/26: dread f2 [0,4194304] 0 2024-04-22T15:49:25.668 INFO:tasks.workunit.client.0.smithi016.stdout:6/27: write f2 [2187096,88825] 0 2024-04-22T15:49:25.672 INFO:tasks.workunit.client.0.smithi016.stdout:1/11: truncate f2 372971 0 2024-04-22T15:49:25.674 INFO:tasks.workunit.client.0.smithi016.stdout:6/28: dwrite f2 [0,4194304] 0 2024-04-22T15:49:25.682 INFO:tasks.workunit.client.0.smithi016.stdout:6/29: dwrite f2 [0,4194304] 0 2024-04-22T15:49:25.699 INFO:tasks.workunit.client.0.smithi016.stdout:7/22: truncate f2 1885509 0 2024-04-22T15:49:25.699 INFO:tasks.workunit.client.0.smithi016.stdout:7/23: chown d3 573227 1 2024-04-22T15:49:25.714 INFO:tasks.workunit.client.0.smithi016.stdout:2/20: dwrite d2/f5 [4194304,4194304] 0 2024-04-22T15:49:25.867 INFO:tasks.workunit.client.0.smithi016.stdout:0/19: dwrite f4 [0,4194304] 0 2024-04-22T15:49:25.964 INFO:tasks.workunit.client.0.smithi016.stdout:9/26: dwrite f2 [0,4194304] 0 2024-04-22T15:49:25.965 INFO:tasks.workunit.client.0.smithi016.stdout:7/24: rmdir d3 39 2024-04-22T15:49:25.965 INFO:tasks.workunit.client.0.smithi016.stdout:6/30: chown l1 18810 1 2024-04-22T15:49:25.967 INFO:tasks.workunit.client.0.smithi016.stdout:0/20: fsync f4 0 2024-04-22T15:49:25.968 INFO:tasks.workunit.client.0.smithi016.stdout:6/31: truncate d3/f6 651192 0 2024-04-22T15:49:25.970 INFO:tasks.workunit.client.0.smithi016.stdout:7/25: dread f2 [0,4194304] 0 2024-04-22T15:49:25.972 INFO:tasks.workunit.client.0.smithi016.stdout:9/27: dread f2 [0,4194304] 0 2024-04-22T15:49:25.973 INFO:tasks.workunit.client.0.smithi016.stdout:9/28: write f2 [2332244,108593] 0 2024-04-22T15:49:25.977 INFO:tasks.workunit.client.0.smithi016.stdout:0/21: write f4 [4677276,31968] 0 2024-04-22T15:49:25.978 INFO:tasks.workunit.client.0.smithi016.stdout:0/22: stat f4 0 2024-04-22T15:49:25.978 INFO:tasks.workunit.client.0.smithi016.stdout:2/21: symlink d2/d9/lb 0 2024-04-22T15:49:25.980 INFO:tasks.workunit.client.0.smithi016.stdout:1/12: dread f2 [0,4194304] 0 2024-04-22T15:49:25.980 INFO:tasks.workunit.client.0.smithi016.stdout:9/29: dread f2 [0,4194304] 0 2024-04-22T15:49:25.980 INFO:tasks.workunit.client.0.smithi016.stdout:0/23: chown c5 70 1 2024-04-22T15:49:25.981 INFO:tasks.workunit.client.0.smithi016.stdout:2/22: mknod d2/cc 0 2024-04-22T15:49:25.981 INFO:tasks.workunit.client.0.smithi016.stdout:1/13: stat f2 0 2024-04-22T15:49:25.982 INFO:tasks.workunit.client.0.smithi016.stdout:7/26: chown d3/d4/f6 0 1 2024-04-22T15:49:25.983 INFO:tasks.workunit.client.0.smithi016.stdout:0/24: rename l3 to l6 0 2024-04-22T15:49:25.984 INFO:tasks.workunit.client.0.smithi016.stdout:2/23: chown d2/d9/lb 529330 1 2024-04-22T15:49:25.986 INFO:tasks.workunit.client.0.smithi016.stdout:2/24: dread - d2/f4 zero size 2024-04-22T15:49:25.987 INFO:tasks.workunit.client.0.smithi016.stdout:0/25: rename c5 to c7 0 2024-04-22T15:49:25.989 INFO:tasks.workunit.client.0.smithi016.stdout:6/32: dwrite f2 [0,4194304] 0 2024-04-22T15:49:25.990 INFO:tasks.workunit.client.0.smithi016.stdout:7/27: link f2 d3/d4/f7 0 2024-04-22T15:49:25.991 INFO:tasks.workunit.client.0.smithi016.stdout:9/30: dread f2 [0,4194304] 0 2024-04-22T15:49:25.995 INFO:tasks.workunit.client.0.smithi016.stdout:2/25: dread d2/f8 [4194304,4194304] 0 2024-04-22T15:49:25.996 INFO:tasks.workunit.client.0.smithi016.stdout:9/31: rename l4 to l6 0 2024-04-22T15:49:25.997 INFO:tasks.workunit.client.0.smithi016.stdout:2/26: fdatasync d2/f4 0 2024-04-22T15:49:25.998 INFO:tasks.workunit.client.0.smithi016.stdout:7/28: mknod d3/c8 0 2024-04-22T15:49:26.002 INFO:tasks.workunit.client.0.smithi016.stdout:1/14: dwrite f2 [0,4194304] 0 2024-04-22T15:49:26.003 INFO:tasks.workunit.client.0.smithi016.stdout:1/15: chown f2 9826 1 2024-04-22T15:49:26.004 INFO:tasks.workunit.client.0.smithi016.stdout:9/32: write f2 [3402894,94070] 0 2024-04-22T15:49:26.007 INFO:tasks.workunit.client.0.smithi016.stdout:7/29: mkdir d3/d4/d9 0 2024-04-22T15:49:26.007 INFO:tasks.workunit.client.0.smithi016.stdout:0/26: truncate f4 353182 0 2024-04-22T15:49:26.007 INFO:tasks.workunit.client.0.smithi016.stdout:1/16: chown f2 282923835 1 2024-04-22T15:49:26.008 INFO:tasks.workunit.client.0.smithi016.stdout:6/33: link l1 d3/l7 0 2024-04-22T15:49:26.009 INFO:tasks.workunit.client.0.smithi016.stdout:2/27: dwrite d2/f6 [0,4194304] 0 2024-04-22T15:49:26.010 INFO:tasks.workunit.client.0.smithi016.stdout:9/33: creat f7 x:0 0 0 2024-04-22T15:49:26.014 INFO:tasks.workunit.client.0.smithi016.stdout:1/17: mkdir d3 0 2024-04-22T15:49:26.015 INFO:tasks.workunit.client.0.smithi016.stdout:1/18: read f2 [1894464,26032] 0 2024-04-22T15:49:26.016 INFO:tasks.workunit.client.0.smithi016.stdout:7/30: dread f0 [0,4194304] 0 2024-04-22T15:49:26.018 INFO:tasks.workunit.client.0.smithi016.stdout:9/34: symlink l8 0 2024-04-22T15:49:26.018 INFO:tasks.workunit.client.0.smithi016.stdout:9/35: chown f7 9819035 1 2024-04-22T15:49:26.019 INFO:tasks.workunit.client.0.smithi016.stdout:9/36: stat l0 0 2024-04-22T15:49:26.021 INFO:tasks.workunit.client.0.smithi016.stdout:0/27: fsync f4 0 2024-04-22T15:49:26.022 INFO:tasks.workunit.client.0.smithi016.stdout:2/28: mknod d2/d9/cd 0 2024-04-22T15:49:26.026 INFO:tasks.workunit.client.0.smithi016.stdout:1/19: mkdir d3/d4 0 2024-04-22T15:49:26.027 INFO:tasks.workunit.client.0.smithi016.stdout:7/31: link d3/d4/f6 d3/fa 0 2024-04-22T15:49:26.031 INFO:tasks.workunit.client.0.smithi016.stdout:6/34: getdents d3 0 2024-04-22T15:49:26.031 INFO:tasks.workunit.client.0.smithi016.stdout:6/35: write f2 [881323,99416] 0 2024-04-22T15:49:26.032 INFO:tasks.workunit.client.0.smithi016.stdout:1/20: unlink f2 0 2024-04-22T15:49:26.034 INFO:tasks.workunit.client.0.smithi016.stdout:7/32: unlink f2 0 2024-04-22T15:49:26.037 INFO:tasks.workunit.client.0.smithi016.stdout:7/33: rmdir d3 39 2024-04-22T15:49:26.038 INFO:tasks.workunit.client.0.smithi016.stdout:6/36: dwrite f2 [4194304,4194304] 0 2024-04-22T15:49:26.039 INFO:tasks.workunit.client.0.smithi016.stdout:6/37: chown d3/c5 381543 1 2024-04-22T15:49:26.039 INFO:tasks.workunit.client.0.smithi016.stdout:1/21: symlink d3/d4/l5 0 2024-04-22T15:49:26.043 INFO:tasks.workunit.client.0.smithi016.stdout:9/37: truncate f2 2390815 0 2024-04-22T15:49:26.045 INFO:tasks.workunit.client.0.smithi016.stdout:1/22: creat d3/d4/f6 x:0 0 0 2024-04-22T15:49:26.046 INFO:tasks.workunit.client.0.smithi016.stdout:6/38: getdents d3 0 2024-04-22T15:49:26.050 INFO:tasks.workunit.client.0.smithi016.stdout:1/23: creat d3/f7 x:0 0 0 2024-04-22T15:49:26.051 INFO:tasks.workunit.client.0.smithi016.stdout:7/34: link d3/fa d3/d4/d9/fb 0 2024-04-22T15:49:26.052 INFO:tasks.workunit.client.0.smithi016.stdout:1/24: creat d3/f8 x:0 0 0 2024-04-22T15:49:26.054 INFO:tasks.workunit.client.0.smithi016.stdout:1/25: creat d3/f9 x:0 0 0 2024-04-22T15:49:26.056 INFO:tasks.workunit.client.0.smithi016.stdout:1/26: mkdir d3/d4/da 0 2024-04-22T15:49:26.057 INFO:tasks.workunit.client.0.smithi016.stdout:1/27: dread - d3/f9 zero size 2024-04-22T15:49:26.058 INFO:tasks.workunit.client.0.smithi016.stdout:7/35: dwrite f0 [0,4194304] 0 2024-04-22T15:49:26.061 INFO:tasks.workunit.client.0.smithi016.stdout:7/36: mkdir d3/dc 0 2024-04-22T15:49:26.067 INFO:tasks.workunit.client.0.smithi016.stdout:1/28: mkdir d3/d4/da/db 0 2024-04-22T15:49:26.072 INFO:tasks.workunit.client.0.smithi016.stdout:9/38: dread f2 [0,4194304] 0 2024-04-22T15:49:26.075 INFO:tasks.workunit.client.0.smithi016.stdout:1/29: mknod d3/d4/da/db/cc 0 2024-04-22T15:49:26.075 INFO:tasks.workunit.client.0.smithi016.stdout:1/30: read - d3/f9 zero size 2024-04-22T15:49:26.077 INFO:tasks.workunit.client.0.smithi016.stdout:1/31: write d3/f7 [387334,113278] 0 2024-04-22T15:49:26.078 INFO:tasks.workunit.client.0.smithi016.stdout:1/32: fdatasync d3/d4/f6 0 2024-04-22T15:49:26.078 INFO:tasks.workunit.client.0.smithi016.stdout:9/39: mkdir d9 0 2024-04-22T15:49:26.082 INFO:tasks.workunit.client.0.smithi016.stdout:1/33: unlink d3/f9 0 2024-04-22T15:49:26.083 INFO:tasks.workunit.client.0.smithi016.stdout:1/34: chown d3/d4/da 8146 1 2024-04-22T15:49:26.084 INFO:tasks.workunit.client.0.smithi016.stdout:7/37: chown d3/d4 242026216 1 2024-04-22T15:49:26.093 INFO:tasks.workunit.client.0.smithi016.stdout:9/40: dwrite f2 [0,4194304] 0 2024-04-22T15:49:26.095 INFO:tasks.workunit.client.0.smithi016.stdout:7/38: creat d3/d4/fd x:0 0 0 2024-04-22T15:49:26.095 INFO:tasks.workunit.client.0.smithi016.stdout:1/35: getdents d3/d4/da/db 0 2024-04-22T15:49:26.096 INFO:tasks.workunit.client.0.smithi016.stdout:1/36: dread - d3/d4/f6 zero size 2024-04-22T15:49:26.097 INFO:tasks.workunit.client.0.smithi016.stdout:1/37: write d3/d4/f6 [768221,110589] 0 2024-04-22T15:49:26.099 INFO:tasks.workunit.client.0.smithi016.stdout:9/41: creat d9/fa x:0 0 0 2024-04-22T15:49:26.100 INFO:tasks.workunit.client.0.smithi016.stdout:1/38: symlink d3/d4/da/ld 0 2024-04-22T15:49:26.100 INFO:tasks.workunit.client.0.smithi016.stdout:9/42: write f7 [734705,58495] 0 2024-04-22T15:49:26.101 INFO:tasks.workunit.client.0.smithi016.stdout:9/43: write f2 [3113001,51051] 0 2024-04-22T15:49:26.104 INFO:tasks.workunit.client.0.smithi016.stdout:1/39: dread d3/f7 [0,4194304] 0 2024-04-22T15:49:26.105 INFO:tasks.workunit.client.0.smithi016.stdout:1/40: readlink d3/d4/da/ld 0 2024-04-22T15:49:26.106 INFO:tasks.workunit.client.0.smithi016.stdout:9/44: creat d9/fb x:0 0 0 2024-04-22T15:49:26.106 INFO:tasks.workunit.client.0.smithi016.stdout:9/45: chown l5 56430129 1 2024-04-22T15:49:26.107 INFO:tasks.workunit.client.0.smithi016.stdout:9/46: chown l5 1254 1 2024-04-22T15:49:26.107 INFO:tasks.workunit.client.0.smithi016.stdout:9/47: chown d9/fb 883533 1 2024-04-22T15:49:26.110 INFO:tasks.workunit.client.0.smithi016.stdout:1/41: dread d3/d4/f6 [0,4194304] 0 2024-04-22T15:49:26.110 INFO:tasks.workunit.client.0.smithi016.stdout:9/48: fsync d9/fa 0 2024-04-22T15:49:26.114 INFO:tasks.workunit.client.0.smithi016.stdout:1/42: creat d3/d4/fe x:0 0 0 2024-04-22T15:49:26.115 INFO:tasks.workunit.client.0.smithi016.stdout:7/39: dwrite d3/d4/fd [0,4194304] 0 2024-04-22T15:49:26.115 INFO:tasks.workunit.client.0.smithi016.stdout:1/43: chown d3/f7 7429151 1 2024-04-22T15:49:26.116 INFO:tasks.workunit.client.0.smithi016.stdout:1/44: dread d3/f7 [0,4194304] 0 2024-04-22T15:49:26.118 INFO:tasks.workunit.client.0.smithi016.stdout:9/49: link d9/fb d9/fc 0 2024-04-22T15:49:26.118 INFO:tasks.workunit.client.0.smithi016.stdout:1/45: dread d3/f7 [0,4194304] 0 2024-04-22T15:49:26.119 INFO:tasks.workunit.client.0.smithi016.stdout:1/46: truncate d3/f7 1005177 0 2024-04-22T15:49:26.128 INFO:tasks.workunit.client.0.smithi016.stdout:9/50: dwrite f2 [0,4194304] 0 2024-04-22T15:49:26.129 INFO:tasks.workunit.client.0.smithi016.stdout:9/51: chown l0 12330779 1 2024-04-22T15:49:26.132 INFO:tasks.workunit.client.0.smithi016.stdout:9/52: write f7 [1764404,72283] 0 2024-04-22T15:49:26.134 INFO:tasks.workunit.client.0.smithi016.stdout:1/47: dwrite d3/f7 [0,4194304] 0 2024-04-22T15:49:26.136 INFO:tasks.workunit.client.0.smithi016.stdout:7/40: dwrite d3/d4/d9/fb [0,4194304] 0 2024-04-22T15:49:26.137 INFO:tasks.workunit.client.0.smithi016.stdout:1/48: rename d3/d4/da to d3/d4/da/db/df 22 2024-04-22T15:49:26.138 INFO:tasks.workunit.client.0.smithi016.stdout:9/53: symlink d9/ld 0 2024-04-22T15:49:26.138 INFO:tasks.workunit.client.0.smithi016.stdout:1/49: write d3/f8 [781297,94404] 0 2024-04-22T15:49:26.140 INFO:tasks.workunit.client.0.smithi016.stdout:9/54: write f7 [2827493,30743] 0 2024-04-22T15:49:26.141 INFO:tasks.workunit.client.0.smithi016.stdout:9/55: fsync d9/fa 0 2024-04-22T15:49:26.145 INFO:tasks.workunit.client.0.smithi016.stdout:9/56: unlink d9/ld 0 2024-04-22T15:49:26.149 INFO:tasks.workunit.client.0.smithi016.stdout:9/57: fsync d9/fb 0 2024-04-22T15:49:26.150 INFO:tasks.workunit.client.0.smithi016.stdout:1/50: dwrite d3/d4/fe [0,4194304] 0 2024-04-22T15:49:26.150 INFO:tasks.workunit.client.0.smithi016.stdout:9/58: dread - d9/fb zero size 2024-04-22T15:49:26.151 INFO:tasks.workunit.client.0.smithi016.stdout:9/59: readlink l5 0 2024-04-22T15:49:26.153 INFO:tasks.workunit.client.0.smithi016.stdout:9/60: mkdir d9/de 0 2024-04-22T15:49:26.158 INFO:tasks.workunit.client.0.smithi016.stdout:1/51: fsync d3/f7 0 2024-04-22T15:49:26.160 INFO:tasks.workunit.client.0.smithi016.stdout:7/41: dwrite d3/d4/f7 [0,4194304] 0 2024-04-22T15:49:26.161 INFO:tasks.workunit.client.0.smithi016.stdout:1/52: rmdir d3/d4/da 39 2024-04-22T15:49:26.164 INFO:tasks.workunit.client.0.smithi016.stdout:9/61: getdents d9 0 2024-04-22T15:49:26.165 INFO:tasks.workunit.client.0.smithi016.stdout:9/62: fsync f2 0 2024-04-22T15:49:26.168 INFO:tasks.workunit.client.0.smithi016.stdout:1/53: dwrite d3/d4/fe [0,4194304] 0 2024-04-22T15:49:26.171 INFO:tasks.workunit.client.0.smithi016.stdout:9/63: creat d9/ff x:0 0 0 2024-04-22T15:49:26.172 INFO:tasks.workunit.client.0.smithi016.stdout:9/64: read - d9/fa zero size 2024-04-22T15:49:26.174 INFO:tasks.workunit.client.0.smithi016.stdout:7/42: getdents d3 0 2024-04-22T15:49:26.175 INFO:tasks.workunit.client.0.smithi016.stdout:1/54: mknod d3/d4/da/db/c10 0 2024-04-22T15:49:26.176 INFO:tasks.workunit.client.0.smithi016.stdout:9/65: symlink d9/l10 0 2024-04-22T15:49:26.184 INFO:tasks.workunit.client.0.smithi016.stdout:7/43: dwrite d3/d4/d9/fb [0,4194304] 0 2024-04-22T15:49:26.185 INFO:tasks.workunit.client.0.smithi016.stdout:1/55: dwrite d3/f8 [0,4194304] 0 2024-04-22T15:49:26.186 INFO:tasks.workunit.client.0.smithi016.stdout:9/66: dread f2 [0,4194304] 0 2024-04-22T15:49:26.188 INFO:tasks.workunit.client.0.smithi016.stdout:7/44: mkdir d3/de 0 2024-04-22T15:49:26.194 INFO:tasks.workunit.client.0.smithi016.stdout:9/67: write d9/fb [426999,120500] 0 2024-04-22T15:49:26.195 INFO:tasks.workunit.client.0.smithi016.stdout:1/56: truncate d3/f7 407840 0 2024-04-22T15:49:26.195 INFO:tasks.workunit.client.0.smithi016.stdout:9/68: readlink d9/l10 0 2024-04-22T15:49:26.196 INFO:tasks.workunit.client.0.smithi016.stdout:7/45: mkdir d3/d4/d9/df 0 2024-04-22T15:49:26.196 INFO:tasks.workunit.client.0.smithi016.stdout:1/57: write d3/f8 [2250588,57192] 0 2024-04-22T15:49:26.197 INFO:tasks.workunit.client.0.smithi016.stdout:7/46: chown d3/c8 735976974 1 2024-04-22T15:49:26.198 INFO:tasks.workunit.client.0.smithi016.stdout:9/69: write f7 [1021031,112094] 0 2024-04-22T15:49:26.199 INFO:tasks.workunit.client.0.smithi016.stdout:7/47: fsync d3/d4/f7 0 2024-04-22T15:49:26.199 INFO:tasks.workunit.client.0.smithi016.stdout:9/70: stat d9/l10 0 2024-04-22T15:49:26.199 INFO:tasks.workunit.client.0.smithi016.stdout:1/58: write d3/d4/f6 [942056,29629] 0 2024-04-22T15:49:26.200 INFO:tasks.workunit.client.0.smithi016.stdout:1/59: chown d3/d4/da/db/c10 788453 1 2024-04-22T15:49:26.203 INFO:tasks.workunit.client.0.smithi016.stdout:7/48: fdatasync d3/d4/f6 0 2024-04-22T15:49:26.203 INFO:tasks.workunit.client.0.smithi016.stdout:1/60: fdatasync d3/d4/f6 0 2024-04-22T15:49:26.205 INFO:tasks.workunit.client.0.smithi016.stdout:9/71: creat d9/de/f11 x:0 0 0 2024-04-22T15:49:26.207 INFO:tasks.workunit.client.0.smithi016.stdout:7/49: creat d3/d4/d9/f10 x:0 0 0 2024-04-22T15:49:26.209 INFO:tasks.workunit.client.0.smithi016.stdout:1/61: creat d3/d4/da/f11 x:0 0 0 2024-04-22T15:49:26.209 INFO:tasks.workunit.client.0.smithi016.stdout:9/72: symlink d9/l12 0 2024-04-22T15:49:26.210 INFO:tasks.workunit.client.0.smithi016.stdout:9/73: read d9/fc [83095,64222] 0 2024-04-22T15:49:26.212 INFO:tasks.workunit.client.0.smithi016.stdout:7/50: creat d3/d4/d9/df/f11 x:0 0 0 2024-04-22T15:49:26.213 INFO:tasks.workunit.client.0.smithi016.stdout:9/74: dread d9/fc [0,4194304] 0 2024-04-22T15:49:26.213 INFO:tasks.workunit.client.0.smithi016.stdout:1/62: mknod d3/d4/c12 0 2024-04-22T15:49:26.214 INFO:tasks.workunit.client.0.smithi016.stdout:9/75: chown d9/fa 259444507 1 2024-04-22T15:49:26.214 INFO:tasks.workunit.client.0.smithi016.stdout:7/51: symlink d3/l12 0 2024-04-22T15:49:26.214 INFO:tasks.workunit.client.0.smithi016.stdout:9/76: write d9/ff [34044,18282] 0 2024-04-22T15:49:26.215 INFO:tasks.workunit.client.0.smithi016.stdout:7/52: chown d3/fa 395692 1 2024-04-22T15:49:26.216 INFO:tasks.workunit.client.0.smithi016.stdout:9/77: dread - d9/fa zero size 2024-04-22T15:49:26.217 INFO:tasks.workunit.client.0.smithi016.stdout:9/78: write d9/de/f11 [1030347,114954] 0 2024-04-22T15:49:26.218 INFO:tasks.workunit.client.0.smithi016.stdout:9/79: dread d9/fb [0,4194304] 0 2024-04-22T15:49:26.219 INFO:tasks.workunit.client.0.smithi016.stdout:9/80: dread - d9/fa zero size 2024-04-22T15:49:26.220 INFO:tasks.workunit.client.0.smithi016.stdout:9/81: write f2 [1102265,12052] 0 2024-04-22T15:49:26.220 INFO:tasks.workunit.client.0.smithi016.stdout:7/53: link f0 d3/d4/d9/f13 0 2024-04-22T15:49:26.220 INFO:tasks.workunit.client.0.smithi016.stdout:9/82: truncate d9/fc 1135214 0 2024-04-22T15:49:26.221 INFO:tasks.workunit.client.0.smithi016.stdout:9/83: chown d9/de/f11 2289777 1 2024-04-22T15:49:26.222 INFO:tasks.workunit.client.0.smithi016.stdout:9/84: write f2 [3813914,117222] 0 2024-04-22T15:49:26.222 INFO:tasks.workunit.client.0.smithi016.stdout:7/54: mkdir d3/d14 0 2024-04-22T15:49:26.223 INFO:tasks.workunit.client.0.smithi016.stdout:9/85: write f7 [3273665,103194] 0 2024-04-22T15:49:26.223 INFO:tasks.workunit.client.0.smithi016.stdout:7/55: dread - d3/d4/d9/f10 zero size 2024-04-22T15:49:26.225 INFO:tasks.workunit.client.0.smithi016.stdout:1/63: rmdir d3/d4/da 39 2024-04-22T15:49:26.227 INFO:tasks.workunit.client.0.smithi016.stdout:9/86: symlink d9/l13 0 2024-04-22T15:49:26.228 INFO:tasks.workunit.client.0.smithi016.stdout:7/56: mkdir d3/d4/d9/d15 0 2024-04-22T15:49:26.232 INFO:tasks.workunit.client.0.smithi016.stdout:1/64: unlink d3/d4/da/f11 0 2024-04-22T15:49:26.236 INFO:tasks.workunit.client.0.smithi016.stdout:1/65: dread d3/f7 [0,4194304] 0 2024-04-22T15:49:26.236 INFO:tasks.workunit.client.0.smithi016.stdout:9/87: dwrite d9/fc [0,4194304] 0 2024-04-22T15:49:26.240 INFO:tasks.workunit.client.0.smithi016.stdout:1/66: unlink d3/d4/f6 0 2024-04-22T15:49:26.241 INFO:tasks.workunit.client.0.smithi016.stdout:9/88: creat d9/f14 x:0 0 0 2024-04-22T15:49:26.242 INFO:tasks.workunit.client.0.smithi016.stdout:9/89: write d9/de/f11 [2131813,47994] 0 2024-04-22T15:49:26.246 INFO:tasks.workunit.client.0.smithi016.stdout:1/67: dwrite d3/d4/fe [0,4194304] 0 2024-04-22T15:49:26.250 INFO:tasks.workunit.client.0.smithi016.stdout:1/68: truncate d3/f7 161514 0 2024-04-22T15:49:26.255 INFO:tasks.workunit.client.0.smithi016.stdout:9/90: rmdir d9 39 2024-04-22T15:49:26.257 INFO:tasks.workunit.client.0.smithi016.stdout:9/91: write d9/fc [2820124,118678] 0 2024-04-22T15:49:26.258 INFO:tasks.workunit.client.0.smithi016.stdout:9/92: chown d9/l10 131504 1 2024-04-22T15:49:26.266 INFO:tasks.workunit.client.0.smithi016.stdout:9/93: truncate d9/de/f11 340383 0 2024-04-22T15:49:26.269 INFO:tasks.workunit.client.0.smithi016.stdout:1/69: dwrite d3/f7 [0,4194304] 0 2024-04-22T15:49:26.270 INFO:tasks.workunit.client.0.smithi016.stdout:9/94: rename d9/fb to d9/f15 0 2024-04-22T15:49:26.270 INFO:tasks.workunit.client.0.smithi016.stdout:9/95: chown d9/f14 7 1 2024-04-22T15:49:26.275 INFO:tasks.workunit.client.0.smithi016.stdout:9/96: creat d9/de/f16 x:0 0 0 2024-04-22T15:49:26.276 INFO:tasks.workunit.client.0.smithi016.stdout:1/70: symlink d3/d4/l13 0 2024-04-22T15:49:26.277 INFO:tasks.workunit.client.0.smithi016.stdout:1/71: write d3/d4/fe [3196759,128067] 0 2024-04-22T15:49:26.281 INFO:tasks.workunit.client.0.smithi016.stdout:9/97: fsync d9/de/f16 0 2024-04-22T15:49:26.282 INFO:tasks.workunit.client.0.smithi016.stdout:1/72: dread d3/f8 [0,4194304] 0 2024-04-22T15:49:26.286 INFO:tasks.workunit.client.0.smithi016.stdout:1/73: rename d3/d4/da/db/cc to d3/c14 0 2024-04-22T15:49:26.290 INFO:tasks.workunit.client.0.smithi016.stdout:1/74: mkdir d3/d4/d15 0 2024-04-22T15:49:26.293 INFO:tasks.workunit.client.0.smithi016.stdout:1/75: creat d3/d4/d15/f16 x:0 0 0 2024-04-22T15:49:26.298 INFO:tasks.workunit.client.0.smithi016.stdout:9/98: dwrite d9/de/f11 [0,4194304] 0 2024-04-22T15:49:26.299 INFO:tasks.workunit.client.0.smithi016.stdout:1/76: link d3/d4/da/db/c10 d3/d4/da/db/c17 0 2024-04-22T15:49:26.299 INFO:tasks.workunit.client.0.smithi016.stdout:9/99: write f2 [225573,68174] 0 2024-04-22T15:49:26.308 INFO:tasks.workunit.client.0.smithi016.stdout:1/77: dwrite d3/f7 [0,4194304] 0 2024-04-22T15:49:26.311 INFO:tasks.workunit.client.0.smithi016.stdout:1/78: mkdir d3/d4/da/db/d18 0 2024-04-22T15:49:26.311 INFO:tasks.workunit.client.0.smithi016.stdout:9/100: dwrite d9/de/f11 [4194304,4194304] 0 2024-04-22T15:49:26.313 INFO:tasks.workunit.client.0.smithi016.stdout:9/101: write d9/de/f16 [495427,9627] 0 2024-04-22T15:49:26.328 INFO:tasks.workunit.client.0.smithi016.stdout:1/79: rename d3/d4/da/db to d3/d4/d19 0 2024-04-22T15:49:26.331 INFO:tasks.workunit.client.0.smithi016.stdout:1/80: link d3/d4/l5 d3/d4/d19/l1a 0 2024-04-22T15:49:26.333 INFO:tasks.workunit.client.0.smithi016.stdout:1/81: symlink d3/d4/l1b 0 2024-04-22T15:49:26.337 INFO:tasks.workunit.client.0.smithi016.stdout:1/82: rmdir d3/d4/d19/d18 0 2024-04-22T15:49:26.339 INFO:tasks.workunit.client.0.smithi016.stdout:1/83: creat d3/d4/f1c x:0 0 0 2024-04-22T15:49:26.342 INFO:tasks.workunit.client.0.smithi016.stdout:1/84: dread d3/f8 [0,4194304] 0 2024-04-22T15:49:26.349 INFO:tasks.workunit.client.0.smithi016.stdout:1/85: fsync d3/d4/f1c 0 2024-04-22T15:49:26.350 INFO:tasks.workunit.client.0.smithi016.stdout:1/86: read d3/d4/fe [2034929,73498] 0 2024-04-22T15:49:26.351 INFO:tasks.workunit.client.0.smithi016.stdout:1/87: write d3/d4/f1c [267087,41076] 0 2024-04-22T15:49:26.356 INFO:tasks.workunit.client.0.smithi016.stdout:1/88: creat d3/d4/d19/f1d x:0 0 0 2024-04-22T15:49:26.362 INFO:tasks.workunit.client.0.smithi016.stdout:1/89: creat d3/d4/f1e x:0 0 0 2024-04-22T15:49:26.363 INFO:tasks.workunit.client.0.smithi016.stdout:1/90: write d3/f7 [4463780,129477] 0 2024-04-22T15:49:26.363 INFO:tasks.workunit.client.0.smithi016.stdout:1/91: chown d3/d4 2789317 1 2024-04-22T15:49:26.366 INFO:tasks.workunit.client.0.smithi016.stdout:1/92: symlink d3/d4/l1f 0 2024-04-22T15:49:26.369 INFO:tasks.workunit.client.0.smithi016.stdout:1/93: dread d3/f8 [0,4194304] 0 2024-04-22T15:49:26.371 INFO:tasks.workunit.client.0.smithi016.stdout:1/94: mkdir d3/d4/d19/d20 0 2024-04-22T15:49:26.449 INFO:tasks.workunit.client.0.smithi016.stdout:1/95: dwrite d3/f7 [0,4194304] 0 2024-04-22T15:49:26.457 INFO:tasks.workunit.client.0.smithi016.stdout:1/96: dwrite d3/d4/d15/f16 [0,4194304] 0 2024-04-22T15:49:26.458 INFO:tasks.workunit.client.0.smithi016.stdout:1/97: fsync d3/d4/f1c 0 2024-04-22T15:49:26.462 INFO:tasks.workunit.client.0.smithi016.stdout:1/98: chown d3/d4/d15/f16 36 1 2024-04-22T15:49:26.462 INFO:tasks.workunit.client.0.smithi016.stdout:1/99: chown d3/d4/l5 558 1 2024-04-22T15:49:26.463 INFO:tasks.workunit.client.0.smithi016.stdout:1/100: stat d3/d4/d19 0 2024-04-22T15:49:26.470 INFO:tasks.workunit.client.0.smithi016.stdout:1/101: creat d3/d4/d19/d20/f21 x:0 0 0 2024-04-22T15:49:26.470 INFO:tasks.workunit.client.0.smithi016.stdout:1/102: dread - d3/d4/d19/d20/f21 zero size 2024-04-22T15:49:26.471 INFO:tasks.workunit.client.0.smithi016.stdout:1/103: write d3/d4/d19/d20/f21 [217387,47608] 0 2024-04-22T15:49:26.473 INFO:tasks.workunit.client.0.smithi016.stdout:1/104: write d3/d4/f1e [729067,16163] 0 2024-04-22T15:49:26.483 INFO:tasks.workunit.client.0.smithi016.stdout:1/105: link d3/d4/da/ld d3/d4/l22 0 2024-04-22T15:49:26.484 INFO:tasks.workunit.client.0.smithi016.stdout:1/106: write d3/d4/f1e [891558,8625] 0 2024-04-22T15:49:26.487 INFO:tasks.workunit.client.0.smithi016.stdout:1/107: write d3/f8 [1791602,102474] 0 2024-04-22T15:49:26.490 INFO:tasks.workunit.client.0.smithi016.stdout:1/108: creat d3/f23 x:0 0 0 2024-04-22T15:49:26.491 INFO:tasks.workunit.client.0.smithi016.stdout:1/109: write d3/f23 [408150,127657] 0 2024-04-22T15:49:26.495 INFO:tasks.workunit.client.0.smithi016.stdout:1/110: dread d3/d4/d15/f16 [0,4194304] 0 2024-04-22T15:49:26.501 INFO:tasks.workunit.client.0.smithi016.stdout:1/111: creat d3/d4/f24 x:0 0 0 2024-04-22T15:49:26.504 INFO:tasks.workunit.client.0.smithi016.stdout:1/112: read d3/d4/fe [3897835,42444] 0 2024-04-22T15:49:26.504 INFO:tasks.workunit.client.0.smithi016.stdout:1/113: write d3/d4/d19/f1d [631315,89335] 0 2024-04-22T15:49:26.512 INFO:tasks.workunit.client.0.smithi016.stdout:1/114: dwrite d3/d4/d15/f16 [0,4194304] 0 2024-04-22T15:49:26.518 INFO:tasks.workunit.client.0.smithi016.stdout:1/115: rename d3/d4/c12 to d3/d4/d15/c25 0 2024-04-22T15:49:26.521 INFO:tasks.workunit.client.0.smithi016.stdout:1/116: dread d3/f8 [0,4194304] 0 2024-04-22T15:49:26.565 INFO:tasks.workunit.client.0.smithi016.stdout:1/117: getdents d3/d4 0 2024-04-22T15:49:26.567 INFO:tasks.workunit.client.0.smithi016.stdout:1/118: creat d3/d4/da/f26 x:0 0 0 2024-04-22T15:49:26.573 INFO:tasks.workunit.client.0.smithi016.stdout:1/119: dwrite d3/d4/f1e [0,4194304] 0 2024-04-22T15:49:26.575 INFO:tasks.workunit.client.0.smithi016.stdout:1/120: read d3/f8 [4177022,110059] 0 2024-04-22T15:49:26.581 INFO:tasks.workunit.client.0.smithi016.stdout:1/121: getdents d3/d4/d19 0 2024-04-22T15:49:26.583 INFO:tasks.workunit.client.0.smithi016.stdout:1/122: chown d3/d4/da/ld 0 1 2024-04-22T15:49:26.584 INFO:tasks.workunit.client.0.smithi016.stdout:1/123: mknod d3/c27 0 2024-04-22T15:49:26.592 INFO:tasks.workunit.client.0.smithi016.stdout:1/124: rmdir d3 39 2024-04-22T15:49:26.593 INFO:tasks.workunit.client.0.smithi016.stdout:1/125: chown d3/d4/d19/d20/f21 7978973 1 2024-04-22T15:49:26.597 INFO:tasks.workunit.client.0.smithi016.stdout:1/126: dread d3/f8 [0,4194304] 0 2024-04-22T15:49:26.598 INFO:tasks.workunit.client.0.smithi016.stdout:1/127: mkdir d3/d28 0 2024-04-22T15:49:26.599 INFO:tasks.workunit.client.0.smithi016.stdout:1/128: truncate d3/d4/f1e 4235898 0 2024-04-22T15:49:26.603 INFO:tasks.workunit.client.0.smithi016.stdout:1/129: getdents d3/d4/da 0 2024-04-22T15:49:26.603 INFO:tasks.workunit.client.0.smithi016.stdout:1/130: stat d3/d4 0 2024-04-22T15:49:26.604 INFO:tasks.workunit.client.0.smithi016.stdout:1/131: creat d3/f29 x:0 0 0 2024-04-22T15:49:26.606 INFO:tasks.workunit.client.0.smithi016.stdout:1/132: rmdir d3/d4/d19/d20 39 2024-04-22T15:49:26.610 INFO:tasks.workunit.client.0.smithi016.stdout:1/133: truncate d3/d4/f1c 144911 0 2024-04-22T15:49:26.612 INFO:tasks.workunit.client.0.smithi016.stdout:1/134: mknod d3/d4/da/c2a 0 2024-04-22T15:49:26.613 INFO:tasks.workunit.client.0.smithi016.stdout:1/135: rmdir d3/d4/d15 39 2024-04-22T15:49:26.621 INFO:tasks.workunit.client.0.smithi016.stdout:1/136: mkdir d3/d4/d19/d20/d2b 0 2024-04-22T15:49:26.628 INFO:tasks.workunit.client.0.smithi016.stdout:1/137: dwrite d3/d4/f1e [0,4194304] 0 2024-04-22T15:49:26.629 INFO:tasks.workunit.client.0.smithi016.stdout:1/138: write d3/f29 [187917,104699] 0 2024-04-22T15:49:26.631 INFO:tasks.workunit.client.0.smithi016.stdout:1/139: fdatasync d3/d4/d15/f16 0 2024-04-22T15:49:26.633 INFO:tasks.workunit.client.0.smithi016.stdout:1/140: rmdir d3/d4/d19 39 2024-04-22T15:49:26.633 INFO:tasks.workunit.client.0.smithi016.stdout:1/141: write d3/f29 [564976,69917] 0 2024-04-22T15:49:26.638 INFO:tasks.workunit.client.0.smithi016.stdout:1/142: mknod d3/d4/d19/d20/d2b/c2c 0 2024-04-22T15:49:26.642 INFO:tasks.workunit.client.0.smithi016.stdout:1/143: creat d3/d4/da/f2d x:0 0 0 2024-04-22T15:49:26.653 INFO:tasks.workunit.client.0.smithi016.stdout:1/144: write d3/d4/f1c [473106,81896] 0 2024-04-22T15:49:26.655 INFO:tasks.workunit.client.0.smithi016.stdout:1/145: mkdir d3/d2e 0 2024-04-22T15:49:26.656 INFO:tasks.workunit.client.0.smithi016.stdout:1/146: chown d3/d4/da 1150 1 2024-04-22T15:49:26.658 INFO:tasks.workunit.client.0.smithi016.stdout:1/147: unlink d3/c14 0 2024-04-22T15:49:26.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:26 smithi016 ceph-mon[26620]: pgmap v25: 65 pgs: 65 active+clean; 188 MiB data, 781 MiB used, 536 GiB / 536 GiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 279 op/s 2024-04-22T15:49:26.670 INFO:tasks.workunit.client.0.smithi016.stdout:1/148: dwrite d3/d4/f1e [0,4194304] 0 2024-04-22T15:49:26.782 INFO:tasks.workunit.client.0.smithi016.stdout:1/149: dwrite d3/f8 [4194304,4194304] 0 2024-04-22T15:49:26.784 INFO:tasks.workunit.client.0.smithi016.stdout:1/150: symlink d3/l2f 0 2024-04-22T15:49:26.786 INFO:tasks.workunit.client.0.smithi016.stdout:1/151: symlink d3/d4/d15/l30 0 2024-04-22T15:49:26.789 INFO:tasks.workunit.client.0.smithi016.stdout:1/152: mknod d3/d28/c31 0 2024-04-22T15:49:26.792 INFO:tasks.workunit.client.0.smithi016.stdout:1/153: mkdir d3/d2e/d32 0 2024-04-22T15:49:26.795 INFO:tasks.workunit.client.0.smithi016.stdout:1/154: dread d3/f29 [0,4194304] 0 2024-04-22T15:49:26.796 INFO:tasks.workunit.client.0.smithi016.stdout:1/155: dread d3/f29 [0,4194304] 0 2024-04-22T15:49:26.802 INFO:tasks.workunit.client.0.smithi016.stdout:1/156: dwrite d3/d4/da/f2d [0,4194304] 0 2024-04-22T15:49:26.810 INFO:tasks.workunit.client.0.smithi016.stdout:1/157: dwrite d3/d4/d19/f1d [0,4194304] 0 2024-04-22T15:49:26.819 INFO:tasks.workunit.client.0.smithi016.stdout:1/158: dwrite d3/d4/f1e [4194304,4194304] 0 2024-04-22T15:49:26.823 INFO:tasks.workunit.client.0.smithi016.stdout:1/159: creat d3/d4/d19/d20/f33 x:0 0 0 2024-04-22T15:49:26.826 INFO:tasks.workunit.client.0.smithi016.stdout:1/160: mknod d3/c34 0 2024-04-22T15:49:26.827 INFO:tasks.workunit.client.0.smithi016.stdout:1/161: dread - d3/d4/d19/d20/f33 zero size 2024-04-22T15:49:26.829 INFO:tasks.workunit.client.0.smithi016.stdout:1/162: dread d3/d4/d19/d20/f21 [0,4194304] 0 2024-04-22T15:49:26.833 INFO:tasks.workunit.client.0.smithi016.stdout:1/163: creat d3/d4/d19/f35 x:0 0 0 2024-04-22T15:49:26.835 INFO:tasks.workunit.client.0.smithi016.stdout:1/164: symlink d3/l36 0 2024-04-22T15:49:26.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:26 smithi027 ceph-mon[34922]: pgmap v25: 65 pgs: 65 active+clean; 188 MiB data, 781 MiB used, 536 GiB / 536 GiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 279 op/s 2024-04-22T15:49:26.836 INFO:tasks.workunit.client.0.smithi016.stdout:1/165: chown d3/d4/l22 19909527 1 2024-04-22T15:49:26.941 INFO:tasks.workunit.client.0.smithi016.stdout:1/166: dwrite d3/d4/d19/f1d [4194304,4194304] 0 2024-04-22T15:49:26.943 INFO:tasks.workunit.client.0.smithi016.stdout:1/167: dread d3/f29 [0,4194304] 0 2024-04-22T15:49:26.948 INFO:tasks.workunit.client.0.smithi016.stdout:1/168: unlink d3/d4/l1f 0 2024-04-22T15:49:26.949 INFO:tasks.workunit.client.0.smithi016.stdout:1/169: chown d3/d2e 926794 1 2024-04-22T15:49:26.953 INFO:tasks.workunit.client.0.smithi016.stdout:1/170: fdatasync d3/d4/fe 0 2024-04-22T15:49:26.954 INFO:tasks.workunit.client.0.smithi016.stdout:1/171: write d3/d4/da/f26 [342048,75523] 0 2024-04-22T15:49:26.955 INFO:tasks.workunit.client.0.smithi016.stdout:1/172: readlink d3/d4/l5 0 2024-04-22T15:49:26.956 INFO:tasks.workunit.client.0.smithi016.stdout:1/173: fsync d3/d4/da/f2d 0 2024-04-22T15:49:26.957 INFO:tasks.workunit.client.0.smithi016.stdout:1/174: stat d3/d4/d19/d20/f21 0 2024-04-22T15:49:26.961 INFO:tasks.workunit.client.0.smithi016.stdout:1/175: mkdir d3/d4/d37 0 2024-04-22T15:49:26.963 INFO:tasks.workunit.client.0.smithi016.stdout:1/176: mkdir d3/d4/d15/d38 0 2024-04-22T15:49:26.969 INFO:tasks.workunit.client.0.smithi016.stdout:1/177: truncate d3/d4/f1e 5567112 0 2024-04-22T15:49:26.973 INFO:tasks.workunit.client.0.smithi016.stdout:1/178: rename d3/d4/d15/c25 to d3/d4/d19/d20/d2b/c39 0 2024-04-22T15:49:26.979 INFO:tasks.workunit.client.0.smithi016.stdout:1/179: rmdir d3/d4/d19 39 2024-04-22T15:49:26.982 INFO:tasks.workunit.client.0.smithi016.stdout:1/180: unlink d3/d4/l22 0 2024-04-22T15:49:26.992 INFO:tasks.workunit.client.0.smithi016.stdout:1/181: truncate d3/d4/f1e 5714380 0 2024-04-22T15:49:27.004 INFO:tasks.workunit.client.0.smithi016.stdout:1/182: dwrite d3/d4/f1e [4194304,4194304] 0 2024-04-22T15:49:27.006 INFO:tasks.workunit.client.0.smithi016.stdout:1/183: mkdir d3/d4/d3a 0 2024-04-22T15:49:27.008 INFO:tasks.workunit.client.0.smithi016.stdout:1/184: dread d3/d4/d19/d20/f21 [0,4194304] 0 2024-04-22T15:49:27.009 INFO:tasks.workunit.client.0.smithi016.stdout:1/185: stat d3/d4/da/f26 0 2024-04-22T15:49:27.015 INFO:tasks.workunit.client.0.smithi016.stdout:1/186: dwrite d3/d4/d15/f16 [0,4194304] 0 2024-04-22T15:49:27.017 INFO:tasks.workunit.client.0.smithi016.stdout:1/187: chown d3/d4/l13 171754889 1 2024-04-22T15:49:27.018 INFO:tasks.workunit.client.0.smithi016.stdout:1/188: write d3/f8 [8836400,46216] 0 2024-04-22T15:49:27.021 INFO:tasks.workunit.client.0.smithi016.stdout:1/189: symlink d3/d28/l3b 0 2024-04-22T15:49:27.024 INFO:tasks.workunit.client.0.smithi016.stdout:1/190: creat d3/d4/d15/d38/f3c x:0 0 0 2024-04-22T15:49:27.025 INFO:tasks.workunit.client.0.smithi016.stdout:1/191: chown d3/d4/d19/d20/f33 2136561 1 2024-04-22T15:49:27.027 INFO:tasks.workunit.client.0.smithi016.stdout:1/192: symlink d3/d4/d19/d20/l3d 0 2024-04-22T15:49:27.027 INFO:tasks.workunit.client.0.smithi016.stdout:1/193: chown d3/d4/l5 199930 1 2024-04-22T15:49:27.028 INFO:tasks.workunit.client.0.smithi016.stdout:1/194: symlink d3/d4/d3a/l3e 0 2024-04-22T15:49:27.029 INFO:tasks.workunit.client.0.smithi016.stdout:1/195: stat d3/d4/d19/c17 0 2024-04-22T15:49:27.031 INFO:tasks.workunit.client.0.smithi016.stdout:1/196: mknod d3/d4/da/c3f 0 2024-04-22T15:49:27.033 INFO:tasks.workunit.client.0.smithi016.stdout:1/197: creat d3/d4/d37/f40 x:0 0 0 2024-04-22T15:49:27.034 INFO:tasks.workunit.client.0.smithi016.stdout:1/198: truncate d3/d4/d19/d20/f33 638253 0 2024-04-22T15:49:27.036 INFO:tasks.workunit.client.0.smithi016.stdout:1/199: creat d3/d4/d37/f41 x:0 0 0 2024-04-22T15:49:27.046 INFO:tasks.workunit.client.0.smithi016.stdout:1/200: dwrite d3/d4/d19/f35 [0,4194304] 0 2024-04-22T15:49:27.046 INFO:tasks.workunit.client.0.smithi016.stdout:1/201: creat d3/d2e/f42 x:0 0 0 2024-04-22T15:49:27.047 INFO:tasks.workunit.client.0.smithi016.stdout:1/202: truncate d3/d4/d19/d20/f21 414262 0 2024-04-22T15:49:27.048 INFO:tasks.workunit.client.0.smithi016.stdout:1/203: dread - d3/d4/d15/d38/f3c zero size 2024-04-22T15:49:27.055 INFO:tasks.workunit.client.0.smithi016.stdout:1/204: link d3/d4/d19/d20/d2b/c39 d3/d4/d19/d20/d2b/c43 0 2024-04-22T15:49:27.056 INFO:tasks.workunit.client.0.smithi016.stdout:1/205: write d3/d4/da/f2d [1813167,18727] 0 2024-04-22T15:49:27.060 INFO:tasks.workunit.client.0.smithi016.stdout:1/206: creat d3/d4/d19/d20/f44 x:0 0 0 2024-04-22T15:49:27.064 INFO:tasks.workunit.client.0.smithi016.stdout:1/207: rename d3/d2e/d32 to d3/d4/d45 0 2024-04-22T15:49:27.064 INFO:tasks.workunit.client.0.smithi016.stdout:1/208: fsync d3/f7 0 2024-04-22T15:49:27.071 INFO:tasks.workunit.client.0.smithi016.stdout:1/209: dwrite d3/d4/d19/d20/f44 [0,4194304] 0 2024-04-22T15:49:27.073 INFO:tasks.workunit.client.0.smithi016.stdout:1/210: mkdir d3/d28/d46 0 2024-04-22T15:49:27.077 INFO:tasks.workunit.client.0.smithi016.stdout:1/211: mknod d3/d2e/c47 0 2024-04-22T15:49:27.077 INFO:tasks.workunit.client.0.smithi016.stdout:1/212: chown d3/f8 3568 1 2024-04-22T15:49:27.079 INFO:tasks.workunit.client.0.smithi016.stdout:1/213: creat d3/d4/d19/d20/f48 x:0 0 0 2024-04-22T15:49:27.082 INFO:tasks.workunit.client.0.smithi016.stdout:1/214: unlink d3/d28/l3b 0 2024-04-22T15:49:27.090 INFO:tasks.workunit.client.0.smithi016.stdout:1/215: dwrite d3/d2e/f42 [0,4194304] 0 2024-04-22T15:49:27.093 INFO:tasks.workunit.client.0.smithi016.stdout:1/216: symlink d3/d4/d37/l49 0 2024-04-22T15:49:27.093 INFO:tasks.workunit.client.0.smithi016.stdout:1/217: read d3/d4/fe [683783,42043] 0 2024-04-22T15:49:27.098 INFO:tasks.workunit.client.0.smithi016.stdout:1/218: rename d3/d4/da/f2d to d3/d4/d19/d20/d2b/f4a 0 2024-04-22T15:49:27.099 INFO:tasks.workunit.client.0.smithi016.stdout:1/219: chown d3/d4/d19/d20/f44 180 1 2024-04-22T15:49:27.101 INFO:tasks.workunit.client.0.smithi016.stdout:1/220: fsync d3/d4/f24 0 2024-04-22T15:49:27.103 INFO:tasks.workunit.client.0.smithi016.stdout:1/221: readlink d3/d4/da/ld 0 2024-04-22T15:49:27.105 INFO:tasks.workunit.client.0.smithi016.stdout:1/222: rmdir d3/d4/d3a 39 2024-04-22T15:49:27.105 INFO:tasks.workunit.client.0.smithi016.stdout:1/223: write d3/d2e/f42 [1447327,88054] 0 2024-04-22T15:49:27.107 INFO:tasks.workunit.client.0.smithi016.stdout:1/224: rmdir d3/d4/d19 39 2024-04-22T15:49:27.110 INFO:tasks.workunit.client.0.smithi016.stdout:1/225: dread d3/d4/d19/d20/f21 [0,4194304] 0 2024-04-22T15:49:27.112 INFO:tasks.workunit.client.0.smithi016.stdout:1/226: stat d3/d4/l1b 0 2024-04-22T15:49:27.112 INFO:tasks.workunit.client.0.smithi016.stdout:1/227: chown d3/d4/d19/d20/f21 257 1 2024-04-22T15:49:27.182 INFO:tasks.workunit.client.0.smithi016.stdout:1/228: dwrite d3/d4/f1e [8388608,4194304] 0 2024-04-22T15:49:27.183 INFO:tasks.workunit.client.0.smithi016.stdout:1/229: write d3/f23 [1155088,42775] 0 2024-04-22T15:49:27.184 INFO:tasks.workunit.client.0.smithi016.stdout:1/230: readlink d3/d4/d19/l1a 0 2024-04-22T15:49:27.184 INFO:tasks.workunit.client.0.smithi016.stdout:1/231: chown d3/d4/f24 466615724 1 2024-04-22T15:49:27.189 INFO:tasks.workunit.client.0.smithi016.stdout:1/232: dread d3/d2e/f42 [0,4194304] 0 2024-04-22T15:49:27.191 INFO:tasks.workunit.client.0.smithi016.stdout:1/233: mknod d3/d4/d37/c4b 0 2024-04-22T15:49:27.192 INFO:tasks.workunit.client.0.smithi016.stdout:1/234: fsync d3/f23 0 2024-04-22T15:49:27.193 INFO:tasks.workunit.client.0.smithi016.stdout:1/235: mkdir d3/d4/d15/d4c 0 2024-04-22T15:49:27.200 INFO:tasks.workunit.client.0.smithi016.stdout:1/236: dwrite d3/d4/d15/d38/f3c [0,4194304] 0 2024-04-22T15:49:27.202 INFO:tasks.workunit.client.0.smithi016.stdout:1/237: write d3/d4/d37/f40 [396057,46284] 0 2024-04-22T15:49:27.217 INFO:tasks.workunit.client.0.smithi016.stdout:1/238: truncate d3/d4/d15/d38/f3c 2356778 0 2024-04-22T15:49:27.217 INFO:tasks.workunit.client.0.smithi016.stdout:1/239: readlink d3/l36 0 2024-04-22T15:49:27.220 INFO:tasks.workunit.client.0.smithi016.stdout:1/240: mknod d3/d4/d3a/c4d 0 2024-04-22T15:49:27.220 INFO:tasks.workunit.client.0.smithi016.stdout:1/241: write d3/d4/d15/f16 [468654,65295] 0 2024-04-22T15:49:27.223 INFO:tasks.workunit.client.0.smithi016.stdout:1/242: rmdir d3/d4/da 39 2024-04-22T15:49:27.225 INFO:tasks.workunit.client.0.smithi016.stdout:1/243: symlink d3/d4/d3a/l4e 0 2024-04-22T15:49:27.226 INFO:tasks.workunit.client.0.smithi016.stdout:1/244: creat d3/f4f x:0 0 0 2024-04-22T15:49:27.230 INFO:tasks.workunit.client.0.smithi016.stdout:1/245: unlink d3/d4/d15/d38/f3c 0 2024-04-22T15:49:27.239 INFO:tasks.workunit.client.0.smithi016.stdout:1/246: truncate d3/d2e/f42 2151929 0 2024-04-22T15:49:27.240 INFO:tasks.workunit.client.0.smithi016.stdout:1/247: rename d3/d4/d15 to d3/d4/d15/d4c/d50 22 2024-04-22T15:49:27.240 INFO:tasks.workunit.client.0.smithi016.stdout:1/248: write d3/d4/d37/f41 [397728,111322] 0 2024-04-22T15:49:27.244 INFO:tasks.workunit.client.0.smithi016.stdout:1/249: rename d3/d4/f1c to d3/d2e/f51 0 2024-04-22T15:49:27.246 INFO:tasks.workunit.client.0.smithi016.stdout:1/250: write d3/f29 [666581,23641] 0 2024-04-22T15:49:27.248 INFO:tasks.workunit.client.0.smithi016.stdout:1/251: mknod d3/d4/d19/d20/d2b/c52 0 2024-04-22T15:49:27.252 INFO:tasks.workunit.client.0.smithi016.stdout:1/252: getdents d3/d4/d19 0 2024-04-22T15:49:27.258 INFO:tasks.workunit.client.0.smithi016.stdout:1/253: dwrite d3/d4/f24 [0,4194304] 0 2024-04-22T15:49:27.262 INFO:tasks.workunit.client.0.smithi016.stdout:1/254: mknod d3/d4/d19/c53 0 2024-04-22T15:49:27.262 INFO:tasks.workunit.client.0.smithi016.stdout:1/255: write d3/f23 [2044167,72320] 0 2024-04-22T15:49:27.264 INFO:tasks.workunit.client.0.smithi016.stdout:1/256: dread d3/f29 [0,4194304] 0 2024-04-22T15:49:27.283 INFO:tasks.workunit.client.0.smithi016.stdout:1/257: creat d3/d4/d19/f54 x:0 0 0 2024-04-22T15:49:27.283 INFO:tasks.workunit.client.0.smithi016.stdout:1/258: write d3/d4/f24 [4494937,57118] 0 2024-04-22T15:49:27.283 INFO:tasks.workunit.client.0.smithi016.stdout:1/259: creat d3/d4/d15/d4c/f55 x:0 0 0 2024-04-22T15:49:27.284 INFO:tasks.workunit.client.0.smithi016.stdout:1/260: symlink d3/d4/d3a/l56 0 2024-04-22T15:49:27.287 INFO:tasks.workunit.client.0.smithi016.stdout:1/261: write d3/d4/d19/d20/f21 [1310748,48975] 0 2024-04-22T15:49:27.290 INFO:tasks.workunit.client.0.smithi016.stdout:1/262: creat d3/d4/da/f57 x:0 0 0 2024-04-22T15:49:27.291 INFO:tasks.workunit.client.0.smithi016.stdout:1/263: creat d3/d4/d45/f58 x:0 0 0 2024-04-22T15:49:27.292 INFO:tasks.workunit.client.0.smithi016.stdout:1/264: dread - d3/f4f zero size 2024-04-22T15:49:27.298 INFO:tasks.workunit.client.0.smithi016.stdout:1/265: dwrite d3/d4/d19/d20/f44 [0,4194304] 0 2024-04-22T15:49:27.300 INFO:tasks.workunit.client.0.smithi016.stdout:1/266: creat d3/d4/d19/d20/d2b/f59 x:0 0 0 2024-04-22T15:49:27.303 INFO:tasks.workunit.client.0.smithi016.stdout:1/267: mknod d3/c5a 0 2024-04-22T15:49:27.305 INFO:tasks.workunit.client.0.smithi016.stdout:1/268: symlink d3/d4/d19/d20/l5b 0 2024-04-22T15:49:27.306 INFO:tasks.workunit.client.0.smithi016.stdout:1/269: write d3/d4/d19/d20/f33 [818419,68716] 0 2024-04-22T15:49:27.324 INFO:tasks.workunit.client.0.smithi016.stdout:1/270: dwrite d3/f29 [0,4194304] 0 2024-04-22T15:49:27.324 INFO:tasks.workunit.client.0.smithi016.stdout:1/271: readlink d3/d4/d3a/l3e 0 2024-04-22T15:49:27.327 INFO:tasks.workunit.client.0.smithi016.stdout:1/272: mknod d3/d4/d3a/c5c 0 2024-04-22T15:49:27.330 INFO:tasks.workunit.client.0.smithi016.stdout:1/273: unlink d3/d4/d19/d20/d2b/c43 0 2024-04-22T15:49:27.333 INFO:tasks.workunit.client.0.smithi016.stdout:1/274: creat d3/d4/d15/f5d x:0 0 0 2024-04-22T15:49:27.446 INFO:tasks.workunit.client.0.smithi016.stdout:3/2: sync 2024-04-22T15:49:27.449 INFO:tasks.workunit.client.0.smithi016.stdout:3/3: creat f0 x:0 0 0 2024-04-22T15:49:27.449 INFO:tasks.workunit.client.0.smithi016.stdout:3/4: stat f0 0 2024-04-22T15:49:27.450 INFO:tasks.workunit.client.0.smithi016.stdout:3/5: chown f0 788174 1 2024-04-22T15:49:27.451 INFO:tasks.workunit.client.0.smithi016.stdout:3/6: creat f1 x:0 0 0 2024-04-22T15:49:27.453 INFO:tasks.workunit.client.0.smithi016.stdout:3/7: creat f2 x:0 0 0 2024-04-22T15:49:27.454 INFO:tasks.workunit.client.0.smithi016.stdout:3/8: write f1 [734814,91550] 0 2024-04-22T15:49:27.454 INFO:tasks.workunit.client.0.smithi016.stdout:3/9: dread - f0 zero size 2024-04-22T15:49:27.455 INFO:tasks.workunit.client.0.smithi016.stdout:3/10: dread - f0 zero size 2024-04-22T15:49:27.455 INFO:tasks.workunit.client.0.smithi016.stdout:3/11: readlink - no filename 2024-04-22T15:49:27.461 INFO:tasks.workunit.client.0.smithi016.stdout:3/12: dwrite f2 [0,4194304] 0 2024-04-22T15:49:27.463 INFO:tasks.workunit.client.0.smithi016.stdout:3/13: creat f3 x:0 0 0 2024-04-22T15:49:27.464 INFO:tasks.workunit.client.0.smithi016.stdout:3/14: write f3 [265140,72052] 0 2024-04-22T15:49:27.467 INFO:tasks.workunit.client.0.smithi016.stdout:3/15: creat f4 x:0 0 0 2024-04-22T15:49:27.539 INFO:tasks.workunit.client.0.smithi016.stdout:4/10: sync 2024-04-22T15:49:27.539 INFO:tasks.workunit.client.0.smithi016.stdout:4/11: write - no filename 2024-04-22T15:49:27.539 INFO:tasks.workunit.client.0.smithi016.stdout:4/12: dwrite - no filename 2024-04-22T15:49:27.539 INFO:tasks.workunit.client.0.smithi016.stdout:4/13: dwrite - no filename 2024-04-22T15:49:27.539 INFO:tasks.workunit.client.0.smithi016.stdout:8/23: sync 2024-04-22T15:49:27.592 INFO:tasks.workunit.client.0.smithi016.stdout:3/16: fsync f3 0 2024-04-22T15:49:27.594 INFO:tasks.workunit.client.0.smithi016.stdout:3/17: symlink l5 0 2024-04-22T15:49:27.597 INFO:tasks.workunit.client.0.smithi016.stdout:3/18: dread f1 [0,4194304] 0 2024-04-22T15:49:27.598 INFO:tasks.workunit.client.0.smithi016.stdout:3/19: mknod c6 0 2024-04-22T15:49:27.599 INFO:tasks.workunit.client.0.smithi016.stdout:3/20: write f3 [1068179,71057] 0 2024-04-22T15:49:27.599 INFO:tasks.workunit.client.0.smithi016.stdout:3/21: dread - f4 zero size 2024-04-22T15:49:27.600 INFO:tasks.workunit.client.0.smithi016.stdout:3/22: write f3 [1707622,11385] 0 2024-04-22T15:49:27.614 INFO:tasks.workunit.client.0.smithi016.stdout:7/57: sync 2024-04-22T15:49:27.614 INFO:tasks.workunit.client.0.smithi016.stdout:6/39: sync 2024-04-22T15:49:27.614 INFO:tasks.workunit.client.0.smithi016.stdout:5/22: sync 2024-04-22T15:49:27.614 INFO:tasks.workunit.client.0.smithi016.stdout:9/102: sync 2024-04-22T15:49:27.614 INFO:tasks.workunit.client.0.smithi016.stdout:0/28: sync 2024-04-22T15:49:27.614 INFO:tasks.workunit.client.0.smithi016.stdout:2/29: sync 2024-04-22T15:49:27.614 INFO:tasks.workunit.client.0.smithi016.stdout:1/275: sync 2024-04-22T15:49:27.616 INFO:tasks.workunit.client.0.smithi016.stdout:6/40: write d3/f6 [293718,105899] 0 2024-04-22T15:49:27.618 INFO:tasks.workunit.client.0.smithi016.stdout:2/30: mkdir d2/de 0 2024-04-22T15:49:27.618 INFO:tasks.workunit.client.0.smithi016.stdout:2/31: stat d2/de 0 2024-04-22T15:49:27.620 INFO:tasks.workunit.client.0.smithi016.stdout:5/23: rename f5 to f6 0 2024-04-22T15:49:27.624 INFO:tasks.workunit.client.0.smithi016.stdout:1/276: dread d3/d2e/f42 [0,4194304] 0 2024-04-22T15:49:27.625 INFO:tasks.workunit.client.0.smithi016.stdout:0/29: rename l6 to l8 0 2024-04-22T15:49:27.628 INFO:tasks.workunit.client.0.smithi016.stdout:7/58: rename d3/d4/d9/f10 to d3/dc/f16 0 2024-04-22T15:49:27.629 INFO:tasks.workunit.client.0.smithi016.stdout:6/41: chown d3/l4 118 1 2024-04-22T15:49:27.631 INFO:tasks.workunit.client.0.smithi016.stdout:2/32: mknod d2/cf 0 2024-04-22T15:49:27.633 INFO:tasks.workunit.client.0.smithi016.stdout:5/24: symlink l7 0 2024-04-22T15:49:27.633 INFO:tasks.workunit.client.0.smithi016.stdout:5/25: stat f2 0 2024-04-22T15:49:27.633 INFO:tasks.workunit.client.0.smithi016.stdout:5/26: fdatasync f2 0 2024-04-22T15:49:27.636 INFO:tasks.workunit.client.0.smithi016.stdout:1/277: mknod d3/d2e/c5e 0 2024-04-22T15:49:27.637 INFO:tasks.workunit.client.0.smithi016.stdout:0/30: truncate f4 44231 0 2024-04-22T15:49:27.642 INFO:tasks.workunit.client.0.smithi016.stdout:6/42: rename f2 to d3/f8 0 2024-04-22T15:49:27.643 INFO:tasks.workunit.client.0.smithi016.stdout:6/43: write d3/f6 [851304,49207] 0 2024-04-22T15:49:27.643 INFO:tasks.workunit.client.0.smithi016.stdout:2/33: creat d2/d9/f10 x:0 0 0 2024-04-22T15:49:27.644 INFO:tasks.workunit.client.0.smithi016.stdout:2/34: dread - d2/f4 zero size 2024-04-22T15:49:27.644 INFO:tasks.workunit.client.0.smithi016.stdout:2/35: fdatasync d2/f7 0 2024-04-22T15:49:27.646 INFO:tasks.workunit.client.0.smithi016.stdout:2/36: write d2/f7 [46062,31765] 0 2024-04-22T15:49:27.646 INFO:tasks.workunit.client.0.smithi016.stdout:9/103: link d9/l10 d9/l17 0 2024-04-22T15:49:27.647 INFO:tasks.workunit.client.0.smithi016.stdout:5/27: write f6 [94067,51822] 0 2024-04-22T15:49:27.650 INFO:tasks.workunit.client.0.smithi016.stdout:9/104: write d9/de/f16 [556587,109651] 0 2024-04-22T15:49:27.653 INFO:tasks.workunit.client.0.smithi016.stdout:9/105: chown d9/l13 80568 1 2024-04-22T15:49:27.653 INFO:tasks.workunit.client.0.smithi016.stdout:1/278: rename d3/d4/d3a/l3e to d3/d4/d19/d20/l5f 0 2024-04-22T15:49:27.653 INFO:tasks.workunit.client.0.smithi016.stdout:6/44: dwrite d3/f6 [0,4194304] 0 2024-04-22T15:49:27.654 INFO:tasks.workunit.client.0.smithi016.stdout:6/45: read d3/f8 [3017156,121990] 0 2024-04-22T15:49:27.657 INFO:tasks.workunit.client.0.smithi016.stdout:8/24: sync 2024-04-22T15:49:27.657 INFO:tasks.workunit.client.0.smithi016.stdout:4/14: sync 2024-04-22T15:49:27.657 INFO:tasks.workunit.client.0.smithi016.stdout:4/15: dwrite - no filename 2024-04-22T15:49:27.657 INFO:tasks.workunit.client.0.smithi016.stdout:5/28: dread f2 [0,4194304] 0 2024-04-22T15:49:27.658 INFO:tasks.workunit.client.0.smithi016.stdout:8/25: dread - f1 zero size 2024-04-22T15:49:27.660 INFO:tasks.workunit.client.0.smithi016.stdout:2/37: creat d2/f11 x:0 0 0 2024-04-22T15:49:27.661 INFO:tasks.workunit.client.0.smithi016.stdout:5/29: dread f6 [0,4194304] 0 2024-04-22T15:49:27.663 INFO:tasks.workunit.client.0.smithi016.stdout:7/59: creat d3/d4/f17 x:0 0 0 2024-04-22T15:49:27.665 INFO:tasks.workunit.client.0.smithi016.stdout:7/60: readlink d3/l12 0 2024-04-22T15:49:27.666 INFO:tasks.workunit.client.0.smithi016.stdout:9/106: dwrite d9/de/f11 [0,4194304] 0 2024-04-22T15:49:27.667 INFO:tasks.workunit.client.0.smithi016.stdout:6/46: dread d3/f8 [4194304,4194304] 0 2024-04-22T15:49:27.681 INFO:tasks.workunit.client.0.smithi016.stdout:2/38: creat d2/d9/f12 x:0 0 0 2024-04-22T15:49:27.681 INFO:tasks.workunit.client.0.smithi016.stdout:5/30: mknod c8 0 2024-04-22T15:49:27.681 INFO:tasks.workunit.client.0.smithi016.stdout:4/16: symlink l0 0 2024-04-22T15:49:27.682 INFO:tasks.workunit.client.0.smithi016.stdout:4/17: dread - no filename 2024-04-22T15:49:27.682 INFO:tasks.workunit.client.0.smithi016.stdout:4/18: dread - no filename 2024-04-22T15:49:27.683 INFO:tasks.workunit.client.0.smithi016.stdout:8/26: dwrite f1 [0,4194304] 0 2024-04-22T15:49:27.688 INFO:tasks.workunit.client.0.smithi016.stdout:6/47: dwrite d3/f6 [0,4194304] 0 2024-04-22T15:49:27.689 INFO:tasks.workunit.client.0.smithi016.stdout:6/48: write d3/f6 [1965721,52881] 0 2024-04-22T15:49:27.707 INFO:tasks.workunit.client.0.smithi016.stdout:3/23: getdents . 0 2024-04-22T15:49:27.709 INFO:tasks.workunit.client.0.smithi016.stdout:8/27: fdatasync f1 0 2024-04-22T15:49:27.710 INFO:tasks.workunit.client.0.smithi016.stdout:8/28: write f1 [2451985,102788] 0 2024-04-22T15:49:27.710 INFO:tasks.workunit.client.0.smithi016.stdout:8/29: readlink - no filename 2024-04-22T15:49:27.712 INFO:tasks.workunit.client.0.smithi016.stdout:8/30: write f1 [649717,58255] 0 2024-04-22T15:49:27.713 INFO:tasks.workunit.client.0.smithi016.stdout:8/31: rename d2 to d2/d4 22 2024-04-22T15:49:27.714 INFO:tasks.workunit.client.0.smithi016.stdout:3/24: dwrite f2 [4194304,4194304] 0 2024-04-22T15:49:27.731 INFO:tasks.workunit.client.0.smithi016.stdout:0/31: dread f4 [0,4194304] 0 2024-04-22T15:49:27.749 INFO:tasks.workunit.client.0.smithi016.stdout:1/279: sync 2024-04-22T15:49:27.756 INFO:tasks.workunit.client.0.smithi016.stdout:1/280: dwrite d3/d4/d19/f54 [0,4194304] 0 2024-04-22T15:49:27.764 INFO:tasks.workunit.client.0.smithi016.stdout:1/281: dwrite d3/d4/d15/f5d [0,4194304] 0 2024-04-22T15:49:27.770 INFO:tasks.workunit.client.0.smithi016.stdout:1/282: dwrite d3/d4/d19/f54 [0,4194304] 0 2024-04-22T15:49:27.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:27 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:49:27.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:27 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:49:27.934 INFO:tasks.workunit.client.0.smithi016.stdout:5/31: dread f6 [0,4194304] 0 2024-04-22T15:49:27.935 INFO:tasks.workunit.client.0.smithi016.stdout:2/39: symlink d2/d9/l13 0 2024-04-22T15:49:27.935 INFO:tasks.workunit.client.0.smithi016.stdout:7/61: rmdir d3/d4/d9/df 39 2024-04-22T15:49:27.935 INFO:tasks.workunit.client.0.smithi016.stdout:4/19: mknod c1 0 2024-04-22T15:49:27.936 INFO:tasks.workunit.client.0.smithi016.stdout:5/32: write f0 [124195,36169] 0 2024-04-22T15:49:27.936 INFO:tasks.workunit.client.0.smithi016.stdout:4/20: chown l0 109338112 1 2024-04-22T15:49:27.939 INFO:tasks.workunit.client.0.smithi016.stdout:6/49: symlink d3/l9 0 2024-04-22T15:49:27.942 INFO:tasks.workunit.client.0.smithi016.stdout:3/25: mkdir d7 0 2024-04-22T15:49:27.943 INFO:tasks.workunit.client.0.smithi016.stdout:3/26: write f0 [520779,31173] 0 2024-04-22T15:49:27.943 INFO:tasks.workunit.client.0.smithi016.stdout:3/27: chown f0 24 1 2024-04-22T15:49:27.943 INFO:tasks.workunit.client.0.smithi016.stdout:8/32: rename d2/c3 to d2/c5 0 2024-04-22T15:49:27.949 INFO:tasks.workunit.client.0.smithi016.stdout:3/28: dread f2 [0,4194304] 0 2024-04-22T15:49:27.950 INFO:tasks.workunit.client.0.smithi016.stdout:8/33: dread f1 [0,4194304] 0 2024-04-22T15:49:27.950 INFO:tasks.workunit.client.0.smithi016.stdout:1/283: creat d3/d4/d19/d20/f60 x:0 0 0 2024-04-22T15:49:27.952 INFO:tasks.workunit.client.0.smithi016.stdout:1/284: chown d3/d2e/f51 56 1 2024-04-22T15:49:27.953 INFO:tasks.workunit.client.0.smithi016.stdout:3/29: dread f1 [0,4194304] 0 2024-04-22T15:49:27.953 INFO:tasks.workunit.client.0.smithi016.stdout:0/32: dwrite f4 [0,4194304] 0 2024-04-22T15:49:27.955 INFO:tasks.workunit.client.0.smithi016.stdout:7/62: read d3/d4/f6 [979481,21152] 0 2024-04-22T15:49:27.956 INFO:tasks.workunit.client.0.smithi016.stdout:2/40: unlink d2/d9/cd 0 2024-04-22T15:49:27.957 INFO:tasks.workunit.client.0.smithi016.stdout:4/21: creat f2 x:0 0 0 2024-04-22T15:49:27.959 INFO:tasks.workunit.client.0.smithi016.stdout:2/41: write d2/d9/f12 [705593,65306] 0 2024-04-22T15:49:27.959 INFO:tasks.workunit.client.0.smithi016.stdout:4/22: chown f2 2191 1 2024-04-22T15:49:27.961 INFO:tasks.workunit.client.0.smithi016.stdout:8/34: dwrite f1 [0,4194304] 0 2024-04-22T15:49:27.962 INFO:tasks.workunit.client.0.smithi016.stdout:4/23: chown f2 60958 1 2024-04-22T15:49:27.962 INFO:tasks.workunit.client.0.smithi016.stdout:6/50: write d3/f8 [4651702,8109] 0 2024-04-22T15:49:27.968 INFO:tasks.workunit.client.0.smithi016.stdout:1/285: dwrite d3/d4/d19/d20/f48 [0,4194304] 0 2024-04-22T15:49:27.971 INFO:tasks.workunit.client.0.smithi016.stdout:3/30: unlink c6 0 2024-04-22T15:49:27.971 INFO:tasks.workunit.client.0.smithi016.stdout:1/286: dread - d3/d4/da/f57 zero size 2024-04-22T15:49:27.971 INFO:tasks.workunit.client.0.smithi016.stdout:0/33: fdatasync f4 0 2024-04-22T15:49:27.976 INFO:tasks.workunit.client.0.smithi016.stdout:4/24: dwrite f2 [0,4194304] 0 2024-04-22T15:49:27.982 INFO:tasks.workunit.client.0.smithi016.stdout:8/35: symlink d2/l6 0 2024-04-22T15:49:27.984 INFO:tasks.workunit.client.0.smithi016.stdout:6/51: mkdir d3/da 0 2024-04-22T15:49:27.985 INFO:tasks.workunit.client.0.smithi016.stdout:9/107: getdents d9/de 0 2024-04-22T15:49:27.987 INFO:tasks.workunit.client.0.smithi016.stdout:9/108: chown f2 767569705 1 2024-04-22T15:49:27.988 INFO:tasks.workunit.client.0.smithi016.stdout:3/31: rename f3 to d7/f8 0 2024-04-22T15:49:27.990 INFO:tasks.workunit.client.0.smithi016.stdout:0/34: creat f9 x:0 0 0 2024-04-22T15:49:27.990 INFO:tasks.workunit.client.0.smithi016.stdout:3/32: chown f1 63895 1 2024-04-22T15:49:27.992 INFO:tasks.workunit.client.0.smithi016.stdout:3/33: write f0 [243619,49181] 0 2024-04-22T15:49:27.993 INFO:tasks.workunit.client.0.smithi016.stdout:1/287: symlink d3/d4/d19/d20/d2b/l61 0 2024-04-22T15:49:27.993 INFO:tasks.workunit.client.0.smithi016.stdout:6/52: dwrite d3/f8 [4194304,4194304] 0 2024-04-22T15:49:27.994 INFO:tasks.workunit.client.0.smithi016.stdout:4/25: creat f3 x:0 0 0 2024-04-22T15:49:27.995 INFO:tasks.workunit.client.0.smithi016.stdout:2/42: mknod d2/de/c14 0 2024-04-22T15:49:28.001 INFO:tasks.workunit.client.0.smithi016.stdout:6/53: dwrite d3/f6 [0,4194304] 0 2024-04-22T15:49:28.002 INFO:tasks.workunit.client.0.smithi016.stdout:6/54: write d3/f8 [6796319,110856] 0 2024-04-22T15:49:28.003 INFO:tasks.workunit.client.0.smithi016.stdout:9/109: creat d9/de/f18 x:0 0 0 2024-04-22T15:49:28.004 INFO:tasks.workunit.client.0.smithi016.stdout:0/35: readlink l8 0 2024-04-22T15:49:28.009 INFO:tasks.workunit.client.0.smithi016.stdout:7/63: rmdir d3/de 0 2024-04-22T15:49:28.012 INFO:tasks.workunit.client.0.smithi016.stdout:3/34: rename f2 to d7/f9 0 2024-04-22T15:49:28.016 INFO:tasks.workunit.client.0.smithi016.stdout:4/26: mknod c4 0 2024-04-22T15:49:28.019 INFO:tasks.workunit.client.0.smithi016.stdout:9/110: mknod d9/c19 0 2024-04-22T15:49:28.020 INFO:tasks.workunit.client.0.smithi016.stdout:9/111: readlink l8 0 2024-04-22T15:49:28.020 INFO:tasks.workunit.client.0.smithi016.stdout:9/112: stat l3 0 2024-04-22T15:49:28.021 INFO:tasks.workunit.client.0.smithi016.stdout:0/36: rename l8 to la 0 2024-04-22T15:49:28.022 INFO:tasks.workunit.client.0.smithi016.stdout:4/27: dwrite f2 [0,4194304] 0 2024-04-22T15:49:28.023 INFO:tasks.workunit.client.0.smithi016.stdout:9/113: write d9/de/f16 [1469504,13624] 0 2024-04-22T15:49:28.023 INFO:tasks.workunit.client.0.smithi016.stdout:4/28: read - f3 zero size 2024-04-22T15:49:28.027 INFO:tasks.workunit.client.0.smithi016.stdout:2/43: link d2/f11 d2/f15 0 2024-04-22T15:49:28.028 INFO:tasks.workunit.client.0.smithi016.stdout:6/55: mkdir d3/da/db 0 2024-04-22T15:49:28.029 INFO:tasks.workunit.client.0.smithi016.stdout:7/64: mknod d3/d14/c18 0 2024-04-22T15:49:28.030 INFO:tasks.workunit.client.0.smithi016.stdout:0/37: chown c7 67688917 1 2024-04-22T15:49:28.032 INFO:tasks.workunit.client.0.smithi016.stdout:2/44: dread f0 [0,4194304] 0 2024-04-22T15:49:28.033 INFO:tasks.workunit.client.0.smithi016.stdout:9/114: creat d9/de/f1a x:0 0 0 2024-04-22T15:49:28.033 INFO:tasks.workunit.client.0.smithi016.stdout:6/56: dread d3/f8 [4194304,4194304] 0 2024-04-22T15:49:28.034 INFO:tasks.workunit.client.0.smithi016.stdout:6/57: write d3/f6 [1921388,76] 0 2024-04-22T15:49:28.038 INFO:tasks.workunit.client.0.smithi016.stdout:0/38: rename f9 to fb 0 2024-04-22T15:49:28.039 INFO:tasks.workunit.client.0.smithi016.stdout:2/45: fdatasync d2/f8 0 2024-04-22T15:49:28.040 INFO:tasks.workunit.client.0.smithi016.stdout:6/58: dwrite d3/f6 [0,4194304] 0 2024-04-22T15:49:28.040 INFO:tasks.workunit.client.0.smithi016.stdout:9/115: creat d9/f1b x:0 0 0 2024-04-22T15:49:28.045 INFO:tasks.workunit.client.0.smithi016.stdout:7/65: mknod d3/d4/d9/df/c19 0 2024-04-22T15:49:28.050 INFO:tasks.workunit.client.0.smithi016.stdout:2/46: creat d2/d9/f16 x:0 0 0 2024-04-22T15:49:28.051 INFO:tasks.workunit.client.0.smithi016.stdout:6/59: fdatasync d3/f8 0 2024-04-22T15:49:28.052 INFO:tasks.workunit.client.0.smithi016.stdout:9/116: creat d9/f1c x:0 0 0 2024-04-22T15:49:28.054 INFO:tasks.workunit.client.0.smithi016.stdout:2/47: unlink d2/d9/l13 0 2024-04-22T15:49:28.054 INFO:tasks.workunit.client.0.smithi016.stdout:2/48: fsync d2/f4 0 2024-04-22T15:49:28.055 INFO:tasks.workunit.client.0.smithi016.stdout:6/60: unlink d3/l4 0 2024-04-22T15:49:28.055 INFO:tasks.workunit.client.0.smithi016.stdout:6/61: chown d3 888 1 2024-04-22T15:49:28.060 INFO:tasks.workunit.client.0.smithi016.stdout:7/66: rename d3/c5 to d3/d4/c1a 0 2024-04-22T15:49:28.060 INFO:tasks.workunit.client.0.smithi016.stdout:2/49: dwrite d2/f6 [0,4194304] 0 2024-04-22T15:49:28.060 INFO:tasks.workunit.client.0.smithi016.stdout:2/50: readlink d2/l3 0 2024-04-22T15:49:28.061 INFO:tasks.workunit.client.0.smithi016.stdout:7/67: write d3/d4/d9/fb [962255,82744] 0 2024-04-22T15:49:28.065 INFO:tasks.workunit.client.0.smithi016.stdout:2/51: mknod d2/c17 0 2024-04-22T15:49:28.066 INFO:tasks.workunit.client.0.smithi016.stdout:2/52: read - d2/d9/f16 zero size 2024-04-22T15:49:28.067 INFO:tasks.workunit.client.0.smithi016.stdout:6/62: mkdir d3/da/db/dc 0 2024-04-22T15:49:28.067 INFO:tasks.workunit.client.0.smithi016.stdout:2/53: write d2/f8 [4716905,113155] 0 2024-04-22T15:49:28.067 INFO:tasks.workunit.client.0.smithi016.stdout:2/54: dread - d2/f11 zero size 2024-04-22T15:49:28.134 INFO:tasks.workunit.client.0.smithi016.stdout:8/36: stat d2/c5 0 2024-04-22T15:49:28.136 INFO:tasks.workunit.client.0.smithi016.stdout:8/37: write f1 [3937622,39717] 0 2024-04-22T15:49:28.138 INFO:tasks.workunit.client.0.smithi016.stdout:8/38: unlink d2/l6 0 2024-04-22T15:49:28.142 INFO:tasks.workunit.client.0.smithi016.stdout:8/39: link c0 d2/c7 0 2024-04-22T15:49:28.144 INFO:tasks.workunit.client.0.smithi016.stdout:8/40: creat d2/f8 x:0 0 0 2024-04-22T15:49:28.145 INFO:tasks.workunit.client.0.smithi016.stdout:8/41: write d2/f8 [1010729,112089] 0 2024-04-22T15:49:28.145 INFO:tasks.workunit.client.0.smithi016.stdout:8/42: chown d2 16741 1 2024-04-22T15:49:28.149 INFO:tasks.workunit.client.0.smithi016.stdout:8/43: rename c0 to d2/c9 0 2024-04-22T15:49:28.151 INFO:tasks.workunit.client.0.smithi016.stdout:5/33: truncate f0 519800 0 2024-04-22T15:49:28.162 INFO:tasks.workunit.client.0.smithi016.stdout:3/35: fsync d7/f9 0 2024-04-22T15:49:28.165 INFO:tasks.workunit.client.0.smithi016.stdout:3/36: creat d7/fa x:0 0 0 2024-04-22T15:49:28.167 INFO:tasks.workunit.client.0.smithi016.stdout:3/37: unlink d7/f9 0 2024-04-22T15:49:28.171 INFO:tasks.workunit.client.0.smithi016.stdout:3/38: mknod d7/cb 0 2024-04-22T15:49:28.172 INFO:tasks.workunit.client.0.smithi016.stdout:0/39: unlink la 0 2024-04-22T15:49:28.172 INFO:tasks.workunit.client.0.smithi016.stdout:0/40: readlink - no filename 2024-04-22T15:49:28.174 INFO:tasks.workunit.client.0.smithi016.stdout:3/39: creat d7/fc x:0 0 0 2024-04-22T15:49:28.174 INFO:tasks.workunit.client.0.smithi016.stdout:3/40: dread - f4 zero size 2024-04-22T15:49:28.175 INFO:tasks.workunit.client.0.smithi016.stdout:0/41: mkdir dc 0 2024-04-22T15:49:28.177 INFO:tasks.workunit.client.0.smithi016.stdout:3/41: symlink d7/ld 0 2024-04-22T15:49:28.179 INFO:tasks.workunit.client.0.smithi016.stdout:0/42: unlink c2 0 2024-04-22T15:49:28.182 INFO:tasks.workunit.client.0.smithi016.stdout:3/42: symlink d7/le 0 2024-04-22T15:49:28.183 INFO:tasks.workunit.client.0.smithi016.stdout:3/43: truncate f0 598783 0 2024-04-22T15:49:28.184 INFO:tasks.workunit.client.0.smithi016.stdout:9/117: getdents d9/de 0 2024-04-22T15:49:28.185 INFO:tasks.workunit.client.0.smithi016.stdout:0/43: mknod dc/cd 0 2024-04-22T15:49:28.185 INFO:tasks.workunit.client.0.smithi016.stdout:0/44: readlink - no filename 2024-04-22T15:49:28.186 INFO:tasks.workunit.client.0.smithi016.stdout:9/118: fsync d9/fc 0 2024-04-22T15:49:28.187 INFO:tasks.workunit.client.0.smithi016.stdout:9/119: fsync f7 0 2024-04-22T15:49:28.190 INFO:tasks.workunit.client.0.smithi016.stdout:3/44: dwrite d7/fa [0,4194304] 0 2024-04-22T15:49:28.191 INFO:tasks.workunit.client.0.smithi016.stdout:9/120: unlink d9/f15 0 2024-04-22T15:49:28.195 INFO:tasks.workunit.client.0.smithi016.stdout:9/121: symlink d9/l1d 0 2024-04-22T15:49:28.196 INFO:tasks.workunit.client.0.smithi016.stdout:6/63: truncate d3/f8 7793224 0 2024-04-22T15:49:28.197 INFO:tasks.workunit.client.0.smithi016.stdout:1/288: sync 2024-04-22T15:49:28.197 INFO:tasks.workunit.client.0.smithi016.stdout:4/29: sync 2024-04-22T15:49:28.197 INFO:tasks.workunit.client.0.smithi016.stdout:6/64: chown d3/da/db 74647 1 2024-04-22T15:49:28.201 INFO:tasks.workunit.client.0.smithi016.stdout:4/30: creat f5 x:0 0 0 2024-04-22T15:49:28.202 INFO:tasks.workunit.client.0.smithi016.stdout:9/122: link d9/l1d d9/l1e 0 2024-04-22T15:49:28.203 INFO:tasks.workunit.client.0.smithi016.stdout:9/123: chown d9/de 325728262 1 2024-04-22T15:49:28.204 INFO:tasks.workunit.client.0.smithi016.stdout:2/55: getdents d2 0 2024-04-22T15:49:28.208 INFO:tasks.workunit.client.0.smithi016.stdout:4/31: link c4 c6 0 2024-04-22T15:49:28.212 INFO:tasks.workunit.client.0.smithi016.stdout:4/32: mkdir d7 0 2024-04-22T15:49:28.216 INFO:tasks.workunit.client.0.smithi016.stdout:1/289: dwrite d3/d4/d15/d4c/f55 [0,4194304] 0 2024-04-22T15:49:28.222 INFO:tasks.workunit.client.0.smithi016.stdout:4/33: creat d7/f8 x:0 0 0 2024-04-22T15:49:28.226 INFO:tasks.workunit.client.0.smithi016.stdout:7/68: dwrite d3/d4/d9/f13 [4194304,4194304] 0 2024-04-22T15:49:28.227 INFO:tasks.workunit.client.0.smithi016.stdout:9/124: dwrite d9/f1b [0,4194304] 0 2024-04-22T15:49:28.228 INFO:tasks.workunit.client.0.smithi016.stdout:4/34: chown d7/f8 3211582 1 2024-04-22T15:49:28.230 INFO:tasks.workunit.client.0.smithi016.stdout:2/56: dwrite d2/d9/f12 [0,4194304] 0 2024-04-22T15:49:28.232 INFO:tasks.workunit.client.0.smithi016.stdout:7/69: mkdir d3/dc/d1b 0 2024-04-22T15:49:28.233 INFO:tasks.workunit.client.0.smithi016.stdout:1/290: rename d3/d4/d19/c53 to d3/d4/d45/c62 0 2024-04-22T15:49:28.233 INFO:tasks.workunit.client.0.smithi016.stdout:5/34: write f6 [479544,86083] 0 2024-04-22T15:49:28.234 INFO:tasks.workunit.client.0.smithi016.stdout:5/35: dread - f3 zero size 2024-04-22T15:49:28.235 INFO:tasks.workunit.client.0.smithi016.stdout:1/291: stat d3/d4/d37 0 2024-04-22T15:49:28.237 INFO:tasks.workunit.client.0.smithi016.stdout:8/44: unlink d2/c9 0 2024-04-22T15:49:28.239 INFO:tasks.workunit.client.0.smithi016.stdout:9/125: mknod d9/c1f 0 2024-04-22T15:49:28.240 INFO:tasks.workunit.client.0.smithi016.stdout:9/126: stat f2 0 2024-04-22T15:49:28.244 INFO:tasks.workunit.client.0.smithi016.stdout:4/35: dread f2 [0,4194304] 0 2024-04-22T15:49:28.246 INFO:tasks.workunit.client.0.smithi016.stdout:2/57: creat d2/f18 x:0 0 0 2024-04-22T15:49:28.247 INFO:tasks.workunit.client.0.smithi016.stdout:5/36: unlink c8 0 2024-04-22T15:49:28.248 INFO:tasks.workunit.client.0.smithi016.stdout:1/292: dwrite d3/d4/d19/d20/f33 [0,4194304] 0 2024-04-22T15:49:28.249 INFO:tasks.workunit.client.0.smithi016.stdout:5/37: readlink l7 0 2024-04-22T15:49:28.251 INFO:tasks.workunit.client.0.smithi016.stdout:8/45: creat d2/fa x:0 0 0 2024-04-22T15:49:28.253 INFO:tasks.workunit.client.0.smithi016.stdout:7/70: dwrite d3/d4/f7 [0,4194304] 0 2024-04-22T15:49:28.253 INFO:tasks.workunit.client.0.smithi016.stdout:9/127: readlink d9/l1d 0 2024-04-22T15:49:28.254 INFO:tasks.workunit.client.0.smithi016.stdout:4/36: symlink d7/l9 0 2024-04-22T15:49:28.255 INFO:tasks.workunit.client.0.smithi016.stdout:3/45: rmdir d7 39 2024-04-22T15:49:28.257 INFO:tasks.workunit.client.0.smithi016.stdout:8/46: fdatasync d2/fa 0 2024-04-22T15:49:28.258 INFO:tasks.workunit.client.0.smithi016.stdout:4/37: read - f5 zero size 2024-04-22T15:49:28.259 INFO:tasks.workunit.client.0.smithi016.stdout:1/293: write d3/d4/d19/d20/f33 [22139,117998] 0 2024-04-22T15:49:28.263 INFO:tasks.workunit.client.0.smithi016.stdout:9/128: mkdir d9/d20 0 2024-04-22T15:49:28.264 INFO:tasks.workunit.client.0.smithi016.stdout:1/294: fsync d3/f4f 0 2024-04-22T15:49:28.266 INFO:tasks.workunit.client.0.smithi016.stdout:7/71: write d3/d4/d9/df/f11 [581933,34283] 0 2024-04-22T15:49:28.267 INFO:tasks.workunit.client.0.smithi016.stdout:3/46: read d7/f8 [1135357,113122] 0 2024-04-22T15:49:28.271 INFO:tasks.workunit.client.0.smithi016.stdout:4/38: symlink d7/la 0 2024-04-22T15:49:28.271 INFO:tasks.workunit.client.0.smithi016.stdout:2/58: dwrite d2/d9/f10 [0,4194304] 0 2024-04-22T15:49:28.273 INFO:tasks.workunit.client.0.smithi016.stdout:4/39: dread - f5 zero size 2024-04-22T15:49:28.274 INFO:tasks.workunit.client.0.smithi016.stdout:8/47: dread f1 [0,4194304] 0 2024-04-22T15:49:28.276 INFO:tasks.workunit.client.0.smithi016.stdout:9/129: symlink d9/de/l21 0 2024-04-22T15:49:28.281 INFO:tasks.workunit.client.0.smithi016.stdout:1/295: mkdir d3/d4/d15/d38/d63 0 2024-04-22T15:49:28.283 INFO:tasks.workunit.client.0.smithi016.stdout:4/40: mknod d7/cb 0 2024-04-22T15:49:28.285 INFO:tasks.workunit.client.0.smithi016.stdout:2/59: mknod d2/d9/c19 0 2024-04-22T15:49:28.291 INFO:tasks.workunit.client.0.smithi016.stdout:7/72: mknod d3/d4/c1c 0 2024-04-22T15:49:28.294 INFO:tasks.workunit.client.0.smithi016.stdout:7/73: dread d3/d4/fd [0,4194304] 0 2024-04-22T15:49:28.295 INFO:tasks.workunit.client.0.smithi016.stdout:1/296: unlink d3/d2e/f42 0 2024-04-22T15:49:28.296 INFO:tasks.workunit.client.0.smithi016.stdout:1/297: fdatasync d3/f23 0 2024-04-22T15:49:28.300 INFO:tasks.workunit.client.0.smithi016.stdout:3/47: link d7/le d7/lf 0 2024-04-22T15:49:28.302 INFO:tasks.workunit.client.0.smithi016.stdout:7/74: symlink d3/d4/d9/df/l1d 0 2024-04-22T15:49:28.302 INFO:tasks.workunit.client.0.smithi016.stdout:8/48: rmdir d2 39 2024-04-22T15:49:28.304 INFO:tasks.workunit.client.0.smithi016.stdout:4/41: dwrite f2 [0,4194304] 0 2024-04-22T15:49:28.311 INFO:tasks.workunit.client.0.smithi016.stdout:1/298: dread d3/d4/d19/f35 [0,4194304] 0 2024-04-22T15:49:28.311 INFO:tasks.workunit.client.0.smithi016.stdout:3/48: symlink d7/l10 0 2024-04-22T15:49:28.319 INFO:tasks.workunit.client.0.smithi016.stdout:7/75: creat d3/dc/f1e x:0 0 0 2024-04-22T15:49:28.319 INFO:tasks.workunit.client.0.smithi016.stdout:4/42: rename c1 to d7/cc 0 2024-04-22T15:49:28.320 INFO:tasks.workunit.client.0.smithi016.stdout:3/49: dwrite d7/fa [0,4194304] 0 2024-04-22T15:49:28.321 INFO:tasks.workunit.client.0.smithi016.stdout:1/299: dwrite d3/d4/d15/d4c/f55 [4194304,4194304] 0 2024-04-22T15:49:28.329 INFO:tasks.workunit.client.0.smithi016.stdout:4/43: symlink d7/ld 0 2024-04-22T15:49:28.330 INFO:tasks.workunit.client.0.smithi016.stdout:4/44: write d7/f8 [177531,50888] 0 2024-04-22T15:49:28.330 INFO:tasks.workunit.client.0.smithi016.stdout:3/50: rmdir d7 39 2024-04-22T15:49:28.331 INFO:tasks.workunit.client.0.smithi016.stdout:3/51: fsync f0 0 2024-04-22T15:49:28.334 INFO:tasks.workunit.client.0.smithi016.stdout:1/300: rmdir d3/d4/d19 39 2024-04-22T15:49:28.346 INFO:tasks.workunit.client.0.smithi016.stdout:5/38: truncate f0 1270521 0 2024-04-22T15:49:28.347 INFO:tasks.workunit.client.0.smithi016.stdout:5/39: dread - f3 zero size 2024-04-22T15:49:28.348 INFO:tasks.workunit.client.0.smithi016.stdout:7/76: creat d3/d4/f1f x:0 0 0 2024-04-22T15:49:28.349 INFO:tasks.workunit.client.0.smithi016.stdout:4/45: rename c4 to d7/ce 0 2024-04-22T15:49:28.350 INFO:tasks.workunit.client.0.smithi016.stdout:1/301: dwrite d3/d4/d19/d20/f44 [0,4194304] 0 2024-04-22T15:49:28.350 INFO:tasks.workunit.client.0.smithi016.stdout:4/46: fsync f3 0 2024-04-22T15:49:28.351 INFO:tasks.workunit.client.0.smithi016.stdout:9/130: dread - d9/de/f18 zero size 2024-04-22T15:49:28.352 INFO:tasks.workunit.client.0.smithi016.stdout:1/302: truncate d3/d4/d19/d20/f48 4544613 0 2024-04-22T15:49:28.355 INFO:tasks.workunit.client.0.smithi016.stdout:1/303: chown d3/d4/d19/d20/d2b/c2c 1749 1 2024-04-22T15:49:28.357 INFO:tasks.workunit.client.0.smithi016.stdout:3/52: creat d7/f11 x:0 0 0 2024-04-22T15:49:28.359 INFO:tasks.workunit.client.0.smithi016.stdout:4/47: dwrite d7/f8 [0,4194304] 0 2024-04-22T15:49:28.361 INFO:tasks.workunit.client.0.smithi016.stdout:5/40: symlink l9 0 2024-04-22T15:49:28.361 INFO:tasks.workunit.client.0.smithi016.stdout:7/77: fsync d3/dc/f16 0 2024-04-22T15:49:28.362 INFO:tasks.workunit.client.0.smithi016.stdout:5/41: chown f4 2995 1 2024-04-22T15:49:28.363 INFO:tasks.workunit.client.0.smithi016.stdout:5/42: fsync f2 0 2024-04-22T15:49:28.363 INFO:tasks.workunit.client.0.smithi016.stdout:8/49: mkdir d2/db 0 2024-04-22T15:49:28.363 INFO:tasks.workunit.client.0.smithi016.stdout:8/50: readlink - no filename 2024-04-22T15:49:28.364 INFO:tasks.workunit.client.0.smithi016.stdout:3/53: rename f4 to d7/f12 0 2024-04-22T15:49:28.366 INFO:tasks.workunit.client.0.smithi016.stdout:9/131: chown d9/f1c 0 1 2024-04-22T15:49:28.368 INFO:tasks.workunit.client.0.smithi016.stdout:4/48: dread f2 [0,4194304] 0 2024-04-22T15:49:28.369 INFO:tasks.workunit.client.0.smithi016.stdout:3/54: mknod d7/c13 0 2024-04-22T15:49:28.370 INFO:tasks.workunit.client.0.smithi016.stdout:3/55: chown f1 40158 1 2024-04-22T15:49:28.371 INFO:tasks.workunit.client.0.smithi016.stdout:7/78: rmdir d3/d4/d9/d15 0 2024-04-22T15:49:28.372 INFO:tasks.workunit.client.0.smithi016.stdout:4/49: write f2 [4524533,126819] 0 2024-04-22T15:49:28.373 INFO:tasks.workunit.client.0.smithi016.stdout:1/304: getdents d3/d4/d15/d4c 0 2024-04-22T15:49:28.374 INFO:tasks.workunit.client.0.smithi016.stdout:3/56: fdatasync d7/f8 0 2024-04-22T15:49:28.374 INFO:tasks.workunit.client.0.smithi016.stdout:7/79: chown d3/d14/c18 3 1 2024-04-22T15:49:28.375 INFO:tasks.workunit.client.0.smithi016.stdout:5/43: dwrite f6 [0,4194304] 0 2024-04-22T15:49:28.376 INFO:tasks.workunit.client.0.smithi016.stdout:1/305: write d3/f4f [960171,94627] 0 2024-04-22T15:49:28.376 INFO:tasks.workunit.client.0.smithi016.stdout:7/80: chown d3/d4/c1c 30947 1 2024-04-22T15:49:28.377 INFO:tasks.workunit.client.0.smithi016.stdout:3/57: fdatasync f1 0 2024-04-22T15:49:28.379 INFO:tasks.workunit.client.0.smithi016.stdout:7/81: write d3/d4/f6 [2393062,111129] 0 2024-04-22T15:49:28.381 INFO:tasks.workunit.client.0.smithi016.stdout:4/50: rename l0 to d7/lf 0 2024-04-22T15:49:28.384 INFO:tasks.workunit.client.0.smithi016.stdout:5/44: dread f6 [0,4194304] 0 2024-04-22T15:49:28.384 INFO:tasks.workunit.client.0.smithi016.stdout:9/132: rename d9/ff to d9/f22 0 2024-04-22T15:49:28.385 INFO:tasks.workunit.client.0.smithi016.stdout:5/45: dread - f3 zero size 2024-04-22T15:49:28.386 INFO:tasks.workunit.client.0.smithi016.stdout:5/46: chown l9 21150 1 2024-04-22T15:49:28.386 INFO:tasks.workunit.client.0.smithi016.stdout:9/133: read - d9/de/f1a zero size 2024-04-22T15:49:28.387 INFO:tasks.workunit.client.0.smithi016.stdout:3/58: dwrite f0 [0,4194304] 0 2024-04-22T15:49:28.387 INFO:tasks.workunit.client.0.smithi016.stdout:9/134: chown d9/f1c 801 1 2024-04-22T15:49:28.395 INFO:tasks.workunit.client.0.smithi016.stdout:8/51: chown d2 1 1 2024-04-22T15:49:28.402 INFO:tasks.workunit.client.0.smithi016.stdout:4/51: creat d7/f10 x:0 0 0 2024-04-22T15:49:28.409 INFO:tasks.workunit.client.0.smithi016.stdout:3/59: rename d7/fa to d7/f14 0 2024-04-22T15:49:28.409 INFO:tasks.workunit.client.0.smithi016.stdout:3/60: stat f0 0 2024-04-22T15:49:28.410 INFO:tasks.workunit.client.0.smithi016.stdout:5/47: dread f0 [0,4194304] 0 2024-04-22T15:49:28.411 INFO:tasks.workunit.client.0.smithi016.stdout:9/135: mkdir d9/de/d23 0 2024-04-22T15:49:28.411 INFO:tasks.workunit.client.0.smithi016.stdout:9/136: dread - d9/de/f1a zero size 2024-04-22T15:49:28.414 INFO:tasks.workunit.client.0.smithi016.stdout:8/52: write f1 [2898050,10391] 0 2024-04-22T15:49:28.417 INFO:tasks.workunit.client.0.smithi016.stdout:4/52: rmdir d7 39 2024-04-22T15:49:28.422 INFO:tasks.workunit.client.0.smithi016.stdout:5/48: unlink f4 0 2024-04-22T15:49:28.425 INFO:tasks.workunit.client.0.smithi016.stdout:5/49: dread f2 [0,4194304] 0 2024-04-22T15:49:28.426 INFO:tasks.workunit.client.0.smithi016.stdout:8/53: stat d2/c5 0 2024-04-22T15:49:28.427 INFO:tasks.workunit.client.0.smithi016.stdout:1/306: rename d3/d4/d15/f5d to d3/d4/d19/f64 0 2024-04-22T15:49:28.429 INFO:tasks.workunit.client.0.smithi016.stdout:4/53: write d7/f10 [734879,131012] 0 2024-04-22T15:49:28.430 INFO:tasks.workunit.client.0.smithi016.stdout:4/54: write f3 [260769,3000] 0 2024-04-22T15:49:28.430 INFO:tasks.workunit.client.0.smithi016.stdout:4/55: write f3 [402400,353] 0 2024-04-22T15:49:28.431 INFO:tasks.workunit.client.0.smithi016.stdout:4/56: write f2 [5391146,110316] 0 2024-04-22T15:49:28.433 INFO:tasks.workunit.client.0.smithi016.stdout:4/57: write d7/f10 [658252,34459] 0 2024-04-22T15:49:28.433 INFO:tasks.workunit.client.0.smithi016.stdout:4/58: stat d7/la 0 2024-04-22T15:49:28.434 INFO:tasks.workunit.client.0.smithi016.stdout:4/59: write d7/f10 [1611350,36392] 0 2024-04-22T15:49:28.434 INFO:tasks.workunit.client.0.smithi016.stdout:9/137: mkdir d9/de/d23/d24 0 2024-04-22T15:49:28.436 INFO:tasks.workunit.client.0.smithi016.stdout:7/82: getdents d3/d4/d9/df 0 2024-04-22T15:49:28.436 INFO:tasks.workunit.client.0.smithi016.stdout:7/83: fdatasync f0 0 2024-04-22T15:49:28.439 INFO:tasks.workunit.client.0.smithi016.stdout:5/50: write f6 [4943193,65654] 0 2024-04-22T15:49:28.440 INFO:tasks.workunit.client.0.smithi016.stdout:5/51: fdatasync f6 0 2024-04-22T15:49:28.441 INFO:tasks.workunit.client.0.smithi016.stdout:8/54: read d2/f8 [1027417,104734] 0 2024-04-22T15:49:28.448 INFO:tasks.workunit.client.0.smithi016.stdout:3/61: truncate f0 2948970 0 2024-04-22T15:49:28.453 INFO:tasks.workunit.client.0.smithi016.stdout:9/138: fdatasync d9/f14 0 2024-04-22T15:49:28.459 INFO:tasks.workunit.client.0.smithi016.stdout:1/307: dwrite d3/d4/fe [0,4194304] 0 2024-04-22T15:49:28.464 INFO:tasks.workunit.client.0.smithi016.stdout:1/308: dread d3/d4/d15/d4c/f55 [4194304,4194304] 0 2024-04-22T15:49:28.466 INFO:tasks.workunit.client.0.smithi016.stdout:1/309: write d3/d4/f1e [6096552,111242] 0 2024-04-22T15:49:28.466 INFO:tasks.workunit.client.0.smithi016.stdout:3/62: write d7/f12 [29366,103736] 0 2024-04-22T15:49:28.470 INFO:tasks.workunit.client.0.smithi016.stdout:4/60: link d7/f8 d7/f11 0 2024-04-22T15:49:28.471 INFO:tasks.workunit.client.0.smithi016.stdout:7/84: symlink d3/dc/d1b/l20 0 2024-04-22T15:49:28.475 INFO:tasks.workunit.client.0.smithi016.stdout:7/85: dread d3/d4/f6 [0,4194304] 0 2024-04-22T15:49:28.476 INFO:tasks.workunit.client.0.smithi016.stdout:8/55: creat d2/db/fc x:0 0 0 2024-04-22T15:49:28.476 INFO:tasks.workunit.client.0.smithi016.stdout:8/56: readlink - no filename 2024-04-22T15:49:28.476 INFO:tasks.workunit.client.0.smithi016.stdout:0/45: sync 2024-04-22T15:49:28.482 INFO:tasks.workunit.client.0.smithi016.stdout:7/86: dwrite d3/fa [0,4194304] 0 2024-04-22T15:49:28.482 INFO:tasks.workunit.client.0.smithi016.stdout:9/139: unlink d9/l1d 0 2024-04-22T15:49:28.484 INFO:tasks.workunit.client.0.smithi016.stdout:8/57: dwrite d2/db/fc [0,4194304] 0 2024-04-22T15:49:28.485 INFO:tasks.workunit.client.0.smithi016.stdout:4/61: mknod d7/c12 0 2024-04-22T15:49:28.491 INFO:tasks.workunit.client.0.smithi016.stdout:0/46: rmdir dc 39 2024-04-22T15:49:28.492 INFO:tasks.workunit.client.0.smithi016.stdout:0/47: write f4 [3667289,68918] 0 2024-04-22T15:49:28.495 INFO:tasks.workunit.client.0.smithi016.stdout:3/63: rename f0 to d7/f15 0 2024-04-22T15:49:28.495 INFO:tasks.workunit.client.0.smithi016.stdout:3/64: rename d7 to d7/d16 22 2024-04-22T15:49:28.506 INFO:tasks.workunit.client.0.smithi016.stdout:4/62: symlink d7/l13 0 2024-04-22T15:49:28.507 INFO:tasks.workunit.client.0.smithi016.stdout:7/87: fdatasync d3/fa 0 2024-04-22T15:49:28.508 INFO:tasks.workunit.client.0.smithi016.stdout:5/52: getdents . 0 2024-04-22T15:49:28.509 INFO:tasks.workunit.client.0.smithi016.stdout:5/53: read f6 [585570,63670] 0 2024-04-22T15:49:28.514 INFO:tasks.workunit.client.0.smithi016.stdout:3/65: chown d7/le 10276484 1 2024-04-22T15:49:28.514 INFO:tasks.workunit.client.0.smithi016.stdout:1/310: rmdir d3/d28/d46 0 2024-04-22T15:49:28.515 INFO:tasks.workunit.client.0.smithi016.stdout:0/48: dwrite fb [0,4194304] 0 2024-04-22T15:49:28.515 INFO:tasks.workunit.client.0.smithi016.stdout:0/49: write fb [2258122,42218] 0 2024-04-22T15:49:28.518 INFO:tasks.workunit.client.0.smithi016.stdout:9/140: mkdir d9/de/d23/d24/d25 0 2024-04-22T15:49:28.518 INFO:tasks.workunit.client.0.smithi016.stdout:2/60: sync 2024-04-22T15:49:28.518 INFO:tasks.workunit.client.0.smithi016.stdout:6/65: sync 2024-04-22T15:49:28.519 INFO:tasks.workunit.client.0.smithi016.stdout:8/58: symlink d2/ld 0 2024-04-22T15:49:28.523 INFO:tasks.workunit.client.0.smithi016.stdout:4/63: creat d7/f14 x:0 0 0 2024-04-22T15:49:28.529 INFO:tasks.workunit.client.0.smithi016.stdout:1/311: symlink d3/d28/l65 0 2024-04-22T15:49:28.538 INFO:tasks.workunit.client.0.smithi016.stdout:2/61: mkdir d2/d1a 0 2024-04-22T15:49:28.540 INFO:tasks.workunit.client.0.smithi016.stdout:6/66: mkdir d3/da/dd 0 2024-04-22T15:49:28.541 INFO:tasks.workunit.client.0.smithi016.stdout:8/59: rename d2 to d2/de 22 2024-04-22T15:49:28.543 INFO:tasks.workunit.client.0.smithi016.stdout:4/64: mknod d7/c15 0 2024-04-22T15:49:28.547 INFO:tasks.workunit.client.0.smithi016.stdout:9/141: readlink d9/l12 0 2024-04-22T15:49:28.550 INFO:tasks.workunit.client.0.smithi016.stdout:4/65: creat d7/f16 x:0 0 0 2024-04-22T15:49:28.551 INFO:tasks.workunit.client.0.smithi016.stdout:7/88: creat d3/d4/d9/f21 x:0 0 0 2024-04-22T15:49:28.556 INFO:tasks.workunit.client.0.smithi016.stdout:4/66: dwrite d7/f10 [0,4194304] 0 2024-04-22T15:49:28.558 INFO:tasks.workunit.client.0.smithi016.stdout:9/142: mkdir d9/de/d26 0 2024-04-22T15:49:28.560 INFO:tasks.workunit.client.0.smithi016.stdout:2/62: mknod d2/d1a/c1b 0 2024-04-22T15:49:28.566 INFO:tasks.workunit.client.0.smithi016.stdout:8/60: rename d2/c7 to d2/db/cf 0 2024-04-22T15:49:28.570 INFO:tasks.workunit.client.0.smithi016.stdout:3/66: getdents d7 0 2024-04-22T15:49:28.572 INFO:tasks.workunit.client.0.smithi016.stdout:7/89: mkdir d3/dc/d22 0 2024-04-22T15:49:28.572 INFO:tasks.workunit.client.0.smithi016.stdout:7/90: fsync d3/d4/d9/fb 0 2024-04-22T15:49:28.573 INFO:tasks.workunit.client.0.smithi016.stdout:0/50: link dc/cd dc/ce 0 2024-04-22T15:49:28.576 INFO:tasks.workunit.client.0.smithi016.stdout:4/67: creat d7/f17 x:0 0 0 2024-04-22T15:49:28.577 INFO:tasks.workunit.client.0.smithi016.stdout:9/143: dread f2 [0,4194304] 0 2024-04-22T15:49:28.578 INFO:tasks.workunit.client.0.smithi016.stdout:2/63: mknod d2/c1c 0 2024-04-22T15:49:28.578 INFO:tasks.workunit.client.0.smithi016.stdout:9/144: write f7 [4228378,110958] 0 2024-04-22T15:49:28.580 INFO:tasks.workunit.client.0.smithi016.stdout:1/312: truncate d3/d4/d15/d4c/f55 6952602 0 2024-04-22T15:49:28.581 INFO:tasks.workunit.client.0.smithi016.stdout:8/61: write d2/fa [298829,48281] 0 2024-04-22T15:49:28.583 INFO:tasks.workunit.client.0.smithi016.stdout:3/67: mkdir d7/d17 0 2024-04-22T15:49:28.586 INFO:tasks.workunit.client.0.smithi016.stdout:7/91: creat d3/f23 x:0 0 0 2024-04-22T15:49:28.589 INFO:tasks.workunit.client.0.smithi016.stdout:4/68: mknod d7/c18 0 2024-04-22T15:49:28.590 INFO:tasks.workunit.client.0.smithi016.stdout:2/64: symlink d2/d9/l1d 0 2024-04-22T15:49:28.592 INFO:tasks.workunit.client.0.smithi016.stdout:6/67: dwrite d3/f8 [0,4194304] 0 2024-04-22T15:49:28.593 INFO:tasks.workunit.client.0.smithi016.stdout:6/68: write d3/f8 [8582063,60062] 0 2024-04-22T15:49:28.593 INFO:tasks.workunit.client.0.smithi016.stdout:1/313: unlink d3/d4/da/f57 0 2024-04-22T15:49:28.594 INFO:tasks.workunit.client.0.smithi016.stdout:6/69: fsync d3/f6 0 2024-04-22T15:49:28.596 INFO:tasks.workunit.client.0.smithi016.stdout:7/92: unlink d3/d4/d9/f21 0 2024-04-22T15:49:28.598 INFO:tasks.workunit.client.0.smithi016.stdout:7/93: write f0 [4315773,44990] 0 2024-04-22T15:49:28.598 INFO:tasks.workunit.client.0.smithi016.stdout:0/51: unlink dc/cd 0 2024-04-22T15:49:28.600 INFO:tasks.workunit.client.0.smithi016.stdout:4/69: rename f3 to d7/f19 0 2024-04-22T15:49:28.603 INFO:tasks.workunit.client.0.smithi016.stdout:6/70: dwrite d3/f8 [0,4194304] 0 2024-04-22T15:49:28.605 INFO:tasks.workunit.client.0.smithi016.stdout:6/71: write d3/f8 [3647143,96009] 0 2024-04-22T15:49:28.606 INFO:tasks.workunit.client.0.smithi016.stdout:0/52: dread fb [0,4194304] 0 2024-04-22T15:49:28.607 INFO:tasks.workunit.client.0.smithi016.stdout:0/53: chown dc 634452562 1 2024-04-22T15:49:28.608 INFO:tasks.workunit.client.0.smithi016.stdout:3/68: mkdir d7/d17/d18 0 2024-04-22T15:49:28.608 INFO:tasks.workunit.client.0.smithi016.stdout:6/72: write d3/f8 [5066459,94821] 0 2024-04-22T15:49:28.610 INFO:tasks.workunit.client.0.smithi016.stdout:4/70: dwrite d7/f14 [0,4194304] 0 2024-04-22T15:49:28.616 INFO:tasks.workunit.client.0.smithi016.stdout:6/73: dread d3/f8 [0,4194304] 0 2024-04-22T15:49:28.620 INFO:tasks.workunit.client.0.smithi016.stdout:7/94: rmdir d3/d4/d9/df 39 2024-04-22T15:49:28.625 INFO:tasks.workunit.client.0.smithi016.stdout:2/65: rename d2/c1c to d2/d1a/c1e 0 2024-04-22T15:49:28.625 INFO:tasks.workunit.client.0.smithi016.stdout:2/66: stat d2/d9/fa 0 2024-04-22T15:49:28.631 INFO:tasks.workunit.client.0.smithi016.stdout:0/54: mknod dc/cf 0 2024-04-22T15:49:28.635 INFO:tasks.workunit.client.0.smithi016.stdout:0/55: dread f4 [0,4194304] 0 2024-04-22T15:49:28.640 INFO:tasks.workunit.client.0.smithi016.stdout:4/71: symlink d7/l1a 0 2024-04-22T15:49:28.644 INFO:tasks.workunit.client.0.smithi016.stdout:9/145: getdents d9/de/d23 0 2024-04-22T15:49:28.646 INFO:tasks.workunit.client.0.smithi016.stdout:6/74: rename d3/da/db/dc to d3/da/de 0 2024-04-22T15:49:28.652 INFO:tasks.workunit.client.0.smithi016.stdout:5/54: sync 2024-04-22T15:49:28.653 INFO:tasks.workunit.client.0.smithi016.stdout:5/55: write f6 [1191069,46569] 0 2024-04-22T15:49:28.655 INFO:tasks.workunit.client.0.smithi016.stdout:5/56: dread f0 [0,4194304] 0 2024-04-22T15:49:28.656 INFO:tasks.workunit.client.0.smithi016.stdout:3/69: symlink d7/d17/d18/l19 0 2024-04-22T15:49:28.658 INFO:tasks.workunit.client.0.smithi016.stdout:1/314: creat d3/d4/d19/f66 x:0 0 0 2024-04-22T15:49:28.659 INFO:tasks.workunit.client.0.smithi016.stdout:8/62: truncate d2/db/fc 3414966 0 2024-04-22T15:49:28.662 INFO:tasks.workunit.client.0.smithi016.stdout:7/95: symlink d3/dc/d22/l24 0 2024-04-22T15:49:28.668 INFO:tasks.workunit.client.0.smithi016.stdout:6/75: creat d3/da/de/ff x:0 0 0 2024-04-22T15:49:28.673 INFO:tasks.workunit.client.0.smithi016.stdout:5/57: symlink la 0 2024-04-22T15:49:28.674 INFO:tasks.workunit.client.0.smithi016.stdout:3/70: symlink d7/l1a 0 2024-04-22T15:49:28.676 INFO:tasks.workunit.client.0.smithi016.stdout:3/71: read d7/f8 [976691,37127] 0 2024-04-22T15:49:28.676 INFO:tasks.workunit.client.0.smithi016.stdout:2/67: rmdir d2 39 2024-04-22T15:49:28.678 INFO:tasks.workunit.client.0.smithi016.stdout:8/63: unlink d2/fa 0 2024-04-22T15:49:28.683 INFO:tasks.workunit.client.0.smithi016.stdout:9/146: truncate d9/fc 2998897 0 2024-04-22T15:49:28.684 INFO:tasks.workunit.client.0.smithi016.stdout:0/56: symlink dc/l10 0 2024-04-22T15:49:28.687 INFO:tasks.workunit.client.0.smithi016.stdout:8/64: dwrite f1 [0,4194304] 0 2024-04-22T15:49:28.688 INFO:tasks.workunit.client.0.smithi016.stdout:5/58: write f3 [581389,104817] 0 2024-04-22T15:49:28.689 INFO:tasks.workunit.client.0.smithi016.stdout:5/59: readlink l7 0 2024-04-22T15:49:28.689 INFO:tasks.workunit.client.0.smithi016.stdout:1/315: mkdir d3/d4/d15/d38/d63/d67 0 2024-04-22T15:49:28.690 INFO:tasks.workunit.client.0.smithi016.stdout:6/76: dread d3/f8 [0,4194304] 0 2024-04-22T15:49:28.691 INFO:tasks.workunit.client.0.smithi016.stdout:5/60: fsync f3 0 2024-04-22T15:49:28.705 INFO:tasks.workunit.client.0.smithi016.stdout:0/57: truncate fb 4891141 0 2024-04-22T15:49:28.714 INFO:tasks.workunit.client.0.smithi016.stdout:1/316: unlink d3/c34 0 2024-04-22T15:49:28.718 INFO:tasks.workunit.client.0.smithi016.stdout:5/61: mknod cb 0 2024-04-22T15:49:28.719 INFO:tasks.workunit.client.0.smithi016.stdout:4/72: dwrite d7/f8 [0,4194304] 0 2024-04-22T15:49:28.720 INFO:tasks.workunit.client.0.smithi016.stdout:4/73: chown d7/c15 21 1 2024-04-22T15:49:28.721 INFO:tasks.workunit.client.0.smithi016.stdout:2/68: mknod d2/c1f 0 2024-04-22T15:49:28.721 INFO:tasks.workunit.client.0.smithi016.stdout:2/69: dread - d2/f4 zero size 2024-04-22T15:49:28.723 INFO:tasks.workunit.client.0.smithi016.stdout:7/96: creat d3/d4/d9/f25 x:0 0 0 2024-04-22T15:49:28.728 INFO:tasks.workunit.client.0.smithi016.stdout:4/74: dwrite d7/f11 [0,4194304] 0 2024-04-22T15:49:28.728 INFO:tasks.workunit.client.0.smithi016.stdout:1/317: mknod d3/d4/d19/d20/d2b/c68 0 2024-04-22T15:49:28.729 INFO:tasks.workunit.client.0.smithi016.stdout:4/75: dread - d7/f16 zero size 2024-04-22T15:49:28.730 INFO:tasks.workunit.client.0.smithi016.stdout:6/77: link d3/f8 d3/da/dd/f10 0 2024-04-22T15:49:28.736 INFO:tasks.workunit.client.0.smithi016.stdout:4/76: dread d7/f8 [0,4194304] 0 2024-04-22T15:49:28.736 INFO:tasks.workunit.client.0.smithi016.stdout:2/70: unlink d2/d9/f10 0 2024-04-22T15:49:28.738 INFO:tasks.workunit.client.0.smithi016.stdout:2/71: fdatasync d2/f7 0 2024-04-22T15:49:28.740 INFO:tasks.workunit.client.0.smithi016.stdout:1/318: dwrite d3/d4/d19/d20/d2b/f59 [0,4194304] 0 2024-04-22T15:49:28.741 INFO:tasks.workunit.client.0.smithi016.stdout:7/97: rmdir d3/dc/d22 39 2024-04-22T15:49:28.744 INFO:tasks.workunit.client.0.smithi016.stdout:1/319: dread d3/d4/d19/d20/d2b/f59 [0,4194304] 0 2024-04-22T15:49:28.749 INFO:tasks.workunit.client.0.smithi016.stdout:6/78: creat d3/da/f11 x:0 0 0 2024-04-22T15:49:28.751 INFO:tasks.workunit.client.0.smithi016.stdout:3/72: dwrite d7/f15 [0,4194304] 0 2024-04-22T15:49:28.756 INFO:tasks.workunit.client.0.smithi016.stdout:4/77: symlink d7/l1b 0 2024-04-22T15:49:28.760 INFO:tasks.workunit.client.0.smithi016.stdout:2/72: creat d2/d1a/f20 x:0 0 0 2024-04-22T15:49:28.765 INFO:tasks.workunit.client.0.smithi016.stdout:1/320: rmdir d3/d4/d19/d20 39 2024-04-22T15:49:28.770 INFO:tasks.workunit.client.0.smithi016.stdout:6/79: rename d3/da/de to d3/da/d12 0 2024-04-22T15:49:28.772 INFO:tasks.workunit.client.0.smithi016.stdout:3/73: mkdir d7/d1b 0 2024-04-22T15:49:28.774 INFO:tasks.workunit.client.0.smithi016.stdout:0/58: write f4 [5087240,33586] 0 2024-04-22T15:49:28.775 INFO:tasks.workunit.client.0.smithi016.stdout:4/78: fdatasync d7/f19 0 2024-04-22T15:49:28.775 INFO:tasks.workunit.client.0.smithi016.stdout:4/79: dread - f5 zero size 2024-04-22T15:49:28.776 INFO:tasks.workunit.client.0.smithi016.stdout:4/80: chown d7/c18 290181464 1 2024-04-22T15:49:28.777 INFO:tasks.workunit.client.0.smithi016.stdout:2/73: symlink d2/l21 0 2024-04-22T15:49:28.779 INFO:tasks.workunit.client.0.smithi016.stdout:9/147: sync 2024-04-22T15:49:28.787 INFO:tasks.workunit.client.0.smithi016.stdout:5/62: getdents . 0 2024-04-22T15:49:28.787 INFO:tasks.workunit.client.0.smithi016.stdout:5/63: readlink l7 0 2024-04-22T15:49:28.788 INFO:tasks.workunit.client.0.smithi016.stdout:6/80: mkdir d3/da/d12/d13 0 2024-04-22T15:49:28.788 INFO:tasks.workunit.client.0.smithi016.stdout:1/321: dread d3/d4/d19/d20/d2b/f4a [0,4194304] 0 2024-04-22T15:49:28.790 INFO:tasks.workunit.client.0.smithi016.stdout:3/74: rename d7/le to d7/d1b/l1c 0 2024-04-22T15:49:28.793 INFO:tasks.workunit.client.0.smithi016.stdout:4/81: symlink d7/l1c 0 2024-04-22T15:49:28.795 INFO:tasks.workunit.client.0.smithi016.stdout:2/74: mknod d2/de/c22 0 2024-04-22T15:49:28.795 INFO:tasks.workunit.client.0.smithi016.stdout:2/75: truncate d2/f18 291871 0 2024-04-22T15:49:28.796 INFO:tasks.workunit.client.0.smithi016.stdout:2/76: chown d2/d1a 751 1 2024-04-22T15:49:28.797 INFO:tasks.workunit.client.0.smithi016.stdout:9/148: write f2 [2557634,115356] 0 2024-04-22T15:49:28.798 INFO:tasks.workunit.client.0.smithi016.stdout:7/98: creat d3/d4/d9/f26 x:0 0 0 2024-04-22T15:49:28.801 INFO:tasks.workunit.client.0.smithi016.stdout:5/64: mknod cc 0 2024-04-22T15:49:28.804 INFO:tasks.workunit.client.0.smithi016.stdout:1/322: mknod d3/d4/d15/c69 0 2024-04-22T15:49:28.807 INFO:tasks.workunit.client.0.smithi016.stdout:3/75: rename d7/l10 to d7/d1b/l1d 0 2024-04-22T15:49:28.811 INFO:tasks.workunit.client.0.smithi016.stdout:9/149: symlink d9/de/l27 0 2024-04-22T15:49:28.814 INFO:tasks.workunit.client.0.smithi016.stdout:5/65: creat fd x:0 0 0 2024-04-22T15:49:28.816 INFO:tasks.workunit.client.0.smithi016.stdout:6/81: unlink d3/da/f11 0 2024-04-22T15:49:28.817 INFO:tasks.workunit.client.0.smithi016.stdout:1/323: mknod d3/d4/da/c6a 0 2024-04-22T15:49:28.818 INFO:tasks.workunit.client.0.smithi016.stdout:3/76: creat d7/f1e x:0 0 0 2024-04-22T15:49:28.820 INFO:tasks.workunit.client.0.smithi016.stdout:1/324: dread d3/d4/da/f26 [0,4194304] 0 2024-04-22T15:49:28.820 INFO:tasks.workunit.client.0.smithi016.stdout:1/325: chown d3/d4/l5 48 1 2024-04-22T15:49:28.823 INFO:tasks.workunit.client.0.smithi016.stdout:8/65: truncate d2/db/fc 3960362 0 2024-04-22T15:49:28.825 INFO:tasks.workunit.client.0.smithi016.stdout:7/99: symlink d3/dc/l27 0 2024-04-22T15:49:28.827 INFO:tasks.workunit.client.0.smithi016.stdout:5/66: unlink f3 0 2024-04-22T15:49:28.827 INFO:tasks.workunit.client.0.smithi016.stdout:5/67: stat l9 0 2024-04-22T15:49:28.832 INFO:tasks.workunit.client.0.smithi016.stdout:3/77: rmdir d7/d17 39 2024-04-22T15:49:28.834 INFO:tasks.workunit.client.0.smithi016.stdout:0/59: creat dc/f11 x:0 0 0 2024-04-22T15:49:28.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:28 smithi027 ceph-mon[34922]: pgmap v26: 65 pgs: 65 active+clean; 266 MiB data, 935 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 13 MiB/s wr, 321 op/s 2024-04-22T15:49:28.836 INFO:tasks.workunit.client.0.smithi016.stdout:1/326: mkdir d3/d4/d3a/d6b 0 2024-04-22T15:49:28.836 INFO:tasks.workunit.client.0.smithi016.stdout:1/327: dread - d3/d4/d19/d20/f60 zero size 2024-04-22T15:49:28.837 INFO:tasks.workunit.client.0.smithi016.stdout:1/328: readlink d3/d4/l13 0 2024-04-22T15:49:28.839 INFO:tasks.workunit.client.0.smithi016.stdout:7/100: creat d3/f28 x:0 0 0 2024-04-22T15:49:28.840 INFO:tasks.workunit.client.0.smithi016.stdout:9/150: read d9/fc [2702520,120852] 0 2024-04-22T15:49:28.841 INFO:tasks.workunit.client.0.smithi016.stdout:8/66: dread d2/f8 [0,4194304] 0 2024-04-22T15:49:28.841 INFO:tasks.workunit.client.0.smithi016.stdout:9/151: write d9/de/f11 [5381894,54200] 0 2024-04-22T15:49:28.844 INFO:tasks.workunit.client.0.smithi016.stdout:5/68: symlink le 0 2024-04-22T15:49:28.844 INFO:tasks.workunit.client.0.smithi016.stdout:5/69: read f6 [230431,107413] 0 2024-04-22T15:49:28.879 INFO:tasks.workunit.client.0.smithi016.stdout:9/152: creat d9/de/f28 x:0 0 0 2024-04-22T15:49:28.879 INFO:tasks.workunit.client.0.smithi016.stdout:2/77: truncate d2/f6 1619683 0 2024-04-22T15:49:28.879 INFO:tasks.workunit.client.0.smithi016.stdout:5/70: mkdir df 0 2024-04-22T15:49:28.880 INFO:tasks.workunit.client.0.smithi016.stdout:3/78: creat d7/d17/d18/f1f x:0 0 0 2024-04-22T15:49:28.881 INFO:tasks.workunit.client.0.smithi016.stdout:8/67: chown d2/db/fc 3805 1 2024-04-22T15:49:28.885 INFO:tasks.workunit.client.0.smithi016.stdout:2/78: mkdir d2/d23 0 2024-04-22T15:49:28.886 INFO:tasks.workunit.client.0.smithi016.stdout:7/101: truncate d3/d4/f7 2687104 0 2024-04-22T15:49:28.888 INFO:tasks.workunit.client.0.smithi016.stdout:7/102: fdatasync d3/fa 0 2024-04-22T15:49:28.888 INFO:tasks.workunit.client.0.smithi016.stdout:9/153: creat d9/de/d23/f29 x:0 0 0 2024-04-22T15:49:28.888 INFO:tasks.workunit.client.0.smithi016.stdout:6/82: dwrite d3/f8 [0,4194304] 0 2024-04-22T15:49:28.889 INFO:tasks.workunit.client.0.smithi016.stdout:3/79: dwrite d7/f1e [0,4194304] 0 2024-04-22T15:49:28.895 INFO:tasks.workunit.client.0.smithi016.stdout:5/71: write f0 [18331,72552] 0 2024-04-22T15:49:28.896 INFO:tasks.workunit.client.0.smithi016.stdout:0/60: symlink dc/l12 0 2024-04-22T15:49:28.898 INFO:tasks.workunit.client.0.smithi016.stdout:6/83: dread d3/f6 [0,4194304] 0 2024-04-22T15:49:28.899 INFO:tasks.workunit.client.0.smithi016.stdout:9/154: dwrite d9/de/f18 [0,4194304] 0 2024-04-22T15:49:28.900 INFO:tasks.workunit.client.0.smithi016.stdout:9/155: chown d9/de/d23/d24 69 1 2024-04-22T15:49:28.902 INFO:tasks.workunit.client.0.smithi016.stdout:6/84: dread d3/f6 [0,4194304] 0 2024-04-22T15:49:28.904 INFO:tasks.workunit.client.0.smithi016.stdout:2/79: symlink d2/d23/l24 0 2024-04-22T15:49:28.912 INFO:tasks.workunit.client.0.smithi016.stdout:2/80: mkdir d2/d1a/d25 0 2024-04-22T15:49:28.914 INFO:tasks.workunit.client.0.smithi016.stdout:0/61: dread fb [0,4194304] 0 2024-04-22T15:49:28.915 INFO:tasks.workunit.client.0.smithi016.stdout:8/68: getdents d2/db 0 2024-04-22T15:49:28.916 INFO:tasks.workunit.client.0.smithi016.stdout:9/156: write d9/f22 [1032024,90861] 0 2024-04-22T15:49:28.916 INFO:tasks.workunit.client.0.smithi016.stdout:2/81: unlink d2/f15 0 2024-04-22T15:49:28.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:28 smithi016 ceph-mon[26620]: pgmap v26: 65 pgs: 65 active+clean; 266 MiB data, 935 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 13 MiB/s wr, 321 op/s 2024-04-22T15:49:28.917 INFO:tasks.workunit.client.0.smithi016.stdout:2/82: chown d2/de/c22 525796886 1 2024-04-22T15:49:28.917 INFO:tasks.workunit.client.0.smithi016.stdout:3/80: getdents d7/d1b 0 2024-04-22T15:49:28.923 INFO:tasks.workunit.client.0.smithi016.stdout:3/81: write f1 [128332,57684] 0 2024-04-22T15:49:28.923 INFO:tasks.workunit.client.0.smithi016.stdout:4/82: sync 2024-04-22T15:49:28.924 INFO:tasks.workunit.client.0.smithi016.stdout:2/83: creat d2/f26 x:0 0 0 2024-04-22T15:49:28.924 INFO:tasks.workunit.client.0.smithi016.stdout:3/82: truncate d7/f12 945876 0 2024-04-22T15:49:28.924 INFO:tasks.workunit.client.0.smithi016.stdout:0/62: mkdir dc/d13 0 2024-04-22T15:49:28.924 INFO:tasks.workunit.client.0.smithi016.stdout:9/157: creat d9/de/d26/f2a x:0 0 0 2024-04-22T15:49:28.924 INFO:tasks.workunit.client.0.smithi016.stdout:9/158: chown d9/de/l21 142 1 2024-04-22T15:49:28.924 INFO:tasks.workunit.client.0.smithi016.stdout:9/159: readlink l1 0 2024-04-22T15:49:28.925 INFO:tasks.workunit.client.0.smithi016.stdout:4/83: creat d7/f1d x:0 0 0 2024-04-22T15:49:28.925 INFO:tasks.workunit.client.0.smithi016.stdout:4/84: chown f5 24924 1 2024-04-22T15:49:28.926 INFO:tasks.workunit.client.0.smithi016.stdout:4/85: dread - d7/f1d zero size 2024-04-22T15:49:28.926 INFO:tasks.workunit.client.0.smithi016.stdout:3/83: chown d7/lf 10701 1 2024-04-22T15:49:28.928 INFO:tasks.workunit.client.0.smithi016.stdout:2/84: mkdir d2/d23/d27 0 2024-04-22T15:49:28.929 INFO:tasks.workunit.client.0.smithi016.stdout:5/72: write f6 [709731,66351] 0 2024-04-22T15:49:28.931 INFO:tasks.workunit.client.0.smithi016.stdout:5/73: dread f0 [0,4194304] 0 2024-04-22T15:49:28.931 INFO:tasks.workunit.client.0.smithi016.stdout:3/84: rename d7 to d7/d20 22 2024-04-22T15:49:28.932 INFO:tasks.workunit.client.0.smithi016.stdout:3/85: write d7/f12 [749416,22917] 0 2024-04-22T15:49:28.933 INFO:tasks.workunit.client.0.smithi016.stdout:6/85: truncate d3/da/dd/f10 4876005 0 2024-04-22T15:49:28.934 INFO:tasks.workunit.client.0.smithi016.stdout:0/63: mkdir dc/d13/d14 0 2024-04-22T15:49:28.936 INFO:tasks.workunit.client.0.smithi016.stdout:0/64: fsync dc/f11 0 2024-04-22T15:49:28.937 INFO:tasks.workunit.client.0.smithi016.stdout:2/85: rename d2/d23/l24 to d2/de/l28 0 2024-04-22T15:49:28.939 INFO:tasks.workunit.client.0.smithi016.stdout:0/65: read fb [1845512,95497] 0 2024-04-22T15:49:28.940 INFO:tasks.workunit.client.0.smithi016.stdout:9/160: getdents d9/d20 0 2024-04-22T15:49:28.943 INFO:tasks.workunit.client.0.smithi016.stdout:0/66: chown dc/f11 175981 1 2024-04-22T15:49:28.944 INFO:tasks.workunit.client.0.smithi016.stdout:5/74: dwrite f0 [0,4194304] 0 2024-04-22T15:49:28.949 INFO:tasks.workunit.client.0.smithi016.stdout:0/67: dread - dc/f11 zero size 2024-04-22T15:49:28.949 INFO:tasks.workunit.client.0.smithi016.stdout:4/86: getdents d7 0 2024-04-22T15:49:28.951 INFO:tasks.workunit.client.0.smithi016.stdout:8/69: dwrite d2/f8 [0,4194304] 0 2024-04-22T15:49:28.954 INFO:tasks.workunit.client.0.smithi016.stdout:3/86: dwrite d7/f11 [0,4194304] 0 2024-04-22T15:49:28.959 INFO:tasks.workunit.client.0.smithi016.stdout:4/87: write d7/f14 [1311334,23981] 0 2024-04-22T15:49:28.967 INFO:tasks.workunit.client.0.smithi016.stdout:6/86: dread d3/f8 [0,4194304] 0 2024-04-22T15:49:28.967 INFO:tasks.workunit.client.0.smithi016.stdout:4/88: fdatasync d7/f16 0 2024-04-22T15:49:28.967 INFO:tasks.workunit.client.0.smithi016.stdout:3/87: mkdir d7/d1b/d21 0 2024-04-22T15:49:28.968 INFO:tasks.workunit.client.0.smithi016.stdout:8/70: mkdir d2/d10 0 2024-04-22T15:49:28.969 INFO:tasks.workunit.client.0.smithi016.stdout:3/88: stat d7/lf 0 2024-04-22T15:49:28.970 INFO:tasks.workunit.client.0.smithi016.stdout:1/329: sync 2024-04-22T15:49:28.971 INFO:tasks.workunit.client.0.smithi016.stdout:5/75: dread f2 [0,4194304] 0 2024-04-22T15:49:28.973 INFO:tasks.workunit.client.0.smithi016.stdout:0/68: dwrite f4 [0,4194304] 0 2024-04-22T15:49:28.977 INFO:tasks.workunit.client.0.smithi016.stdout:0/69: readlink dc/l10 0 2024-04-22T15:49:28.977 INFO:tasks.workunit.client.0.smithi016.stdout:1/330: write d3/d4/fe [3730825,29097] 0 2024-04-22T15:49:28.980 INFO:tasks.workunit.client.0.smithi016.stdout:5/76: rename f0 to df/f10 0 2024-04-22T15:49:28.980 INFO:tasks.workunit.client.0.smithi016.stdout:9/161: dread d9/fc [0,4194304] 0 2024-04-22T15:49:28.982 INFO:tasks.workunit.client.0.smithi016.stdout:2/86: dread d2/d9/fa [4194304,4194304] 0 2024-04-22T15:49:28.982 INFO:tasks.workunit.client.0.smithi016.stdout:0/70: write dc/f11 [700034,10090] 0 2024-04-22T15:49:28.983 INFO:tasks.workunit.client.0.smithi016.stdout:5/77: write fd [41239,50207] 0 2024-04-22T15:49:28.985 INFO:tasks.workunit.client.0.smithi016.stdout:4/89: dread d7/f10 [0,4194304] 0 2024-04-22T15:49:28.985 INFO:tasks.workunit.client.0.smithi016.stdout:8/71: dread f1 [0,4194304] 0 2024-04-22T15:49:28.985 INFO:tasks.workunit.client.0.smithi016.stdout:1/331: rmdir d3 39 2024-04-22T15:49:28.986 INFO:tasks.workunit.client.0.smithi016.stdout:6/87: dread d3/f8 [0,4194304] 0 2024-04-22T15:49:28.987 INFO:tasks.workunit.client.0.smithi016.stdout:3/89: dread d7/f14 [0,4194304] 0 2024-04-22T15:49:28.991 INFO:tasks.workunit.client.0.smithi016.stdout:2/87: chown d2/f6 10 1 2024-04-22T15:49:28.996 INFO:tasks.workunit.client.0.smithi016.stdout:6/88: mkdir d3/da/db/d14 0 2024-04-22T15:49:28.996 INFO:tasks.workunit.client.0.smithi016.stdout:4/90: chown d7/ce 14 1 2024-04-22T15:49:28.996 INFO:tasks.workunit.client.0.smithi016.stdout:0/71: mkdir dc/d13/d14/d15 0 2024-04-22T15:49:28.999 INFO:tasks.workunit.client.0.smithi016.stdout:1/332: readlink d3/d4/d37/l49 0 2024-04-22T15:49:29.004 INFO:tasks.workunit.client.0.smithi016.stdout:2/88: write f0 [7025674,3399] 0 2024-04-22T15:49:29.004 INFO:tasks.workunit.client.0.smithi016.stdout:5/78: dwrite f6 [4194304,4194304] 0 2024-04-22T15:49:29.006 INFO:tasks.workunit.client.0.smithi016.stdout:9/162: dwrite d9/f22 [0,4194304] 0 2024-04-22T15:49:29.009 INFO:tasks.workunit.client.0.smithi016.stdout:4/91: rename d7/f17 to d7/f1e 0 2024-04-22T15:49:29.010 INFO:tasks.workunit.client.0.smithi016.stdout:8/72: dwrite d2/f8 [4194304,4194304] 0 2024-04-22T15:49:29.013 INFO:tasks.workunit.client.0.smithi016.stdout:3/90: dwrite f1 [0,4194304] 0 2024-04-22T15:49:29.015 INFO:tasks.workunit.client.0.smithi016.stdout:6/89: creat d3/da/d12/f15 x:0 0 0 2024-04-22T15:49:29.016 INFO:tasks.workunit.client.0.smithi016.stdout:5/79: rmdir df 39 2024-04-22T15:49:29.019 INFO:tasks.workunit.client.0.smithi016.stdout:3/91: read d7/f11 [1029713,83513] 0 2024-04-22T15:49:29.021 INFO:tasks.workunit.client.0.smithi016.stdout:2/89: dread d2/f5 [4194304,4194304] 0 2024-04-22T15:49:29.023 INFO:tasks.workunit.client.0.smithi016.stdout:6/90: truncate d3/da/d12/f15 816791 0 2024-04-22T15:49:29.024 INFO:tasks.workunit.client.0.smithi016.stdout:1/333: dread d3/d4/d37/f40 [0,4194304] 0 2024-04-22T15:49:29.025 INFO:tasks.workunit.client.0.smithi016.stdout:3/92: write d7/f14 [2248660,29083] 0 2024-04-22T15:49:29.025 INFO:tasks.workunit.client.0.smithi016.stdout:2/90: symlink d2/de/l29 0 2024-04-22T15:49:29.025 INFO:tasks.workunit.client.0.smithi016.stdout:6/91: fsync d3/da/d12/ff 0 2024-04-22T15:49:29.027 INFO:tasks.workunit.client.0.smithi016.stdout:2/91: fdatasync d2/f7 0 2024-04-22T15:49:29.027 INFO:tasks.workunit.client.0.smithi016.stdout:8/73: getdents d2/d10 0 2024-04-22T15:49:29.027 INFO:tasks.workunit.client.0.smithi016.stdout:9/163: getdents d9/de/d23/d24/d25 0 2024-04-22T15:49:29.028 INFO:tasks.workunit.client.0.smithi016.stdout:6/92: chown d3/l9 15 1 2024-04-22T15:49:29.028 INFO:tasks.workunit.client.0.smithi016.stdout:4/92: link d7/l1a d7/l1f 0 2024-04-22T15:49:29.030 INFO:tasks.workunit.client.0.smithi016.stdout:4/93: truncate d7/f19 997477 0 2024-04-22T15:49:29.031 INFO:tasks.workunit.client.0.smithi016.stdout:6/93: read - d3/da/d12/ff zero size 2024-04-22T15:49:29.031 INFO:tasks.workunit.client.0.smithi016.stdout:8/74: write f1 [3766762,103380] 0 2024-04-22T15:49:29.033 INFO:tasks.workunit.client.0.smithi016.stdout:9/164: symlink d9/de/l2b 0 2024-04-22T15:49:29.034 INFO:tasks.workunit.client.0.smithi016.stdout:4/94: write d7/f19 [282381,60885] 0 2024-04-22T15:49:29.035 INFO:tasks.workunit.client.0.smithi016.stdout:8/75: write d2/f8 [9189336,23122] 0 2024-04-22T15:49:29.035 INFO:tasks.workunit.client.0.smithi016.stdout:2/92: fdatasync d2/d9/fa 0 2024-04-22T15:49:29.037 INFO:tasks.workunit.client.0.smithi016.stdout:6/94: rmdir d3/da 39 2024-04-22T15:49:29.039 INFO:tasks.workunit.client.0.smithi016.stdout:9/165: dread - d9/fa zero size 2024-04-22T15:49:29.039 INFO:tasks.workunit.client.0.smithi016.stdout:5/80: dwrite df/f10 [0,4194304] 0 2024-04-22T15:49:29.043 INFO:tasks.workunit.client.0.smithi016.stdout:8/76: dread d2/f8 [0,4194304] 0 2024-04-22T15:49:29.046 INFO:tasks.workunit.client.0.smithi016.stdout:5/81: creat df/f11 x:0 0 0 2024-04-22T15:49:29.046 INFO:tasks.workunit.client.0.smithi016.stdout:5/82: chown cb 1 1 2024-04-22T15:49:29.048 INFO:tasks.workunit.client.0.smithi016.stdout:4/95: dread d7/f11 [0,4194304] 0 2024-04-22T15:49:29.049 INFO:tasks.workunit.client.0.smithi016.stdout:3/93: getdents d7 0 2024-04-22T15:49:29.050 INFO:tasks.workunit.client.0.smithi016.stdout:4/96: chown d7/cb 883156169 1 2024-04-22T15:49:29.051 INFO:tasks.workunit.client.0.smithi016.stdout:1/334: rename d3/d4/d19/d20/l3d to d3/d4/l6c 0 2024-04-22T15:49:29.052 INFO:tasks.workunit.client.0.smithi016.stdout:1/335: write d3/d4/fe [2581324,72468] 0 2024-04-22T15:49:29.054 INFO:tasks.workunit.client.0.smithi016.stdout:9/166: creat d9/de/d23/d24/d25/f2c x:0 0 0 2024-04-22T15:49:29.055 INFO:tasks.workunit.client.0.smithi016.stdout:8/77: unlink d2/db/cf 0 2024-04-22T15:49:29.055 INFO:tasks.workunit.client.0.smithi016.stdout:8/78: chown d2/ld 111 1 2024-04-22T15:49:29.056 INFO:tasks.workunit.client.0.smithi016.stdout:3/94: creat d7/f22 x:0 0 0 2024-04-22T15:49:29.058 INFO:tasks.workunit.client.0.smithi016.stdout:2/93: rename d2/d1a/d25 to d2/de/d2a 0 2024-04-22T15:49:29.059 INFO:tasks.workunit.client.0.smithi016.stdout:2/94: dread d2/f18 [0,4194304] 0 2024-04-22T15:49:29.060 INFO:tasks.workunit.client.0.smithi016.stdout:6/95: mkdir d3/da/db/d14/d16 0 2024-04-22T15:49:29.063 INFO:tasks.workunit.client.0.smithi016.stdout:1/336: creat d3/d2e/f6d x:0 0 0 2024-04-22T15:49:29.065 INFO:tasks.workunit.client.0.smithi016.stdout:8/79: mknod d2/db/c11 0 2024-04-22T15:49:29.066 INFO:tasks.workunit.client.0.smithi016.stdout:2/95: dwrite d2/f26 [0,4194304] 0 2024-04-22T15:49:29.069 INFO:tasks.workunit.client.0.smithi016.stdout:7/103: sync 2024-04-22T15:49:29.071 INFO:tasks.workunit.client.0.smithi016.stdout:2/96: dwrite d2/d9/f12 [0,4194304] 0 2024-04-22T15:49:29.072 INFO:tasks.workunit.client.0.smithi016.stdout:6/96: creat d3/da/db/f17 x:0 0 0 2024-04-22T15:49:29.081 INFO:tasks.workunit.client.0.smithi016.stdout:5/83: truncate f6 5006117 0 2024-04-22T15:49:29.087 INFO:tasks.workunit.client.0.smithi016.stdout:2/97: creat d2/f2b x:0 0 0 2024-04-22T15:49:29.097 INFO:tasks.workunit.client.0.smithi016.stdout:4/97: link d7/l1f d7/l20 0 2024-04-22T15:49:29.099 INFO:tasks.workunit.client.0.smithi016.stdout:7/104: mknod d3/d4/d9/df/c29 0 2024-04-22T15:49:29.100 INFO:tasks.workunit.client.0.smithi016.stdout:2/98: mkdir d2/d2c 0 2024-04-22T15:49:29.101 INFO:tasks.workunit.client.0.smithi016.stdout:4/98: dread d7/f19 [0,4194304] 0 2024-04-22T15:49:29.103 INFO:tasks.workunit.client.0.smithi016.stdout:6/97: creat d3/da/dd/f18 x:0 0 0 2024-04-22T15:49:29.103 INFO:tasks.workunit.client.0.smithi016.stdout:9/167: link d9/l12 d9/l2d 0 2024-04-22T15:49:29.104 INFO:tasks.workunit.client.0.smithi016.stdout:1/337: link d3/d4/da/ld d3/d4/d3a/l6e 0 2024-04-22T15:49:29.105 INFO:tasks.workunit.client.0.smithi016.stdout:4/99: dread d7/f14 [0,4194304] 0 2024-04-22T15:49:29.111 INFO:tasks.workunit.client.0.smithi016.stdout:9/168: write d9/f14 [235351,28301] 0 2024-04-22T15:49:29.112 INFO:tasks.workunit.client.0.smithi016.stdout:9/169: read d9/de/f18 [2387482,103904] 0 2024-04-22T15:49:29.113 INFO:tasks.workunit.client.0.smithi016.stdout:9/170: read d9/f22 [3185989,81402] 0 2024-04-22T15:49:29.113 INFO:tasks.workunit.client.0.smithi016.stdout:1/338: creat d3/d4/d15/f6f x:0 0 0 2024-04-22T15:49:29.114 INFO:tasks.workunit.client.0.smithi016.stdout:9/171: write d9/de/f1a [344303,35211] 0 2024-04-22T15:49:29.115 INFO:tasks.workunit.client.0.smithi016.stdout:1/339: write d3/d4/d19/d20/f48 [3049913,74852] 0 2024-04-22T15:49:29.120 INFO:tasks.workunit.client.0.smithi016.stdout:3/95: truncate d7/f1e 2725243 0 2024-04-22T15:49:29.120 INFO:tasks.workunit.client.0.smithi016.stdout:1/340: dread d3/d4/d19/f54 [0,4194304] 0 2024-04-22T15:49:29.122 INFO:tasks.workunit.client.0.smithi016.stdout:6/98: mknod d3/da/db/d14/d16/c19 0 2024-04-22T15:49:29.122 INFO:tasks.workunit.client.0.smithi016.stdout:5/84: stat f6 0 2024-04-22T15:49:29.127 INFO:tasks.workunit.client.0.smithi016.stdout:9/172: creat d9/de/d23/d24/f2e x:0 0 0 2024-04-22T15:49:29.127 INFO:tasks.workunit.client.0.smithi016.stdout:0/72: sync 2024-04-22T15:49:29.129 INFO:tasks.workunit.client.0.smithi016.stdout:0/73: write f4 [5900646,119649] 0 2024-04-22T15:49:29.131 INFO:tasks.workunit.client.0.smithi016.stdout:7/105: creat d3/d4/d9/f2a x:0 0 0 2024-04-22T15:49:29.131 INFO:tasks.workunit.client.0.smithi016.stdout:0/74: fsync f4 0 2024-04-22T15:49:29.135 INFO:tasks.workunit.client.0.smithi016.stdout:5/85: mknod df/c12 0 2024-04-22T15:49:29.136 INFO:tasks.workunit.client.0.smithi016.stdout:5/86: fdatasync df/f11 0 2024-04-22T15:49:29.136 INFO:tasks.workunit.client.0.smithi016.stdout:3/96: dread d7/f12 [0,4194304] 0 2024-04-22T15:49:29.137 INFO:tasks.workunit.client.0.smithi016.stdout:6/99: fsync d3/da/dd/f10 0 2024-04-22T15:49:29.138 INFO:tasks.workunit.client.0.smithi016.stdout:5/87: write f2 [2890610,70184] 0 2024-04-22T15:49:29.139 INFO:tasks.workunit.client.0.smithi016.stdout:9/173: readlink d9/l1e 0 2024-04-22T15:49:29.146 INFO:tasks.workunit.client.0.smithi016.stdout:0/75: rename dc/f11 to dc/d13/f16 0 2024-04-22T15:49:29.146 INFO:tasks.workunit.client.0.smithi016.stdout:2/99: getdents d2/d1a 0 2024-04-22T15:49:29.148 INFO:tasks.workunit.client.0.smithi016.stdout:3/97: fdatasync d7/f14 0 2024-04-22T15:49:29.150 INFO:tasks.workunit.client.0.smithi016.stdout:4/100: getdents d7 0 2024-04-22T15:49:29.155 INFO:tasks.workunit.client.0.smithi016.stdout:6/100: creat d3/da/f1a x:0 0 0 2024-04-22T15:49:29.155 INFO:tasks.workunit.client.0.smithi016.stdout:5/88: mknod df/c13 0 2024-04-22T15:49:29.156 INFO:tasks.workunit.client.0.smithi016.stdout:1/341: link d3/d4/d3a/l56 d3/d4/d45/l70 0 2024-04-22T15:49:29.157 INFO:tasks.workunit.client.0.smithi016.stdout:4/101: readlink d7/l9 0 2024-04-22T15:49:29.157 INFO:tasks.workunit.client.0.smithi016.stdout:5/89: fdatasync fd 0 2024-04-22T15:49:29.161 INFO:tasks.workunit.client.0.smithi016.stdout:5/90: write df/f10 [1896142,32892] 0 2024-04-22T15:49:29.162 INFO:tasks.workunit.client.0.smithi016.stdout:1/342: read d3/d4/f1e [790410,72311] 0 2024-04-22T15:49:29.163 INFO:tasks.workunit.client.0.smithi016.stdout:0/76: dwrite f4 [0,4194304] 0 2024-04-22T15:49:29.163 INFO:tasks.workunit.client.0.smithi016.stdout:4/102: mkdir d7/d21 0 2024-04-22T15:49:29.163 INFO:tasks.workunit.client.0.smithi016.stdout:2/100: dwrite d2/f2b [0,4194304] 0 2024-04-22T15:49:29.168 INFO:tasks.workunit.client.0.smithi016.stdout:6/101: dread d3/f8 [0,4194304] 0 2024-04-22T15:49:29.171 INFO:tasks.workunit.client.0.smithi016.stdout:5/91: read fd [32330,116917] 0 2024-04-22T15:49:29.171 INFO:tasks.workunit.client.0.smithi016.stdout:5/92: chown df/c13 22670150 1 2024-04-22T15:49:29.172 INFO:tasks.workunit.client.0.smithi016.stdout:5/93: stat df/f10 0 2024-04-22T15:49:29.173 INFO:tasks.workunit.client.0.smithi016.stdout:4/103: mkdir d7/d22 0 2024-04-22T15:49:29.174 INFO:tasks.workunit.client.0.smithi016.stdout:0/77: creat dc/d13/f17 x:0 0 0 2024-04-22T15:49:29.176 INFO:tasks.workunit.client.0.smithi016.stdout:1/343: symlink d3/d4/d15/l71 0 2024-04-22T15:49:29.176 INFO:tasks.workunit.client.0.smithi016.stdout:2/101: dread d2/f5 [4194304,4194304] 0 2024-04-22T15:49:29.177 INFO:tasks.workunit.client.0.smithi016.stdout:6/102: mkdir d3/da/dd/d1b 0 2024-04-22T15:49:29.178 INFO:tasks.workunit.client.0.smithi016.stdout:8/80: sync 2024-04-22T15:49:29.179 INFO:tasks.workunit.client.0.smithi016.stdout:4/104: readlink d7/l1a 0 2024-04-22T15:49:29.180 INFO:tasks.workunit.client.0.smithi016.stdout:0/78: stat dc/ce 0 2024-04-22T15:49:29.181 INFO:tasks.workunit.client.0.smithi016.stdout:2/102: mknod d2/d1a/c2d 0 2024-04-22T15:49:29.183 INFO:tasks.workunit.client.0.smithi016.stdout:7/106: dwrite d3/d4/f7 [0,4194304] 0 2024-04-22T15:49:29.183 INFO:tasks.workunit.client.0.smithi016.stdout:4/105: write d7/f16 [621297,66180] 0 2024-04-22T15:49:29.184 INFO:tasks.workunit.client.0.smithi016.stdout:8/81: mkdir d2/db/d12 0 2024-04-22T15:49:29.187 INFO:tasks.workunit.client.0.smithi016.stdout:7/107: dread - d3/d4/f17 zero size 2024-04-22T15:49:29.187 INFO:tasks.workunit.client.0.smithi016.stdout:0/79: creat dc/d13/d14/f18 x:0 0 0 2024-04-22T15:49:29.189 INFO:tasks.workunit.client.0.smithi016.stdout:2/103: fsync d2/f6 0 2024-04-22T15:49:29.189 INFO:tasks.workunit.client.0.smithi016.stdout:7/108: fdatasync d3/f28 0 2024-04-22T15:49:29.189 INFO:tasks.workunit.client.0.smithi016.stdout:3/98: write d7/f1e [2982546,119728] 0 2024-04-22T15:49:29.190 INFO:tasks.workunit.client.0.smithi016.stdout:6/103: link d3/da/f1a d3/da/d12/f1c 0 2024-04-22T15:49:29.193 INFO:tasks.workunit.client.0.smithi016.stdout:7/109: readlink d3/dc/d1b/l20 0 2024-04-22T15:49:29.194 INFO:tasks.workunit.client.0.smithi016.stdout:8/82: fdatasync d2/db/fc 0 2024-04-22T15:49:29.196 INFO:tasks.workunit.client.0.smithi016.stdout:6/104: fsync d3/da/dd/f10 0 2024-04-22T15:49:29.197 INFO:tasks.workunit.client.0.smithi016.stdout:1/344: dwrite d3/d4/d19/d20/f21 [0,4194304] 0 2024-04-22T15:49:29.197 INFO:tasks.workunit.client.0.smithi016.stdout:8/83: fsync f1 0 2024-04-22T15:49:29.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.196+0000 7f13808fa700 1 -- 172.21.15.16:0/208349992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f137c101750 msgr2=0x7f137c103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:29.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.196+0000 7f13808fa700 1 --2- 172.21.15.16:0/208349992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f137c101750 0x7f137c103b40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f136c009910 tx=0x7f136c009c20 comp rx=0 tx=0).stop 2024-04-22T15:49:29.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.196+0000 7f13808fa700 1 -- 172.21.15.16:0/208349992 shutdown_connections 2024-04-22T15:49:29.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.196+0000 7f13808fa700 1 --2- 172.21.15.16:0/208349992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f137c104080 0x7f137c106470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:29.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.196+0000 7f13808fa700 1 --2- 172.21.15.16:0/208349992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f137c101750 0x7f137c103b40 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:29.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.196+0000 7f13808fa700 1 -- 172.21.15.16:0/208349992 >> 172.21.15.16:0/208349992 conn(0x7f137c0fb160 msgr2=0x7f137c0fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:29.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.196+0000 7f13808fa700 1 -- 172.21.15.16:0/208349992 shutdown_connections 2024-04-22T15:49:29.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.196+0000 7f13808fa700 1 -- 172.21.15.16:0/208349992 wait complete. 2024-04-22T15:49:29.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.198+0000 7f13808fa700 1 Processor -- start 2024-04-22T15:49:29.202 INFO:tasks.workunit.client.0.smithi016.stdout:4/106: mkdir d7/d21/d23 0 2024-04-22T15:49:29.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.198+0000 7f13808fa700 1 -- start start 2024-04-22T15:49:29.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.198+0000 7f13808fa700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f137c101750 0x7f137c19cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:29.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.198+0000 7f13808fa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f137c104080 0x7f137c19d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:29.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.198+0000 7f137a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f137c101750 0x7f137c19cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:29.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.198+0000 7f137a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f137c101750 0x7f137c19cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:44762/0 (socket says 172.21.15.16:44762) 2024-04-22T15:49:29.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.198+0000 7f137a59c700 1 -- 172.21.15.16:0/24961081 learned_addr learned my addr 172.21.15.16:0/24961081 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:49:29.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.199+0000 7f13808fa700 1 -- 172.21.15.16:0/24961081 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137c19d7c0 con 0x7f137c104080 2024-04-22T15:49:29.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.199+0000 7f13808fa700 1 -- 172.21.15.16:0/24961081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137c1a21d0 con 0x7f137c101750 2024-04-22T15:49:29.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.199+0000 7f1379d9b700 1 --2- 172.21.15.16:0/24961081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f137c104080 0x7f137c19d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:29.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.199+0000 7f137a59c700 1 -- 172.21.15.16:0/24961081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f137c104080 msgr2=0x7f137c19d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:29.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.199+0000 7f137a59c700 1 --2- 172.21.15.16:0/24961081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f137c104080 0x7f137c19d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:29.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.199+0000 7f137a59c700 1 -- 172.21.15.16:0/24961081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f136c0095f0 con 0x7f137c101750 2024-04-22T15:49:29.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.200+0000 7f137a59c700 1 --2- 172.21.15.16:0/24961081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f137c101750 0x7f137c19cc60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f136c00beb0 tx=0x7f136c00bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:29.204 INFO:tasks.workunit.client.0.smithi016.stdout:4/107: dread - f5 zero size 2024-04-22T15:49:29.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.200+0000 7f136b7fe700 1 -- 172.21.15.16:0/24961081 <== mon.1 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f136c01d070 con 0x7f137c101750 2024-04-22T15:49:29.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.200+0000 7f136b7fe700 1 -- 172.21.15.16:0/24961081 <== mon.1 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f136c004c80 con 0x7f137c101750 2024-04-22T15:49:29.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.200+0000 7f136b7fe700 1 -- 172.21.15.16:0/24961081 <== mon.1 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f136c00f6f0 con 0x7f137c101750 2024-04-22T15:49:29.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.200+0000 7f13808fa700 1 -- 172.21.15.16:0/24961081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f137c1a2370 con 0x7f137c101750 2024-04-22T15:49:29.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.200+0000 7f13808fa700 1 -- 172.21.15.16:0/24961081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f137c1a2770 con 0x7f137c101750 2024-04-22T15:49:29.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.202+0000 7f13808fa700 1 -- 172.21.15.16:0/24961081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f137c1097b0 con 0x7f137c101750 2024-04-22T15:49:29.207 INFO:tasks.workunit.client.0.smithi016.stdout:0/80: mknod dc/d13/d14/d15/c19 0 2024-04-22T15:49:29.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.206+0000 7f136b7fe700 1 -- 172.21.15.16:0/24961081 <== mon.1 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) v1 ==== 91557+0+0 (secure 0 0 0) 0x7f136c004df0 con 0x7f137c101750 2024-04-22T15:49:29.210 INFO:tasks.workunit.client.0.smithi016.stdout:2/104: creat d2/d23/d27/f2e x:0 0 0 2024-04-22T15:49:29.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.207+0000 7f136b7fe700 1 --2- 172.21.15.16:0/24961081 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f136406d880 0x7f136406fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:29.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.207+0000 7f136b7fe700 1 -- 172.21.15.16:0/24961081 <== mon.1 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f136c08ebc0 con 0x7f137c101750 2024-04-22T15:49:29.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.207+0000 7f1379d9b700 1 --2- 172.21.15.16:0/24961081 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f136406d880 0x7f136406fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:29.211 INFO:tasks.workunit.client.0.smithi016.stdout:3/99: dwrite d7/f14 [0,4194304] 0 2024-04-22T15:49:29.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.208+0000 7f1379d9b700 1 --2- 172.21.15.16:0/24961081 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f136406d880 0x7f136406fd40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1370006fd0 tx=0x7f1370009190 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:29.213 INFO:tasks.workunit.client.0.smithi016.stdout:7/110: mknod d3/dc/c2b 0 2024-04-22T15:49:29.213 INFO:tasks.workunit.client.0.smithi016.stdout:0/81: write dc/d13/f17 [503663,3325] 0 2024-04-22T15:49:29.213 INFO:tasks.workunit.client.0.smithi016.stdout:2/105: stat d2/f7 0 2024-04-22T15:49:29.213 INFO:tasks.workunit.client.0.smithi016.stdout:4/108: mkdir d7/d24 0 2024-04-22T15:49:29.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.211+0000 7f136b7fe700 1 -- 172.21.15.16:0/24961081 <== mon.1 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f136c057e90 con 0x7f137c101750 2024-04-22T15:49:29.214 INFO:tasks.workunit.client.0.smithi016.stdout:1/345: truncate d3/d4/d19/d20/d2b/f59 2655575 0 2024-04-22T15:49:29.217 INFO:tasks.workunit.client.0.smithi016.stdout:0/82: symlink dc/d13/l1a 0 2024-04-22T15:49:29.217 INFO:tasks.workunit.client.0.smithi016.stdout:7/111: rename f0 to d3/d14/f2c 0 2024-04-22T15:49:29.218 INFO:tasks.workunit.client.0.smithi016.stdout:0/83: stat dc/ce 0 2024-04-22T15:49:29.218 INFO:tasks.workunit.client.0.smithi016.stdout:6/105: creat d3/f1d x:0 0 0 2024-04-22T15:49:29.219 INFO:tasks.workunit.client.0.smithi016.stdout:4/109: unlink d7/ce 0 2024-04-22T15:49:29.220 INFO:tasks.workunit.client.0.smithi016.stdout:8/84: creat d2/f13 x:0 0 0 2024-04-22T15:49:29.221 INFO:tasks.workunit.client.0.smithi016.stdout:1/346: creat d3/d4/d37/f72 x:0 0 0 2024-04-22T15:49:29.221 INFO:tasks.workunit.client.0.smithi016.stdout:4/110: dread - f5 zero size 2024-04-22T15:49:29.223 INFO:tasks.workunit.client.0.smithi016.stdout:2/106: mkdir d2/d9/d2f 0 2024-04-22T15:49:29.225 INFO:tasks.workunit.client.0.smithi016.stdout:7/112: rmdir d3/d4 39 2024-04-22T15:49:29.227 INFO:tasks.workunit.client.0.smithi016.stdout:6/106: dread d3/da/dd/f10 [0,4194304] 0 2024-04-22T15:49:29.228 INFO:tasks.workunit.client.0.smithi016.stdout:5/94: sync 2024-04-22T15:49:29.229 INFO:tasks.workunit.client.0.smithi016.stdout:9/174: sync 2024-04-22T15:49:29.229 INFO:tasks.workunit.client.0.smithi016.stdout:8/85: fsync d2/db/fc 0 2024-04-22T15:49:29.229 INFO:tasks.workunit.client.0.smithi016.stdout:3/100: rename d7/f11 to d7/d17/d18/f23 0 2024-04-22T15:49:29.231 INFO:tasks.workunit.client.0.smithi016.stdout:4/111: rename d7/l9 to d7/l25 0 2024-04-22T15:49:29.236 INFO:tasks.workunit.client.0.smithi016.stdout:2/107: dread d2/f5 [4194304,4194304] 0 2024-04-22T15:49:29.239 INFO:tasks.workunit.client.0.smithi016.stdout:0/84: dwrite dc/d13/f16 [0,4194304] 0 2024-04-22T15:49:29.240 INFO:tasks.workunit.client.0.smithi016.stdout:6/107: dwrite d3/f8 [0,4194304] 0 2024-04-22T15:49:29.240 INFO:tasks.workunit.client.0.smithi016.stdout:5/95: creat df/f14 x:0 0 0 2024-04-22T15:49:29.243 INFO:tasks.workunit.client.0.smithi016.stdout:0/85: chown dc/d13/d14/d15/c19 142424862 1 2024-04-22T15:49:29.243 INFO:tasks.workunit.client.0.smithi016.stdout:5/96: dread - df/f11 zero size 2024-04-22T15:49:29.247 INFO:tasks.workunit.client.0.smithi016.stdout:5/97: write fd [1013913,104182] 0 2024-04-22T15:49:29.252 INFO:tasks.workunit.client.0.smithi016.stdout:7/113: symlink d3/dc/d22/l2d 0 2024-04-22T15:49:29.254 INFO:tasks.workunit.client.0.smithi016.stdout:2/108: mknod d2/d23/c30 0 2024-04-22T15:49:29.258 INFO:tasks.workunit.client.0.smithi016.stdout:1/347: rename d3/d4/d15/d4c/f55 to d3/d2e/f73 0 2024-04-22T15:49:29.260 INFO:tasks.workunit.client.0.smithi016.stdout:6/108: rmdir d3 39 2024-04-22T15:49:29.262 INFO:tasks.workunit.client.0.smithi016.stdout:0/86: rmdir dc/d13 39 2024-04-22T15:49:29.269 INFO:tasks.workunit.client.0.smithi016.stdout:2/109: creat d2/d23/d27/f31 x:0 0 0 2024-04-22T15:49:29.271 INFO:tasks.workunit.client.0.smithi016.stdout:1/348: fdatasync d3/d4/d19/d20/d2b/f4a 0 2024-04-22T15:49:29.277 INFO:tasks.workunit.client.0.smithi016.stdout:3/101: symlink d7/d17/l24 0 2024-04-22T15:49:29.278 INFO:tasks.workunit.client.0.smithi016.stdout:6/109: write d3/f6 [1405804,81475] 0 2024-04-22T15:49:29.278 INFO:tasks.workunit.client.0.smithi016.stdout:1/349: write d3/d4/d15/f6f [149756,103461] 0 2024-04-22T15:49:29.279 INFO:tasks.workunit.client.0.smithi016.stdout:8/86: dwrite d2/f8 [4194304,4194304] 0 2024-04-22T15:49:29.281 INFO:tasks.workunit.client.0.smithi016.stdout:9/175: getdents d9 0 2024-04-22T15:49:29.285 INFO:tasks.workunit.client.0.smithi016.stdout:2/110: rename d2/f6 to d2/d23/d27/f32 0 2024-04-22T15:49:29.285 INFO:tasks.workunit.client.0.smithi016.stdout:8/87: chown d2/db/d12 3210 1 2024-04-22T15:49:29.287 INFO:tasks.workunit.client.0.smithi016.stdout:5/98: write f6 [5173876,61729] 0 2024-04-22T15:49:29.293 INFO:tasks.workunit.client.0.smithi016.stdout:2/111: fdatasync d2/d23/d27/f2e 0 2024-04-22T15:49:29.294 INFO:tasks.workunit.client.0.smithi016.stdout:1/350: stat d3/d4/d3a/l56 0 2024-04-22T15:49:29.296 INFO:tasks.workunit.client.0.smithi016.stdout:6/110: fsync d3/da/f1a 0 2024-04-22T15:49:29.296 INFO:tasks.workunit.client.0.smithi016.stdout:4/112: dwrite d7/f19 [0,4194304] 0 2024-04-22T15:49:29.297 INFO:tasks.workunit.client.0.smithi016.stdout:5/99: write f6 [3079601,111457] 0 2024-04-22T15:49:29.297 INFO:tasks.workunit.client.0.smithi016.stdout:8/88: fsync d2/db/fc 0 2024-04-22T15:49:29.298 INFO:tasks.workunit.client.0.smithi016.stdout:7/114: dwrite d3/d4/d9/f25 [0,4194304] 0 2024-04-22T15:49:29.305 INFO:tasks.workunit.client.0.smithi016.stdout:0/87: dwrite dc/d13/d14/f18 [0,4194304] 0 2024-04-22T15:49:29.306 INFO:tasks.workunit.client.0.smithi016.stdout:6/111: fdatasync d3/da/f1a 0 2024-04-22T15:49:29.306 INFO:tasks.workunit.client.0.smithi016.stdout:5/100: truncate df/f11 197360 0 2024-04-22T15:49:29.309 INFO:tasks.workunit.client.0.smithi016.stdout:5/101: dread - df/f14 zero size 2024-04-22T15:49:29.311 INFO:tasks.workunit.client.0.smithi016.stdout:4/113: write f2 [4868758,43138] 0 2024-04-22T15:49:29.311 INFO:tasks.workunit.client.0.smithi016.stdout:1/351: creat d3/d28/f74 x:0 0 0 2024-04-22T15:49:29.311 INFO:tasks.workunit.client.0.smithi016.stdout:0/88: write f4 [2273554,59863] 0 2024-04-22T15:49:29.312 INFO:tasks.workunit.client.0.smithi016.stdout:7/115: rmdir d3/d4 39 2024-04-22T15:49:29.314 INFO:tasks.workunit.client.0.smithi016.stdout:3/102: dread d7/d17/d18/f23 [0,4194304] 0 2024-04-22T15:49:29.315 INFO:tasks.workunit.client.0.smithi016.stdout:2/112: chown d2/d23/d27/f32 747352 1 2024-04-22T15:49:29.318 INFO:tasks.workunit.client.0.smithi016.stdout:8/89: creat d2/d10/f14 x:0 0 0 2024-04-22T15:49:29.318 INFO:tasks.workunit.client.0.smithi016.stdout:4/114: write d7/f19 [2819980,60015] 0 2024-04-22T15:49:29.319 INFO:tasks.workunit.client.0.smithi016.stdout:8/90: fsync d2/f13 0 2024-04-22T15:49:29.321 INFO:tasks.workunit.client.0.smithi016.stdout:2/113: mkdir d2/d23/d27/d33 0 2024-04-22T15:49:29.322 INFO:tasks.workunit.client.0.smithi016.stdout:7/116: write d3/f23 [538783,66006] 0 2024-04-22T15:49:29.323 INFO:tasks.workunit.client.0.smithi016.stdout:1/352: chown d3/d2e/f6d 158176636 1 2024-04-22T15:49:29.323 INFO:tasks.workunit.client.0.smithi016.stdout:2/114: chown d2/d23/c30 99 1 2024-04-22T15:49:29.323 INFO:tasks.workunit.client.0.smithi016.stdout:8/91: dread - d2/f13 zero size 2024-04-22T15:49:29.324 INFO:tasks.workunit.client.0.smithi016.stdout:6/112: creat d3/da/db/d14/f1e x:0 0 0 2024-04-22T15:49:29.326 INFO:tasks.workunit.client.0.smithi016.stdout:0/89: mknod dc/c1b 0 2024-04-22T15:49:29.327 INFO:tasks.workunit.client.0.smithi016.stdout:5/102: mkdir df/d15 0 2024-04-22T15:49:29.329 INFO:tasks.workunit.client.0.smithi016.stdout:9/176: truncate d9/de/f16 1404299 0 2024-04-22T15:49:29.329 INFO:tasks.workunit.client.0.smithi016.stdout:5/103: chown f2 0 1 2024-04-22T15:49:29.330 INFO:tasks.workunit.client.0.smithi016.stdout:2/115: mkdir d2/d23/d27/d34 0 2024-04-22T15:49:29.334 INFO:tasks.workunit.client.0.smithi016.stdout:3/103: chown d7/l1a 3 1 2024-04-22T15:49:29.334 INFO:tasks.workunit.client.0.smithi016.stdout:4/115: dread d7/f8 [0,4194304] 0 2024-04-22T15:49:29.334 INFO:tasks.workunit.client.0.smithi016.stdout:6/113: symlink d3/da/db/l1f 0 2024-04-22T15:49:29.335 INFO:tasks.workunit.client.0.smithi016.stdout:6/114: stat d3/da/f1a 0 2024-04-22T15:49:29.336 INFO:tasks.workunit.client.0.smithi016.stdout:2/116: write d2/d23/d27/f31 [901836,112086] 0 2024-04-22T15:49:29.340 INFO:tasks.workunit.client.0.smithi016.stdout:1/353: rename d3/d4/d15/f6f to d3/d4/d15/d38/d63/f75 0 2024-04-22T15:49:29.341 INFO:tasks.workunit.client.0.smithi016.stdout:7/117: symlink d3/d4/d9/df/l2e 0 2024-04-22T15:49:29.345 INFO:tasks.workunit.client.0.smithi016.stdout:3/104: dread f1 [0,4194304] 0 2024-04-22T15:49:29.347 INFO:tasks.workunit.client.0.smithi016.stdout:7/118: write d3/d4/d9/f2a [168212,100980] 0 2024-04-22T15:49:29.348 INFO:tasks.workunit.client.0.smithi016.stdout:9/177: dwrite f2 [0,4194304] 0 2024-04-22T15:49:29.348 INFO:tasks.workunit.client.0.smithi016.stdout:5/104: dread f6 [0,4194304] 0 2024-04-22T15:49:29.349 INFO:tasks.workunit.client.0.smithi016.stdout:6/115: creat d3/da/db/d14/d16/f20 x:0 0 0 2024-04-22T15:49:29.351 INFO:tasks.workunit.client.0.smithi016.stdout:8/92: rmdir d2/db/d12 0 2024-04-22T15:49:29.352 INFO:tasks.workunit.client.0.smithi016.stdout:6/116: chown d3/da/db/l1f 6 1 2024-04-22T15:49:29.353 INFO:tasks.workunit.client.0.smithi016.stdout:8/93: fsync d2/d10/f14 0 2024-04-22T15:49:29.353 INFO:tasks.workunit.client.0.smithi016.stdout:4/116: getdents d7/d21/d23 0 2024-04-22T15:49:29.355 INFO:tasks.workunit.client.0.smithi016.stdout:6/117: write d3/da/db/d14/f1e [596625,92093] 0 2024-04-22T15:49:29.356 INFO:tasks.workunit.client.0.smithi016.stdout:1/354: dread d3/d4/fe [0,4194304] 0 2024-04-22T15:49:29.356 INFO:tasks.workunit.client.0.smithi016.stdout:6/118: readlink d3/l9 0 2024-04-22T15:49:29.359 INFO:tasks.workunit.client.0.smithi016.stdout:0/90: getdents dc 0 2024-04-22T15:49:29.362 INFO:tasks.workunit.client.0.smithi016.stdout:9/178: creat d9/de/d23/d24/f2f x:0 0 0 2024-04-22T15:49:29.364 INFO:tasks.workunit.client.0.smithi016.stdout:7/119: dread d3/d4/fd [0,4194304] 0 2024-04-22T15:49:29.366 INFO:tasks.workunit.client.0.smithi016.stdout:8/94: mknod d2/d10/c15 0 2024-04-22T15:49:29.367 INFO:tasks.workunit.client.0.smithi016.stdout:8/95: fsync d2/f8 0 2024-04-22T15:49:29.368 INFO:tasks.workunit.client.0.smithi016.stdout:1/355: rename d3/f29 to d3/d4/d15/d4c/f76 0 2024-04-22T15:49:29.373 INFO:tasks.workunit.client.0.smithi016.stdout:0/91: creat dc/d13/d14/d15/f1c x:0 0 0 2024-04-22T15:49:29.375 INFO:tasks.workunit.client.0.smithi016.stdout:2/117: dwrite d2/f11 [0,4194304] 0 2024-04-22T15:49:29.375 INFO:tasks.workunit.client.0.smithi016.stdout:0/92: stat dc/l10 0 2024-04-22T15:49:29.375 INFO:tasks.workunit.client.0.smithi016.stdout:4/117: creat d7/d24/f26 x:0 0 0 2024-04-22T15:49:29.376 INFO:tasks.workunit.client.0.smithi016.stdout:2/118: rename d2 to d2/de/d35 22 2024-04-22T15:49:29.376 INFO:tasks.workunit.client.0.smithi016.stdout:4/118: dread - d7/f1e zero size 2024-04-22T15:49:29.381 INFO:tasks.workunit.client.0.smithi016.stdout:9/179: mkdir d9/de/d23/d30 0 2024-04-22T15:49:29.382 INFO:tasks.workunit.client.0.smithi016.stdout:7/120: symlink d3/d14/l2f 0 2024-04-22T15:49:29.383 INFO:tasks.workunit.client.0.smithi016.stdout:5/105: truncate f2 1639508 0 2024-04-22T15:49:29.384 INFO:tasks.workunit.client.0.smithi016.stdout:8/96: creat d2/db/f16 x:0 0 0 2024-04-22T15:49:29.386 INFO:tasks.workunit.client.0.smithi016.stdout:3/105: getdents d7/d1b/d21 0 2024-04-22T15:49:29.388 INFO:tasks.workunit.client.0.smithi016.stdout:2/119: symlink d2/d9/l36 0 2024-04-22T15:49:29.390 INFO:tasks.workunit.client.0.smithi016.stdout:4/119: dwrite d7/f1d [0,4194304] 0 2024-04-22T15:49:29.390 INFO:tasks.workunit.client.0.smithi016.stdout:0/93: rename dc/d13/d14/d15 to dc/d13/d14/d1d 0 2024-04-22T15:49:29.393 INFO:tasks.workunit.client.0.smithi016.stdout:7/121: dread d3/d4/d9/df/f11 [0,4194304] 0 2024-04-22T15:49:29.402 INFO:tasks.workunit.client.0.smithi016.stdout:6/119: truncate d3/da/d12/f15 796139 0 2024-04-22T15:49:29.403 INFO:tasks.workunit.client.0.smithi016.stdout:8/97: dread f1 [0,4194304] 0 2024-04-22T15:49:29.407 INFO:tasks.workunit.client.0.smithi016.stdout:4/120: dwrite f2 [0,4194304] 0 2024-04-22T15:49:29.407 INFO:tasks.workunit.client.0.smithi016.stdout:9/180: dwrite d9/de/f18 [0,4194304] 0 2024-04-22T15:49:29.409 INFO:tasks.workunit.client.0.smithi016.stdout:0/94: creat dc/f1e x:0 0 0 2024-04-22T15:49:29.409 INFO:tasks.workunit.client.0.smithi016.stdout:6/120: truncate d3/f1d 1020798 0 2024-04-22T15:49:29.412 INFO:tasks.workunit.client.0.smithi016.stdout:7/122: dwrite d3/d4/d9/f2a [0,4194304] 0 2024-04-22T15:49:29.418 INFO:tasks.workunit.client.0.smithi016.stdout:3/106: mknod d7/c25 0 2024-04-22T15:49:29.418 INFO:tasks.workunit.client.0.smithi016.stdout:9/181: truncate d9/de/d26/f2a 982107 0 2024-04-22T15:49:29.419 INFO:tasks.workunit.client.0.smithi016.stdout:2/120: creat d2/d2c/f37 x:0 0 0 2024-04-22T15:49:29.421 INFO:tasks.workunit.client.0.smithi016.stdout:9/182: chown l0 5 1 2024-04-22T15:49:29.423 INFO:tasks.workunit.client.0.smithi016.stdout:8/98: unlink d2/f13 0 2024-04-22T15:49:29.425 INFO:tasks.workunit.client.0.smithi016.stdout:6/121: dread d3/da/dd/f10 [0,4194304] 0 2024-04-22T15:49:29.429 INFO:tasks.workunit.client.0.smithi016.stdout:7/123: dwrite d3/f28 [0,4194304] 0 2024-04-22T15:49:29.431 INFO:tasks.workunit.client.0.smithi016.stdout:1/356: getdents d3/d4/d45 0 2024-04-22T15:49:29.437 INFO:tasks.workunit.client.0.smithi016.stdout:6/122: write d3/da/db/d14/d16/f20 [397291,17706] 0 2024-04-22T15:49:29.439 INFO:tasks.workunit.client.0.smithi016.stdout:4/121: dwrite d7/f1e [0,4194304] 0 2024-04-22T15:49:29.442 INFO:tasks.workunit.client.0.smithi016.stdout:5/106: dread f2 [0,4194304] 0 2024-04-22T15:49:29.442 INFO:tasks.workunit.client.0.smithi016.stdout:9/183: chown d9/l17 0 1 2024-04-22T15:49:29.442 INFO:tasks.workunit.client.0.smithi016.stdout:2/121: symlink d2/de/d2a/l38 0 2024-04-22T15:49:29.443 INFO:tasks.workunit.client.0.smithi016.stdout:8/99: unlink d2/d10/f14 0 2024-04-22T15:49:29.443 INFO:tasks.workunit.client.0.smithi016.stdout:9/184: fdatasync f2 0 2024-04-22T15:49:29.444 INFO:tasks.workunit.client.0.smithi016.stdout:9/185: stat l3 0 2024-04-22T15:49:29.448 INFO:tasks.workunit.client.0.smithi016.stdout:7/124: dwrite d3/d4/f7 [0,4194304] 0 2024-04-22T15:49:29.452 INFO:tasks.workunit.client.0.smithi016.stdout:2/122: dread d2/f8 [4194304,4194304] 0 2024-04-22T15:49:29.453 INFO:tasks.workunit.client.0.smithi016.stdout:3/107: mkdir d7/d17/d26 0 2024-04-22T15:49:29.454 INFO:tasks.workunit.client.0.smithi016.stdout:0/95: write fb [5420104,126576] 0 2024-04-22T15:49:29.456 INFO:tasks.workunit.client.0.smithi016.stdout:6/123: mkdir d3/da/d21 0 2024-04-22T15:49:29.457 INFO:tasks.workunit.client.0.smithi016.stdout:6/124: dread - d3/da/db/f17 zero size 2024-04-22T15:49:29.459 INFO:tasks.workunit.client.0.smithi016.stdout:7/125: dwrite d3/d4/f1f [0,4194304] 0 2024-04-22T15:49:29.468 INFO:tasks.workunit.client.0.smithi016.stdout:9/186: creat d9/de/f31 x:0 0 0 2024-04-22T15:49:29.472 INFO:tasks.workunit.client.0.smithi016.stdout:0/96: readlink dc/l10 0 2024-04-22T15:49:29.473 INFO:tasks.workunit.client.0.smithi016.stdout:5/107: dwrite f6 [0,4194304] 0 2024-04-22T15:49:29.475 INFO:tasks.workunit.client.0.smithi016.stdout:7/126: dwrite d3/d4/d9/f26 [0,4194304] 0 2024-04-22T15:49:29.476 INFO:tasks.workunit.client.0.smithi016.stdout:3/108: mkdir d7/d17/d18/d27 0 2024-04-22T15:49:29.479 INFO:tasks.workunit.client.0.smithi016.stdout:1/357: dread d3/d4/d19/f64 [0,4194304] 0 2024-04-22T15:49:29.481 INFO:tasks.workunit.client.0.smithi016.stdout:0/97: dread fb [0,4194304] 0 2024-04-22T15:49:29.482 INFO:tasks.workunit.client.0.smithi016.stdout:4/122: creat d7/d22/f27 x:0 0 0 2024-04-22T15:49:29.485 INFO:tasks.workunit.client.0.smithi016.stdout:1/358: dread d3/d4/d19/d20/f44 [0,4194304] 0 2024-04-22T15:49:29.485 INFO:tasks.workunit.client.0.smithi016.stdout:9/187: creat d9/de/d23/d24/d25/f32 x:0 0 0 2024-04-22T15:49:29.490 INFO:tasks.workunit.client.0.smithi016.stdout:2/123: truncate d2/f8 8371175 0 2024-04-22T15:49:29.493 INFO:tasks.workunit.client.0.smithi016.stdout:5/108: rename fd to df/d15/f16 0 2024-04-22T15:49:29.498 INFO:tasks.workunit.client.0.smithi016.stdout:3/109: dread d7/f1e [0,4194304] 0 2024-04-22T15:49:29.503 INFO:tasks.workunit.client.0.smithi016.stdout:0/98: read dc/d13/d14/f18 [846222,128079] 0 2024-04-22T15:49:29.506 INFO:tasks.workunit.client.0.smithi016.stdout:4/123: fdatasync d7/f14 0 2024-04-22T15:49:29.512 INFO:tasks.workunit.client.0.smithi016.stdout:5/109: symlink df/d15/l17 0 2024-04-22T15:49:29.514 INFO:tasks.workunit.client.0.smithi016.stdout:1/359: rename d3/d4/d15/d4c to d3/d4/d3a/d6b/d77 0 2024-04-22T15:49:29.515 INFO:tasks.workunit.client.0.smithi016.stdout:1/360: fdatasync d3/f7 0 2024-04-22T15:49:29.515 INFO:tasks.workunit.client.0.smithi016.stdout:5/110: dread df/d15/f16 [0,4194304] 0 2024-04-22T15:49:29.516 INFO:tasks.workunit.client.0.smithi016.stdout:5/111: readlink l7 0 2024-04-22T15:49:29.518 INFO:tasks.workunit.client.0.smithi016.stdout:7/127: symlink d3/d4/d9/l30 0 2024-04-22T15:49:29.527 INFO:tasks.workunit.client.0.smithi016.stdout:8/100: getdents d2/d10 0 2024-04-22T15:49:29.528 INFO:tasks.workunit.client.0.smithi016.stdout:8/101: write d2/db/f16 [738827,31719] 0 2024-04-22T15:49:29.529 INFO:tasks.workunit.client.0.smithi016.stdout:0/99: mkdir dc/d13/d14/d1d/d1f 0 2024-04-22T15:49:29.532 INFO:tasks.workunit.client.0.smithi016.stdout:8/102: dread f1 [0,4194304] 0 2024-04-22T15:49:29.540 INFO:tasks.workunit.client.0.smithi016.stdout:6/125: dwrite d3/da/d12/f15 [0,4194304] 0 2024-04-22T15:49:29.540 INFO:tasks.workunit.client.0.smithi016.stdout:6/126: truncate d3/f1d 1261895 0 2024-04-22T15:49:29.540 INFO:tasks.workunit.client.0.smithi016.stdout:8/103: dread f1 [0,4194304] 0 2024-04-22T15:49:29.540 INFO:tasks.workunit.client.0.smithi016.stdout:6/127: fdatasync d3/da/dd/f18 0 2024-04-22T15:49:29.540 INFO:tasks.workunit.client.0.smithi016.stdout:8/104: truncate f1 4659801 0 2024-04-22T15:49:29.540 INFO:tasks.workunit.client.0.smithi016.stdout:8/105: chown d2/db/f16 827737 1 2024-04-22T15:49:29.540 INFO:tasks.workunit.client.0.smithi016.stdout:1/361: dread d3/d4/d19/d20/d2b/f59 [0,4194304] 0 2024-04-22T15:49:29.541 INFO:tasks.workunit.client.0.smithi016.stdout:5/112: symlink df/l18 0 2024-04-22T15:49:29.541 INFO:tasks.workunit.client.0.smithi016.stdout:1/362: dread - d3/d2e/f6d zero size 2024-04-22T15:49:29.542 INFO:tasks.workunit.client.0.smithi016.stdout:5/113: write df/f14 [178445,14855] 0 2024-04-22T15:49:29.543 INFO:tasks.workunit.client.0.smithi016.stdout:5/114: chown df/f14 40974 1 2024-04-22T15:49:29.544 INFO:tasks.workunit.client.0.smithi016.stdout:7/128: mkdir d3/dc/d1b/d31 0 2024-04-22T15:49:29.549 INFO:tasks.workunit.client.0.smithi016.stdout:0/100: mknod dc/d13/d14/c20 0 2024-04-22T15:49:29.550 INFO:tasks.workunit.client.0.smithi016.stdout:0/101: dread - dc/d13/d14/d1d/f1c zero size 2024-04-22T15:49:29.551 INFO:tasks.workunit.client.0.smithi016.stdout:4/124: getdents d7/d21/d23 0 2024-04-22T15:49:29.554 INFO:tasks.workunit.client.0.smithi016.stdout:6/128: unlink d3/c5 0 2024-04-22T15:49:29.557 INFO:tasks.workunit.client.0.smithi016.stdout:1/363: rename d3/l36 to d3/d4/d3a/l78 0 2024-04-22T15:49:29.558 INFO:tasks.workunit.client.0.smithi016.stdout:5/115: mknod df/d15/c19 0 2024-04-22T15:49:29.560 INFO:tasks.workunit.client.0.smithi016.stdout:3/110: creat d7/d1b/d21/f28 x:0 0 0 2024-04-22T15:49:29.562 INFO:tasks.workunit.client.0.smithi016.stdout:0/102: unlink dc/d13/d14/d1d/f1c 0 2024-04-22T15:49:29.562 INFO:tasks.workunit.client.0.smithi016.stdout:0/103: truncate dc/d13/f17 1023585 0 2024-04-22T15:49:29.565 INFO:tasks.workunit.client.0.smithi016.stdout:8/106: truncate d2/f8 6300240 0 2024-04-22T15:49:29.568 INFO:tasks.workunit.client.0.smithi016.stdout:4/125: rename d7/cb to d7/d22/c28 0 2024-04-22T15:49:29.568 INFO:tasks.workunit.client.0.smithi016.stdout:4/126: write f2 [5043762,65045] 0 2024-04-22T15:49:29.570 INFO:tasks.workunit.client.0.smithi016.stdout:7/129: mkdir d3/d32 0 2024-04-22T15:49:29.572 INFO:tasks.workunit.client.0.smithi016.stdout:3/111: symlink d7/d17/l29 0 2024-04-22T15:49:29.574 INFO:tasks.workunit.client.0.smithi016.stdout:6/129: dread d3/da/db/d14/d16/f20 [0,4194304] 0 2024-04-22T15:49:29.577 INFO:tasks.workunit.client.0.smithi016.stdout:1/364: unlink d3/d2e/f73 0 2024-04-22T15:49:29.577 INFO:tasks.workunit.client.0.smithi016.stdout:5/116: dwrite f2 [0,4194304] 0 2024-04-22T15:49:29.578 INFO:tasks.workunit.client.0.smithi016.stdout:1/365: truncate d3/f4f 1648271 0 2024-04-22T15:49:29.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.579+0000 7f13808fa700 1 -- 172.21.15.16:0/24961081 --> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f137c068c00 con 0x7f136406d880 2024-04-22T15:49:29.583 INFO:tasks.workunit.client.0.smithi016.stdout:3/112: unlink d7/f1e 0 2024-04-22T15:49:29.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.581+0000 7f136b7fe700 1 -- 172.21.15.16:0/24961081 <== mgr.24395 v2:172.21.15.27:6828/2676414545 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f137c068c00 con 0x7f136406d880 2024-04-22T15:49:29.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.584+0000 7f13808fa700 1 -- 172.21.15.16:0/24961081 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f136406d880 msgr2=0x7f136406fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:29.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.584+0000 7f13808fa700 1 --2- 172.21.15.16:0/24961081 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f136406d880 0x7f136406fd40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1370006fd0 tx=0x7f1370009190 comp rx=0 tx=0).stop 2024-04-22T15:49:29.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.584+0000 7f13808fa700 1 -- 172.21.15.16:0/24961081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f137c101750 msgr2=0x7f137c19cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:29.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.584+0000 7f13808fa700 1 --2- 172.21.15.16:0/24961081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f137c101750 0x7f137c19cc60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f136c00beb0 tx=0x7f136c00bee0 comp rx=0 tx=0).stop 2024-04-22T15:49:29.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.584+0000 7f13808fa700 1 -- 172.21.15.16:0/24961081 shutdown_connections 2024-04-22T15:49:29.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.584+0000 7f13808fa700 1 --2- 172.21.15.16:0/24961081 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f136406d880 0x7f136406fd40 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:29.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.584+0000 7f13808fa700 1 --2- 172.21.15.16:0/24961081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f137c101750 0x7f137c19cc60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:29.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.584+0000 7f13808fa700 1 --2- 172.21.15.16:0/24961081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f137c104080 0x7f137c19d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:29.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.585+0000 7f13808fa700 1 -- 172.21.15.16:0/24961081 >> 172.21.15.16:0/24961081 conn(0x7f137c0fb160 msgr2=0x7f137c0fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:29.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.585+0000 7f13808fa700 1 -- 172.21.15.16:0/24961081 shutdown_connections 2024-04-22T15:49:29.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.585+0000 7f13808fa700 1 -- 172.21.15.16:0/24961081 wait complete. 2024-04-22T15:49:29.589 INFO:tasks.workunit.client.0.smithi016.stdout:6/130: mknod d3/da/dd/c22 0 2024-04-22T15:49:29.591 INFO:tasks.workunit.client.0.smithi016.stdout:4/127: creat d7/d21/d23/f29 x:0 0 0 2024-04-22T15:49:29.594 INFO:tasks.workunit.client.0.smithi016.stdout:7/130: mkdir d3/dc/d33 0 2024-04-22T15:49:29.597 INFO:tasks.workunit.client.0.smithi016.stdout:4/128: dwrite f2 [0,4194304] 0 2024-04-22T15:49:29.598 INFO:tasks.workunit.client.0.smithi016.stdout:6/131: creat d3/da/dd/f23 x:0 0 0 2024-04-22T15:49:29.600 INFO:tasks.workunit.client.0.smithi016.stdout:6/132: dread d3/f1d [0,4194304] 0 2024-04-22T15:49:29.601 INFO:tasks.workunit.client.0.smithi016.stdout:6/133: chown d3/da/db/d14 32464 1 2024-04-22T15:49:29.603 INFO:tasks.workunit.client.0.smithi016.stdout:6/134: write d3/da/dd/f23 [13445,80076] 0 2024-04-22T15:49:29.603 INFO:tasks.workunit.client.0.smithi016.stdout:1/366: creat d3/d4/d15/d38/d63/d67/f79 x:0 0 0 2024-04-22T15:49:29.604 INFO:tasks.workunit.client.0.smithi016.stdout:6/135: write d3/f1d [2123248,6398] 0 2024-04-22T15:49:29.605 INFO:tasks.workunit.client.0.smithi016.stdout:1/367: write d3/d4/d19/d20/f33 [1796482,20676] 0 2024-04-22T15:49:29.608 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:49:29.608 INFO:tasks.workunit.client.0.smithi016.stdout:3/113: unlink d7/cb 0 2024-04-22T15:49:29.609 INFO:tasks.workunit.client.0.smithi016.stdout:6/136: read d3/f1d [815930,107610] 0 2024-04-22T15:49:29.610 INFO:tasks.workunit.client.0.smithi016.stdout:8/107: rename d2/c5 to d2/c17 0 2024-04-22T15:49:29.618 INFO:tasks.workunit.client.0.smithi016.stdout:1/368: symlink d3/d4/d3a/d6b/l7a 0 2024-04-22T15:49:29.620 INFO:tasks.workunit.client.0.smithi016.stdout:3/114: fdatasync d7/fc 0 2024-04-22T15:49:29.626 INFO:tasks.workunit.client.0.smithi016.stdout:7/131: link d3/d4/f1f d3/f34 0 2024-04-22T15:49:29.629 INFO:tasks.workunit.client.0.smithi016.stdout:1/369: dread d3/d4/d19/d20/f33 [0,4194304] 0 2024-04-22T15:49:29.631 INFO:tasks.workunit.client.0.smithi016.stdout:3/115: creat d7/d17/d18/f2a x:0 0 0 2024-04-22T15:49:29.633 INFO:tasks.workunit.client.0.smithi016.stdout:8/108: getdents d2/d10 0 2024-04-22T15:49:29.634 INFO:tasks.workunit.client.0.smithi016.stdout:0/104: dwrite fb [4194304,4194304] 0 2024-04-22T15:49:29.637 INFO:tasks.workunit.client.0.smithi016.stdout:7/132: rename d3/d4/d9/l30 to d3/dc/d1b/l35 0 2024-04-22T15:49:29.638 INFO:tasks.workunit.client.0.smithi016.stdout:7/133: dread - d3/dc/f16 zero size 2024-04-22T15:49:29.638 INFO:tasks.workunit.client.0.smithi016.stdout:0/105: dread fb [0,4194304] 0 2024-04-22T15:49:29.640 INFO:tasks.workunit.client.0.smithi016.stdout:7/134: write d3/f28 [3817317,130090] 0 2024-04-22T15:49:29.641 INFO:tasks.workunit.client.0.smithi016.stdout:1/370: rmdir d3/d4/d37 39 2024-04-22T15:49:29.646 INFO:tasks.workunit.client.0.smithi016.stdout:6/137: creat d3/f24 x:0 0 0 2024-04-22T15:49:29.647 INFO:tasks.workunit.client.0.smithi016.stdout:8/109: creat d2/db/f18 x:0 0 0 2024-04-22T15:49:29.648 INFO:tasks.workunit.client.0.smithi016.stdout:8/110: truncate d2/db/f18 96393 0 2024-04-22T15:49:29.648 INFO:tasks.workunit.client.0.smithi016.stdout:8/111: readlink d2/ld 0 2024-04-22T15:49:29.649 INFO:tasks.workunit.client.0.smithi016.stdout:8/112: chown d2/db/c11 2318957 1 2024-04-22T15:49:29.650 INFO:tasks.workunit.client.0.smithi016.stdout:8/113: stat d2/d10/c15 0 2024-04-22T15:49:29.653 INFO:tasks.workunit.client.0.smithi016.stdout:0/106: symlink dc/d13/d14/l21 0 2024-04-22T15:49:29.663 INFO:tasks.workunit.client.0.smithi016.stdout:1/371: dwrite d3/d4/d19/f1d [0,4194304] 0 2024-04-22T15:49:29.664 INFO:tasks.workunit.client.0.smithi016.stdout:6/138: rename d3/da/f1a to d3/da/dd/f25 0 2024-04-22T15:49:29.668 INFO:tasks.workunit.client.0.smithi016.stdout:8/114: symlink d2/d10/l19 0 2024-04-22T15:49:29.668 INFO:tasks.workunit.client.0.smithi016.stdout:8/115: fdatasync d2/db/f18 0 2024-04-22T15:49:29.670 INFO:tasks.workunit.client.0.smithi016.stdout:8/116: write d2/db/f16 [1737629,58675] 0 2024-04-22T15:49:29.680 INFO:tasks.workunit.client.0.smithi016.stdout:0/107: getdents dc/d13/d14/d1d/d1f 0 2024-04-22T15:49:29.686 INFO:tasks.workunit.client.0.smithi016.stdout:7/135: creat d3/d4/f36 x:0 0 0 2024-04-22T15:49:29.687 INFO:tasks.workunit.client.0.smithi016.stdout:0/108: dwrite dc/f1e [0,4194304] 0 2024-04-22T15:49:29.692 INFO:tasks.workunit.client.0.smithi016.stdout:3/116: getdents d7 0 2024-04-22T15:49:29.693 INFO:tasks.workunit.client.0.smithi016.stdout:3/117: readlink d7/d1b/l1d 0 2024-04-22T15:49:29.694 INFO:tasks.workunit.client.0.smithi016.stdout:6/139: mknod d3/c26 0 2024-04-22T15:49:29.695 INFO:tasks.workunit.client.0.smithi016.stdout:6/140: dread - d3/f24 zero size 2024-04-22T15:49:29.695 INFO:tasks.workunit.client.0.smithi016.stdout:7/136: dwrite d3/fa [4194304,4194304] 0 2024-04-22T15:49:29.696 INFO:tasks.workunit.client.0.smithi016.stdout:3/118: write d7/d1b/d21/f28 [555391,38996] 0 2024-04-22T15:49:29.702 INFO:tasks.workunit.client.0.smithi016.stdout:8/117: fdatasync d2/f8 0 2024-04-22T15:49:29.704 INFO:tasks.workunit.client.0.smithi016.stdout:7/137: dwrite d3/d4/f7 [4194304,4194304] 0 2024-04-22T15:49:29.715 INFO:tasks.workunit.client.0.smithi016.stdout:3/119: symlink d7/d1b/d21/l2b 0 2024-04-22T15:49:29.716 INFO:tasks.workunit.client.0.smithi016.stdout:3/120: chown d7/fc 146241347 1 2024-04-22T15:49:29.718 INFO:tasks.workunit.client.0.smithi016.stdout:8/118: symlink d2/d10/l1a 0 2024-04-22T15:49:29.722 INFO:tasks.workunit.client.0.smithi016.stdout:8/119: dread d2/f8 [0,4194304] 0 2024-04-22T15:49:29.724 INFO:tasks.workunit.client.0.smithi016.stdout:1/372: getdents d3/d4/d15 0 2024-04-22T15:49:29.725 INFO:tasks.workunit.client.0.smithi016.stdout:8/120: dread d2/f8 [0,4194304] 0 2024-04-22T15:49:29.727 INFO:tasks.workunit.client.0.smithi016.stdout:8/121: chown d2/db/fc 352 1 2024-04-22T15:49:29.727 INFO:tasks.workunit.client.0.smithi016.stdout:8/122: chown f1 1864239260 1 2024-04-22T15:49:29.727 INFO:tasks.workunit.client.0.smithi016.stdout:8/123: fdatasync f1 0 2024-04-22T15:49:29.728 INFO:tasks.workunit.client.0.smithi016.stdout:7/138: mknod d3/dc/d1b/d31/c37 0 2024-04-22T15:49:29.728 INFO:tasks.workunit.client.0.smithi016.stdout:7/139: chown d3/d4/f17 30 1 2024-04-22T15:49:29.733 INFO:tasks.workunit.client.0.smithi016.stdout:3/121: link d7/f14 d7/f2c 0 2024-04-22T15:49:29.742 INFO:tasks.workunit.client.0.smithi016.stdout:6/141: getdents d3/da/db 0 2024-04-22T15:49:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.738+0000 7f240b978700 1 -- 172.21.15.16:0/3028754571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2404100ca0 msgr2=0x7f24041010c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.738+0000 7f240b978700 1 --2- 172.21.15.16:0/3028754571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2404100ca0 0x7f24041010c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f2400009910 tx=0x7f2400009c20 comp rx=0 tx=0).stop 2024-04-22T15:49:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.738+0000 7f240b978700 1 -- 172.21.15.16:0/3028754571 shutdown_connections 2024-04-22T15:49:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.738+0000 7f240b978700 1 --2- 172.21.15.16:0/3028754571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2404101ea0 0x7f2404102300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.738+0000 7f240b978700 1 --2- 172.21.15.16:0/3028754571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2404100ca0 0x7f24041010c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.738+0000 7f240b978700 1 -- 172.21.15.16:0/3028754571 >> 172.21.15.16:0/3028754571 conn(0x7f24040fc260 msgr2=0x7f24040fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.738+0000 7f240b978700 1 -- 172.21.15.16:0/3028754571 shutdown_connections 2024-04-22T15:49:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.738+0000 7f240b978700 1 -- 172.21.15.16:0/3028754571 wait complete. 2024-04-22T15:49:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.739+0000 7f240b978700 1 Processor -- start 2024-04-22T15:49:29.744 INFO:tasks.workunit.client.0.smithi016.stdout:1/373: dread d3/d4/f1e [4194304,4194304] 0 2024-04-22T15:49:29.744 INFO:tasks.workunit.client.0.smithi016.stdout:7/140: dread d3/fa [0,4194304] 0 2024-04-22T15:49:29.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.739+0000 7f240b978700 1 -- start start 2024-04-22T15:49:29.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.739+0000 7f240b978700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2404100ca0 0x7f240419cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:29.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.739+0000 7f240b978700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2404101ea0 0x7f240419d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:29.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.739+0000 7f240b978700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240419d620 con 0x7f2404100ca0 2024-04-22T15:49:29.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.739+0000 7f240b978700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240419d760 con 0x7f2404101ea0 2024-04-22T15:49:29.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.740+0000 7f2408f13700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2404101ea0 0x7f240419d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:29.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.740+0000 7f2408f13700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2404101ea0 0x7f240419d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:44786/0 (socket says 172.21.15.16:44786) 2024-04-22T15:49:29.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.740+0000 7f2408f13700 1 -- 172.21.15.16:0/388474185 learned_addr learned my addr 172.21.15.16:0/388474185 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:49:29.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.740+0000 7f2409714700 1 --2- 172.21.15.16:0/388474185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2404100ca0 0x7f240419cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:29.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.740+0000 7f2408f13700 1 -- 172.21.15.16:0/388474185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2404100ca0 msgr2=0x7f240419cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:29.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.740+0000 7f2408f13700 1 --2- 172.21.15.16:0/388474185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2404100ca0 0x7f240419cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:29.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.740+0000 7f2408f13700 1 -- 172.21.15.16:0/388474185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24000095f0 con 0x7f2404101ea0 2024-04-22T15:49:29.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.740+0000 7f2409714700 1 --2- 172.21.15.16:0/388474185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2404100ca0 0x7f240419cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:49:29.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.740+0000 7f2408f13700 1 --2- 172.21.15.16:0/388474185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2404101ea0 0x7f240419d000 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f23f400e920 tx=0x7f23f400ec30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:29.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.741+0000 7f23fa7fc700 1 -- 172.21.15.16:0/388474185 <== mon.1 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23f400c9d0 con 0x7f2404101ea0 2024-04-22T15:49:29.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.741+0000 7f240b978700 1 -- 172.21.15.16:0/388474185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24041a2210 con 0x7f2404101ea0 2024-04-22T15:49:29.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.741+0000 7f23fa7fc700 1 -- 172.21.15.16:0/388474185 <== mon.1 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f23f4010430 con 0x7f2404101ea0 2024-04-22T15:49:29.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.741+0000 7f23fa7fc700 1 -- 172.21.15.16:0/388474185 <== mon.1 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23f4018650 con 0x7f2404101ea0 2024-04-22T15:49:29.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.741+0000 7f240b978700 1 -- 172.21.15.16:0/388474185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24041a2760 con 0x7f2404101ea0 2024-04-22T15:49:29.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.742+0000 7f240b978700 1 -- 172.21.15.16:0/388474185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2404105f80 con 0x7f2404101ea0 2024-04-22T15:49:29.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.742+0000 7f23fa7fc700 1 -- 172.21.15.16:0/388474185 <== mon.1 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) v1 ==== 91557+0+0 (secure 0 0 0) 0x7f23f400cb30 con 0x7f2404101ea0 2024-04-22T15:49:29.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.743+0000 7f23fa7fc700 1 --2- 172.21.15.16:0/388474185 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f23f006d850 0x7f23f006fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:29.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.743+0000 7f23fa7fc700 1 -- 172.21.15.16:0/388474185 <== mon.1 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f23f4014070 con 0x7f2404101ea0 2024-04-22T15:49:29.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.743+0000 7f2409714700 1 --2- 172.21.15.16:0/388474185 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f23f006d850 0x7f23f006fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:29.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.744+0000 7f2409714700 1 --2- 172.21.15.16:0/388474185 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f23f006d850 0x7f23f006fd10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f240000b6f0 tx=0x7f2400005940 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:29.748 INFO:tasks.workunit.client.0.smithi016.stdout:7/141: creat d3/dc/d22/f38 x:0 0 0 2024-04-22T15:49:29.748 INFO:tasks.workunit.client.0.smithi016.stdout:0/109: truncate f4 2277604 0 2024-04-22T15:49:29.751 INFO:tasks.workunit.client.0.smithi016.stdout:6/142: creat d3/da/d12/d13/f27 x:0 0 0 2024-04-22T15:49:29.751 INFO:tasks.workunit.client.0.smithi016.stdout:0/110: dread dc/f1e [0,4194304] 0 2024-04-22T15:49:29.752 INFO:tasks.workunit.client.0.smithi016.stdout:0/111: read fb [7930767,36397] 0 2024-04-22T15:49:29.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:29.751+0000 7f23fa7fc700 1 -- 172.21.15.16:0/388474185 <== mon.1 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f23f4053e20 con 0x7f2404101ea0 2024-04-22T15:49:29.755 INFO:tasks.workunit.client.0.smithi016.stdout:7/142: fsync d3/d4/d9/fb 0 2024-04-22T15:49:29.759 INFO:tasks.workunit.client.0.smithi016.stdout:6/143: mknod d3/da/dd/c28 0 2024-04-22T15:49:29.762 INFO:tasks.workunit.client.0.smithi016.stdout:1/374: dread d3/d4/d19/f35 [0,4194304] 0 2024-04-22T15:49:29.764 INFO:tasks.workunit.client.0.smithi016.stdout:1/375: fsync d3/d4/d19/d20/f48 0 2024-04-22T15:49:29.767 INFO:tasks.workunit.client.0.smithi016.stdout:8/124: dwrite d2/db/fc [0,4194304] 0 2024-04-22T15:49:29.771 INFO:tasks.workunit.client.0.smithi016.stdout:7/143: unlink d3/d4/c1a 0 2024-04-22T15:49:29.772 INFO:tasks.workunit.client.0.smithi016.stdout:8/125: dread d2/db/fc [0,4194304] 0 2024-04-22T15:49:29.772 INFO:tasks.workunit.client.0.smithi016.stdout:7/144: chown d3/d4/d9/df 7806260 1 2024-04-22T15:49:29.777 INFO:tasks.workunit.client.0.smithi016.stdout:1/376: mknod d3/d28/c7b 0 2024-04-22T15:49:29.778 INFO:tasks.workunit.client.0.smithi016.stdout:6/144: dread d3/f6 [0,4194304] 0 2024-04-22T15:49:29.778 INFO:tasks.workunit.client.0.smithi016.stdout:1/377: truncate d3/d2e/f51 1603310 0 2024-04-22T15:49:29.789 INFO:tasks.workunit.client.0.smithi016.stdout:7/145: dwrite d3/d4/d9/f13 [4194304,4194304] 0 2024-04-22T15:49:29.791 INFO:tasks.workunit.client.0.smithi016.stdout:3/122: mknod d7/d17/d26/c2d 0 2024-04-22T15:49:29.792 INFO:tasks.workunit.client.0.smithi016.stdout:0/112: write f4 [3000400,53729] 0 2024-04-22T15:49:29.795 INFO:tasks.workunit.client.0.smithi016.stdout:6/145: rename d3/da/dd/f25 to d3/da/dd/f29 0 2024-04-22T15:49:29.797 INFO:tasks.workunit.client.0.smithi016.stdout:1/378: dwrite d3/d4/fe [0,4194304] 0 2024-04-22T15:49:29.798 INFO:tasks.workunit.client.0.smithi016.stdout:6/146: write d3/da/d12/ff [118299,27229] 0 2024-04-22T15:49:29.799 INFO:tasks.workunit.client.0.smithi016.stdout:7/146: dread d3/d4/d9/f25 [0,4194304] 0 2024-04-22T15:49:29.800 INFO:tasks.workunit.client.0.smithi016.stdout:3/123: creat d7/d17/f2e x:0 0 0 2024-04-22T15:49:29.802 INFO:tasks.workunit.client.0.smithi016.stdout:0/113: rmdir dc 39 2024-04-22T15:49:29.802 INFO:tasks.workunit.client.0.smithi016.stdout:6/147: chown d3/da/db/f17 134837 1 2024-04-22T15:49:29.804 INFO:tasks.workunit.client.0.smithi016.stdout:6/148: write d3/da/d12/d13/f27 [1008827,229] 0 2024-04-22T15:49:29.810 INFO:tasks.workunit.client.0.smithi016.stdout:7/147: rename d3/d4/f6 to d3/dc/d22/f39 0 2024-04-22T15:49:29.810 INFO:tasks.workunit.client.0.smithi016.stdout:7/148: fsync d3/dc/f1e 0 2024-04-22T15:49:29.812 INFO:tasks.workunit.client.0.smithi016.stdout:3/124: write d7/fc [616229,102736] 0 2024-04-22T15:49:29.813 INFO:tasks.workunit.client.0.smithi016.stdout:0/114: chown dc/d13/d14/d1d 107635291 1 2024-04-22T15:49:29.814 INFO:tasks.workunit.client.0.smithi016.stdout:8/126: getdents d2/db 0 2024-04-22T15:49:29.815 INFO:tasks.workunit.client.0.smithi016.stdout:6/149: mknod d3/da/d12/d13/c2a 0 2024-04-22T15:49:29.817 INFO:tasks.workunit.client.0.smithi016.stdout:6/150: write d3/da/d12/d13/f27 [1013784,27414] 0 2024-04-22T15:49:29.818 INFO:tasks.workunit.client.0.smithi016.stdout:1/379: truncate d3/f23 1811823 0 2024-04-22T15:49:29.820 INFO:tasks.workunit.client.0.smithi016.stdout:8/127: dread d2/db/f16 [0,4194304] 0 2024-04-22T15:49:29.824 INFO:tasks.workunit.client.0.smithi016.stdout:0/115: mkdir dc/d13/d22 0 2024-04-22T15:49:29.825 INFO:tasks.workunit.client.0.smithi016.stdout:0/116: stat dc/d13/d22 0 2024-04-22T15:49:29.825 INFO:tasks.workunit.client.0.smithi016.stdout:0/117: chown f4 30786755 1 2024-04-22T15:49:29.827 INFO:tasks.workunit.client.0.smithi016.stdout:3/125: dwrite d7/f15 [0,4194304] 0 2024-04-22T15:49:29.828 INFO:tasks.workunit.client.0.smithi016.stdout:3/126: readlink d7/d17/l24 0 2024-04-22T15:49:29.828 INFO:tasks.workunit.client.0.smithi016.stdout:6/151: read d3/da/dd/f10 [933765,48689] 0 2024-04-22T15:49:29.829 INFO:tasks.workunit.client.0.smithi016.stdout:1/380: mknod d3/d4/d15/d38/d63/d67/c7c 0 2024-04-22T15:49:29.830 INFO:tasks.workunit.client.0.smithi016.stdout:6/152: write d3/da/d12/ff [1032383,13149] 0 2024-04-22T15:49:29.833 INFO:tasks.workunit.client.0.smithi016.stdout:6/153: dread d3/f6 [0,4194304] 0 2024-04-22T15:49:29.834 INFO:tasks.workunit.client.0.smithi016.stdout:0/118: mknod dc/d13/d14/d1d/c23 0 2024-04-22T15:49:29.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:29 smithi027 ceph-mon[34922]: Upgrade: Updating mgr.smithi016.rnsnnr 2024-04-22T15:49:29.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:29 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:29.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:29 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi016.rnsnnr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:49:29.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:29 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi016.rnsnnr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:49:29.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:29 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:49:29.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:29 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:49:29.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:29 smithi027 ceph-mon[34922]: Deploying daemon mgr.smithi016.rnsnnr on smithi016 2024-04-22T15:49:29.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:29 smithi027 ceph-mon[34922]: pgmap v27: 65 pgs: 65 active+clean; 349 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.4 MiB/s rd, 25 MiB/s wr, 341 op/s 2024-04-22T15:49:29.840 INFO:tasks.workunit.client.0.smithi016.stdout:1/381: symlink d3/d4/d15/l7d 0 2024-04-22T15:49:29.840 INFO:tasks.workunit.client.0.smithi016.stdout:9/188: sync 2024-04-22T15:49:29.842 INFO:tasks.workunit.client.0.smithi016.stdout:3/127: dwrite f1 [0,4194304] 0 2024-04-22T15:49:29.843 INFO:tasks.workunit.client.0.smithi016.stdout:3/128: fdatasync d7/fc 0 2024-04-22T15:49:29.847 INFO:tasks.workunit.client.0.smithi016.stdout:0/119: symlink dc/d13/l24 0 2024-04-22T15:49:29.850 INFO:tasks.workunit.client.0.smithi016.stdout:1/382: mkdir d3/d4/da/d7e 0 2024-04-22T15:49:29.852 INFO:tasks.workunit.client.0.smithi016.stdout:7/149: link d3/c8 d3/d4/d9/c3a 0 2024-04-22T15:49:29.854 INFO:tasks.workunit.client.0.smithi016.stdout:9/189: symlink d9/de/d26/l33 0 2024-04-22T15:49:29.854 INFO:tasks.workunit.client.0.smithi016.stdout:9/190: dread - d9/de/d23/d24/f2f zero size 2024-04-22T15:49:29.860 INFO:tasks.workunit.client.0.smithi016.stdout:9/191: dwrite d9/de/f11 [0,4194304] 0 2024-04-22T15:49:29.861 INFO:tasks.workunit.client.0.smithi016.stdout:9/192: chown d9/de/f1a 0 1 2024-04-22T15:49:29.864 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:29 smithi016 ceph-mon[26620]: Upgrade: Updating mgr.smithi016.rnsnnr 2024-04-22T15:49:29.864 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:29 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:29.864 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:29 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi016.rnsnnr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:49:29.865 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:29 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi016.rnsnnr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:49:29.865 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:29 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:49:29.865 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:29 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:49:29.865 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:29 smithi016 ceph-mon[26620]: Deploying daemon mgr.smithi016.rnsnnr on smithi016 2024-04-22T15:49:29.865 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:29 smithi016 ceph-mon[26620]: pgmap v27: 65 pgs: 65 active+clean; 349 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.4 MiB/s rd, 25 MiB/s wr, 341 op/s 2024-04-22T15:49:29.866 INFO:tasks.workunit.client.0.smithi016.stdout:0/120: mknod dc/d13/d14/c25 0 2024-04-22T15:49:29.868 INFO:tasks.workunit.client.0.smithi016.stdout:6/154: write d3/f1d [1471740,77579] 0 2024-04-22T15:49:29.868 INFO:tasks.workunit.client.0.smithi016.stdout:0/121: write dc/d13/f16 [5154941,56459] 0 2024-04-22T15:49:29.868 INFO:tasks.workunit.client.0.smithi016.stdout:1/383: rename d3/d4/d3a/d6b/l7a to d3/d4/d3a/d6b/d77/l7f 0 2024-04-22T15:49:29.869 INFO:tasks.workunit.client.0.smithi016.stdout:7/150: mknod d3/dc/d22/c3b 0 2024-04-22T15:49:29.870 INFO:tasks.workunit.client.0.smithi016.stdout:0/122: stat fb 0 2024-04-22T15:49:29.872 INFO:tasks.workunit.client.0.smithi016.stdout:8/128: getdents d2/d10 0 2024-04-22T15:49:29.873 INFO:tasks.workunit.client.0.smithi016.stdout:9/193: dwrite d9/de/d23/d24/d25/f32 [0,4194304] 0 2024-04-22T15:49:29.877 INFO:tasks.workunit.client.0.smithi016.stdout:3/129: creat d7/f2f x:0 0 0 2024-04-22T15:49:29.879 INFO:tasks.workunit.client.0.smithi016.stdout:0/123: mkdir dc/d13/d14/d26 0 2024-04-22T15:49:29.880 INFO:tasks.workunit.client.0.smithi016.stdout:8/129: mknod d2/db/c1b 0 2024-04-22T15:49:29.882 INFO:tasks.workunit.client.0.smithi016.stdout:9/194: rename d9/f22 to d9/de/d23/f34 0 2024-04-22T15:49:29.883 INFO:tasks.workunit.client.0.smithi016.stdout:9/195: chown f2 13 1 2024-04-22T15:49:29.884 INFO:tasks.workunit.client.0.smithi016.stdout:8/130: dread d2/db/fc [0,4194304] 0 2024-04-22T15:49:29.885 INFO:tasks.workunit.client.0.smithi016.stdout:7/151: getdents d3/d32 0 2024-04-22T15:49:29.892 INFO:tasks.workunit.client.0.smithi016.stdout:9/196: creat d9/de/d23/f35 x:0 0 0 2024-04-22T15:49:29.892 INFO:tasks.workunit.client.0.smithi016.stdout:7/152: dwrite d3/f23 [0,4194304] 0 2024-04-22T15:49:29.894 INFO:tasks.workunit.client.0.smithi016.stdout:8/131: read d2/db/f16 [1369266,104455] 0 2024-04-22T15:49:29.895 INFO:tasks.workunit.client.0.smithi016.stdout:3/130: dwrite d7/f15 [4194304,4194304] 0 2024-04-22T15:49:29.900 INFO:tasks.workunit.client.0.smithi016.stdout:6/155: link d3/c26 d3/c2b 0 2024-04-22T15:49:29.900 INFO:tasks.workunit.client.0.smithi016.stdout:2/124: sync 2024-04-22T15:49:29.902 INFO:tasks.workunit.client.0.smithi016.stdout:2/125: chown d2/de/d2a/l38 18992 1 2024-04-22T15:49:29.903 INFO:tasks.workunit.client.0.smithi016.stdout:2/126: fsync d2/d9/f16 0 2024-04-22T15:49:29.904 INFO:tasks.workunit.client.0.smithi016.stdout:9/197: mkdir d9/de/d23/d24/d25/d36 0 2024-04-22T15:49:29.908 INFO:tasks.workunit.client.0.smithi016.stdout:8/132: mkdir d2/db/d1c 0 2024-04-22T15:49:29.908 INFO:tasks.workunit.client.0.smithi016.stdout:6/156: dwrite d3/da/db/f17 [0,4194304] 0 2024-04-22T15:49:29.909 INFO:tasks.workunit.client.0.smithi016.stdout:1/384: getdents d3/d4/d19/d20 0 2024-04-22T15:49:29.910 INFO:tasks.workunit.client.0.smithi016.stdout:8/133: write f1 [5202455,125040] 0 2024-04-22T15:49:29.911 INFO:tasks.workunit.client.0.smithi016.stdout:0/124: rename fb to dc/f27 0 2024-04-22T15:49:29.917 INFO:tasks.workunit.client.0.smithi016.stdout:9/198: mkdir d9/d37 0 2024-04-22T15:49:29.920 INFO:tasks.workunit.client.0.smithi016.stdout:3/131: fdatasync d7/f14 0 2024-04-22T15:49:29.921 INFO:tasks.workunit.client.0.smithi016.stdout:1/385: mkdir d3/d2e/d80 0 2024-04-22T15:49:29.927 INFO:tasks.workunit.client.0.smithi016.stdout:8/134: rename d2/d10/l19 to d2/db/l1d 0 2024-04-22T15:49:29.928 INFO:tasks.workunit.client.0.smithi016.stdout:8/135: truncate d2/db/f18 269884 0 2024-04-22T15:49:29.930 INFO:tasks.workunit.client.0.smithi016.stdout:2/127: dread d2/f26 [0,4194304] 0 2024-04-22T15:49:29.930 INFO:tasks.workunit.client.0.smithi016.stdout:2/128: readlink d2/d9/l1d 0 2024-04-22T15:49:29.931 INFO:tasks.workunit.client.0.smithi016.stdout:2/129: write f0 [1443792,107696] 0 2024-04-22T15:49:29.932 INFO:tasks.workunit.client.0.smithi016.stdout:1/386: truncate d3/d4/d19/f35 4074967 0 2024-04-22T15:49:29.934 INFO:tasks.workunit.client.0.smithi016.stdout:6/157: mknod d3/da/d21/c2c 0 2024-04-22T15:49:29.935 INFO:tasks.workunit.client.0.smithi016.stdout:9/199: creat d9/d20/f38 x:0 0 0 2024-04-22T15:49:29.935 INFO:tasks.workunit.client.0.smithi016.stdout:8/136: mkdir d2/d10/d1e 0 2024-04-22T15:49:29.939 INFO:tasks.workunit.client.0.smithi016.stdout:6/158: mknod d3/da/db/d14/d16/c2d 0 2024-04-22T15:49:29.941 INFO:tasks.workunit.client.0.smithi016.stdout:9/200: creat d9/de/d23/d24/f39 x:0 0 0 2024-04-22T15:49:29.941 INFO:tasks.workunit.client.0.smithi016.stdout:8/137: symlink d2/db/l1f 0 2024-04-22T15:49:29.942 INFO:tasks.workunit.client.0.smithi016.stdout:2/130: mkdir d2/d9/d2f/d39 0 2024-04-22T15:49:29.943 INFO:tasks.workunit.client.0.smithi016.stdout:8/138: chown d2/db/c11 36944342 1 2024-04-22T15:49:29.944 INFO:tasks.workunit.client.0.smithi016.stdout:2/131: read - d2/d1a/f20 zero size 2024-04-22T15:49:29.945 INFO:tasks.workunit.client.0.smithi016.stdout:7/153: dwrite d3/d4/fd [0,4194304] 0 2024-04-22T15:49:29.945 INFO:tasks.workunit.client.0.smithi016.stdout:6/159: readlink d3/l7 0 2024-04-22T15:49:29.947 INFO:tasks.workunit.client.0.smithi016.stdout:9/201: mknod d9/de/c3a 0 2024-04-22T15:49:29.947 INFO:tasks.workunit.client.0.smithi016.stdout:9/202: readlink d9/l2d 0 2024-04-22T15:49:29.948 INFO:tasks.workunit.client.0.smithi016.stdout:1/387: getdents d3/d4/d19/d20 0 2024-04-22T15:49:29.950 INFO:tasks.workunit.client.0.smithi016.stdout:8/139: fsync d2/db/f16 0 2024-04-22T15:49:29.951 INFO:tasks.workunit.client.0.smithi016.stdout:5/117: sync 2024-04-22T15:49:29.951 INFO:tasks.workunit.client.0.smithi016.stdout:4/129: sync 2024-04-22T15:49:29.952 INFO:tasks.workunit.client.0.smithi016.stdout:2/132: mknod d2/d23/d27/c3a 0 2024-04-22T15:49:29.954 INFO:tasks.workunit.client.0.smithi016.stdout:6/160: fdatasync d3/da/dd/f10 0 2024-04-22T15:49:29.956 INFO:tasks.workunit.client.0.smithi016.stdout:6/161: chown d3 5780 1 2024-04-22T15:49:29.956 INFO:tasks.workunit.client.0.smithi016.stdout:8/140: dread d2/f8 [4194304,4194304] 0 2024-04-22T15:49:29.959 INFO:tasks.workunit.client.0.smithi016.stdout:9/203: unlink d9/de/f11 0 2024-04-22T15:49:29.960 INFO:tasks.workunit.client.0.smithi016.stdout:3/132: truncate d7/f2c 185052 0 2024-04-22T15:49:29.964 INFO:tasks.workunit.client.0.smithi016.stdout:4/130: dwrite d7/f16 [0,4194304] 0 2024-04-22T15:49:29.964 INFO:tasks.workunit.client.0.smithi016.stdout:9/204: read d9/de/d23/f34 [3424749,53476] 0 2024-04-22T15:49:29.964 INFO:tasks.workunit.client.0.smithi016.stdout:2/133: dwrite d2/f11 [4194304,4194304] 0 2024-04-22T15:49:29.966 INFO:tasks.workunit.client.0.smithi016.stdout:5/118: mkdir df/d15/d1a 0 2024-04-22T15:49:29.972 INFO:tasks.workunit.client.0.smithi016.stdout:8/141: creat d2/db/f20 x:0 0 0 2024-04-22T15:49:29.975 INFO:tasks.workunit.client.0.smithi016.stdout:1/388: write d3/d4/d37/f40 [122546,74967] 0 2024-04-22T15:49:29.977 INFO:tasks.workunit.client.0.smithi016.stdout:4/131: fdatasync d7/f10 0 2024-04-22T15:49:29.979 INFO:tasks.workunit.client.0.smithi016.stdout:9/205: creat d9/f3b x:0 0 0 2024-04-22T15:49:29.980 INFO:tasks.workunit.client.0.smithi016.stdout:2/134: creat d2/d23/f3b x:0 0 0 2024-04-22T15:49:29.980 INFO:tasks.workunit.client.0.smithi016.stdout:2/135: dread - d2/d1a/f20 zero size 2024-04-22T15:49:29.987 INFO:tasks.workunit.client.0.smithi016.stdout:3/133: rmdir d7/d1b 39 2024-04-22T15:49:29.990 INFO:tasks.workunit.client.0.smithi016.stdout:4/132: symlink d7/d24/l2a 0 2024-04-22T15:49:29.992 INFO:tasks.workunit.client.0.smithi016.stdout:5/119: symlink df/d15/d1a/l1b 0 2024-04-22T15:49:29.992 INFO:tasks.workunit.client.0.smithi016.stdout:4/133: truncate d7/d21/d23/f29 29336 0 2024-04-22T15:49:29.994 INFO:tasks.workunit.client.0.smithi016.stdout:2/136: creat d2/d2c/f3c x:0 0 0 2024-04-22T15:49:29.995 INFO:tasks.workunit.client.0.smithi016.stdout:7/154: getdents d3/d4/d9 0 2024-04-22T15:49:29.995 INFO:tasks.workunit.client.0.smithi016.stdout:5/120: dread f6 [4194304,4194304] 0 2024-04-22T15:49:29.997 INFO:tasks.workunit.client.0.smithi016.stdout:7/155: chown d3/dc/d22/f38 160220 1 2024-04-22T15:49:30.000 INFO:tasks.workunit.client.0.smithi016.stdout:7/156: truncate d3/dc/f16 1016291 0 2024-04-22T15:49:30.000 INFO:tasks.workunit.client.0.smithi016.stdout:4/134: dwrite d7/d24/f26 [0,4194304] 0 2024-04-22T15:49:30.000 INFO:tasks.workunit.client.0.smithi016.stdout:6/162: truncate d3/f6 3395175 0 2024-04-22T15:49:30.001 INFO:tasks.workunit.client.0.smithi016.stdout:7/157: chown d3/d14 1 1 2024-04-22T15:49:30.002 INFO:tasks.workunit.client.0.smithi016.stdout:4/135: chown d7/f14 938807665 1 2024-04-22T15:49:30.003 INFO:tasks.workunit.client.0.smithi016.stdout:9/206: creat d9/d37/f3c x:0 0 0 2024-04-22T15:49:30.006 INFO:tasks.workunit.client.0.smithi016.stdout:0/125: getdents dc/d13/d14/d1d 0 2024-04-22T15:49:30.007 INFO:tasks.workunit.client.0.smithi016.stdout:2/137: fdatasync d2/d23/d27/f32 0 2024-04-22T15:49:30.011 INFO:tasks.workunit.client.0.smithi016.stdout:8/142: link d2/ld d2/db/l21 0 2024-04-22T15:49:30.012 INFO:tasks.workunit.client.0.smithi016.stdout:6/163: mknod d3/da/c2e 0 2024-04-22T15:49:30.013 INFO:tasks.workunit.client.0.smithi016.stdout:1/389: link d3/d4/d19/f54 d3/d4/d3a/d6b/d77/f81 0 2024-04-22T15:49:30.014 INFO:tasks.workunit.client.0.smithi016.stdout:2/138: dwrite d2/d2c/f3c [0,4194304] 0 2024-04-22T15:49:30.016 INFO:tasks.workunit.client.0.smithi016.stdout:2/139: read d2/d23/d27/f32 [856291,11529] 0 2024-04-22T15:49:30.018 INFO:tasks.workunit.client.0.smithi016.stdout:2/140: read f0 [1157048,15876] 0 2024-04-22T15:49:30.021 INFO:tasks.workunit.client.0.smithi016.stdout:7/158: dwrite d3/dc/d22/f39 [0,4194304] 0 2024-04-22T15:49:30.022 INFO:tasks.workunit.client.0.smithi016.stdout:0/126: mkdir dc/d13/d28 0 2024-04-22T15:49:30.023 INFO:tasks.workunit.client.0.smithi016.stdout:0/127: readlink dc/d13/l24 0 2024-04-22T15:49:30.032 INFO:tasks.workunit.client.0.smithi016.stdout:6/164: rename d3/l9 to d3/da/db/d14/d16/l2f 0 2024-04-22T15:49:30.034 INFO:tasks.workunit.client.0.smithi016.stdout:1/390: fsync d3/d4/da/f26 0 2024-04-22T15:49:30.037 INFO:tasks.workunit.client.0.smithi016.stdout:4/136: truncate d7/f10 3816100 0 2024-04-22T15:49:30.041 INFO:tasks.workunit.client.0.smithi016.stdout:1/391: dwrite d3/d28/f74 [0,4194304] 0 2024-04-22T15:49:30.043 INFO:tasks.workunit.client.0.smithi016.stdout:2/141: creat d2/d1a/f3d x:0 0 0 2024-04-22T15:49:30.044 INFO:tasks.workunit.client.0.smithi016.stdout:2/142: write d2/d9/f16 [912547,59573] 0 2024-04-22T15:49:30.045 INFO:tasks.workunit.client.0.smithi016.stdout:0/128: fdatasync dc/d13/d14/f18 0 2024-04-22T15:49:30.046 INFO:tasks.workunit.client.0.smithi016.stdout:2/143: readlink d2/d9/l36 0 2024-04-22T15:49:30.052 INFO:tasks.workunit.client.0.smithi016.stdout:3/134: link d7/fc d7/f30 0 2024-04-22T15:49:30.054 INFO:tasks.workunit.client.0.smithi016.stdout:6/165: mknod d3/da/dd/c30 0 2024-04-22T15:49:30.055 INFO:tasks.workunit.client.0.smithi016.stdout:6/166: write d3/da/d12/d13/f27 [978739,40038] 0 2024-04-22T15:49:30.060 INFO:tasks.workunit.client.0.smithi016.stdout:9/207: dwrite d9/de/f16 [0,4194304] 0 2024-04-22T15:49:30.062 INFO:tasks.workunit.client.0.smithi016.stdout:7/159: symlink d3/d32/l3c 0 2024-04-22T15:49:30.064 INFO:tasks.workunit.client.0.smithi016.stdout:5/121: getdents df 0 2024-04-22T15:49:30.064 INFO:tasks.workunit.client.0.smithi016.stdout:0/129: truncate dc/f1e 415895 0 2024-04-22T15:49:30.068 INFO:tasks.workunit.client.0.smithi016.stdout:8/143: rmdir d2/d10/d1e 0 2024-04-22T15:49:30.068 INFO:tasks.workunit.client.0.smithi016.stdout:8/144: chown d2/f8 276093 1 2024-04-22T15:49:30.078 INFO:tasks.workunit.client.0.smithi016.stdout:1/392: mkdir d3/d2e/d80/d82 0 2024-04-22T15:49:30.089 INFO:tasks.workunit.client.0.smithi016.stdout:4/137: dwrite d7/f10 [0,4194304] 0 2024-04-22T15:49:30.089 INFO:tasks.workunit.client.0.smithi016.stdout:4/138: stat d7/f1d 0 2024-04-22T15:49:30.090 INFO:tasks.workunit.client.0.smithi016.stdout:4/139: chown d7/l1a 7672 1 2024-04-22T15:49:30.095 INFO:tasks.workunit.client.0.smithi016.stdout:2/144: rmdir d2/d9/d2f/d39 0 2024-04-22T15:49:30.098 INFO:tasks.workunit.client.0.smithi016.stdout:8/145: write d2/db/f16 [1491849,48490] 0 2024-04-22T15:49:30.099 INFO:tasks.workunit.client.0.smithi016.stdout:0/130: truncate dc/d13/d14/f18 1300367 0 2024-04-22T15:49:30.101 INFO:tasks.workunit.client.0.smithi016.stdout:4/140: mknod d7/d21/c2b 0 2024-04-22T15:49:30.102 INFO:tasks.workunit.client.0.smithi016.stdout:0/131: chown dc/d13/f17 1 1 2024-04-22T15:49:30.103 INFO:tasks.workunit.client.0.smithi016.stdout:6/167: rename d3/da/db/d14/f1e to d3/f31 0 2024-04-22T15:49:30.103 INFO:tasks.workunit.client.0.smithi016.stdout:8/146: write d2/db/f16 [1865981,111110] 0 2024-04-22T15:49:30.105 INFO:tasks.workunit.client.0.smithi016.stdout:8/147: chown d2 1894306 1 2024-04-22T15:49:30.106 INFO:tasks.workunit.client.0.smithi016.stdout:4/141: chown d7/d24/f26 165 1 2024-04-22T15:49:30.107 INFO:tasks.workunit.client.0.smithi016.stdout:5/122: getdents df/d15/d1a 0 2024-04-22T15:49:30.109 INFO:tasks.workunit.client.0.smithi016.stdout:9/208: rename d9/f14 to d9/de/d23/d24/d25/f3d 0 2024-04-22T15:49:30.111 INFO:tasks.workunit.client.0.smithi016.stdout:2/145: dwrite d2/f2b [4194304,4194304] 0 2024-04-22T15:49:30.117 INFO:tasks.workunit.client.0.smithi016.stdout:3/135: dwrite d7/f12 [0,4194304] 0 2024-04-22T15:49:30.125 INFO:tasks.workunit.client.0.smithi016.stdout:3/136: write d7/f15 [7895992,61162] 0 2024-04-22T15:49:30.125 INFO:tasks.workunit.client.0.smithi016.stdout:4/142: dwrite f2 [4194304,4194304] 0 2024-04-22T15:49:30.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.123+0000 7f240b978700 1 -- 172.21.15.16:0/388474185 --> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f24040611d0 con 0x7f23f006d850 2024-04-22T15:49:30.128 INFO:tasks.workunit.client.0.smithi016.stdout:7/160: rename d3/d4/f7 to d3/d32/f3d 0 2024-04-22T15:49:30.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.125+0000 7f23fa7fc700 1 -- 172.21.15.16:0/388474185 <== mgr.24395 v2:172.21.15.27:6828/2676414545 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f24040611d0 con 0x7f23f006d850 2024-04-22T15:49:30.129 INFO:tasks.workunit.client.0.smithi016.stdout:4/143: fsync d7/d24/f26 0 2024-04-22T15:49:30.130 INFO:tasks.workunit.client.0.smithi016.stdout:9/209: creat d9/d20/f3e x:0 0 0 2024-04-22T15:49:30.130 INFO:tasks.workunit.client.0.smithi016.stdout:5/123: symlink df/d15/d1a/l1c 0 2024-04-22T15:49:30.131 INFO:tasks.workunit.client.0.smithi016.stdout:4/144: fsync d7/f16 0 2024-04-22T15:49:30.132 INFO:tasks.workunit.client.0.smithi016.stdout:6/168: dread d3/f1d [0,4194304] 0 2024-04-22T15:49:30.133 INFO:tasks.workunit.client.0.smithi016.stdout:4/145: fsync d7/d24/f26 0 2024-04-22T15:49:30.134 INFO:tasks.workunit.client.0.smithi016.stdout:2/146: unlink d2/d1a/f3d 0 2024-04-22T15:49:30.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.132+0000 7f240b978700 1 -- 172.21.15.16:0/388474185 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f23f006d850 msgr2=0x7f23f006fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:30.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.132+0000 7f240b978700 1 --2- 172.21.15.16:0/388474185 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f23f006d850 0x7f23f006fd10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f240000b6f0 tx=0x7f2400005940 comp rx=0 tx=0).stop 2024-04-22T15:49:30.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.132+0000 7f240b978700 1 -- 172.21.15.16:0/388474185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2404101ea0 msgr2=0x7f240419d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:30.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.132+0000 7f240b978700 1 --2- 172.21.15.16:0/388474185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2404101ea0 0x7f240419d000 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f23f400e920 tx=0x7f23f400ec30 comp rx=0 tx=0).stop 2024-04-22T15:49:30.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.132+0000 7f240b978700 1 -- 172.21.15.16:0/388474185 shutdown_connections 2024-04-22T15:49:30.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.132+0000 7f240b978700 1 --2- 172.21.15.16:0/388474185 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7f23f006d850 0x7f23f006fd10 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:30.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.132+0000 7f240b978700 1 --2- 172.21.15.16:0/388474185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2404100ca0 0x7f240419cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:30.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.132+0000 7f240b978700 1 --2- 172.21.15.16:0/388474185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2404101ea0 0x7f240419d000 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:30.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.132+0000 7f240b978700 1 -- 172.21.15.16:0/388474185 >> 172.21.15.16:0/388474185 conn(0x7f24040fc260 msgr2=0x7f24041030c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:30.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.132+0000 7f240b978700 1 -- 172.21.15.16:0/388474185 shutdown_connections 2024-04-22T15:49:30.136 INFO:tasks.workunit.client.0.smithi016.stdout:0/132: symlink dc/d13/d14/d1d/d1f/l29 0 2024-04-22T15:49:30.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.132+0000 7f240b978700 1 -- 172.21.15.16:0/388474185 wait complete. 2024-04-22T15:49:30.138 INFO:tasks.workunit.client.0.smithi016.stdout:1/393: rename d3/d4/d19/d20/d2b/f59 to d3/d4/da/f83 0 2024-04-22T15:49:30.139 INFO:tasks.workunit.client.0.smithi016.stdout:3/137: write d7/d17/f2e [7608,50519] 0 2024-04-22T15:49:30.139 INFO:tasks.workunit.client.0.smithi016.stdout:4/146: chown d7/d24/f26 55 1 2024-04-22T15:49:30.142 INFO:tasks.workunit.client.0.smithi016.stdout:4/147: stat d7/l1f 0 2024-04-22T15:49:30.142 INFO:tasks.workunit.client.0.smithi016.stdout:8/148: creat d2/f22 x:0 0 0 2024-04-22T15:49:30.144 INFO:tasks.workunit.client.0.smithi016.stdout:5/124: mkdir df/d15/d1a/d1d 0 2024-04-22T15:49:30.146 INFO:tasks.workunit.client.0.smithi016.stdout:6/169: creat d3/da/db/d14/d16/f32 x:0 0 0 2024-04-22T15:49:30.146 INFO:tasks.workunit.client.0.smithi016.stdout:9/210: dwrite d9/de/f18 [4194304,4194304] 0 2024-04-22T15:49:30.149 INFO:tasks.workunit.client.0.smithi016.stdout:9/211: stat l5 0 2024-04-22T15:49:30.149 INFO:tasks.workunit.client.0.smithi016.stdout:7/161: creat d3/dc/d33/f3e x:0 0 0 2024-04-22T15:49:30.151 INFO:tasks.workunit.client.0.smithi016.stdout:0/133: dwrite dc/d13/f17 [0,4194304] 0 2024-04-22T15:49:30.153 INFO:tasks.workunit.client.0.smithi016.stdout:3/138: mkdir d7/d17/d18/d31 0 2024-04-22T15:49:30.154 INFO:tasks.workunit.client.0.smithi016.stdout:4/148: mknod d7/d24/c2c 0 2024-04-22T15:49:30.155 INFO:tasks.workunit.client.0.smithi016.stdout:9/212: write d9/f3b [728720,108199] 0 2024-04-22T15:49:30.155 INFO:tasks.workunit.client.0.smithi016.stdout:6/170: mkdir d3/da/d21/d33 0 2024-04-22T15:49:30.158 INFO:tasks.workunit.client.0.smithi016.stdout:9/213: fsync d9/d37/f3c 0 2024-04-22T15:49:30.159 INFO:tasks.workunit.client.0.smithi016.stdout:7/162: creat d3/dc/d22/f3f x:0 0 0 2024-04-22T15:49:30.160 INFO:tasks.workunit.client.0.smithi016.stdout:8/149: unlink d2/c17 0 2024-04-22T15:49:30.164 INFO:tasks.workunit.client.0.smithi016.stdout:2/147: rmdir d2/d23/d27/d33 0 2024-04-22T15:49:30.164 INFO:tasks.workunit.client.0.smithi016.stdout:7/163: dread - d3/dc/d22/f3f zero size 2024-04-22T15:49:30.165 INFO:tasks.workunit.client.0.smithi016.stdout:5/125: dwrite df/f11 [0,4194304] 0 2024-04-22T15:49:30.167 INFO:tasks.workunit.client.0.smithi016.stdout:8/150: write d2/f22 [57627,66449] 0 2024-04-22T15:49:30.168 INFO:tasks.workunit.client.0.smithi016.stdout:8/151: fsync f1 0 2024-04-22T15:49:30.170 INFO:tasks.workunit.client.0.smithi016.stdout:2/148: creat d2/f3e x:0 0 0 2024-04-22T15:49:30.171 INFO:tasks.workunit.client.0.smithi016.stdout:2/149: chown d2/d1a 357589 1 2024-04-22T15:49:30.172 INFO:tasks.workunit.client.0.smithi016.stdout:3/139: dwrite d7/f12 [0,4194304] 0 2024-04-22T15:49:30.173 INFO:tasks.workunit.client.0.smithi016.stdout:5/126: creat df/d15/d1a/f1e x:0 0 0 2024-04-22T15:49:30.173 INFO:tasks.workunit.client.0.smithi016.stdout:2/150: readlink d2/l3 0 2024-04-22T15:49:30.177 INFO:tasks.workunit.client.0.smithi016.stdout:8/152: mkdir d2/d10/d23 0 2024-04-22T15:49:30.178 INFO:tasks.workunit.client.0.smithi016.stdout:5/127: rmdir df/d15 39 2024-04-22T15:49:30.180 INFO:tasks.workunit.client.0.smithi016.stdout:5/128: write f2 [4895964,31972] 0 2024-04-22T15:49:30.180 INFO:tasks.workunit.client.0.smithi016.stdout:2/151: creat d2/d9/d2f/f3f x:0 0 0 2024-04-22T15:49:30.181 INFO:tasks.workunit.client.0.smithi016.stdout:9/214: dwrite d9/de/d23/f34 [0,4194304] 0 2024-04-22T15:49:30.182 INFO:tasks.workunit.client.0.smithi016.stdout:3/140: mkdir d7/d17/d18/d32 0 2024-04-22T15:49:30.184 INFO:tasks.workunit.client.0.smithi016.stdout:9/215: fdatasync d9/de/d23/d24/d25/f32 0 2024-04-22T15:49:30.191 INFO:tasks.workunit.client.0.smithi016.stdout:9/216: dread d9/de/d23/f34 [0,4194304] 0 2024-04-22T15:49:30.193 INFO:tasks.workunit.client.0.smithi016.stdout:5/129: fsync df/d15/f16 0 2024-04-22T15:49:30.193 INFO:tasks.workunit.client.0.smithi016.stdout:1/394: write d3/d4/d19/f54 [3973918,45728] 0 2024-04-22T15:49:30.194 INFO:tasks.workunit.client.0.smithi016.stdout:8/153: creat d2/f24 x:0 0 0 2024-04-22T15:49:30.198 INFO:tasks.workunit.client.0.smithi016.stdout:9/217: creat d9/f3f x:0 0 0 2024-04-22T15:49:30.198 INFO:tasks.workunit.client.0.smithi016.stdout:5/130: rmdir df/d15 39 2024-04-22T15:49:30.199 INFO:tasks.workunit.client.0.smithi016.stdout:9/218: fsync d9/de/f28 0 2024-04-22T15:49:30.199 INFO:tasks.workunit.client.0.smithi016.stdout:1/395: mkdir d3/d84 0 2024-04-22T15:49:30.202 INFO:tasks.workunit.client.0.smithi016.stdout:8/154: readlink d2/ld 0 2024-04-22T15:49:30.204 INFO:tasks.workunit.client.0.smithi016.stdout:0/134: dwrite dc/d13/f17 [4194304,4194304] 0 2024-04-22T15:49:30.204 INFO:tasks.workunit.client.0.smithi016.stdout:9/219: creat d9/d20/f40 x:0 0 0 2024-04-22T15:49:30.205 INFO:tasks.workunit.client.0.smithi016.stdout:4/149: truncate d7/f19 2514264 0 2024-04-22T15:49:30.207 INFO:tasks.workunit.client.0.smithi016.stdout:3/141: dread d7/d17/d18/f23 [0,4194304] 0 2024-04-22T15:49:30.208 INFO:tasks.workunit.client.0.smithi016.stdout:1/396: mknod d3/d4/d3a/d6b/d77/c85 0 2024-04-22T15:49:30.211 INFO:tasks.workunit.client.0.smithi016.stdout:7/164: dread d3/f28 [0,4194304] 0 2024-04-22T15:49:30.211 INFO:tasks.workunit.client.0.smithi016.stdout:5/131: rename df/f14 to df/f1f 0 2024-04-22T15:49:30.213 INFO:tasks.workunit.client.0.smithi016.stdout:8/155: mkdir d2/db/d25 0 2024-04-22T15:49:30.213 INFO:tasks.workunit.client.0.smithi016.stdout:4/150: write f5 [206630,108711] 0 2024-04-22T15:49:30.215 INFO:tasks.workunit.client.0.smithi016.stdout:4/151: chown d7/d21 2843791 1 2024-04-22T15:49:30.215 INFO:tasks.workunit.client.0.smithi016.stdout:9/220: creat d9/de/d23/d24/d25/f41 x:0 0 0 2024-04-22T15:49:30.216 INFO:tasks.workunit.client.0.smithi016.stdout:6/171: dwrite d3/f8 [0,4194304] 0 2024-04-22T15:49:30.217 INFO:tasks.workunit.client.0.smithi016.stdout:3/142: creat d7/d17/d26/f33 x:0 0 0 2024-04-22T15:49:30.219 INFO:tasks.workunit.client.0.smithi016.stdout:6/172: chown d3/da/dd/d1b 475 1 2024-04-22T15:49:30.221 INFO:tasks.workunit.client.0.smithi016.stdout:0/135: rename dc/d13/d14/l21 to dc/d13/l2a 0 2024-04-22T15:49:30.222 INFO:tasks.workunit.client.0.smithi016.stdout:8/156: fsync d2/f24 0 2024-04-22T15:49:30.223 INFO:tasks.workunit.client.0.smithi016.stdout:6/173: chown d3/f24 6174 1 2024-04-22T15:49:30.223 INFO:tasks.workunit.client.0.smithi016.stdout:4/152: dread d7/f10 [0,4194304] 0 2024-04-22T15:49:30.226 INFO:tasks.workunit.client.0.smithi016.stdout:5/132: chown df/d15/d1a/l1c 163 1 2024-04-22T15:49:30.226 INFO:tasks.workunit.client.0.smithi016.stdout:7/165: dread d3/d4/d9/fb [0,4194304] 0 2024-04-22T15:49:30.227 INFO:tasks.workunit.client.0.smithi016.stdout:6/174: read d3/f31 [388787,121614] 0 2024-04-22T15:49:30.232 INFO:tasks.workunit.client.0.smithi016.stdout:8/157: dwrite d2/f24 [0,4194304] 0 2024-04-22T15:49:30.234 INFO:tasks.workunit.client.0.smithi016.stdout:1/397: dread d3/d4/d19/f1d [4194304,4194304] 0 2024-04-22T15:49:30.235 INFO:tasks.workunit.client.0.smithi016.stdout:4/153: symlink d7/l2d 0 2024-04-22T15:49:30.236 INFO:tasks.workunit.client.0.smithi016.stdout:1/398: write d3/d28/f74 [4876900,105699] 0 2024-04-22T15:49:30.237 INFO:tasks.workunit.client.0.smithi016.stdout:1/399: chown d3/c27 3 1 2024-04-22T15:49:30.239 INFO:tasks.workunit.client.0.smithi016.stdout:1/400: truncate d3/d4/d19/f66 951802 0 2024-04-22T15:49:30.240 INFO:tasks.workunit.client.0.smithi016.stdout:1/401: fsync d3/f4f 0 2024-04-22T15:49:30.242 INFO:tasks.workunit.client.0.smithi016.stdout:3/143: rmdir d7/d17/d26 39 2024-04-22T15:49:30.242 INFO:tasks.workunit.client.0.smithi016.stdout:4/154: dread f2 [4194304,4194304] 0 2024-04-22T15:49:30.249 INFO:tasks.workunit.client.0.smithi016.stdout:7/166: dwrite d3/d4/d9/df/f11 [0,4194304] 0 2024-04-22T15:49:30.252 INFO:tasks.workunit.client.0.smithi016.stdout:6/175: dwrite d3/f31 [0,4194304] 0 2024-04-22T15:49:30.255 INFO:tasks.workunit.client.0.smithi016.stdout:5/133: write f6 [3651967,50173] 0 2024-04-22T15:49:30.256 INFO:tasks.workunit.client.0.smithi016.stdout:5/134: write df/f10 [4124898,60261] 0 2024-04-22T15:49:30.261 INFO:tasks.workunit.client.0.smithi016.stdout:4/155: mknod d7/d21/d23/c2e 0 2024-04-22T15:49:30.264 INFO:tasks.workunit.client.0.smithi016.stdout:1/402: rename d3/d4/d19/f64 to d3/d4/d15/d38/d63/d67/f86 0 2024-04-22T15:49:30.266 INFO:tasks.workunit.client.0.smithi016.stdout:5/135: dwrite f6 [4194304,4194304] 0 2024-04-22T15:49:30.267 INFO:tasks.workunit.client.0.smithi016.stdout:8/158: mknod d2/d10/d23/c26 0 2024-04-22T15:49:30.268 INFO:tasks.workunit.client.0.smithi016.stdout:8/159: chown d2/db/c11 1 1 2024-04-22T15:49:30.270 INFO:tasks.workunit.client.0.smithi016.stdout:7/167: unlink d3/dc/d22/f3f 0 2024-04-22T15:49:30.271 INFO:tasks.workunit.client.0.smithi016.stdout:7/168: fdatasync d3/f23 0 2024-04-22T15:49:30.273 INFO:tasks.workunit.client.0.smithi016.stdout:3/144: symlink d7/d1b/l34 0 2024-04-22T15:49:30.274 INFO:tasks.workunit.client.0.smithi016.stdout:3/145: stat d7/d17/d18/l19 0 2024-04-22T15:49:30.274 INFO:tasks.workunit.client.0.smithi016.stdout:1/403: rename d3/d4/d19/d20/l5b to d3/d4/d37/l87 0 2024-04-22T15:49:30.276 INFO:tasks.workunit.client.0.smithi016.stdout:5/136: fsync df/f1f 0 2024-04-22T15:49:30.276 INFO:tasks.workunit.client.0.smithi016.stdout:5/137: readlink l7 0 2024-04-22T15:49:30.277 INFO:tasks.workunit.client.0.smithi016.stdout:5/138: chown df/d15/d1a/l1b 34073239 1 2024-04-22T15:49:30.277 INFO:tasks.workunit.client.0.smithi016.stdout:5/139: chown df/d15 417936044 1 2024-04-22T15:49:30.279 INFO:tasks.workunit.client.0.smithi016.stdout:6/176: mknod d3/da/d21/d33/c34 0 2024-04-22T15:49:30.283 INFO:tasks.workunit.client.0.smithi016.stdout:0/136: getdents dc/d13/d14 0 2024-04-22T15:49:30.283 INFO:tasks.workunit.client.0.smithi016.stdout:0/137: fsync dc/d13/f17 0 2024-04-22T15:49:30.284 INFO:tasks.workunit.client.0.smithi016.stdout:7/169: symlink d3/d4/d9/df/l40 0 2024-04-22T15:49:30.289 INFO:tasks.workunit.client.0.smithi016.stdout:5/140: creat df/d15/d1a/f20 x:0 0 0 2024-04-22T15:49:30.291 INFO:tasks.workunit.client.0.smithi016.stdout:6/177: mkdir d3/da/d21/d33/d35 0 2024-04-22T15:49:30.296 INFO:tasks.workunit.client.0.smithi016.stdout:7/170: creat d3/d4/d9/df/f41 x:0 0 0 2024-04-22T15:49:30.298 INFO:tasks.workunit.client.0.smithi016.stdout:3/146: write d7/d17/d18/f23 [3437955,96701] 0 2024-04-22T15:49:30.299 INFO:tasks.workunit.client.0.smithi016.stdout:1/404: truncate d3/d4/d19/d20/f48 3887230 0 2024-04-22T15:49:30.302 INFO:tasks.workunit.client.0.smithi016.stdout:8/160: creat d2/f27 x:0 0 0 2024-04-22T15:49:30.302 INFO:tasks.workunit.client.0.smithi016.stdout:5/141: rmdir df/d15/d1a 39 2024-04-22T15:49:30.304 INFO:tasks.workunit.client.0.smithi016.stdout:6/178: mknod d3/da/db/d14/d16/c36 0 2024-04-22T15:49:30.304 INFO:tasks.workunit.client.0.smithi016.stdout:1/405: dread d3/d4/fe [0,4194304] 0 2024-04-22T15:49:30.304 INFO:tasks.workunit.client.0.smithi016.stdout:5/142: chown df/f11 93158 1 2024-04-22T15:49:30.304 INFO:tasks.workunit.client.0.smithi016.stdout:0/138: symlink dc/d13/d22/l2b 0 2024-04-22T15:49:30.306 INFO:tasks.workunit.client.0.smithi016.stdout:6/179: chown d3/da/dd/f23 635 1 2024-04-22T15:49:30.306 INFO:tasks.workunit.client.0.smithi016.stdout:0/139: rename dc to dc/d13/d28/d2c 22 2024-04-22T15:49:30.307 INFO:tasks.workunit.client.0.smithi016.stdout:7/171: mkdir d3/d4/d9/df/d42 0 2024-04-22T15:49:30.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.305+0000 7fd454e0c700 1 -- 172.21.15.16:0/4051820904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd450101170 msgr2=0x7fd450101590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:30.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.305+0000 7fd454e0c700 1 --2- 172.21.15.16:0/4051820904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd450101170 0x7fd450101590 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd440009910 tx=0x7fd440009c20 comp rx=0 tx=0).stop 2024-04-22T15:49:30.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.305+0000 7fd454e0c700 1 -- 172.21.15.16:0/4051820904 shutdown_connections 2024-04-22T15:49:30.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.305+0000 7fd454e0c700 1 --2- 172.21.15.16:0/4051820904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4500fed00 0x7fd4500ff160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:30.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.305+0000 7fd454e0c700 1 --2- 172.21.15.16:0/4051820904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd450101170 0x7fd450101590 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:30.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.305+0000 7fd454e0c700 1 -- 172.21.15.16:0/4051820904 >> 172.21.15.16:0/4051820904 conn(0x7fd4500fa8c0 msgr2=0x7fd4500fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:30.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.305+0000 7fd454e0c700 1 -- 172.21.15.16:0/4051820904 shutdown_connections 2024-04-22T15:49:30.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.305+0000 7fd454e0c700 1 -- 172.21.15.16:0/4051820904 wait complete. 2024-04-22T15:49:30.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.306+0000 7fd454e0c700 1 Processor -- start 2024-04-22T15:49:30.309 INFO:tasks.workunit.client.0.smithi016.stdout:7/172: read - d3/dc/f1e zero size 2024-04-22T15:49:30.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.306+0000 7fd454e0c700 1 -- start start 2024-04-22T15:49:30.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.307+0000 7fd454e0c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4500fed00 0x7fd450105950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:30.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.307+0000 7fd454e0c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd450101170 0x7fd450105e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:30.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.307+0000 7fd454e0c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd450102440 con 0x7fd4500fed00 2024-04-22T15:49:30.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.307+0000 7fd454e0c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4501025b0 con 0x7fd450101170 2024-04-22T15:49:30.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.307+0000 7fd44e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4500fed00 0x7fd450105950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:30.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.307+0000 7fd44dd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd450101170 0x7fd450105e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:30.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.307+0000 7fd44dd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd450101170 0x7fd450105e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:44802/0 (socket says 172.21.15.16:44802) 2024-04-22T15:49:30.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.307+0000 7fd44dd9b700 1 -- 172.21.15.16:0/808903036 learned_addr learned my addr 172.21.15.16:0/808903036 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:49:30.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.307+0000 7fd44dd9b700 1 -- 172.21.15.16:0/808903036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4500fed00 msgr2=0x7fd450105950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:30.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.307+0000 7fd44dd9b700 1 --2- 172.21.15.16:0/808903036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4500fed00 0x7fd450105950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:30.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.307+0000 7fd44dd9b700 1 -- 172.21.15.16:0/808903036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4400095f0 con 0x7fd450101170 2024-04-22T15:49:30.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.307+0000 7fd44e59c700 1 --2- 172.21.15.16:0/808903036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4500fed00 0x7fd450105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:49:30.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.308+0000 7fd44dd9b700 1 --2- 172.21.15.16:0/808903036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd450101170 0x7fd450105e90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd44400e920 tx=0x7fd44400ec30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:30.312 INFO:tasks.workunit.client.0.smithi016.stdout:3/147: creat d7/d1b/f35 x:0 0 0 2024-04-22T15:49:30.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.308+0000 7fd43f7fe700 1 -- 172.21.15.16:0/808903036 <== mon.1 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd44400c9d0 con 0x7fd450101170 2024-04-22T15:49:30.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.308+0000 7fd43f7fe700 1 -- 172.21.15.16:0/808903036 <== mon.1 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd444010430 con 0x7fd450101170 2024-04-22T15:49:30.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.308+0000 7fd454e0c700 1 -- 172.21.15.16:0/808903036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd450102890 con 0x7fd450101170 2024-04-22T15:49:30.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.308+0000 7fd43f7fe700 1 -- 172.21.15.16:0/808903036 <== mon.1 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd444018650 con 0x7fd450101170 2024-04-22T15:49:30.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.308+0000 7fd454e0c700 1 -- 172.21.15.16:0/808903036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd450102de0 con 0x7fd450101170 2024-04-22T15:49:30.313 INFO:tasks.workunit.client.0.smithi016.stdout:8/161: dwrite f1 [0,4194304] 0 2024-04-22T15:49:30.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.309+0000 7fd454e0c700 1 -- 172.21.15.16:0/808903036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd450103000 con 0x7fd450101170 2024-04-22T15:49:30.313 INFO:tasks.workunit.client.0.smithi016.stdout:3/148: chown d7/d1b/d21 4732762 1 2024-04-22T15:49:30.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.310+0000 7fd43f7fe700 1 -- 172.21.15.16:0/808903036 <== mon.1 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) v1 ==== 91557+0+0 (secure 0 0 0) 0x7fd4440105a0 con 0x7fd450101170 2024-04-22T15:49:30.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.311+0000 7fd43f7fe700 1 --2- 172.21.15.16:0/808903036 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7fd43806d8a0 0x7fd43806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:30.314 INFO:tasks.workunit.client.0.smithi016.stdout:0/140: symlink dc/d13/d14/d1d/l2d 0 2024-04-22T15:49:30.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.311+0000 7fd43f7fe700 1 -- 172.21.15.16:0/808903036 <== mon.1 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fd444014070 con 0x7fd450101170 2024-04-22T15:49:30.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.311+0000 7fd44e59c700 1 --2- 172.21.15.16:0/808903036 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7fd43806d8a0 0x7fd43806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:30.315 INFO:tasks.workunit.client.0.smithi016.stdout:7/173: symlink d3/dc/d22/l43 0 2024-04-22T15:49:30.315 INFO:tasks.workunit.client.0.smithi016.stdout:5/143: truncate df/d15/d1a/f20 983173 0 2024-04-22T15:49:30.316 INFO:tasks.workunit.client.0.smithi016.stdout:8/162: dread d2/db/f18 [0,4194304] 0 2024-04-22T15:49:30.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.313+0000 7fd44e59c700 1 --2- 172.21.15.16:0/808903036 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7fd43806d8a0 0x7fd43806fd60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd440009fd0 tx=0x7fd440005940 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:30.317 INFO:tasks.workunit.client.0.smithi016.stdout:5/144: chown l9 126029 1 2024-04-22T15:49:30.321 INFO:tasks.workunit.client.0.smithi016.stdout:8/163: dread d2/db/f16 [0,4194304] 0 2024-04-22T15:49:30.321 INFO:tasks.workunit.client.0.smithi016.stdout:7/174: rmdir d3/dc/d1b 39 2024-04-22T15:49:30.322 INFO:tasks.workunit.client.0.smithi016.stdout:3/149: dwrite d7/d17/d18/f1f [0,4194304] 0 2024-04-22T15:49:30.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.319+0000 7fd43f7fe700 1 -- 172.21.15.16:0/808903036 <== mon.1 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7fd444053e20 con 0x7fd450101170 2024-04-22T15:49:30.324 INFO:tasks.workunit.client.0.smithi016.stdout:3/150: readlink d7/d1b/d21/l2b 0 2024-04-22T15:49:30.325 INFO:tasks.workunit.client.0.smithi016.stdout:5/145: creat df/d15/d1a/f21 x:0 0 0 2024-04-22T15:49:30.331 INFO:tasks.workunit.client.0.smithi016.stdout:6/180: getdents d3/da/d12/d13 0 2024-04-22T15:49:30.331 INFO:tasks.workunit.client.0.smithi016.stdout:3/151: dread d7/d1b/d21/f28 [0,4194304] 0 2024-04-22T15:49:30.339 INFO:tasks.workunit.client.0.smithi016.stdout:3/152: dwrite f1 [0,4194304] 0 2024-04-22T15:49:30.340 INFO:tasks.workunit.client.0.smithi016.stdout:3/153: write d7/f15 [7284286,76145] 0 2024-04-22T15:49:30.343 INFO:tasks.workunit.client.0.smithi016.stdout:6/181: mknod d3/da/db/c37 0 2024-04-22T15:49:30.344 INFO:tasks.workunit.client.0.smithi016.stdout:8/164: truncate d2/db/f18 5622 0 2024-04-22T15:49:30.346 INFO:tasks.workunit.client.0.smithi016.stdout:0/141: getdents dc/d13/d14/d1d/d1f 0 2024-04-22T15:49:30.349 INFO:tasks.workunit.client.0.smithi016.stdout:3/154: read - d7/d17/d26/f33 zero size 2024-04-22T15:49:30.353 INFO:tasks.workunit.client.0.smithi016.stdout:6/182: rename d3/da/d21 to d3/da/d12/d13/d38 0 2024-04-22T15:49:30.353 INFO:tasks.workunit.client.0.smithi016.stdout:3/155: rename d7 to d7/d17/d18/d36 22 2024-04-22T15:49:30.353 INFO:tasks.workunit.client.0.smithi016.stdout:7/175: rmdir d3/d4/d9/df/d42 0 2024-04-22T15:49:30.354 INFO:tasks.workunit.client.0.smithi016.stdout:6/183: stat d3/da/db/d14/d16/f32 0 2024-04-22T15:49:30.355 INFO:tasks.workunit.client.0.smithi016.stdout:0/142: creat dc/d13/d22/f2e x:0 0 0 2024-04-22T15:49:30.355 INFO:tasks.workunit.client.0.smithi016.stdout:8/165: mkdir d2/db/d25/d28 0 2024-04-22T15:49:30.356 INFO:tasks.workunit.client.0.smithi016.stdout:0/143: stat dc/d13/d14/d26 0 2024-04-22T15:49:30.356 INFO:tasks.workunit.client.0.smithi016.stdout:5/146: getdents df/d15 0 2024-04-22T15:49:30.357 INFO:tasks.workunit.client.0.smithi016.stdout:5/147: chown f6 1973 1 2024-04-22T15:49:30.361 INFO:tasks.workunit.client.0.smithi016.stdout:7/176: rename d3/d4/d9/df/f41 to d3/dc/d33/f44 0 2024-04-22T15:49:30.364 INFO:tasks.workunit.client.0.smithi016.stdout:6/184: mknod d3/da/dd/d1b/c39 0 2024-04-22T15:49:30.364 INFO:tasks.workunit.client.0.smithi016.stdout:6/185: stat d3/da/dd/d1b 0 2024-04-22T15:49:30.366 INFO:tasks.workunit.client.0.smithi016.stdout:0/144: symlink dc/l2f 0 2024-04-22T15:49:30.367 INFO:tasks.workunit.client.0.smithi016.stdout:7/177: write d3/dc/d33/f44 [371265,77885] 0 2024-04-22T15:49:30.368 INFO:tasks.workunit.client.0.smithi016.stdout:7/178: chown d3/dc/d22/c3b 14944787 1 2024-04-22T15:49:30.369 INFO:tasks.workunit.client.0.smithi016.stdout:8/166: mkdir d2/db/d1c/d29 0 2024-04-22T15:49:30.370 INFO:tasks.workunit.client.0.smithi016.stdout:3/156: rmdir d7/d17/d18/d31 0 2024-04-22T15:49:30.372 INFO:tasks.workunit.client.0.smithi016.stdout:5/148: mkdir df/d22 0 2024-04-22T15:49:30.375 INFO:tasks.workunit.client.0.smithi016.stdout:8/167: mkdir d2/db/d1c/d2a 0 2024-04-22T15:49:30.377 INFO:tasks.workunit.client.0.smithi016.stdout:8/168: write d2/f27 [822737,30839] 0 2024-04-22T15:49:30.378 INFO:tasks.workunit.client.0.smithi016.stdout:0/145: rmdir dc/d13/d14/d1d 39 2024-04-22T15:49:30.381 INFO:tasks.workunit.client.0.smithi016.stdout:7/179: rename d3/dc/d22/l24 to d3/l45 0 2024-04-22T15:49:30.384 INFO:tasks.workunit.client.0.smithi016.stdout:0/146: mkdir dc/d13/d14/d1d/d30 0 2024-04-22T15:49:30.385 INFO:tasks.workunit.client.0.smithi016.stdout:8/169: dread d2/f8 [0,4194304] 0 2024-04-22T15:49:30.387 INFO:tasks.workunit.client.0.smithi016.stdout:3/157: dwrite d7/f8 [0,4194304] 0 2024-04-22T15:49:30.387 INFO:tasks.workunit.client.0.smithi016.stdout:7/180: creat d3/d4/f46 x:0 0 0 2024-04-22T15:49:30.389 INFO:tasks.workunit.client.0.smithi016.stdout:0/147: creat dc/d13/f31 x:0 0 0 2024-04-22T15:49:30.390 INFO:tasks.workunit.client.0.smithi016.stdout:5/149: dread f2 [0,4194304] 0 2024-04-22T15:49:30.391 INFO:tasks.workunit.client.0.smithi016.stdout:0/148: write dc/d13/f17 [2945974,100694] 0 2024-04-22T15:49:30.391 INFO:tasks.workunit.client.0.smithi016.stdout:5/150: dread df/d15/d1a/f20 [0,4194304] 0 2024-04-22T15:49:30.402 INFO:tasks.workunit.client.0.smithi016.stdout:7/181: symlink d3/dc/d1b/d31/l47 0 2024-04-22T15:49:30.403 INFO:tasks.workunit.client.0.smithi016.stdout:7/182: write d3/d4/d9/f2a [3295336,19530] 0 2024-04-22T15:49:30.404 INFO:tasks.workunit.client.0.smithi016.stdout:3/158: creat d7/d17/d18/d32/f37 x:0 0 0 2024-04-22T15:49:30.404 INFO:tasks.workunit.client.0.smithi016.stdout:7/183: chown d3/d14/l2f 17951 1 2024-04-22T15:49:30.405 INFO:tasks.workunit.client.0.smithi016.stdout:0/149: getdents dc/d13/d14/d1d/d30 0 2024-04-22T15:49:30.412 INFO:tasks.workunit.client.0.smithi016.stdout:3/159: read d7/d1b/d21/f28 [413576,78334] 0 2024-04-22T15:49:30.412 INFO:tasks.workunit.client.0.smithi016.stdout:3/160: stat d7/d17/l24 0 2024-04-22T15:49:30.414 INFO:tasks.workunit.client.0.smithi016.stdout:7/184: creat d3/dc/d1b/d31/f48 x:0 0 0 2024-04-22T15:49:30.419 INFO:tasks.workunit.client.0.smithi016.stdout:0/150: dread f4 [0,4194304] 0 2024-04-22T15:49:30.421 INFO:tasks.workunit.client.0.smithi016.stdout:8/170: rmdir d2/db 39 2024-04-22T15:49:30.430 INFO:tasks.workunit.client.0.smithi016.stdout:7/185: dwrite d3/dc/d22/f39 [8388608,4194304] 0 2024-04-22T15:49:30.434 INFO:tasks.workunit.client.0.smithi016.stdout:0/151: mkdir dc/d13/d14/d26/d32 0 2024-04-22T15:49:30.436 INFO:tasks.workunit.client.0.smithi016.stdout:5/151: truncate f2 1166097 0 2024-04-22T15:49:30.437 INFO:tasks.workunit.client.0.smithi016.stdout:7/186: creat d3/dc/d1b/f49 x:0 0 0 2024-04-22T15:49:30.437 INFO:tasks.workunit.client.0.smithi016.stdout:5/152: write df/d15/d1a/f21 [900106,101498] 0 2024-04-22T15:49:30.441 INFO:tasks.workunit.client.0.smithi016.stdout:8/171: fdatasync d2/db/f18 0 2024-04-22T15:49:30.445 INFO:tasks.workunit.client.0.smithi016.stdout:5/153: unlink df/d15/f16 0 2024-04-22T15:49:30.448 INFO:tasks.workunit.client.0.smithi016.stdout:7/187: mkdir d3/d4/d4a 0 2024-04-22T15:49:30.449 INFO:tasks.workunit.client.0.smithi016.stdout:3/161: truncate d7/f8 1825932 0 2024-04-22T15:49:30.450 INFO:tasks.workunit.client.0.smithi016.stdout:3/162: truncate d7/d17/f2e 300184 0 2024-04-22T15:49:30.454 INFO:tasks.workunit.client.0.smithi016.stdout:7/188: fsync d3/d4/d9/f25 0 2024-04-22T15:49:30.455 INFO:tasks.workunit.client.0.smithi016.stdout:7/189: fdatasync d3/f23 0 2024-04-22T15:49:30.456 INFO:tasks.workunit.client.0.smithi016.stdout:5/154: creat df/d22/f23 x:0 0 0 2024-04-22T15:49:30.462 INFO:tasks.workunit.client.0.smithi016.stdout:8/172: rename d2/ld to d2/l2b 0 2024-04-22T15:49:30.477 INFO:tasks.workunit.client.0.smithi016.stdout:8/173: dwrite d2/db/fc [0,4194304] 0 2024-04-22T15:49:30.478 INFO:tasks.workunit.client.0.smithi016.stdout:5/155: readlink df/d15/d1a/l1c 0 2024-04-22T15:49:30.479 INFO:tasks.workunit.client.0.smithi016.stdout:8/174: write d2/f24 [2492700,50468] 0 2024-04-22T15:49:30.480 INFO:tasks.workunit.client.0.smithi016.stdout:8/175: chown d2/f24 0 1 2024-04-22T15:49:30.480 INFO:tasks.workunit.client.0.smithi016.stdout:8/176: chown d2 7 1 2024-04-22T15:49:30.487 INFO:tasks.workunit.client.0.smithi016.stdout:7/190: rename d3/d4/fd to d3/d4/f4b 0 2024-04-22T15:49:30.493 INFO:tasks.workunit.client.0.smithi016.stdout:5/156: rename df/d15/d1a/f21 to df/d22/f24 0 2024-04-22T15:49:30.494 INFO:tasks.workunit.client.0.smithi016.stdout:3/163: link l5 d7/l38 0 2024-04-22T15:49:30.498 INFO:tasks.workunit.client.0.smithi016.stdout:7/191: mkdir d3/d32/d4c 0 2024-04-22T15:49:30.499 INFO:tasks.workunit.client.0.smithi016.stdout:7/192: write d3/dc/f1e [862735,88673] 0 2024-04-22T15:49:30.502 INFO:tasks.workunit.client.0.smithi016.stdout:8/177: rename d2/db/d1c/d2a to d2/d10/d23/d2c 0 2024-04-22T15:49:30.503 INFO:tasks.workunit.client.0.smithi016.stdout:8/178: dread d2/db/f18 [0,4194304] 0 2024-04-22T15:49:30.504 INFO:tasks.workunit.client.0.smithi016.stdout:5/157: creat df/d22/f25 x:0 0 0 2024-04-22T15:49:30.506 INFO:tasks.workunit.client.0.smithi016.stdout:5/158: truncate df/f1f 991035 0 2024-04-22T15:49:30.506 INFO:tasks.workunit.client.0.smithi016.stdout:5/159: chown df/d22/f24 526448799 1 2024-04-22T15:49:30.507 INFO:tasks.workunit.client.0.smithi016.stdout:5/160: readlink l9 0 2024-04-22T15:49:30.508 INFO:tasks.workunit.client.0.smithi016.stdout:5/161: write df/f1f [617269,28557] 0 2024-04-22T15:49:30.512 INFO:tasks.workunit.client.0.smithi016.stdout:7/193: dread d3/f34 [0,4194304] 0 2024-04-22T15:49:30.513 INFO:tasks.workunit.client.0.smithi016.stdout:8/179: mknod d2/db/d25/d28/c2d 0 2024-04-22T15:49:30.519 INFO:tasks.workunit.client.0.smithi016.stdout:7/194: creat d3/dc/d33/f4d x:0 0 0 2024-04-22T15:49:30.519 INFO:tasks.workunit.client.0.smithi016.stdout:8/180: dwrite d2/db/f20 [0,4194304] 0 2024-04-22T15:49:30.524 INFO:tasks.workunit.client.0.smithi016.stdout:7/195: symlink d3/d4/d9/df/l4e 0 2024-04-22T15:49:30.524 INFO:tasks.workunit.client.0.smithi016.stdout:7/196: stat d3/d14/f2c 0 2024-04-22T15:49:30.526 INFO:tasks.workunit.client.0.smithi016.stdout:5/162: link l9 df/d15/d1a/d1d/l26 0 2024-04-22T15:49:30.530 INFO:tasks.workunit.client.0.smithi016.stdout:8/181: rename d2/db/l1f to d2/db/d25/l2e 0 2024-04-22T15:49:30.531 INFO:tasks.workunit.client.0.smithi016.stdout:3/164: link l5 d7/l39 0 2024-04-22T15:49:30.532 INFO:tasks.workunit.client.0.smithi016.stdout:3/165: fdatasync d7/d17/f2e 0 2024-04-22T15:49:30.535 INFO:tasks.workunit.client.0.smithi016.stdout:7/197: dwrite d3/dc/d33/f3e [0,4194304] 0 2024-04-22T15:49:30.535 INFO:tasks.workunit.client.0.smithi016.stdout:2/152: sync 2024-04-22T15:49:30.536 INFO:tasks.workunit.client.0.smithi016.stdout:2/153: dread - d2/f4 zero size 2024-04-22T15:49:30.539 INFO:tasks.workunit.client.0.smithi016.stdout:7/198: stat d3/d4/f46 0 2024-04-22T15:49:30.541 INFO:tasks.workunit.client.0.smithi016.stdout:3/166: rmdir d7/d1b 39 2024-04-22T15:49:30.543 INFO:tasks.workunit.client.0.smithi016.stdout:5/163: creat df/f27 x:0 0 0 2024-04-22T15:49:30.544 INFO:tasks.workunit.client.0.smithi016.stdout:8/182: dwrite d2/f27 [0,4194304] 0 2024-04-22T15:49:30.545 INFO:tasks.workunit.client.0.smithi016.stdout:7/199: rmdir d3/dc/d22 39 2024-04-22T15:49:30.546 INFO:tasks.workunit.client.0.smithi016.stdout:5/164: chown le 15066 1 2024-04-22T15:49:30.551 INFO:tasks.workunit.client.0.smithi016.stdout:2/154: dwrite d2/f3e [0,4194304] 0 2024-04-22T15:49:30.553 INFO:tasks.workunit.client.0.smithi016.stdout:3/167: mknod d7/c3a 0 2024-04-22T15:49:30.556 INFO:tasks.workunit.client.0.smithi016.stdout:8/183: mknod d2/db/d1c/c2f 0 2024-04-22T15:49:30.558 INFO:tasks.workunit.client.0.smithi016.stdout:7/200: dwrite d3/dc/d33/f44 [0,4194304] 0 2024-04-22T15:49:30.562 INFO:tasks.workunit.client.0.smithi016.stdout:5/165: rename df/d15/d1a/d1d to df/d15/d1a/d28 0 2024-04-22T15:49:30.566 INFO:tasks.workunit.client.0.smithi016.stdout:2/155: rename d2/f26 to d2/de/f40 0 2024-04-22T15:49:30.567 INFO:tasks.workunit.client.0.smithi016.stdout:2/156: write d2/f4 [976139,76290] 0 2024-04-22T15:49:30.569 INFO:tasks.workunit.client.0.smithi016.stdout:3/168: creat d7/d17/d26/f3b x:0 0 0 2024-04-22T15:49:30.575 INFO:tasks.workunit.client.0.smithi016.stdout:5/166: symlink df/d22/l29 0 2024-04-22T15:49:30.576 INFO:tasks.workunit.client.0.smithi016.stdout:2/157: creat d2/de/d2a/f41 x:0 0 0 2024-04-22T15:49:30.578 INFO:tasks.workunit.client.0.smithi016.stdout:8/184: mkdir d2/d30 0 2024-04-22T15:49:30.579 INFO:tasks.workunit.client.0.smithi016.stdout:7/201: creat d3/d4/d4a/f4f x:0 0 0 2024-04-22T15:49:30.580 INFO:tasks.workunit.client.0.smithi016.stdout:5/167: creat df/d22/f2a x:0 0 0 2024-04-22T15:49:30.581 INFO:tasks.workunit.client.0.smithi016.stdout:2/158: creat d2/de/d2a/f42 x:0 0 0 2024-04-22T15:49:30.581 INFO:tasks.workunit.client.0.smithi016.stdout:3/169: rmdir d7/d1b 39 2024-04-22T15:49:30.582 INFO:tasks.workunit.client.0.smithi016.stdout:3/170: truncate d7/f22 655170 0 2024-04-22T15:49:30.583 INFO:tasks.workunit.client.0.smithi016.stdout:8/185: fsync d2/db/f16 0 2024-04-22T15:49:30.585 INFO:tasks.workunit.client.0.smithi016.stdout:5/168: dread df/f11 [0,4194304] 0 2024-04-22T15:49:30.585 INFO:tasks.workunit.client.0.smithi016.stdout:7/202: creat d3/d4/d9/f50 x:0 0 0 2024-04-22T15:49:30.593 INFO:tasks.workunit.client.0.smithi016.stdout:3/171: mkdir d7/d17/d18/d32/d3c 0 2024-04-22T15:49:30.596 INFO:tasks.workunit.client.0.smithi016.stdout:5/169: creat df/d15/d1a/d28/f2b x:0 0 0 2024-04-22T15:49:30.600 INFO:tasks.workunit.client.0.smithi016.stdout:7/203: rename d3/dc/c2b to d3/d32/d4c/c51 0 2024-04-22T15:49:30.603 INFO:tasks.workunit.client.0.smithi016.stdout:3/172: dread d7/d1b/d21/f28 [0,4194304] 0 2024-04-22T15:49:30.604 INFO:tasks.workunit.client.0.smithi016.stdout:5/170: creat df/d15/d1a/d28/f2c x:0 0 0 2024-04-22T15:49:30.604 INFO:tasks.workunit.client.0.smithi016.stdout:5/171: truncate df/f27 400475 0 2024-04-22T15:49:30.630 INFO:tasks.workunit.client.0.smithi016.stdout:0/152: sync 2024-04-22T15:49:30.630 INFO:tasks.workunit.client.0.smithi016.stdout:4/156: sync 2024-04-22T15:49:30.630 INFO:tasks.workunit.client.0.smithi016.stdout:6/186: sync 2024-04-22T15:49:30.631 INFO:tasks.workunit.client.0.smithi016.stdout:1/406: sync 2024-04-22T15:49:30.631 INFO:tasks.workunit.client.0.smithi016.stdout:9/221: sync 2024-04-22T15:49:30.631 INFO:tasks.workunit.client.0.smithi016.stdout:4/157: fdatasync d7/f1d 0 2024-04-22T15:49:30.633 INFO:tasks.workunit.client.0.smithi016.stdout:0/153: truncate dc/d13/f31 844076 0 2024-04-22T15:49:30.633 INFO:tasks.workunit.client.0.smithi016.stdout:4/158: stat d7/d24/c2c 0 2024-04-22T15:49:30.644 INFO:tasks.workunit.client.0.smithi016.stdout:7/204: creat d3/d14/f52 x:0 0 0 2024-04-22T15:49:30.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.644+0000 7fd454e0c700 1 -- 172.21.15.16:0/808903036 --> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd450109b20 con 0x7fd43806d8a0 2024-04-22T15:49:30.650 INFO:tasks.workunit.client.0.smithi016.stdout:5/172: unlink cb 0 2024-04-22T15:49:30.653 INFO:tasks.workunit.client.0.smithi016.stdout:9/222: symlink d9/de/l42 0 2024-04-22T15:49:30.655 INFO:tasks.workunit.client.0.smithi016.stdout:1/407: rename d3/d28/f74 to d3/d4/da/f88 0 2024-04-22T15:49:30.658 INFO:tasks.workunit.client.0.smithi016.stdout:4/159: rmdir d7/d21 39 2024-04-22T15:49:30.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.658+0000 7fd43f7fe700 1 -- 172.21.15.16:0/808903036 <== mgr.24395 v2:172.21.15.27:6828/2676414545 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fd450109b20 con 0x7fd43806d8a0 2024-04-22T15:49:30.661 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:49:30.661 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.smithi016 smithi016 *:9093,9094 running (6m) 31s ago 8m 24.8M - 0.25.0 c8568f914cd2 fab40987f1b0 2024-04-22T15:49:30.661 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi016 smithi016 running (8m) 31s ago 8m 7612k - 18.2.1 5be31c24972a f8b627d3f41e 2024-04-22T15:49:30.661 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi027 smithi027 running (7m) 22s ago 7m 7604k - 18.2.1 5be31c24972a f00fd40b4220 2024-04-22T15:49:30.661 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi016 smithi016 running (8m) 31s ago 8m 6656k - 18.2.1 5be31c24972a 6b93db1fa0d9 2024-04-22T15:49:30.661 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi027 smithi027 running (7m) 22s ago 7m 6665k - 18.2.1 5be31c24972a d3bd439f30a0 2024-04-22T15:49:30.662 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.smithi016 smithi016 *:3000 running (6m) 31s ago 8m 85.5M - 9.4.7 954c08fa6188 a85bdbf5073d 2024-04-22T15:49:30.662 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.fkkejm smithi016 running (2m) 31s ago 2m 12.9M - 18.2.1 5be31c24972a 4153ac94cea0 2024-04-22T15:49:30.662 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.pgvphu smithi016 running (2m) 31s ago 2m 62.3M - 18.2.1 5be31c24972a 93dc072068a2 2024-04-22T15:49:30.662 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.dgqyes smithi027 running (2m) 22s ago 2m 15.9M - 18.2.1 5be31c24972a 41a9220430fd 2024-04-22T15:49:30.662 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.iobjds smithi027 running (2m) 22s ago 2m 98.2M - 18.2.1 5be31c24972a c1c8d6770f02 2024-04-22T15:49:30.662 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi016.rnsnnr smithi016 *:9283,8765,8443 running (10m) 31s ago 10m 437M - 18.2.1 5be31c24972a 1cf5a7efea72 2024-04-22T15:49:30.662 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi027.ukdgzs smithi027 *:8443,9283,8765 running (59s) 22s ago 7m 491M - 19.0.0-3223-g5883d494 3d983c26e278 8dfb23a29706 2024-04-22T15:49:30.662 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi016 smithi016 running (10m) 31s ago 10m 53.3M 2048M 18.2.1 5be31c24972a 72af40873cb6 2024-04-22T15:49:30.662 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi027 smithi027 running (7m) 22s ago 7m 40.5M 2048M 18.2.1 5be31c24972a 9f21ce7fa7ef 2024-04-22T15:49:30.662 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi016 smithi016 *:9100 running (8m) 31s ago 8m 16.0M - 1.5.0 0da6a335fe13 1937a0bc1959 2024-04-22T15:49:30.662 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi027 smithi027 *:9100 running (7m) 22s ago 7m 15.7M - 1.5.0 0da6a335fe13 00440c991b3f 2024-04-22T15:49:30.662 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (6m) 31s ago 6m 69.0M 1217M 18.2.1 5be31c24972a 3c6ebc4ea041 2024-04-22T15:49:30.662 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (6m) 31s ago 6m 65.0M 1217M 18.2.1 5be31c24972a 6aa2e6796784 2024-04-22T15:49:30.662 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (5m) 31s ago 5m 65.6M 1217M 18.2.1 5be31c24972a 3d9d310aa9eb 2024-04-22T15:49:30.663 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi027 running (5m) 22s ago 5m 67.5M 2240M 18.2.1 5be31c24972a cdf9b7aaf865 2024-04-22T15:49:30.663 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (4m) 22s ago 4m 61.5M 2240M 18.2.1 5be31c24972a 629397d49c76 2024-04-22T15:49:30.663 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi027 running (4m) 22s ago 4m 64.3M 2240M 18.2.1 5be31c24972a 23adcf675b04 2024-04-22T15:49:30.663 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.smithi016 smithi016 *:9095 starting - - - - 2024-04-22T15:49:30.663 INFO:tasks.workunit.client.0.smithi016.stdout:0/154: rmdir dc/d13/d22 39 2024-04-22T15:49:30.663 INFO:tasks.workunit.client.0.smithi016.stdout:2/159: dwrite f0 [0,4194304] 0 2024-04-22T15:49:30.664 INFO:tasks.workunit.client.0.smithi016.stdout:2/160: write d2/f3e [3058172,55383] 0 2024-04-22T15:49:30.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.662+0000 7fd454e0c700 1 -- 172.21.15.16:0/808903036 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7fd43806d8a0 msgr2=0x7fd43806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:30.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.662+0000 7fd454e0c700 1 --2- 172.21.15.16:0/808903036 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7fd43806d8a0 0x7fd43806fd60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd440009fd0 tx=0x7fd440005940 comp rx=0 tx=0).stop 2024-04-22T15:49:30.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.662+0000 7fd454e0c700 1 -- 172.21.15.16:0/808903036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd450101170 msgr2=0x7fd450105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:30.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.662+0000 7fd454e0c700 1 --2- 172.21.15.16:0/808903036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd450101170 0x7fd450105e90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd44400e920 tx=0x7fd44400ec30 comp rx=0 tx=0).stop 2024-04-22T15:49:30.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.662+0000 7fd454e0c700 1 -- 172.21.15.16:0/808903036 shutdown_connections 2024-04-22T15:49:30.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.662+0000 7fd454e0c700 1 --2- 172.21.15.16:0/808903036 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7fd43806d8a0 0x7fd43806fd60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:30.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.662+0000 7fd454e0c700 1 --2- 172.21.15.16:0/808903036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4500fed00 0x7fd450105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:30.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.662+0000 7fd454e0c700 1 --2- 172.21.15.16:0/808903036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd450101170 0x7fd450105e90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:30.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.662+0000 7fd454e0c700 1 -- 172.21.15.16:0/808903036 >> 172.21.15.16:0/808903036 conn(0x7fd4500fa8c0 msgr2=0x7fd450108400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:30.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.662+0000 7fd454e0c700 1 -- 172.21.15.16:0/808903036 shutdown_connections 2024-04-22T15:49:30.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.663+0000 7fd454e0c700 1 -- 172.21.15.16:0/808903036 wait complete. 2024-04-22T15:49:30.669 INFO:tasks.workunit.client.0.smithi016.stdout:5/173: unlink cc 0 2024-04-22T15:49:30.671 INFO:tasks.workunit.client.0.smithi016.stdout:8/186: dwrite d2/f27 [4194304,4194304] 0 2024-04-22T15:49:30.672 INFO:tasks.workunit.client.0.smithi016.stdout:8/187: fdatasync f1 0 2024-04-22T15:49:30.673 INFO:tasks.workunit.client.0.smithi016.stdout:0/155: dread dc/d13/f17 [4194304,4194304] 0 2024-04-22T15:49:30.674 INFO:tasks.workunit.client.0.smithi016.stdout:6/187: rename d3/da/dd/c30 to d3/da/d12/d13/c3a 0 2024-04-22T15:49:30.680 INFO:tasks.workunit.client.0.smithi016.stdout:5/174: dwrite df/d22/f23 [0,4194304] 0 2024-04-22T15:49:30.682 INFO:tasks.workunit.client.0.smithi016.stdout:4/160: fdatasync d7/d21/d23/f29 0 2024-04-22T15:49:30.694 INFO:tasks.workunit.client.0.smithi016.stdout:7/205: unlink d3/dc/d1b/l35 0 2024-04-22T15:49:30.704 INFO:tasks.workunit.client.0.smithi016.stdout:0/156: write f4 [3984662,20064] 0 2024-04-22T15:49:30.709 INFO:tasks.workunit.client.0.smithi016.stdout:1/408: fsync d3/d4/da/f83 0 2024-04-22T15:49:30.715 INFO:tasks.workunit.client.0.smithi016.stdout:5/175: mknod df/d22/c2d 0 2024-04-22T15:49:30.725 INFO:tasks.workunit.client.0.smithi016.stdout:7/206: creat d3/d32/d4c/f53 x:0 0 0 2024-04-22T15:49:30.726 INFO:tasks.workunit.client.0.smithi016.stdout:4/161: dread d7/f14 [0,4194304] 0 2024-04-22T15:49:30.727 INFO:tasks.workunit.client.0.smithi016.stdout:0/157: truncate dc/f27 8482517 0 2024-04-22T15:49:30.728 INFO:tasks.workunit.client.0.smithi016.stdout:0/158: truncate f4 4042514 0 2024-04-22T15:49:30.728 INFO:tasks.workunit.client.0.smithi016.stdout:8/188: mknod d2/db/d1c/d29/c31 0 2024-04-22T15:49:30.729 INFO:tasks.workunit.client.0.smithi016.stdout:8/189: write f1 [5430790,19503] 0 2024-04-22T15:49:30.732 INFO:tasks.workunit.client.0.smithi016.stdout:1/409: rmdir d3/d4/d15/d38/d63 39 2024-04-22T15:49:30.736 INFO:tasks.workunit.client.0.smithi016.stdout:1/410: dread d3/d2e/f51 [0,4194304] 0 2024-04-22T15:49:30.739 INFO:tasks.workunit.client.0.smithi016.stdout:7/207: creat d3/d32/d4c/f54 x:0 0 0 2024-04-22T15:49:30.748 INFO:tasks.workunit.client.0.smithi016.stdout:0/159: dwrite dc/f27 [4194304,4194304] 0 2024-04-22T15:49:30.751 INFO:tasks.workunit.client.0.smithi016.stdout:5/176: link df/d22/f2a df/d15/d1a/d28/f2e 0 2024-04-22T15:49:30.751 INFO:tasks.workunit.client.0.smithi016.stdout:3/173: write d7/f8 [2645202,73309] 0 2024-04-22T15:49:30.751 INFO:tasks.workunit.client.0.smithi016.stdout:7/208: creat d3/d4/d4a/f55 x:0 0 0 2024-04-22T15:49:30.758 INFO:tasks.workunit.client.0.smithi016.stdout:7/209: dwrite d3/dc/f1e [0,4194304] 0 2024-04-22T15:49:30.761 INFO:tasks.workunit.client.0.smithi016.stdout:1/411: symlink d3/d4/d19/l89 0 2024-04-22T15:49:30.769 INFO:tasks.workunit.client.0.smithi016.stdout:0/160: creat dc/d13/d14/d26/d32/f33 x:0 0 0 2024-04-22T15:49:30.770 INFO:tasks.workunit.client.0.smithi016.stdout:3/174: creat d7/d17/d18/d27/f3d x:0 0 0 2024-04-22T15:49:30.771 INFO:tasks.workunit.client.0.smithi016.stdout:3/175: fdatasync d7/d17/d18/f1f 0 2024-04-22T15:49:30.772 INFO:tasks.workunit.client.0.smithi016.stdout:1/412: rename d3/d2e/f51 to d3/d2e/f8a 0 2024-04-22T15:49:30.774 INFO:tasks.workunit.client.0.smithi016.stdout:5/177: rename df/d15/l17 to df/d15/d1a/d28/l2f 0 2024-04-22T15:49:30.779 INFO:tasks.workunit.client.0.smithi016.stdout:3/176: chown d7/f2c 364286 1 2024-04-22T15:49:30.779 INFO:tasks.workunit.client.0.smithi016.stdout:3/177: write d7/f22 [1010183,104545] 0 2024-04-22T15:49:30.780 INFO:tasks.workunit.client.0.smithi016.stdout:1/413: creat d3/d28/f8b x:0 0 0 2024-04-22T15:49:30.785 INFO:tasks.workunit.client.0.smithi016.stdout:0/161: getdents dc/d13/d14/d1d/d1f 0 2024-04-22T15:49:30.785 INFO:tasks.workunit.client.0.smithi016.stdout:0/162: stat dc/f27 0 2024-04-22T15:49:30.790 INFO:tasks.workunit.client.0.smithi016.stdout:1/414: dread d3/d4/fe [0,4194304] 0 2024-04-22T15:49:30.796 INFO:tasks.workunit.client.0.smithi016.stdout:7/210: rename d3/d14/f52 to d3/d4/f56 0 2024-04-22T15:49:30.800 INFO:tasks.workunit.client.0.smithi016.stdout:3/178: creat d7/d17/f3e x:0 0 0 2024-04-22T15:49:30.804 INFO:tasks.workunit.client.0.smithi016.stdout:1/415: dread d3/f7 [0,4194304] 0 2024-04-22T15:49:30.805 INFO:tasks.workunit.client.0.smithi016.stdout:0/163: link dc/d13/f16 dc/d13/d14/f34 0 2024-04-22T15:49:30.807 INFO:tasks.workunit.client.0.smithi016.stdout:3/179: rename d7/d17/d26/f3b to d7/d17/d18/f3f 0 2024-04-22T15:49:30.808 INFO:tasks.workunit.client.0.smithi016.stdout:3/180: write d7/f12 [422335,48677] 0 2024-04-22T15:49:30.809 INFO:tasks.workunit.client.0.smithi016.stdout:7/211: unlink d3/dc/d22/f38 0 2024-04-22T15:49:30.810 INFO:tasks.workunit.client.0.smithi016.stdout:2/161: write d2/d9/fa [6480830,33425] 0 2024-04-22T15:49:30.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.808+0000 7fac21730700 1 -- 172.21.15.16:0/191964609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac1c101f40 msgr2=0x7fac1c1023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:30.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.808+0000 7fac21730700 1 --2- 172.21.15.16:0/191964609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac1c101f40 0x7fac1c1023a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fac10009910 tx=0x7fac10009c20 comp rx=0 tx=0).stop 2024-04-22T15:49:30.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.808+0000 7fac21730700 1 -- 172.21.15.16:0/191964609 shutdown_connections 2024-04-22T15:49:30.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.808+0000 7fac21730700 1 --2- 172.21.15.16:0/191964609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac1c101f40 0x7fac1c1023a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:30.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.808+0000 7fac21730700 1 --2- 172.21.15.16:0/191964609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac1c0fef60 0x7fac1c0ff380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:30.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.808+0000 7fac21730700 1 -- 172.21.15.16:0/191964609 >> 172.21.15.16:0/191964609 conn(0x7fac1c0fa8c0 msgr2=0x7fac1c0fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:30.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.809+0000 7fac21730700 1 -- 172.21.15.16:0/191964609 shutdown_connections 2024-04-22T15:49:30.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.809+0000 7fac21730700 1 -- 172.21.15.16:0/191964609 wait complete. 2024-04-22T15:49:30.813 INFO:tasks.workunit.client.0.smithi016.stdout:0/164: dread dc/f27 [4194304,4194304] 0 2024-04-22T15:49:30.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.810+0000 7fac21730700 1 Processor -- start 2024-04-22T15:49:30.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.810+0000 7fac21730700 1 -- start start 2024-04-22T15:49:30.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.811+0000 7fac21730700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac1c0fef60 0x7fac1c1018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:30.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.811+0000 7fac21730700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac1c101f40 0x7fac1c0ffef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:30.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.811+0000 7fac21730700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c100430 con 0x7fac1c101f40 2024-04-22T15:49:30.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.811+0000 7fac21730700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c100570 con 0x7fac1c0fef60 2024-04-22T15:49:30.814 INFO:tasks.workunit.client.0.smithi016.stdout:1/416: dwrite d3/d4/d19/d20/f21 [0,4194304] 0 2024-04-22T15:49:30.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.811+0000 7fac1a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac1c101f40 0x7fac1c0ffef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:30.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.811+0000 7fac1affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac1c0fef60 0x7fac1c1018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:30.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.811+0000 7fac1a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac1c101f40 0x7fac1c0ffef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42568/0 (socket says 172.21.15.16:42568) 2024-04-22T15:49:30.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.811+0000 7fac1a7fc700 1 -- 172.21.15.16:0/881798420 learned_addr learned my addr 172.21.15.16:0/881798420 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:49:30.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.811+0000 7fac1affd700 1 -- 172.21.15.16:0/881798420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac1c101f40 msgr2=0x7fac1c0ffef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:30.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.811+0000 7fac1affd700 1 --2- 172.21.15.16:0/881798420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac1c101f40 0x7fac1c0ffef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:30.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.811+0000 7fac1affd700 1 -- 172.21.15.16:0/881798420 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0c009520 con 0x7fac1c0fef60 2024-04-22T15:49:30.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.811+0000 7fac1a7fc700 1 --2- 172.21.15.16:0/881798420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac1c101f40 0x7fac1c0ffef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:49:30.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.812+0000 7fac1affd700 1 --2- 172.21.15.16:0/881798420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac1c0fef60 0x7fac1c1018a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fac0c00e940 tx=0x7fac0c00ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:30.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.813+0000 7fac03fff700 1 -- 172.21.15.16:0/881798420 <== mon.1 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac0c00cb40 con 0x7fac1c0fef60 2024-04-22T15:49:30.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.813+0000 7fac21730700 1 -- 172.21.15.16:0/881798420 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac100095f0 con 0x7fac1c0fef60 2024-04-22T15:49:30.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.813+0000 7fac21730700 1 -- 172.21.15.16:0/881798420 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac1c100c10 con 0x7fac1c0fef60 2024-04-22T15:49:30.817 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:30 smithi016 ceph-mon[26620]: from='client.24441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:49:30.817 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:30 smithi016 ceph-mon[26620]: from='client.24445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:49:30.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.813+0000 7fac03fff700 1 -- 172.21.15.16:0/881798420 <== mon.1 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fac0c00ee90 con 0x7fac1c0fef60 2024-04-22T15:49:30.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.813+0000 7fac03fff700 1 -- 172.21.15.16:0/881798420 <== mon.1 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac0c018650 con 0x7fac1c0fef60 2024-04-22T15:49:30.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.815+0000 7fac21730700 1 -- 172.21.15.16:0/881798420 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac08005320 con 0x7fac1c0fef60 2024-04-22T15:49:30.818 INFO:tasks.workunit.client.0.smithi016.stdout:0/165: write f4 [2941777,28765] 0 2024-04-22T15:49:30.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.815+0000 7fac03fff700 1 -- 172.21.15.16:0/881798420 <== mon.1 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) v1 ==== 91557+0+0 (secure 0 0 0) 0x7fac0c018840 con 0x7fac1c0fef60 2024-04-22T15:49:30.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.816+0000 7fac03fff700 1 --2- 172.21.15.16:0/881798420 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7fac0406d8e0 0x7fac0406fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:30.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.816+0000 7fac03fff700 1 -- 172.21.15.16:0/881798420 <== mon.1 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fac0c014070 con 0x7fac1c0fef60 2024-04-22T15:49:30.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.816+0000 7fac1a7fc700 1 --2- 172.21.15.16:0/881798420 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7fac0406d8e0 0x7fac0406fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:30.821 INFO:tasks.workunit.client.0.smithi016.stdout:6/188: truncate d3/da/dd/f23 52388 0 2024-04-22T15:49:30.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.818+0000 7fac1a7fc700 1 --2- 172.21.15.16:0/881798420 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7fac0406d8e0 0x7fac0406fda0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fac1000b6f0 tx=0x7fac10009f90 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:30.823 INFO:tasks.workunit.client.0.smithi016.stdout:3/181: dwrite d7/f12 [0,4194304] 0 2024-04-22T15:49:30.825 INFO:tasks.workunit.client.0.smithi016.stdout:2/162: dread d2/f3e [0,4194304] 0 2024-04-22T15:49:30.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:30.824+0000 7fac03fff700 1 -- 172.21.15.16:0/881798420 <== mon.1 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7fac0c05a2f0 con 0x7fac1c0fef60 2024-04-22T15:49:30.828 INFO:tasks.workunit.client.0.smithi016.stdout:1/417: mkdir d3/d4/d15/d8c 0 2024-04-22T15:49:30.830 INFO:tasks.workunit.client.0.smithi016.stdout:6/189: symlink d3/da/l3b 0 2024-04-22T15:49:30.831 INFO:tasks.workunit.client.0.smithi016.stdout:2/163: dread - d2/de/d2a/f41 zero size 2024-04-22T15:49:30.833 INFO:tasks.workunit.client.0.smithi016.stdout:1/418: write d3/d4/d19/d20/d2b/f4a [3626380,125920] 0 2024-04-22T15:49:30.833 INFO:tasks.workunit.client.0.smithi016.stdout:3/182: symlink d7/d17/d18/d32/l40 0 2024-04-22T15:49:30.835 INFO:tasks.workunit.client.0.smithi016.stdout:0/166: dwrite f4 [0,4194304] 0 2024-04-22T15:49:30.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:30 smithi027 ceph-mon[34922]: from='client.24441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:49:30.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:30 smithi027 ceph-mon[34922]: from='client.24445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:49:30.838 INFO:tasks.workunit.client.0.smithi016.stdout:4/162: dwrite d7/f11 [0,4194304] 0 2024-04-22T15:49:30.841 INFO:tasks.workunit.client.0.smithi016.stdout:2/164: dwrite d2/d9/fa [4194304,4194304] 0 2024-04-22T15:49:30.841 INFO:tasks.workunit.client.0.smithi016.stdout:8/190: truncate d2/f22 114020 0 2024-04-22T15:49:30.844 INFO:tasks.workunit.client.0.smithi016.stdout:6/190: chown d3/da/dd/f23 1629 1 2024-04-22T15:49:30.844 INFO:tasks.workunit.client.0.smithi016.stdout:6/191: fsync d3/f24 0 2024-04-22T15:49:30.846 INFO:tasks.workunit.client.0.smithi016.stdout:6/192: readlink d3/l7 0 2024-04-22T15:49:30.847 INFO:tasks.workunit.client.0.smithi016.stdout:9/223: sync 2024-04-22T15:49:30.849 INFO:tasks.workunit.client.0.smithi016.stdout:9/224: chown d9/de/d23/d24/d25/f2c 216 1 2024-04-22T15:49:30.850 INFO:tasks.workunit.client.0.smithi016.stdout:4/163: rmdir d7/d21/d23 39 2024-04-22T15:49:30.854 INFO:tasks.workunit.client.0.smithi016.stdout:1/419: symlink d3/d4/d15/d38/d63/d67/l8d 0 2024-04-22T15:49:30.856 INFO:tasks.workunit.client.0.smithi016.stdout:9/225: dwrite d9/f1b [0,4194304] 0 2024-04-22T15:49:30.860 INFO:tasks.workunit.client.0.smithi016.stdout:0/167: mknod dc/d13/d28/c35 0 2024-04-22T15:49:30.861 INFO:tasks.workunit.client.0.smithi016.stdout:0/168: write f4 [41820,81242] 0 2024-04-22T15:49:30.862 INFO:tasks.workunit.client.0.smithi016.stdout:0/169: chown dc/d13/l1a 0 1 2024-04-22T15:49:30.872 INFO:tasks.workunit.client.0.smithi016.stdout:1/420: symlink d3/d4/d15/l8e 0 2024-04-22T15:49:30.882 INFO:tasks.workunit.client.0.smithi016.stdout:0/170: creat dc/d13/d14/d26/f36 x:0 0 0 2024-04-22T15:49:30.882 INFO:tasks.workunit.client.0.smithi016.stdout:0/171: chown dc/l10 98 1 2024-04-22T15:49:30.883 INFO:tasks.workunit.client.0.smithi016.stdout:4/164: mkdir d7/d21/d23/d2f 0 2024-04-22T15:49:30.883 INFO:tasks.workunit.client.0.smithi016.stdout:0/172: stat dc/d13/d14/d1d/d1f/l29 0 2024-04-22T15:49:30.886 INFO:tasks.workunit.client.0.smithi016.stdout:4/165: dread d7/f14 [0,4194304] 0 2024-04-22T15:49:30.888 INFO:tasks.workunit.client.0.smithi016.stdout:8/191: rename d2/db/d1c/d29 to d2/d32 0 2024-04-22T15:49:30.892 INFO:tasks.workunit.client.0.smithi016.stdout:0/173: rmdir dc/d13/d14/d26/d32 39 2024-04-22T15:49:30.896 INFO:tasks.workunit.client.0.smithi016.stdout:4/166: dread d7/f1d [0,4194304] 0 2024-04-22T15:49:30.897 INFO:tasks.workunit.client.0.smithi016.stdout:2/165: getdents d2/d9 0 2024-04-22T15:49:30.898 INFO:tasks.workunit.client.0.smithi016.stdout:9/226: rename d9/d20/f3e to d9/de/f43 0 2024-04-22T15:49:30.900 INFO:tasks.workunit.client.0.smithi016.stdout:8/192: symlink d2/db/d25/l33 0 2024-04-22T15:49:30.903 INFO:tasks.workunit.client.0.smithi016.stdout:2/166: dwrite d2/f7 [0,4194304] 0 2024-04-22T15:49:30.906 INFO:tasks.workunit.client.0.smithi016.stdout:9/227: unlink l8 0 2024-04-22T15:49:30.908 INFO:tasks.workunit.client.0.smithi016.stdout:0/174: dwrite dc/d13/d22/f2e [0,4194304] 0 2024-04-22T15:49:30.911 INFO:tasks.workunit.client.0.smithi016.stdout:1/421: getdents d3/d4 0 2024-04-22T15:49:30.912 INFO:tasks.workunit.client.0.smithi016.stdout:4/167: symlink d7/d21/l30 0 2024-04-22T15:49:30.914 INFO:tasks.workunit.client.0.smithi016.stdout:2/167: symlink d2/d9/d2f/l43 0 2024-04-22T15:49:30.923 INFO:tasks.workunit.client.0.smithi016.stdout:7/212: dwrite d3/dc/d33/f44 [4194304,4194304] 0 2024-04-22T15:49:30.924 INFO:tasks.workunit.client.0.smithi016.stdout:2/168: dread d2/f4 [0,4194304] 0 2024-04-22T15:49:30.926 INFO:tasks.workunit.client.0.smithi016.stdout:9/228: link d9/d37/f3c d9/de/d26/f44 0 2024-04-22T15:49:30.927 INFO:tasks.workunit.client.0.smithi016.stdout:4/168: dread d7/f11 [0,4194304] 0 2024-04-22T15:49:30.931 INFO:tasks.workunit.client.0.smithi016.stdout:2/169: write d2/d2c/f3c [1321669,69340] 0 2024-04-22T15:49:30.931 INFO:tasks.workunit.client.0.smithi016.stdout:1/422: dwrite d3/d4/d37/f41 [0,4194304] 0 2024-04-22T15:49:30.931 INFO:tasks.workunit.client.0.smithi016.stdout:1/423: fsync d3/f4f 0 2024-04-22T15:49:30.934 INFO:tasks.workunit.client.0.smithi016.stdout:0/175: creat dc/d13/f37 x:0 0 0 2024-04-22T15:49:30.939 INFO:tasks.workunit.client.0.smithi016.stdout:9/229: mknod d9/de/d26/c45 0 2024-04-22T15:49:30.940 INFO:tasks.workunit.client.0.smithi016.stdout:9/230: dread - d9/de/d26/f44 zero size 2024-04-22T15:49:30.941 INFO:tasks.workunit.client.0.smithi016.stdout:4/169: chown d7/l25 3 1 2024-04-22T15:49:30.942 INFO:tasks.workunit.client.0.smithi016.stdout:2/170: readlink d2/de/l28 0 2024-04-22T15:49:30.943 INFO:tasks.workunit.client.0.smithi016.stdout:2/171: write d2/f7 [4057717,109057] 0 2024-04-22T15:49:30.946 INFO:tasks.workunit.client.0.smithi016.stdout:1/424: symlink d3/d4/d19/d20/d2b/l8f 0 2024-04-22T15:49:30.950 INFO:tasks.workunit.client.0.smithi016.stdout:4/170: truncate d7/f10 3179347 0 2024-04-22T15:49:30.954 INFO:tasks.workunit.client.0.smithi016.stdout:0/176: symlink dc/d13/d14/d1d/d30/l38 0 2024-04-22T15:49:30.955 INFO:tasks.workunit.client.0.smithi016.stdout:3/183: truncate d7/d17/d18/f23 3013602 0 2024-04-22T15:49:30.956 INFO:tasks.workunit.client.0.smithi016.stdout:0/177: chown dc/d13/d22 13004 1 2024-04-22T15:49:30.960 INFO:tasks.workunit.client.0.smithi016.stdout:2/172: dwrite d2/d23/f3b [0,4194304] 0 2024-04-22T15:49:30.961 INFO:tasks.workunit.client.0.smithi016.stdout:7/213: dwrite d3/d4/d9/fb [8388608,4194304] 0 2024-04-22T15:49:30.962 INFO:tasks.workunit.client.0.smithi016.stdout:3/184: truncate d7/f14 326067 0 2024-04-22T15:49:30.963 INFO:tasks.workunit.client.0.smithi016.stdout:1/425: dwrite d3/d4/f24 [0,4194304] 0 2024-04-22T15:49:30.963 INFO:tasks.workunit.client.0.smithi016.stdout:3/185: readlink d7/d17/d18/l19 0 2024-04-22T15:49:30.965 INFO:tasks.workunit.client.0.smithi016.stdout:4/171: mknod d7/d21/c31 0 2024-04-22T15:49:30.965 INFO:tasks.workunit.client.0.smithi016.stdout:1/426: write d3/d4/d3a/d6b/d77/f81 [841316,102204] 0 2024-04-22T15:49:30.967 INFO:tasks.workunit.client.0.smithi016.stdout:4/172: stat d7/f14 0 2024-04-22T15:49:30.968 INFO:tasks.workunit.client.0.smithi016.stdout:0/178: creat dc/d13/d14/d1d/d1f/f39 x:0 0 0 2024-04-22T15:49:30.970 INFO:tasks.workunit.client.0.smithi016.stdout:2/173: rmdir d2/de/d2a 39 2024-04-22T15:49:30.979 INFO:tasks.workunit.client.0.smithi016.stdout:7/214: rename d3/d32/d4c/f53 to d3/dc/d33/f57 0 2024-04-22T15:49:30.979 INFO:tasks.workunit.client.0.smithi016.stdout:2/174: dwrite d2/d23/d27/f2e [0,4194304] 0 2024-04-22T15:49:30.983 INFO:tasks.workunit.client.0.smithi016.stdout:5/178: dwrite df/d15/d1a/f20 [0,4194304] 0 2024-04-22T15:49:30.988 INFO:tasks.workunit.client.0.smithi016.stdout:3/186: stat d7/f2c 0 2024-04-22T15:49:30.991 INFO:tasks.workunit.client.0.smithi016.stdout:8/193: write d2/db/f16 [150439,88240] 0 2024-04-22T15:49:30.995 INFO:tasks.workunit.client.0.smithi016.stdout:1/427: mkdir d3/d4/d15/d8c/d90 0 2024-04-22T15:49:30.998 INFO:tasks.workunit.client.0.smithi016.stdout:1/428: write d3/f8 [5346735,73168] 0 2024-04-22T15:49:30.998 INFO:tasks.workunit.client.0.smithi016.stdout:7/215: mknod d3/d4/d9/c58 0 2024-04-22T15:49:31.000 INFO:tasks.workunit.client.0.smithi016.stdout:3/187: dwrite d7/d17/f3e [0,4194304] 0 2024-04-22T15:49:31.000 INFO:tasks.workunit.client.0.smithi016.stdout:8/194: rename d2/db/d25 to d2/d10/d34 0 2024-04-22T15:49:31.004 INFO:tasks.workunit.client.0.smithi016.stdout:6/193: dwrite d3/da/dd/f23 [0,4194304] 0 2024-04-22T15:49:31.007 INFO:tasks.workunit.client.0.smithi016.stdout:4/173: getdents d7 0 2024-04-22T15:49:31.011 INFO:tasks.workunit.client.0.smithi016.stdout:7/216: chown d3/d4/d9/fb 13229412 1 2024-04-22T15:49:31.013 INFO:tasks.workunit.client.0.smithi016.stdout:8/195: dread d2/f24 [0,4194304] 0 2024-04-22T15:49:31.015 INFO:tasks.workunit.client.0.smithi016.stdout:1/429: dread d3/d4/d37/f40 [0,4194304] 0 2024-04-22T15:49:31.016 INFO:tasks.workunit.client.0.smithi016.stdout:4/174: dread d7/f11 [0,4194304] 0 2024-04-22T15:49:31.017 INFO:tasks.workunit.client.0.smithi016.stdout:2/175: dread d2/d2c/f3c [0,4194304] 0 2024-04-22T15:49:31.017 INFO:tasks.workunit.client.0.smithi016.stdout:1/430: read d3/d4/d19/f66 [647408,6918] 0 2024-04-22T15:49:31.018 INFO:tasks.workunit.client.0.smithi016.stdout:6/194: dread d3/da/d12/ff [0,4194304] 0 2024-04-22T15:49:31.023 INFO:tasks.workunit.client.0.smithi016.stdout:1/431: read d3/d4/d19/d20/d2b/f4a [1938851,218] 0 2024-04-22T15:49:31.024 INFO:tasks.workunit.client.0.smithi016.stdout:8/196: creat d2/d32/f35 x:0 0 0 2024-04-22T15:49:31.025 INFO:tasks.workunit.client.0.smithi016.stdout:7/217: dwrite d3/d32/f3d [4194304,4194304] 0 2024-04-22T15:49:31.027 INFO:tasks.workunit.client.0.smithi016.stdout:8/197: write d2/db/f20 [3428905,109358] 0 2024-04-22T15:49:31.027 INFO:tasks.workunit.client.0.smithi016.stdout:9/231: truncate d9/de/f18 179422 0 2024-04-22T15:49:31.029 INFO:tasks.workunit.client.0.smithi016.stdout:4/175: unlink f2 0 2024-04-22T15:49:31.032 INFO:tasks.workunit.client.0.smithi016.stdout:8/198: dread d2/db/f20 [0,4194304] 0 2024-04-22T15:49:31.033 INFO:tasks.workunit.client.0.smithi016.stdout:6/195: chown d3/da/dd/f29 8039957 1 2024-04-22T15:49:31.034 INFO:tasks.workunit.client.0.smithi016.stdout:6/196: read - d3/f24 zero size 2024-04-22T15:49:31.034 INFO:tasks.workunit.client.0.smithi016.stdout:8/199: readlink d2/d10/d34/l33 0 2024-04-22T15:49:31.035 INFO:tasks.workunit.client.0.smithi016.stdout:2/176: rename d2/d1a/c1b to d2/d9/c44 0 2024-04-22T15:49:31.036 INFO:tasks.workunit.client.0.smithi016.stdout:2/177: readlink d2/de/l28 0 2024-04-22T15:49:31.038 INFO:tasks.workunit.client.0.smithi016.stdout:8/200: dread d2/f27 [0,4194304] 0 2024-04-22T15:49:31.039 INFO:tasks.workunit.client.0.smithi016.stdout:8/201: read d2/f27 [4846683,122970] 0 2024-04-22T15:49:31.039 INFO:tasks.workunit.client.0.smithi016.stdout:1/432: rmdir d3/d4/d37 39 2024-04-22T15:49:31.040 INFO:tasks.workunit.client.0.smithi016.stdout:7/218: write d3/d4/f4b [3584945,109627] 0 2024-04-22T15:49:31.043 INFO:tasks.workunit.client.0.smithi016.stdout:4/176: dread d7/d24/f26 [0,4194304] 0 2024-04-22T15:49:31.047 INFO:tasks.workunit.client.0.smithi016.stdout:4/177: dread f5 [0,4194304] 0 2024-04-22T15:49:31.055 INFO:tasks.workunit.client.0.smithi016.stdout:8/202: readlink d2/l2b 0 2024-04-22T15:49:31.056 INFO:tasks.workunit.client.0.smithi016.stdout:7/219: chown d3/d4/d9/f26 147227 1 2024-04-22T15:49:31.063 INFO:tasks.workunit.client.0.smithi016.stdout:2/178: symlink d2/de/l45 0 2024-04-22T15:49:31.066 INFO:tasks.workunit.client.0.smithi016.stdout:8/203: rename d2/d10/l1a to d2/d32/l36 0 2024-04-22T15:49:31.068 INFO:tasks.workunit.client.0.smithi016.stdout:2/179: dread d2/d9/fa [4194304,4194304] 0 2024-04-22T15:49:31.072 INFO:tasks.workunit.client.0.smithi016.stdout:7/220: fsync d3/d4/f1f 0 2024-04-22T15:49:31.078 INFO:tasks.workunit.client.0.smithi016.stdout:7/221: dwrite d3/dc/d1b/d31/f48 [0,4194304] 0 2024-04-22T15:49:31.080 INFO:tasks.workunit.client.0.smithi016.stdout:1/433: mkdir d3/d4/d3a/d91 0 2024-04-22T15:49:31.083 INFO:tasks.workunit.client.0.smithi016.stdout:2/180: mkdir d2/d1a/d46 0 2024-04-22T15:49:31.086 INFO:tasks.workunit.client.0.smithi016.stdout:8/204: dwrite d2/f27 [0,4194304] 0 2024-04-22T15:49:31.090 INFO:tasks.workunit.client.0.smithi016.stdout:9/232: truncate d9/de/f16 1123508 0 2024-04-22T15:49:31.090 INFO:tasks.workunit.client.0.smithi016.stdout:9/233: stat d9/de/d23/d24/d25/f3d 0 2024-04-22T15:49:31.092 INFO:tasks.workunit.client.0.smithi016.stdout:7/222: unlink d3/dc/d33/f57 0 2024-04-22T15:49:31.092 INFO:tasks.workunit.client.0.smithi016.stdout:9/234: chown d9/de/d23/d24/d25/f41 833 1 2024-04-22T15:49:31.093 INFO:tasks.workunit.client.0.smithi016.stdout:9/235: chown d9/de 419677 1 2024-04-22T15:49:31.095 INFO:tasks.workunit.client.0.smithi016.stdout:7/223: truncate d3/dc/d33/f4d 419947 0 2024-04-22T15:49:31.095 INFO:tasks.workunit.client.0.smithi016.stdout:2/181: mkdir d2/d23/d47 0 2024-04-22T15:49:31.097 INFO:tasks.workunit.client.0.smithi016.stdout:7/224: read - d3/d4/f56 zero size 2024-04-22T15:49:31.100 INFO:tasks.workunit.client.0.smithi016.stdout:8/205: mkdir d2/d37 0 2024-04-22T15:49:31.102 INFO:tasks.workunit.client.0.smithi016.stdout:6/197: dwrite d3/f6 [0,4194304] 0 2024-04-22T15:49:31.104 INFO:tasks.workunit.client.0.smithi016.stdout:1/434: link d3/c27 d3/d4/d15/d38/d63/c92 0 2024-04-22T15:49:31.105 INFO:tasks.workunit.client.0.smithi016.stdout:2/182: mknod d2/d1a/d46/c48 0 2024-04-22T15:49:31.106 INFO:tasks.workunit.client.0.smithi016.stdout:1/435: chown d3/d4/d15/l7d 33 1 2024-04-22T15:49:31.106 INFO:tasks.workunit.client.0.smithi016.stdout:3/188: truncate d7/d17/d18/f23 3205754 0 2024-04-22T15:49:31.107 INFO:tasks.workunit.client.0.smithi016.stdout:1/436: chown d3/d4/d3a/c5c 0 1 2024-04-22T15:49:31.107 INFO:tasks.workunit.client.0.smithi016.stdout:8/206: creat d2/d10/d34/d28/f38 x:0 0 0 2024-04-22T15:49:31.109 INFO:tasks.workunit.client.0.smithi016.stdout:7/225: mkdir d3/d4/d59 0 2024-04-22T15:49:31.111 INFO:tasks.workunit.client.0.smithi016.stdout:9/236: dread d9/de/d23/f34 [0,4194304] 0 2024-04-22T15:49:31.118 INFO:tasks.workunit.client.0.smithi016.stdout:8/207: creat d2/d10/f39 x:0 0 0 2024-04-22T15:49:31.120 INFO:tasks.workunit.client.0.smithi016.stdout:7/226: readlink d3/d4/d9/df/l1d 0 2024-04-22T15:49:31.121 INFO:tasks.workunit.client.0.smithi016.stdout:3/189: rename d7/f2c to d7/d17/d18/f41 0 2024-04-22T15:49:31.121 INFO:tasks.workunit.client.0.smithi016.stdout:8/208: write d2/db/f18 [701731,126878] 0 2024-04-22T15:49:31.121 INFO:tasks.workunit.client.0.smithi016.stdout:9/237: dwrite d9/de/f31 [0,4194304] 0 2024-04-22T15:49:31.124 INFO:tasks.workunit.client.0.smithi016.stdout:1/437: write d3/d4/da/f88 [696119,99746] 0 2024-04-22T15:49:31.131 INFO:tasks.workunit.client.0.smithi016.stdout:3/190: mkdir d7/d17/d18/d32/d42 0 2024-04-22T15:49:31.133 INFO:tasks.workunit.client.0.smithi016.stdout:1/438: truncate d3/d4/d37/f72 250829 0 2024-04-22T15:49:31.135 INFO:tasks.workunit.client.0.smithi016.stdout:8/209: creat d2/d30/f3a x:0 0 0 2024-04-22T15:49:31.137 INFO:tasks.workunit.client.0.smithi016.stdout:3/191: unlink d7/f2f 0 2024-04-22T15:49:31.140 INFO:tasks.workunit.client.0.smithi016.stdout:2/183: dwrite d2/f8 [0,4194304] 0 2024-04-22T15:49:31.141 INFO:tasks.workunit.client.0.smithi016.stdout:8/210: mknod d2/d30/c3b 0 2024-04-22T15:49:31.146 INFO:tasks.workunit.client.0.smithi016.stdout:7/227: getdents d3/d32 0 2024-04-22T15:49:31.148 INFO:tasks.workunit.client.0.smithi016.stdout:2/184: write d2/de/d2a/f42 [751666,119869] 0 2024-04-22T15:49:31.150 INFO:tasks.workunit.client.0.smithi016.stdout:7/228: unlink d3/d4/f36 0 2024-04-22T15:49:31.151 INFO:tasks.workunit.client.0.smithi016.stdout:7/229: readlink d3/l12 0 2024-04-22T15:49:31.153 INFO:tasks.workunit.client.0.smithi016.stdout:2/185: creat d2/de/d2a/f49 x:0 0 0 2024-04-22T15:49:31.154 INFO:tasks.workunit.client.0.smithi016.stdout:2/186: write f0 [3760621,80376] 0 2024-04-22T15:49:31.154 INFO:tasks.workunit.client.0.smithi016.stdout:3/192: rename d7/c13 to d7/d17/c43 0 2024-04-22T15:49:31.155 INFO:tasks.workunit.client.0.smithi016.stdout:2/187: chown d2/l21 1602 1 2024-04-22T15:49:31.157 INFO:tasks.workunit.client.0.smithi016.stdout:2/188: fdatasync d2/d23/d27/f31 0 2024-04-22T15:49:31.159 INFO:tasks.workunit.client.0.smithi016.stdout:3/193: mkdir d7/d17/d18/d32/d44 0 2024-04-22T15:49:31.161 INFO:tasks.workunit.client.0.smithi016.stdout:1/439: dread d3/d4/d19/d20/f48 [0,4194304] 0 2024-04-22T15:49:31.165 INFO:tasks.workunit.client.0.smithi016.stdout:3/194: dwrite d7/d17/f2e [0,4194304] 0 2024-04-22T15:49:31.165 INFO:tasks.workunit.client.0.smithi016.stdout:1/440: mknod d3/d4/d19/d20/d2b/c93 0 2024-04-22T15:49:31.166 INFO:tasks.workunit.client.0.smithi016.stdout:3/195: chown d7/f8 4904188 1 2024-04-22T15:49:31.169 INFO:tasks.workunit.client.0.smithi016.stdout:1/441: creat d3/d4/d3a/d6b/f94 x:0 0 0 2024-04-22T15:49:31.172 INFO:tasks.workunit.client.0.smithi016.stdout:7/230: rename d3/d4 to d3/dc/d1b/d5a 0 2024-04-22T15:49:31.172 INFO:tasks.workunit.client.0.smithi016.stdout:7/231: stat d3/dc/d33/f3e 0 2024-04-22T15:49:31.174 INFO:tasks.workunit.client.0.smithi016.stdout:1/442: mkdir d3/d4/d19/d20/d95 0 2024-04-22T15:49:31.178 INFO:tasks.workunit.client.0.smithi016.stdout:1/443: chown d3/d4/l13 30115 1 2024-04-22T15:49:31.180 INFO:tasks.workunit.client.0.smithi016.stdout:3/196: getdents d7/d17 0 2024-04-22T15:49:31.181 INFO:tasks.workunit.client.0.smithi016.stdout:2/189: truncate d2/f11 8293586 0 2024-04-22T15:49:31.189 INFO:tasks.workunit.client.0.smithi016.stdout:7/232: creat d3/dc/d1b/d5a/f5b x:0 0 0 2024-04-22T15:49:31.192 INFO:tasks.workunit.client.0.smithi016.stdout:2/190: creat d2/d23/d47/f4a x:0 0 0 2024-04-22T15:49:31.199 INFO:tasks.workunit.client.0.smithi016.stdout:2/191: dwrite d2/f5 [0,4194304] 0 2024-04-22T15:49:31.206 INFO:tasks.workunit.client.0.smithi016.stdout:3/197: dwrite d7/d17/d18/f41 [0,4194304] 0 2024-04-22T15:49:31.207 INFO:tasks.workunit.client.0.smithi016.stdout:1/444: dwrite d3/d4/f1e [12582912,4194304] 0 2024-04-22T15:49:31.208 INFO:tasks.workunit.client.0.smithi016.stdout:3/198: write d7/f12 [1233853,25113] 0 2024-04-22T15:49:31.212 INFO:tasks.workunit.client.0.smithi016.stdout:1/445: write d3/d4/d3a/d6b/d77/f81 [2152957,84425] 0 2024-04-22T15:49:31.213 INFO:tasks.workunit.client.0.smithi016.stdout:1/446: chown d3/d4/d3a/d6b/f94 15 1 2024-04-22T15:49:31.213 INFO:tasks.workunit.client.0.smithi016.stdout:2/192: write d2/d23/d27/f32 [1517596,71117] 0 2024-04-22T15:49:31.218 INFO:tasks.workunit.client.0.smithi016.stdout:1/447: mkdir d3/d4/d3a/d96 0 2024-04-22T15:49:31.220 INFO:tasks.workunit.client.0.smithi016.stdout:3/199: unlink d7/l38 0 2024-04-22T15:49:31.221 INFO:tasks.workunit.client.0.smithi016.stdout:2/193: creat d2/d23/d27/d34/f4b x:0 0 0 2024-04-22T15:49:31.224 INFO:tasks.workunit.client.0.smithi016.stdout:1/448: mkdir d3/d84/d97 0 2024-04-22T15:49:31.227 INFO:tasks.workunit.client.0.smithi016.stdout:1/449: creat d3/d28/f98 x:0 0 0 2024-04-22T15:49:31.228 INFO:tasks.workunit.client.0.smithi016.stdout:3/200: dread d7/d17/f3e [0,4194304] 0 2024-04-22T15:49:31.230 INFO:tasks.workunit.client.0.smithi016.stdout:3/201: write d7/d17/d18/f41 [1762896,86610] 0 2024-04-22T15:49:31.230 INFO:tasks.workunit.client.0.smithi016.stdout:2/194: dwrite d2/d1a/f20 [0,4194304] 0 2024-04-22T15:49:31.231 INFO:tasks.workunit.client.0.smithi016.stdout:3/202: readlink d7/d1b/l1d 0 2024-04-22T15:49:31.239 INFO:tasks.workunit.client.0.smithi016.stdout:2/195: rename d2/d9/d2f to d2/d1a/d4c 0 2024-04-22T15:49:31.240 INFO:tasks.workunit.client.0.smithi016.stdout:3/203: dwrite d7/f15 [0,4194304] 0 2024-04-22T15:49:31.243 INFO:tasks.workunit.client.0.smithi016.stdout:2/196: mkdir d2/d4d 0 2024-04-22T15:49:31.243 INFO:tasks.workunit.client.0.smithi016.stdout:1/450: write d3/d4/d19/d20/f48 [4386855,126877] 0 2024-04-22T15:49:31.246 INFO:tasks.workunit.client.0.smithi016.stdout:3/204: dwrite d7/d17/d26/f33 [0,4194304] 0 2024-04-22T15:49:31.247 INFO:tasks.workunit.client.0.smithi016.stdout:3/205: read - d7/d17/d18/f2a zero size 2024-04-22T15:49:31.252 INFO:tasks.workunit.client.0.smithi016.stdout:3/206: mknod d7/c45 0 2024-04-22T15:49:31.254 INFO:tasks.workunit.client.0.smithi016.stdout:1/451: getdents d3/d28 0 2024-04-22T15:49:31.255 INFO:tasks.workunit.client.0.smithi016.stdout:3/207: symlink d7/l46 0 2024-04-22T15:49:31.260 INFO:tasks.workunit.client.0.smithi016.stdout:3/208: rename d7/d17/d26 to d7/d17/d18/d32/d3c/d47 0 2024-04-22T15:49:31.272 INFO:tasks.workunit.client.0.smithi016.stdout:3/209: creat d7/d17/d18/d32/d3c/d47/f48 x:0 0 0 2024-04-22T15:49:31.273 INFO:tasks.workunit.client.0.smithi016.stdout:1/452: dwrite d3/d4/da/f83 [0,4194304] 0 2024-04-22T15:49:31.273 INFO:tasks.workunit.client.0.smithi016.stdout:3/210: stat d7/d17/d18/f2a 0 2024-04-22T15:49:31.278 INFO:tasks.workunit.client.0.smithi016.stdout:3/211: mknod d7/c49 0 2024-04-22T15:49:31.280 INFO:tasks.workunit.client.0.smithi016.stdout:1/453: link d3/f8 d3/d4/d3a/d6b/d77/f99 0 2024-04-22T15:49:31.282 INFO:tasks.workunit.client.0.smithi016.stdout:3/212: mknod d7/c4a 0 2024-04-22T15:49:31.286 INFO:tasks.workunit.client.0.smithi016.stdout:3/213: dread d7/f14 [0,4194304] 0 2024-04-22T15:49:31.286 INFO:tasks.workunit.client.0.smithi016.stdout:1/454: rename d3/d4/d45/l70 to d3/d2e/d80/d82/l9a 0 2024-04-22T15:49:31.287 INFO:tasks.workunit.client.0.smithi016.stdout:3/214: fdatasync d7/d17/d18/d32/f37 0 2024-04-22T15:49:31.288 INFO:tasks.workunit.client.0.smithi016.stdout:3/215: chown d7/d17/d18/d32/d42 87277880 1 2024-04-22T15:49:31.291 INFO:tasks.workunit.client.0.smithi016.stdout:3/216: creat d7/d1b/f4b x:0 0 0 2024-04-22T15:49:31.295 INFO:tasks.workunit.client.0.smithi016.stdout:1/455: truncate d3/f8 304804 0 2024-04-22T15:49:31.297 INFO:tasks.workunit.client.0.smithi016.stdout:1/456: chown d3/d4/l6c 952 1 2024-04-22T15:49:31.298 INFO:tasks.workunit.client.0.smithi016.stdout:1/457: fsync d3/d4/da/f83 0 2024-04-22T15:49:31.301 INFO:tasks.workunit.client.0.smithi016.stdout:3/217: dread f1 [0,4194304] 0 2024-04-22T15:49:31.301 INFO:tasks.workunit.client.0.smithi016.stdout:1/458: symlink d3/d4/d3a/d6b/l9b 0 2024-04-22T15:49:31.301 INFO:tasks.workunit.client.0.smithi016.stdout:1/459: stat d3/d4/da 0 2024-04-22T15:49:31.302 INFO:tasks.workunit.client.0.smithi016.stdout:3/218: fsync d7/f15 0 2024-04-22T15:49:31.302 INFO:tasks.workunit.client.0.smithi016.stdout:3/219: chown d7/c3a 1465 1 2024-04-22T15:49:31.303 INFO:tasks.workunit.client.0.smithi016.stdout:3/220: chown d7/c4a 3655 1 2024-04-22T15:49:31.304 INFO:tasks.workunit.client.0.smithi016.stdout:3/221: chown d7/d17/d18/f2a 7529 1 2024-04-22T15:49:31.305 INFO:tasks.workunit.client.0.smithi016.stdout:3/222: dread - d7/d17/d18/d32/f37 zero size 2024-04-22T15:49:31.310 INFO:tasks.workunit.client.0.smithi016.stdout:1/460: write d3/d4/d19/d20/d2b/f4a [909665,128420] 0 2024-04-22T15:49:31.312 INFO:tasks.workunit.client.0.smithi016.stdout:3/223: dwrite d7/d1b/f4b [0,4194304] 0 2024-04-22T15:49:31.317 INFO:tasks.workunit.client.0.smithi016.stdout:1/461: dwrite d3/d28/f98 [0,4194304] 0 2024-04-22T15:49:31.320 INFO:tasks.workunit.client.0.smithi016.stdout:3/224: getdents d7/d17/d18/d32 0 2024-04-22T15:49:31.323 INFO:tasks.workunit.client.0.smithi016.stdout:1/462: truncate d3/f8 383953 0 2024-04-22T15:49:31.327 INFO:tasks.workunit.client.0.smithi016.stdout:1/463: write d3/d4/d15/d38/d63/f75 [980526,72532] 0 2024-04-22T15:49:31.332 INFO:tasks.workunit.client.0.smithi016.stdout:1/464: rename d3/d4/d3a/d6b/d77/c85 to d3/d28/c9c 0 2024-04-22T15:49:31.334 INFO:tasks.workunit.client.0.smithi016.stdout:1/465: mkdir d3/d4/d19/d20/d9d 0 2024-04-22T15:49:31.336 INFO:tasks.workunit.client.0.smithi016.stdout:1/466: mkdir d3/d4/d3a/d9e 0 2024-04-22T15:49:31.337 INFO:tasks.workunit.client.0.smithi016.stdout:3/225: dread d7/d17/d18/f23 [0,4194304] 0 2024-04-22T15:49:31.338 INFO:tasks.workunit.client.0.smithi016.stdout:1/467: mkdir d3/d2e/d9f 0 2024-04-22T15:49:31.339 INFO:tasks.workunit.client.0.smithi016.stdout:1/468: dread - d3/d4/d15/d38/d63/d67/f79 zero size 2024-04-22T15:49:31.339 INFO:tasks.workunit.client.0.smithi016.stdout:3/226: write d7/d17/f3e [4002545,17679] 0 2024-04-22T15:49:31.342 INFO:tasks.workunit.client.0.smithi016.stdout:3/227: write f1 [2841120,30395] 0 2024-04-22T15:49:31.344 INFO:tasks.workunit.client.0.smithi016.stdout:3/228: mknod d7/c4c 0 2024-04-22T15:49:31.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.343+0000 7fac21730700 1 -- 172.21.15.16:0/881798420 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fac08005cc0 con 0x7fac1c0fef60 2024-04-22T15:49:31.346 INFO:tasks.workunit.client.0.smithi016.stdout:3/229: fsync d7/d17/d18/f1f 0 2024-04-22T15:49:31.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.343+0000 7fac03fff700 1 -- 172.21.15.16:0/881798420 <== mon.1 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+766 (secure 0 0 0) 0x7fac0c059e80 con 0x7fac1c0fef60 2024-04-22T15:49:31.347 INFO:tasks.workunit.client.0.smithi016.stdout:1/469: dwrite d3/d2e/f6d [0,4194304] 0 2024-04-22T15:49:31.347 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:49:31.347 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-22T15:49:31.347 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T15:49:31.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:49:31.347 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-22T15:49:31.347 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T15:49:31.347 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 1 2024-04-22T15:49:31.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:49:31.348 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-22T15:49:31.348 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-22T15:49:31.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:49:31.348 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-22T15:49:31.348 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T15:49:31.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:49:31.348 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-22T15:49:31.348 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 13, 2024-04-22T15:49:31.348 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 1 2024-04-22T15:49:31.348 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-22T15:49:31.348 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:49:31.351 INFO:tasks.workunit.client.0.smithi016.stdout:3/230: mknod d7/d17/d18/c4d 0 2024-04-22T15:49:31.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.348+0000 7fac21730700 1 -- 172.21.15.16:0/881798420 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7fac0406d8e0 msgr2=0x7fac0406fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:31.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.348+0000 7fac21730700 1 --2- 172.21.15.16:0/881798420 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7fac0406d8e0 0x7fac0406fda0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fac1000b6f0 tx=0x7fac10009f90 comp rx=0 tx=0).stop 2024-04-22T15:49:31.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.348+0000 7fac21730700 1 -- 172.21.15.16:0/881798420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac1c0fef60 msgr2=0x7fac1c1018a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:31.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.348+0000 7fac21730700 1 --2- 172.21.15.16:0/881798420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac1c0fef60 0x7fac1c1018a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fac0c00e940 tx=0x7fac0c00ec50 comp rx=0 tx=0).stop 2024-04-22T15:49:31.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.348+0000 7fac21730700 1 -- 172.21.15.16:0/881798420 shutdown_connections 2024-04-22T15:49:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.348+0000 7fac21730700 1 --2- 172.21.15.16:0/881798420 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7fac0406d8e0 0x7fac0406fda0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.348+0000 7fac21730700 1 --2- 172.21.15.16:0/881798420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac1c0fef60 0x7fac1c1018a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.348+0000 7fac21730700 1 --2- 172.21.15.16:0/881798420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac1c101f40 0x7fac1c0ffef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.348+0000 7fac21730700 1 -- 172.21.15.16:0/881798420 >> 172.21.15.16:0/881798420 conn(0x7fac1c0fa8c0 msgr2=0x7fac1c105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.349+0000 7fac21730700 1 -- 172.21.15.16:0/881798420 shutdown_connections 2024-04-22T15:49:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.349+0000 7fac21730700 1 -- 172.21.15.16:0/881798420 wait complete. 2024-04-22T15:49:31.353 INFO:tasks.workunit.client.0.smithi016.stdout:1/470: stat d3/d4/da/ld 0 2024-04-22T15:49:31.357 INFO:tasks.workunit.client.0.smithi016.stdout:3/231: dwrite d7/f15 [0,4194304] 0 2024-04-22T15:49:31.358 INFO:tasks.workunit.client.0.smithi016.stdout:0/179: sync 2024-04-22T15:49:31.360 INFO:tasks.workunit.client.0.smithi016.stdout:1/471: rename d3/d4/d19/d20/d2b/l8f to d3/d4/d15/d38/la0 0 2024-04-22T15:49:31.366 INFO:tasks.workunit.client.0.smithi016.stdout:3/232: dwrite d7/f22 [0,4194304] 0 2024-04-22T15:49:31.411 INFO:tasks.workunit.client.0.smithi016.stdout:3/233: dread d7/d17/f2e [0,4194304] 0 2024-04-22T15:49:31.414 INFO:tasks.workunit.client.0.smithi016.stdout:0/180: dwrite dc/f1e [0,4194304] 0 2024-04-22T15:49:31.416 INFO:tasks.workunit.client.0.smithi016.stdout:3/234: dread d7/d17/d18/f23 [0,4194304] 0 2024-04-22T15:49:31.417 INFO:tasks.workunit.client.0.smithi016.stdout:0/181: fdatasync dc/d13/d14/f18 0 2024-04-22T15:49:31.421 INFO:tasks.workunit.client.0.smithi016.stdout:3/235: creat d7/d17/d18/f4e x:0 0 0 2024-04-22T15:49:31.422 INFO:tasks.workunit.client.0.smithi016.stdout:0/182: rename dc/d13/d14/d26/f36 to dc/d13/d14/d1d/d1f/f3a 0 2024-04-22T15:49:31.425 INFO:tasks.workunit.client.0.smithi016.stdout:0/183: write dc/f27 [3077631,84551] 0 2024-04-22T15:49:31.433 INFO:tasks.workunit.client.0.smithi016.stdout:0/184: mknod dc/d13/d14/d26/d32/c3b 0 2024-04-22T15:49:31.433 INFO:tasks.workunit.client.0.smithi016.stdout:0/185: chown dc/d13/d14/d1d/d1f/f39 4770366 1 2024-04-22T15:49:31.434 INFO:tasks.workunit.client.0.smithi016.stdout:0/186: write dc/d13/d14/d1d/d1f/f39 [777686,9864] 0 2024-04-22T15:49:31.435 INFO:tasks.workunit.client.0.smithi016.stdout:0/187: write dc/d13/f31 [1483294,30928] 0 2024-04-22T15:49:31.441 INFO:tasks.workunit.client.0.smithi016.stdout:3/236: truncate d7/d17/f2e 782373 0 2024-04-22T15:49:31.443 INFO:tasks.workunit.client.0.smithi016.stdout:3/237: creat d7/d17/d18/d27/f4f x:0 0 0 2024-04-22T15:49:31.450 INFO:tasks.workunit.client.0.smithi016.stdout:3/238: getdents d7/d17/d18/d32 0 2024-04-22T15:49:31.451 INFO:tasks.workunit.client.0.smithi016.stdout:3/239: dread - d7/d17/d18/d27/f3d zero size 2024-04-22T15:49:31.452 INFO:tasks.workunit.client.0.smithi016.stdout:3/240: write d7/d17/d18/d27/f3d [1032419,14773] 0 2024-04-22T15:49:31.455 INFO:tasks.workunit.client.0.smithi016.stdout:3/241: mknod d7/c50 0 2024-04-22T15:49:31.456 INFO:tasks.workunit.client.0.smithi016.stdout:3/242: dread - d7/d17/d18/f4e zero size 2024-04-22T15:49:31.459 INFO:tasks.workunit.client.0.smithi016.stdout:3/243: unlink d7/d17/d18/f1f 0 2024-04-22T15:49:31.459 INFO:tasks.workunit.client.0.smithi016.stdout:3/244: stat d7/c4a 0 2024-04-22T15:49:31.466 INFO:tasks.workunit.client.0.smithi016.stdout:3/245: dwrite d7/d17/d18/f3f [0,4194304] 0 2024-04-22T15:49:31.468 INFO:tasks.workunit.client.0.smithi016.stdout:5/179: sync 2024-04-22T15:49:31.470 INFO:tasks.workunit.client.0.smithi016.stdout:5/180: write df/f1f [952619,102200] 0 2024-04-22T15:49:31.476 INFO:tasks.workunit.client.0.smithi016.stdout:5/181: dread df/f1f [0,4194304] 0 2024-04-22T15:49:31.482 INFO:tasks.workunit.client.0.smithi016.stdout:5/182: dwrite df/d22/f23 [0,4194304] 0 2024-04-22T15:49:31.490 INFO:tasks.workunit.client.0.smithi016.stdout:5/183: link df/d22/f2a df/f30 0 2024-04-22T15:49:31.491 INFO:tasks.workunit.client.0.smithi016.stdout:5/184: write df/f27 [936496,111871] 0 2024-04-22T15:49:31.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.490+0000 7fd80500a700 1 -- 172.21.15.16:0/870292032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd800100ca0 msgr2=0x7fd8001010c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:31.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.490+0000 7fd80500a700 1 --2- 172.21.15.16:0/870292032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd800100ca0 0x7fd8001010c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd7f0009910 tx=0x7fd7f0009c20 comp rx=0 tx=0).stop 2024-04-22T15:49:31.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.490+0000 7fd80500a700 1 -- 172.21.15.16:0/870292032 shutdown_connections 2024-04-22T15:49:31.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.490+0000 7fd80500a700 1 --2- 172.21.15.16:0/870292032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd800101ea0 0x7fd800102300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:31.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.490+0000 7fd80500a700 1 --2- 172.21.15.16:0/870292032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd800100ca0 0x7fd8001010c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:31.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.490+0000 7fd80500a700 1 -- 172.21.15.16:0/870292032 >> 172.21.15.16:0/870292032 conn(0x7fd8000fc260 msgr2=0x7fd8000fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:31.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.490+0000 7fd80500a700 1 -- 172.21.15.16:0/870292032 shutdown_connections 2024-04-22T15:49:31.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.490+0000 7fd80500a700 1 -- 172.21.15.16:0/870292032 wait complete. 2024-04-22T15:49:31.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.491+0000 7fd80500a700 1 Processor -- start 2024-04-22T15:49:31.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.492+0000 7fd80500a700 1 -- start start 2024-04-22T15:49:31.495 INFO:tasks.workunit.client.0.smithi016.stdout:5/185: creat df/d15/d1a/f31 x:0 0 0 2024-04-22T15:49:31.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.492+0000 7fd80500a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd800100ca0 0x7fd8001964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:31.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.492+0000 7fd80500a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd800101ea0 0x7fd800196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:31.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.492+0000 7fd80500a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd800197030 con 0x7fd800100ca0 2024-04-22T15:49:31.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.492+0000 7fd80500a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd800197170 con 0x7fd800101ea0 2024-04-22T15:49:31.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.492+0000 7fd7fed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd800100ca0 0x7fd8001964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:31.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.492+0000 7fd7fe59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd800101ea0 0x7fd800196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:31.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.492+0000 7fd7fed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd800100ca0 0x7fd8001964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42592/0 (socket says 172.21.15.16:42592) 2024-04-22T15:49:31.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.492+0000 7fd7fed9d700 1 -- 172.21.15.16:0/3622970158 learned_addr learned my addr 172.21.15.16:0/3622970158 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:49:31.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.493+0000 7fd7fed9d700 1 -- 172.21.15.16:0/3622970158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd800101ea0 msgr2=0x7fd800196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:31.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.493+0000 7fd7fed9d700 1 --2- 172.21.15.16:0/3622970158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd800101ea0 0x7fd800196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:31.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.493+0000 7fd7fed9d700 1 -- 172.21.15.16:0/3622970158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f00095f0 con 0x7fd800100ca0 2024-04-22T15:49:31.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.493+0000 7fd7fed9d700 1 --2- 172.21.15.16:0/3622970158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd800100ca0 0x7fd8001964d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd7f000bd70 tx=0x7fd7f000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:31.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.493+0000 7fd7e7fff700 1 -- 172.21.15.16:0/3622970158 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7f001d070 con 0x7fd800100ca0 2024-04-22T15:49:31.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.493+0000 7fd80500a700 1 -- 172.21.15.16:0/3622970158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd80019bbc0 con 0x7fd800100ca0 2024-04-22T15:49:31.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.493+0000 7fd7e7fff700 1 -- 172.21.15.16:0/3622970158 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd7f000f460 con 0x7fd800100ca0 2024-04-22T15:49:31.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.493+0000 7fd7e7fff700 1 -- 172.21.15.16:0/3622970158 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7f0021620 con 0x7fd800100ca0 2024-04-22T15:49:31.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.493+0000 7fd80500a700 1 -- 172.21.15.16:0/3622970158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd80019c0b0 con 0x7fd800100ca0 2024-04-22T15:49:31.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.495+0000 7fd80500a700 1 -- 172.21.15.16:0/3622970158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd800109800 con 0x7fd800100ca0 2024-04-22T15:49:31.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.495+0000 7fd7e7fff700 1 -- 172.21.15.16:0/3622970158 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) v1 ==== 91557+0+0 (secure 0 0 0) 0x7fd7f002b440 con 0x7fd800100ca0 2024-04-22T15:49:31.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.495+0000 7fd7e7fff700 1 --2- 172.21.15.16:0/3622970158 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7fd7e806d840 0x7fd7e806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:49:31.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.495+0000 7fd7e7fff700 1 -- 172.21.15.16:0/3622970158 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fd7f008e530 con 0x7fd800100ca0 2024-04-22T15:49:31.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.495+0000 7fd7fe59c700 1 --2- 172.21.15.16:0/3622970158 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7fd7e806d840 0x7fd7e806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:49:31.499 INFO:tasks.workunit.client.0.smithi016.stdout:5/186: link df/f1f df/d15/d1a/d28/f32 0 2024-04-22T15:49:31.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.496+0000 7fd7fe59c700 1 --2- 172.21.15.16:0/3622970158 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7fd7e806d840 0x7fd7e806fd00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd7f4005fd0 tx=0x7fd7f4005f60 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:49:31.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.501+0000 7fd7e7fff700 1 -- 172.21.15.16:0/3622970158 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7fd7f005b2a0 con 0x7fd800100ca0 2024-04-22T15:49:31.507 INFO:tasks.workunit.client.0.smithi016.stdout:5/187: dwrite df/f1f [0,4194304] 0 2024-04-22T15:49:31.515 INFO:tasks.workunit.client.0.smithi016.stdout:3/246: write d7/fc [1694911,78313] 0 2024-04-22T15:49:31.518 INFO:tasks.workunit.client.0.smithi016.stdout:3/247: getdents d7/d17/d18/d32/d42 0 2024-04-22T15:49:31.520 INFO:tasks.workunit.client.0.smithi016.stdout:3/248: mkdir d7/d17/d18/d32/d3c/d51 0 2024-04-22T15:49:31.525 INFO:tasks.workunit.client.0.smithi016.stdout:3/249: dread d7/fc [0,4194304] 0 2024-04-22T15:49:31.529 INFO:tasks.workunit.client.0.smithi016.stdout:3/250: dread d7/d1b/d21/f28 [0,4194304] 0 2024-04-22T15:49:31.531 INFO:tasks.workunit.client.0.smithi016.stdout:3/251: creat d7/d17/d18/d32/d3c/d47/f52 x:0 0 0 2024-04-22T15:49:31.536 INFO:tasks.workunit.client.0.smithi016.stdout:5/188: dwrite df/d22/f2a [0,4194304] 0 2024-04-22T15:49:31.536 INFO:tasks.workunit.client.0.smithi016.stdout:3/252: getdents d7/d17 0 2024-04-22T15:49:31.540 INFO:tasks.workunit.client.0.smithi016.stdout:5/189: mkdir df/d15/d1a/d28/d33 0 2024-04-22T15:49:31.540 INFO:tasks.workunit.client.0.smithi016.stdout:5/190: rename df/d15 to df/d15/d1a/d28/d34 22 2024-04-22T15:49:31.547 INFO:tasks.workunit.client.0.smithi016.stdout:5/191: dwrite df/d15/d1a/f20 [0,4194304] 0 2024-04-22T15:49:31.554 INFO:tasks.workunit.client.0.smithi016.stdout:5/192: dwrite df/f1f [0,4194304] 0 2024-04-22T15:49:31.555 INFO:tasks.workunit.client.0.smithi016.stdout:3/253: truncate d7/d1b/f4b 1427149 0 2024-04-22T15:49:31.558 INFO:tasks.workunit.client.0.smithi016.stdout:5/193: symlink df/d15/d1a/d28/l35 0 2024-04-22T15:49:31.559 INFO:tasks.workunit.client.0.smithi016.stdout:3/254: creat d7/d17/d18/d27/f53 x:0 0 0 2024-04-22T15:49:31.560 INFO:tasks.workunit.client.0.smithi016.stdout:3/255: readlink d7/d17/d18/d32/l40 0 2024-04-22T15:49:31.561 INFO:tasks.workunit.client.0.smithi016.stdout:5/194: write df/d15/d1a/d28/f2b [985183,112449] 0 2024-04-22T15:49:31.562 INFO:tasks.workunit.client.0.smithi016.stdout:3/256: write d7/d17/d18/d27/f3d [423556,103744] 0 2024-04-22T15:49:31.564 INFO:tasks.workunit.client.0.smithi016.stdout:9/238: sync 2024-04-22T15:49:31.564 INFO:tasks.workunit.client.0.smithi016.stdout:2/197: sync 2024-04-22T15:49:31.564 INFO:tasks.workunit.client.0.smithi016.stdout:7/233: sync 2024-04-22T15:49:31.564 INFO:tasks.workunit.client.0.smithi016.stdout:8/211: sync 2024-04-22T15:49:31.564 INFO:tasks.workunit.client.0.smithi016.stdout:4/178: sync 2024-04-22T15:49:31.565 INFO:tasks.workunit.client.0.smithi016.stdout:6/198: sync 2024-04-22T15:49:31.567 INFO:tasks.workunit.client.0.smithi016.stdout:5/195: dread df/f1f [0,4194304] 0 2024-04-22T15:49:31.567 INFO:tasks.workunit.client.0.smithi016.stdout:3/257: truncate d7/d17/d18/d32/d3c/d47/f48 47238 0 2024-04-22T15:49:31.567 INFO:tasks.workunit.client.0.smithi016.stdout:8/212: readlink d2/d32/l36 0 2024-04-22T15:49:31.569 INFO:tasks.workunit.client.0.smithi016.stdout:7/234: mknod d3/d32/c5c 0 2024-04-22T15:49:31.570 INFO:tasks.workunit.client.0.smithi016.stdout:3/258: dread d7/d1b/d21/f28 [0,4194304] 0 2024-04-22T15:49:31.571 INFO:tasks.workunit.client.0.smithi016.stdout:9/239: mknod d9/de/d23/d24/c46 0 2024-04-22T15:49:31.571 INFO:tasks.workunit.client.0.smithi016.stdout:3/259: chown d7/d17/d18/d32/d3c 0 1 2024-04-22T15:49:31.574 INFO:tasks.workunit.client.0.smithi016.stdout:6/199: creat d3/da/db/d14/d16/f3c x:0 0 0 2024-04-22T15:49:31.575 INFO:tasks.workunit.client.0.smithi016.stdout:4/179: rename d7/d24/f26 to d7/d21/d23/d2f/f32 0 2024-04-22T15:49:31.579 INFO:tasks.workunit.client.0.smithi016.stdout:9/240: dwrite d9/de/d23/d24/d25/f32 [4194304,4194304] 0 2024-04-22T15:49:31.579 INFO:tasks.workunit.client.0.smithi016.stdout:8/213: creat d2/db/f3c x:0 0 0 2024-04-22T15:49:31.583 INFO:tasks.workunit.client.0.smithi016.stdout:7/235: rename d3/d32/l3c to d3/d32/l5d 0 2024-04-22T15:49:31.584 INFO:tasks.workunit.client.0.smithi016.stdout:2/198: write d2/f4 [488002,21004] 0 2024-04-22T15:49:31.584 INFO:tasks.workunit.client.0.smithi016.stdout:3/260: creat d7/d17/d18/d27/f54 x:0 0 0 2024-04-22T15:49:31.588 INFO:tasks.workunit.client.0.smithi016.stdout:6/200: rename d3/f24 to d3/da/db/d14/d16/f3d 0 2024-04-22T15:49:31.589 INFO:tasks.workunit.client.0.smithi016.stdout:4/180: mkdir d7/d21/d23/d2f/d33 0 2024-04-22T15:49:31.592 INFO:tasks.workunit.client.0.smithi016.stdout:9/241: rmdir d9/de/d23/d24 39 2024-04-22T15:49:31.593 INFO:tasks.workunit.client.0.smithi016.stdout:8/214: creat d2/db/d1c/f3d x:0 0 0 2024-04-22T15:49:31.594 INFO:tasks.workunit.client.0.smithi016.stdout:9/242: write d9/f3f [140707,18175] 0 2024-04-22T15:49:31.594 INFO:tasks.workunit.client.0.smithi016.stdout:2/199: dwrite d2/f5 [4194304,4194304] 0 2024-04-22T15:49:31.595 INFO:tasks.workunit.client.0.smithi016.stdout:5/196: mknod df/d15/d1a/d28/d33/c36 0 2024-04-22T15:49:31.598 INFO:tasks.workunit.client.0.smithi016.stdout:3/261: mkdir d7/d1b/d21/d55 0 2024-04-22T15:49:31.603 INFO:tasks.workunit.client.0.smithi016.stdout:3/262: write d7/d17/d18/d32/d3c/d47/f33 [2074722,66883] 0 2024-04-22T15:49:31.606 INFO:tasks.workunit.client.0.smithi016.stdout:4/181: truncate d7/f19 949238 0 2024-04-22T15:49:31.606 INFO:tasks.workunit.client.0.smithi016.stdout:2/200: dwrite d2/d1a/d4c/f3f [0,4194304] 0 2024-04-22T15:49:31.606 INFO:tasks.workunit.client.0.smithi016.stdout:0/188: sync 2024-04-22T15:49:31.606 INFO:tasks.workunit.client.0.smithi016.stdout:1/472: sync 2024-04-22T15:49:31.608 INFO:tasks.workunit.client.0.smithi016.stdout:3/263: readlink d7/d1b/l1d 0 2024-04-22T15:49:31.609 INFO:tasks.workunit.client.0.smithi016.stdout:9/243: symlink d9/de/d26/l47 0 2024-04-22T15:49:31.612 INFO:tasks.workunit.client.0.smithi016.stdout:8/215: rename d2/d10/d23 to d2/d10/d34/d28/d3e 0 2024-04-22T15:49:31.614 INFO:tasks.workunit.client.0.smithi016.stdout:8/216: stat d2/db/d1c/f3d 0 2024-04-22T15:49:31.616 INFO:tasks.workunit.client.0.smithi016.stdout:6/201: dwrite d3/da/d12/ff [0,4194304] 0 2024-04-22T15:49:31.616 INFO:tasks.workunit.client.0.smithi016.stdout:8/217: rename d2 to d2/d32/d3f 22 2024-04-22T15:49:31.618 INFO:tasks.workunit.client.0.smithi016.stdout:0/189: fdatasync dc/d13/f16 0 2024-04-22T15:49:31.618 INFO:tasks.workunit.client.0.smithi016.stdout:9/244: creat d9/de/f48 x:0 0 0 2024-04-22T15:49:31.620 INFO:tasks.workunit.client.0.smithi016.stdout:3/264: unlink d7/d1b/d21/f28 0 2024-04-22T15:49:31.620 INFO:tasks.workunit.client.0.smithi016.stdout:0/190: readlink dc/d13/l1a 0 2024-04-22T15:49:31.627 INFO:tasks.workunit.client.0.smithi016.stdout:1/473: creat d3/d4/d3a/d91/fa1 x:0 0 0 2024-04-22T15:49:31.629 INFO:tasks.workunit.client.0.smithi016.stdout:0/191: creat dc/d13/d28/f3c x:0 0 0 2024-04-22T15:49:31.638 INFO:tasks.workunit.client.0.smithi016.stdout:8/218: truncate d2/f8 2060314 0 2024-04-22T15:49:31.640 INFO:tasks.workunit.client.0.smithi016.stdout:6/202: write d3/da/db/d14/d16/f20 [160875,14574] 0 2024-04-22T15:49:31.640 INFO:tasks.workunit.client.0.smithi016.stdout:7/236: link d3/dc/d1b/d5a/c1c d3/dc/d1b/d5a/c5e 0 2024-04-22T15:49:31.644 INFO:tasks.workunit.client.0.smithi016.stdout:1/474: creat d3/d4/da/fa2 x:0 0 0 2024-04-22T15:49:31.645 INFO:tasks.workunit.client.0.smithi016.stdout:0/192: unlink dc/d13/d14/d1d/d30/l38 0 2024-04-22T15:49:31.648 INFO:tasks.workunit.client.0.smithi016.stdout:2/201: link d2/d1a/c2d d2/d1a/d4c/c4e 0 2024-04-22T15:49:31.648 INFO:tasks.workunit.client.0.smithi016.stdout:0/193: write dc/d13/f31 [1942114,47666] 0 2024-04-22T15:49:31.649 INFO:tasks.workunit.client.0.smithi016.stdout:5/197: getdents df/d15/d1a/d28/d33 0 2024-04-22T15:49:31.650 INFO:tasks.workunit.client.0.smithi016.stdout:6/203: dwrite d3/f8 [4194304,4194304] 0 2024-04-22T15:49:31.651 INFO:tasks.workunit.client.0.smithi016.stdout:8/219: symlink d2/d32/l40 0 2024-04-22T15:49:31.654 INFO:tasks.workunit.client.0.smithi016.stdout:4/182: truncate d7/f8 576410 0 2024-04-22T15:49:31.654 INFO:tasks.workunit.client.0.smithi016.stdout:5/198: chown df/d15/d1a/d28/f2b 136 1 2024-04-22T15:49:31.654 INFO:tasks.workunit.client.0.smithi016.stdout:8/220: chown d2/d32/f35 10 1 2024-04-22T15:49:31.654 INFO:tasks.workunit.client.0.smithi016.stdout:3/265: rmdir d7/d17/d18/d32/d44 0 2024-04-22T15:49:31.655 INFO:tasks.workunit.client.0.smithi016.stdout:1/475: symlink d3/d2e/la3 0 2024-04-22T15:49:31.658 INFO:tasks.workunit.client.0.smithi016.stdout:2/202: unlink d2/d9/fa 0 2024-04-22T15:49:31.661 INFO:tasks.workunit.client.0.smithi016.stdout:6/204: creat d3/da/f3e x:0 0 0 2024-04-22T15:49:31.667 INFO:tasks.workunit.client.0.smithi016.stdout:8/221: creat d2/d10/f41 x:0 0 0 2024-04-22T15:49:31.668 INFO:tasks.workunit.client.0.smithi016.stdout:3/266: chown d7/d17/f2e 107 1 2024-04-22T15:49:31.668 INFO:tasks.workunit.client.0.smithi016.stdout:6/205: dread d3/da/dd/f23 [0,4194304] 0 2024-04-22T15:49:31.676 INFO:tasks.workunit.client.0.smithi016.stdout:2/203: dread d2/f2b [0,4194304] 0 2024-04-22T15:49:31.678 INFO:tasks.workunit.client.0.smithi016.stdout:2/204: truncate d2/de/d2a/f41 815586 0 2024-04-22T15:49:31.686 INFO:tasks.workunit.client.0.smithi016.stdout:3/267: chown d7/l39 940749 1 2024-04-22T15:49:31.693 INFO:tasks.workunit.client.0.smithi016.stdout:9/245: getdents d9/de/d23/d24 0 2024-04-22T15:49:31.694 INFO:tasks.workunit.client.0.smithi016.stdout:9/246: chown l1 1414 1 2024-04-22T15:49:31.694 INFO:tasks.workunit.client.0.smithi016.stdout:8/222: rmdir d2/d10/d34/d28/d3e 39 2024-04-22T15:49:31.695 INFO:tasks.workunit.client.0.smithi016.stdout:8/223: readlink d2/d32/l36 0 2024-04-22T15:49:31.695 INFO:tasks.workunit.client.0.smithi016.stdout:7/237: truncate d3/dc/d33/f3e 3823293 0 2024-04-22T15:49:31.695 INFO:tasks.workunit.client.0.smithi016.stdout:7/238: stat d3 0 2024-04-22T15:49:31.701 INFO:tasks.workunit.client.0.smithi016.stdout:2/205: creat d2/f4f x:0 0 0 2024-04-22T15:49:31.701 INFO:tasks.workunit.client.0.smithi016.stdout:2/206: chown d2/d9/l36 5 1 2024-04-22T15:49:31.705 INFO:tasks.workunit.client.0.smithi016.stdout:3/268: rename d7/c4a to d7/d17/d18/d27/c56 0 2024-04-22T15:49:31.706 INFO:tasks.workunit.client.0.smithi016.stdout:2/207: dwrite d2/de/d2a/f49 [0,4194304] 0 2024-04-22T15:49:31.709 INFO:tasks.workunit.client.0.smithi016.stdout:1/476: rmdir d3/d4/d15/d8c/d90 0 2024-04-22T15:49:31.711 INFO:tasks.workunit.client.0.smithi016.stdout:9/247: unlink d9/f3f 0 2024-04-22T15:49:31.711 INFO:tasks.workunit.client.0.smithi016.stdout:3/269: dread d7/f15 [0,4194304] 0 2024-04-22T15:49:31.712 INFO:tasks.workunit.client.0.smithi016.stdout:3/270: chown d7/d17/l29 460 1 2024-04-22T15:49:31.714 INFO:tasks.workunit.client.0.smithi016.stdout:7/239: mkdir d3/dc/d1b/d31/d5f 0 2024-04-22T15:49:31.716 INFO:tasks.workunit.client.0.smithi016.stdout:0/194: truncate dc/d13/f31 1234132 0 2024-04-22T15:49:31.719 INFO:tasks.workunit.client.0.smithi016.stdout:2/208: mknod d2/d1a/c50 0 2024-04-22T15:49:31.720 INFO:tasks.workunit.client.0.smithi016.stdout:5/199: truncate f6 1771020 0 2024-04-22T15:49:31.721 INFO:tasks.workunit.client.0.smithi016.stdout:3/271: dwrite f1 [4194304,4194304] 0 2024-04-22T15:49:31.775 INFO:tasks.workunit.client.0.smithi016.stdout:0/195: truncate dc/d13/d14/f18 847038 0 2024-04-22T15:49:31.776 INFO:tasks.workunit.client.0.smithi016.stdout:2/209: mkdir d2/d51 0 2024-04-22T15:49:31.778 INFO:tasks.workunit.client.0.smithi016.stdout:0/196: stat dc/d13/d14/d1d/d1f 0 2024-04-22T15:49:31.780 INFO:tasks.workunit.client.0.smithi016.stdout:3/272: fdatasync d7/d17/d18/f23 0 2024-04-22T15:49:31.782 INFO:tasks.workunit.client.0.smithi016.stdout:6/206: write d3/da/d12/f1c [86880,36879] 0 2024-04-22T15:49:31.784 INFO:tasks.workunit.client.0.smithi016.stdout:9/248: fsync d9/de/f16 0 2024-04-22T15:49:31.784 INFO:tasks.workunit.client.0.smithi016.stdout:7/240: symlink d3/dc/d1b/d5a/d59/l60 0 2024-04-22T15:49:31.788 INFO:tasks.workunit.client.0.smithi016.stdout:3/273: write d7/d17/f3e [3037315,21225] 0 2024-04-22T15:49:31.795 INFO:tasks.workunit.client.0.smithi016.stdout:2/210: dwrite d2/d9/f16 [0,4194304] 0 2024-04-22T15:49:31.799 INFO:tasks.workunit.client.0.smithi016.stdout:4/183: dwrite d7/f11 [0,4194304] 0 2024-04-22T15:49:31.799 INFO:tasks.workunit.client.0.smithi016.stdout:7/241: write d3/dc/d1b/d5a/f5b [888240,123279] 0 2024-04-22T15:49:31.799 INFO:tasks.workunit.client.0.smithi016.stdout:3/274: write d7/d1b/f35 [673070,116747] 0 2024-04-22T15:49:31.803 INFO:tasks.workunit.client.0.smithi016.stdout:5/200: dwrite df/d22/f24 [0,4194304] 0 2024-04-22T15:49:31.806 INFO:tasks.workunit.client.0.smithi016.stdout:2/211: fsync d2/f2b 0 2024-04-22T15:49:31.817 INFO:tasks.workunit.client.0.smithi016.stdout:3/275: creat d7/d17/d18/d32/f57 x:0 0 0 2024-04-22T15:49:31.818 INFO:tasks.workunit.client.0.smithi016.stdout:6/207: dwrite d3/da/db/d14/d16/f3c [0,4194304] 0 2024-04-22T15:49:31.822 INFO:tasks.workunit.client.0.smithi016.stdout:8/224: dwrite d2/f22 [0,4194304] 0 2024-04-22T15:49:31.824 INFO:tasks.workunit.client.0.smithi016.stdout:5/201: rename df/c13 to df/d15/d1a/d28/d33/c37 0 2024-04-22T15:49:31.824 INFO:tasks.workunit.client.0.smithi016.stdout:0/197: dread dc/d13/f16 [0,4194304] 0 2024-04-22T15:49:31.824 INFO:tasks.workunit.client.0.smithi016.stdout:9/249: getdents d9/d20 0 2024-04-22T15:49:31.825 INFO:tasks.workunit.client.0.smithi016.stdout:5/202: dread - df/d22/f25 zero size 2024-04-22T15:49:31.826 INFO:tasks.workunit.client.0.smithi016.stdout:6/208: write d3/da/db/d14/d16/f20 [1428798,51266] 0 2024-04-22T15:49:31.827 INFO:tasks.workunit.client.0.smithi016.stdout:0/198: chown dc/d13/d14/f34 211571 1 2024-04-22T15:49:31.829 INFO:tasks.workunit.client.0.smithi016.stdout:5/203: write df/d22/f25 [55091,80327] 0 2024-04-22T15:49:31.831 INFO:tasks.workunit.client.0.smithi016.stdout:4/184: write d7/f10 [3066538,12399] 0 2024-04-22T15:49:31.832 INFO:tasks.workunit.client.0.smithi016.stdout:3/276: fsync d7/d17/d18/f23 0 2024-04-22T15:49:31.833 INFO:tasks.workunit.client.0.smithi016.stdout:4/185: chown d7/f1d 135520 1 2024-04-22T15:49:31.837 INFO:tasks.workunit.client.0.smithi016.stdout:0/199: unlink dc/d13/l1a 0 2024-04-22T15:49:31.838 INFO:tasks.workunit.client.0.smithi016.stdout:6/209: creat d3/da/d12/d13/f3f x:0 0 0 2024-04-22T15:49:31.839 INFO:tasks.workunit.client.0.smithi016.stdout:9/250: dwrite d9/de/d23/d24/f2e [0,4194304] 0 2024-04-22T15:49:31.841 INFO:tasks.workunit.client.0.smithi016.stdout:6/210: fdatasync d3/f6 0 2024-04-22T15:49:31.842 INFO:tasks.workunit.client.0.smithi016.stdout:7/242: getdents d3/dc/d1b/d5a/d9 0 2024-04-22T15:49:31.844 INFO:tasks.workunit.client.0.smithi016.stdout:3/277: write d7/f14 [2513104,52460] 0 2024-04-22T15:49:31.846 INFO:tasks.workunit.client.0.smithi016.stdout:5/204: rename le to df/l38 0 2024-04-22T15:49:31.849 INFO:tasks.workunit.client.0.smithi016.stdout:3/278: write d7/f15 [4323330,57010] 0 2024-04-22T15:49:31.849 INFO:tasks.workunit.client.0.smithi016.stdout:4/186: dwrite d7/d22/f27 [0,4194304] 0 2024-04-22T15:49:31.851 INFO:tasks.workunit.client.0.smithi016.stdout:3/279: fsync d7/d17/d18/d27/f54 0 2024-04-22T15:49:31.852 INFO:tasks.workunit.client.0.smithi016.stdout:3/280: write d7/f22 [444614,97002] 0 2024-04-22T15:49:31.853 INFO:tasks.workunit.client.0.smithi016.stdout:8/225: rename f1 to d2/d10/d34/d28/d3e/f42 0 2024-04-22T15:49:31.854 INFO:tasks.workunit.client.0.smithi016.stdout:0/200: rename dc to dc/d13/d14/d1d/d30/d3d 22 2024-04-22T15:49:31.856 INFO:tasks.workunit.client.0.smithi016.stdout:3/281: write d7/d17/d18/d32/f37 [713121,115024] 0 2024-04-22T15:49:31.858 INFO:tasks.workunit.client.0.smithi016.stdout:5/205: creat df/d15/d1a/d28/d33/f39 x:0 0 0 2024-04-22T15:49:31.859 INFO:tasks.workunit.client.0.smithi016.stdout:0/201: dread - dc/d13/d28/f3c zero size 2024-04-22T15:49:31.862 INFO:tasks.workunit.client.0.smithi016.stdout:9/251: link d9/de/d23/f34 d9/d37/f49 0 2024-04-22T15:49:31.863 INFO:tasks.workunit.client.0.smithi016.stdout:2/212: truncate d2/d23/d27/f32 128191 0 2024-04-22T15:49:31.863 INFO:tasks.workunit.client.0.smithi016.stdout:3/282: dread - d7/d17/d18/d32/d3c/d47/f52 zero size 2024-04-22T15:49:31.865 INFO:tasks.workunit.client.0.smithi016.stdout:6/211: truncate d3/f6 2439613 0 2024-04-22T15:49:31.868 INFO:tasks.workunit.client.0.smithi016.stdout:5/206: rename df/d15/d1a/d28/f2e to df/d15/d1a/d28/f3a 0 2024-04-22T15:49:31.868 INFO:tasks.workunit.client.0.smithi016.stdout:8/226: dwrite d2/db/d1c/f3d [0,4194304] 0 2024-04-22T15:49:31.873 INFO:tasks.workunit.client.0.smithi016.stdout:9/252: fsync d9/d37/f3c 0 2024-04-22T15:49:31.874 INFO:tasks.workunit.client.0.smithi016.stdout:4/187: symlink d7/d21/d23/d2f/d33/l34 0 2024-04-22T15:49:31.875 INFO:tasks.workunit.client.0.smithi016.stdout:0/202: dwrite dc/d13/d14/d26/d32/f33 [0,4194304] 0 2024-04-22T15:49:31.875 INFO:tasks.workunit.client.0.smithi016.stdout:2/213: rmdir d2/d9 39 2024-04-22T15:49:31.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.874+0000 7fd80500a700 1 -- 172.21.15.16:0/3622970158 --> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8001067f0 con 0x7fd7e806d840 2024-04-22T15:49:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.876+0000 7fd7e7fff700 1 -- 172.21.15.16:0/3622970158 <== mgr.24395 v2:172.21.15.27:6828/2676414545 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fd8001067f0 con 0x7fd7e806d840 2024-04-22T15:49:31.879 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:49:31.879 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:49:31.879 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-22T15:49:31.879 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T15:49:31.880 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [], 2024-04-22T15:49:31.880 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "1/2 daemons upgraded", 2024-04-22T15:49:31.880 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading mgr daemons", 2024-04-22T15:49:31.880 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-22T15:49:31.880 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:49:31.880 INFO:tasks.workunit.client.0.smithi016.stdout:4/188: truncate d7/f10 3893522 0 2024-04-22T15:49:31.882 INFO:tasks.workunit.client.0.smithi016.stdout:3/283: dwrite d7/d17/d18/d27/f4f [0,4194304] 0 2024-04-22T15:49:31.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.880+0000 7fd80500a700 1 -- 172.21.15.16:0/3622970158 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7fd7e806d840 msgr2=0x7fd7e806fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:31.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.880+0000 7fd80500a700 1 --2- 172.21.15.16:0/3622970158 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7fd7e806d840 0x7fd7e806fd00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd7f4005fd0 tx=0x7fd7f4005f60 comp rx=0 tx=0).stop 2024-04-22T15:49:31.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.880+0000 7fd80500a700 1 -- 172.21.15.16:0/3622970158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd800100ca0 msgr2=0x7fd8001964d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:49:31.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.881+0000 7fd80500a700 1 --2- 172.21.15.16:0/3622970158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd800100ca0 0x7fd8001964d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd7f000bd70 tx=0x7fd7f000be50 comp rx=0 tx=0).stop 2024-04-22T15:49:31.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.881+0000 7fd80500a700 1 -- 172.21.15.16:0/3622970158 shutdown_connections 2024-04-22T15:49:31.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.881+0000 7fd80500a700 1 --2- 172.21.15.16:0/3622970158 >> [v2:172.21.15.27:6828/2676414545,v1:172.21.15.27:6829/2676414545] conn(0x7fd7e806d840 0x7fd7e806fd00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:31.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.881+0000 7fd80500a700 1 --2- 172.21.15.16:0/3622970158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd800100ca0 0x7fd8001964d0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:31.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.881+0000 7fd80500a700 1 --2- 172.21.15.16:0/3622970158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd800101ea0 0x7fd800196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:49:31.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.881+0000 7fd80500a700 1 -- 172.21.15.16:0/3622970158 >> 172.21.15.16:0/3622970158 conn(0x7fd8000fc260 msgr2=0x7fd8001050d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:49:31.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.881+0000 7fd80500a700 1 -- 172.21.15.16:0/3622970158 shutdown_connections 2024-04-22T15:49:31.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:49:31.881+0000 7fd80500a700 1 -- 172.21.15.16:0/3622970158 wait complete. 2024-04-22T15:49:31.888 INFO:tasks.workunit.client.0.smithi016.stdout:9/253: creat d9/d37/f4a x:0 0 0 2024-04-22T15:49:31.895 INFO:tasks.workunit.client.0.smithi016.stdout:0/203: unlink dc/d13/f37 0 2024-04-22T15:49:31.896 INFO:tasks.workunit.client.0.smithi016.stdout:9/254: dwrite d9/de/d23/d24/f2f [0,4194304] 0 2024-04-22T15:49:31.897 INFO:tasks.workunit.client.0.smithi016.stdout:2/214: rename d2/d1a/c50 to d2/d23/d27/c52 0 2024-04-22T15:49:31.898 INFO:tasks.workunit.client.0.smithi016.stdout:4/189: truncate f5 811484 0 2024-04-22T15:49:31.899 INFO:tasks.workunit.client.0.smithi016.stdout:0/204: write dc/d13/d14/d26/d32/f33 [4706702,125414] 0 2024-04-22T15:49:31.903 INFO:tasks.workunit.client.0.smithi016.stdout:2/215: mkdir d2/d1a/d53 0 2024-04-22T15:49:31.905 INFO:tasks.workunit.client.0.smithi016.stdout:3/284: symlink d7/d17/d18/d32/d3c/d51/l58 0 2024-04-22T15:49:31.907 INFO:tasks.workunit.client.0.smithi016.stdout:0/205: dwrite dc/d13/d14/d1d/d1f/f39 [0,4194304] 0 2024-04-22T15:49:31.909 INFO:tasks.workunit.client.0.smithi016.stdout:8/227: getdents d2/d10/d34 0 2024-04-22T15:49:31.910 INFO:tasks.workunit.client.0.smithi016.stdout:5/207: getdents df/d15/d1a/d28 0 2024-04-22T15:49:31.912 INFO:tasks.workunit.client.0.smithi016.stdout:0/206: write dc/d13/d14/d26/d32/f33 [878082,12777] 0 2024-04-22T15:49:31.913 INFO:tasks.workunit.client.0.smithi016.stdout:3/285: creat d7/d17/d18/d32/d3c/f59 x:0 0 0 2024-04-22T15:49:31.914 INFO:tasks.workunit.client.0.smithi016.stdout:0/207: write dc/d13/d22/f2e [272704,63468] 0 2024-04-22T15:49:31.915 INFO:tasks.workunit.client.0.smithi016.stdout:9/255: rename d9/l10 to d9/de/d23/l4b 0 2024-04-22T15:49:31.915 INFO:tasks.workunit.client.0.smithi016.stdout:2/216: dread d2/f5 [0,4194304] 0 2024-04-22T15:49:31.919 INFO:tasks.workunit.client.0.smithi016.stdout:5/208: creat df/d22/f3b x:0 0 0 2024-04-22T15:49:31.921 INFO:tasks.workunit.client.0.smithi016.stdout:3/286: dwrite d7/f14 [0,4194304] 0 2024-04-22T15:49:31.927 INFO:tasks.workunit.client.0.smithi016.stdout:6/212: truncate d3/da/db/d14/d16/f3c 1501364 0 2024-04-22T15:49:31.927 INFO:tasks.workunit.client.0.smithi016.stdout:6/213: stat d3/da/d12/d13 0 2024-04-22T15:49:31.930 INFO:tasks.workunit.client.0.smithi016.stdout:5/209: write df/d22/f2a [5095926,96654] 0 2024-04-22T15:49:31.930 INFO:tasks.workunit.client.0.smithi016.stdout:5/210: fdatasync df/d22/f3b 0 2024-04-22T15:49:31.933 INFO:tasks.workunit.client.0.smithi016.stdout:2/217: symlink d2/d9/l54 0 2024-04-22T15:49:31.933 INFO:tasks.workunit.client.0.smithi016.stdout:2/218: readlink d2/de/l29 0 2024-04-22T15:49:31.938 INFO:tasks.workunit.client.0.smithi016.stdout:2/219: read d2/d23/f3b [1885363,14361] 0 2024-04-22T15:49:31.938 INFO:tasks.workunit.client.0.smithi016.stdout:8/228: write d2/db/f20 [2217093,21503] 0 2024-04-22T15:49:31.939 INFO:tasks.workunit.client.0.smithi016.stdout:5/211: creat df/d15/d1a/d28/d33/f3c x:0 0 0 2024-04-22T15:49:31.940 INFO:tasks.workunit.client.0.smithi016.stdout:6/214: dread d3/da/d12/f15 [0,4194304] 0 2024-04-22T15:49:31.940 INFO:tasks.workunit.client.0.smithi016.stdout:5/212: write df/f30 [3554942,88355] 0 2024-04-22T15:49:31.941 INFO:tasks.workunit.client.0.smithi016.stdout:6/215: readlink d3/da/db/l1f 0 2024-04-22T15:49:31.942 INFO:tasks.workunit.client.0.smithi016.stdout:6/216: fdatasync d3/da/dd/f18 0 2024-04-22T15:49:31.943 INFO:tasks.workunit.client.0.smithi016.stdout:2/220: dread d2/de/d2a/f49 [0,4194304] 0 2024-04-22T15:49:31.945 INFO:tasks.workunit.client.0.smithi016.stdout:0/208: link dc/d13/d22/l2b dc/d13/d14/d1d/d30/l3e 0 2024-04-22T15:49:31.948 INFO:tasks.workunit.client.0.smithi016.stdout:0/209: chown dc/d13/d14/d1d/d1f/f3a 0 1 2024-04-22T15:49:31.949 INFO:tasks.workunit.client.0.smithi016.stdout:8/229: mkdir d2/d10/d43 0 2024-04-22T15:49:31.950 INFO:tasks.workunit.client.0.smithi016.stdout:2/221: dwrite d2/d23/d27/f31 [0,4194304] 0 2024-04-22T15:49:31.952 INFO:tasks.workunit.client.0.smithi016.stdout:8/230: truncate d2/d10/d34/d28/f38 444853 0 2024-04-22T15:49:31.954 INFO:tasks.workunit.client.0.smithi016.stdout:3/287: creat d7/f5a x:0 0 0 2024-04-22T15:49:31.955 INFO:tasks.workunit.client.0.smithi016.stdout:8/231: chown d2/d10/d34/d28/d3e/f42 210 1 2024-04-22T15:49:31.955 INFO:tasks.workunit.client.0.smithi016.stdout:9/256: getdents d9/de/d23/d24/d25 0 2024-04-22T15:49:31.955 INFO:tasks.workunit.client.0.smithi016.stdout:3/288: chown d7/f22 13 1 2024-04-22T15:49:31.956 INFO:tasks.workunit.client.0.smithi016.stdout:6/217: dread d3/da/dd/f10 [0,4194304] 0 2024-04-22T15:49:31.961 INFO:tasks.workunit.client.0.smithi016.stdout:2/222: mkdir d2/d23/d47/d55 0 2024-04-22T15:49:31.963 INFO:tasks.workunit.client.0.smithi016.stdout:9/257: symlink d9/d37/l4c 0 2024-04-22T15:49:31.965 INFO:tasks.workunit.client.0.smithi016.stdout:6/218: creat d3/da/d12/f40 x:0 0 0 2024-04-22T15:49:31.967 INFO:tasks.workunit.client.0.smithi016.stdout:5/213: rename df/d15/d1a/d28/l35 to df/l3d 0 2024-04-22T15:49:31.968 INFO:tasks.workunit.client.0.smithi016.stdout:8/232: mkdir d2/d10/d34/d28/d44 0 2024-04-22T15:49:31.968 INFO:tasks.workunit.client.0.smithi016.stdout:5/214: read - df/d15/d1a/d28/d33/f3c zero size 2024-04-22T15:49:31.969 INFO:tasks.workunit.client.0.smithi016.stdout:5/215: truncate df/d22/f23 4562134 0 2024-04-22T15:49:31.971 INFO:tasks.workunit.client.0.smithi016.stdout:3/289: rename d7/d1b/f35 to d7/d17/d18/f5b 0 2024-04-22T15:49:31.973 INFO:tasks.workunit.client.0.smithi016.stdout:3/290: dread - d7/d17/d18/d32/d3c/f59 zero size 2024-04-22T15:49:31.974 INFO:tasks.workunit.client.0.smithi016.stdout:5/216: dread df/d15/d1a/d28/f3a [0,4194304] 0 2024-04-22T15:49:31.975 INFO:tasks.workunit.client.0.smithi016.stdout:5/217: chown df/d22/f2a 902889463 1 2024-04-22T15:49:31.976 INFO:tasks.workunit.client.0.smithi016.stdout:5/218: chown l7 21 1 2024-04-22T15:49:31.978 INFO:tasks.workunit.client.0.smithi016.stdout:8/233: symlink d2/d10/d43/l45 0 2024-04-22T15:49:31.978 INFO:tasks.workunit.client.0.smithi016.stdout:5/219: fdatasync df/d15/d1a/d28/d33/f39 0 2024-04-22T15:49:31.979 INFO:tasks.workunit.client.0.smithi016.stdout:8/234: write d2/db/f16 [2814300,9038] 0 2024-04-22T15:49:31.979 INFO:tasks.workunit.client.0.smithi016.stdout:2/223: rename d2/d1a/d4c/l43 to d2/d1a/l56 0 2024-04-22T15:49:31.979 INFO:tasks.workunit.client.0.smithi016.stdout:3/291: dwrite d7/d17/d18/d27/f4f [0,4194304] 0 2024-04-22T15:49:31.982 INFO:tasks.workunit.client.0.smithi016.stdout:8/235: dread - d2/db/f3c zero size 2024-04-22T15:49:31.983 INFO:tasks.workunit.client.0.smithi016.stdout:0/210: write dc/d13/d14/f34 [2584858,116666] 0 2024-04-22T15:49:31.983 INFO:tasks.workunit.client.0.smithi016.stdout:3/292: fdatasync d7/d17/d18/d32/d3c/d47/f52 0 2024-04-22T15:49:31.984 INFO:tasks.workunit.client.0.smithi016.stdout:8/236: chown d2/d32/c31 1038070 1 2024-04-22T15:49:31.985 INFO:tasks.workunit.client.0.smithi016.stdout:3/293: write d7/d17/d18/d27/f53 [127312,47933] 0 2024-04-22T15:49:31.985 INFO:tasks.workunit.client.0.smithi016.stdout:8/237: truncate d2/d10/f39 43017 0 2024-04-22T15:49:31.987 INFO:tasks.workunit.client.0.smithi016.stdout:8/238: chown d2/d10/d34/l2e 752725 1 2024-04-22T15:49:31.987 INFO:tasks.workunit.client.0.smithi016.stdout:8/239: chown d2/d10/d34/d28/d3e/f42 1576553 1 2024-04-22T15:49:31.988 INFO:tasks.workunit.client.0.smithi016.stdout:9/258: rename d9/de/d26/l33 to d9/de/d23/d24/d25/l4d 0 2024-04-22T15:49:31.989 INFO:tasks.workunit.client.0.smithi016.stdout:8/240: truncate d2/d10/f39 887049 0 2024-04-22T15:49:31.990 INFO:tasks.workunit.client.0.smithi016.stdout:8/241: readlink d2/d10/d34/l33 0 2024-04-22T15:49:31.990 INFO:tasks.workunit.client.0.smithi016.stdout:8/242: chown d2/d10/d34/d28/d44 511588129 1 2024-04-22T15:49:31.993 INFO:tasks.workunit.client.0.smithi016.stdout:3/294: mkdir d7/d17/d18/d27/d5c 0 2024-04-22T15:49:31.993 INFO:tasks.workunit.client.0.smithi016.stdout:3/295: write f1 [7712178,37053] 0 2024-04-22T15:49:31.997 INFO:tasks.workunit.client.0.smithi016.stdout:9/259: fsync d9/de/d23/d24/d25/f3d 0 2024-04-22T15:49:31.999 INFO:tasks.workunit.client.0.smithi016.stdout:2/224: mknod d2/d4d/c57 0 2024-04-22T15:49:32.001 INFO:tasks.workunit.client.0.smithi016.stdout:2/225: write d2/d23/d27/f31 [5156740,38505] 0 2024-04-22T15:49:32.003 INFO:tasks.workunit.client.0.smithi016.stdout:5/220: rename df/l3d to df/d15/d1a/l3e 0 2024-04-22T15:49:32.003 INFO:tasks.workunit.client.0.smithi016.stdout:6/219: dwrite d3/f6 [0,4194304] 0 2024-04-22T15:49:32.010 INFO:tasks.workunit.client.0.smithi016.stdout:9/260: readlink d9/de/d23/d24/d25/l4d 0 2024-04-22T15:49:32.010 INFO:tasks.workunit.client.0.smithi016.stdout:9/261: readlink d9/l1e 0 2024-04-22T15:49:32.011 INFO:tasks.workunit.client.0.smithi016.stdout:3/296: symlink d7/d17/d18/d32/d42/l5d 0 2024-04-22T15:49:32.012 INFO:tasks.workunit.client.0.smithi016.stdout:0/211: dread f4 [0,4194304] 0 2024-04-22T15:49:32.015 INFO:tasks.workunit.client.0.smithi016.stdout:2/226: dwrite d2/d2c/f3c [4194304,4194304] 0 2024-04-22T15:49:32.016 INFO:tasks.workunit.client.0.smithi016.stdout:6/220: write d3/da/d12/f15 [3387210,115238] 0 2024-04-22T15:49:32.018 INFO:tasks.workunit.client.0.smithi016.stdout:9/262: mkdir d9/de/d23/d24/d4e 0 2024-04-22T15:49:32.018 INFO:tasks.workunit.client.0.smithi016.stdout:0/212: rmdir dc 39 2024-04-22T15:49:32.020 INFO:tasks.workunit.client.0.smithi016.stdout:5/221: rename f6 to df/d15/d1a/d28/f3f 0 2024-04-22T15:49:32.022 INFO:tasks.workunit.client.0.smithi016.stdout:5/222: fsync df/f27 0 2024-04-22T15:49:32.022 INFO:tasks.workunit.client.0.smithi016.stdout:9/263: write d9/fa [110719,26464] 0 2024-04-22T15:49:32.023 INFO:tasks.workunit.client.0.smithi016.stdout:8/243: dread d2/d10/d34/d28/d3e/f42 [0,4194304] 0 2024-04-22T15:49:32.023 INFO:tasks.workunit.client.0.smithi016.stdout:5/223: chown df/f30 3118247 1 2024-04-22T15:49:32.024 INFO:tasks.workunit.client.0.smithi016.stdout:2/227: chown d2/d23/d27/f32 2865716 1 2024-04-22T15:49:32.026 INFO:tasks.workunit.client.0.smithi016.stdout:2/228: truncate d2/f4 1419501 0 2024-04-22T15:49:32.028 INFO:tasks.workunit.client.0.smithi016.stdout:0/213: rmdir dc/d13/d22 39 2024-04-22T15:49:32.031 INFO:tasks.workunit.client.0.smithi016.stdout:2/229: fdatasync d2/de/d2a/f49 0 2024-04-22T15:49:32.032 INFO:tasks.workunit.client.0.smithi016.stdout:9/264: dwrite d9/d37/f4a [0,4194304] 0 2024-04-22T15:49:32.035 INFO:tasks.workunit.client.0.smithi016.stdout:8/244: link d2/d10/d43/l45 d2/d10/l46 0 2024-04-22T15:49:32.037 INFO:tasks.workunit.client.0.smithi016.stdout:9/265: truncate d9/de/d23/f29 553132 0 2024-04-22T15:49:32.037 INFO:tasks.workunit.client.0.smithi016.stdout:5/224: getdents df/d22 0 2024-04-22T15:49:32.039 INFO:tasks.workunit.client.0.smithi016.stdout:0/214: symlink dc/d13/d22/l3f 0 2024-04-22T15:49:32.042 INFO:tasks.workunit.client.0.smithi016.stdout:2/230: dwrite d2/d23/d27/f31 [4194304,4194304] 0 2024-04-22T15:49:32.047 INFO:tasks.workunit.client.0.smithi016.stdout:6/221: dread d3/da/db/f17 [0,4194304] 0 2024-04-22T15:49:32.048 INFO:tasks.workunit.client.0.smithi016.stdout:8/245: rmdir d2/d37 0 2024-04-22T15:49:32.048 INFO:tasks.workunit.client.0.smithi016.stdout:9/266: dwrite d9/fa [0,4194304] 0 2024-04-22T15:49:32.050 INFO:tasks.workunit.client.0.smithi016.stdout:2/231: write d2/d23/f3b [2400636,6109] 0 2024-04-22T15:49:32.053 INFO:tasks.workunit.client.0.smithi016.stdout:6/222: write d3/da/db/f17 [1242441,99119] 0 2024-04-22T15:49:32.057 INFO:tasks.workunit.client.0.smithi016.stdout:5/225: write f2 [498777,117055] 0 2024-04-22T15:49:32.057 INFO:tasks.workunit.client.0.smithi016.stdout:5/226: readlink la 0 2024-04-22T15:49:32.058 INFO:tasks.workunit.client.0.smithi016.stdout:2/232: symlink d2/d1a/l58 0 2024-04-22T15:49:32.058 INFO:tasks.workunit.client.0.smithi016.stdout:2/233: chown d2/d23/d47 911 1 2024-04-22T15:49:32.059 INFO:tasks.workunit.client.0.smithi016.stdout:0/215: getdents dc/d13/d14/d26 0 2024-04-22T15:49:32.059 INFO:tasks.workunit.client.0.smithi016.stdout:5/227: truncate df/d15/d1a/d28/f2b 1732945 0 2024-04-22T15:49:32.062 INFO:tasks.workunit.client.0.smithi016.stdout:6/223: mknod d3/da/d12/d13/d38/d33/c41 0 2024-04-22T15:49:32.062 INFO:tasks.workunit.client.0.smithi016.stdout:2/234: mkdir d2/de/d2a/d59 0 2024-04-22T15:49:32.064 INFO:tasks.workunit.client.0.smithi016.stdout:5/228: truncate df/f11 510056 0 2024-04-22T15:49:32.066 INFO:tasks.workunit.client.0.smithi016.stdout:0/216: mkdir dc/d13/d14/d1d/d40 0 2024-04-22T15:49:32.068 INFO:tasks.workunit.client.0.smithi016.stdout:8/246: dread d2/db/f16 [0,4194304] 0 2024-04-22T15:49:32.069 INFO:tasks.workunit.client.0.smithi016.stdout:9/267: write d9/de/f18 [1190425,118984] 0 2024-04-22T15:49:32.070 INFO:tasks.workunit.client.0.smithi016.stdout:2/235: dwrite d2/f4 [0,4194304] 0 2024-04-22T15:49:32.070 INFO:tasks.workunit.client.0.smithi016.stdout:0/217: rmdir dc 39 2024-04-22T15:49:32.071 INFO:tasks.workunit.client.0.smithi016.stdout:6/224: getdents d3/da/d12/d13/d38/d33/d35 0 2024-04-22T15:49:32.071 INFO:tasks.workunit.client.0.smithi016.stdout:5/229: write df/f1f [5093988,29057] 0 2024-04-22T15:49:32.075 INFO:tasks.workunit.client.0.smithi016.stdout:2/236: write d2/d23/d27/d34/f4b [1017750,46180] 0 2024-04-22T15:49:32.076 INFO:tasks.workunit.client.0.smithi016.stdout:6/225: write d3/da/d12/ff [991851,3287] 0 2024-04-22T15:49:32.079 INFO:tasks.workunit.client.0.smithi016.stdout:8/247: rename d2/d10/d34/d28/d3e/d2c to d2/d10/d47 0 2024-04-22T15:49:32.081 INFO:tasks.workunit.client.0.smithi016.stdout:0/218: symlink dc/d13/d14/d1d/d30/l41 0 2024-04-22T15:49:32.085 INFO:tasks.workunit.client.0.smithi016.stdout:6/226: rmdir d3/da/db/d14/d16 39 2024-04-22T15:49:32.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:31 smithi027 ceph-mon[34922]: from='client.24449 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:49:32.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:31 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/881798420' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:49:32.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:31 smithi027 ceph-mon[34922]: pgmap v28: 65 pgs: 65 active+clean; 349 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s rd, 24 MiB/s wr, 257 op/s 2024-04-22T15:49:32.088 INFO:tasks.workunit.client.0.smithi016.stdout:9/268: dwrite d9/de/d23/d24/d25/f2c [0,4194304] 0 2024-04-22T15:49:32.088 INFO:tasks.workunit.client.0.smithi016.stdout:5/230: dwrite df/d15/d1a/d28/f32 [0,4194304] 0 2024-04-22T15:49:32.090 INFO:tasks.workunit.client.0.smithi016.stdout:0/219: creat dc/d13/d14/d1d/f42 x:0 0 0 2024-04-22T15:49:32.091 INFO:tasks.workunit.client.0.smithi016.stdout:2/237: dwrite d2/de/d2a/f42 [0,4194304] 0 2024-04-22T15:49:32.091 INFO:tasks.workunit.client.0.smithi016.stdout:8/248: mkdir d2/d48 0 2024-04-22T15:49:32.095 INFO:tasks.workunit.client.0.smithi016.stdout:9/269: write f2 [3839115,123937] 0 2024-04-22T15:49:32.098 INFO:tasks.workunit.client.0.smithi016.stdout:8/249: chown d2/d10/l46 7 1 2024-04-22T15:49:32.099 INFO:tasks.workunit.client.0.smithi016.stdout:2/238: creat d2/d51/f5a x:0 0 0 2024-04-22T15:49:32.100 INFO:tasks.workunit.client.0.smithi016.stdout:2/239: stat d2/de/d2a/d59 0 2024-04-22T15:49:32.101 INFO:tasks.workunit.client.0.smithi016.stdout:9/270: dwrite f2 [0,4194304] 0 2024-04-22T15:49:32.102 INFO:tasks.workunit.client.0.smithi016.stdout:6/227: creat d3/da/db/d14/d16/f42 x:0 0 0 2024-04-22T15:49:32.103 INFO:tasks.workunit.client.0.smithi016.stdout:6/228: fdatasync d3/da/dd/f29 0 2024-04-22T15:49:32.106 INFO:tasks.workunit.client.0.smithi016.stdout:2/240: dread d2/de/d2a/f42 [0,4194304] 0 2024-04-22T15:49:32.108 INFO:tasks.workunit.client.0.smithi016.stdout:8/250: creat d2/d48/f49 x:0 0 0 2024-04-22T15:49:32.109 INFO:tasks.workunit.client.0.smithi016.stdout:6/229: symlink d3/da/db/l43 0 2024-04-22T15:49:32.109 INFO:tasks.workunit.client.0.smithi016.stdout:0/220: link dc/d13/d14/d1d/c23 dc/d13/d14/d1d/d40/c43 0 2024-04-22T15:49:32.111 INFO:tasks.workunit.client.0.smithi016.stdout:9/271: mkdir d9/de/d23/d24/d4e/d4f 0 2024-04-22T15:49:32.112 INFO:tasks.workunit.client.0.smithi016.stdout:2/241: creat d2/d2c/f5b x:0 0 0 2024-04-22T15:49:32.114 INFO:tasks.workunit.client.0.smithi016.stdout:0/221: creat dc/d13/d14/d1d/d30/f44 x:0 0 0 2024-04-22T15:49:32.117 INFO:tasks.workunit.client.0.smithi016.stdout:0/222: symlink dc/d13/d14/d1d/l45 0 2024-04-22T15:49:32.117 INFO:tasks.workunit.client.0.smithi016.stdout:0/223: chown dc/d13/l24 0 1 2024-04-22T15:49:32.117 INFO:tasks.workunit.client.0.smithi016.stdout:0/224: fdatasync dc/f27 0 2024-04-22T15:49:32.119 INFO:tasks.workunit.client.0.smithi016.stdout:0/225: fdatasync dc/d13/d14/d1d/d1f/f3a 0 2024-04-22T15:49:32.120 INFO:tasks.workunit.client.0.smithi016.stdout:0/226: rmdir dc/d13/d28 39 2024-04-22T15:49:32.137 INFO:tasks.workunit.client.0.smithi016.stdout:8/251: rename d2/d10/d34 to d2/d10/d43/d4a 0 2024-04-22T15:49:32.137 INFO:tasks.workunit.client.0.smithi016.stdout:8/252: chown d2/db 609513 1 2024-04-22T15:49:32.139 INFO:tasks.workunit.client.0.smithi016.stdout:8/253: creat d2/d10/d43/f4b x:0 0 0 2024-04-22T15:49:32.146 INFO:tasks.workunit.client.0.smithi016.stdout:8/254: dwrite d2/db/f3c [0,4194304] 0 2024-04-22T15:49:32.146 INFO:tasks.workunit.client.0.smithi016.stdout:0/227: getdents dc/d13/d14/d1d/d40 0 2024-04-22T15:49:32.149 INFO:tasks.workunit.client.0.smithi016.stdout:8/255: write d2/db/f16 [3507872,806] 0 2024-04-22T15:49:32.151 INFO:tasks.workunit.client.0.smithi016.stdout:8/256: write d2/db/f20 [4624188,36780] 0 2024-04-22T15:49:32.154 INFO:tasks.workunit.client.0.smithi016.stdout:0/228: dwrite dc/d13/d22/f2e [0,4194304] 0 2024-04-22T15:49:32.156 INFO:tasks.workunit.client.0.smithi016.stdout:0/229: chown dc/d13/d14/c25 2557 1 2024-04-22T15:49:32.156 INFO:tasks.workunit.client.0.smithi016.stdout:8/257: dread d2/db/f3c [0,4194304] 0 2024-04-22T15:49:32.157 INFO:tasks.workunit.client.0.smithi016.stdout:6/230: truncate d3/f31 2240790 0 2024-04-22T15:49:32.158 INFO:tasks.workunit.client.0.smithi016.stdout:0/230: write dc/d13/d14/d1d/d1f/f39 [2665295,34896] 0 2024-04-22T15:49:32.159 INFO:tasks.workunit.client.0.smithi016.stdout:6/231: write d3/da/d12/d13/f27 [1323485,89415] 0 2024-04-22T15:49:32.160 INFO:tasks.workunit.client.0.smithi016.stdout:9/272: truncate d9/de/d23/d24/d25/f32 5364634 0 2024-04-22T15:49:32.164 INFO:tasks.workunit.client.0.smithi016.stdout:2/242: truncate d2/d23/d27/f2e 67902 0 2024-04-22T15:49:32.165 INFO:tasks.workunit.client.0.smithi016.stdout:8/258: creat d2/db/f4c x:0 0 0 2024-04-22T15:49:32.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:31 smithi016 ceph-mon[26620]: from='client.24449 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:49:32.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:31 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/881798420' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:49:32.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:31 smithi016 ceph-mon[26620]: pgmap v28: 65 pgs: 65 active+clean; 349 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s rd, 24 MiB/s wr, 257 op/s 2024-04-22T15:49:32.168 INFO:tasks.workunit.client.0.smithi016.stdout:6/232: mkdir d3/da/db/d44 0 2024-04-22T15:49:32.169 INFO:tasks.workunit.client.0.smithi016.stdout:9/273: fsync d9/de/d23/f34 0 2024-04-22T15:49:32.169 INFO:tasks.workunit.client.0.smithi016.stdout:8/259: dread d2/db/f3c [0,4194304] 0 2024-04-22T15:49:32.183 INFO:tasks.workunit.client.0.smithi016.stdout:8/260: dwrite d2/d10/d43/d4a/d28/d3e/f42 [4194304,4194304] 0 2024-04-22T15:49:32.184 INFO:tasks.workunit.client.0.smithi016.stdout:8/261: write d2/d10/f41 [103792,31968] 0 2024-04-22T15:49:32.185 INFO:tasks.workunit.client.0.smithi016.stdout:6/233: getdents d3/da/dd/d1b 0 2024-04-22T15:49:32.191 INFO:tasks.workunit.client.0.smithi016.stdout:6/234: rmdir d3/da/db/d14 39 2024-04-22T15:49:32.191 INFO:tasks.workunit.client.0.smithi016.stdout:0/231: dread dc/f27 [0,4194304] 0 2024-04-22T15:49:32.192 INFO:tasks.workunit.client.0.smithi016.stdout:6/235: chown d3/da/db/d44 0 1 2024-04-22T15:49:32.192 INFO:tasks.workunit.client.0.smithi016.stdout:0/232: readlink dc/d13/d14/d1d/l2d 0 2024-04-22T15:49:32.197 INFO:tasks.workunit.client.0.smithi016.stdout:6/236: creat d3/da/f45 x:0 0 0 2024-04-22T15:49:32.197 INFO:tasks.workunit.client.0.smithi016.stdout:0/233: write dc/d13/d28/f3c [610013,45833] 0 2024-04-22T15:49:32.198 INFO:tasks.workunit.client.0.smithi016.stdout:0/234: stat dc/f27 0 2024-04-22T15:49:32.200 INFO:tasks.workunit.client.0.smithi016.stdout:6/237: mkdir d3/da/d12/d13/d46 0 2024-04-22T15:49:32.201 INFO:tasks.workunit.client.0.smithi016.stdout:0/235: creat dc/d13/d14/d26/f46 x:0 0 0 2024-04-22T15:49:32.202 INFO:tasks.workunit.client.0.smithi016.stdout:8/262: getdents d2/d32 0 2024-04-22T15:49:32.204 INFO:tasks.workunit.client.0.smithi016.stdout:2/243: truncate d2/d1a/d4c/f3f 1039549 0 2024-04-22T15:49:32.208 INFO:tasks.workunit.client.0.smithi016.stdout:6/238: symlink d3/da/db/d44/l47 0 2024-04-22T15:49:32.211 INFO:tasks.workunit.client.0.smithi016.stdout:0/236: rename dc/l12 to dc/d13/d14/d1d/d40/l47 0 2024-04-22T15:49:32.218 INFO:tasks.workunit.client.0.smithi016.stdout:0/237: mkdir dc/d13/d14/d1d/d1f/d48 0 2024-04-22T15:49:32.220 INFO:tasks.workunit.client.0.smithi016.stdout:0/238: read - dc/d13/d14/d1d/d1f/f3a zero size 2024-04-22T15:49:32.220 INFO:tasks.workunit.client.0.smithi016.stdout:2/244: creat d2/de/f5c x:0 0 0 2024-04-22T15:49:32.221 INFO:tasks.workunit.client.0.smithi016.stdout:6/239: creat d3/da/d12/d13/d46/f48 x:0 0 0 2024-04-22T15:49:32.222 INFO:tasks.workunit.client.0.smithi016.stdout:0/239: chown dc/d13/d14/d1d/c19 28318459 1 2024-04-22T15:49:32.222 INFO:tasks.workunit.client.0.smithi016.stdout:2/245: write d2/d23/d47/f4a [859216,6707] 0 2024-04-22T15:49:32.223 INFO:tasks.workunit.client.0.smithi016.stdout:6/240: stat d3/da/d12/d13/d46 0 2024-04-22T15:49:32.224 INFO:tasks.workunit.client.0.smithi016.stdout:8/263: dwrite d2/f24 [0,4194304] 0 2024-04-22T15:49:32.227 INFO:tasks.workunit.client.0.smithi016.stdout:0/240: symlink dc/d13/d14/d1d/d1f/l49 0 2024-04-22T15:49:32.228 INFO:tasks.workunit.client.0.smithi016.stdout:0/241: read - dc/d13/d14/d1d/d30/f44 zero size 2024-04-22T15:49:32.228 INFO:tasks.workunit.client.0.smithi016.stdout:0/242: chown dc/d13/d28 0 1 2024-04-22T15:49:32.235 INFO:tasks.workunit.client.0.smithi016.stdout:0/243: mkdir dc/d13/d14/d1d/d40/d4a 0 2024-04-22T15:49:32.237 INFO:tasks.workunit.client.0.smithi016.stdout:2/246: rename d2/de/d2a/f41 to d2/de/f5d 0 2024-04-22T15:49:32.238 INFO:tasks.workunit.client.0.smithi016.stdout:8/264: creat d2/d10/d43/d4a/d28/f4d x:0 0 0 2024-04-22T15:49:32.239 INFO:tasks.workunit.client.0.smithi016.stdout:6/241: link d3/da/db/d14/d16/c19 d3/da/db/d44/c49 0 2024-04-22T15:49:32.242 INFO:tasks.workunit.client.0.smithi016.stdout:0/244: symlink dc/d13/d14/l4b 0 2024-04-22T15:49:32.242 INFO:tasks.workunit.client.0.smithi016.stdout:0/245: fdatasync dc/d13/f16 0 2024-04-22T15:49:32.242 INFO:tasks.workunit.client.0.smithi016.stdout:8/265: symlink d2/db/l4e 0 2024-04-22T15:49:32.244 INFO:tasks.workunit.client.0.smithi016.stdout:6/242: unlink d3/da/l3b 0 2024-04-22T15:49:32.244 INFO:tasks.workunit.client.0.smithi016.stdout:2/247: getdents d2/d1a/d53 0 2024-04-22T15:49:32.249 INFO:tasks.workunit.client.0.smithi016.stdout:8/266: symlink d2/d10/d43/d4a/d28/d3e/l4f 0 2024-04-22T15:49:32.250 INFO:tasks.workunit.client.0.smithi016.stdout:8/267: truncate d2/f22 4228521 0 2024-04-22T15:49:32.253 INFO:tasks.workunit.client.0.smithi016.stdout:0/246: write dc/d13/d14/d26/d32/f33 [1184454,109084] 0 2024-04-22T15:49:32.256 INFO:tasks.workunit.client.0.smithi016.stdout:8/268: symlink d2/db/l50 0 2024-04-22T15:49:32.258 INFO:tasks.workunit.client.0.smithi016.stdout:2/248: link d2/f7 d2/d1a/f5e 0 2024-04-22T15:49:32.263 INFO:tasks.workunit.client.0.smithi016.stdout:6/243: rename d3/da/db/c37 to d3/c4a 0 2024-04-22T15:49:32.263 INFO:tasks.workunit.client.0.smithi016.stdout:2/249: dwrite d2/d23/d27/f31 [4194304,4194304] 0 2024-04-22T15:49:32.264 INFO:tasks.workunit.client.0.smithi016.stdout:6/244: read d3/da/d12/ff [2700278,102674] 0 2024-04-22T15:49:32.271 INFO:tasks.workunit.client.0.smithi016.stdout:6/245: mknod d3/da/d12/d13/d38/d33/d35/c4b 0 2024-04-22T15:49:32.271 INFO:tasks.workunit.client.0.smithi016.stdout:2/250: dwrite d2/f2b [0,4194304] 0 2024-04-22T15:49:32.276 INFO:tasks.workunit.client.0.smithi016.stdout:2/251: creat d2/d23/d27/f5f x:0 0 0 2024-04-22T15:49:32.278 INFO:tasks.workunit.client.0.smithi016.stdout:6/246: mkdir d3/da/db/d14/d4c 0 2024-04-22T15:49:32.279 INFO:tasks.workunit.client.0.smithi016.stdout:6/247: stat d3/da/d12/d13 0 2024-04-22T15:49:32.282 INFO:tasks.workunit.client.0.smithi016.stdout:2/252: mkdir d2/d23/d27/d34/d60 0 2024-04-22T15:49:32.283 INFO:tasks.workunit.client.0.smithi016.stdout:1/477: sync 2024-04-22T15:49:32.283 INFO:tasks.workunit.client.0.smithi016.stdout:7/243: sync 2024-04-22T15:49:32.284 INFO:tasks.workunit.client.0.smithi016.stdout:8/269: truncate d2/f24 2966373 0 2024-04-22T15:49:32.284 INFO:tasks.workunit.client.0.smithi016.stdout:2/253: write d2/d23/d27/d34/f4b [1801731,74168] 0 2024-04-22T15:49:32.285 INFO:tasks.workunit.client.0.smithi016.stdout:7/244: write d3/f23 [3347336,111672] 0 2024-04-22T15:49:32.287 INFO:tasks.workunit.client.0.smithi016.stdout:1/478: symlink d3/d4/d37/la4 0 2024-04-22T15:49:32.288 INFO:tasks.workunit.client.0.smithi016.stdout:2/254: chown d2/de/d2a/f42 16662023 1 2024-04-22T15:49:32.290 INFO:tasks.workunit.client.0.smithi016.stdout:0/247: dwrite dc/d13/d14/f18 [0,4194304] 0 2024-04-22T15:49:32.291 INFO:tasks.workunit.client.0.smithi016.stdout:2/255: fsync d2/d2c/f3c 0 2024-04-22T15:49:32.292 INFO:tasks.workunit.client.0.smithi016.stdout:1/479: chown d3/d4/d19/d20/d2b/c93 0 1 2024-04-22T15:49:32.293 INFO:tasks.workunit.client.0.smithi016.stdout:2/256: stat d2/d23/c30 0 2024-04-22T15:49:32.294 INFO:tasks.workunit.client.0.smithi016.stdout:0/248: dread - dc/d13/d14/d1d/d30/f44 zero size 2024-04-22T15:49:32.296 INFO:tasks.workunit.client.0.smithi016.stdout:6/248: write d3/f8 [4700214,37432] 0 2024-04-22T15:49:32.296 INFO:tasks.workunit.client.0.smithi016.stdout:1/480: chown d3/d4/d19/l1a 3912 1 2024-04-22T15:49:32.297 INFO:tasks.workunit.client.0.smithi016.stdout:8/270: dwrite d2/d10/d43/d4a/d28/d3e/f42 [8388608,4194304] 0 2024-04-22T15:49:32.302 INFO:tasks.workunit.client.0.smithi016.stdout:0/249: creat dc/d13/d14/d1d/d1f/f4c x:0 0 0 2024-04-22T15:49:32.304 INFO:tasks.workunit.client.0.smithi016.stdout:6/249: mknod d3/da/db/d44/c4d 0 2024-04-22T15:49:32.305 INFO:tasks.workunit.client.0.smithi016.stdout:7/245: write d3/f34 [181739,38441] 0 2024-04-22T15:49:32.306 INFO:tasks.workunit.client.0.smithi016.stdout:2/257: getdents d2/de/d2a/d59 0 2024-04-22T15:49:32.311 INFO:tasks.workunit.client.0.smithi016.stdout:0/250: rename dc/d13/d14/d1d/l45 to dc/d13/d14/d1d/l4d 0 2024-04-22T15:49:32.312 INFO:tasks.workunit.client.0.smithi016.stdout:8/271: symlink d2/d10/d43/d4a/d28/l51 0 2024-04-22T15:49:32.314 INFO:tasks.workunit.client.0.smithi016.stdout:8/272: write d2/d30/f3a [762759,69886] 0 2024-04-22T15:49:32.316 INFO:tasks.workunit.client.0.smithi016.stdout:0/251: creat dc/d13/d14/d26/d32/f4e x:0 0 0 2024-04-22T15:49:32.317 INFO:tasks.workunit.client.0.smithi016.stdout:6/250: dwrite d3/f1d [0,4194304] 0 2024-04-22T15:49:32.320 INFO:tasks.workunit.client.0.smithi016.stdout:1/481: write d3/d4/d19/f35 [1931427,61624] 0 2024-04-22T15:49:32.321 INFO:tasks.workunit.client.0.smithi016.stdout:8/273: rmdir d2/d48 39 2024-04-22T15:49:32.323 INFO:tasks.workunit.client.0.smithi016.stdout:6/251: symlink d3/da/dd/l4e 0 2024-04-22T15:49:32.324 INFO:tasks.workunit.client.0.smithi016.stdout:0/252: dwrite dc/f27 [4194304,4194304] 0 2024-04-22T15:49:32.325 INFO:tasks.workunit.client.0.smithi016.stdout:1/482: creat d3/d2e/d80/fa5 x:0 0 0 2024-04-22T15:49:32.328 INFO:tasks.workunit.client.0.smithi016.stdout:0/253: write dc/d13/d14/d1d/d30/f44 [749398,42852] 0 2024-04-22T15:49:32.328 INFO:tasks.workunit.client.0.smithi016.stdout:7/246: truncate d3/d32/f3d 6945068 0 2024-04-22T15:49:32.330 INFO:tasks.workunit.client.0.smithi016.stdout:6/252: rename d3/da/d12/d13/d46/f48 to d3/da/d12/d13/d46/f4f 0 2024-04-22T15:49:32.333 INFO:tasks.workunit.client.0.smithi016.stdout:8/274: mkdir d2/d10/d43/d4a/d28/d52 0 2024-04-22T15:49:32.334 INFO:tasks.workunit.client.0.smithi016.stdout:8/275: stat d2/d10/f41 0 2024-04-22T15:49:32.335 INFO:tasks.workunit.client.0.smithi016.stdout:6/253: unlink d3/da/d12/d13/d38/d33/c34 0 2024-04-22T15:49:32.337 INFO:tasks.workunit.client.0.smithi016.stdout:1/483: dread d3/d4/d3a/d6b/d77/f76 [0,4194304] 0 2024-04-22T15:49:32.338 INFO:tasks.workunit.client.0.smithi016.stdout:1/484: stat d3/d2e/c47 0 2024-04-22T15:49:32.338 INFO:tasks.workunit.client.0.smithi016.stdout:0/254: creat dc/d13/d14/d26/f4f x:0 0 0 2024-04-22T15:49:32.340 INFO:tasks.workunit.client.0.smithi016.stdout:1/485: truncate d3/d2e/d80/fa5 80552 0 2024-04-22T15:49:32.340 INFO:tasks.workunit.client.0.smithi016.stdout:0/255: chown dc/d13/d14/f18 190 1 2024-04-22T15:49:32.340 INFO:tasks.workunit.client.0.smithi016.stdout:8/276: creat d2/d10/d43/d4a/d28/d44/f53 x:0 0 0 2024-04-22T15:49:32.341 INFO:tasks.workunit.client.0.smithi016.stdout:6/254: mknod d3/da/db/d14/c50 0 2024-04-22T15:49:32.342 INFO:tasks.workunit.client.0.smithi016.stdout:6/255: fsync d3/da/db/f17 0 2024-04-22T15:49:32.343 INFO:tasks.workunit.client.0.smithi016.stdout:8/277: rename d2/db/f16 to d2/d10/d43/d4a/d28/d52/f54 0 2024-04-22T15:49:32.345 INFO:tasks.workunit.client.0.smithi016.stdout:0/256: dread dc/d13/d14/d26/d32/f33 [0,4194304] 0 2024-04-22T15:49:32.346 INFO:tasks.workunit.client.0.smithi016.stdout:0/257: chown dc/d13/f16 507629525 1 2024-04-22T15:49:32.346 INFO:tasks.workunit.client.0.smithi016.stdout:6/256: creat d3/da/dd/f51 x:0 0 0 2024-04-22T15:49:32.349 INFO:tasks.workunit.client.0.smithi016.stdout:8/278: creat d2/d48/f55 x:0 0 0 2024-04-22T15:49:32.350 INFO:tasks.workunit.client.0.smithi016.stdout:8/279: readlink d2/d10/d43/d4a/d28/l51 0 2024-04-22T15:49:32.351 INFO:tasks.workunit.client.0.smithi016.stdout:0/258: unlink dc/ce 0 2024-04-22T15:49:32.354 INFO:tasks.workunit.client.0.smithi016.stdout:8/280: mkdir d2/d10/d43/d4a/d28/d56 0 2024-04-22T15:49:32.355 INFO:tasks.workunit.client.0.smithi016.stdout:8/281: write d2/db/f20 [4374151,86752] 0 2024-04-22T15:49:32.360 INFO:tasks.workunit.client.0.smithi016.stdout:0/259: truncate dc/d13/f16 3984642 0 2024-04-22T15:49:32.360 INFO:tasks.workunit.client.0.smithi016.stdout:0/260: dread - dc/d13/d14/d26/f46 zero size 2024-04-22T15:49:32.363 INFO:tasks.workunit.client.0.smithi016.stdout:0/261: dread f4 [0,4194304] 0 2024-04-22T15:49:32.364 INFO:tasks.workunit.client.0.smithi016.stdout:0/262: fsync dc/d13/d28/f3c 0 2024-04-22T15:49:32.369 INFO:tasks.workunit.client.0.smithi016.stdout:0/263: dwrite dc/d13/d14/f18 [0,4194304] 0 2024-04-22T15:49:32.370 INFO:tasks.workunit.client.0.smithi016.stdout:6/257: getdents d3/da/db/d14 0 2024-04-22T15:49:32.372 INFO:tasks.workunit.client.0.smithi016.stdout:0/264: chown dc/d13/d22/l2b 76431698 1 2024-04-22T15:49:32.374 INFO:tasks.workunit.client.0.smithi016.stdout:6/258: rmdir d3/da/d12/d13/d38/d33/d35 39 2024-04-22T15:49:32.379 INFO:tasks.workunit.client.0.smithi016.stdout:6/259: rmdir d3/da/dd/d1b 39 2024-04-22T15:49:32.382 INFO:tasks.workunit.client.0.smithi016.stdout:6/260: stat d3/da/dd/d1b 0 2024-04-22T15:49:32.383 INFO:tasks.workunit.client.0.smithi016.stdout:0/265: dread dc/f1e [0,4194304] 0 2024-04-22T15:49:32.386 INFO:tasks.workunit.client.0.smithi016.stdout:6/261: unlink d3/c2b 0 2024-04-22T15:49:32.386 INFO:tasks.workunit.client.0.smithi016.stdout:4/190: sync 2024-04-22T15:49:32.386 INFO:tasks.workunit.client.0.smithi016.stdout:4/191: fdatasync d7/f11 0 2024-04-22T15:49:32.391 INFO:tasks.workunit.client.0.smithi016.stdout:4/192: mknod d7/d24/c35 0 2024-04-22T15:49:32.392 INFO:tasks.workunit.client.0.smithi016.stdout:8/282: truncate d2/d10/d43/d4a/d28/d3e/f42 10594091 0 2024-04-22T15:49:32.395 INFO:tasks.workunit.client.0.smithi016.stdout:0/266: link dc/d13/d14/d1d/d1f/f4c dc/d13/d28/f50 0 2024-04-22T15:49:32.399 INFO:tasks.workunit.client.0.smithi016.stdout:4/193: unlink d7/lf 0 2024-04-22T15:49:32.401 INFO:tasks.workunit.client.0.smithi016.stdout:8/283: rename d2/d10/d43/d4a/d28/d3e/l4f to d2/d10/d43/d4a/l57 0 2024-04-22T15:49:32.402 INFO:tasks.workunit.client.0.smithi016.stdout:0/267: mknod dc/d13/d14/d26/d32/c51 0 2024-04-22T15:49:32.406 INFO:tasks.workunit.client.0.smithi016.stdout:4/194: rename d7/d21/d23/d2f/f32 to d7/d24/f36 0 2024-04-22T15:49:32.412 INFO:tasks.workunit.client.0.smithi016.stdout:6/262: getdents d3/da/d12/d13/d46 0 2024-04-22T15:49:32.415 INFO:tasks.workunit.client.0.smithi016.stdout:4/195: rename d7/l2d to d7/d21/d23/l37 0 2024-04-22T15:49:32.418 INFO:tasks.workunit.client.0.smithi016.stdout:8/284: creat d2/d10/d43/d4a/d28/f58 x:0 0 0 2024-04-22T15:49:32.419 INFO:tasks.workunit.client.0.smithi016.stdout:8/285: stat d2/d10/d43/d4a/d28/f58 0 2024-04-22T15:49:32.423 INFO:tasks.workunit.client.0.smithi016.stdout:0/268: creat dc/d13/f52 x:0 0 0 2024-04-22T15:49:32.430 INFO:tasks.workunit.client.0.smithi016.stdout:8/286: dwrite d2/d10/d43/d4a/d28/d52/f54 [0,4194304] 0 2024-04-22T15:49:32.432 INFO:tasks.workunit.client.0.smithi016.stdout:0/269: fdatasync dc/d13/d14/f34 0 2024-04-22T15:49:32.435 INFO:tasks.workunit.client.0.smithi016.stdout:6/263: truncate d3/da/d12/f15 4148890 0 2024-04-22T15:49:32.436 INFO:tasks.workunit.client.0.smithi016.stdout:4/196: getdents d7/d22 0 2024-04-22T15:49:32.440 INFO:tasks.workunit.client.0.smithi016.stdout:8/287: creat d2/d10/d43/d4a/f59 x:0 0 0 2024-04-22T15:49:32.446 INFO:tasks.workunit.client.0.smithi016.stdout:4/197: mknod d7/d22/c38 0 2024-04-22T15:49:32.447 INFO:tasks.workunit.client.0.smithi016.stdout:4/198: stat d7/d22 0 2024-04-22T15:49:32.448 INFO:tasks.workunit.client.0.smithi016.stdout:0/270: getdents dc/d13/d14/d1d/d40/d4a 0 2024-04-22T15:49:32.450 INFO:tasks.workunit.client.0.smithi016.stdout:6/264: creat d3/da/db/d14/d4c/f52 x:0 0 0 2024-04-22T15:49:32.452 INFO:tasks.workunit.client.0.smithi016.stdout:4/199: mkdir d7/d21/d23/d39 0 2024-04-22T15:49:32.453 INFO:tasks.workunit.client.0.smithi016.stdout:4/200: stat d7/d21/d23/d2f/d33/l34 0 2024-04-22T15:49:32.454 INFO:tasks.workunit.client.0.smithi016.stdout:4/201: write d7/d22/f27 [4880346,99081] 0 2024-04-22T15:49:32.455 INFO:tasks.workunit.client.0.smithi016.stdout:4/202: fdatasync d7/f16 0 2024-04-22T15:49:32.456 INFO:tasks.workunit.client.0.smithi016.stdout:8/288: creat d2/f5a x:0 0 0 2024-04-22T15:49:32.457 INFO:tasks.workunit.client.0.smithi016.stdout:8/289: chown d2/d48/f49 1094389 1 2024-04-22T15:49:32.458 INFO:tasks.workunit.client.0.smithi016.stdout:4/203: rmdir d7/d22 39 2024-04-22T15:49:32.459 INFO:tasks.workunit.client.0.smithi016.stdout:0/271: dwrite dc/d13/d14/d26/d32/f33 [0,4194304] 0 2024-04-22T15:49:32.463 INFO:tasks.workunit.client.0.smithi016.stdout:4/204: mkdir d7/d21/d23/d2f/d3a 0 2024-04-22T15:49:32.466 INFO:tasks.workunit.client.0.smithi016.stdout:6/265: rename d3/da/d12/d13/d38/d33 to d3/d53 0 2024-04-22T15:49:32.467 INFO:tasks.workunit.client.0.smithi016.stdout:8/290: dwrite d2/d10/d43/d4a/d28/f4d [0,4194304] 0 2024-04-22T15:49:32.470 INFO:tasks.workunit.client.0.smithi016.stdout:4/205: symlink d7/d21/d23/l3b 0 2024-04-22T15:49:32.474 INFO:tasks.workunit.client.0.smithi016.stdout:0/272: rename dc/d13/d28/f3c to dc/d13/d14/d1d/d1f/f53 0 2024-04-22T15:49:32.477 INFO:tasks.workunit.client.0.smithi016.stdout:6/266: mknod d3/da/db/d44/c54 0 2024-04-22T15:49:32.478 INFO:tasks.workunit.client.0.smithi016.stdout:6/267: chown d3/d53 47691786 1 2024-04-22T15:49:32.480 INFO:tasks.workunit.client.0.smithi016.stdout:8/291: mknod d2/db/d1c/c5b 0 2024-04-22T15:49:32.482 INFO:tasks.workunit.client.0.smithi016.stdout:4/206: rmdir d7/d21/d23 39 2024-04-22T15:49:32.485 INFO:tasks.workunit.client.0.smithi016.stdout:0/273: unlink dc/d13/d14/d1d/d30/l41 0 2024-04-22T15:49:32.486 INFO:tasks.workunit.client.0.smithi016.stdout:4/207: dwrite d7/f11 [0,4194304] 0 2024-04-22T15:49:32.488 INFO:tasks.workunit.client.0.smithi016.stdout:6/268: creat d3/da/db/d14/d16/f55 x:0 0 0 2024-04-22T15:49:32.492 INFO:tasks.workunit.client.0.smithi016.stdout:0/274: mknod dc/d13/d14/d26/d32/c54 0 2024-04-22T15:49:32.494 INFO:tasks.workunit.client.0.smithi016.stdout:6/269: rmdir d3/da/db/d14 39 2024-04-22T15:49:32.500 INFO:tasks.workunit.client.0.smithi016.stdout:4/208: mkdir d7/d3c 0 2024-04-22T15:49:32.502 INFO:tasks.workunit.client.0.smithi016.stdout:8/292: creat d2/f5c x:0 0 0 2024-04-22T15:49:32.503 INFO:tasks.workunit.client.0.smithi016.stdout:4/209: creat d7/d24/f3d x:0 0 0 2024-04-22T15:49:32.504 INFO:tasks.workunit.client.0.smithi016.stdout:8/293: truncate d2/d10/d43/d4a/d28/f58 198793 0 2024-04-22T15:49:32.505 INFO:tasks.workunit.client.0.smithi016.stdout:8/294: write d2/d32/f35 [800122,130132] 0 2024-04-22T15:49:32.506 INFO:tasks.workunit.client.0.smithi016.stdout:8/295: truncate d2/f5a 223468 0 2024-04-22T15:49:32.509 INFO:tasks.workunit.client.0.smithi016.stdout:6/270: rename d3/d53/d35/c4b to d3/da/dd/c56 0 2024-04-22T15:49:32.513 INFO:tasks.workunit.client.0.smithi016.stdout:8/296: dwrite d2/d10/d43/d4a/d28/d44/f53 [0,4194304] 0 2024-04-22T15:49:32.514 INFO:tasks.workunit.client.0.smithi016.stdout:3/297: sync 2024-04-22T15:49:32.514 INFO:tasks.workunit.client.0.smithi016.stdout:5/231: sync 2024-04-22T15:49:32.514 INFO:tasks.workunit.client.0.smithi016.stdout:9/274: sync 2024-04-22T15:49:32.514 INFO:tasks.workunit.client.0.smithi016.stdout:6/271: mkdir d3/da/dd/d1b/d57 0 2024-04-22T15:49:32.517 INFO:tasks.workunit.client.0.smithi016.stdout:4/210: getdents d7/d21/d23 0 2024-04-22T15:49:32.517 INFO:tasks.workunit.client.0.smithi016.stdout:9/275: write d9/d20/f40 [517989,86121] 0 2024-04-22T15:49:32.522 INFO:tasks.workunit.client.0.smithi016.stdout:0/275: dread dc/d13/f17 [0,4194304] 0 2024-04-22T15:49:32.523 INFO:tasks.workunit.client.0.smithi016.stdout:8/297: dwrite d2/d10/d43/d4a/d28/f38 [0,4194304] 0 2024-04-22T15:49:32.524 INFO:tasks.workunit.client.0.smithi016.stdout:6/272: write d3/da/db/d14/d16/f20 [1073837,112749] 0 2024-04-22T15:49:32.524 INFO:tasks.workunit.client.0.smithi016.stdout:5/232: rename df/d15/d1a/f31 to df/d15/d1a/d28/d33/f40 0 2024-04-22T15:49:32.525 INFO:tasks.workunit.client.0.smithi016.stdout:5/233: fdatasync df/f10 0 2024-04-22T15:49:32.527 INFO:tasks.workunit.client.0.smithi016.stdout:0/276: truncate dc/d13/d14/d26/d32/f4e 560768 0 2024-04-22T15:49:32.535 INFO:tasks.workunit.client.0.smithi016.stdout:5/234: dwrite df/d22/f23 [0,4194304] 0 2024-04-22T15:49:32.537 INFO:tasks.workunit.client.0.smithi016.stdout:0/277: dwrite dc/d13/d14/d1d/d1f/f39 [0,4194304] 0 2024-04-22T15:49:32.538 INFO:tasks.workunit.client.0.smithi016.stdout:8/298: mknod d2/d10/d47/c5d 0 2024-04-22T15:49:32.538 INFO:tasks.workunit.client.0.smithi016.stdout:0/278: chown dc/d13/d14/d26/f4f 738326 1 2024-04-22T15:49:32.540 INFO:tasks.workunit.client.0.smithi016.stdout:0/279: read - dc/d13/d14/d26/f4f zero size 2024-04-22T15:49:32.541 INFO:tasks.workunit.client.0.smithi016.stdout:0/280: readlink dc/d13/d14/d1d/d30/l3e 0 2024-04-22T15:49:32.542 INFO:tasks.workunit.client.0.smithi016.stdout:6/273: mkdir d3/da/db/d14/d4c/d58 0 2024-04-22T15:49:32.543 INFO:tasks.workunit.client.0.smithi016.stdout:6/274: read - d3/da/db/d14/d16/f32 zero size 2024-04-22T15:49:32.551 INFO:tasks.workunit.client.0.smithi016.stdout:3/298: creat d7/f5e x:0 0 0 2024-04-22T15:49:32.554 INFO:tasks.workunit.client.0.smithi016.stdout:5/235: symlink df/d22/l41 0 2024-04-22T15:49:32.557 INFO:tasks.workunit.client.0.smithi016.stdout:3/299: dwrite d7/d17/d18/f3f [0,4194304] 0 2024-04-22T15:49:32.557 INFO:tasks.workunit.client.0.smithi016.stdout:9/276: rename d9/de/d23/l4b to d9/de/d23/d24/l50 0 2024-04-22T15:49:32.563 INFO:tasks.workunit.client.0.smithi016.stdout:5/236: dwrite df/d15/d1a/d28/f32 [0,4194304] 0 2024-04-22T15:49:32.565 INFO:tasks.workunit.client.0.smithi016.stdout:5/237: fsync df/d22/f25 0 2024-04-22T15:49:32.568 INFO:tasks.workunit.client.0.smithi016.stdout:6/275: dwrite d3/da/d12/d13/d46/f4f [0,4194304] 0 2024-04-22T15:49:32.570 INFO:tasks.workunit.client.0.smithi016.stdout:8/299: fdatasync d2/f24 0 2024-04-22T15:49:32.572 INFO:tasks.workunit.client.0.smithi016.stdout:3/300: creat d7/d17/d18/d32/d3c/d51/f5f x:0 0 0 2024-04-22T15:49:32.576 INFO:tasks.workunit.client.0.smithi016.stdout:4/211: getdents d7/d21/d23/d2f 0 2024-04-22T15:49:32.577 INFO:tasks.workunit.client.0.smithi016.stdout:4/212: chown d7/l13 0 1 2024-04-22T15:49:32.583 INFO:tasks.workunit.client.0.smithi016.stdout:5/238: rename df/f30 to df/d22/f42 0 2024-04-22T15:49:32.587 INFO:tasks.workunit.client.0.smithi016.stdout:9/277: symlink d9/de/d23/d24/d25/d36/l51 0 2024-04-22T15:49:32.589 INFO:tasks.workunit.client.0.smithi016.stdout:5/239: dwrite df/d15/d1a/d28/f2b [0,4194304] 0 2024-04-22T15:49:32.589 INFO:tasks.workunit.client.0.smithi016.stdout:3/301: unlink d7/d17/d18/d32/d3c/d47/c2d 0 2024-04-22T15:49:32.591 INFO:tasks.workunit.client.0.smithi016.stdout:3/302: read d7/f22 [2359198,83546] 0 2024-04-22T15:49:32.592 INFO:tasks.workunit.client.0.smithi016.stdout:3/303: readlink d7/d17/d18/d32/d42/l5d 0 2024-04-22T15:49:32.593 INFO:tasks.workunit.client.0.smithi016.stdout:8/300: rename d2/db/c1b to d2/d10/c5e 0 2024-04-22T15:49:32.599 INFO:tasks.workunit.client.0.smithi016.stdout:6/276: unlink d3/f31 0 2024-04-22T15:49:32.603 INFO:tasks.workunit.client.0.smithi016.stdout:5/240: unlink df/d22/f23 0 2024-04-22T15:49:32.603 INFO:tasks.workunit.client.0.smithi016.stdout:5/241: chown f2 935596 1 2024-04-22T15:49:32.610 INFO:tasks.workunit.client.0.smithi016.stdout:3/304: rename d7/l39 to d7/d17/d18/d32/d3c/d51/l60 0 2024-04-22T15:49:32.615 INFO:tasks.workunit.client.0.smithi016.stdout:2/258: sync 2024-04-22T15:49:32.615 INFO:tasks.workunit.client.0.smithi016.stdout:1/486: sync 2024-04-22T15:49:32.615 INFO:tasks.workunit.client.0.smithi016.stdout:7/247: sync 2024-04-22T15:49:32.616 INFO:tasks.workunit.client.0.smithi016.stdout:6/277: unlink d3/da/db/d14/d16/f32 0 2024-04-22T15:49:32.618 INFO:tasks.workunit.client.0.smithi016.stdout:9/278: rename d9/de/d23/d24/d25/f32 to d9/de/d23/d24/d25/f52 0 2024-04-22T15:49:32.625 INFO:tasks.workunit.client.0.smithi016.stdout:8/301: rmdir d2/db 39 2024-04-22T15:49:32.629 INFO:tasks.workunit.client.0.smithi016.stdout:2/259: mknod d2/d23/d47/c61 0 2024-04-22T15:49:32.630 INFO:tasks.workunit.client.0.smithi016.stdout:2/260: chown d2 3383 1 2024-04-22T15:49:32.630 INFO:tasks.workunit.client.0.smithi016.stdout:1/487: dread d3/d4/d15/f16 [0,4194304] 0 2024-04-22T15:49:32.630 INFO:tasks.workunit.client.0.smithi016.stdout:2/261: chown d2/d1a/l58 640 1 2024-04-22T15:49:32.634 INFO:tasks.workunit.client.0.smithi016.stdout:4/213: truncate d7/f8 2936948 0 2024-04-22T15:49:32.636 INFO:tasks.workunit.client.0.smithi016.stdout:5/242: rename df/d15/d1a/d28/l26 to df/d15/d1a/l43 0 2024-04-22T15:49:32.648 INFO:tasks.workunit.client.0.smithi016.stdout:3/305: mknod d7/c61 0 2024-04-22T15:49:32.648 INFO:tasks.workunit.client.0.smithi016.stdout:7/248: creat d3/dc/d1b/d31/d5f/f61 x:0 0 0 2024-04-22T15:49:32.649 INFO:tasks.workunit.client.0.smithi016.stdout:4/214: mknod d7/d21/d23/c3e 0 2024-04-22T15:49:32.649 INFO:tasks.workunit.client.0.smithi016.stdout:5/243: rename l7 to df/d15/d1a/d28/d33/l44 0 2024-04-22T15:49:32.651 INFO:tasks.workunit.client.0.smithi016.stdout:8/302: readlink d2/db/l1d 0 2024-04-22T15:49:32.653 INFO:tasks.workunit.client.0.smithi016.stdout:6/278: creat d3/f59 x:0 0 0 2024-04-22T15:49:32.653 INFO:tasks.workunit.client.0.smithi016.stdout:7/249: write d3/dc/d1b/d5a/f1f [2746615,77788] 0 2024-04-22T15:49:32.657 INFO:tasks.workunit.client.0.smithi016.stdout:3/306: getdents d7/d17/d18/d27/d5c 0 2024-04-22T15:49:32.659 INFO:tasks.workunit.client.0.smithi016.stdout:8/303: write d2/d10/d43/d4a/d28/f58 [279533,124359] 0 2024-04-22T15:49:32.661 INFO:tasks.workunit.client.0.smithi016.stdout:2/262: dwrite d2/de/d2a/f42 [0,4194304] 0 2024-04-22T15:49:32.661 INFO:tasks.workunit.client.0.smithi016.stdout:3/307: fdatasync d7/d17/d18/f4e 0 2024-04-22T15:49:32.665 INFO:tasks.workunit.client.0.smithi016.stdout:4/215: link d7/l1a d7/d21/d23/d2f/l3f 0 2024-04-22T15:49:32.666 INFO:tasks.workunit.client.0.smithi016.stdout:5/244: dwrite f2 [0,4194304] 0 2024-04-22T15:49:32.667 INFO:tasks.workunit.client.0.smithi016.stdout:7/250: rename d3/dc/d1b/d5a/d9/df/l1d to d3/dc/d1b/l62 0 2024-04-22T15:49:32.669 INFO:tasks.workunit.client.0.smithi016.stdout:6/279: creat d3/da/dd/f5a x:0 0 0 2024-04-22T15:49:32.670 INFO:tasks.workunit.client.0.smithi016.stdout:6/280: fdatasync d3/da/dd/f5a 0 2024-04-22T15:49:32.671 INFO:tasks.workunit.client.0.smithi016.stdout:3/308: read d7/f8 [2273149,99801] 0 2024-04-22T15:49:32.671 INFO:tasks.workunit.client.0.smithi016.stdout:8/304: creat d2/d10/d43/d4a/f5f x:0 0 0 2024-04-22T15:49:32.673 INFO:tasks.workunit.client.0.smithi016.stdout:6/281: chown d3/da/db/d14/d16/f55 39710159 1 2024-04-22T15:49:32.673 INFO:tasks.workunit.client.0.smithi016.stdout:1/488: dread d3/d4/f24 [0,4194304] 0 2024-04-22T15:49:32.675 INFO:tasks.workunit.client.0.smithi016.stdout:2/263: creat d2/d51/f62 x:0 0 0 2024-04-22T15:49:32.676 INFO:tasks.workunit.client.0.smithi016.stdout:6/282: write d3/da/d12/d13/d46/f4f [5145591,89259] 0 2024-04-22T15:49:32.680 INFO:tasks.workunit.client.0.smithi016.stdout:6/283: read d3/f6 [643451,34079] 0 2024-04-22T15:49:32.680 INFO:tasks.workunit.client.0.smithi016.stdout:5/245: chown df/l38 12 1 2024-04-22T15:49:32.683 INFO:tasks.workunit.client.0.smithi016.stdout:5/246: stat df/d15/d1a/l1c 0 2024-04-22T15:49:32.683 INFO:tasks.workunit.client.0.smithi016.stdout:7/251: creat d3/dc/d1b/d31/d5f/f63 x:0 0 0 2024-04-22T15:49:32.684 INFO:tasks.workunit.client.0.smithi016.stdout:3/309: creat d7/d17/d18/d32/d3c/d47/f62 x:0 0 0 2024-04-22T15:49:32.684 INFO:tasks.workunit.client.0.smithi016.stdout:7/252: write d3/dc/d1b/d31/f48 [5171389,83958] 0 2024-04-22T15:49:32.697 INFO:tasks.workunit.client.0.smithi016.stdout:1/489: symlink d3/d4/d19/d20/d9d/la6 0 2024-04-22T15:49:32.699 INFO:tasks.workunit.client.0.smithi016.stdout:8/305: truncate d2/d10/d43/d4a/d28/d3e/f42 11405937 0 2024-04-22T15:49:32.700 INFO:tasks.workunit.client.0.smithi016.stdout:8/306: dread - d2/d48/f55 zero size 2024-04-22T15:49:32.701 INFO:tasks.workunit.client.0.smithi016.stdout:0/281: sync 2024-04-22T15:49:32.701 INFO:tasks.workunit.client.0.smithi016.stdout:7/253: mkdir d3/dc/d1b/d5a/d59/d64 0 2024-04-22T15:49:32.702 INFO:tasks.workunit.client.0.smithi016.stdout:3/310: mkdir d7/d17/d18/d32/d42/d63 0 2024-04-22T15:49:32.702 INFO:tasks.workunit.client.0.smithi016.stdout:7/254: write d3/dc/d1b/d5a/f46 [17885,125602] 0 2024-04-22T15:49:32.703 INFO:tasks.workunit.client.0.smithi016.stdout:3/311: readlink d7/d1b/l1c 0 2024-04-22T15:49:32.710 INFO:tasks.workunit.client.0.smithi016.stdout:4/216: dread d7/f8 [0,4194304] 0 2024-04-22T15:49:32.711 INFO:tasks.workunit.client.0.smithi016.stdout:7/255: dwrite d3/dc/d1b/d5a/d9/f25 [0,4194304] 0 2024-04-22T15:49:32.712 INFO:tasks.workunit.client.0.smithi016.stdout:1/490: symlink d3/d28/la7 0 2024-04-22T15:49:32.714 INFO:tasks.workunit.client.0.smithi016.stdout:8/307: creat d2/d30/f60 x:0 0 0 2024-04-22T15:49:32.719 INFO:tasks.workunit.client.0.smithi016.stdout:3/312: rmdir d7/d17/d18/d32/d3c/d51 39 2024-04-22T15:49:32.719 INFO:tasks.workunit.client.0.smithi016.stdout:7/256: dwrite d3/dc/d1b/d5a/d9/f25 [0,4194304] 0 2024-04-22T15:49:32.722 INFO:tasks.workunit.client.0.smithi016.stdout:7/257: dread - d3/dc/d1b/d31/d5f/f61 zero size 2024-04-22T15:49:32.723 INFO:tasks.workunit.client.0.smithi016.stdout:7/258: fdatasync d3/dc/d1b/d5a/f4b 0 2024-04-22T15:49:32.726 INFO:tasks.workunit.client.0.smithi016.stdout:5/247: creat df/d15/f45 x:0 0 0 2024-04-22T15:49:32.735 INFO:tasks.workunit.client.0.smithi016.stdout:0/282: mkdir dc/d13/d14/d55 0 2024-04-22T15:49:32.735 INFO:tasks.workunit.client.0.smithi016.stdout:1/491: read d3/d4/d3a/d6b/d77/f99 [45464,76194] 0 2024-04-22T15:49:32.736 INFO:tasks.workunit.client.0.smithi016.stdout:1/492: dread - d3/d4/da/fa2 zero size 2024-04-22T15:49:32.737 INFO:tasks.workunit.client.0.smithi016.stdout:3/313: unlink d7/c4c 0 2024-04-22T15:49:32.737 INFO:tasks.workunit.client.0.smithi016.stdout:1/493: read - d3/d4/d15/d38/d63/d67/f79 zero size 2024-04-22T15:49:32.739 INFO:tasks.workunit.client.0.smithi016.stdout:3/314: dread d7/d17/d18/d27/f53 [0,4194304] 0 2024-04-22T15:49:32.739 INFO:tasks.workunit.client.0.smithi016.stdout:6/284: rename d3/c26 to d3/da/dd/c5b 0 2024-04-22T15:49:32.748 INFO:tasks.workunit.client.0.smithi016.stdout:8/308: rename d2/d30/f3a to d2/d10/f61 0 2024-04-22T15:49:32.749 INFO:tasks.workunit.client.0.smithi016.stdout:8/309: fsync d2/d10/f39 0 2024-04-22T15:49:32.752 INFO:tasks.workunit.client.0.smithi016.stdout:6/285: mknod d3/da/d12/d13/d46/c5c 0 2024-04-22T15:49:32.752 INFO:tasks.workunit.client.0.smithi016.stdout:6/286: fdatasync d3/da/d12/f1c 0 2024-04-22T15:49:32.755 INFO:tasks.workunit.client.0.smithi016.stdout:5/248: symlink df/d15/d1a/l46 0 2024-04-22T15:49:32.756 INFO:tasks.workunit.client.0.smithi016.stdout:5/249: truncate df/d15/d1a/d28/f2c 921425 0 2024-04-22T15:49:32.759 INFO:tasks.workunit.client.0.smithi016.stdout:0/283: rename dc/f27 to dc/d13/d14/d1d/d40/f56 0 2024-04-22T15:49:32.761 INFO:tasks.workunit.client.0.smithi016.stdout:0/284: write dc/d13/d14/d26/f46 [118229,102006] 0 2024-04-22T15:49:32.762 INFO:tasks.workunit.client.0.smithi016.stdout:3/315: symlink d7/d17/d18/d32/d3c/d51/l64 0 2024-04-22T15:49:32.763 INFO:tasks.workunit.client.0.smithi016.stdout:3/316: readlink d7/d17/d18/l19 0 2024-04-22T15:49:32.764 INFO:tasks.workunit.client.0.smithi016.stdout:7/259: link d3/dc/d1b/d5a/d4a/f4f d3/dc/f65 0 2024-04-22T15:49:32.768 INFO:tasks.workunit.client.0.smithi016.stdout:0/285: mkdir dc/d13/d14/d1d/d57 0 2024-04-22T15:49:32.770 INFO:tasks.workunit.client.0.smithi016.stdout:7/260: mknod d3/dc/d1b/d5a/d9/c66 0 2024-04-22T15:49:32.772 INFO:tasks.workunit.client.0.smithi016.stdout:6/287: rename d3/da/d12/d13/c3a to d3/da/d12/d13/d46/c5d 0 2024-04-22T15:49:32.773 INFO:tasks.workunit.client.0.smithi016.stdout:3/317: symlink d7/d17/d18/d32/d42/d63/l65 0 2024-04-22T15:49:32.774 INFO:tasks.workunit.client.0.smithi016.stdout:6/288: read - d3/da/d12/f40 zero size 2024-04-22T15:49:32.775 INFO:tasks.workunit.client.0.smithi016.stdout:7/261: mknod d3/d32/c67 0 2024-04-22T15:49:32.775 INFO:tasks.workunit.client.0.smithi016.stdout:3/318: fdatasync d7/d17/d18/d27/f4f 0 2024-04-22T15:49:32.776 INFO:tasks.workunit.client.0.smithi016.stdout:6/289: chown d3/da/db/d14/d4c/d58 20 1 2024-04-22T15:49:32.777 INFO:tasks.workunit.client.0.smithi016.stdout:0/286: dwrite dc/d13/d14/d1d/d30/f44 [0,4194304] 0 2024-04-22T15:49:32.779 INFO:tasks.workunit.client.0.smithi016.stdout:7/262: fdatasync d3/dc/d1b/d5a/f56 0 2024-04-22T15:49:32.785 INFO:tasks.workunit.client.0.smithi016.stdout:6/290: symlink d3/da/d12/d13/l5e 0 2024-04-22T15:49:32.787 INFO:tasks.workunit.client.0.smithi016.stdout:5/250: rename df/c12 to df/d15/c47 0 2024-04-22T15:49:32.788 INFO:tasks.workunit.client.0.smithi016.stdout:5/251: chown df/d15 34330411 1 2024-04-22T15:49:32.789 INFO:tasks.workunit.client.0.smithi016.stdout:7/263: mknod d3/dc/d1b/d5a/d4a/c68 0 2024-04-22T15:49:32.790 INFO:tasks.workunit.client.0.smithi016.stdout:6/291: creat d3/da/db/d14/d16/f5f x:0 0 0 2024-04-22T15:49:32.792 INFO:tasks.workunit.client.0.smithi016.stdout:6/292: write d3/da/dd/f5a [465513,90464] 0 2024-04-22T15:49:32.796 INFO:tasks.workunit.client.0.smithi016.stdout:5/252: dwrite f2 [0,4194304] 0 2024-04-22T15:49:32.800 INFO:tasks.workunit.client.0.smithi016.stdout:5/253: symlink df/d15/d1a/d28/d33/l48 0 2024-04-22T15:49:32.802 INFO:tasks.workunit.client.0.smithi016.stdout:5/254: fdatasync df/d22/f3b 0 2024-04-22T15:49:32.809 INFO:tasks.workunit.client.0.smithi016.stdout:6/293: dwrite d3/da/db/d14/d16/f3d [0,4194304] 0 2024-04-22T15:49:32.811 INFO:tasks.workunit.client.0.smithi016.stdout:4/217: dwrite d7/f8 [0,4194304] 0 2024-04-22T15:49:32.821 INFO:tasks.workunit.client.0.smithi016.stdout:4/218: creat d7/d22/f40 x:0 0 0 2024-04-22T15:49:32.823 INFO:tasks.workunit.client.0.smithi016.stdout:6/294: dwrite d3/da/db/d14/d4c/f52 [0,4194304] 0 2024-04-22T15:49:32.830 INFO:tasks.workunit.client.0.smithi016.stdout:4/219: mkdir d7/d24/d41 0 2024-04-22T15:49:32.830 INFO:tasks.workunit.client.0.smithi016.stdout:4/220: write d7/f1d [4706226,126693] 0 2024-04-22T15:49:32.831 INFO:tasks.workunit.client.0.smithi016.stdout:6/295: mknod d3/da/db/c60 0 2024-04-22T15:49:32.840 INFO:tasks.workunit.client.0.smithi016.stdout:4/221: rmdir d7/d21/d23/d2f/d3a 0 2024-04-22T15:49:32.841 INFO:tasks.workunit.client.0.smithi016.stdout:4/222: chown d7/d22/f40 252301 1 2024-04-22T15:49:32.842 INFO:tasks.workunit.client.0.smithi016.stdout:6/296: getdents d3/da/db/d14/d4c 0 2024-04-22T15:49:32.844 INFO:tasks.workunit.client.0.smithi016.stdout:0/287: rmdir dc 39 2024-04-22T15:49:32.846 INFO:tasks.workunit.client.0.smithi016.stdout:4/223: mkdir d7/d21/d42 0 2024-04-22T15:49:32.847 INFO:tasks.workunit.client.0.smithi016.stdout:4/224: write d7/f1e [4790934,10628] 0 2024-04-22T15:49:32.847 INFO:tasks.workunit.client.0.smithi016.stdout:4/225: stat d7/ld 0 2024-04-22T15:49:32.849 INFO:tasks.workunit.client.0.smithi016.stdout:9/279: sync 2024-04-22T15:49:32.857 INFO:tasks.workunit.client.0.smithi016.stdout:9/280: creat d9/de/d23/d24/f53 x:0 0 0 2024-04-22T15:49:32.859 INFO:tasks.workunit.client.0.smithi016.stdout:0/288: rename dc/d13/d22/l3f to dc/d13/d14/d1d/d57/l58 0 2024-04-22T15:49:32.862 INFO:tasks.workunit.client.0.smithi016.stdout:7/264: unlink d3/dc/d1b/d5a/d9/df/l2e 0 2024-04-22T15:49:32.862 INFO:tasks.workunit.client.0.smithi016.stdout:4/226: mknod d7/d3c/c43 0 2024-04-22T15:49:32.863 INFO:tasks.workunit.client.0.smithi016.stdout:4/227: readlink d7/l13 0 2024-04-22T15:49:32.866 INFO:tasks.workunit.client.0.smithi016.stdout:9/281: rename d9/c1f to d9/de/d23/d24/c54 0 2024-04-22T15:49:32.870 INFO:tasks.workunit.client.0.smithi016.stdout:5/255: write df/d22/f42 [2036679,98214] 0 2024-04-22T15:49:32.870 INFO:tasks.workunit.client.0.smithi016.stdout:7/265: chown d3/dc/d1b/d5a/c5e 32296 1 2024-04-22T15:49:32.871 INFO:tasks.workunit.client.0.smithi016.stdout:4/228: creat d7/d3c/f44 x:0 0 0 2024-04-22T15:49:32.872 INFO:tasks.workunit.client.0.smithi016.stdout:4/229: chown d7/d21/d42 308 1 2024-04-22T15:49:32.873 INFO:tasks.workunit.client.0.smithi016.stdout:4/230: write d7/f1d [2304223,118945] 0 2024-04-22T15:49:32.876 INFO:tasks.workunit.client.0.smithi016.stdout:5/256: chown df/d15/d1a/l3e 932848 1 2024-04-22T15:49:32.879 INFO:tasks.workunit.client.0.smithi016.stdout:4/231: mkdir d7/d21/d45 0 2024-04-22T15:49:32.880 INFO:tasks.workunit.client.0.smithi016.stdout:4/232: truncate d7/d24/f3d 17121 0 2024-04-22T15:49:32.883 INFO:tasks.workunit.client.0.smithi016.stdout:9/282: dread d9/de/f31 [0,4194304] 0 2024-04-22T15:49:32.883 INFO:tasks.workunit.client.0.smithi016.stdout:6/297: write d3/da/d12/ff [3279026,99913] 0 2024-04-22T15:49:32.884 INFO:tasks.workunit.client.0.smithi016.stdout:5/257: symlink df/l49 0 2024-04-22T15:49:32.885 INFO:tasks.workunit.client.0.smithi016.stdout:5/258: dread - df/d15/f45 zero size 2024-04-22T15:49:32.885 INFO:tasks.workunit.client.0.smithi016.stdout:5/259: readlink df/l38 0 2024-04-22T15:49:32.886 INFO:tasks.workunit.client.0.smithi016.stdout:4/233: fsync d7/d24/f36 0 2024-04-22T15:49:32.887 INFO:tasks.workunit.client.0.smithi016.stdout:9/283: chown l6 47534892 1 2024-04-22T15:49:32.888 INFO:tasks.workunit.client.0.smithi016.stdout:0/289: readlink dc/d13/d14/d1d/d57/l58 0 2024-04-22T15:49:32.891 INFO:tasks.workunit.client.0.smithi016.stdout:0/290: write dc/d13/d14/d26/f4f [788715,90956] 0 2024-04-22T15:49:32.892 INFO:tasks.workunit.client.0.smithi016.stdout:6/298: dwrite d3/da/db/f17 [0,4194304] 0 2024-04-22T15:49:32.897 INFO:tasks.workunit.client.0.smithi016.stdout:0/291: chown dc/d13/d14/d1d/l4d 90 1 2024-04-22T15:49:32.898 INFO:tasks.workunit.client.0.smithi016.stdout:4/234: rmdir d7 39 2024-04-22T15:49:32.898 INFO:tasks.workunit.client.0.smithi016.stdout:0/292: truncate dc/f1e 4716019 0 2024-04-22T15:49:32.901 INFO:tasks.workunit.client.0.smithi016.stdout:7/266: truncate d3/dc/d1b/d5a/d9/f2a 1333150 0 2024-04-22T15:49:32.901 INFO:tasks.workunit.client.0.smithi016.stdout:9/284: dread d9/f1b [0,4194304] 0 2024-04-22T15:49:32.902 INFO:tasks.workunit.client.0.smithi016.stdout:6/299: creat d3/da/db/f61 x:0 0 0 2024-04-22T15:49:32.903 INFO:tasks.workunit.client.0.smithi016.stdout:7/267: chown d3/dc/d1b/d5a/d9/df 11 1 2024-04-22T15:49:32.903 INFO:tasks.workunit.client.0.smithi016.stdout:6/300: dread - d3/f59 zero size 2024-04-22T15:49:32.905 INFO:tasks.workunit.client.0.smithi016.stdout:4/235: symlink d7/d3c/l46 0 2024-04-22T15:49:32.906 INFO:tasks.workunit.client.0.smithi016.stdout:4/236: rename d7/d21 to d7/d21/d23/d2f/d33/d47 22 2024-04-22T15:49:32.908 INFO:tasks.workunit.client.0.smithi016.stdout:7/268: unlink d3/dc/d1b/l20 0 2024-04-22T15:49:32.911 INFO:tasks.workunit.client.0.smithi016.stdout:7/269: mkdir d3/dc/d1b/d5a/d69 0 2024-04-22T15:49:32.914 INFO:tasks.workunit.client.0.smithi016.stdout:0/293: write dc/d13/d28/f50 [916295,72860] 0 2024-04-22T15:49:32.916 INFO:tasks.workunit.client.0.smithi016.stdout:7/270: creat d3/dc/d1b/d5a/d4a/f6a x:0 0 0 2024-04-22T15:49:32.916 INFO:tasks.workunit.client.0.smithi016.stdout:0/294: chown dc/d13/d28/c35 67289 1 2024-04-22T15:49:32.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:32 smithi016 ceph-mon[26620]: from='client.14672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:49:32.917 INFO:tasks.workunit.client.0.smithi016.stdout:7/271: dread - d3/dc/d1b/d5a/d4a/f4f zero size 2024-04-22T15:49:32.918 INFO:tasks.workunit.client.0.smithi016.stdout:9/285: dread d9/de/d23/f34 [0,4194304] 0 2024-04-22T15:49:32.920 INFO:tasks.workunit.client.0.smithi016.stdout:7/272: read d3/f34 [4030593,123468] 0 2024-04-22T15:49:32.922 INFO:tasks.workunit.client.0.smithi016.stdout:4/237: dread d7/d22/f27 [0,4194304] 0 2024-04-22T15:49:32.922 INFO:tasks.workunit.client.0.smithi016.stdout:0/295: mkdir dc/d13/d14/d1d/d40/d4a/d59 0 2024-04-22T15:49:32.923 INFO:tasks.workunit.client.0.smithi016.stdout:7/273: write d3/dc/d1b/d31/f48 [1895034,54920] 0 2024-04-22T15:49:32.923 INFO:tasks.workunit.client.0.smithi016.stdout:9/286: mknod d9/de/d23/d24/c55 0 2024-04-22T15:49:32.924 INFO:tasks.workunit.client.0.smithi016.stdout:6/301: dwrite d3/da/d12/f15 [0,4194304] 0 2024-04-22T15:49:32.927 INFO:tasks.workunit.client.0.smithi016.stdout:0/296: dread dc/d13/d14/d26/d32/f4e [0,4194304] 0 2024-04-22T15:49:32.928 INFO:tasks.workunit.client.0.smithi016.stdout:4/238: chown d7/d21/d23/d2f/l3f 110 1 2024-04-22T15:49:32.932 INFO:tasks.workunit.client.0.smithi016.stdout:0/297: symlink dc/d13/d22/l5a 0 2024-04-22T15:49:32.932 INFO:tasks.workunit.client.0.smithi016.stdout:7/274: truncate d3/dc/f65 644317 0 2024-04-22T15:49:32.936 INFO:tasks.workunit.client.0.smithi016.stdout:9/287: dwrite d9/de/f31 [0,4194304] 0 2024-04-22T15:49:32.937 INFO:tasks.workunit.client.0.smithi016.stdout:4/239: creat d7/d21/d23/d39/f48 x:0 0 0 2024-04-22T15:49:32.937 INFO:tasks.workunit.client.0.smithi016.stdout:0/298: rmdir dc/d13/d14 39 2024-04-22T15:49:32.941 INFO:tasks.workunit.client.0.smithi016.stdout:6/302: getdents d3/da/db/d14/d16 0 2024-04-22T15:49:32.943 INFO:tasks.workunit.client.0.smithi016.stdout:7/275: rename d3/d32/d4c/f54 to d3/dc/d22/f6b 0 2024-04-22T15:49:32.944 INFO:tasks.workunit.client.0.smithi016.stdout:6/303: truncate d3/da/db/d14/d16/f55 560452 0 2024-04-22T15:49:32.944 INFO:tasks.workunit.client.0.smithi016.stdout:9/288: dwrite d9/de/d23/d24/f39 [0,4194304] 0 2024-04-22T15:49:32.945 INFO:tasks.workunit.client.0.smithi016.stdout:9/289: write f2 [630080,21866] 0 2024-04-22T15:49:32.950 INFO:tasks.workunit.client.0.smithi016.stdout:4/240: mkdir d7/d21/d45/d49 0 2024-04-22T15:49:32.953 INFO:tasks.workunit.client.0.smithi016.stdout:6/304: creat d3/da/dd/d1b/f62 x:0 0 0 2024-04-22T15:49:32.954 INFO:tasks.workunit.client.0.smithi016.stdout:9/290: fsync d9/de/f16 0 2024-04-22T15:49:32.955 INFO:tasks.workunit.client.0.smithi016.stdout:9/291: truncate d9/f3b 1661414 0 2024-04-22T15:49:32.956 INFO:tasks.workunit.client.0.smithi016.stdout:9/292: dread - d9/de/f28 zero size 2024-04-22T15:49:32.959 INFO:tasks.workunit.client.0.smithi016.stdout:0/299: creat dc/d13/d14/d26/d32/f5b x:0 0 0 2024-04-22T15:49:32.959 INFO:tasks.workunit.client.0.smithi016.stdout:6/305: dread d3/da/db/d14/d16/f20 [0,4194304] 0 2024-04-22T15:49:32.960 INFO:tasks.workunit.client.0.smithi016.stdout:4/241: mknod d7/d21/d23/d39/c4a 0 2024-04-22T15:49:32.960 INFO:tasks.workunit.client.0.smithi016.stdout:4/242: chown d7/f14 181961 1 2024-04-22T15:49:32.961 INFO:tasks.workunit.client.0.smithi016.stdout:7/276: creat d3/dc/d1b/d5a/d59/d64/f6c x:0 0 0 2024-04-22T15:49:32.965 INFO:tasks.workunit.client.0.smithi016.stdout:0/300: rmdir dc/d13 39 2024-04-22T15:49:32.965 INFO:tasks.workunit.client.0.smithi016.stdout:6/306: mknod d3/da/d12/d13/d46/c63 0 2024-04-22T15:49:32.967 INFO:tasks.workunit.client.0.smithi016.stdout:4/243: rename d7/cc to d7/d22/c4b 0 2024-04-22T15:49:32.967 INFO:tasks.workunit.client.0.smithi016.stdout:7/277: creat d3/dc/d1b/f6d x:0 0 0 2024-04-22T15:49:32.969 INFO:tasks.workunit.client.0.smithi016.stdout:0/301: stat dc/d13/d14/d55 0 2024-04-22T15:49:32.970 INFO:tasks.workunit.client.0.smithi016.stdout:6/307: mknod d3/da/db/d44/c64 0 2024-04-22T15:49:32.970 INFO:tasks.workunit.client.0.smithi016.stdout:4/244: rmdir d7/d21/d45 39 2024-04-22T15:49:32.970 INFO:tasks.workunit.client.0.smithi016.stdout:6/308: write d3/da/f45 [557443,73838] 0 2024-04-22T15:49:32.973 INFO:tasks.workunit.client.0.smithi016.stdout:0/302: fsync f4 0 2024-04-22T15:49:32.975 INFO:tasks.workunit.client.0.smithi016.stdout:9/293: getdents d9/de 0 2024-04-22T15:49:32.977 INFO:tasks.workunit.client.0.smithi016.stdout:6/309: symlink d3/da/dd/d1b/l65 0 2024-04-22T15:49:32.977 INFO:tasks.workunit.client.0.smithi016.stdout:6/310: read d3/da/db/f17 [2911657,70835] 0 2024-04-22T15:49:32.978 INFO:tasks.workunit.client.0.smithi016.stdout:6/311: readlink d3/l7 0 2024-04-22T15:49:32.978 INFO:tasks.workunit.client.0.smithi016.stdout:0/303: symlink dc/d13/d14/d1d/d30/l5c 0 2024-04-22T15:49:32.979 INFO:tasks.workunit.client.0.smithi016.stdout:0/304: write dc/d13/d22/f2e [274506,68277] 0 2024-04-22T15:49:32.979 INFO:tasks.workunit.client.0.smithi016.stdout:7/278: dread d3/dc/d1b/d5a/d9/f2a [0,4194304] 0 2024-04-22T15:49:32.980 INFO:tasks.workunit.client.0.smithi016.stdout:6/312: dread d3/da/d12/f1c [0,4194304] 0 2024-04-22T15:49:32.981 INFO:tasks.workunit.client.0.smithi016.stdout:2/264: sync 2024-04-22T15:49:32.982 INFO:tasks.workunit.client.0.smithi016.stdout:6/313: truncate d3/da/db/d14/d4c/f52 4358943 0 2024-04-22T15:49:32.983 INFO:tasks.workunit.client.0.smithi016.stdout:6/314: truncate d3/da/d12/f1c 229075 0 2024-04-22T15:49:32.986 INFO:tasks.workunit.client.0.smithi016.stdout:7/279: chown d3/d32/f3d 3590 1 2024-04-22T15:49:32.987 INFO:tasks.workunit.client.0.smithi016.stdout:0/305: rmdir dc/d13/d14/d1d 39 2024-04-22T15:49:32.989 INFO:tasks.workunit.client.0.smithi016.stdout:2/265: mknod d2/d23/d27/d34/c63 0 2024-04-22T15:49:32.991 INFO:tasks.workunit.client.0.smithi016.stdout:7/280: dread d3/dc/d1b/d5a/f5b [0,4194304] 0 2024-04-22T15:49:32.991 INFO:tasks.workunit.client.0.smithi016.stdout:4/245: link d7/f19 d7/d21/d42/f4c 0 2024-04-22T15:49:32.991 INFO:tasks.workunit.client.0.smithi016.stdout:2/266: read d2/f4 [3786646,110381] 0 2024-04-22T15:49:32.991 INFO:tasks.workunit.client.0.smithi016.stdout:6/315: symlink d3/da/db/d14/d4c/l66 0 2024-04-22T15:49:32.994 INFO:tasks.workunit.client.0.smithi016.stdout:2/267: mkdir d2/d4d/d64 0 2024-04-22T15:49:32.996 INFO:tasks.workunit.client.0.smithi016.stdout:2/268: truncate d2/d1a/f20 5127880 0 2024-04-22T15:49:32.996 INFO:tasks.workunit.client.0.smithi016.stdout:2/269: chown d2/d1a/d4c 1813 1 2024-04-22T15:49:32.997 INFO:tasks.workunit.client.0.smithi016.stdout:0/306: mkdir dc/d13/d5d 0 2024-04-22T15:49:32.997 INFO:tasks.workunit.client.0.smithi016.stdout:2/270: write d2/d23/d47/f4a [1140068,39491] 0 2024-04-22T15:49:32.998 INFO:tasks.workunit.client.0.smithi016.stdout:7/281: mkdir d3/dc/d1b/d5a/d4a/d6e 0 2024-04-22T15:49:32.999 INFO:tasks.workunit.client.0.smithi016.stdout:6/316: rmdir d3/da/db/d14/d4c 39 2024-04-22T15:49:33.004 INFO:tasks.workunit.client.0.smithi016.stdout:4/246: rename d7/f14 to d7/d21/d45/d49/f4d 0 2024-04-22T15:49:33.008 INFO:tasks.workunit.client.0.smithi016.stdout:4/247: dread d7/d22/f27 [0,4194304] 0 2024-04-22T15:49:33.008 INFO:tasks.workunit.client.0.smithi016.stdout:7/282: readlink d3/d32/l5d 0 2024-04-22T15:49:33.009 INFO:tasks.workunit.client.0.smithi016.stdout:4/248: write d7/d24/f3d [824715,9691] 0 2024-04-22T15:49:33.009 INFO:tasks.workunit.client.0.smithi016.stdout:4/249: fsync d7/f10 0 2024-04-22T15:49:33.010 INFO:tasks.workunit.client.0.smithi016.stdout:4/250: chown d7/f11 0 1 2024-04-22T15:49:33.014 INFO:tasks.workunit.client.0.smithi016.stdout:2/271: dread d2/d9/f16 [0,4194304] 0 2024-04-22T15:49:33.022 INFO:tasks.workunit.client.0.smithi016.stdout:6/317: mkdir d3/da/dd/d67 0 2024-04-22T15:49:33.023 INFO:tasks.workunit.client.0.smithi016.stdout:6/318: truncate d3/da/dd/f29 500115 0 2024-04-22T15:49:33.025 INFO:tasks.workunit.client.0.smithi016.stdout:0/307: truncate dc/d13/d22/f2e 3447592 0 2024-04-22T15:49:33.026 INFO:tasks.workunit.client.0.smithi016.stdout:7/283: creat d3/dc/f6f x:0 0 0 2024-04-22T15:49:33.028 INFO:tasks.workunit.client.0.smithi016.stdout:4/251: rename d7/d21/d23 to d7/d4e 0 2024-04-22T15:49:33.029 INFO:tasks.workunit.client.0.smithi016.stdout:4/252: readlink d7/d4e/l37 0 2024-04-22T15:49:33.030 INFO:tasks.workunit.client.0.smithi016.stdout:2/272: dread d2/f3e [0,4194304] 0 2024-04-22T15:49:33.031 INFO:tasks.workunit.client.0.smithi016.stdout:6/319: dwrite d3/da/dd/f10 [8388608,4194304] 0 2024-04-22T15:49:33.032 INFO:tasks.workunit.client.0.smithi016.stdout:7/284: stat d3/dc/d1b/l62 0 2024-04-22T15:49:33.035 INFO:tasks.workunit.client.0.smithi016.stdout:2/273: mknod d2/de/d2a/c65 0 2024-04-22T15:49:33.043 INFO:tasks.workunit.client.0.smithi016.stdout:0/308: dwrite dc/d13/d14/d1d/d1f/f53 [0,4194304] 0 2024-04-22T15:49:33.045 INFO:tasks.workunit.client.0.smithi016.stdout:6/320: dwrite d3/da/db/f61 [0,4194304] 0 2024-04-22T15:49:33.053 INFO:tasks.workunit.client.0.smithi016.stdout:0/309: link dc/d13/d14/d1d/f42 dc/d13/d14/d26/f5e 0 2024-04-22T15:49:33.056 INFO:tasks.workunit.client.0.smithi016.stdout:4/253: dread d7/d21/d45/d49/f4d [0,4194304] 0 2024-04-22T15:49:33.059 INFO:tasks.workunit.client.0.smithi016.stdout:7/285: dwrite d3/f34 [0,4194304] 0 2024-04-22T15:49:33.059 INFO:tasks.workunit.client.0.smithi016.stdout:6/321: dwrite d3/da/dd/d1b/f62 [0,4194304] 0 2024-04-22T15:49:33.062 INFO:tasks.workunit.client.0.smithi016.stdout:0/310: creat dc/d13/d5d/f5f x:0 0 0 2024-04-22T15:49:33.063 INFO:tasks.workunit.client.0.smithi016.stdout:2/274: dread d2/f5 [0,4194304] 0 2024-04-22T15:49:33.064 INFO:tasks.workunit.client.0.smithi016.stdout:7/286: write d3/dc/d1b/d5a/d9/df/f11 [4818951,65457] 0 2024-04-22T15:49:33.065 INFO:tasks.workunit.client.0.smithi016.stdout:4/254: creat d7/d24/d41/f4f x:0 0 0 2024-04-22T15:49:33.067 INFO:tasks.workunit.client.0.smithi016.stdout:7/287: chown d3/d32/c5c 286028566 1 2024-04-22T15:49:33.067 INFO:tasks.workunit.client.0.smithi016.stdout:6/322: mkdir d3/da/db/d44/d68 0 2024-04-22T15:49:33.067 INFO:tasks.workunit.client.0.smithi016.stdout:7/288: stat d3/dc/d1b 0 2024-04-22T15:49:33.070 INFO:tasks.workunit.client.0.smithi016.stdout:2/275: creat d2/d23/d47/f66 x:0 0 0 2024-04-22T15:49:33.071 INFO:tasks.workunit.client.0.smithi016.stdout:0/311: mknod dc/d13/d14/d1d/d40/c60 0 2024-04-22T15:49:33.073 INFO:tasks.workunit.client.0.smithi016.stdout:0/312: chown dc/d13/d14/d1d/d57/l58 0 1 2024-04-22T15:49:33.074 INFO:tasks.workunit.client.0.smithi016.stdout:4/255: unlink d7/l20 0 2024-04-22T15:49:33.075 INFO:tasks.workunit.client.0.smithi016.stdout:6/323: creat d3/da/f69 x:0 0 0 2024-04-22T15:49:33.076 INFO:tasks.workunit.client.0.smithi016.stdout:6/324: dread - d3/da/f69 zero size 2024-04-22T15:49:33.078 INFO:tasks.workunit.client.0.smithi016.stdout:7/289: creat d3/dc/d1b/d5a/d9/df/f70 x:0 0 0 2024-04-22T15:49:33.078 INFO:tasks.workunit.client.0.smithi016.stdout:7/290: chown d3/l12 132591 1 2024-04-22T15:49:33.079 INFO:tasks.workunit.client.0.smithi016.stdout:4/256: mkdir d7/d4e/d2f/d50 0 2024-04-22T15:49:33.082 INFO:tasks.workunit.client.0.smithi016.stdout:6/325: creat d3/da/db/d44/d68/f6a x:0 0 0 2024-04-22T15:49:33.082 INFO:tasks.workunit.client.0.smithi016.stdout:7/291: write d3/dc/d1b/d5a/d9/fb [5713303,108108] 0 2024-04-22T15:49:33.082 INFO:tasks.workunit.client.0.smithi016.stdout:1/494: sync 2024-04-22T15:49:33.083 INFO:tasks.workunit.client.0.smithi016.stdout:6/326: dread - d3/da/f3e zero size 2024-04-22T15:49:33.084 INFO:tasks.workunit.client.0.smithi016.stdout:1/495: chown d3/d4/d19/d20/d2b/c52 69396 1 2024-04-22T15:49:33.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:32 smithi027 ceph-mon[34922]: from='client.14672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:49:33.087 INFO:tasks.workunit.client.0.smithi016.stdout:4/257: dwrite d7/d24/f36 [0,4194304] 0 2024-04-22T15:49:33.087 INFO:tasks.workunit.client.0.smithi016.stdout:7/292: dread d3/dc/f65 [0,4194304] 0 2024-04-22T15:49:33.087 INFO:tasks.workunit.client.0.smithi016.stdout:6/327: chown d3/da/db/d14/d16/c36 12531 1 2024-04-22T15:49:33.087 INFO:tasks.workunit.client.0.smithi016.stdout:4/258: chown d7/d4e/f29 103 1 2024-04-22T15:49:33.089 INFO:tasks.workunit.client.0.smithi016.stdout:7/293: write d3/dc/d1b/d5a/f1f [4807517,21352] 0 2024-04-22T15:49:33.090 INFO:tasks.workunit.client.0.smithi016.stdout:4/259: readlink d7/d4e/d2f/d33/l34 0 2024-04-22T15:49:33.091 INFO:tasks.workunit.client.0.smithi016.stdout:6/328: chown d3/da/d12/d13/d46/f4f 15963 1 2024-04-22T15:49:33.091 INFO:tasks.workunit.client.0.smithi016.stdout:4/260: write d7/f16 [4883101,69359] 0 2024-04-22T15:49:33.092 INFO:tasks.workunit.client.0.smithi016.stdout:6/329: fsync d3/da/d12/d13/f27 0 2024-04-22T15:49:33.093 INFO:tasks.workunit.client.0.smithi016.stdout:1/496: creat d3/d4/d15/d38/fa8 x:0 0 0 2024-04-22T15:49:33.094 INFO:tasks.workunit.client.0.smithi016.stdout:7/294: dread d3/dc/d1b/d5a/f46 [0,4194304] 0 2024-04-22T15:49:33.095 INFO:tasks.workunit.client.0.smithi016.stdout:1/497: fsync d3/d2e/d80/fa5 0 2024-04-22T15:49:33.097 INFO:tasks.workunit.client.0.smithi016.stdout:6/330: rmdir d3/da/db/d14/d16 39 2024-04-22T15:49:33.101 INFO:tasks.workunit.client.0.smithi016.stdout:1/498: creat d3/d4/d15/d38/d63/fa9 x:0 0 0 2024-04-22T15:49:33.102 INFO:tasks.workunit.client.0.smithi016.stdout:0/313: dwrite dc/d13/d22/f2e [0,4194304] 0 2024-04-22T15:49:33.104 INFO:tasks.workunit.client.0.smithi016.stdout:7/295: rename d3/dc/d1b/d5a/d4a/c68 to d3/dc/d1b/d5a/d9/c71 0 2024-04-22T15:49:33.104 INFO:tasks.workunit.client.0.smithi016.stdout:6/331: symlink d3/da/db/d14/d16/l6b 0 2024-04-22T15:49:33.105 INFO:tasks.workunit.client.0.smithi016.stdout:7/296: fdatasync d3/dc/d1b/d5a/d59/d64/f6c 0 2024-04-22T15:49:33.106 INFO:tasks.workunit.client.0.smithi016.stdout:0/314: mkdir dc/d13/d14/d1d/d40/d4a/d61 0 2024-04-22T15:49:33.107 INFO:tasks.workunit.client.0.smithi016.stdout:0/315: stat dc/d13/d14/d1d/d30/f44 0 2024-04-22T15:49:33.109 INFO:tasks.workunit.client.0.smithi016.stdout:7/297: creat d3/d32/f72 x:0 0 0 2024-04-22T15:49:33.111 INFO:tasks.workunit.client.0.smithi016.stdout:1/499: creat d3/d4/faa x:0 0 0 2024-04-22T15:49:33.111 INFO:tasks.workunit.client.0.smithi016.stdout:6/332: mkdir d3/da/dd/d67/d6c 0 2024-04-22T15:49:33.112 INFO:tasks.workunit.client.0.smithi016.stdout:6/333: write d3/f59 [450404,98178] 0 2024-04-22T15:49:33.115 INFO:tasks.workunit.client.0.smithi016.stdout:0/316: symlink dc/l62 0 2024-04-22T15:49:33.115 INFO:tasks.workunit.client.0.smithi016.stdout:7/298: creat d3/dc/d1b/d5a/d4a/d6e/f73 x:0 0 0 2024-04-22T15:49:33.121 INFO:tasks.workunit.client.0.smithi016.stdout:7/299: dwrite d3/dc/d1b/d5a/f17 [0,4194304] 0 2024-04-22T15:49:33.125 INFO:tasks.workunit.client.0.smithi016.stdout:0/317: getdents dc/d13/d14/d1d/d30 0 2024-04-22T15:49:33.130 INFO:tasks.workunit.client.0.smithi016.stdout:0/318: dwrite dc/d13/d14/d1d/d1f/f53 [0,4194304] 0 2024-04-22T15:49:33.133 INFO:tasks.workunit.client.0.smithi016.stdout:0/319: creat dc/d13/d14/d1d/f63 x:0 0 0 2024-04-22T15:49:33.134 INFO:tasks.workunit.client.0.smithi016.stdout:6/334: getdents d3/da/dd/d67 0 2024-04-22T15:49:33.134 INFO:tasks.workunit.client.0.smithi016.stdout:0/320: chown dc/l62 270436380 1 2024-04-22T15:49:33.134 INFO:tasks.workunit.client.0.smithi016.stdout:6/335: chown d3/da/d12/d13/d38 37 1 2024-04-22T15:49:33.136 INFO:tasks.workunit.client.0.smithi016.stdout:0/321: read dc/d13/d14/d1d/d1f/f53 [3444352,89096] 0 2024-04-22T15:49:33.137 INFO:tasks.workunit.client.0.smithi016.stdout:6/336: creat d3/da/d12/d13/d46/f6d x:0 0 0 2024-04-22T15:49:33.138 INFO:tasks.workunit.client.0.smithi016.stdout:0/322: write dc/d13/d28/f50 [1888556,53492] 0 2024-04-22T15:49:33.140 INFO:tasks.workunit.client.0.smithi016.stdout:0/323: readlink dc/d13/d14/d1d/d1f/l29 0 2024-04-22T15:49:33.143 INFO:tasks.workunit.client.0.smithi016.stdout:1/500: dwrite d3/d4/d19/d20/f44 [0,4194304] 0 2024-04-22T15:49:33.147 INFO:tasks.workunit.client.0.smithi016.stdout:0/324: creat dc/d13/d14/d1d/d1f/d48/f64 x:0 0 0 2024-04-22T15:49:33.151 INFO:tasks.workunit.client.0.smithi016.stdout:1/501: dwrite d3/d4/d15/d38/d63/f75 [0,4194304] 0 2024-04-22T15:49:33.152 INFO:tasks.workunit.client.0.smithi016.stdout:1/502: chown d3/d4/d3a/d9e 2 1 2024-04-22T15:49:33.152 INFO:tasks.workunit.client.0.smithi016.stdout:1/503: readlink d3/d4/d19/d20/d9d/la6 0 2024-04-22T15:49:33.153 INFO:tasks.workunit.client.0.smithi016.stdout:6/337: write d3/da/db/f17 [4030962,89077] 0 2024-04-22T15:49:33.155 INFO:tasks.workunit.client.0.smithi016.stdout:6/338: creat d3/da/d12/d13/f6e x:0 0 0 2024-04-22T15:49:33.161 INFO:tasks.workunit.client.0.smithi016.stdout:6/339: dread d3/da/d12/d13/f27 [0,4194304] 0 2024-04-22T15:49:33.164 INFO:tasks.workunit.client.0.smithi016.stdout:6/340: creat d3/da/db/d14/d4c/d58/f6f x:0 0 0 2024-04-22T15:49:33.166 INFO:tasks.workunit.client.0.smithi016.stdout:6/341: unlink d3/da/dd/c22 0 2024-04-22T15:49:33.171 INFO:tasks.workunit.client.0.smithi016.stdout:6/342: dwrite d3/da/f45 [0,4194304] 0 2024-04-22T15:49:33.176 INFO:tasks.workunit.client.0.smithi016.stdout:6/343: dread d3/da/d12/d13/d46/f4f [4194304,4194304] 0 2024-04-22T15:49:33.176 INFO:tasks.workunit.client.0.smithi016.stdout:6/344: write d3/da/d12/f40 [192353,26011] 0 2024-04-22T15:49:33.180 INFO:tasks.workunit.client.0.smithi016.stdout:6/345: unlink d3/da/d12/d13/d38/c2c 0 2024-04-22T15:49:33.185 INFO:tasks.workunit.client.0.smithi016.stdout:6/346: rename d3/da/d12/d13/f6e to d3/da/dd/f70 0 2024-04-22T15:49:33.187 INFO:tasks.workunit.client.0.smithi016.stdout:6/347: mknod d3/da/db/d44/d68/c71 0 2024-04-22T15:49:33.188 INFO:tasks.workunit.client.0.smithi016.stdout:6/348: creat d3/da/f72 x:0 0 0 2024-04-22T15:49:33.208 INFO:tasks.workunit.client.0.smithi016.stdout:6/349: truncate d3/da/d12/f15 1188941 0 2024-04-22T15:49:33.209 INFO:tasks.workunit.client.0.smithi016.stdout:6/350: mknod d3/da/db/d14/d4c/c73 0 2024-04-22T15:49:33.211 INFO:tasks.workunit.client.0.smithi016.stdout:6/351: mknod d3/da/dd/d1b/c74 0 2024-04-22T15:49:33.212 INFO:tasks.workunit.client.0.smithi016.stdout:6/352: creat d3/da/db/d14/d4c/d58/f75 x:0 0 0 2024-04-22T15:49:33.214 INFO:tasks.workunit.client.0.smithi016.stdout:6/353: creat d3/da/d12/d13/f76 x:0 0 0 2024-04-22T15:49:33.215 INFO:tasks.workunit.client.0.smithi016.stdout:6/354: stat d3/da/d12/d13/d46/f4f 0 2024-04-22T15:49:33.237 INFO:tasks.workunit.client.0.smithi016.stdout:8/310: sync 2024-04-22T15:49:33.237 INFO:tasks.workunit.client.0.smithi016.stdout:9/294: sync 2024-04-22T15:49:33.237 INFO:tasks.workunit.client.0.smithi016.stdout:5/260: sync 2024-04-22T15:49:33.237 INFO:tasks.workunit.client.0.smithi016.stdout:3/319: sync 2024-04-22T15:49:33.242 INFO:tasks.workunit.client.0.smithi016.stdout:5/261: creat df/d15/f4a x:0 0 0 2024-04-22T15:49:33.245 INFO:tasks.workunit.client.0.smithi016.stdout:5/262: creat df/d15/d1a/f4b x:0 0 0 2024-04-22T15:49:33.246 INFO:tasks.workunit.client.0.smithi016.stdout:5/263: rmdir df/d15/d1a/d28 39 2024-04-22T15:49:33.249 INFO:tasks.workunit.client.0.smithi016.stdout:5/264: write df/d15/d1a/d28/f32 [5766809,112705] 0 2024-04-22T15:49:33.254 INFO:tasks.workunit.client.0.smithi016.stdout:5/265: creat df/d15/d1a/d28/d33/f4c x:0 0 0 2024-04-22T15:49:33.255 INFO:tasks.workunit.client.0.smithi016.stdout:3/320: dread f1 [4194304,4194304] 0 2024-04-22T15:49:33.258 INFO:tasks.workunit.client.0.smithi016.stdout:5/266: mkdir df/d15/d1a/d4d 0 2024-04-22T15:49:33.268 INFO:tasks.workunit.client.0.smithi016.stdout:9/295: truncate f2 14564 0 2024-04-22T15:49:33.268 INFO:tasks.workunit.client.0.smithi016.stdout:9/296: stat d9/f1c 0 2024-04-22T15:49:33.270 INFO:tasks.workunit.client.0.smithi016.stdout:9/297: creat d9/d20/f56 x:0 0 0 2024-04-22T15:49:33.271 INFO:tasks.workunit.client.0.smithi016.stdout:5/267: getdents df/d15 0 2024-04-22T15:49:33.273 INFO:tasks.workunit.client.0.smithi016.stdout:5/268: stat l9 0 2024-04-22T15:49:33.274 INFO:tasks.workunit.client.0.smithi016.stdout:5/269: write f2 [3985153,39252] 0 2024-04-22T15:49:33.280 INFO:tasks.workunit.client.0.smithi016.stdout:5/270: dwrite f2 [0,4194304] 0 2024-04-22T15:49:33.283 INFO:tasks.workunit.client.0.smithi016.stdout:3/321: dwrite d7/d1b/f4b [0,4194304] 0 2024-04-22T15:49:33.284 INFO:tasks.workunit.client.0.smithi016.stdout:3/322: chown d7/d17/d18/d32/f57 7983145 1 2024-04-22T15:49:33.285 INFO:tasks.workunit.client.0.smithi016.stdout:5/271: mkdir df/d15/d1a/d4e 0 2024-04-22T15:49:33.288 INFO:tasks.workunit.client.0.smithi016.stdout:5/272: mknod df/d15/d1a/d28/d33/c4f 0 2024-04-22T15:49:33.288 INFO:tasks.workunit.client.0.smithi016.stdout:5/273: fsync df/d22/f2a 0 2024-04-22T15:49:33.290 INFO:tasks.workunit.client.0.smithi016.stdout:3/323: link f1 d7/d17/d18/d32/d3c/f66 0 2024-04-22T15:49:33.293 INFO:tasks.workunit.client.0.smithi016.stdout:5/274: mkdir df/d15/d1a/d50 0 2024-04-22T15:49:33.295 INFO:tasks.workunit.client.0.smithi016.stdout:3/324: rename d7/f30 to d7/d17/d18/f67 0 2024-04-22T15:49:33.302 INFO:tasks.workunit.client.0.smithi016.stdout:5/275: dwrite f2 [0,4194304] 0 2024-04-22T15:49:33.306 INFO:tasks.workunit.client.0.smithi016.stdout:9/298: dread d9/de/d23/d24/d25/f2c [0,4194304] 0 2024-04-22T15:49:33.307 INFO:tasks.workunit.client.0.smithi016.stdout:5/276: rename df/d15/c47 to df/d15/d1a/d50/c51 0 2024-04-22T15:49:33.310 INFO:tasks.workunit.client.0.smithi016.stdout:9/299: mkdir d9/d57 0 2024-04-22T15:49:33.318 INFO:tasks.workunit.client.0.smithi016.stdout:9/300: dwrite d9/d20/f40 [0,4194304] 0 2024-04-22T15:49:33.319 INFO:tasks.workunit.client.0.smithi016.stdout:3/325: dread d7/f8 [0,4194304] 0 2024-04-22T15:49:33.319 INFO:tasks.workunit.client.0.smithi016.stdout:9/301: read - d9/d20/f56 zero size 2024-04-22T15:49:33.321 INFO:tasks.workunit.client.0.smithi016.stdout:9/302: write d9/d20/f40 [170909,67548] 0 2024-04-22T15:49:33.322 INFO:tasks.workunit.client.0.smithi016.stdout:2/276: sync 2024-04-22T15:49:33.327 INFO:tasks.workunit.client.0.smithi016.stdout:3/326: mkdir d7/d17/d18/d32/d42/d68 0 2024-04-22T15:49:33.327 INFO:tasks.workunit.client.0.smithi016.stdout:2/277: dwrite d2/de/d2a/f49 [0,4194304] 0 2024-04-22T15:49:33.329 INFO:tasks.workunit.client.0.smithi016.stdout:2/278: chown d2/d51/f62 14755 1 2024-04-22T15:49:33.338 INFO:tasks.workunit.client.0.smithi016.stdout:3/327: rename d7/d17/d18/d32/d3c/f59 to d7/f69 0 2024-04-22T15:49:33.339 INFO:tasks.workunit.client.0.smithi016.stdout:3/328: fdatasync d7/d17/d18/d32/f57 0 2024-04-22T15:49:33.340 INFO:tasks.workunit.client.0.smithi016.stdout:5/277: link df/d15/d1a/l46 df/d15/d1a/d4d/l52 0 2024-04-22T15:49:33.341 INFO:tasks.workunit.client.0.smithi016.stdout:5/278: write df/d15/d1a/d28/d33/f39 [998076,95643] 0 2024-04-22T15:49:33.342 INFO:tasks.workunit.client.0.smithi016.stdout:2/279: truncate f0 7002296 0 2024-04-22T15:49:33.343 INFO:tasks.workunit.client.0.smithi016.stdout:2/280: stat d2/d1a/d46 0 2024-04-22T15:49:33.344 INFO:tasks.workunit.client.0.smithi016.stdout:2/281: write d2/de/f5c [759869,89534] 0 2024-04-22T15:49:33.393 INFO:tasks.workunit.client.0.smithi016.stdout:2/282: dwrite d2/de/f5c [0,4194304] 0 2024-04-22T15:49:33.397 INFO:tasks.workunit.client.0.smithi016.stdout:2/283: rename d2/d1a/d46 to d2/d23/d27/d67 0 2024-04-22T15:49:33.398 INFO:tasks.workunit.client.0.smithi016.stdout:5/279: dwrite df/d15/d1a/d28/f3f [0,4194304] 0 2024-04-22T15:49:33.399 INFO:tasks.workunit.client.0.smithi016.stdout:5/280: chown df/d22/c2d 1887 1 2024-04-22T15:49:33.400 INFO:tasks.workunit.client.0.smithi016.stdout:5/281: fdatasync df/d15/d1a/d28/f2b 0 2024-04-22T15:49:33.400 INFO:tasks.workunit.client.0.smithi016.stdout:2/284: read d2/d2c/f3c [6869517,93110] 0 2024-04-22T15:49:33.411 INFO:tasks.workunit.client.0.smithi016.stdout:5/282: link df/d15/d1a/d28/f3a df/d15/d1a/f53 0 2024-04-22T15:49:33.414 INFO:tasks.workunit.client.0.smithi016.stdout:4/261: sync 2024-04-22T15:49:33.416 INFO:tasks.workunit.client.0.smithi016.stdout:4/262: mkdir d7/d24/d51 0 2024-04-22T15:49:33.420 INFO:tasks.workunit.client.0.smithi016.stdout:4/263: mknod d7/d21/d45/c52 0 2024-04-22T15:49:33.423 INFO:tasks.workunit.client.0.smithi016.stdout:4/264: mknod d7/d4e/c53 0 2024-04-22T15:49:33.423 INFO:tasks.workunit.client.0.smithi016.stdout:4/265: stat d7/d24/c2c 0 2024-04-22T15:49:33.423 INFO:tasks.workunit.client.0.smithi016.stdout:4/266: stat d7/c18 0 2024-04-22T15:49:33.424 INFO:tasks.workunit.client.0.smithi016.stdout:4/267: stat d7/d21/d45 0 2024-04-22T15:49:33.426 INFO:tasks.workunit.client.0.smithi016.stdout:4/268: creat d7/d22/f54 x:0 0 0 2024-04-22T15:49:33.428 INFO:tasks.workunit.client.0.smithi016.stdout:2/285: dread f0 [0,4194304] 0 2024-04-22T15:49:33.430 INFO:tasks.workunit.client.0.smithi016.stdout:4/269: creat d7/d21/d42/f55 x:0 0 0 2024-04-22T15:49:33.431 INFO:tasks.workunit.client.0.smithi016.stdout:4/270: stat d7/d21/d42/f55 0 2024-04-22T15:49:33.431 INFO:tasks.workunit.client.0.smithi016.stdout:5/283: dread df/d22/f42 [0,4194304] 0 2024-04-22T15:49:33.432 INFO:tasks.workunit.client.0.smithi016.stdout:5/284: write df/d15/d1a/d28/f32 [1108769,114343] 0 2024-04-22T15:49:33.436 INFO:tasks.workunit.client.0.smithi016.stdout:5/285: rename df/d15/d1a/d28/d33/c4f to df/d15/d1a/d4d/c54 0 2024-04-22T15:49:33.438 INFO:tasks.workunit.client.0.smithi016.stdout:4/271: getdents d7/d21 0 2024-04-22T15:49:33.438 INFO:tasks.workunit.client.0.smithi016.stdout:5/286: mknod df/d15/d1a/d4d/c55 0 2024-04-22T15:49:33.441 INFO:tasks.workunit.client.0.smithi016.stdout:5/287: write df/d15/f45 [560965,45751] 0 2024-04-22T15:49:33.445 INFO:tasks.workunit.client.0.smithi016.stdout:5/288: creat df/d15/d1a/d4d/f56 x:0 0 0 2024-04-22T15:49:33.450 INFO:tasks.workunit.client.0.smithi016.stdout:5/289: dwrite df/d15/d1a/f1e [0,4194304] 0 2024-04-22T15:49:33.461 INFO:tasks.workunit.client.0.smithi016.stdout:4/272: truncate d7/d24/f3d 80447 0 2024-04-22T15:49:33.462 INFO:tasks.workunit.client.0.smithi016.stdout:5/290: creat df/f57 x:0 0 0 2024-04-22T15:49:33.463 INFO:tasks.workunit.client.0.smithi016.stdout:5/291: readlink df/d15/d1a/l1b 0 2024-04-22T15:49:33.463 INFO:tasks.workunit.client.0.smithi016.stdout:5/292: chown df/d15/d1a/d28/f3f 192342 1 2024-04-22T15:49:33.464 INFO:tasks.workunit.client.0.smithi016.stdout:5/293: chown df/d15/d1a/d28/f3f 30 1 2024-04-22T15:49:33.465 INFO:tasks.workunit.client.0.smithi016.stdout:5/294: truncate df/d22/f25 425207 0 2024-04-22T15:49:33.466 INFO:tasks.workunit.client.0.smithi016.stdout:4/273: rename d7/d22/f40 to d7/d21/d45/d49/f56 0 2024-04-22T15:49:33.471 INFO:tasks.workunit.client.0.smithi016.stdout:4/274: dwrite d7/f8 [4194304,4194304] 0 2024-04-22T15:49:33.471 INFO:tasks.workunit.client.0.smithi016.stdout:5/295: unlink df/d15/d1a/d4d/f56 0 2024-04-22T15:49:33.472 INFO:tasks.workunit.client.0.smithi016.stdout:4/275: read d7/d21/d45/d49/f4d [3892500,69545] 0 2024-04-22T15:49:33.479 INFO:tasks.workunit.client.0.smithi016.stdout:4/276: rename d7/d24/f36 to d7/d24/d41/f57 0 2024-04-22T15:49:33.479 INFO:tasks.workunit.client.0.smithi016.stdout:4/277: write d7/f11 [3712280,24347] 0 2024-04-22T15:49:33.497 INFO:tasks.workunit.client.0.smithi016.stdout:4/278: link d7/d4e/d2f/l3f d7/l58 0 2024-04-22T15:49:33.497 INFO:tasks.workunit.client.0.smithi016.stdout:4/279: chown d7/d4e/d2f/d50 2 1 2024-04-22T15:49:33.497 INFO:tasks.workunit.client.0.smithi016.stdout:4/280: fdatasync d7/f1e 0 2024-04-22T15:49:33.498 INFO:tasks.workunit.client.0.smithi016.stdout:4/281: stat d7/d4e/c3e 0 2024-04-22T15:49:33.499 INFO:tasks.workunit.client.0.smithi016.stdout:4/282: write d7/f10 [3792308,120469] 0 2024-04-22T15:49:33.500 INFO:tasks.workunit.client.0.smithi016.stdout:5/296: truncate df/d22/f42 3610362 0 2024-04-22T15:49:33.506 INFO:tasks.workunit.client.0.smithi016.stdout:5/297: mkdir df/d15/d1a/d4e/d58 0 2024-04-22T15:49:33.507 INFO:tasks.workunit.client.0.smithi016.stdout:4/283: dwrite d7/d21/d45/d49/f56 [0,4194304] 0 2024-04-22T15:49:33.508 INFO:tasks.workunit.client.0.smithi016.stdout:5/298: fsync df/d15/d1a/d28/d33/f39 0 2024-04-22T15:49:33.509 INFO:tasks.workunit.client.0.smithi016.stdout:5/299: chown df/l38 213727 1 2024-04-22T15:49:33.510 INFO:tasks.workunit.client.0.smithi016.stdout:5/300: write df/d15/d1a/d28/f32 [1458877,40886] 0 2024-04-22T15:49:33.512 INFO:tasks.workunit.client.0.smithi016.stdout:5/301: readlink df/d15/d1a/d28/l2f 0 2024-04-22T15:49:33.513 INFO:tasks.workunit.client.0.smithi016.stdout:4/284: dread d7/f11 [4194304,4194304] 0 2024-04-22T15:49:33.517 INFO:tasks.workunit.client.0.smithi016.stdout:5/302: mknod df/d15/d1a/d50/c59 0 2024-04-22T15:49:33.519 INFO:tasks.workunit.client.0.smithi016.stdout:4/285: mkdir d7/d21/d42/d59 0 2024-04-22T15:49:33.531 INFO:tasks.workunit.client.0.smithi016.stdout:7/300: sync 2024-04-22T15:49:33.531 INFO:tasks.workunit.client.0.smithi016.stdout:1/504: sync 2024-04-22T15:49:33.532 INFO:tasks.workunit.client.0.smithi016.stdout:6/355: sync 2024-04-22T15:49:33.532 INFO:tasks.workunit.client.0.smithi016.stdout:0/325: sync 2024-04-22T15:49:33.535 INFO:tasks.workunit.client.0.smithi016.stdout:6/356: creat d3/da/d12/d13/d46/f77 x:0 0 0 2024-04-22T15:49:33.535 INFO:tasks.workunit.client.0.smithi016.stdout:6/357: fdatasync d3/da/dd/d1b/f62 0 2024-04-22T15:49:33.539 INFO:tasks.workunit.client.0.smithi016.stdout:0/326: rename dc/d13/d14/d1d to dc/d13/d28/d65 0 2024-04-22T15:49:33.547 INFO:tasks.workunit.client.0.smithi016.stdout:6/358: rename d3/da/db/d14/d16/f42 to d3/da/d12/d13/f78 0 2024-04-22T15:49:33.548 INFO:tasks.workunit.client.0.smithi016.stdout:0/327: truncate dc/d13/d14/f34 2381235 0 2024-04-22T15:49:33.549 INFO:tasks.workunit.client.0.smithi016.stdout:5/303: truncate df/d15/d1a/d28/f32 1262559 0 2024-04-22T15:49:33.549 INFO:tasks.workunit.client.0.smithi016.stdout:0/328: stat dc/d13/d28/d65/d30/f44 0 2024-04-22T15:49:33.549 INFO:tasks.workunit.client.0.smithi016.stdout:5/304: read - df/f57 zero size 2024-04-22T15:49:33.550 INFO:tasks.workunit.client.0.smithi016.stdout:5/305: readlink df/d22/l29 0 2024-04-22T15:49:33.550 INFO:tasks.workunit.client.0.smithi016.stdout:0/329: read dc/d13/f17 [5673118,108324] 0 2024-04-22T15:49:33.551 INFO:tasks.workunit.client.0.smithi016.stdout:7/301: creat d3/f74 x:0 0 0 2024-04-22T15:49:33.551 INFO:tasks.workunit.client.0.smithi016.stdout:7/302: write d3/f74 [142776,74879] 0 2024-04-22T15:49:33.552 INFO:tasks.workunit.client.0.smithi016.stdout:4/286: truncate d7/f11 4940560 0 2024-04-22T15:49:33.553 INFO:tasks.workunit.client.0.smithi016.stdout:1/505: link d3/d28/c7b d3/d4/d3a/d6b/cab 0 2024-04-22T15:49:33.559 INFO:tasks.workunit.client.0.smithi016.stdout:6/359: rename d3/da/d12/d13/d46/f6d to d3/da/dd/d67/f79 0 2024-04-22T15:49:33.564 INFO:tasks.workunit.client.0.smithi016.stdout:7/303: read d3/dc/d1b/d5a/d9/f25 [2125799,124156] 0 2024-04-22T15:49:33.565 INFO:tasks.workunit.client.0.smithi016.stdout:7/304: write d3/dc/d1b/d31/d5f/f63 [27707,43252] 0 2024-04-22T15:49:33.572 INFO:tasks.workunit.client.0.smithi016.stdout:3/329: sync 2024-04-22T15:49:33.572 INFO:tasks.workunit.client.0.smithi016.stdout:9/303: sync 2024-04-22T15:49:33.572 INFO:tasks.workunit.client.0.smithi016.stdout:8/311: sync 2024-04-22T15:49:33.572 INFO:tasks.workunit.client.0.smithi016.stdout:2/286: sync 2024-04-22T15:49:33.573 INFO:tasks.workunit.client.0.smithi016.stdout:3/330: fsync d7/d1b/f4b 0 2024-04-22T15:49:33.574 INFO:tasks.workunit.client.0.smithi016.stdout:9/304: chown d9/f1b 1789778 1 2024-04-22T15:49:33.577 INFO:tasks.workunit.client.0.smithi016.stdout:2/287: dread d2/d9/f16 [0,4194304] 0 2024-04-22T15:49:33.579 INFO:tasks.workunit.client.0.smithi016.stdout:1/506: mknod d3/d2e/d9f/cac 0 2024-04-22T15:49:33.581 INFO:tasks.workunit.client.0.smithi016.stdout:7/305: rmdir d3/dc/d1b/d5a/d59 39 2024-04-22T15:49:33.582 INFO:tasks.workunit.client.0.smithi016.stdout:9/305: dwrite d9/de/d23/f29 [0,4194304] 0 2024-04-22T15:49:33.582 INFO:tasks.workunit.client.0.smithi016.stdout:1/507: write d3/d4/d19/d20/f48 [107326,59364] 0 2024-04-22T15:49:33.583 INFO:tasks.workunit.client.0.smithi016.stdout:5/306: dread df/d15/d1a/f53 [0,4194304] 0 2024-04-22T15:49:33.585 INFO:tasks.workunit.client.0.smithi016.stdout:9/306: write d9/de/f1a [107902,37954] 0 2024-04-22T15:49:33.586 INFO:tasks.workunit.client.0.smithi016.stdout:5/307: write df/d15/d1a/f4b [393456,130047] 0 2024-04-22T15:49:33.591 INFO:tasks.workunit.client.0.smithi016.stdout:9/307: dwrite d9/fa [0,4194304] 0 2024-04-22T15:49:33.591 INFO:tasks.workunit.client.0.smithi016.stdout:9/308: dread - d9/de/f28 zero size 2024-04-22T15:49:33.596 INFO:tasks.workunit.client.0.smithi016.stdout:8/312: dread d2/db/fc [0,4194304] 0 2024-04-22T15:49:33.598 INFO:tasks.workunit.client.0.smithi016.stdout:3/331: rename d7/d17/l24 to d7/d17/d18/d32/l6a 0 2024-04-22T15:49:33.601 INFO:tasks.workunit.client.0.smithi016.stdout:4/287: dwrite d7/d24/f3d [0,4194304] 0 2024-04-22T15:49:33.603 INFO:tasks.workunit.client.0.smithi016.stdout:2/288: symlink d2/d4d/l68 0 2024-04-22T15:49:33.605 INFO:tasks.workunit.client.0.smithi016.stdout:6/360: truncate d3/da/f45 3814701 0 2024-04-22T15:49:33.615 INFO:tasks.workunit.client.0.smithi016.stdout:0/330: link dc/l62 dc/d13/d14/d26/l66 0 2024-04-22T15:49:33.617 INFO:tasks.workunit.client.0.smithi016.stdout:5/308: chown df/d15/d1a/d4d/c54 0 1 2024-04-22T15:49:33.617 INFO:tasks.workunit.client.0.smithi016.stdout:5/309: dread - df/d22/f3b zero size 2024-04-22T15:49:33.618 INFO:tasks.workunit.client.0.smithi016.stdout:5/310: write df/d15/d1a/f20 [423262,18018] 0 2024-04-22T15:49:33.628 INFO:tasks.workunit.client.0.smithi016.stdout:8/313: rmdir d2/d10/d43 39 2024-04-22T15:49:33.633 INFO:tasks.workunit.client.0.smithi016.stdout:1/508: rename d3/d4/da/c3f to d3/d4/d45/cad 0 2024-04-22T15:49:33.640 INFO:tasks.workunit.client.0.smithi016.stdout:3/332: dwrite d7/d17/d18/f5b [0,4194304] 0 2024-04-22T15:49:33.642 INFO:tasks.workunit.client.0.smithi016.stdout:7/306: symlink d3/dc/d1b/d5a/d69/l75 0 2024-04-22T15:49:33.652 INFO:tasks.workunit.client.0.smithi016.stdout:9/309: creat d9/de/d23/d30/f58 x:0 0 0 2024-04-22T15:49:33.652 INFO:tasks.workunit.client.0.smithi016.stdout:9/310: write f7 [3328291,57167] 0 2024-04-22T15:49:33.653 INFO:tasks.workunit.client.0.smithi016.stdout:9/311: read - d9/d20/f56 zero size 2024-04-22T15:49:33.654 INFO:tasks.workunit.client.0.smithi016.stdout:9/312: read - d9/de/d26/f44 zero size 2024-04-22T15:49:33.655 INFO:tasks.workunit.client.0.smithi016.stdout:8/314: chown d2/d10/d43/d4a/d28/f38 1239310 1 2024-04-22T15:49:33.657 INFO:tasks.workunit.client.0.smithi016.stdout:6/361: rename d3/da/db/l43 to d3/da/db/d44/d68/l7a 0 2024-04-22T15:49:33.657 INFO:tasks.workunit.client.0.smithi016.stdout:6/362: chown d3/da/d12/d13/f76 191452959 1 2024-04-22T15:49:33.658 INFO:tasks.workunit.client.0.smithi016.stdout:1/509: write d3/d4/d19/f1d [4041976,99788] 0 2024-04-22T15:49:33.660 INFO:tasks.workunit.client.0.smithi016.stdout:4/288: fsync f5 0 2024-04-22T15:49:33.665 INFO:tasks.workunit.client.0.smithi016.stdout:2/289: mknod d2/de/d2a/d59/c69 0 2024-04-22T15:49:33.666 INFO:tasks.workunit.client.0.smithi016.stdout:0/331: dwrite dc/d13/d14/d26/f5e [0,4194304] 0 2024-04-22T15:49:33.666 INFO:tasks.workunit.client.0.smithi016.stdout:2/290: chown d2/de/d2a 43243 1 2024-04-22T15:49:33.667 INFO:tasks.workunit.client.0.smithi016.stdout:3/333: mknod d7/d17/d18/d32/c6b 0 2024-04-22T15:49:33.668 INFO:tasks.workunit.client.0.smithi016.stdout:3/334: truncate d7/fc 2279218 0 2024-04-22T15:49:33.669 INFO:tasks.workunit.client.0.smithi016.stdout:7/307: unlink d3/dc/d1b/d5a/d4a/f6a 0 2024-04-22T15:49:33.671 INFO:tasks.workunit.client.0.smithi016.stdout:7/308: chown d3/dc/d1b/d5a/d4a/f55 1555001 1 2024-04-22T15:49:33.672 INFO:tasks.workunit.client.0.smithi016.stdout:7/309: write d3/dc/d1b/f6d [203857,88501] 0 2024-04-22T15:49:33.674 INFO:tasks.workunit.client.0.smithi016.stdout:7/310: write d3/dc/d1b/d31/d5f/f61 [938319,87694] 0 2024-04-22T15:49:33.675 INFO:tasks.workunit.client.0.smithi016.stdout:8/315: dread d2/db/f3c [0,4194304] 0 2024-04-22T15:49:33.676 INFO:tasks.workunit.client.0.smithi016.stdout:5/311: symlink df/l5a 0 2024-04-22T15:49:33.681 INFO:tasks.workunit.client.0.smithi016.stdout:1/510: creat d3/d2e/d80/d82/fae x:0 0 0 2024-04-22T15:49:33.682 INFO:tasks.workunit.client.0.smithi016.stdout:1/511: write d3/d4/d15/d38/d63/f75 [2519341,130790] 0 2024-04-22T15:49:33.684 INFO:tasks.workunit.client.0.smithi016.stdout:4/289: chown d7/d22/c4b 653878509 1 2024-04-22T15:49:33.687 INFO:tasks.workunit.client.0.smithi016.stdout:4/290: dread d7/d21/d45/d49/f56 [0,4194304] 0 2024-04-22T15:49:33.688 INFO:tasks.workunit.client.0.smithi016.stdout:4/291: chown d7/d3c/f44 31024 1 2024-04-22T15:49:33.688 INFO:tasks.workunit.client.0.smithi016.stdout:0/332: creat dc/d13/d28/d65/d57/f67 x:0 0 0 2024-04-22T15:49:33.693 INFO:tasks.workunit.client.0.smithi016.stdout:2/291: creat d2/d23/d27/f6a x:0 0 0 2024-04-22T15:49:33.700 INFO:tasks.workunit.client.0.smithi016.stdout:8/316: fdatasync d2/d10/f61 0 2024-04-22T15:49:33.701 INFO:tasks.workunit.client.0.smithi016.stdout:4/292: dwrite d7/f1e [4194304,4194304] 0 2024-04-22T15:49:33.707 INFO:tasks.workunit.client.0.smithi016.stdout:9/313: write d9/d37/f49 [507400,62207] 0 2024-04-22T15:49:33.708 INFO:tasks.workunit.client.0.smithi016.stdout:9/314: dread - d9/de/f43 zero size 2024-04-22T15:49:33.721 INFO:tasks.workunit.client.0.smithi016.stdout:5/312: mkdir df/d5b 0 2024-04-22T15:49:33.723 INFO:tasks.workunit.client.0.smithi016.stdout:4/293: mknod d7/d21/d42/c5a 0 2024-04-22T15:49:33.725 INFO:tasks.workunit.client.0.smithi016.stdout:8/317: mkdir d2/d10/d43/d4a/d28/d44/d62 0 2024-04-22T15:49:33.725 INFO:tasks.workunit.client.0.smithi016.stdout:8/318: write d2/db/f20 [3274089,22695] 0 2024-04-22T15:49:33.729 INFO:tasks.workunit.client.0.smithi016.stdout:9/315: dread d9/de/d23/d24/f2f [0,4194304] 0 2024-04-22T15:49:33.731 INFO:tasks.workunit.client.0.smithi016.stdout:9/316: read d9/de/f16 [1055130,18780] 0 2024-04-22T15:49:33.731 INFO:tasks.workunit.client.0.smithi016.stdout:9/317: write f7 [756660,43279] 0 2024-04-22T15:49:33.735 INFO:tasks.workunit.client.0.smithi016.stdout:2/292: link d2/d2c/f37 d2/d2c/f6b 0 2024-04-22T15:49:33.736 INFO:tasks.workunit.client.0.smithi016.stdout:2/293: chown d2/d9/f12 211 1 2024-04-22T15:49:33.736 INFO:tasks.workunit.client.0.smithi016.stdout:9/318: dwrite d9/d37/f4a [0,4194304] 0 2024-04-22T15:49:33.737 INFO:tasks.workunit.client.0.smithi016.stdout:9/319: dread - d9/de/f48 zero size 2024-04-22T15:49:33.738 INFO:tasks.workunit.client.0.smithi016.stdout:9/320: write d9/d20/f56 [489534,25932] 0 2024-04-22T15:49:33.739 INFO:tasks.workunit.client.0.smithi016.stdout:3/335: rename d7/d17/d18/d32/d3c/d47/f62 to d7/d17/d18/d32/d3c/f6c 0 2024-04-22T15:49:33.739 INFO:tasks.workunit.client.0.smithi016.stdout:9/321: write d9/d20/f56 [1061205,69169] 0 2024-04-22T15:49:33.745 INFO:tasks.workunit.client.0.smithi016.stdout:6/363: getdents d3/da/db/d14/d4c/d58 0 2024-04-22T15:49:33.746 INFO:tasks.workunit.client.0.smithi016.stdout:1/512: dwrite d3/d4/da/f26 [0,4194304] 0 2024-04-22T15:49:33.747 INFO:tasks.workunit.client.0.smithi016.stdout:6/364: fsync d3/da/db/d14/d16/f5f 0 2024-04-22T15:49:33.749 INFO:tasks.workunit.client.0.smithi016.stdout:6/365: truncate d3/da/d12/d13/f76 512147 0 2024-04-22T15:49:33.759 INFO:tasks.workunit.client.0.smithi016.stdout:0/333: creat dc/f68 x:0 0 0 2024-04-22T15:49:33.760 INFO:tasks.workunit.client.0.smithi016.stdout:0/334: readlink dc/d13/d22/l5a 0 2024-04-22T15:49:33.761 INFO:tasks.workunit.client.0.smithi016.stdout:7/311: getdents d3/d14 0 2024-04-22T15:49:33.762 INFO:tasks.workunit.client.0.smithi016.stdout:7/312: write d3/dc/d1b/f49 [926152,127616] 0 2024-04-22T15:49:33.763 INFO:tasks.workunit.client.0.smithi016.stdout:2/294: creat d2/d23/f6c x:0 0 0 2024-04-22T15:49:33.882 INFO:tasks.workunit.client.0.smithi016.stdout:4/294: rename d7/c15 to d7/d21/d42/c5b 0 2024-04-22T15:49:33.882 INFO:tasks.workunit.client.0.smithi016.stdout:3/336: mknod d7/d17/d18/d32/c6d 0 2024-04-22T15:49:33.882 INFO:tasks.workunit.client.0.smithi016.stdout:8/319: mknod d2/c63 0 2024-04-22T15:49:33.883 INFO:tasks.workunit.client.0.smithi016.stdout:7/313: symlink d3/dc/d1b/d5a/d4a/l76 0 2024-04-22T15:49:33.884 INFO:tasks.workunit.client.0.smithi016.stdout:2/295: mkdir d2/d4d/d6d 0 2024-04-22T15:49:33.885 INFO:tasks.workunit.client.0.smithi016.stdout:6/366: write d3/f6 [830091,76148] 0 2024-04-22T15:49:33.886 INFO:tasks.workunit.client.0.smithi016.stdout:0/335: mkdir dc/d13/d14/d55/d69 0 2024-04-22T15:49:33.888 INFO:tasks.workunit.client.0.smithi016.stdout:4/295: rename d7/d24/c35 to d7/d4e/d2f/d33/c5c 0 2024-04-22T15:49:33.890 INFO:tasks.workunit.client.0.smithi016.stdout:7/314: dread d3/dc/d1b/d5a/d4a/f4f [0,4194304] 0 2024-04-22T15:49:33.893 INFO:tasks.workunit.client.0.smithi016.stdout:5/313: dwrite df/f1f [0,4194304] 0 2024-04-22T15:49:33.894 INFO:tasks.workunit.client.0.smithi016.stdout:5/314: readlink l9 0 2024-04-22T15:49:33.894 INFO:tasks.workunit.client.0.smithi016.stdout:3/337: dread d7/d17/d18/f23 [0,4194304] 0 2024-04-22T15:49:33.895 INFO:tasks.workunit.client.0.smithi016.stdout:2/296: creat d2/d4d/d64/f6e x:0 0 0 2024-04-22T15:49:33.899 INFO:tasks.workunit.client.0.smithi016.stdout:3/338: write d7/d17/d18/d27/f4f [3144455,123] 0 2024-04-22T15:49:33.900 INFO:tasks.workunit.client.0.smithi016.stdout:6/367: dwrite d3/da/dd/f29 [0,4194304] 0 2024-04-22T15:49:33.908 INFO:tasks.workunit.client.0.smithi016.stdout:4/296: link d7/d21/d42/f55 d7/d24/d41/f5d 0 2024-04-22T15:49:33.913 INFO:tasks.workunit.client.0.smithi016.stdout:3/339: rename d7/d17/d18/f41 to d7/d17/d18/d27/f6e 0 2024-04-22T15:49:33.914 INFO:tasks.workunit.client.0.smithi016.stdout:5/315: symlink df/d15/d1a/l5c 0 2024-04-22T15:49:33.915 INFO:tasks.workunit.client.0.smithi016.stdout:3/340: write d7/d17/d18/d32/d3c/d51/f5f [757867,105655] 0 2024-04-22T15:49:33.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:33 smithi016 ceph-mon[26620]: pgmap v29: 65 pgs: 65 active+clean; 815 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 17 MiB/s rd, 101 MiB/s wr, 440 op/s 2024-04-22T15:49:33.916 INFO:tasks.workunit.client.0.smithi016.stdout:4/297: read d7/d21/d45/d49/f56 [3532,30072] 0 2024-04-22T15:49:33.919 INFO:tasks.workunit.client.0.smithi016.stdout:6/368: mkdir d3/da/dd/d7b 0 2024-04-22T15:49:33.921 INFO:tasks.workunit.client.0.smithi016.stdout:3/341: dwrite d7/f5a [0,4194304] 0 2024-04-22T15:49:33.923 INFO:tasks.workunit.client.0.smithi016.stdout:5/316: fsync df/d15/f4a 0 2024-04-22T15:49:33.924 INFO:tasks.workunit.client.0.smithi016.stdout:7/315: getdents d3/dc/d1b/d5a/d4a 0 2024-04-22T15:49:33.927 INFO:tasks.workunit.client.0.smithi016.stdout:5/317: write df/d15/d1a/f20 [3844850,79493] 0 2024-04-22T15:49:33.927 INFO:tasks.workunit.client.0.smithi016.stdout:6/369: dwrite d3/f6 [0,4194304] 0 2024-04-22T15:49:33.929 INFO:tasks.workunit.client.0.smithi016.stdout:5/318: write df/d15/d1a/d28/f3f [4272189,22509] 0 2024-04-22T15:49:33.932 INFO:tasks.workunit.client.0.smithi016.stdout:4/298: mknod d7/d24/d51/c5e 0 2024-04-22T15:49:33.937 INFO:tasks.workunit.client.0.smithi016.stdout:5/319: write df/d15/d1a/d28/d33/f4c [168342,15789] 0 2024-04-22T15:49:33.939 INFO:tasks.workunit.client.0.smithi016.stdout:5/320: dread - df/f57 zero size 2024-04-22T15:49:33.940 INFO:tasks.workunit.client.0.smithi016.stdout:3/342: dread d7/f14 [0,4194304] 0 2024-04-22T15:49:33.940 INFO:tasks.workunit.client.0.smithi016.stdout:4/299: mknod d7/d21/c5f 0 2024-04-22T15:49:33.942 INFO:tasks.workunit.client.0.smithi016.stdout:6/370: dwrite d3/da/dd/f51 [0,4194304] 0 2024-04-22T15:49:33.942 INFO:tasks.workunit.client.0.smithi016.stdout:0/336: write dc/d13/f16 [3273991,33806] 0 2024-04-22T15:49:33.943 INFO:tasks.workunit.client.0.smithi016.stdout:9/322: dwrite f2 [0,4194304] 0 2024-04-22T15:49:33.943 INFO:tasks.workunit.client.0.smithi016.stdout:7/316: getdents d3/dc/d33 0 2024-04-22T15:49:33.946 INFO:tasks.workunit.client.0.smithi016.stdout:4/300: chown d7/l1a 5086784 1 2024-04-22T15:49:33.948 INFO:tasks.workunit.client.0.smithi016.stdout:3/343: unlink d7/d17/d18/f3f 0 2024-04-22T15:49:33.950 INFO:tasks.workunit.client.0.smithi016.stdout:9/323: creat d9/d20/f59 x:0 0 0 2024-04-22T15:49:33.951 INFO:tasks.workunit.client.0.smithi016.stdout:9/324: chown d9/de/d23/d24/f53 8 1 2024-04-22T15:49:33.952 INFO:tasks.workunit.client.0.smithi016.stdout:6/371: creat d3/da/dd/d67/f7c x:0 0 0 2024-04-22T15:49:33.957 INFO:tasks.workunit.client.0.smithi016.stdout:5/321: mkdir df/d5b/d5d 0 2024-04-22T15:49:33.958 INFO:tasks.workunit.client.0.smithi016.stdout:5/322: write df/d15/d1a/d28/d33/f3c [673310,69122] 0 2024-04-22T15:49:33.976 INFO:tasks.workunit.client.0.smithi016.stdout:7/317: unlink d3/dc/d1b/d31/d5f/f61 0 2024-04-22T15:49:33.976 INFO:tasks.workunit.client.0.smithi016.stdout:9/325: mkdir d9/de/d26/d5a 0 2024-04-22T15:49:33.977 INFO:tasks.workunit.client.0.smithi016.stdout:6/372: mknod d3/da/db/d14/d4c/c7d 0 2024-04-22T15:49:33.979 INFO:tasks.workunit.client.0.smithi016.stdout:7/318: fsync d3/dc/d1b/d31/f48 0 2024-04-22T15:49:33.980 INFO:tasks.workunit.client.0.smithi016.stdout:5/323: rename df/l49 to df/d15/d1a/d4e/l5e 0 2024-04-22T15:49:33.980 INFO:tasks.workunit.client.0.smithi016.stdout:0/337: dread dc/d13/d14/f34 [0,4194304] 0 2024-04-22T15:49:33.980 INFO:tasks.workunit.client.0.smithi016.stdout:3/344: creat d7/d1b/d21/d55/f6f x:0 0 0 2024-04-22T15:49:33.981 INFO:tasks.workunit.client.0.smithi016.stdout:9/326: rmdir d9/d20 39 2024-04-22T15:49:33.982 INFO:tasks.workunit.client.0.smithi016.stdout:5/324: chown df/d22/f3b 30 1 2024-04-22T15:49:33.984 INFO:tasks.workunit.client.0.smithi016.stdout:7/319: rmdir d3/dc/d33 39 2024-04-22T15:49:33.985 INFO:tasks.workunit.client.0.smithi016.stdout:7/320: write d3/f23 [944819,127330] 0 2024-04-22T15:49:33.990 INFO:tasks.workunit.client.0.smithi016.stdout:6/373: rename d3/da/dd/c28 to d3/da/db/d14/d4c/d58/c7e 0 2024-04-22T15:49:33.991 INFO:tasks.workunit.client.0.smithi016.stdout:5/325: dwrite df/d15/d1a/f20 [0,4194304] 0 2024-04-22T15:49:33.991 INFO:tasks.workunit.client.0.smithi016.stdout:0/338: creat dc/d13/d28/f6a x:0 0 0 2024-04-22T15:49:33.992 INFO:tasks.workunit.client.0.smithi016.stdout:0/339: readlink dc/d13/d22/l5a 0 2024-04-22T15:49:33.992 INFO:tasks.workunit.client.0.smithi016.stdout:6/374: chown d3/da/db/d44/d68/c71 28677 1 2024-04-22T15:49:33.994 INFO:tasks.workunit.client.0.smithi016.stdout:6/375: write d3/da/db/d14/d16/f55 [64709,67804] 0 2024-04-22T15:49:33.997 INFO:tasks.workunit.client.0.smithi016.stdout:3/345: dread f1 [0,4194304] 0 2024-04-22T15:49:33.998 INFO:tasks.workunit.client.0.smithi016.stdout:6/376: write d3/da/db/d14/d4c/f52 [2456141,127435] 0 2024-04-22T15:49:34.000 INFO:tasks.workunit.client.0.smithi016.stdout:4/301: getdents d7/d21 0 2024-04-22T15:49:34.004 INFO:tasks.workunit.client.0.smithi016.stdout:7/321: dwrite d3/dc/d1b/d5a/d9/df/f11 [0,4194304] 0 2024-04-22T15:49:34.005 INFO:tasks.workunit.client.0.smithi016.stdout:7/322: readlink d3/d32/l5d 0 2024-04-22T15:49:34.007 INFO:tasks.workunit.client.0.smithi016.stdout:9/327: link d9/d20/f56 d9/de/d23/d24/d4e/f5b 0 2024-04-22T15:49:34.009 INFO:tasks.workunit.client.0.smithi016.stdout:6/377: mknod d3/d53/d35/c7f 0 2024-04-22T15:49:34.009 INFO:tasks.workunit.client.0.smithi016.stdout:6/378: stat l1 0 2024-04-22T15:49:34.010 INFO:tasks.workunit.client.0.smithi016.stdout:6/379: readlink l1 0 2024-04-22T15:49:34.010 INFO:tasks.workunit.client.0.smithi016.stdout:0/340: dwrite dc/d13/d28/d65/d40/f56 [4194304,4194304] 0 2024-04-22T15:49:34.013 INFO:tasks.workunit.client.0.smithi016.stdout:5/326: mkdir df/d5b/d5d/d5f 0 2024-04-22T15:49:34.017 INFO:tasks.workunit.client.0.smithi016.stdout:6/380: dwrite d3/da/db/d14/d16/f3d [0,4194304] 0 2024-04-22T15:49:34.019 INFO:tasks.workunit.client.0.smithi016.stdout:9/328: truncate d9/fc 1099815 0 2024-04-22T15:49:34.020 INFO:tasks.workunit.client.0.smithi016.stdout:9/329: chown d9/de/l21 0 1 2024-04-22T15:49:34.021 INFO:tasks.workunit.client.0.smithi016.stdout:7/323: read d3/dc/d1b/d5a/d9/fb [7456760,127928] 0 2024-04-22T15:49:34.021 INFO:tasks.workunit.client.0.smithi016.stdout:0/341: write dc/d13/f16 [4158290,93683] 0 2024-04-22T15:49:34.026 INFO:tasks.workunit.client.0.smithi016.stdout:0/342: dread dc/d13/d14/d26/f5e [0,4194304] 0 2024-04-22T15:49:34.027 INFO:tasks.workunit.client.0.smithi016.stdout:4/302: link d7/d4e/d39/f48 d7/d4e/f60 0 2024-04-22T15:49:34.028 INFO:tasks.workunit.client.0.smithi016.stdout:5/327: fsync df/d15/d1a/f53 0 2024-04-22T15:49:34.030 INFO:tasks.workunit.client.0.smithi016.stdout:7/324: creat d3/dc/d1b/d31/d5f/f77 x:0 0 0 2024-04-22T15:49:34.033 INFO:tasks.workunit.client.0.smithi016.stdout:6/381: getdents d3/da/dd/d7b 0 2024-04-22T15:49:34.034 INFO:tasks.workunit.client.0.smithi016.stdout:4/303: rename d7/d24/d41/f4f to d7/d4e/d2f/d50/f61 0 2024-04-22T15:49:34.035 INFO:tasks.workunit.client.0.smithi016.stdout:0/343: dread dc/d13/d28/d65/d1f/f53 [0,4194304] 0 2024-04-22T15:49:34.036 INFO:tasks.workunit.client.0.smithi016.stdout:9/330: link d9/de/d23/d24/f39 d9/de/d23/d24/d4e/f5c 0 2024-04-22T15:49:34.037 INFO:tasks.workunit.client.0.smithi016.stdout:0/344: write dc/d13/d28/d65/d1f/f4c [1227180,89148] 0 2024-04-22T15:49:34.037 INFO:tasks.workunit.client.0.smithi016.stdout:9/331: dread - d9/de/d26/f44 zero size 2024-04-22T15:49:34.038 INFO:tasks.workunit.client.0.smithi016.stdout:0/345: truncate dc/f68 780566 0 2024-04-22T15:49:34.041 INFO:tasks.workunit.client.0.smithi016.stdout:5/328: mknod df/d15/c60 0 2024-04-22T15:49:34.042 INFO:tasks.workunit.client.0.smithi016.stdout:6/382: mknod d3/da/db/d44/d68/c80 0 2024-04-22T15:49:34.044 INFO:tasks.workunit.client.0.smithi016.stdout:7/325: dread d3/dc/d1b/d5a/d9/f25 [0,4194304] 0 2024-04-22T15:49:34.044 INFO:tasks.workunit.client.0.smithi016.stdout:6/383: chown d3/da/d12 234392437 1 2024-04-22T15:49:34.044 INFO:tasks.workunit.client.0.smithi016.stdout:4/304: symlink d7/d24/d41/l62 0 2024-04-22T15:49:34.052 INFO:tasks.workunit.client.0.smithi016.stdout:0/346: dwrite dc/d13/d28/d65/f42 [0,4194304] 0 2024-04-22T15:49:34.053 INFO:tasks.workunit.client.0.smithi016.stdout:6/384: creat d3/da/db/d44/f81 x:0 0 0 2024-04-22T15:49:34.054 INFO:tasks.workunit.client.0.smithi016.stdout:7/326: rename d3/d32/c5c to d3/dc/d1b/d5a/d69/c78 0 2024-04-22T15:49:34.055 INFO:tasks.workunit.client.0.smithi016.stdout:4/305: symlink d7/d22/l63 0 2024-04-22T15:49:34.057 INFO:tasks.workunit.client.0.smithi016.stdout:9/332: mkdir d9/d5d 0 2024-04-22T15:49:34.057 INFO:tasks.workunit.client.0.smithi016.stdout:0/347: read - dc/d13/d14/d26/d32/f5b zero size 2024-04-22T15:49:34.059 INFO:tasks.workunit.client.0.smithi016.stdout:7/327: chown d3/dc/d1b/d5a/f17 114 1 2024-04-22T15:49:34.060 INFO:tasks.workunit.client.0.smithi016.stdout:5/329: dread df/d15/d1a/d28/f3f [0,4194304] 0 2024-04-22T15:49:34.062 INFO:tasks.workunit.client.0.smithi016.stdout:5/330: write df/d22/f3b [506566,127757] 0 2024-04-22T15:49:34.064 INFO:tasks.workunit.client.0.smithi016.stdout:9/333: creat d9/de/d23/d24/d25/d36/f5e x:0 0 0 2024-04-22T15:49:34.070 INFO:tasks.workunit.client.0.smithi016.stdout:7/328: creat d3/dc/d1b/d5a/d69/f79 x:0 0 0 2024-04-22T15:49:34.072 INFO:tasks.workunit.client.0.smithi016.stdout:9/334: mknod d9/de/d23/d24/d4e/c5f 0 2024-04-22T15:49:34.078 INFO:tasks.workunit.client.0.smithi016.stdout:6/385: getdents d3/d53 0 2024-04-22T15:49:34.079 INFO:tasks.workunit.client.0.smithi016.stdout:7/329: mkdir d3/d32/d4c/d7a 0 2024-04-22T15:49:34.083 INFO:tasks.workunit.client.0.smithi016.stdout:9/335: unlink d9/de/d23/d24/d25/f52 0 2024-04-22T15:49:34.084 INFO:tasks.workunit.client.0.smithi016.stdout:7/330: creat d3/dc/d1b/d31/d5f/f7b x:0 0 0 2024-04-22T15:49:34.084 INFO:tasks.workunit.client.0.smithi016.stdout:4/306: dwrite d7/f8 [0,4194304] 0 2024-04-22T15:49:34.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:33 smithi027 ceph-mon[34922]: pgmap v29: 65 pgs: 65 active+clean; 815 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 17 MiB/s rd, 101 MiB/s wr, 440 op/s 2024-04-22T15:49:34.086 INFO:tasks.workunit.client.0.smithi016.stdout:0/348: truncate dc/d13/f16 3105525 0 2024-04-22T15:49:34.090 INFO:tasks.workunit.client.0.smithi016.stdout:9/336: unlink d9/de/d23/f29 0 2024-04-22T15:49:34.092 INFO:tasks.workunit.client.0.smithi016.stdout:4/307: chown c6 75 1 2024-04-22T15:49:34.094 INFO:tasks.workunit.client.0.smithi016.stdout:0/349: dwrite dc/d13/d28/d65/d1f/f39 [0,4194304] 0 2024-04-22T15:49:34.100 INFO:tasks.workunit.client.0.smithi016.stdout:6/386: truncate d3/da/d12/f1c 2949845 0 2024-04-22T15:49:34.102 INFO:tasks.workunit.client.0.smithi016.stdout:7/331: link d3/dc/f16 d3/d14/f7c 0 2024-04-22T15:49:34.113 INFO:tasks.workunit.client.0.smithi016.stdout:9/337: link l0 d9/d57/l60 0 2024-04-22T15:49:34.116 INFO:tasks.workunit.client.0.smithi016.stdout:4/308: write d7/d4e/d39/f48 [163633,26351] 0 2024-04-22T15:49:34.119 INFO:tasks.workunit.client.0.smithi016.stdout:9/338: unlink d9/de/f18 0 2024-04-22T15:49:34.120 INFO:tasks.workunit.client.0.smithi016.stdout:9/339: write d9/de/d23/f35 [855994,62976] 0 2024-04-22T15:49:34.120 INFO:tasks.workunit.client.0.smithi016.stdout:6/387: creat d3/da/dd/f82 x:0 0 0 2024-04-22T15:49:34.121 INFO:tasks.workunit.client.0.smithi016.stdout:6/388: write d3/f8 [10730478,61330] 0 2024-04-22T15:49:34.122 INFO:tasks.workunit.client.0.smithi016.stdout:7/332: rename d3/dc/d22/l43 to d3/l7d 0 2024-04-22T15:49:34.122 INFO:tasks.workunit.client.0.smithi016.stdout:0/350: getdents dc/d13/d28/d65/d40 0 2024-04-22T15:49:34.124 INFO:tasks.workunit.client.0.smithi016.stdout:7/333: write d3/d32/f72 [993414,128578] 0 2024-04-22T15:49:34.124 INFO:tasks.workunit.client.0.smithi016.stdout:4/309: creat d7/d22/f64 x:0 0 0 2024-04-22T15:49:34.129 INFO:tasks.workunit.client.0.smithi016.stdout:0/351: mknod dc/d13/d28/d65/d1f/c6b 0 2024-04-22T15:49:34.130 INFO:tasks.workunit.client.0.smithi016.stdout:7/334: creat d3/dc/d1b/d31/d5f/f7e x:0 0 0 2024-04-22T15:49:34.131 INFO:tasks.workunit.client.0.smithi016.stdout:0/352: write dc/d13/d28/d65/d30/f44 [2706120,50667] 0 2024-04-22T15:49:34.131 INFO:tasks.workunit.client.0.smithi016.stdout:7/335: stat d3/d32/d4c/d7a 0 2024-04-22T15:49:34.132 INFO:tasks.workunit.client.0.smithi016.stdout:7/336: dread - d3/dc/d22/f6b zero size 2024-04-22T15:49:34.133 INFO:tasks.workunit.client.0.smithi016.stdout:4/310: rename d7/d22/c28 to d7/d21/d45/d49/c65 0 2024-04-22T15:49:34.134 INFO:tasks.workunit.client.0.smithi016.stdout:6/389: getdents d3/da/dd/d67/d6c 0 2024-04-22T15:49:34.135 INFO:tasks.workunit.client.0.smithi016.stdout:0/353: chown dc/d13/d28/d65/d40/d4a/d61 40622 1 2024-04-22T15:49:34.135 INFO:tasks.workunit.client.0.smithi016.stdout:6/390: write d3/da/dd/f82 [335281,29691] 0 2024-04-22T15:49:34.137 INFO:tasks.workunit.client.0.smithi016.stdout:6/391: read d3/f6 [1322535,77580] 0 2024-04-22T15:49:34.139 INFO:tasks.workunit.client.0.smithi016.stdout:4/311: creat d7/d21/d45/f66 x:0 0 0 2024-04-22T15:49:34.140 INFO:tasks.workunit.client.0.smithi016.stdout:0/354: rmdir dc/d13/d28/d65/d30 39 2024-04-22T15:49:34.141 INFO:tasks.workunit.client.0.smithi016.stdout:9/340: write d9/d37/f3c [771983,89265] 0 2024-04-22T15:49:34.145 INFO:tasks.workunit.client.0.smithi016.stdout:4/312: mkdir d7/d21/d45/d49/d67 0 2024-04-22T15:49:34.146 INFO:tasks.workunit.client.0.smithi016.stdout:0/355: creat dc/d13/d5d/f6c x:0 0 0 2024-04-22T15:49:34.146 INFO:tasks.workunit.client.0.smithi016.stdout:7/337: write d3/fa [1113187,7036] 0 2024-04-22T15:49:34.147 INFO:tasks.workunit.client.0.smithi016.stdout:0/356: chown dc/d13/d14/d26/d32 257018620 1 2024-04-22T15:49:34.148 INFO:tasks.workunit.client.0.smithi016.stdout:0/357: chown dc/d13/d14/d26/f5e 118874109 1 2024-04-22T15:49:34.148 INFO:tasks.workunit.client.0.smithi016.stdout:1/513: sync 2024-04-22T15:49:34.149 INFO:tasks.workunit.client.0.smithi016.stdout:0/358: write dc/f68 [958613,107127] 0 2024-04-22T15:49:34.149 INFO:tasks.workunit.client.0.smithi016.stdout:1/514: chown d3/d4/d3a/d6b/l9b 13472576 1 2024-04-22T15:49:34.150 INFO:tasks.workunit.client.0.smithi016.stdout:4/313: mknod d7/d21/c68 0 2024-04-22T15:49:34.151 INFO:tasks.workunit.client.0.smithi016.stdout:4/314: dread - d7/d22/f54 zero size 2024-04-22T15:49:34.152 INFO:tasks.workunit.client.0.smithi016.stdout:4/315: read d7/f8 [2684051,107775] 0 2024-04-22T15:49:34.154 INFO:tasks.workunit.client.0.smithi016.stdout:1/515: mknod d3/d84/caf 0 2024-04-22T15:49:34.156 INFO:tasks.workunit.client.0.smithi016.stdout:7/338: creat d3/dc/d33/f7f x:0 0 0 2024-04-22T15:49:34.158 INFO:tasks.workunit.client.0.smithi016.stdout:0/359: rename dc/d13/l24 to dc/d13/d28/d65/d40/l6d 0 2024-04-22T15:49:34.159 INFO:tasks.workunit.client.0.smithi016.stdout:6/392: truncate d3/da/db/f61 2102761 0 2024-04-22T15:49:34.160 INFO:tasks.workunit.client.0.smithi016.stdout:6/393: readlink d3/da/db/d44/l47 0 2024-04-22T15:49:34.160 INFO:tasks.workunit.client.0.smithi016.stdout:0/360: write dc/d13/d28/d65/d1f/d48/f64 [490467,54646] 0 2024-04-22T15:49:34.160 INFO:tasks.workunit.client.0.smithi016.stdout:4/316: rmdir d7/d24/d51 39 2024-04-22T15:49:34.161 INFO:tasks.workunit.client.0.smithi016.stdout:4/317: dread - d7/d22/f64 zero size 2024-04-22T15:49:34.162 INFO:tasks.workunit.client.0.smithi016.stdout:4/318: write d7/f10 [696663,124525] 0 2024-04-22T15:49:34.163 INFO:tasks.workunit.client.0.smithi016.stdout:1/516: mknod d3/d4/da/cb0 0 2024-04-22T15:49:34.165 INFO:tasks.workunit.client.0.smithi016.stdout:6/394: mkdir d3/d53/d83 0 2024-04-22T15:49:34.246 INFO:tasks.workunit.client.0.smithi016.stdout:9/341: getdents d9/de/d26 0 2024-04-22T15:49:34.247 INFO:tasks.workunit.client.0.smithi016.stdout:4/319: dread d7/d4e/d39/f48 [0,4194304] 0 2024-04-22T15:49:34.247 INFO:tasks.workunit.client.0.smithi016.stdout:0/361: mknod dc/d13/d14/d55/c6e 0 2024-04-22T15:49:34.247 INFO:tasks.workunit.client.0.smithi016.stdout:1/517: fdatasync d3/d4/d19/f66 0 2024-04-22T15:49:34.248 INFO:tasks.workunit.client.0.smithi016.stdout:6/395: rmdir d3/da/dd 39 2024-04-22T15:49:34.254 INFO:tasks.workunit.client.0.smithi016.stdout:1/518: dread d3/f4f [0,4194304] 0 2024-04-22T15:49:34.254 INFO:tasks.workunit.client.0.smithi016.stdout:0/362: dwrite dc/d13/d28/d65/d1f/f4c [0,4194304] 0 2024-04-22T15:49:34.259 INFO:tasks.workunit.client.0.smithi016.stdout:9/342: getdents d9/d5d 0 2024-04-22T15:49:34.261 INFO:tasks.workunit.client.0.smithi016.stdout:9/343: write d9/de/d26/f2a [133472,92658] 0 2024-04-22T15:49:34.261 INFO:tasks.workunit.client.0.smithi016.stdout:1/519: dwrite d3/d28/f8b [0,4194304] 0 2024-04-22T15:49:34.262 INFO:tasks.workunit.client.0.smithi016.stdout:1/520: readlink d3/d4/d15/l71 0 2024-04-22T15:49:34.269 INFO:tasks.workunit.client.0.smithi016.stdout:1/521: dwrite d3/d4/da/f26 [4194304,4194304] 0 2024-04-22T15:49:34.271 INFO:tasks.workunit.client.0.smithi016.stdout:0/363: mkdir dc/d13/d6f 0 2024-04-22T15:49:34.275 INFO:tasks.workunit.client.0.smithi016.stdout:9/344: mknod d9/de/d23/d24/d25/d36/c61 0 2024-04-22T15:49:34.277 INFO:tasks.workunit.client.0.smithi016.stdout:9/345: stat d9/de/d23/d24/d25/d36/f5e 0 2024-04-22T15:49:34.277 INFO:tasks.workunit.client.0.smithi016.stdout:6/396: dread d3/f8 [4194304,4194304] 0 2024-04-22T15:49:34.280 INFO:tasks.workunit.client.0.smithi016.stdout:6/397: mkdir d3/da/d12/d13/d38/d84 0 2024-04-22T15:49:34.282 INFO:tasks.workunit.client.0.smithi016.stdout:6/398: chown d3/da/db/d14/d16/f5f 4 1 2024-04-22T15:49:34.282 INFO:tasks.workunit.client.0.smithi016.stdout:0/364: mkdir dc/d13/d70 0 2024-04-22T15:49:34.282 INFO:tasks.workunit.client.0.smithi016.stdout:4/320: link d7/d21/d45/d49/f4d d7/f69 0 2024-04-22T15:49:34.288 INFO:tasks.workunit.client.0.smithi016.stdout:4/321: rename d7/f10 to d7/d24/d51/f6a 0 2024-04-22T15:49:34.289 INFO:tasks.workunit.client.0.smithi016.stdout:0/365: creat dc/d13/d28/d65/d40/d4a/d59/f71 x:0 0 0 2024-04-22T15:49:34.291 INFO:tasks.workunit.client.0.smithi016.stdout:0/366: truncate dc/d13/d5d/f5f 175597 0 2024-04-22T15:49:34.292 INFO:tasks.workunit.client.0.smithi016.stdout:6/399: rename d3/da/d12/d13/d46/f4f to d3/da/dd/d1b/f85 0 2024-04-22T15:49:34.292 INFO:tasks.workunit.client.0.smithi016.stdout:9/346: dread d9/f1b [0,4194304] 0 2024-04-22T15:49:34.293 INFO:tasks.workunit.client.0.smithi016.stdout:9/347: readlink d9/de/l21 0 2024-04-22T15:49:34.293 INFO:tasks.workunit.client.0.smithi016.stdout:4/322: unlink d7/d24/d51/c5e 0 2024-04-22T15:49:34.293 INFO:tasks.workunit.client.0.smithi016.stdout:9/348: chown d9/de/d23/d24/d4e/d4f 1 1 2024-04-22T15:49:34.295 INFO:tasks.workunit.client.0.smithi016.stdout:6/400: fsync d3/f6 0 2024-04-22T15:49:34.295 INFO:tasks.workunit.client.0.smithi016.stdout:8/320: sync 2024-04-22T15:49:34.296 INFO:tasks.workunit.client.0.smithi016.stdout:8/321: chown d2/db/d1c 81596624 1 2024-04-22T15:49:34.296 INFO:tasks.workunit.client.0.smithi016.stdout:6/401: truncate d3/da/db/d14/d4c/d58/f75 473515 0 2024-04-22T15:49:34.297 INFO:tasks.workunit.client.0.smithi016.stdout:4/323: symlink d7/d21/d45/l6b 0 2024-04-22T15:49:34.297 INFO:tasks.workunit.client.0.smithi016.stdout:8/322: dread - d2/d10/d43/d4a/f5f zero size 2024-04-22T15:49:34.298 INFO:tasks.workunit.client.0.smithi016.stdout:6/402: read - d3/da/dd/f18 zero size 2024-04-22T15:49:34.302 INFO:tasks.workunit.client.0.smithi016.stdout:9/349: mkdir d9/d5d/d62 0 2024-04-22T15:49:34.307 INFO:tasks.workunit.client.0.smithi016.stdout:8/323: creat d2/d10/d43/d4a/d28/d56/f64 x:0 0 0 2024-04-22T15:49:34.308 INFO:tasks.workunit.client.0.smithi016.stdout:4/324: dwrite d7/d24/d41/f57 [0,4194304] 0 2024-04-22T15:49:34.309 INFO:tasks.workunit.client.0.smithi016.stdout:6/403: getdents d3/da/dd/d1b/d57 0 2024-04-22T15:49:34.310 INFO:tasks.workunit.client.0.smithi016.stdout:9/350: creat d9/de/d23/d24/d4e/d4f/f63 x:0 0 0 2024-04-22T15:49:34.312 INFO:tasks.workunit.client.0.smithi016.stdout:6/404: read d3/da/dd/d1b/f62 [38316,51334] 0 2024-04-22T15:49:34.312 INFO:tasks.workunit.client.0.smithi016.stdout:8/324: mkdir d2/db/d1c/d65 0 2024-04-22T15:49:34.317 INFO:tasks.workunit.client.0.smithi016.stdout:0/367: truncate dc/f1e 2301022 0 2024-04-22T15:49:34.318 INFO:tasks.workunit.client.0.smithi016.stdout:8/325: write d2/d10/d43/d4a/d28/f4d [4260415,2613] 0 2024-04-22T15:49:34.319 INFO:tasks.workunit.client.0.smithi016.stdout:6/405: chown d3/da/dd/c56 1043259 1 2024-04-22T15:49:34.321 INFO:tasks.workunit.client.0.smithi016.stdout:9/351: dwrite d9/de/f1a [0,4194304] 0 2024-04-22T15:49:34.323 INFO:tasks.workunit.client.0.smithi016.stdout:4/325: link d7/d24/c2c d7/d24/c6c 0 2024-04-22T15:49:34.324 INFO:tasks.workunit.client.0.smithi016.stdout:4/326: fsync d7/d3c/f44 0 2024-04-22T15:49:34.325 INFO:tasks.workunit.client.0.smithi016.stdout:0/368: creat dc/d13/d70/f72 x:0 0 0 2024-04-22T15:49:34.326 INFO:tasks.workunit.client.0.smithi016.stdout:0/369: chown dc/d13/d28/f6a 8188 1 2024-04-22T15:49:34.326 INFO:tasks.workunit.client.0.smithi016.stdout:9/352: creat d9/d37/f64 x:0 0 0 2024-04-22T15:49:34.329 INFO:tasks.workunit.client.0.smithi016.stdout:8/326: rename d2/db to d2/d10/d43/d4a/d28/d44/d66 0 2024-04-22T15:49:34.330 INFO:tasks.workunit.client.0.smithi016.stdout:6/406: truncate d3/da/db/d14/d16/f20 932386 0 2024-04-22T15:49:34.331 INFO:tasks.workunit.client.0.smithi016.stdout:4/327: creat d7/d21/d45/f6d x:0 0 0 2024-04-22T15:49:34.337 INFO:tasks.workunit.client.0.smithi016.stdout:9/353: getdents d9/d5d/d62 0 2024-04-22T15:49:34.340 INFO:tasks.workunit.client.0.smithi016.stdout:4/328: rename d7/l1c to d7/d21/d42/d59/l6e 0 2024-04-22T15:49:34.342 INFO:tasks.workunit.client.0.smithi016.stdout:6/407: creat d3/da/db/d14/f86 x:0 0 0 2024-04-22T15:49:34.343 INFO:tasks.workunit.client.0.smithi016.stdout:9/354: symlink d9/de/d23/d24/d25/l65 0 2024-04-22T15:49:34.344 INFO:tasks.workunit.client.0.smithi016.stdout:0/370: dread dc/d13/d28/d65/d30/f44 [0,4194304] 0 2024-04-22T15:49:34.345 INFO:tasks.workunit.client.0.smithi016.stdout:9/355: chown d9/de/d23/d30/f58 10267773 1 2024-04-22T15:49:34.347 INFO:tasks.workunit.client.0.smithi016.stdout:4/329: unlink d7/d21/l30 0 2024-04-22T15:49:34.347 INFO:tasks.workunit.client.0.smithi016.stdout:0/371: truncate dc/d13/d14/d26/d32/f33 4977979 0 2024-04-22T15:49:34.348 INFO:tasks.workunit.client.0.smithi016.stdout:9/356: truncate d9/de/d23/d24/d25/f41 197088 0 2024-04-22T15:49:34.354 INFO:tasks.workunit.client.0.smithi016.stdout:4/330: unlink d7/d24/c2c 0 2024-04-22T15:49:34.356 INFO:tasks.workunit.client.0.smithi016.stdout:0/372: dwrite dc/d13/d28/d65/d57/f67 [0,4194304] 0 2024-04-22T15:49:34.358 INFO:tasks.workunit.client.0.smithi016.stdout:9/357: symlink d9/d57/l66 0 2024-04-22T15:49:34.359 INFO:tasks.workunit.client.0.smithi016.stdout:4/331: chown d7/d21/d45/d49/f4d 5 1 2024-04-22T15:49:34.362 INFO:tasks.workunit.client.0.smithi016.stdout:0/373: creat dc/d13/d28/d65/d40/f73 x:0 0 0 2024-04-22T15:49:34.362 INFO:tasks.workunit.client.0.smithi016.stdout:9/358: symlink d9/de/d23/d30/l67 0 2024-04-22T15:49:34.368 INFO:tasks.workunit.client.0.smithi016.stdout:0/374: unlink dc/d13/d28/d65/d40/c60 0 2024-04-22T15:49:34.370 INFO:tasks.workunit.client.0.smithi016.stdout:9/359: unlink d9/de/f48 0 2024-04-22T15:49:34.371 INFO:tasks.workunit.client.0.smithi016.stdout:9/360: write d9/d20/f40 [2522682,110961] 0 2024-04-22T15:49:34.372 INFO:tasks.workunit.client.0.smithi016.stdout:0/375: dread dc/d13/d28/d65/d30/f44 [0,4194304] 0 2024-04-22T15:49:34.377 INFO:tasks.workunit.client.0.smithi016.stdout:9/361: mknod d9/d37/c68 0 2024-04-22T15:49:34.378 INFO:tasks.workunit.client.0.smithi016.stdout:9/362: chown d9/de/d23/d30/l67 11191954 1 2024-04-22T15:49:34.379 INFO:tasks.workunit.client.0.smithi016.stdout:9/363: write d9/d20/f59 [919589,13766] 0 2024-04-22T15:49:34.381 INFO:tasks.workunit.client.0.smithi016.stdout:4/332: getdents d7 0 2024-04-22T15:49:34.382 INFO:tasks.workunit.client.0.smithi016.stdout:0/376: mknod dc/d13/d6f/c74 0 2024-04-22T15:49:34.385 INFO:tasks.workunit.client.0.smithi016.stdout:0/377: dread dc/d13/d28/d65/f42 [0,4194304] 0 2024-04-22T15:49:34.385 INFO:tasks.workunit.client.0.smithi016.stdout:9/364: rmdir d9/d20 39 2024-04-22T15:49:34.386 INFO:tasks.workunit.client.0.smithi016.stdout:4/333: mkdir d7/d4e/d2f/d6f 0 2024-04-22T15:49:34.388 INFO:tasks.workunit.client.0.smithi016.stdout:4/334: truncate d7/d21/d45/f66 955495 0 2024-04-22T15:49:34.391 INFO:tasks.workunit.client.0.smithi016.stdout:9/365: rename d9/de/d23/d24/d4e/f5c to d9/de/d23/d24/d4e/d4f/f69 0 2024-04-22T15:49:34.394 INFO:tasks.workunit.client.0.smithi016.stdout:0/378: fsync dc/f1e 0 2024-04-22T15:49:34.395 INFO:tasks.workunit.client.0.smithi016.stdout:9/366: symlink d9/de/l6a 0 2024-04-22T15:49:34.399 INFO:tasks.workunit.client.0.smithi016.stdout:9/367: dread - d9/f1c zero size 2024-04-22T15:49:34.399 INFO:tasks.workunit.client.0.smithi016.stdout:0/379: dwrite dc/d13/d28/d65/d57/f67 [4194304,4194304] 0 2024-04-22T15:49:34.400 INFO:tasks.workunit.client.0.smithi016.stdout:9/368: read d9/f3b [1324248,74712] 0 2024-04-22T15:49:34.403 INFO:tasks.workunit.client.0.smithi016.stdout:0/380: mkdir dc/d13/d28/d65/d30/d75 0 2024-04-22T15:49:34.404 INFO:tasks.workunit.client.0.smithi016.stdout:9/369: creat d9/de/d23/d24/d25/d36/f6b x:0 0 0 2024-04-22T15:49:34.406 INFO:tasks.workunit.client.0.smithi016.stdout:2/297: sync 2024-04-22T15:49:34.406 INFO:tasks.workunit.client.0.smithi016.stdout:3/346: sync 2024-04-22T15:49:34.406 INFO:tasks.workunit.client.0.smithi016.stdout:5/331: sync 2024-04-22T15:49:34.408 INFO:tasks.workunit.client.0.smithi016.stdout:5/332: stat df/l5a 0 2024-04-22T15:49:34.408 INFO:tasks.workunit.client.0.smithi016.stdout:4/335: dwrite d7/f11 [0,4194304] 0 2024-04-22T15:49:34.412 INFO:tasks.workunit.client.0.smithi016.stdout:5/333: write df/d15/d1a/f4b [638352,4207] 0 2024-04-22T15:49:34.412 INFO:tasks.workunit.client.0.smithi016.stdout:3/347: write d7/d17/d18/d32/f57 [959890,56138] 0 2024-04-22T15:49:34.415 INFO:tasks.workunit.client.0.smithi016.stdout:2/298: creat d2/d23/d27/d34/d60/f6f x:0 0 0 2024-04-22T15:49:34.415 INFO:tasks.workunit.client.0.smithi016.stdout:9/370: dread d9/de/d23/d24/d25/f3d [0,4194304] 0 2024-04-22T15:49:34.419 INFO:tasks.workunit.client.0.smithi016.stdout:0/381: dread dc/d13/d28/d65/d1f/f4c [0,4194304] 0 2024-04-22T15:49:34.420 INFO:tasks.workunit.client.0.smithi016.stdout:2/299: mknod d2/d23/d47/c70 0 2024-04-22T15:49:34.421 INFO:tasks.workunit.client.0.smithi016.stdout:9/371: mknod d9/de/d23/d24/d4e/c6c 0 2024-04-22T15:49:34.422 INFO:tasks.workunit.client.0.smithi016.stdout:2/300: write d2/d23/d47/f66 [264560,62970] 0 2024-04-22T15:49:34.423 INFO:tasks.workunit.client.0.smithi016.stdout:5/334: link df/d15/f45 df/d15/d1a/d28/d33/f61 0 2024-04-22T15:49:34.425 INFO:tasks.workunit.client.0.smithi016.stdout:5/335: chown df/d15/d1a/l1b 4 1 2024-04-22T15:49:34.425 INFO:tasks.workunit.client.0.smithi016.stdout:4/336: dread d7/d21/d42/f4c [0,4194304] 0 2024-04-22T15:49:34.426 INFO:tasks.workunit.client.0.smithi016.stdout:5/336: chown df/d22/l41 872 1 2024-04-22T15:49:34.427 INFO:tasks.workunit.client.0.smithi016.stdout:0/382: dwrite dc/d13/d14/d26/d32/f5b [0,4194304] 0 2024-04-22T15:49:34.501 INFO:tasks.workunit.client.0.smithi016.stdout:9/372: symlink d9/d20/l6d 0 2024-04-22T15:49:34.501 INFO:tasks.workunit.client.0.smithi016.stdout:0/383: readlink dc/d13/d28/d65/d1f/l29 0 2024-04-22T15:49:34.502 INFO:tasks.workunit.client.0.smithi016.stdout:3/348: truncate d7/d17/d18/d32/d3c/d47/f33 436013 0 2024-04-22T15:49:34.502 INFO:tasks.workunit.client.0.smithi016.stdout:5/337: chown df/d15/d1a/d50/c51 105 1 2024-04-22T15:49:34.502 INFO:tasks.workunit.client.0.smithi016.stdout:4/337: rename d7/d4e/l37 to d7/d4e/d2f/d33/l70 0 2024-04-22T15:49:34.503 INFO:tasks.workunit.client.0.smithi016.stdout:3/349: fsync d7/d17/f3e 0 2024-04-22T15:49:34.505 INFO:tasks.workunit.client.0.smithi016.stdout:9/373: creat d9/d5d/d62/f6e x:0 0 0 2024-04-22T15:49:34.507 INFO:tasks.workunit.client.0.smithi016.stdout:5/338: creat df/d15/d1a/d28/d33/f62 x:0 0 0 2024-04-22T15:49:34.509 INFO:tasks.workunit.client.0.smithi016.stdout:4/338: mknod d7/d21/d45/c71 0 2024-04-22T15:49:34.509 INFO:tasks.workunit.client.0.smithi016.stdout:4/339: chown d7 140649 1 2024-04-22T15:49:34.514 INFO:tasks.workunit.client.0.smithi016.stdout:4/340: dwrite d7/f1e [8388608,4194304] 0 2024-04-22T15:49:34.515 INFO:tasks.workunit.client.0.smithi016.stdout:9/374: creat d9/d20/f6f x:0 0 0 2024-04-22T15:49:34.515 INFO:tasks.workunit.client.0.smithi016.stdout:4/341: readlink d7/d4e/d2f/l3f 0 2024-04-22T15:49:34.516 INFO:tasks.workunit.client.0.smithi016.stdout:9/375: chown d9/de/f28 0 1 2024-04-22T15:49:34.516 INFO:tasks.workunit.client.0.smithi016.stdout:4/342: chown d7/f16 14 1 2024-04-22T15:49:34.520 INFO:tasks.workunit.client.0.smithi016.stdout:5/339: creat df/d5b/d5d/d5f/f63 x:0 0 0 2024-04-22T15:49:34.521 INFO:tasks.workunit.client.0.smithi016.stdout:5/340: read df/d22/f2a [2466549,75335] 0 2024-04-22T15:49:34.522 INFO:tasks.workunit.client.0.smithi016.stdout:9/376: mknod d9/de/d23/d30/c70 0 2024-04-22T15:49:34.523 INFO:tasks.workunit.client.0.smithi016.stdout:9/377: chown d9/l1e 51154218 1 2024-04-22T15:49:34.523 INFO:tasks.workunit.client.0.smithi016.stdout:4/343: symlink d7/d4e/d39/l72 0 2024-04-22T15:49:34.527 INFO:tasks.workunit.client.0.smithi016.stdout:5/341: dwrite df/d15/d1a/d28/d33/f39 [0,4194304] 0 2024-04-22T15:49:34.528 INFO:tasks.workunit.client.0.smithi016.stdout:5/342: write f2 [186384,21791] 0 2024-04-22T15:49:34.532 INFO:tasks.workunit.client.0.smithi016.stdout:4/344: getdents d7/d21/d45/d49/d67 0 2024-04-22T15:49:34.539 INFO:tasks.workunit.client.0.smithi016.stdout:4/345: write d7/d24/d51/f6a [277723,111784] 0 2024-04-22T15:49:34.548 INFO:tasks.workunit.client.0.smithi016.stdout:1/522: sync 2024-04-22T15:49:34.548 INFO:tasks.workunit.client.0.smithi016.stdout:7/339: sync 2024-04-22T15:49:34.549 INFO:tasks.workunit.client.0.smithi016.stdout:7/340: readlink d3/d32/l5d 0 2024-04-22T15:49:34.550 INFO:tasks.workunit.client.0.smithi016.stdout:7/341: dread - d3/dc/d1b/d5a/d9/df/f70 zero size 2024-04-22T15:49:34.550 INFO:tasks.workunit.client.0.smithi016.stdout:1/523: unlink d3/d4/d19/d20/f44 0 2024-04-22T15:49:34.551 INFO:tasks.workunit.client.0.smithi016.stdout:1/524: chown d3/d4/d15/l71 986346 1 2024-04-22T15:49:34.554 INFO:tasks.workunit.client.0.smithi016.stdout:1/525: symlink d3/d4/d3a/lb1 0 2024-04-22T15:49:34.555 INFO:tasks.workunit.client.0.smithi016.stdout:7/342: mknod d3/dc/d1b/d5a/c80 0 2024-04-22T15:49:34.556 INFO:tasks.workunit.client.0.smithi016.stdout:7/343: dread - d3/dc/d1b/d5a/d9/df/f70 zero size 2024-04-22T15:49:34.557 INFO:tasks.workunit.client.0.smithi016.stdout:7/344: chown d3/dc/d1b/d31/l47 7 1 2024-04-22T15:49:34.558 INFO:tasks.workunit.client.0.smithi016.stdout:7/345: fsync d3/f28 0 2024-04-22T15:49:34.559 INFO:tasks.workunit.client.0.smithi016.stdout:1/526: creat d3/d4/d19/fb2 x:0 0 0 2024-04-22T15:49:34.560 INFO:tasks.workunit.client.0.smithi016.stdout:1/527: read d3/d28/f8b [751849,95377] 0 2024-04-22T15:49:34.565 INFO:tasks.workunit.client.0.smithi016.stdout:1/528: dwrite d3/d4/d19/fb2 [0,4194304] 0 2024-04-22T15:49:34.567 INFO:tasks.workunit.client.0.smithi016.stdout:1/529: write d3/d4/d15/d38/d63/f75 [4643754,117409] 0 2024-04-22T15:49:34.570 INFO:tasks.workunit.client.0.smithi016.stdout:1/530: fdatasync d3/d4/d19/d20/f33 0 2024-04-22T15:49:34.576 INFO:tasks.workunit.client.0.smithi016.stdout:1/531: dwrite d3/d4/d19/d20/f21 [4194304,4194304] 0 2024-04-22T15:49:34.578 INFO:tasks.workunit.client.0.smithi016.stdout:1/532: unlink d3/d2e/f6d 0 2024-04-22T15:49:34.583 INFO:tasks.workunit.client.0.smithi016.stdout:1/533: rmdir d3/d4/d3a/d6b 39 2024-04-22T15:49:34.584 INFO:tasks.workunit.client.0.smithi016.stdout:6/408: sync 2024-04-22T15:49:34.584 INFO:tasks.workunit.client.0.smithi016.stdout:8/327: sync 2024-04-22T15:49:34.584 INFO:tasks.workunit.client.0.smithi016.stdout:2/301: sync 2024-04-22T15:49:34.584 INFO:tasks.workunit.client.0.smithi016.stdout:1/534: write d3/d4/d19/d20/f33 [2763151,131004] 0 2024-04-22T15:49:34.587 INFO:tasks.workunit.client.0.smithi016.stdout:2/302: dread d2/de/f5d [0,4194304] 0 2024-04-22T15:49:34.588 INFO:tasks.workunit.client.0.smithi016.stdout:6/409: creat d3/da/db/d14/d16/f87 x:0 0 0 2024-04-22T15:49:34.588 INFO:tasks.workunit.client.0.smithi016.stdout:6/410: chown d3/da/db 465866822 1 2024-04-22T15:49:34.589 INFO:tasks.workunit.client.0.smithi016.stdout:1/535: mkdir d3/d4/d3a/d91/db3 0 2024-04-22T15:49:34.589 INFO:tasks.workunit.client.0.smithi016.stdout:6/411: write d3/da/d12/d13/f27 [2137504,85379] 0 2024-04-22T15:49:34.590 INFO:tasks.workunit.client.0.smithi016.stdout:6/412: chown d3/da/d12 38 1 2024-04-22T15:49:34.592 INFO:tasks.workunit.client.0.smithi016.stdout:8/328: write d2/d10/d43/d4a/d28/d44/d66/fc [949340,2562] 0 2024-04-22T15:49:34.592 INFO:tasks.workunit.client.0.smithi016.stdout:8/329: chown d2/d10/d43/d4a/d28/d56 419 1 2024-04-22T15:49:34.593 INFO:tasks.workunit.client.0.smithi016.stdout:2/303: truncate d2/f18 398030 0 2024-04-22T15:49:34.596 INFO:tasks.workunit.client.0.smithi016.stdout:6/413: mknod d3/da/d12/d13/c88 0 2024-04-22T15:49:34.597 INFO:tasks.workunit.client.0.smithi016.stdout:0/384: dwrite dc/d13/d14/d26/d32/f4e [0,4194304] 0 2024-04-22T15:49:34.599 INFO:tasks.workunit.client.0.smithi016.stdout:8/330: mkdir d2/d10/d47/d67 0 2024-04-22T15:49:34.599 INFO:tasks.workunit.client.0.smithi016.stdout:1/536: creat d3/d84/d97/fb4 x:0 0 0 2024-04-22T15:49:34.638 INFO:tasks.workunit.client.0.smithi016.stdout:3/350: write d7/d17/f2e [447268,35417] 0 2024-04-22T15:49:34.639 INFO:tasks.workunit.client.0.smithi016.stdout:8/331: write d2/d10/d43/d4a/d28/d56/f64 [388544,55334] 0 2024-04-22T15:49:34.641 INFO:tasks.workunit.client.0.smithi016.stdout:9/378: write d9/de/d23/d24/d4e/f5b [1591506,48349] 0 2024-04-22T15:49:34.641 INFO:tasks.workunit.client.0.smithi016.stdout:4/346: truncate d7/f1e 9499924 0 2024-04-22T15:49:34.642 INFO:tasks.workunit.client.0.smithi016.stdout:0/385: dread dc/d13/d28/d65/d1f/f53 [0,4194304] 0 2024-04-22T15:49:34.644 INFO:tasks.workunit.client.0.smithi016.stdout:7/346: truncate d3/dc/f1e 72665 0 2024-04-22T15:49:34.646 INFO:tasks.workunit.client.0.smithi016.stdout:3/351: truncate d7/d17/d18/d27/f4f 4930520 0 2024-04-22T15:49:34.647 INFO:tasks.workunit.client.0.smithi016.stdout:7/347: readlink d3/dc/d1b/l62 0 2024-04-22T15:49:34.650 INFO:tasks.workunit.client.0.smithi016.stdout:1/537: creat d3/d4/d3a/d91/fb5 x:0 0 0 2024-04-22T15:49:34.650 INFO:tasks.workunit.client.0.smithi016.stdout:4/347: creat d7/d22/f73 x:0 0 0 2024-04-22T15:49:34.652 INFO:tasks.workunit.client.0.smithi016.stdout:5/343: dwrite df/d15/d1a/d28/d33/f61 [0,4194304] 0 2024-04-22T15:49:34.652 INFO:tasks.workunit.client.0.smithi016.stdout:3/352: creat d7/d17/d18/f70 x:0 0 0 2024-04-22T15:49:34.652 INFO:tasks.workunit.client.0.smithi016.stdout:7/348: mknod d3/dc/d33/c81 0 2024-04-22T15:49:34.653 INFO:tasks.workunit.client.0.smithi016.stdout:1/538: chown d3/d4/d3a/d91 3 1 2024-04-22T15:49:34.657 INFO:tasks.workunit.client.0.smithi016.stdout:9/379: dwrite d9/de/f28 [0,4194304] 0 2024-04-22T15:49:34.659 INFO:tasks.workunit.client.0.smithi016.stdout:0/386: dwrite dc/d13/d28/d65/d1f/d48/f64 [0,4194304] 0 2024-04-22T15:49:34.663 INFO:tasks.workunit.client.0.smithi016.stdout:0/387: fdatasync dc/d13/d5d/f6c 0 2024-04-22T15:49:34.664 INFO:tasks.workunit.client.0.smithi016.stdout:6/414: link d3/da/d12/f15 d3/da/d12/d13/d46/f89 0 2024-04-22T15:49:34.665 INFO:tasks.workunit.client.0.smithi016.stdout:6/415: dread - d3/da/f69 zero size 2024-04-22T15:49:34.667 INFO:tasks.workunit.client.0.smithi016.stdout:1/539: truncate d3/d4/d3a/d6b/f94 526612 0 2024-04-22T15:49:34.669 INFO:tasks.workunit.client.0.smithi016.stdout:5/344: write df/d15/d1a/d28/f3f [706434,123819] 0 2024-04-22T15:49:34.670 INFO:tasks.workunit.client.0.smithi016.stdout:3/353: mkdir d7/d17/d18/d32/d42/d63/d71 0 2024-04-22T15:49:34.671 INFO:tasks.workunit.client.0.smithi016.stdout:9/380: creat d9/d37/f71 x:0 0 0 2024-04-22T15:49:34.671 INFO:tasks.workunit.client.0.smithi016.stdout:5/345: chown df/d22/f2a 125157 1 2024-04-22T15:49:34.672 INFO:tasks.workunit.client.0.smithi016.stdout:4/348: creat d7/d24/d41/f74 x:0 0 0 2024-04-22T15:49:34.672 INFO:tasks.workunit.client.0.smithi016.stdout:3/354: stat d7/d17/d18/d27/f6e 0 2024-04-22T15:49:34.673 INFO:tasks.workunit.client.0.smithi016.stdout:9/381: chown l3 1 1 2024-04-22T15:49:34.674 INFO:tasks.workunit.client.0.smithi016.stdout:8/332: getdents d2/d10 0 2024-04-22T15:49:34.674 INFO:tasks.workunit.client.0.smithi016.stdout:3/355: stat d7/d17 0 2024-04-22T15:49:34.676 INFO:tasks.workunit.client.0.smithi016.stdout:0/388: creat dc/d13/d28/d65/d40/f76 x:0 0 0 2024-04-22T15:49:34.679 INFO:tasks.workunit.client.0.smithi016.stdout:1/540: dwrite d3/d4/d19/d20/f60 [0,4194304] 0 2024-04-22T15:49:34.679 INFO:tasks.workunit.client.0.smithi016.stdout:0/389: stat dc/d13/d14/d26/d32/f5b 0 2024-04-22T15:49:34.681 INFO:tasks.workunit.client.0.smithi016.stdout:8/333: chown d2/f8 7519 1 2024-04-22T15:49:34.683 INFO:tasks.workunit.client.0.smithi016.stdout:1/541: stat d3/d4/d37/l49 0 2024-04-22T15:49:34.684 INFO:tasks.workunit.client.0.smithi016.stdout:0/390: write dc/d13/d70/f72 [241955,124638] 0 2024-04-22T15:49:34.685 INFO:tasks.workunit.client.0.smithi016.stdout:8/334: read - d2/d10/d43/d4a/f5f zero size 2024-04-22T15:49:34.686 INFO:tasks.workunit.client.0.smithi016.stdout:9/382: mknod d9/c72 0 2024-04-22T15:49:34.687 INFO:tasks.workunit.client.0.smithi016.stdout:6/416: dwrite d3/da/dd/f70 [0,4194304] 0 2024-04-22T15:49:34.692 INFO:tasks.workunit.client.0.smithi016.stdout:0/391: symlink dc/d13/d22/l77 0 2024-04-22T15:49:34.692 INFO:tasks.workunit.client.0.smithi016.stdout:3/356: dread d7/d17/d18/f67 [0,4194304] 0 2024-04-22T15:49:34.693 INFO:tasks.workunit.client.0.smithi016.stdout:0/392: fdatasync dc/d13/d28/d65/d57/f67 0 2024-04-22T15:49:34.694 INFO:tasks.workunit.client.0.smithi016.stdout:3/357: truncate d7/d17/d18/d32/d3c/f6c 817588 0 2024-04-22T15:49:34.702 INFO:tasks.workunit.client.0.smithi016.stdout:6/417: mknod d3/da/dd/d67/c8a 0 2024-04-22T15:49:34.703 INFO:tasks.workunit.client.0.smithi016.stdout:8/335: symlink d2/d10/d43/d4a/d28/d56/l68 0 2024-04-22T15:49:34.704 INFO:tasks.workunit.client.0.smithi016.stdout:7/349: fsync d3/dc/f1e 0 2024-04-22T15:49:34.705 INFO:tasks.workunit.client.0.smithi016.stdout:7/350: chown d3/dc/d22 394 1 2024-04-22T15:49:34.705 INFO:tasks.workunit.client.0.smithi016.stdout:8/336: fsync d2/d10/d43/d4a/d28/d44/d66/f18 0 2024-04-22T15:49:34.706 INFO:tasks.workunit.client.0.smithi016.stdout:7/351: chown d3/dc/f16 6933308 1 2024-04-22T15:49:34.708 INFO:tasks.workunit.client.0.smithi016.stdout:0/393: mkdir dc/d13/d28/d65/d40/d4a/d59/d78 0 2024-04-22T15:49:34.709 INFO:tasks.workunit.client.0.smithi016.stdout:7/352: read d3/dc/d1b/f6d [280278,84308] 0 2024-04-22T15:49:34.709 INFO:tasks.workunit.client.0.smithi016.stdout:3/358: stat d7/d17/d18/d27/c56 0 2024-04-22T15:49:34.709 INFO:tasks.workunit.client.0.smithi016.stdout:4/349: dwrite d7/d21/d42/f4c [0,4194304] 0 2024-04-22T15:49:34.711 INFO:tasks.workunit.client.0.smithi016.stdout:9/383: creat d9/de/d26/d5a/f73 x:0 0 0 2024-04-22T15:49:34.712 INFO:tasks.workunit.client.0.smithi016.stdout:9/384: dread - d9/d20/f38 zero size 2024-04-22T15:49:34.712 INFO:tasks.workunit.client.0.smithi016.stdout:1/542: truncate d3/d4/f1e 15332820 0 2024-04-22T15:49:34.713 INFO:tasks.workunit.client.0.smithi016.stdout:9/385: readlink d9/de/d23/d30/l67 0 2024-04-22T15:49:34.715 INFO:tasks.workunit.client.0.smithi016.stdout:1/543: read d3/d4/d15/d38/d63/f75 [323736,123559] 0 2024-04-22T15:49:34.720 INFO:tasks.workunit.client.0.smithi016.stdout:6/418: rmdir d3 39 2024-04-22T15:49:34.721 INFO:tasks.workunit.client.0.smithi016.stdout:1/544: dwrite d3/d4/d19/d20/d2b/f4a [0,4194304] 0 2024-04-22T15:49:34.730 INFO:tasks.workunit.client.0.smithi016.stdout:0/394: symlink dc/d13/d70/l79 0 2024-04-22T15:49:34.734 INFO:tasks.workunit.client.0.smithi016.stdout:7/353: symlink d3/dc/d1b/d31/l82 0 2024-04-22T15:49:34.737 INFO:tasks.workunit.client.0.smithi016.stdout:9/386: creat d9/de/d23/d24/f74 x:0 0 0 2024-04-22T15:49:34.738 INFO:tasks.workunit.client.0.smithi016.stdout:9/387: chown d9/de/d23/d24/d4e/d4f/f63 415807 1 2024-04-22T15:49:34.739 INFO:tasks.workunit.client.0.smithi016.stdout:6/419: stat d3/da/db/l1f 0 2024-04-22T15:49:34.739 INFO:tasks.workunit.client.0.smithi016.stdout:6/420: readlink d3/l7 0 2024-04-22T15:49:34.741 INFO:tasks.workunit.client.0.smithi016.stdout:4/350: mknod d7/d21/c75 0 2024-04-22T15:49:34.742 INFO:tasks.workunit.client.0.smithi016.stdout:9/388: dread d9/de/d23/d24/f2f [0,4194304] 0 2024-04-22T15:49:34.750 INFO:tasks.workunit.client.0.smithi016.stdout:4/351: dwrite d7/d24/d41/f57 [0,4194304] 0 2024-04-22T15:49:34.754 INFO:tasks.workunit.client.0.smithi016.stdout:3/359: link d7/d17/d18/d32/c6d d7/d1b/d21/d55/c72 0 2024-04-22T15:49:34.755 INFO:tasks.workunit.client.0.smithi016.stdout:6/421: dread d3/da/db/d14/d4c/f52 [0,4194304] 0 2024-04-22T15:49:34.755 INFO:tasks.workunit.client.0.smithi016.stdout:7/354: symlink d3/dc/d1b/d5a/d59/l83 0 2024-04-22T15:49:34.755 INFO:tasks.workunit.client.0.smithi016.stdout:0/395: write dc/d13/d14/d26/f5e [4747765,68220] 0 2024-04-22T15:49:34.757 INFO:tasks.workunit.client.0.smithi016.stdout:8/337: truncate d2/d10/d43/d4a/d28/d44/d66/d1c/f3d 160710 0 2024-04-22T15:49:34.757 INFO:tasks.workunit.client.0.smithi016.stdout:7/355: dread - d3/dc/d1b/d5a/d9/f50 zero size 2024-04-22T15:49:34.759 INFO:tasks.workunit.client.0.smithi016.stdout:3/360: creat d7/d1b/d21/f73 x:0 0 0 2024-04-22T15:49:34.759 INFO:tasks.workunit.client.0.smithi016.stdout:0/396: truncate dc/d13/f17 9209858 0 2024-04-22T15:49:34.760 INFO:tasks.workunit.client.0.smithi016.stdout:7/356: readlink d3/l7d 0 2024-04-22T15:49:34.761 INFO:tasks.workunit.client.0.smithi016.stdout:4/352: creat d7/d24/f76 x:0 0 0 2024-04-22T15:49:34.761 INFO:tasks.workunit.client.0.smithi016.stdout:3/361: fdatasync d7/d17/d18/d27/f3d 0 2024-04-22T15:49:34.762 INFO:tasks.workunit.client.0.smithi016.stdout:9/389: dread d9/de/d23/d24/d25/f2c [0,4194304] 0 2024-04-22T15:49:34.762 INFO:tasks.workunit.client.0.smithi016.stdout:8/338: symlink d2/d10/l69 0 2024-04-22T15:49:34.763 INFO:tasks.workunit.client.0.smithi016.stdout:7/357: chown d3/dc/d1b/d31/d5f/f77 276548891 1 2024-04-22T15:49:34.763 INFO:tasks.workunit.client.0.smithi016.stdout:9/390: truncate d9/de/d23/f35 1071779 0 2024-04-22T15:49:34.764 INFO:tasks.workunit.client.0.smithi016.stdout:0/397: symlink dc/d13/d28/l7a 0 2024-04-22T15:49:34.765 INFO:tasks.workunit.client.0.smithi016.stdout:0/398: write dc/d13/d28/d65/d57/f67 [7650896,9444] 0 2024-04-22T15:49:34.766 INFO:tasks.workunit.client.0.smithi016.stdout:3/362: write d7/fc [1017112,55402] 0 2024-04-22T15:49:34.768 INFO:tasks.workunit.client.0.smithi016.stdout:0/399: write f4 [4845789,78020] 0 2024-04-22T15:49:34.768 INFO:tasks.workunit.client.0.smithi016.stdout:7/358: mkdir d3/dc/d33/d84 0 2024-04-22T15:49:34.769 INFO:tasks.workunit.client.0.smithi016.stdout:9/391: dread d9/de/f16 [0,4194304] 0 2024-04-22T15:49:34.770 INFO:tasks.workunit.client.0.smithi016.stdout:0/400: write dc/d13/d28/d65/d40/f56 [4851849,55893] 0 2024-04-22T15:49:34.771 INFO:tasks.workunit.client.0.smithi016.stdout:7/359: read d3/dc/d22/f39 [4365866,27265] 0 2024-04-22T15:49:34.771 INFO:tasks.workunit.client.0.smithi016.stdout:9/392: dread - d9/de/d23/d24/d25/d36/f5e zero size 2024-04-22T15:49:34.773 INFO:tasks.workunit.client.0.smithi016.stdout:7/360: write d3/dc/d1b/d5a/d9/fb [3585881,71039] 0 2024-04-22T15:49:34.775 INFO:tasks.workunit.client.0.smithi016.stdout:3/363: creat d7/f74 x:0 0 0 2024-04-22T15:49:34.776 INFO:tasks.workunit.client.0.smithi016.stdout:3/364: fsync d7/f5e 0 2024-04-22T15:49:34.778 INFO:tasks.workunit.client.0.smithi016.stdout:3/365: readlink d7/d1b/d21/l2b 0 2024-04-22T15:49:34.780 INFO:tasks.workunit.client.0.smithi016.stdout:9/393: dwrite d9/d5d/d62/f6e [0,4194304] 0 2024-04-22T15:49:34.782 INFO:tasks.workunit.client.0.smithi016.stdout:3/366: truncate d7/d17/d18/d32/d3c/d51/f5f 1181604 0 2024-04-22T15:49:34.783 INFO:tasks.workunit.client.0.smithi016.stdout:9/394: truncate d9/de/d23/d24/d25/f41 281510 0 2024-04-22T15:49:34.786 INFO:tasks.workunit.client.0.smithi016.stdout:0/401: rename dc/d13/d28/d65/d1f/f4c to dc/d13/d14/d26/f7b 0 2024-04-22T15:49:34.786 INFO:tasks.workunit.client.0.smithi016.stdout:2/304: sync 2024-04-22T15:49:34.787 INFO:tasks.workunit.client.0.smithi016.stdout:3/367: dread d7/fc [0,4194304] 0 2024-04-22T15:49:34.788 INFO:tasks.workunit.client.0.smithi016.stdout:7/361: dread d3/dc/d22/f39 [0,4194304] 0 2024-04-22T15:49:34.790 INFO:tasks.workunit.client.0.smithi016.stdout:8/339: mkdir d2/d10/d43/d4a/d6a 0 2024-04-22T15:49:34.793 INFO:tasks.workunit.client.0.smithi016.stdout:9/395: creat d9/de/d23/d30/f75 x:0 0 0 2024-04-22T15:49:34.793 INFO:tasks.workunit.client.0.smithi016.stdout:4/353: dwrite d7/f69 [4194304,4194304] 0 2024-04-22T15:49:34.795 INFO:tasks.workunit.client.0.smithi016.stdout:7/362: write d3/dc/d1b/d31/d5f/f7e [254410,5141] 0 2024-04-22T15:49:34.796 INFO:tasks.workunit.client.0.smithi016.stdout:0/402: unlink dc/d13/d5d/f5f 0 2024-04-22T15:49:34.798 INFO:tasks.workunit.client.0.smithi016.stdout:8/340: readlink d2/d10/d43/d4a/d28/d44/d66/l4e 0 2024-04-22T15:49:34.799 INFO:tasks.workunit.client.0.smithi016.stdout:4/354: chown d7/d4e/d39/f48 0 1 2024-04-22T15:49:34.800 INFO:tasks.workunit.client.0.smithi016.stdout:8/341: chown d2/d32/f35 5343 1 2024-04-22T15:49:34.800 INFO:tasks.workunit.client.0.smithi016.stdout:2/305: dwrite d2/d2c/f5b [0,4194304] 0 2024-04-22T15:49:34.802 INFO:tasks.workunit.client.0.smithi016.stdout:9/396: rmdir d9/de/d26 39 2024-04-22T15:49:34.805 INFO:tasks.workunit.client.0.smithi016.stdout:3/368: mkdir d7/d75 0 2024-04-22T15:49:34.811 INFO:tasks.workunit.client.0.smithi016.stdout:2/306: unlink d2/d9/lb 0 2024-04-22T15:49:34.811 INFO:tasks.workunit.client.0.smithi016.stdout:3/369: dwrite d7/d17/d18/f2a [0,4194304] 0 2024-04-22T15:49:34.812 INFO:tasks.workunit.client.0.smithi016.stdout:2/307: dread - d2/d4d/d64/f6e zero size 2024-04-22T15:49:34.813 INFO:tasks.workunit.client.0.smithi016.stdout:2/308: write d2/d51/f5a [50279,12907] 0 2024-04-22T15:49:34.814 INFO:tasks.workunit.client.0.smithi016.stdout:9/397: creat d9/de/d23/d24/f76 x:0 0 0 2024-04-22T15:49:34.815 INFO:tasks.workunit.client.0.smithi016.stdout:2/309: chown d2/d4d 986 1 2024-04-22T15:49:34.816 INFO:tasks.workunit.client.0.smithi016.stdout:2/310: readlink d2/d1a/l58 0 2024-04-22T15:49:34.826 INFO:tasks.workunit.client.0.smithi016.stdout:9/398: rename d9/d37/c68 to d9/de/d23/d24/d4e/d4f/c77 0 2024-04-22T15:49:34.827 INFO:tasks.workunit.client.0.smithi016.stdout:4/355: link d7/d4e/d39/l72 d7/l77 0 2024-04-22T15:49:34.827 INFO:tasks.workunit.client.0.smithi016.stdout:2/311: dwrite d2/d9/f16 [0,4194304] 0 2024-04-22T15:49:34.829 INFO:tasks.workunit.client.0.smithi016.stdout:2/312: write d2/d23/d27/f6a [309033,68215] 0 2024-04-22T15:49:34.829 INFO:tasks.workunit.client.0.smithi016.stdout:3/370: creat d7/d75/f76 x:0 0 0 2024-04-22T15:49:34.830 INFO:tasks.workunit.client.0.smithi016.stdout:8/342: dread d2/d10/d43/d4a/d28/d44/d66/d1c/f3d [0,4194304] 0 2024-04-22T15:49:34.830 INFO:tasks.workunit.client.0.smithi016.stdout:3/371: fsync d7/d1b/d21/f73 0 2024-04-22T15:49:34.837 INFO:tasks.workunit.client.0.smithi016.stdout:2/313: stat d2/d1a/l56 0 2024-04-22T15:49:34.837 INFO:tasks.workunit.client.0.smithi016.stdout:3/372: dwrite d7/d17/d18/d32/d3c/d51/f5f [0,4194304] 0 2024-04-22T15:49:34.840 INFO:tasks.workunit.client.0.smithi016.stdout:0/403: link dc/d13/d28/d65/d40/c43 dc/d13/c7c 0 2024-04-22T15:49:34.841 INFO:tasks.workunit.client.0.smithi016.stdout:4/356: symlink d7/d21/d45/d49/d67/l78 0 2024-04-22T15:49:34.845 INFO:tasks.workunit.client.0.smithi016.stdout:9/399: creat d9/de/d23/f78 x:0 0 0 2024-04-22T15:49:34.846 INFO:tasks.workunit.client.0.smithi016.stdout:3/373: mkdir d7/d77 0 2024-04-22T15:49:34.847 INFO:tasks.workunit.client.0.smithi016.stdout:3/374: stat d7/d1b/d21 0 2024-04-22T15:49:34.848 INFO:tasks.workunit.client.0.smithi016.stdout:4/357: mknod d7/d24/c79 0 2024-04-22T15:49:34.849 INFO:tasks.workunit.client.0.smithi016.stdout:8/343: dread d2/d10/d43/d4a/d28/d3e/f42 [8388608,4194304] 0 2024-04-22T15:49:34.849 INFO:tasks.workunit.client.0.smithi016.stdout:9/400: creat d9/de/d23/d24/d4e/f79 x:0 0 0 2024-04-22T15:49:34.851 INFO:tasks.workunit.client.0.smithi016.stdout:3/375: write d7/d17/d18/d27/f6e [2746001,127730] 0 2024-04-22T15:49:34.852 INFO:tasks.workunit.client.0.smithi016.stdout:2/314: getdents d2/d23/d27 0 2024-04-22T15:49:34.852 INFO:tasks.workunit.client.0.smithi016.stdout:4/358: creat d7/d4e/d2f/d33/f7a x:0 0 0 2024-04-22T15:49:34.853 INFO:tasks.workunit.client.0.smithi016.stdout:9/401: readlink d9/l17 0 2024-04-22T15:49:34.854 INFO:tasks.workunit.client.0.smithi016.stdout:3/376: write d7/d17/d18/d32/f37 [1809382,73526] 0 2024-04-22T15:49:34.854 INFO:tasks.workunit.client.0.smithi016.stdout:9/402: dread - d9/f1c zero size 2024-04-22T15:49:34.858 INFO:tasks.workunit.client.0.smithi016.stdout:2/315: creat d2/d23/d47/f71 x:0 0 0 2024-04-22T15:49:34.860 INFO:tasks.workunit.client.0.smithi016.stdout:4/359: mknod d7/d21/d45/d49/d67/c7b 0 2024-04-22T15:49:34.860 INFO:tasks.workunit.client.0.smithi016.stdout:3/377: rmdir d7/d17/d18/d32/d3c/d51 39 2024-04-22T15:49:34.862 INFO:tasks.workunit.client.0.smithi016.stdout:8/344: symlink d2/d10/d43/d4a/d6a/l6b 0 2024-04-22T15:49:34.862 INFO:tasks.workunit.client.0.smithi016.stdout:3/378: dread d7/d17/d18/d32/d3c/d47/f48 [0,4194304] 0 2024-04-22T15:49:34.864 INFO:tasks.workunit.client.0.smithi016.stdout:2/316: symlink d2/d4d/d6d/l72 0 2024-04-22T15:49:34.866 INFO:tasks.workunit.client.0.smithi016.stdout:8/345: mknod d2/d48/c6c 0 2024-04-22T15:49:34.868 INFO:tasks.workunit.client.0.smithi016.stdout:3/379: symlink d7/d1b/l78 0 2024-04-22T15:49:34.872 INFO:tasks.workunit.client.0.smithi016.stdout:8/346: mknod d2/d30/c6d 0 2024-04-22T15:49:34.876 INFO:tasks.workunit.client.0.smithi016.stdout:4/360: getdents d7/d21 0 2024-04-22T15:49:34.878 INFO:tasks.workunit.client.0.smithi016.stdout:8/347: dwrite d2/d10/d43/d4a/d28/f4d [0,4194304] 0 2024-04-22T15:49:34.878 INFO:tasks.workunit.client.0.smithi016.stdout:4/361: fdatasync d7/d24/d51/f6a 0 2024-04-22T15:49:34.884 INFO:tasks.workunit.client.0.smithi016.stdout:4/362: dwrite d7/f69 [0,4194304] 0 2024-04-22T15:49:34.886 INFO:tasks.workunit.client.0.smithi016.stdout:3/380: dread d7/f5a [0,4194304] 0 2024-04-22T15:49:34.888 INFO:tasks.workunit.client.0.smithi016.stdout:8/348: unlink d2/d30/c6d 0 2024-04-22T15:49:34.892 INFO:tasks.workunit.client.0.smithi016.stdout:2/317: getdents d2/d4d/d64 0 2024-04-22T15:49:34.892 INFO:tasks.workunit.client.0.smithi016.stdout:4/363: dwrite d7/d21/d45/f6d [0,4194304] 0 2024-04-22T15:49:34.893 INFO:tasks.workunit.client.0.smithi016.stdout:4/364: fdatasync d7/f8 0 2024-04-22T15:49:34.894 INFO:tasks.workunit.client.0.smithi016.stdout:5/346: sync 2024-04-22T15:49:34.897 INFO:tasks.workunit.client.0.smithi016.stdout:3/381: rename d7/d17/d18/d32/f37 to d7/d1b/d21/d55/f79 0 2024-04-22T15:49:34.901 INFO:tasks.workunit.client.0.smithi016.stdout:3/382: dread d7/d17/f2e [0,4194304] 0 2024-04-22T15:49:34.903 INFO:tasks.workunit.client.0.smithi016.stdout:8/349: symlink d2/l6e 0 2024-04-22T15:49:34.904 INFO:tasks.workunit.client.0.smithi016.stdout:4/365: fsync f5 0 2024-04-22T15:49:34.904 INFO:tasks.workunit.client.0.smithi016.stdout:8/350: truncate d2/d30/f60 676068 0 2024-04-22T15:49:34.905 INFO:tasks.workunit.client.0.smithi016.stdout:5/347: symlink df/d5b/l64 0 2024-04-22T15:49:34.906 INFO:tasks.workunit.client.0.smithi016.stdout:8/351: dread d2/d30/f60 [0,4194304] 0 2024-04-22T15:49:34.909 INFO:tasks.workunit.client.0.smithi016.stdout:4/366: mkdir d7/d21/d7c 0 2024-04-22T15:49:34.911 INFO:tasks.workunit.client.0.smithi016.stdout:5/348: unlink df/d22/f2a 0 2024-04-22T15:49:34.913 INFO:tasks.workunit.client.0.smithi016.stdout:3/383: creat d7/d17/d18/d32/d42/d68/f7a x:0 0 0 2024-04-22T15:49:34.913 INFO:tasks.workunit.client.0.smithi016.stdout:8/352: mkdir d2/d10/d43/d4a/d28/d44/d6f 0 2024-04-22T15:49:34.915 INFO:tasks.workunit.client.0.smithi016.stdout:8/353: write d2/d10/d43/d4a/d28/f38 [2271990,47343] 0 2024-04-22T15:49:34.916 INFO:tasks.workunit.client.0.smithi016.stdout:5/349: creat df/d15/d1a/d4d/f65 x:0 0 0 2024-04-22T15:49:34.917 INFO:tasks.workunit.client.0.smithi016.stdout:8/354: truncate d2/d10/d43/d4a/d28/f4d 5225233 0 2024-04-22T15:49:34.917 INFO:tasks.workunit.client.0.smithi016.stdout:5/350: chown df/d22/f25 0 1 2024-04-22T15:49:34.918 INFO:tasks.workunit.client.0.smithi016.stdout:5/351: write df/d15/d1a/d28/f2c [1091444,117910] 0 2024-04-22T15:49:34.921 INFO:tasks.workunit.client.0.smithi016.stdout:4/367: truncate d7/f1e 6399622 0 2024-04-22T15:49:34.923 INFO:tasks.workunit.client.0.smithi016.stdout:8/355: fdatasync d2/d10/d43/d4a/d28/d3e/f42 0 2024-04-22T15:49:34.926 INFO:tasks.workunit.client.0.smithi016.stdout:3/384: rmdir d7/d1b 39 2024-04-22T15:49:34.931 INFO:tasks.workunit.client.0.smithi016.stdout:8/356: symlink d2/d10/d43/d4a/d28/d44/l70 0 2024-04-22T15:49:34.935 INFO:tasks.workunit.client.0.smithi016.stdout:3/385: write d7/d1b/d21/f73 [405051,64754] 0 2024-04-22T15:49:34.940 INFO:tasks.workunit.client.0.smithi016.stdout:4/368: rename d7/d21/d42 to d7/d21/d42/d59/d7d 22 2024-04-22T15:49:34.940 INFO:tasks.workunit.client.0.smithi016.stdout:5/352: dread df/d15/d1a/f53 [0,4194304] 0 2024-04-22T15:49:34.944 INFO:tasks.workunit.client.0.smithi016.stdout:3/386: dwrite d7/f5a [4194304,4194304] 0 2024-04-22T15:49:34.955 INFO:tasks.workunit.client.0.smithi016.stdout:4/369: link d7/d3c/c43 d7/d4e/d2f/d33/c7e 0 2024-04-22T15:49:34.957 INFO:tasks.workunit.client.0.smithi016.stdout:3/387: dwrite d7/d17/d18/f67 [0,4194304] 0 2024-04-22T15:49:34.960 INFO:tasks.workunit.client.0.smithi016.stdout:6/422: sync 2024-04-22T15:49:34.960 INFO:tasks.workunit.client.0.smithi016.stdout:1/545: sync 2024-04-22T15:49:34.963 INFO:tasks.workunit.client.0.smithi016.stdout:4/370: dread d7/d24/d41/f57 [0,4194304] 0 2024-04-22T15:49:34.963 INFO:tasks.workunit.client.0.smithi016.stdout:8/357: write d2/f24 [2830077,55112] 0 2024-04-22T15:49:34.965 INFO:tasks.workunit.client.0.smithi016.stdout:1/546: write d3/d4/da/fa2 [260072,5668] 0 2024-04-22T15:49:34.965 INFO:tasks.workunit.client.0.smithi016.stdout:8/358: stat d2/l6e 0 2024-04-22T15:49:34.968 INFO:tasks.workunit.client.0.smithi016.stdout:1/547: chown d3/d4/d19/f35 55 1 2024-04-22T15:49:34.972 INFO:tasks.workunit.client.0.smithi016.stdout:6/423: creat d3/da/dd/d1b/f8b x:0 0 0 2024-04-22T15:49:34.974 INFO:tasks.workunit.client.0.smithi016.stdout:3/388: rename d7/d17/d18/d32/d3c/d51/l60 to d7/d75/l7b 0 2024-04-22T15:49:34.975 INFO:tasks.workunit.client.0.smithi016.stdout:1/548: unlink d3/d4/d3a/c4d 0 2024-04-22T15:49:34.978 INFO:tasks.workunit.client.0.smithi016.stdout:6/424: symlink d3/da/db/d14/d4c/d58/l8c 0 2024-04-22T15:49:34.979 INFO:tasks.workunit.client.0.smithi016.stdout:3/389: creat d7/d1b/f7c x:0 0 0 2024-04-22T15:49:34.981 INFO:tasks.workunit.client.0.smithi016.stdout:1/549: creat d3/d84/d97/fb6 x:0 0 0 2024-04-22T15:49:34.982 INFO:tasks.workunit.client.0.smithi016.stdout:3/390: write d7/fc [4660941,95362] 0 2024-04-22T15:49:34.984 INFO:tasks.workunit.client.0.smithi016.stdout:6/425: symlink d3/da/db/d14/d4c/d58/l8d 0 2024-04-22T15:49:34.985 INFO:tasks.workunit.client.0.smithi016.stdout:5/353: dwrite df/d15/d1a/d28/d33/f39 [0,4194304] 0 2024-04-22T15:49:34.986 INFO:tasks.workunit.client.0.smithi016.stdout:8/359: getdents d2/d10/d43/d4a/d28/d56 0 2024-04-22T15:49:34.990 INFO:tasks.workunit.client.0.smithi016.stdout:3/391: write d7/f69 [191820,3734] 0 2024-04-22T15:49:34.992 INFO:tasks.workunit.client.0.smithi016.stdout:8/360: write d2/d10/d43/d4a/d28/d56/f64 [676627,96802] 0 2024-04-22T15:49:34.992 INFO:tasks.workunit.client.0.smithi016.stdout:4/371: dread d7/f1e [4194304,4194304] 0 2024-04-22T15:49:34.993 INFO:tasks.workunit.client.0.smithi016.stdout:1/550: rename d3/d4/d19/f54 to d3/d4/d3a/d91/fb7 0 2024-04-22T15:49:34.993 INFO:tasks.workunit.client.0.smithi016.stdout:7/363: sync 2024-04-22T15:49:34.998 INFO:tasks.workunit.client.0.smithi016.stdout:7/364: creat d3/d14/f85 x:0 0 0 2024-04-22T15:49:35.002 INFO:tasks.workunit.client.0.smithi016.stdout:5/354: dwrite df/d22/f25 [0,4194304] 0 2024-04-22T15:49:35.004 INFO:tasks.workunit.client.0.smithi016.stdout:8/361: creat d2/d10/d43/d4a/d28/d56/f71 x:0 0 0 2024-04-22T15:49:35.004 INFO:tasks.workunit.client.0.smithi016.stdout:3/392: dwrite d7/f69 [0,4194304] 0 2024-04-22T15:49:35.004 INFO:tasks.workunit.client.0.smithi016.stdout:4/372: mknod d7/d21/d45/c7f 0 2024-04-22T15:49:35.006 INFO:tasks.workunit.client.0.smithi016.stdout:7/365: rmdir d3/dc/d1b/d5a/d59 39 2024-04-22T15:49:35.007 INFO:tasks.workunit.client.0.smithi016.stdout:6/426: truncate d3/f1d 3382628 0 2024-04-22T15:49:35.009 INFO:tasks.workunit.client.0.smithi016.stdout:3/393: write d7/d75/f76 [975790,114165] 0 2024-04-22T15:49:35.012 INFO:tasks.workunit.client.0.smithi016.stdout:4/373: mkdir d7/d21/d42/d80 0 2024-04-22T15:49:35.013 INFO:tasks.workunit.client.0.smithi016.stdout:8/362: mknod d2/d10/d47/c72 0 2024-04-22T15:49:35.013 INFO:tasks.workunit.client.0.smithi016.stdout:1/551: dwrite d3/d4/d19/f35 [0,4194304] 0 2024-04-22T15:49:35.016 INFO:tasks.workunit.client.0.smithi016.stdout:7/366: creat d3/dc/d1b/d5a/d9/df/f86 x:0 0 0 2024-04-22T15:49:35.016 INFO:tasks.workunit.client.0.smithi016.stdout:4/374: fdatasync d7/d21/d45/d49/f4d 0 2024-04-22T15:49:35.018 INFO:tasks.workunit.client.0.smithi016.stdout:8/363: readlink d2/d10/d43/d4a/d6a/l6b 0 2024-04-22T15:49:35.023 INFO:tasks.workunit.client.0.smithi016.stdout:4/375: mkdir d7/d4e/d81 0 2024-04-22T15:49:35.025 INFO:tasks.workunit.client.0.smithi016.stdout:5/355: dwrite df/d15/d1a/d28/f2c [0,4194304] 0 2024-04-22T15:49:35.025 INFO:tasks.workunit.client.0.smithi016.stdout:8/364: read d2/d10/d43/d4a/d28/f58 [161773,69662] 0 2024-04-22T15:49:35.027 INFO:tasks.workunit.client.0.smithi016.stdout:3/394: dread d7/d17/d18/f67 [0,4194304] 0 2024-04-22T15:49:35.030 INFO:tasks.workunit.client.0.smithi016.stdout:5/356: chown df/d15 3 1 2024-04-22T15:49:35.032 INFO:tasks.workunit.client.0.smithi016.stdout:6/427: link d3/c4a d3/d53/c8e 0 2024-04-22T15:49:35.033 INFO:tasks.workunit.client.0.smithi016.stdout:8/365: stat d2/d10/d43/d4a/d28/f38 0 2024-04-22T15:49:35.034 INFO:tasks.workunit.client.0.smithi016.stdout:4/376: rmdir d7/d21/d45/d49 39 2024-04-22T15:49:35.035 INFO:tasks.workunit.client.0.smithi016.stdout:7/367: dwrite d3/dc/d22/f6b [0,4194304] 0 2024-04-22T15:49:35.036 INFO:tasks.workunit.client.0.smithi016.stdout:5/357: write df/d15/d1a/d28/d33/f3c [359115,83845] 0 2024-04-22T15:49:35.037 INFO:tasks.workunit.client.0.smithi016.stdout:1/552: rename d3/d4/l5 to d3/d4/d19/lb8 0 2024-04-22T15:49:35.037 INFO:tasks.workunit.client.0.smithi016.stdout:7/368: fdatasync d3/dc/d33/f4d 0 2024-04-22T15:49:35.041 INFO:tasks.workunit.client.0.smithi016.stdout:3/395: creat d7/d1b/d21/d55/f7d x:0 0 0 2024-04-22T15:49:35.043 INFO:tasks.workunit.client.0.smithi016.stdout:4/377: fsync d7/d24/d41/f5d 0 2024-04-22T15:49:35.043 INFO:tasks.workunit.client.0.smithi016.stdout:3/396: write d7/d17/d18/d27/f54 [957906,123605] 0 2024-04-22T15:49:35.045 INFO:tasks.workunit.client.0.smithi016.stdout:3/397: read d7/f5a [4773943,42536] 0 2024-04-22T15:49:35.047 INFO:tasks.workunit.client.0.smithi016.stdout:8/366: dwrite d2/d10/d43/d4a/d28/d44/d66/f18 [0,4194304] 0 2024-04-22T15:49:35.047 INFO:tasks.workunit.client.0.smithi016.stdout:3/398: write d7/f14 [3779951,16744] 0 2024-04-22T15:49:35.050 INFO:tasks.workunit.client.0.smithi016.stdout:1/553: rename d3/d4/f24 to d3/d4/d37/fb9 0 2024-04-22T15:49:35.053 INFO:tasks.workunit.client.0.smithi016.stdout:4/378: unlink d7/l1f 0 2024-04-22T15:49:35.055 INFO:tasks.workunit.client.0.smithi016.stdout:0/404: sync 2024-04-22T15:49:35.055 INFO:tasks.workunit.client.0.smithi016.stdout:2/318: sync 2024-04-22T15:49:35.055 INFO:tasks.workunit.client.0.smithi016.stdout:9/403: sync 2024-04-22T15:49:35.055 INFO:tasks.workunit.client.0.smithi016.stdout:9/404: fsync d9/d20/f38 0 2024-04-22T15:49:35.062 INFO:tasks.workunit.client.0.smithi016.stdout:9/405: dwrite d9/de/d23/f78 [0,4194304] 0 2024-04-22T15:49:35.065 INFO:tasks.workunit.client.0.smithi016.stdout:8/367: creat d2/d30/f73 x:0 0 0 2024-04-22T15:49:35.065 INFO:tasks.workunit.client.0.smithi016.stdout:3/399: chown d7/ld 12526 1 2024-04-22T15:49:35.068 INFO:tasks.workunit.client.0.smithi016.stdout:6/428: dwrite d3/f6 [0,4194304] 0 2024-04-22T15:49:35.068 INFO:tasks.workunit.client.0.smithi016.stdout:8/368: read - d2/d10/d43/d4a/d28/d56/f71 zero size 2024-04-22T15:49:35.071 INFO:tasks.workunit.client.0.smithi016.stdout:8/369: write d2/d30/f73 [423891,109462] 0 2024-04-22T15:49:35.072 INFO:tasks.workunit.client.0.smithi016.stdout:7/369: dwrite d3/d14/f7c [0,4194304] 0 2024-04-22T15:49:35.074 INFO:tasks.workunit.client.0.smithi016.stdout:2/319: rename d2/cc to d2/d4d/d6d/c73 0 2024-04-22T15:49:35.077 INFO:tasks.workunit.client.0.smithi016.stdout:5/358: symlink df/d15/d1a/d4e/d58/l66 0 2024-04-22T15:49:35.080 INFO:tasks.workunit.client.0.smithi016.stdout:1/554: symlink d3/d4/lba 0 2024-04-22T15:49:35.081 INFO:tasks.workunit.client.0.smithi016.stdout:1/555: chown d3/d4/d45 1475301 1 2024-04-22T15:49:35.081 INFO:tasks.workunit.client.0.smithi016.stdout:8/370: dwrite d2/d10/d43/d4a/d28/d44/d66/fc [0,4194304] 0 2024-04-22T15:49:35.082 INFO:tasks.workunit.client.0.smithi016.stdout:4/379: symlink d7/d21/d45/l82 0 2024-04-22T15:49:35.083 INFO:tasks.workunit.client.0.smithi016.stdout:3/400: write d7/d1b/d21/d55/f79 [461054,2357] 0 2024-04-22T15:49:35.084 INFO:tasks.workunit.client.0.smithi016.stdout:0/405: mkdir dc/d7d 0 2024-04-22T15:49:35.085 INFO:tasks.workunit.client.0.smithi016.stdout:0/406: fdatasync dc/d13/f52 0 2024-04-22T15:49:35.086 INFO:tasks.workunit.client.0.smithi016.stdout:6/429: creat d3/da/dd/d67/f8f x:0 0 0 2024-04-22T15:49:35.089 INFO:tasks.workunit.client.0.smithi016.stdout:7/370: symlink d3/dc/d1b/d5a/d4a/d6e/l87 0 2024-04-22T15:49:35.089 INFO:tasks.workunit.client.0.smithi016.stdout:2/320: symlink d2/d4d/d6d/l74 0 2024-04-22T15:49:35.091 INFO:tasks.workunit.client.0.smithi016.stdout:9/406: mknod d9/de/c7a 0 2024-04-22T15:49:35.092 INFO:tasks.workunit.client.0.smithi016.stdout:5/359: creat df/d15/d1a/d28/d33/f67 x:0 0 0 2024-04-22T15:49:35.092 INFO:tasks.workunit.client.0.smithi016.stdout:3/401: dread d7/d17/d18/d32/d3c/f6c [0,4194304] 0 2024-04-22T15:49:35.094 INFO:tasks.workunit.client.0.smithi016.stdout:7/371: write d3/d32/f72 [102787,125958] 0 2024-04-22T15:49:35.097 INFO:tasks.workunit.client.0.smithi016.stdout:3/402: stat d7/d17/f2e 0 2024-04-22T15:49:35.097 INFO:tasks.workunit.client.0.smithi016.stdout:4/380: dwrite d7/f19 [0,4194304] 0 2024-04-22T15:49:35.098 INFO:tasks.workunit.client.0.smithi016.stdout:8/371: rename d2/d10/d43/d4a/f59 to d2/d10/d43/d4a/d6a/f74 0 2024-04-22T15:49:35.100 INFO:tasks.workunit.client.0.smithi016.stdout:1/556: dwrite d3/d4/d19/f1d [4194304,4194304] 0 2024-04-22T15:49:35.100 INFO:tasks.workunit.client.0.smithi016.stdout:5/360: write df/d15/d1a/d28/d33/f62 [831847,20623] 0 2024-04-22T15:49:35.102 INFO:tasks.workunit.client.0.smithi016.stdout:2/321: creat d2/d23/d27/d67/f75 x:0 0 0 2024-04-22T15:49:35.105 INFO:tasks.workunit.client.0.smithi016.stdout:8/372: chown d2/d10/d43/d4a/d28/d3e/c26 1012627564 1 2024-04-22T15:49:35.110 INFO:tasks.workunit.client.0.smithi016.stdout:0/407: creat dc/f7e x:0 0 0 2024-04-22T15:49:35.112 INFO:tasks.workunit.client.0.smithi016.stdout:6/430: chown d3/d53/d35/c7f 98137 1 2024-04-22T15:49:35.112 INFO:tasks.workunit.client.0.smithi016.stdout:7/372: dwrite d3/f74 [0,4194304] 0 2024-04-22T15:49:35.113 INFO:tasks.workunit.client.0.smithi016.stdout:9/407: dwrite d9/d37/f71 [0,4194304] 0 2024-04-22T15:49:35.116 INFO:tasks.workunit.client.0.smithi016.stdout:0/408: readlink dc/d13/d28/d65/d30/l3e 0 2024-04-22T15:49:35.119 INFO:tasks.workunit.client.0.smithi016.stdout:1/557: dread d3/d4/d19/d20/f60 [0,4194304] 0 2024-04-22T15:49:35.122 INFO:tasks.workunit.client.0.smithi016.stdout:3/403: rename d7/d17/d18/d27/d5c to d7/d17/d18/d32/d42/d7e 0 2024-04-22T15:49:35.123 INFO:tasks.workunit.client.0.smithi016.stdout:8/373: rmdir d2/d10/d43/d4a/d28/d52 39 2024-04-22T15:49:35.127 INFO:tasks.workunit.client.0.smithi016.stdout:4/381: mknod d7/d24/c83 0 2024-04-22T15:49:35.129 INFO:tasks.workunit.client.0.smithi016.stdout:7/373: mkdir d3/dc/d1b/d31/d88 0 2024-04-22T15:49:35.130 INFO:tasks.workunit.client.0.smithi016.stdout:0/409: chown dc/d13/l2a 499388000 1 2024-04-22T15:49:35.133 INFO:tasks.workunit.client.0.smithi016.stdout:4/382: dread d7/f8 [0,4194304] 0 2024-04-22T15:49:35.133 INFO:tasks.workunit.client.0.smithi016.stdout:0/410: write dc/d13/d14/d26/f4f [817337,38749] 0 2024-04-22T15:49:35.134 INFO:tasks.workunit.client.0.smithi016.stdout:6/431: creat d3/da/db/d14/d4c/f90 x:0 0 0 2024-04-22T15:49:35.138 INFO:tasks.workunit.client.0.smithi016.stdout:1/558: chown d3/d4/d45/cad 737 1 2024-04-22T15:49:35.140 INFO:tasks.workunit.client.0.smithi016.stdout:3/404: dread d7/d17/d18/d32/d3c/d47/f48 [0,4194304] 0 2024-04-22T15:49:35.141 INFO:tasks.workunit.client.0.smithi016.stdout:2/322: write d2/f8 [8777819,71751] 0 2024-04-22T15:49:35.144 INFO:tasks.workunit.client.0.smithi016.stdout:1/559: dwrite d3/d4/d3a/d91/fb5 [0,4194304] 0 2024-04-22T15:49:35.148 INFO:tasks.workunit.client.0.smithi016.stdout:3/405: write d7/d17/d18/f67 [4734476,90628] 0 2024-04-22T15:49:35.150 INFO:tasks.workunit.client.0.smithi016.stdout:3/406: write d7/f5e [514140,91730] 0 2024-04-22T15:49:35.150 INFO:tasks.workunit.client.0.smithi016.stdout:2/323: mkdir d2/d4d/d76 0 2024-04-22T15:49:35.152 INFO:tasks.workunit.client.0.smithi016.stdout:8/374: dread d2/f27 [4194304,4194304] 0 2024-04-22T15:49:35.153 INFO:tasks.workunit.client.0.smithi016.stdout:0/411: dwrite dc/d13/d28/d65/d30/f44 [0,4194304] 0 2024-04-22T15:49:35.154 INFO:tasks.workunit.client.0.smithi016.stdout:2/324: write d2/de/f5c [486997,19102] 0 2024-04-22T15:49:35.154 INFO:tasks.workunit.client.0.smithi016.stdout:3/407: write d7/d17/d18/d27/f6e [894445,71195] 0 2024-04-22T15:49:35.161 INFO:tasks.workunit.client.0.smithi016.stdout:9/408: rename d9/d57/l66 to d9/de/d23/l7b 0 2024-04-22T15:49:35.161 INFO:tasks.workunit.client.0.smithi016.stdout:2/325: chown d2/de/c14 3086 1 2024-04-22T15:49:35.161 INFO:tasks.workunit.client.0.smithi016.stdout:7/374: link d3/dc/d1b/d5a/d9/df/f86 d3/dc/d1b/d5a/d59/d64/f89 0 2024-04-22T15:49:35.169 INFO:tasks.workunit.client.0.smithi016.stdout:0/412: readlink dc/d13/d28/d65/d40/l6d 0 2024-04-22T15:49:35.170 INFO:tasks.workunit.client.0.smithi016.stdout:8/375: dwrite d2/d10/d43/d4a/d28/d56/f64 [0,4194304] 0 2024-04-22T15:49:35.171 INFO:tasks.workunit.client.0.smithi016.stdout:6/432: creat d3/da/dd/f91 x:0 0 0 2024-04-22T15:49:35.172 INFO:tasks.workunit.client.0.smithi016.stdout:3/408: dread d7/d1b/d21/d55/f79 [0,4194304] 0 2024-04-22T15:49:35.173 INFO:tasks.workunit.client.0.smithi016.stdout:6/433: dread - d3/da/f69 zero size 2024-04-22T15:49:35.173 INFO:tasks.workunit.client.0.smithi016.stdout:2/326: dwrite d2/f4f [0,4194304] 0 2024-04-22T15:49:35.174 INFO:tasks.workunit.client.0.smithi016.stdout:4/383: rename d7/f11 to d7/d4e/d2f/d6f/f84 0 2024-04-22T15:49:35.180 INFO:tasks.workunit.client.0.smithi016.stdout:7/375: mknod d3/dc/d1b/d5a/d9/df/c8a 0 2024-04-22T15:49:35.180 INFO:tasks.workunit.client.0.smithi016.stdout:6/434: write d3/da/db/d14/d16/f3d [2097908,88183] 0 2024-04-22T15:49:35.183 INFO:tasks.workunit.client.0.smithi016.stdout:0/413: unlink dc/d13/d14/d26/d32/f5b 0 2024-04-22T15:49:35.186 INFO:tasks.workunit.client.0.smithi016.stdout:4/384: dwrite d7/d24/d41/f74 [0,4194304] 0 2024-04-22T15:49:35.186 INFO:tasks.workunit.client.0.smithi016.stdout:8/376: symlink d2/d10/d43/d4a/d6a/l75 0 2024-04-22T15:49:35.187 INFO:tasks.workunit.client.0.smithi016.stdout:0/414: write dc/d13/d28/d65/d57/f67 [1183634,81937] 0 2024-04-22T15:49:35.189 INFO:tasks.workunit.client.0.smithi016.stdout:2/327: mkdir d2/d1a/d4c/d77 0 2024-04-22T15:49:35.194 INFO:tasks.workunit.client.0.smithi016.stdout:4/385: write d7/d22/f54 [763300,123141] 0 2024-04-22T15:49:35.194 INFO:tasks.workunit.client.0.smithi016.stdout:1/560: dread d3/f7 [0,4194304] 0 2024-04-22T15:49:35.197 INFO:tasks.workunit.client.0.smithi016.stdout:4/386: fdatasync d7/d21/d45/f6d 0 2024-04-22T15:49:35.197 INFO:tasks.workunit.client.0.smithi016.stdout:7/376: dwrite d3/dc/d1b/d5a/f1f [0,4194304] 0 2024-04-22T15:49:35.202 INFO:tasks.workunit.client.0.smithi016.stdout:1/561: write d3/d4/d19/d20/f60 [2633956,31976] 0 2024-04-22T15:49:35.203 INFO:tasks.workunit.client.0.smithi016.stdout:4/387: mknod d7/d3c/c85 0 2024-04-22T15:49:35.207 INFO:tasks.workunit.client.0.smithi016.stdout:6/435: dread d3/da/d12/f15 [0,4194304] 0 2024-04-22T15:49:35.208 INFO:tasks.workunit.client.0.smithi016.stdout:8/377: dwrite d2/d10/d43/d4a/d28/d44/d66/f3c [0,4194304] 0 2024-04-22T15:49:35.210 INFO:tasks.workunit.client.0.smithi016.stdout:8/378: fsync d2/d10/d43/d4a/d28/d44/d66/fc 0 2024-04-22T15:49:35.214 INFO:tasks.workunit.client.0.smithi016.stdout:9/409: truncate d9/d37/f4a 3364513 0 2024-04-22T15:49:35.216 INFO:tasks.workunit.client.0.smithi016.stdout:7/377: creat d3/f8b x:0 0 0 2024-04-22T15:49:35.218 INFO:tasks.workunit.client.0.smithi016.stdout:7/378: chown d3/f23 77591 1 2024-04-22T15:49:35.218 INFO:tasks.workunit.client.0.smithi016.stdout:8/379: dread d2/d10/d43/d4a/d28/f4d [0,4194304] 0 2024-04-22T15:49:35.219 INFO:tasks.workunit.client.0.smithi016.stdout:3/409: dwrite d7/d17/f2e [0,4194304] 0 2024-04-22T15:49:35.220 INFO:tasks.workunit.client.0.smithi016.stdout:1/562: dread d3/d4/da/f26 [4194304,4194304] 0 2024-04-22T15:49:35.221 INFO:tasks.workunit.client.0.smithi016.stdout:7/379: write d3/dc/d1b/d5a/d59/d64/f6c [215691,67582] 0 2024-04-22T15:49:35.223 INFO:tasks.workunit.client.0.smithi016.stdout:6/436: symlink d3/l92 0 2024-04-22T15:49:35.225 INFO:tasks.workunit.client.0.smithi016.stdout:2/328: write d2/d9/f12 [1281109,52177] 0 2024-04-22T15:49:35.226 INFO:tasks.workunit.client.0.smithi016.stdout:1/563: dread d3/f7 [0,4194304] 0 2024-04-22T15:49:35.229 INFO:tasks.workunit.client.0.smithi016.stdout:3/410: creat d7/d17/d18/d32/d3c/d47/f7f x:0 0 0 2024-04-22T15:49:35.230 INFO:tasks.workunit.client.0.smithi016.stdout:0/415: truncate dc/d13/d28/d65/f42 1276681 0 2024-04-22T15:49:35.232 INFO:tasks.workunit.client.0.smithi016.stdout:7/380: mkdir d3/dc/d1b/d5a/d69/d8c 0 2024-04-22T15:49:35.232 INFO:tasks.workunit.client.0.smithi016.stdout:9/410: dwrite d9/de/d26/f2a [0,4194304] 0 2024-04-22T15:49:35.234 INFO:tasks.workunit.client.0.smithi016.stdout:7/381: dread - d3/dc/d1b/d5a/d4a/f55 zero size 2024-04-22T15:49:35.235 INFO:tasks.workunit.client.0.smithi016.stdout:6/437: mknod d3/da/db/d44/d68/c93 0 2024-04-22T15:49:35.239 INFO:tasks.workunit.client.0.smithi016.stdout:7/382: dread - d3/dc/d1b/d5a/d59/d64/f89 zero size 2024-04-22T15:49:35.240 INFO:tasks.workunit.client.0.smithi016.stdout:3/411: dread d7/d17/d18/d27/f3d [0,4194304] 0 2024-04-22T15:49:35.241 INFO:tasks.workunit.client.0.smithi016.stdout:4/388: truncate d7/d24/d41/f74 2721509 0 2024-04-22T15:49:35.242 INFO:tasks.workunit.client.0.smithi016.stdout:9/411: mknod d9/de/d23/d24/d25/c7c 0 2024-04-22T15:49:35.245 INFO:tasks.workunit.client.0.smithi016.stdout:2/329: dread d2/f8 [0,4194304] 0 2024-04-22T15:49:35.246 INFO:tasks.workunit.client.0.smithi016.stdout:9/412: dread - d9/de/d23/d24/d25/d36/f6b zero size 2024-04-22T15:49:35.247 INFO:tasks.workunit.client.0.smithi016.stdout:9/413: chown d9/c72 0 1 2024-04-22T15:49:35.248 INFO:tasks.workunit.client.0.smithi016.stdout:1/564: dwrite d3/d4/d3a/d6b/d77/f76 [0,4194304] 0 2024-04-22T15:49:35.248 INFO:tasks.workunit.client.0.smithi016.stdout:7/383: symlink d3/dc/d1b/d31/l8d 0 2024-04-22T15:49:35.250 INFO:tasks.workunit.client.0.smithi016.stdout:3/412: write d7/d17/d18/d32/d3c/f6c [160183,58896] 0 2024-04-22T15:49:35.254 INFO:tasks.workunit.client.0.smithi016.stdout:2/330: creat d2/de/d2a/d59/f78 x:0 0 0 2024-04-22T15:49:35.255 INFO:tasks.workunit.client.0.smithi016.stdout:2/331: truncate d2/d23/d47/f66 831967 0 2024-04-22T15:49:35.256 INFO:tasks.workunit.client.0.smithi016.stdout:9/414: creat d9/de/d23/d24/d25/f7d x:0 0 0 2024-04-22T15:49:35.257 INFO:tasks.workunit.client.0.smithi016.stdout:9/415: stat d9/de/l42 0 2024-04-22T15:49:35.258 INFO:tasks.workunit.client.0.smithi016.stdout:1/565: mkdir d3/d2e/d80/dbb 0 2024-04-22T15:49:35.258 INFO:tasks.workunit.client.0.smithi016.stdout:9/416: stat d9/de/d23/f78 0 2024-04-22T15:49:35.258 INFO:tasks.workunit.client.0.smithi016.stdout:7/384: symlink d3/dc/d1b/d5a/d69/l8e 0 2024-04-22T15:49:35.266 INFO:tasks.workunit.client.0.smithi016.stdout:2/332: dread f0 [0,4194304] 0 2024-04-22T15:49:35.266 INFO:tasks.workunit.client.0.smithi016.stdout:1/566: mknod d3/d2e/cbc 0 2024-04-22T15:49:35.267 INFO:tasks.workunit.client.0.smithi016.stdout:1/567: dread - d3/d84/d97/fb4 zero size 2024-04-22T15:49:35.268 INFO:tasks.workunit.client.0.smithi016.stdout:9/417: mknod d9/d57/c7e 0 2024-04-22T15:49:35.268 INFO:tasks.workunit.client.0.smithi016.stdout:1/568: chown d3/d2e/la3 2 1 2024-04-22T15:49:35.269 INFO:tasks.workunit.client.0.smithi016.stdout:9/418: read d9/de/d26/f2a [1094743,19136] 0 2024-04-22T15:49:35.269 INFO:tasks.workunit.client.0.smithi016.stdout:9/419: truncate d9/d20/f38 571605 0 2024-04-22T15:49:35.270 INFO:tasks.workunit.client.0.smithi016.stdout:9/420: read - d9/de/d23/d24/d25/f7d zero size 2024-04-22T15:49:35.271 INFO:tasks.workunit.client.0.smithi016.stdout:4/389: link d7/d4e/d2f/d33/l34 d7/d3c/l86 0 2024-04-22T15:49:35.276 INFO:tasks.workunit.client.0.smithi016.stdout:0/416: dwrite dc/d13/d28/d65/f42 [0,4194304] 0 2024-04-22T15:49:35.277 INFO:tasks.workunit.client.0.smithi016.stdout:7/385: mknod d3/c8f 0 2024-04-22T15:49:35.278 INFO:tasks.workunit.client.0.smithi016.stdout:0/417: chown dc/d13/d14/d26/d32/c54 307 1 2024-04-22T15:49:35.279 INFO:tasks.workunit.client.0.smithi016.stdout:7/386: truncate d3/dc/d1b/d5a/f56 113611 0 2024-04-22T15:49:35.279 INFO:tasks.workunit.client.0.smithi016.stdout:1/569: mkdir d3/d28/dbd 0 2024-04-22T15:49:35.280 INFO:tasks.workunit.client.0.smithi016.stdout:9/421: chown d9/de/d23/d24/d4e/d4f/c77 37 1 2024-04-22T15:49:35.281 INFO:tasks.workunit.client.0.smithi016.stdout:4/390: symlink d7/d22/l87 0 2024-04-22T15:49:35.282 INFO:tasks.workunit.client.0.smithi016.stdout:3/413: link d7/d1b/d21/d55/f79 d7/d17/f80 0 2024-04-22T15:49:35.285 INFO:tasks.workunit.client.0.smithi016.stdout:2/333: mknod d2/c79 0 2024-04-22T15:49:35.286 INFO:tasks.workunit.client.0.smithi016.stdout:3/414: dread d7/d1b/d21/f73 [0,4194304] 0 2024-04-22T15:49:35.286 INFO:tasks.workunit.client.0.smithi016.stdout:2/334: dread - d2/d4d/d64/f6e zero size 2024-04-22T15:49:35.288 INFO:tasks.workunit.client.0.smithi016.stdout:0/418: mknod dc/d13/d14/d55/c7f 0 2024-04-22T15:49:35.290 INFO:tasks.workunit.client.0.smithi016.stdout:1/570: fsync d3/d4/da/f26 0 2024-04-22T15:49:35.291 INFO:tasks.workunit.client.0.smithi016.stdout:0/419: dread dc/d13/d28/d65/f42 [0,4194304] 0 2024-04-22T15:49:35.293 INFO:tasks.workunit.client.0.smithi016.stdout:0/420: fdatasync dc/d13/d14/d26/f4f 0 2024-04-22T15:49:35.295 INFO:tasks.workunit.client.0.smithi016.stdout:0/421: truncate dc/d13/d28/d65/d1f/f39 4983330 0 2024-04-22T15:49:35.297 INFO:tasks.workunit.client.0.smithi016.stdout:4/391: dwrite d7/d21/d45/d49/f4d [0,4194304] 0 2024-04-22T15:49:35.303 INFO:tasks.workunit.client.0.smithi016.stdout:7/387: write d3/dc/d1b/d5a/f46 [471054,24714] 0 2024-04-22T15:49:35.306 INFO:tasks.workunit.client.0.smithi016.stdout:1/571: dwrite d3/d4/d3a/d6b/d77/f81 [0,4194304] 0 2024-04-22T15:49:35.311 INFO:tasks.workunit.client.0.smithi016.stdout:4/392: mkdir d7/d21/d42/d59/d88 0 2024-04-22T15:49:35.311 INFO:tasks.workunit.client.0.smithi016.stdout:1/572: dread d3/d4/d19/f35 [0,4194304] 0 2024-04-22T15:49:35.312 INFO:tasks.workunit.client.0.smithi016.stdout:7/388: unlink d3/f23 0 2024-04-22T15:49:35.321 INFO:tasks.workunit.client.0.smithi016.stdout:1/573: mknod d3/d4/d15/cbe 0 2024-04-22T15:49:35.322 INFO:tasks.workunit.client.0.smithi016.stdout:9/422: getdents d9/de/d23/d24/d4e/d4f 0 2024-04-22T15:49:35.324 INFO:tasks.workunit.client.0.smithi016.stdout:2/335: getdents d2/d2c 0 2024-04-22T15:49:35.325 INFO:tasks.workunit.client.0.smithi016.stdout:9/423: dread d9/d37/f71 [0,4194304] 0 2024-04-22T15:49:35.326 INFO:tasks.workunit.client.0.smithi016.stdout:3/415: getdents d7/d17/d18/d32/d42/d63 0 2024-04-22T15:49:35.327 INFO:tasks.workunit.client.0.smithi016.stdout:3/416: write d7/d1b/f4b [3375733,53762] 0 2024-04-22T15:49:35.328 INFO:tasks.workunit.client.0.smithi016.stdout:3/417: chown d7/d17/f2e 6747 1 2024-04-22T15:49:35.329 INFO:tasks.workunit.client.0.smithi016.stdout:0/422: truncate dc/d13/d28/d65/d1f/f39 4585280 0 2024-04-22T15:49:35.332 INFO:tasks.workunit.client.0.smithi016.stdout:2/336: creat d2/d1a/d4c/f7a x:0 0 0 2024-04-22T15:49:35.333 INFO:tasks.workunit.client.0.smithi016.stdout:4/393: rename d7/d21/d45/l6b to d7/d24/l89 0 2024-04-22T15:49:35.334 INFO:tasks.workunit.client.0.smithi016.stdout:3/418: dwrite d7/d1b/f4b [0,4194304] 0 2024-04-22T15:49:35.336 INFO:tasks.workunit.client.0.smithi016.stdout:7/389: dread d3/d32/f3d [4194304,4194304] 0 2024-04-22T15:49:35.337 INFO:tasks.workunit.client.0.smithi016.stdout:1/574: mkdir d3/d4/d3a/d9e/dbf 0 2024-04-22T15:49:35.338 INFO:tasks.workunit.client.0.smithi016.stdout:1/575: chown d3/d4/d3a/d6b/f94 67 1 2024-04-22T15:49:35.344 INFO:tasks.workunit.client.0.smithi016.stdout:4/394: chown d7/d4e/d2f/d33/c5c 0 1 2024-04-22T15:49:35.347 INFO:tasks.workunit.client.0.smithi016.stdout:3/419: rename d7/c50 to d7/d17/d18/d27/c81 0 2024-04-22T15:49:35.348 INFO:tasks.workunit.client.0.smithi016.stdout:3/420: readlink d7/d1b/l1d 0 2024-04-22T15:49:35.348 INFO:tasks.workunit.client.0.smithi016.stdout:7/390: creat d3/dc/d1b/d5a/d69/f90 x:0 0 0 2024-04-22T15:49:35.350 INFO:tasks.workunit.client.0.smithi016.stdout:5/361: sync 2024-04-22T15:49:35.350 INFO:tasks.workunit.client.0.smithi016.stdout:0/423: creat dc/d13/d28/d65/d30/d75/f80 x:0 0 0 2024-04-22T15:49:35.351 INFO:tasks.workunit.client.0.smithi016.stdout:1/576: rmdir d3/d4/d3a/d91 39 2024-04-22T15:49:35.352 INFO:tasks.workunit.client.0.smithi016.stdout:5/362: write df/f1f [4919222,128747] 0 2024-04-22T15:49:35.355 INFO:tasks.workunit.client.0.smithi016.stdout:4/395: fsync d7/d24/f3d 0 2024-04-22T15:49:35.356 INFO:tasks.workunit.client.0.smithi016.stdout:7/391: dwrite d3/d32/f72 [0,4194304] 0 2024-04-22T15:49:35.357 INFO:tasks.workunit.client.0.smithi016.stdout:4/396: chown d7/d21/d42/d80 4079 1 2024-04-22T15:49:35.358 INFO:tasks.workunit.client.0.smithi016.stdout:4/397: write d7/d21/d45/d49/f4d [4956170,26591] 0 2024-04-22T15:49:35.361 INFO:tasks.workunit.client.0.smithi016.stdout:1/577: creat d3/d4/d15/fc0 x:0 0 0 2024-04-22T15:49:35.362 INFO:tasks.workunit.client.0.smithi016.stdout:1/578: read d3/d4/d3a/d6b/d77/f76 [3381500,9742] 0 2024-04-22T15:49:35.368 INFO:tasks.workunit.client.0.smithi016.stdout:1/579: dread d3/d2e/f8a [0,4194304] 0 2024-04-22T15:49:35.368 INFO:tasks.workunit.client.0.smithi016.stdout:3/421: dwrite d7/f5a [0,4194304] 0 2024-04-22T15:49:35.371 INFO:tasks.workunit.client.0.smithi016.stdout:4/398: mkdir d7/d4e/d39/d8a 0 2024-04-22T15:49:35.372 INFO:tasks.workunit.client.0.smithi016.stdout:4/399: dread - d7/d4e/d2f/d50/f61 zero size 2024-04-22T15:49:35.374 INFO:tasks.workunit.client.0.smithi016.stdout:2/337: getdents d2/d23/d47 0 2024-04-22T15:49:35.374 INFO:tasks.workunit.client.0.smithi016.stdout:4/400: dread - d7/d3c/f44 zero size 2024-04-22T15:49:35.375 INFO:tasks.workunit.client.0.smithi016.stdout:7/392: dread d3/dc/d1b/d5a/f5b [0,4194304] 0 2024-04-22T15:49:35.376 INFO:tasks.workunit.client.0.smithi016.stdout:5/363: symlink df/d15/d1a/d50/l68 0 2024-04-22T15:49:35.377 INFO:tasks.workunit.client.0.smithi016.stdout:0/424: mknod dc/d13/d28/d65/d40/d4a/d59/d78/c81 0 2024-04-22T15:49:35.378 INFO:tasks.workunit.client.0.smithi016.stdout:3/422: dwrite d7/f5e [0,4194304] 0 2024-04-22T15:49:35.380 INFO:tasks.workunit.client.0.smithi016.stdout:7/393: readlink d3/dc/d1b/d5a/d9/df/l4e 0 2024-04-22T15:49:35.383 INFO:tasks.workunit.client.0.smithi016.stdout:7/394: stat d3/c8f 0 2024-04-22T15:49:35.384 INFO:tasks.workunit.client.0.smithi016.stdout:1/580: creat d3/d4/d3a/d6b/fc1 x:0 0 0 2024-04-22T15:49:35.384 INFO:tasks.workunit.client.0.smithi016.stdout:2/338: mkdir d2/d1a/d7b 0 2024-04-22T15:49:35.390 INFO:tasks.workunit.client.0.smithi016.stdout:7/395: write d3/dc/d1b/d5a/f46 [1220408,108941] 0 2024-04-22T15:49:35.391 INFO:tasks.workunit.client.0.smithi016.stdout:4/401: dwrite d7/d21/d42/f4c [4194304,4194304] 0 2024-04-22T15:49:35.392 INFO:tasks.workunit.client.0.smithi016.stdout:3/423: symlink d7/d17/d18/d32/d3c/l82 0 2024-04-22T15:49:35.394 INFO:tasks.workunit.client.0.smithi016.stdout:2/339: creat d2/d23/d27/d67/f7c x:0 0 0 2024-04-22T15:49:35.395 INFO:tasks.workunit.client.0.smithi016.stdout:5/364: dwrite df/d15/d1a/d28/f2c [0,4194304] 0 2024-04-22T15:49:35.395 INFO:tasks.workunit.client.0.smithi016.stdout:1/581: creat d3/d2e/d80/dbb/fc2 x:0 0 0 2024-04-22T15:49:35.398 INFO:tasks.workunit.client.0.smithi016.stdout:4/402: chown d7/d21/d45/d49/d67/c7b 114 1 2024-04-22T15:49:35.398 INFO:tasks.workunit.client.0.smithi016.stdout:0/425: dwrite dc/d13/d28/d65/d30/f44 [0,4194304] 0 2024-04-22T15:49:35.402 INFO:tasks.workunit.client.0.smithi016.stdout:1/582: rmdir d3/d28 39 2024-04-22T15:49:35.403 INFO:tasks.workunit.client.0.smithi016.stdout:3/424: mkdir d7/d17/d18/d32/d3c/d47/d83 0 2024-04-22T15:49:35.406 INFO:tasks.workunit.client.0.smithi016.stdout:0/426: unlink dc/d13/d28/d65/d1f/l29 0 2024-04-22T15:49:35.407 INFO:tasks.workunit.client.0.smithi016.stdout:2/340: mknod d2/d23/d47/d55/c7d 0 2024-04-22T15:49:35.409 INFO:tasks.workunit.client.0.smithi016.stdout:3/425: chown l5 2907 1 2024-04-22T15:49:35.412 INFO:tasks.workunit.client.0.smithi016.stdout:1/583: write d3/d4/d37/fb9 [2394835,8018] 0 2024-04-22T15:49:35.413 INFO:tasks.workunit.client.0.smithi016.stdout:7/396: dwrite d3/dc/d1b/d5a/d4a/f55 [0,4194304] 0 2024-04-22T15:49:35.415 INFO:tasks.workunit.client.0.smithi016.stdout:4/403: dwrite d7/d24/f3d [0,4194304] 0 2024-04-22T15:49:35.416 INFO:tasks.workunit.client.0.smithi016.stdout:3/426: mknod d7/d17/d18/d32/c84 0 2024-04-22T15:49:35.425 INFO:tasks.workunit.client.0.smithi016.stdout:0/427: dwrite dc/d13/d28/d65/d40/f73 [0,4194304] 0 2024-04-22T15:49:35.426 INFO:tasks.workunit.client.0.smithi016.stdout:1/584: mkdir d3/d4/d3a/d6b/dc3 0 2024-04-22T15:49:35.426 INFO:tasks.workunit.client.0.smithi016.stdout:2/341: write d2/d23/d27/f32 [418936,11436] 0 2024-04-22T15:49:35.431 INFO:tasks.workunit.client.0.smithi016.stdout:4/404: symlink d7/d4e/d81/l8b 0 2024-04-22T15:49:35.432 INFO:tasks.workunit.client.0.smithi016.stdout:4/405: read d7/d21/d45/f66 [702977,88743] 0 2024-04-22T15:49:35.433 INFO:tasks.workunit.client.0.smithi016.stdout:2/342: creat d2/d4d/d6d/f7e x:0 0 0 2024-04-22T15:49:35.433 INFO:tasks.workunit.client.0.smithi016.stdout:2/343: fsync d2/d2c/f5b 0 2024-04-22T15:49:35.435 INFO:tasks.workunit.client.0.smithi016.stdout:1/585: fdatasync d3/d28/f8b 0 2024-04-22T15:49:35.437 INFO:tasks.workunit.client.0.smithi016.stdout:0/428: mkdir dc/d13/d28/d65/d40/d4a/d61/d82 0 2024-04-22T15:49:35.439 INFO:tasks.workunit.client.0.smithi016.stdout:4/406: symlink d7/d24/d41/l8c 0 2024-04-22T15:49:35.439 INFO:tasks.workunit.client.0.smithi016.stdout:0/429: dread - dc/d13/d5d/f6c zero size 2024-04-22T15:49:35.439 INFO:tasks.workunit.client.0.smithi016.stdout:4/407: chown d7/d24 13 1 2024-04-22T15:49:35.440 INFO:tasks.workunit.client.0.smithi016.stdout:4/408: write d7/d24/d51/f6a [4223600,2869] 0 2024-04-22T15:49:35.440 INFO:tasks.workunit.client.0.smithi016.stdout:2/344: rmdir d2/de/d2a/d59 39 2024-04-22T15:49:35.446 INFO:tasks.workunit.client.0.smithi016.stdout:2/345: rename d2/f8 to d2/d4d/f7f 0 2024-04-22T15:49:35.447 INFO:tasks.workunit.client.0.smithi016.stdout:3/427: dread f1 [4194304,4194304] 0 2024-04-22T15:49:35.447 INFO:tasks.workunit.client.0.smithi016.stdout:0/430: dwrite dc/d13/d14/f18 [0,4194304] 0 2024-04-22T15:49:35.449 INFO:tasks.workunit.client.0.smithi016.stdout:0/431: stat dc/d13/d28/d65/d40/f73 0 2024-04-22T15:49:35.449 INFO:tasks.workunit.client.0.smithi016.stdout:2/346: write d2/d23/d27/d67/f75 [103694,93557] 0 2024-04-22T15:49:35.452 INFO:tasks.workunit.client.0.smithi016.stdout:3/428: truncate d7/f22 2297481 0 2024-04-22T15:49:35.455 INFO:tasks.workunit.client.0.smithi016.stdout:7/397: write d3/dc/d1b/d5a/d9/f26 [1292260,100714] 0 2024-04-22T15:49:35.455 INFO:tasks.workunit.client.0.smithi016.stdout:4/409: dread d7/d24/d41/f57 [0,4194304] 0 2024-04-22T15:49:35.455 INFO:tasks.workunit.client.0.smithi016.stdout:0/432: symlink dc/d13/d28/d65/d1f/l83 0 2024-04-22T15:49:35.457 INFO:tasks.workunit.client.0.smithi016.stdout:7/398: chown d3/dc/d1b/f49 4222 1 2024-04-22T15:49:35.462 INFO:tasks.workunit.client.0.smithi016.stdout:0/433: creat dc/d13/d28/d65/d30/f84 x:0 0 0 2024-04-22T15:49:35.464 INFO:tasks.workunit.client.0.smithi016.stdout:0/434: write dc/d13/d14/d26/d32/f33 [1330746,85608] 0 2024-04-22T15:49:35.464 INFO:tasks.workunit.client.0.smithi016.stdout:7/399: symlink d3/dc/d1b/d5a/d4a/d6e/l91 0 2024-04-22T15:49:35.466 INFO:tasks.workunit.client.0.smithi016.stdout:7/400: chown d3/dc/d1b/d31/l82 518008 1 2024-04-22T15:49:35.473 INFO:tasks.workunit.client.0.smithi016.stdout:3/429: getdents d7/d75 0 2024-04-22T15:49:35.474 INFO:tasks.workunit.client.0.smithi016.stdout:1/586: write d3/f23 [1331176,74374] 0 2024-04-22T15:49:35.476 INFO:tasks.workunit.client.0.smithi016.stdout:7/401: fsync d3/dc/d1b/d5a/d9/fb 0 2024-04-22T15:49:35.478 INFO:tasks.workunit.client.0.smithi016.stdout:2/347: dread d2/d23/f3b [0,4194304] 0 2024-04-22T15:49:35.480 INFO:tasks.workunit.client.0.smithi016.stdout:3/430: creat d7/d1b/f85 x:0 0 0 2024-04-22T15:49:35.481 INFO:tasks.workunit.client.0.smithi016.stdout:1/587: truncate d3/f7 3346751 0 2024-04-22T15:49:35.484 INFO:tasks.workunit.client.0.smithi016.stdout:3/431: dwrite d7/f69 [0,4194304] 0 2024-04-22T15:49:35.486 INFO:tasks.workunit.client.0.smithi016.stdout:4/410: link d7/d21/d45/d49/c65 d7/d4e/d2f/d33/c8d 0 2024-04-22T15:49:35.488 INFO:tasks.workunit.client.0.smithi016.stdout:6/438: sync 2024-04-22T15:49:35.488 INFO:tasks.workunit.client.0.smithi016.stdout:8/380: sync 2024-04-22T15:49:35.489 INFO:tasks.workunit.client.0.smithi016.stdout:1/588: dwrite d3/d4/d19/d20/f33 [0,4194304] 0 2024-04-22T15:49:35.490 INFO:tasks.workunit.client.0.smithi016.stdout:6/439: dread - d3/da/dd/d1b/f8b zero size 2024-04-22T15:49:35.492 INFO:tasks.workunit.client.0.smithi016.stdout:1/589: write d3/d4/da/f26 [3815552,85818] 0 2024-04-22T15:49:35.495 INFO:tasks.workunit.client.0.smithi016.stdout:0/435: truncate dc/d13/d14/d26/f46 79133 0 2024-04-22T15:49:35.497 INFO:tasks.workunit.client.0.smithi016.stdout:1/590: dread d3/d4/d3a/d6b/d77/f99 [0,4194304] 0 2024-04-22T15:49:35.503 INFO:tasks.workunit.client.0.smithi016.stdout:3/432: mkdir d7/d17/d18/d32/d42/d63/d86 0 2024-04-22T15:49:35.510 INFO:tasks.workunit.client.0.smithi016.stdout:8/381: rmdir d2 39 2024-04-22T15:49:35.517 INFO:tasks.workunit.client.0.smithi016.stdout:0/436: creat dc/d13/d5d/f85 x:0 0 0 2024-04-22T15:49:35.524 INFO:tasks.workunit.client.0.smithi016.stdout:2/348: write d2/f11 [1385417,69985] 0 2024-04-22T15:49:35.524 INFO:tasks.workunit.client.0.smithi016.stdout:4/411: truncate d7/d24/d41/f74 673073 0 2024-04-22T15:49:35.527 INFO:tasks.workunit.client.0.smithi016.stdout:6/440: mknod d3/da/dd/d7b/c94 0 2024-04-22T15:49:35.528 INFO:tasks.workunit.client.0.smithi016.stdout:1/591: mkdir d3/d4/dc4 0 2024-04-22T15:49:35.529 INFO:tasks.workunit.client.0.smithi016.stdout:7/402: link d3/dc/d22/c3b d3/dc/d1b/d5a/c92 0 2024-04-22T15:49:35.529 INFO:tasks.workunit.client.0.smithi016.stdout:6/441: dread - d3/da/dd/d1b/f8b zero size 2024-04-22T15:49:35.530 INFO:tasks.workunit.client.0.smithi016.stdout:2/349: rmdir d2/d2c 39 2024-04-22T15:49:35.531 INFO:tasks.workunit.client.0.smithi016.stdout:6/442: dread - d3/da/dd/d67/f7c zero size 2024-04-22T15:49:35.531 INFO:tasks.workunit.client.0.smithi016.stdout:8/382: dwrite d2/f5c [0,4194304] 0 2024-04-22T15:49:35.532 INFO:tasks.workunit.client.0.smithi016.stdout:6/443: chown d3/da/db/d14/d16/l2f 3633417 1 2024-04-22T15:49:35.533 INFO:tasks.workunit.client.0.smithi016.stdout:6/444: dread - d3/da/dd/d67/f7c zero size 2024-04-22T15:49:35.536 INFO:tasks.workunit.client.0.smithi016.stdout:3/433: rename d7/ld to d7/d17/d18/d32/d3c/l87 0 2024-04-22T15:49:35.537 INFO:tasks.workunit.client.0.smithi016.stdout:1/592: fdatasync d3/d4/d15/d38/d63/f75 0 2024-04-22T15:49:35.538 INFO:tasks.workunit.client.0.smithi016.stdout:1/593: chown d3/c5a 3820 1 2024-04-22T15:49:35.540 INFO:tasks.workunit.client.0.smithi016.stdout:7/403: fdatasync d3/dc/d1b/d5a/d59/d64/f89 0 2024-04-22T15:49:35.541 INFO:tasks.workunit.client.0.smithi016.stdout:6/445: dwrite d3/da/dd/f91 [0,4194304] 0 2024-04-22T15:49:35.545 INFO:tasks.workunit.client.0.smithi016.stdout:4/412: symlink d7/d24/l8e 0 2024-04-22T15:49:35.547 INFO:tasks.workunit.client.0.smithi016.stdout:8/383: unlink d2/d10/d43/d4a/d28/d44/d66/l50 0 2024-04-22T15:49:35.548 INFO:tasks.workunit.client.0.smithi016.stdout:3/434: dread f1 [0,4194304] 0 2024-04-22T15:49:35.558 INFO:tasks.workunit.client.0.smithi016.stdout:6/446: readlink d3/da/db/d44/d68/l7a 0 2024-04-22T15:49:35.559 INFO:tasks.workunit.client.0.smithi016.stdout:6/447: fdatasync d3/da/db/d14/d16/f3d 0 2024-04-22T15:49:35.560 INFO:tasks.workunit.client.0.smithi016.stdout:7/404: rename d3/dc/d1b/d5a/d9/c66 to d3/dc/d1b/d31/c93 0 2024-04-22T15:49:35.561 INFO:tasks.workunit.client.0.smithi016.stdout:7/405: chown d3/d14/f85 12451317 1 2024-04-22T15:49:35.564 INFO:tasks.workunit.client.0.smithi016.stdout:7/406: dread d3/f74 [0,4194304] 0 2024-04-22T15:49:35.566 INFO:tasks.workunit.client.0.smithi016.stdout:3/435: readlink d7/l1a 0 2024-04-22T15:49:35.567 INFO:tasks.workunit.client.0.smithi016.stdout:8/384: symlink d2/d10/d43/d4a/d28/d44/l76 0 2024-04-22T15:49:35.567 INFO:tasks.workunit.client.0.smithi016.stdout:3/436: read d7/f12 [3115254,95311] 0 2024-04-22T15:49:35.570 INFO:tasks.workunit.client.0.smithi016.stdout:4/413: dwrite d7/d4e/d2f/d6f/f84 [4194304,4194304] 0 2024-04-22T15:49:35.573 INFO:tasks.workunit.client.0.smithi016.stdout:7/407: dwrite d3/dc/d1b/d5a/f56 [0,4194304] 0 2024-04-22T15:49:35.575 INFO:tasks.workunit.client.0.smithi016.stdout:7/408: write d3/dc/d1b/d5a/d9/df/f11 [918657,363] 0 2024-04-22T15:49:35.576 INFO:tasks.workunit.client.0.smithi016.stdout:7/409: chown d3/dc/d1b/d5a/d4a/d6e/f73 14669 1 2024-04-22T15:49:35.582 INFO:tasks.workunit.client.0.smithi016.stdout:1/594: link d3/d4/d15/f16 d3/d4/d15/d8c/fc5 0 2024-04-22T15:49:35.582 INFO:tasks.workunit.client.0.smithi016.stdout:6/448: dread d3/f1d [0,4194304] 0 2024-04-22T15:49:35.583 INFO:tasks.workunit.client.0.smithi016.stdout:1/595: chown d3/d4/d19/d20/d2b/l61 30644 1 2024-04-22T15:49:35.584 INFO:tasks.workunit.client.0.smithi016.stdout:6/449: chown d3/da/db/d14/d4c/l66 376 1 2024-04-22T15:49:35.589 INFO:tasks.workunit.client.0.smithi016.stdout:8/385: dread d2/d10/f61 [0,4194304] 0 2024-04-22T15:49:35.590 INFO:tasks.workunit.client.0.smithi016.stdout:5/365: sync 2024-04-22T15:49:35.590 INFO:tasks.workunit.client.0.smithi016.stdout:9/424: sync 2024-04-22T15:49:35.592 INFO:tasks.workunit.client.0.smithi016.stdout:9/425: dread - d9/de/d23/d24/d25/d36/f5e zero size 2024-04-22T15:49:35.592 INFO:tasks.workunit.client.0.smithi016.stdout:6/450: dread d3/da/dd/f5a [0,4194304] 0 2024-04-22T15:49:35.596 INFO:tasks.workunit.client.0.smithi016.stdout:2/350: link d2/d2c/f37 d2/d9/f80 0 2024-04-22T15:49:35.600 INFO:tasks.workunit.client.0.smithi016.stdout:7/410: rename d3/d32 to d3/dc/d1b/d5a/d9/df/d94 0 2024-04-22T15:49:35.631 INFO:tasks.workunit.client.0.smithi016.stdout:0/437: sync 2024-04-22T15:49:35.665 INFO:tasks.workunit.client.0.smithi016.stdout:5/366: creat df/d15/d1a/d4e/f69 x:0 0 0 2024-04-22T15:49:35.671 INFO:tasks.workunit.client.0.smithi016.stdout:9/426: mknod d9/d5d/d62/c7f 0 2024-04-22T15:49:35.671 INFO:tasks.workunit.client.0.smithi016.stdout:3/437: dwrite d7/d1b/d21/f73 [0,4194304] 0 2024-04-22T15:49:35.673 INFO:tasks.workunit.client.0.smithi016.stdout:6/451: mkdir d3/da/db/d44/d68/d95 0 2024-04-22T15:49:35.675 INFO:tasks.workunit.client.0.smithi016.stdout:2/351: symlink d2/d4d/d6d/l81 0 2024-04-22T15:49:35.675 INFO:tasks.workunit.client.0.smithi016.stdout:9/427: read d9/de/d23/d24/d4e/d4f/f69 [3627276,40639] 0 2024-04-22T15:49:35.677 INFO:tasks.workunit.client.0.smithi016.stdout:7/411: symlink d3/dc/d1b/d31/l95 0 2024-04-22T15:49:35.678 INFO:tasks.workunit.client.0.smithi016.stdout:1/596: symlink d3/d4/d3a/d91/lc6 0 2024-04-22T15:49:35.679 INFO:tasks.workunit.client.0.smithi016.stdout:1/597: write d3/d2e/d80/fa5 [1084038,8795] 0 2024-04-22T15:49:35.681 INFO:tasks.workunit.client.0.smithi016.stdout:1/598: chown d3/d4/d37/c4b 3 1 2024-04-22T15:49:35.690 INFO:tasks.workunit.client.0.smithi016.stdout:4/414: link d7/d4e/c2e d7/d4e/c8f 0 2024-04-22T15:49:35.691 INFO:tasks.workunit.client.0.smithi016.stdout:5/367: symlink df/d15/d1a/d28/d33/l6a 0 2024-04-22T15:49:35.700 INFO:tasks.workunit.client.0.smithi016.stdout:4/415: dread d7/d24/d51/f6a [0,4194304] 0 2024-04-22T15:49:35.702 INFO:tasks.workunit.client.0.smithi016.stdout:4/416: truncate d7/d4e/d2f/d33/f7a 263676 0 2024-04-22T15:49:35.705 INFO:tasks.workunit.client.0.smithi016.stdout:3/438: readlink d7/d17/d18/d32/d3c/l87 0 2024-04-22T15:49:35.705 INFO:tasks.workunit.client.0.smithi016.stdout:6/452: rename d3/da/db/d44/d68/c80 to d3/da/db/d14/d4c/d58/c96 0 2024-04-22T15:49:35.707 INFO:tasks.workunit.client.0.smithi016.stdout:3/439: write d7/d17/d18/f70 [1006017,39203] 0 2024-04-22T15:49:35.711 INFO:tasks.workunit.client.0.smithi016.stdout:2/352: write d2/de/d2a/d59/f78 [1039983,6899] 0 2024-04-22T15:49:35.711 INFO:tasks.workunit.client.0.smithi016.stdout:9/428: write d9/f1b [370580,29763] 0 2024-04-22T15:49:35.713 INFO:tasks.workunit.client.0.smithi016.stdout:6/453: dwrite d3/da/dd/d67/f7c [0,4194304] 0 2024-04-22T15:49:35.713 INFO:tasks.workunit.client.0.smithi016.stdout:9/429: dread - d9/f1c zero size 2024-04-22T15:49:35.713 INFO:tasks.workunit.client.0.smithi016.stdout:2/353: dread - d2/d23/d47/f71 zero size 2024-04-22T15:49:35.716 INFO:tasks.workunit.client.0.smithi016.stdout:7/412: rmdir d3/dc/d1b/d31/d5f 39 2024-04-22T15:49:35.718 INFO:tasks.workunit.client.0.smithi016.stdout:7/413: dread d3/dc/f1e [0,4194304] 0 2024-04-22T15:49:35.718 INFO:tasks.workunit.client.0.smithi016.stdout:7/414: dread - d3/dc/d33/f7f zero size 2024-04-22T15:49:35.718 INFO:tasks.workunit.client.0.smithi016.stdout:7/415: dread - d3/dc/d1b/d5a/d9/df/f70 zero size 2024-04-22T15:49:35.727 INFO:tasks.workunit.client.0.smithi016.stdout:5/368: chown df/d15/d1a/d28/d33/l44 21819 1 2024-04-22T15:49:35.742 INFO:tasks.workunit.client.0.smithi016.stdout:2/354: chown d2/d2c/f6b 34 1 2024-04-22T15:49:35.745 INFO:tasks.workunit.client.0.smithi016.stdout:6/454: unlink d3/da/d12/d13/d46/c5c 0 2024-04-22T15:49:35.745 INFO:tasks.workunit.client.0.smithi016.stdout:9/430: symlink d9/de/d26/d5a/l80 0 2024-04-22T15:49:35.759 INFO:tasks.workunit.client.0.smithi016.stdout:2/355: creat d2/d2c/f82 x:0 0 0 2024-04-22T15:49:35.763 INFO:tasks.workunit.client.0.smithi016.stdout:6/455: unlink d3/da/dd/d1b/l65 0 2024-04-22T15:49:35.769 INFO:tasks.workunit.client.0.smithi016.stdout:4/417: rmdir d7/d4e/d39/d8a 0 2024-04-22T15:49:35.774 INFO:tasks.workunit.client.0.smithi016.stdout:6/456: chown d3/c4a 0 1 2024-04-22T15:49:35.780 INFO:tasks.workunit.client.0.smithi016.stdout:1/599: rename d3/d4/d37/l49 to d3/d4/d15/d38/d63/lc7 0 2024-04-22T15:49:35.781 INFO:tasks.workunit.client.0.smithi016.stdout:4/418: dwrite d7/d22/f27 [0,4194304] 0 2024-04-22T15:49:35.786 INFO:tasks.workunit.client.0.smithi016.stdout:4/419: stat d7/d3c/l86 0 2024-04-22T15:49:35.793 INFO:tasks.workunit.client.0.smithi016.stdout:4/420: dwrite d7/d4e/d2f/d33/f7a [0,4194304] 0 2024-04-22T15:49:35.815 INFO:tasks.workunit.client.0.smithi016.stdout:8/386: truncate d2/d10/d43/d4a/d28/d56/f64 267608 0 2024-04-22T15:49:35.815 INFO:tasks.workunit.client.0.smithi016.stdout:8/387: chown d2/d32 274 1 2024-04-22T15:49:35.822 INFO:tasks.workunit.client.0.smithi016.stdout:6/457: getdents d3/da/db/d14/d4c/d58 0 2024-04-22T15:49:35.827 INFO:tasks.workunit.client.0.smithi016.stdout:0/438: truncate dc/d13/d28/d65/d30/f44 1857836 0 2024-04-22T15:49:35.830 INFO:tasks.workunit.client.0.smithi016.stdout:0/439: creat dc/d13/d28/d65/d40/d4a/d59/d78/f86 x:0 0 0 2024-04-22T15:49:35.832 INFO:tasks.workunit.client.0.smithi016.stdout:6/458: dread d3/da/d12/ff [0,4194304] 0 2024-04-22T15:49:35.832 INFO:tasks.workunit.client.0.smithi016.stdout:6/459: dread - d3/da/d12/d13/f3f zero size 2024-04-22T15:49:35.835 INFO:tasks.workunit.client.0.smithi016.stdout:6/460: mkdir d3/da/d12/d13/d97 0 2024-04-22T15:49:35.839 INFO:tasks.workunit.client.0.smithi016.stdout:5/369: rename df/d15/d1a/d28/d33/l6a to df/d15/l6b 0 2024-04-22T15:49:35.843 INFO:tasks.workunit.client.0.smithi016.stdout:5/370: write df/d15/d1a/f4b [26503,13775] 0 2024-04-22T15:49:35.848 INFO:tasks.workunit.client.0.smithi016.stdout:5/371: creat df/d15/d1a/f6c x:0 0 0 2024-04-22T15:49:35.849 INFO:tasks.workunit.client.0.smithi016.stdout:1/600: rmdir d3/d4/d3a 39 2024-04-22T15:49:35.849 INFO:tasks.workunit.client.0.smithi016.stdout:5/372: write df/d15/d1a/f6c [614520,46312] 0 2024-04-22T15:49:35.853 INFO:tasks.workunit.client.0.smithi016.stdout:5/373: dread df/d22/f3b [0,4194304] 0 2024-04-22T15:49:35.854 INFO:tasks.workunit.client.0.smithi016.stdout:3/440: truncate d7/f14 2935404 0 2024-04-22T15:49:35.857 INFO:tasks.workunit.client.0.smithi016.stdout:5/374: unlink df/d5b/d5d/d5f/f63 0 2024-04-22T15:49:35.864 INFO:tasks.workunit.client.0.smithi016.stdout:9/431: write d9/d37/f71 [2870220,122287] 0 2024-04-22T15:49:35.864 INFO:tasks.workunit.client.0.smithi016.stdout:3/441: creat d7/d17/d18/d32/d3c/f88 x:0 0 0 2024-04-22T15:49:35.866 INFO:tasks.workunit.client.0.smithi016.stdout:5/375: creat df/d15/d1a/d28/f6d x:0 0 0 2024-04-22T15:49:35.866 INFO:tasks.workunit.client.0.smithi016.stdout:3/442: read d7/d17/d18/d32/f57 [850921,60978] 0 2024-04-22T15:49:35.867 INFO:tasks.workunit.client.0.smithi016.stdout:5/376: write df/f27 [1799182,27975] 0 2024-04-22T15:49:35.868 INFO:tasks.workunit.client.0.smithi016.stdout:5/377: write df/d15/d1a/d4e/f69 [320175,2010] 0 2024-04-22T15:49:35.871 INFO:tasks.workunit.client.0.smithi016.stdout:9/432: rename d9/de/d23/d24/d4e/c6c to d9/de/d23/d24/d25/c81 0 2024-04-22T15:49:35.871 INFO:tasks.workunit.client.0.smithi016.stdout:3/443: mknod d7/d1b/d21/d55/c89 0 2024-04-22T15:49:35.872 INFO:tasks.workunit.client.0.smithi016.stdout:9/433: write d9/d20/f56 [30734,42139] 0 2024-04-22T15:49:35.876 INFO:tasks.workunit.client.0.smithi016.stdout:2/356: rmdir d2/de/d2a 39 2024-04-22T15:49:35.877 INFO:tasks.workunit.client.0.smithi016.stdout:3/444: mkdir d7/d17/d18/d32/d8a 0 2024-04-22T15:49:35.878 INFO:tasks.workunit.client.0.smithi016.stdout:3/445: write d7/d1b/d21/f73 [2239154,30380] 0 2024-04-22T15:49:35.881 INFO:tasks.workunit.client.0.smithi016.stdout:2/357: creat d2/d23/d27/d34/d60/f83 x:0 0 0 2024-04-22T15:49:35.888 INFO:tasks.workunit.client.0.smithi016.stdout:7/416: creat d3/dc/d1b/f96 x:0 0 0 2024-04-22T15:49:35.889 INFO:tasks.workunit.client.0.smithi016.stdout:2/358: read d2/de/d2a/f49 [704899,6700] 0 2024-04-22T15:49:35.892 INFO:tasks.workunit.client.0.smithi016.stdout:5/378: getdents df/d15/d1a/d4e/d58 0 2024-04-22T15:49:35.892 INFO:tasks.workunit.client.0.smithi016.stdout:4/421: truncate d7/f69 307247 0 2024-04-22T15:49:35.893 INFO:tasks.workunit.client.0.smithi016.stdout:5/379: truncate df/d15/d1a/d28/f6d 723657 0 2024-04-22T15:49:35.894 INFO:tasks.workunit.client.0.smithi016.stdout:4/422: write d7/d24/f3d [5047812,94135] 0 2024-04-22T15:49:35.897 INFO:tasks.workunit.client.0.smithi016.stdout:3/446: mknod d7/c8b 0 2024-04-22T15:49:35.900 INFO:tasks.workunit.client.0.smithi016.stdout:4/423: symlink d7/l90 0 2024-04-22T15:49:35.905 INFO:tasks.workunit.client.0.smithi016.stdout:2/359: unlink d2/f18 0 2024-04-22T15:49:35.908 INFO:tasks.workunit.client.0.smithi016.stdout:3/447: creat d7/d1b/d21/d55/f8c x:0 0 0 2024-04-22T15:49:35.910 INFO:tasks.workunit.client.0.smithi016.stdout:8/388: dwrite d2/d10/d43/d4a/d28/d3e/f42 [8388608,4194304] 0 2024-04-22T15:49:35.912 INFO:tasks.workunit.client.0.smithi016.stdout:4/424: dread d7/f16 [0,4194304] 0 2024-04-22T15:49:35.912 INFO:tasks.workunit.client.0.smithi016.stdout:8/389: dread d2/d10/f61 [0,4194304] 0 2024-04-22T15:49:35.913 INFO:tasks.workunit.client.0.smithi016.stdout:7/417: getdents d3 0 2024-04-22T15:49:35.920 INFO:tasks.workunit.client.0.smithi016.stdout:4/425: mknod d7/d24/d41/c91 0 2024-04-22T15:49:35.921 INFO:tasks.workunit.client.0.smithi016.stdout:0/440: dwrite dc/d13/d28/d65/d1f/f39 [0,4194304] 0 2024-04-22T15:49:35.925 INFO:tasks.workunit.client.0.smithi016.stdout:5/380: link df/d15/d1a/l1c df/d5b/d5d/l6e 0 2024-04-22T15:49:35.927 INFO:tasks.workunit.client.0.smithi016.stdout:5/381: chown df/d15/f45 3363503 1 2024-04-22T15:49:35.928 INFO:tasks.workunit.client.0.smithi016.stdout:3/448: rename d7/d17/d18/d32/c6d to d7/d17/d18/c8d 0 2024-04-22T15:49:35.928 INFO:tasks.workunit.client.0.smithi016.stdout:6/461: dwrite d3/da/d12/f1c [0,4194304] 0 2024-04-22T15:49:35.929 INFO:tasks.workunit.client.0.smithi016.stdout:4/426: write d7/d4e/d2f/d50/f61 [91705,122952] 0 2024-04-22T15:49:35.930 INFO:tasks.workunit.client.0.smithi016.stdout:1/601: write d3/f8 [440244,74500] 0 2024-04-22T15:49:35.933 INFO:tasks.workunit.client.0.smithi016.stdout:2/360: creat d2/de/f84 x:0 0 0 2024-04-22T15:49:35.934 INFO:tasks.workunit.client.0.smithi016.stdout:6/462: read - d3/da/d12/d13/f3f zero size 2024-04-22T15:49:35.939 INFO:tasks.workunit.client.0.smithi016.stdout:7/418: mknod d3/dc/d33/d84/c97 0 2024-04-22T15:49:35.939 INFO:tasks.workunit.client.0.smithi016.stdout:3/449: dwrite d7/f5a [8388608,4194304] 0 2024-04-22T15:49:35.943 INFO:tasks.workunit.client.0.smithi016.stdout:3/450: chown d7/d17/d18/d32/d42/l5d 3 1 2024-04-22T15:49:35.943 INFO:tasks.workunit.client.0.smithi016.stdout:2/361: dwrite d2/f4f [4194304,4194304] 0 2024-04-22T15:49:35.944 INFO:tasks.workunit.client.0.smithi016.stdout:8/390: rmdir d2/d10/d43/d4a/d28/d44/d62 0 2024-04-22T15:49:35.947 INFO:tasks.workunit.client.0.smithi016.stdout:6/463: write d3/f8 [5831747,29880] 0 2024-04-22T15:49:35.954 INFO:tasks.workunit.client.0.smithi016.stdout:1/602: creat d3/d4/d19/d20/d2b/fc8 x:0 0 0 2024-04-22T15:49:35.958 INFO:tasks.workunit.client.0.smithi016.stdout:3/451: creat d7/d1b/d21/d55/f8e x:0 0 0 2024-04-22T15:49:35.959 INFO:tasks.workunit.client.0.smithi016.stdout:2/362: symlink d2/d23/d47/l85 0 2024-04-22T15:49:35.961 INFO:tasks.workunit.client.0.smithi016.stdout:6/464: rmdir d3/da/db/d14 39 2024-04-22T15:49:35.962 INFO:tasks.workunit.client.0.smithi016.stdout:9/434: truncate f2 4138800 0 2024-04-22T15:49:35.963 INFO:tasks.workunit.client.0.smithi016.stdout:9/435: write d9/de/d23/d24/d4e/f79 [960399,94699] 0 2024-04-22T15:49:35.969 INFO:tasks.workunit.client.0.smithi016.stdout:9/436: dread d9/de/f16 [0,4194304] 0 2024-04-22T15:49:35.970 INFO:tasks.workunit.client.0.smithi016.stdout:2/363: creat d2/d23/d27/d67/f86 x:0 0 0 2024-04-22T15:49:35.971 INFO:tasks.workunit.client.0.smithi016.stdout:2/364: fsync d2/d4d/d64/f6e 0 2024-04-22T15:49:35.971 INFO:tasks.workunit.client.0.smithi016.stdout:3/452: dread d7/d17/d18/d27/f53 [0,4194304] 0 2024-04-22T15:49:35.975 INFO:tasks.workunit.client.0.smithi016.stdout:4/427: link d7/d21/d45/d49/f56 d7/d4e/d81/f92 0 2024-04-22T15:49:35.976 INFO:tasks.workunit.client.0.smithi016.stdout:5/382: link df/d15/d1a/d28/f3a df/d5b/d5d/d5f/f6f 0 2024-04-22T15:49:35.977 INFO:tasks.workunit.client.0.smithi016.stdout:9/437: rmdir d9/de/d23/d24 39 2024-04-22T15:49:35.980 INFO:tasks.workunit.client.0.smithi016.stdout:2/365: symlink d2/d4d/l87 0 2024-04-22T15:49:35.981 INFO:tasks.workunit.client.0.smithi016.stdout:3/453: mkdir d7/d17/d18/d32/d42/d63/d8f 0 2024-04-22T15:49:35.985 INFO:tasks.workunit.client.0.smithi016.stdout:4/428: mkdir d7/d4e/d2f/d50/d93 0 2024-04-22T15:49:35.987 INFO:tasks.workunit.client.0.smithi016.stdout:3/454: dwrite d7/d17/d18/f5b [4194304,4194304] 0 2024-04-22T15:49:35.987 INFO:tasks.workunit.client.0.smithi016.stdout:5/383: mkdir df/d5b/d70 0 2024-04-22T15:49:35.988 INFO:tasks.workunit.client.0.smithi016.stdout:9/438: mknod d9/d20/c82 0 2024-04-22T15:49:35.990 INFO:tasks.workunit.client.0.smithi016.stdout:2/366: mkdir d2/d23/d27/d67/d88 0 2024-04-22T15:49:35.995 INFO:tasks.workunit.client.0.smithi016.stdout:4/429: fdatasync d7/f16 0 2024-04-22T15:49:35.999 INFO:tasks.workunit.client.0.smithi016.stdout:4/430: dread d7/d4e/d2f/d6f/f84 [4194304,4194304] 0 2024-04-22T15:49:36.002 INFO:tasks.workunit.client.0.smithi016.stdout:0/441: truncate dc/d13/d28/d65/d57/f67 1294409 0 2024-04-22T15:49:36.002 INFO:tasks.workunit.client.0.smithi016.stdout:0/442: readlink dc/d13/d28/d65/d40/l6d 0 2024-04-22T15:49:36.002 INFO:tasks.workunit.client.0.smithi016.stdout:2/367: symlink d2/d23/d27/d34/d60/l89 0 2024-04-22T15:49:36.003 INFO:tasks.workunit.client.0.smithi016.stdout:2/368: readlink d2/de/d2a/l38 0 2024-04-22T15:49:36.004 INFO:tasks.workunit.client.0.smithi016.stdout:6/465: getdents d3/da/dd/d1b 0 2024-04-22T15:49:36.008 INFO:tasks.workunit.client.0.smithi016.stdout:0/443: dwrite dc/d13/f52 [0,4194304] 0 2024-04-22T15:49:36.009 INFO:tasks.workunit.client.0.smithi016.stdout:4/431: truncate d7/d21/d42/f55 6926 0 2024-04-22T15:49:36.014 INFO:tasks.workunit.client.0.smithi016.stdout:4/432: write d7/d4e/d2f/d50/f61 [678978,25699] 0 2024-04-22T15:49:36.014 INFO:tasks.workunit.client.0.smithi016.stdout:6/466: dwrite d3/da/d12/d13/f3f [0,4194304] 0 2024-04-22T15:49:36.016 INFO:tasks.workunit.client.0.smithi016.stdout:2/369: rmdir d2/d4d/d6d 39 2024-04-22T15:49:36.020 INFO:tasks.workunit.client.0.smithi016.stdout:5/384: creat df/d15/d1a/f71 x:0 0 0 2024-04-22T15:49:36.020 INFO:tasks.workunit.client.0.smithi016.stdout:3/455: link d7/d1b/d21/l2b d7/d17/d18/d32/d3c/d47/l90 0 2024-04-22T15:49:36.024 INFO:tasks.workunit.client.0.smithi016.stdout:7/419: dwrite d3/dc/d1b/d5a/d9/f2a [0,4194304] 0 2024-04-22T15:49:36.024 INFO:tasks.workunit.client.0.smithi016.stdout:3/456: dread d7/d17/d18/d32/d3c/d47/f48 [0,4194304] 0 2024-04-22T15:49:36.027 INFO:tasks.workunit.client.0.smithi016.stdout:0/444: creat dc/d13/d28/d65/d40/d4a/d59/f87 x:0 0 0 2024-04-22T15:49:36.027 INFO:tasks.workunit.client.0.smithi016.stdout:4/433: creat d7/d24/d41/f94 x:0 0 0 2024-04-22T15:49:36.028 INFO:tasks.workunit.client.0.smithi016.stdout:8/391: truncate d2/d30/f73 301917 0 2024-04-22T15:49:36.030 INFO:tasks.workunit.client.0.smithi016.stdout:6/467: unlink d3/da/dd/d67/f7c 0 2024-04-22T15:49:36.035 INFO:tasks.workunit.client.0.smithi016.stdout:2/370: rename d2/d4d/d64/f6e to d2/d23/d27/d67/d88/f8a 0 2024-04-22T15:49:36.037 INFO:tasks.workunit.client.0.smithi016.stdout:5/385: rmdir df/d5b 39 2024-04-22T15:49:36.038 INFO:tasks.workunit.client.0.smithi016.stdout:3/457: dread d7/fc [0,4194304] 0 2024-04-22T15:49:36.045 INFO:tasks.workunit.client.0.smithi016.stdout:0/445: unlink dc/d13/d14/d26/d32/f4e 0 2024-04-22T15:49:36.046 INFO:tasks.workunit.client.0.smithi016.stdout:4/434: symlink d7/d24/d41/l95 0 2024-04-22T15:49:36.049 INFO:tasks.workunit.client.0.smithi016.stdout:8/392: fdatasync d2/f27 0 2024-04-22T15:49:36.050 INFO:tasks.workunit.client.0.smithi016.stdout:6/468: symlink d3/d53/d35/l98 0 2024-04-22T15:49:36.055 INFO:tasks.workunit.client.0.smithi016.stdout:2/371: dread d2/d9/f12 [0,4194304] 0 2024-04-22T15:49:36.059 INFO:tasks.workunit.client.0.smithi016.stdout:3/458: mknod d7/d17/d18/d32/c91 0 2024-04-22T15:49:36.063 INFO:tasks.workunit.client.0.smithi016.stdout:0/446: mknod dc/d13/d28/d65/d40/d4a/d61/c88 0 2024-04-22T15:49:36.064 INFO:tasks.workunit.client.0.smithi016.stdout:0/447: chown dc/d13/d28/d65/d40/f76 200553972 1 2024-04-22T15:49:36.064 INFO:tasks.workunit.client.0.smithi016.stdout:0/448: chown c7 1866 1 2024-04-22T15:49:36.065 INFO:tasks.workunit.client.0.smithi016.stdout:0/449: write dc/d13/d28/d65/d40/f56 [5098368,48782] 0 2024-04-22T15:49:36.067 INFO:tasks.workunit.client.0.smithi016.stdout:5/386: dread df/d15/d1a/f20 [0,4194304] 0 2024-04-22T15:49:36.073 INFO:tasks.workunit.client.0.smithi016.stdout:5/387: dwrite df/f57 [0,4194304] 0 2024-04-22T15:49:36.075 INFO:tasks.workunit.client.0.smithi016.stdout:3/459: dread d7/d17/d18/f67 [0,4194304] 0 2024-04-22T15:49:36.078 INFO:tasks.workunit.client.0.smithi016.stdout:8/393: mkdir d2/d77 0 2024-04-22T15:49:36.078 INFO:tasks.workunit.client.0.smithi016.stdout:6/469: mknod d3/c99 0 2024-04-22T15:49:36.081 INFO:tasks.workunit.client.0.smithi016.stdout:0/450: mknod dc/d13/d22/c89 0 2024-04-22T15:49:36.081 INFO:tasks.workunit.client.0.smithi016.stdout:6/470: fsync d3/da/dd/f70 0 2024-04-22T15:49:36.085 INFO:tasks.workunit.client.0.smithi016.stdout:7/420: dwrite d3/d14/f2c [4194304,4194304] 0 2024-04-22T15:49:36.085 INFO:tasks.workunit.client.0.smithi016.stdout:5/388: creat df/d15/d1a/d4e/f72 x:0 0 0 2024-04-22T15:49:36.086 INFO:tasks.workunit.client.0.smithi016.stdout:4/435: rmdir d7/d24/d41 39 2024-04-22T15:49:36.087 INFO:tasks.workunit.client.0.smithi016.stdout:4/436: stat d7/d22/l87 0 2024-04-22T15:49:36.089 INFO:tasks.workunit.client.0.smithi016.stdout:8/394: rename d2/d10/d43/d4a/d6a/l75 to d2/d10/d43/d4a/l78 0 2024-04-22T15:49:36.090 INFO:tasks.workunit.client.0.smithi016.stdout:4/437: write d7/d4e/d2f/d50/f61 [1647838,80110] 0 2024-04-22T15:49:36.093 INFO:tasks.workunit.client.0.smithi016.stdout:0/451: mknod dc/d13/d28/d65/d1f/c8a 0 2024-04-22T15:49:36.099 INFO:tasks.workunit.client.0.smithi016.stdout:4/438: dwrite d7/d22/f64 [0,4194304] 0 2024-04-22T15:49:36.102 INFO:tasks.workunit.client.0.smithi016.stdout:2/372: getdents d2/d1a/d4c 0 2024-04-22T15:49:36.105 INFO:tasks.workunit.client.0.smithi016.stdout:8/395: mkdir d2/d10/d43/d4a/d28/d44/d66/d79 0 2024-04-22T15:49:36.107 INFO:tasks.workunit.client.0.smithi016.stdout:0/452: mkdir dc/d13/d6f/d8b 0 2024-04-22T15:49:36.107 INFO:tasks.workunit.client.0.smithi016.stdout:8/396: read d2/d10/d43/d4a/d28/f38 [2513470,13191] 0 2024-04-22T15:49:36.110 INFO:tasks.workunit.client.0.smithi016.stdout:5/389: dread df/d15/d1a/d28/f3f [0,4194304] 0 2024-04-22T15:49:36.111 INFO:tasks.workunit.client.0.smithi016.stdout:3/460: link d7/d17/d18/d32/d42/d63/l65 d7/d17/d18/d32/d42/d7e/l92 0 2024-04-22T15:49:36.114 INFO:tasks.workunit.client.0.smithi016.stdout:4/439: mkdir d7/d3c/d96 0 2024-04-22T15:49:36.116 INFO:tasks.workunit.client.0.smithi016.stdout:2/373: truncate f0 6498643 0 2024-04-22T15:49:36.117 INFO:tasks.workunit.client.0.smithi016.stdout:2/374: read - d2/d51/f62 zero size 2024-04-22T15:49:36.120 INFO:tasks.workunit.client.0.smithi016.stdout:0/453: mkdir dc/d13/d70/d8c 0 2024-04-22T15:49:36.121 INFO:tasks.workunit.client.0.smithi016.stdout:0/454: dread - dc/d13/d5d/f6c zero size 2024-04-22T15:49:36.121 INFO:tasks.workunit.client.0.smithi016.stdout:0/455: truncate dc/f68 1093690 0 2024-04-22T15:49:36.127 INFO:tasks.workunit.client.0.smithi016.stdout:7/421: link d3/dc/d1b/d5a/d9/df/l40 d3/d14/l98 0 2024-04-22T15:49:36.129 INFO:tasks.workunit.client.0.smithi016.stdout:6/471: dwrite d3/da/dd/f23 [0,4194304] 0 2024-04-22T15:49:36.129 INFO:tasks.workunit.client.0.smithi016.stdout:7/422: write d3/dc/d1b/d5a/f4b [3690943,7380] 0 2024-04-22T15:49:36.131 INFO:tasks.workunit.client.0.smithi016.stdout:7/423: chown d3/d14/c18 43 1 2024-04-22T15:49:36.132 INFO:tasks.workunit.client.0.smithi016.stdout:7/424: write d3/fa [11345996,9270] 0 2024-04-22T15:49:36.134 INFO:tasks.workunit.client.0.smithi016.stdout:0/456: creat dc/d13/d28/d65/d57/f8d x:0 0 0 2024-04-22T15:49:36.136 INFO:tasks.workunit.client.0.smithi016.stdout:8/397: dread d2/d10/d43/d4a/d28/d44/d66/f3c [0,4194304] 0 2024-04-22T15:49:36.137 INFO:tasks.workunit.client.0.smithi016.stdout:0/457: write dc/d13/d28/d65/d1f/f3a [106124,38557] 0 2024-04-22T15:49:36.138 INFO:tasks.workunit.client.0.smithi016.stdout:3/461: rename d7/lf to d7/d17/d18/l93 0 2024-04-22T15:49:36.144 INFO:tasks.workunit.client.0.smithi016.stdout:7/425: creat d3/f99 x:0 0 0 2024-04-22T15:49:36.144 INFO:tasks.workunit.client.0.smithi016.stdout:5/390: dread df/d15/d1a/f1e [0,4194304] 0 2024-04-22T15:49:36.145 INFO:tasks.workunit.client.0.smithi016.stdout:8/398: dwrite d2/d10/d43/d4a/d28/d44/d66/fc [4194304,4194304] 0 2024-04-22T15:49:36.146 INFO:tasks.workunit.client.0.smithi016.stdout:8/399: chown d2/d10/d43/d4a/d28/c2d 436082 1 2024-04-22T15:49:36.148 INFO:tasks.workunit.client.0.smithi016.stdout:0/458: mknod dc/d13/d28/d65/d40/d4a/d61/c8e 0 2024-04-22T15:49:36.149 INFO:tasks.workunit.client.0.smithi016.stdout:3/462: symlink d7/d1b/d21/d55/l94 0 2024-04-22T15:49:36.149 INFO:tasks.workunit.client.0.smithi016.stdout:2/375: link d2/d9/l36 d2/d1a/d4c/l8b 0 2024-04-22T15:49:36.150 INFO:tasks.workunit.client.0.smithi016.stdout:0/459: write dc/d13/d28/d65/d40/d4a/d59/f71 [809633,6331] 0 2024-04-22T15:49:36.155 INFO:tasks.workunit.client.0.smithi016.stdout:5/391: stat df/d15/f4a 0 2024-04-22T15:49:36.156 INFO:tasks.workunit.client.0.smithi016.stdout:2/376: dread d2/de/d2a/d59/f78 [0,4194304] 0 2024-04-22T15:49:36.159 INFO:tasks.workunit.client.0.smithi016.stdout:0/460: mknod dc/d13/d22/c8f 0 2024-04-22T15:49:36.165 INFO:tasks.workunit.client.0.smithi016.stdout:2/377: rename d2/d23/d27/d67/f7c to d2/d23/d27/f8c 0 2024-04-22T15:49:36.166 INFO:tasks.workunit.client.0.smithi016.stdout:2/378: chown d2/de/f84 6420 1 2024-04-22T15:49:36.166 INFO:tasks.workunit.client.0.smithi016.stdout:3/463: symlink d7/d17/d18/d32/d42/d63/d86/l95 0 2024-04-22T15:49:36.170 INFO:tasks.workunit.client.0.smithi016.stdout:5/392: mkdir df/d15/d73 0 2024-04-22T15:49:36.173 INFO:tasks.workunit.client.0.smithi016.stdout:3/464: chown d7/d1b/d21/l2b 456 1 2024-04-22T15:49:36.177 INFO:tasks.workunit.client.0.smithi016.stdout:6/472: getdents d3/da/db/d14/d16 0 2024-04-22T15:49:36.180 INFO:tasks.workunit.client.0.smithi016.stdout:2/379: creat d2/d1a/d7b/f8d x:0 0 0 2024-04-22T15:49:36.181 INFO:tasks.workunit.client.0.smithi016.stdout:3/465: mknod d7/d17/c96 0 2024-04-22T15:49:36.181 INFO:tasks.workunit.client.0.smithi016.stdout:3/466: readlink d7/l46 0 2024-04-22T15:49:36.184 INFO:tasks.workunit.client.0.smithi016.stdout:6/473: creat d3/da/db/d44/f9a x:0 0 0 2024-04-22T15:49:36.187 INFO:tasks.workunit.client.0.smithi016.stdout:8/400: truncate d2/d10/d43/d4a/d28/d52/f54 1439841 0 2024-04-22T15:49:36.188 INFO:tasks.workunit.client.0.smithi016.stdout:7/426: creat d3/dc/f9a x:0 0 0 2024-04-22T15:49:36.190 INFO:tasks.workunit.client.0.smithi016.stdout:0/461: write dc/d13/d14/f34 [2140671,83632] 0 2024-04-22T15:49:36.190 INFO:tasks.workunit.client.0.smithi016.stdout:7/427: fdatasync d3/dc/d1b/d5a/d9/df/f70 0 2024-04-22T15:49:36.191 INFO:tasks.workunit.client.0.smithi016.stdout:5/393: creat df/d15/d1a/d28/f74 x:0 0 0 2024-04-22T15:49:36.191 INFO:tasks.workunit.client.0.smithi016.stdout:2/380: mkdir d2/d4d/d8e 0 2024-04-22T15:49:36.193 INFO:tasks.workunit.client.0.smithi016.stdout:3/467: dwrite f1 [4194304,4194304] 0 2024-04-22T15:49:36.193 INFO:tasks.workunit.client.0.smithi016.stdout:5/394: truncate df/d22/f3b 1093450 0 2024-04-22T15:49:36.194 INFO:tasks.workunit.client.0.smithi016.stdout:2/381: chown d2/d23/d47/f66 1826 1 2024-04-22T15:49:36.194 INFO:tasks.workunit.client.0.smithi016.stdout:3/468: read - d7/d1b/d21/d55/f8e zero size 2024-04-22T15:49:36.195 INFO:tasks.workunit.client.0.smithi016.stdout:8/401: fdatasync d2/d30/f73 0 2024-04-22T15:49:36.195 INFO:tasks.workunit.client.0.smithi016.stdout:2/382: write d2/d2c/f5b [21771,68784] 0 2024-04-22T15:49:36.196 INFO:tasks.workunit.client.0.smithi016.stdout:0/462: symlink dc/d13/d28/d65/d30/d75/l90 0 2024-04-22T15:49:36.197 INFO:tasks.workunit.client.0.smithi016.stdout:6/474: creat d3/da/db/d14/f9b x:0 0 0 2024-04-22T15:49:36.197 INFO:tasks.workunit.client.0.smithi016.stdout:3/469: read - d7/d1b/d21/d55/f8e zero size 2024-04-22T15:49:36.202 INFO:tasks.workunit.client.0.smithi016.stdout:2/383: dread d2/d23/d27/f6a [0,4194304] 0 2024-04-22T15:49:36.205 INFO:tasks.workunit.client.0.smithi016.stdout:3/470: dwrite d7/d17/d18/f2a [0,4194304] 0 2024-04-22T15:49:36.213 INFO:tasks.workunit.client.0.smithi016.stdout:6/475: rename d3/da/d12/f1c to d3/da/d12/f9c 0 2024-04-22T15:49:36.217 INFO:tasks.workunit.client.0.smithi016.stdout:2/384: symlink d2/d1a/d4c/l8f 0 2024-04-22T15:49:36.220 INFO:tasks.workunit.client.0.smithi016.stdout:3/471: rmdir d7/d17/d18 39 2024-04-22T15:49:36.224 INFO:tasks.workunit.client.0.smithi016.stdout:2/385: dwrite d2/d23/f6c [0,4194304] 0 2024-04-22T15:49:36.226 INFO:tasks.workunit.client.0.smithi016.stdout:7/428: getdents d3/dc/d1b/d31 0 2024-04-22T15:49:36.226 INFO:tasks.workunit.client.0.smithi016.stdout:6/476: rmdir d3/da/dd/d7b 39 2024-04-22T15:49:36.227 INFO:tasks.workunit.client.0.smithi016.stdout:0/463: dwrite dc/f1e [0,4194304] 0 2024-04-22T15:49:36.231 INFO:tasks.workunit.client.0.smithi016.stdout:0/464: write dc/d13/d28/d65/d1f/d48/f64 [3813960,87316] 0 2024-04-22T15:49:36.232 INFO:tasks.workunit.client.0.smithi016.stdout:8/402: creat d2/f7a x:0 0 0 2024-04-22T15:49:36.234 INFO:tasks.workunit.client.0.smithi016.stdout:0/465: truncate dc/d13/d28/d65/d57/f8d 391881 0 2024-04-22T15:49:36.235 INFO:tasks.workunit.client.0.smithi016.stdout:0/466: stat dc/d13/d28/d65/d40/d4a/d61/c8e 0 2024-04-22T15:49:36.241 INFO:tasks.workunit.client.0.smithi016.stdout:5/395: getdents df/d15/d1a/d28 0 2024-04-22T15:49:36.241 INFO:tasks.workunit.client.0.smithi016.stdout:1/603: sync 2024-04-22T15:49:36.241 INFO:tasks.workunit.client.0.smithi016.stdout:9/439: sync 2024-04-22T15:49:36.242 INFO:tasks.workunit.client.0.smithi016.stdout:7/429: creat d3/dc/d1b/d5a/d59/d64/f9b x:0 0 0 2024-04-22T15:49:36.242 INFO:tasks.workunit.client.0.smithi016.stdout:7/430: stat d3/dc/d1b 0 2024-04-22T15:49:36.245 INFO:tasks.workunit.client.0.smithi016.stdout:6/477: write d3/da/dd/f5a [595465,72466] 0 2024-04-22T15:49:36.247 INFO:tasks.workunit.client.0.smithi016.stdout:0/467: rename c7 to dc/d13/d28/d65/d40/d4a/c91 0 2024-04-22T15:49:36.249 INFO:tasks.workunit.client.0.smithi016.stdout:2/386: symlink d2/d4d/d6d/l90 0 2024-04-22T15:49:36.303 INFO:tasks.workunit.client.0.smithi016.stdout:9/440: dread - d9/de/d23/d24/d4e/d4f/f63 zero size 2024-04-22T15:49:36.303 INFO:tasks.workunit.client.0.smithi016.stdout:7/431: write d3/fa [5043203,37937] 0 2024-04-22T15:49:36.304 INFO:tasks.workunit.client.0.smithi016.stdout:2/387: write d2/d4d/f7f [7897199,59479] 0 2024-04-22T15:49:36.306 INFO:tasks.workunit.client.0.smithi016.stdout:0/468: dwrite dc/d13/d28/d65/d30/d75/f80 [0,4194304] 0 2024-04-22T15:49:36.306 INFO:tasks.workunit.client.0.smithi016.stdout:9/441: read d9/d37/f3c [478666,8433] 0 2024-04-22T15:49:36.309 INFO:tasks.workunit.client.0.smithi016.stdout:8/403: creat d2/d10/d43/d4a/d28/f7b x:0 0 0 2024-04-22T15:49:36.313 INFO:tasks.workunit.client.0.smithi016.stdout:1/604: dwrite d3/d4/d3a/d6b/d77/f81 [0,4194304] 0 2024-04-22T15:49:36.314 INFO:tasks.workunit.client.0.smithi016.stdout:6/478: creat d3/da/dd/f9d x:0 0 0 2024-04-22T15:49:36.319 INFO:tasks.workunit.client.0.smithi016.stdout:9/442: creat d9/de/d23/d24/d4e/f83 x:0 0 0 2024-04-22T15:49:36.322 INFO:tasks.workunit.client.0.smithi016.stdout:1/605: truncate d3/d4/faa 734620 0 2024-04-22T15:49:36.323 INFO:tasks.workunit.client.0.smithi016.stdout:2/388: dwrite d2/d1a/d4c/f7a [0,4194304] 0 2024-04-22T15:49:36.324 INFO:tasks.workunit.client.0.smithi016.stdout:7/432: dwrite d3/dc/d1b/d5a/d9/f50 [0,4194304] 0 2024-04-22T15:49:36.331 INFO:tasks.workunit.client.0.smithi016.stdout:3/472: link d7/d17/d18/d32/d42/d63/l65 d7/d17/d18/d32/d3c/l97 0 2024-04-22T15:49:36.332 INFO:tasks.workunit.client.0.smithi016.stdout:3/473: write d7/d17/d18/d27/f4f [2633314,92917] 0 2024-04-22T15:49:36.332 INFO:tasks.workunit.client.0.smithi016.stdout:3/474: fdatasync d7/f5e 0 2024-04-22T15:49:36.334 INFO:tasks.workunit.client.0.smithi016.stdout:8/404: dread d2/d10/d43/d4a/d28/d44/f53 [0,4194304] 0 2024-04-22T15:49:36.337 INFO:tasks.workunit.client.0.smithi016.stdout:9/443: symlink d9/de/d26/d5a/l84 0 2024-04-22T15:49:36.337 INFO:tasks.workunit.client.0.smithi016.stdout:9/444: readlink d9/d20/l6d 0 2024-04-22T15:49:36.337 INFO:tasks.workunit.client.0.smithi016.stdout:3/475: dread d7/d17/d18/d32/d3c/f66 [4194304,4194304] 0 2024-04-22T15:49:36.339 INFO:tasks.workunit.client.0.smithi016.stdout:3/476: read d7/f5a [5786472,19182] 0 2024-04-22T15:49:36.341 INFO:tasks.workunit.client.0.smithi016.stdout:4/440: sync 2024-04-22T15:49:36.344 INFO:tasks.workunit.client.0.smithi016.stdout:2/389: rename d2/d9/f16 to d2/d4d/d8e/f91 0 2024-04-22T15:49:36.347 INFO:tasks.workunit.client.0.smithi016.stdout:6/479: mkdir d3/da/db/d44/d68/d95/d9e 0 2024-04-22T15:49:36.352 INFO:tasks.workunit.client.0.smithi016.stdout:0/469: getdents dc/d13/d14/d55 0 2024-04-22T15:49:36.353 INFO:tasks.workunit.client.0.smithi016.stdout:0/470: dread - dc/d13/d28/d65/d40/d4a/d59/d78/f86 zero size 2024-04-22T15:49:36.354 INFO:tasks.workunit.client.0.smithi016.stdout:0/471: chown dc/d13/d28/d65/d57/l58 219616 1 2024-04-22T15:49:36.357 INFO:tasks.workunit.client.0.smithi016.stdout:1/606: creat d3/d4/d3a/d9e/dbf/fc9 x:0 0 0 2024-04-22T15:49:36.359 INFO:tasks.workunit.client.0.smithi016.stdout:4/441: stat d7/d4e/c2e 0 2024-04-22T15:49:36.363 INFO:tasks.workunit.client.0.smithi016.stdout:2/390: rename d2/d23/d47 to d2/d23/d92 0 2024-04-22T15:49:36.365 INFO:tasks.workunit.client.0.smithi016.stdout:6/480: rmdir d3/da/dd 39 2024-04-22T15:49:36.369 INFO:tasks.workunit.client.0.smithi016.stdout:8/405: symlink d2/d10/l7c 0 2024-04-22T15:49:36.374 INFO:tasks.workunit.client.0.smithi016.stdout:3/477: getdents d7/d17/d18/d32/d8a 0 2024-04-22T15:49:36.376 INFO:tasks.workunit.client.0.smithi016.stdout:1/607: rmdir d3/d2e 39 2024-04-22T15:49:36.381 INFO:tasks.workunit.client.0.smithi016.stdout:2/391: rmdir d2 39 2024-04-22T15:49:36.382 INFO:tasks.workunit.client.0.smithi016.stdout:9/445: dwrite d9/d37/f4a [0,4194304] 0 2024-04-22T15:49:36.385 INFO:tasks.workunit.client.0.smithi016.stdout:8/406: chown d2/d10/d43/d4a/d28/d3e/c26 1 1 2024-04-22T15:49:36.388 INFO:tasks.workunit.client.0.smithi016.stdout:7/433: link d3/d14/f2c d3/dc/d1b/d5a/d9/f9c 0 2024-04-22T15:49:36.391 INFO:tasks.workunit.client.0.smithi016.stdout:0/472: write dc/d13/f31 [1415984,94437] 0 2024-04-22T15:49:36.394 INFO:tasks.workunit.client.0.smithi016.stdout:6/481: creat d3/d53/d83/f9f x:0 0 0 2024-04-22T15:49:36.395 INFO:tasks.workunit.client.0.smithi016.stdout:4/442: truncate d7/d4e/d2f/d50/f61 791089 0 2024-04-22T15:49:36.397 INFO:tasks.workunit.client.0.smithi016.stdout:3/478: unlink d7/d17/d18/d32/d42/d63/l65 0 2024-04-22T15:49:36.399 INFO:tasks.workunit.client.0.smithi016.stdout:3/479: fsync d7/d17/f2e 0 2024-04-22T15:49:36.399 INFO:tasks.workunit.client.0.smithi016.stdout:8/407: rename d2/d10/d43/d4a/d28/d44/d66/l21 to d2/d10/d43/d4a/d28/d44/d66/d1c/d65/l7d 0 2024-04-22T15:49:36.399 INFO:tasks.workunit.client.0.smithi016.stdout:1/608: mknod d3/d4/d3a/d91/db3/cca 0 2024-04-22T15:49:36.400 INFO:tasks.workunit.client.0.smithi016.stdout:0/473: dwrite dc/d13/d28/d65/d40/d4a/d59/f87 [0,4194304] 0 2024-04-22T15:49:36.402 INFO:tasks.workunit.client.0.smithi016.stdout:1/609: write d3/d84/d97/fb4 [915757,65257] 0 2024-04-22T15:49:36.402 INFO:tasks.workunit.client.0.smithi016.stdout:0/474: write dc/d13/d14/f18 [3891150,7261] 0 2024-04-22T15:49:36.405 INFO:tasks.workunit.client.0.smithi016.stdout:6/482: unlink d3/da/db/d44/c4d 0 2024-04-22T15:49:36.405 INFO:tasks.workunit.client.0.smithi016.stdout:1/610: readlink d3/d4/d3a/d91/lc6 0 2024-04-22T15:49:36.406 INFO:tasks.workunit.client.0.smithi016.stdout:0/475: dread - dc/d13/d28/d65/f63 zero size 2024-04-22T15:49:36.406 INFO:tasks.workunit.client.0.smithi016.stdout:7/434: dread d3/dc/d1b/d5a/d4a/f55 [0,4194304] 0 2024-04-22T15:49:36.407 INFO:tasks.workunit.client.0.smithi016.stdout:1/611: stat d3/d4/d3a/d91/lc6 0 2024-04-22T15:49:36.409 INFO:tasks.workunit.client.0.smithi016.stdout:7/435: truncate d3/dc/d1b/d31/d5f/f63 109147 0 2024-04-22T15:49:36.414 INFO:tasks.workunit.client.0.smithi016.stdout:0/476: dwrite dc/d13/d14/f34 [0,4194304] 0 2024-04-22T15:49:36.415 INFO:tasks.workunit.client.0.smithi016.stdout:0/477: readlink dc/d13/d22/l2b 0 2024-04-22T15:49:36.415 INFO:tasks.workunit.client.0.smithi016.stdout:1/612: dwrite d3/d4/d15/fc0 [0,4194304] 0 2024-04-22T15:49:36.416 INFO:tasks.workunit.client.0.smithi016.stdout:9/446: truncate d9/d20/f56 454885 0 2024-04-22T15:49:36.417 INFO:tasks.workunit.client.0.smithi016.stdout:9/447: stat d9/de/d23/d24/f74 0 2024-04-22T15:49:36.418 INFO:tasks.workunit.client.0.smithi016.stdout:9/448: dread - d9/f1c zero size 2024-04-22T15:49:36.418 INFO:tasks.workunit.client.0.smithi016.stdout:3/480: unlink d7/d17/c96 0 2024-04-22T15:49:36.420 INFO:tasks.workunit.client.0.smithi016.stdout:3/481: write d7/d1b/d21/d55/f8c [407294,127327] 0 2024-04-22T15:49:36.425 INFO:tasks.workunit.client.0.smithi016.stdout:5/396: sync 2024-04-22T15:49:36.426 INFO:tasks.workunit.client.0.smithi016.stdout:8/408: mkdir d2/d10/d43/d4a/d28/d44/d66/d7e 0 2024-04-22T15:49:36.439 INFO:tasks.workunit.client.0.smithi016.stdout:1/613: unlink d3/d4/d37/f72 0 2024-04-22T15:49:36.442 INFO:tasks.workunit.client.0.smithi016.stdout:2/392: creat d2/de/d2a/f93 x:0 0 0 2024-04-22T15:49:36.444 INFO:tasks.workunit.client.0.smithi016.stdout:5/397: unlink la 0 2024-04-22T15:49:36.446 INFO:tasks.workunit.client.0.smithi016.stdout:4/443: link d7/d4e/d39/f48 d7/d4e/d2f/d6f/f97 0 2024-04-22T15:49:36.447 INFO:tasks.workunit.client.0.smithi016.stdout:4/444: chown d7/d3c/d96 78527591 1 2024-04-22T15:49:36.450 INFO:tasks.workunit.client.0.smithi016.stdout:9/449: dread d9/de/d26/f2a [0,4194304] 0 2024-04-22T15:49:36.452 INFO:tasks.workunit.client.0.smithi016.stdout:1/614: mknod d3/d4/d15/d38/d63/ccb 0 2024-04-22T15:49:36.454 INFO:tasks.workunit.client.0.smithi016.stdout:0/478: dread dc/d13/d28/f50 [0,4194304] 0 2024-04-22T15:49:36.454 INFO:tasks.workunit.client.0.smithi016.stdout:0/479: readlink dc/l62 0 2024-04-22T15:49:36.456 INFO:tasks.workunit.client.0.smithi016.stdout:6/483: dread d3/da/f45 [0,4194304] 0 2024-04-22T15:49:36.456 INFO:tasks.workunit.client.0.smithi016.stdout:2/393: rmdir d2/d1a 39 2024-04-22T15:49:36.457 INFO:tasks.workunit.client.0.smithi016.stdout:6/484: readlink d3/da/d12/d13/l5e 0 2024-04-22T15:49:36.457 INFO:tasks.workunit.client.0.smithi016.stdout:6/485: readlink d3/l92 0 2024-04-22T15:49:36.458 INFO:tasks.workunit.client.0.smithi016.stdout:6/486: chown d3/da/db/d14/d16/l6b 108 1 2024-04-22T15:49:36.459 INFO:tasks.workunit.client.0.smithi016.stdout:6/487: dread - d3/da/db/d44/f9a zero size 2024-04-22T15:49:36.459 INFO:tasks.workunit.client.0.smithi016.stdout:8/409: rename d2/d10/d43/d4a/l78 to d2/d10/d43/d4a/d28/d44/d66/d7e/l7f 0 2024-04-22T15:49:36.461 INFO:tasks.workunit.client.0.smithi016.stdout:8/410: dread - d2/f7a zero size 2024-04-22T15:49:36.464 INFO:tasks.workunit.client.0.smithi016.stdout:7/436: dwrite d3/dc/d1b/d5a/d9/f9c [4194304,4194304] 0 2024-04-22T15:49:36.466 INFO:tasks.workunit.client.0.smithi016.stdout:9/450: mknod d9/de/d23/d24/d4e/d4f/c85 0 2024-04-22T15:49:36.468 INFO:tasks.workunit.client.0.smithi016.stdout:1/615: creat d3/d4/d3a/d6b/fcc x:0 0 0 2024-04-22T15:49:36.470 INFO:tasks.workunit.client.0.smithi016.stdout:1/616: fdatasync d3/d4/d3a/d6b/fcc 0 2024-04-22T15:49:36.474 INFO:tasks.workunit.client.0.smithi016.stdout:1/617: chown d3/d4/d15/d38/d63/d67/f79 3851 1 2024-04-22T15:49:36.474 INFO:tasks.workunit.client.0.smithi016.stdout:9/451: dwrite d9/d20/f59 [0,4194304] 0 2024-04-22T15:49:36.476 INFO:tasks.workunit.client.0.smithi016.stdout:1/618: chown d3/d4/d15/l7d 25815 1 2024-04-22T15:49:36.476 INFO:tasks.workunit.client.0.smithi016.stdout:2/394: truncate d2/de/d2a/d59/f78 341533 0 2024-04-22T15:49:36.477 INFO:tasks.workunit.client.0.smithi016.stdout:0/480: dwrite dc/d13/d28/d65/f42 [4194304,4194304] 0 2024-04-22T15:49:36.478 INFO:tasks.workunit.client.0.smithi016.stdout:6/488: unlink d3/da/db/d14/d4c/c7d 0 2024-04-22T15:49:36.483 INFO:tasks.workunit.client.0.smithi016.stdout:3/482: dwrite d7/d1b/d21/d55/f79 [0,4194304] 0 2024-04-22T15:49:36.486 INFO:tasks.workunit.client.0.smithi016.stdout:5/398: truncate df/d15/d1a/d28/f2b 2844996 0 2024-04-22T15:49:36.490 INFO:tasks.workunit.client.0.smithi016.stdout:8/411: creat d2/d10/d43/d4a/d28/d56/f80 x:0 0 0 2024-04-22T15:49:36.493 INFO:tasks.workunit.client.0.smithi016.stdout:4/445: dwrite d7/d4e/f29 [0,4194304] 0 2024-04-22T15:49:36.495 INFO:tasks.workunit.client.0.smithi016.stdout:7/437: creat d3/dc/d1b/d5a/d69/f9d x:0 0 0 2024-04-22T15:49:36.501 INFO:tasks.workunit.client.0.smithi016.stdout:9/452: creat d9/de/d23/d30/f86 x:0 0 0 2024-04-22T15:49:36.502 INFO:tasks.workunit.client.0.smithi016.stdout:9/453: write d9/d20/f38 [363288,45880] 0 2024-04-22T15:49:36.503 INFO:tasks.workunit.client.0.smithi016.stdout:9/454: dread - d9/d37/f64 zero size 2024-04-22T15:49:36.505 INFO:tasks.workunit.client.0.smithi016.stdout:1/619: creat d3/d84/fcd x:0 0 0 2024-04-22T15:49:36.510 INFO:tasks.workunit.client.0.smithi016.stdout:9/455: dwrite d9/d5d/d62/f6e [4194304,4194304] 0 2024-04-22T15:49:36.511 INFO:tasks.workunit.client.0.smithi016.stdout:0/481: mkdir dc/d13/d28/d65/d40/d4a/d92 0 2024-04-22T15:49:36.516 INFO:tasks.workunit.client.0.smithi016.stdout:9/456: dwrite d9/de/f28 [0,4194304] 0 2024-04-22T15:49:36.517 INFO:tasks.workunit.client.0.smithi016.stdout:9/457: dread - d9/de/d23/d30/f58 zero size 2024-04-22T15:49:36.527 INFO:tasks.workunit.client.0.smithi016.stdout:8/412: truncate d2/d10/d43/d4a/d28/d44/d66/d1c/f3d 1055855 0 2024-04-22T15:49:36.534 INFO:tasks.workunit.client.0.smithi016.stdout:1/620: creat d3/d4/d15/fce x:0 0 0 2024-04-22T15:49:36.537 INFO:tasks.workunit.client.0.smithi016.stdout:6/489: mknod d3/da/dd/d67/ca0 0 2024-04-22T15:49:36.541 INFO:tasks.workunit.client.0.smithi016.stdout:0/482: creat dc/d13/d14/d55/f93 x:0 0 0 2024-04-22T15:49:36.548 INFO:tasks.workunit.client.0.smithi016.stdout:9/458: fdatasync d9/f3b 0 2024-04-22T15:49:36.553 INFO:tasks.workunit.client.0.smithi016.stdout:4/446: symlink d7/d24/l98 0 2024-04-22T15:49:36.556 INFO:tasks.workunit.client.0.smithi016.stdout:7/438: creat d3/dc/d1b/d5a/d9/df/d94/d4c/d7a/f9e x:0 0 0 2024-04-22T15:49:36.562 INFO:tasks.workunit.client.0.smithi016.stdout:1/621: unlink d3/d28/l65 0 2024-04-22T15:49:36.569 INFO:tasks.workunit.client.0.smithi016.stdout:3/483: mkdir d7/d17/d18/d32/d3c/d51/d98 0 2024-04-22T15:49:36.570 INFO:tasks.workunit.client.0.smithi016.stdout:9/459: rmdir d9/de/d23/d24 39 2024-04-22T15:49:36.572 INFO:tasks.workunit.client.0.smithi016.stdout:8/413: dwrite d2/d10/d43/d4a/d28/f4d [4194304,4194304] 0 2024-04-22T15:49:36.572 INFO:tasks.workunit.client.0.smithi016.stdout:5/399: rmdir df/d15/d73 0 2024-04-22T15:49:36.573 INFO:tasks.workunit.client.0.smithi016.stdout:5/400: fdatasync df/d22/f24 0 2024-04-22T15:49:36.576 INFO:tasks.workunit.client.0.smithi016.stdout:2/395: getdents d2/d23/d27/d67 0 2024-04-22T15:49:36.577 INFO:tasks.workunit.client.0.smithi016.stdout:6/490: creat d3/da/dd/d1b/d57/fa1 x:0 0 0 2024-04-22T15:49:36.578 INFO:tasks.workunit.client.0.smithi016.stdout:0/483: symlink dc/d13/d14/d26/l94 0 2024-04-22T15:49:36.590 INFO:tasks.workunit.client.0.smithi016.stdout:4/447: unlink d7/d21/c2b 0 2024-04-22T15:49:36.591 INFO:tasks.workunit.client.0.smithi016.stdout:6/491: dread d3/da/dd/f51 [0,4194304] 0 2024-04-22T15:49:36.592 INFO:tasks.workunit.client.0.smithi016.stdout:6/492: fsync d3/da/dd/f18 0 2024-04-22T15:49:36.593 INFO:tasks.workunit.client.0.smithi016.stdout:0/484: dread dc/d13/d14/f18 [0,4194304] 0 2024-04-22T15:49:36.596 INFO:tasks.workunit.client.0.smithi016.stdout:0/485: dread dc/d13/d28/d65/d40/d4a/d59/f87 [0,4194304] 0 2024-04-22T15:49:36.600 INFO:tasks.workunit.client.0.smithi016.stdout:5/401: mkdir df/d15/d1a/d4d/d75 0 2024-04-22T15:49:36.602 INFO:tasks.workunit.client.0.smithi016.stdout:4/448: unlink d7/d4e/d2f/d33/l34 0 2024-04-22T15:49:36.604 INFO:tasks.workunit.client.0.smithi016.stdout:1/622: rename d3/d4/d37/la4 to d3/d4/d15/d38/lcf 0 2024-04-22T15:49:36.606 INFO:tasks.workunit.client.0.smithi016.stdout:2/396: symlink d2/d4d/d76/l94 0 2024-04-22T15:49:36.608 INFO:tasks.workunit.client.0.smithi016.stdout:6/493: unlink d3/da/db/l1f 0 2024-04-22T15:49:36.608 INFO:tasks.workunit.client.0.smithi016.stdout:8/414: write d2/d30/f60 [1552192,105671] 0 2024-04-22T15:49:36.608 INFO:tasks.workunit.client.0.smithi016.stdout:6/494: fdatasync d3/da/f3e 0 2024-04-22T15:49:36.609 INFO:tasks.workunit.client.0.smithi016.stdout:8/415: write d2/f5a [108261,17507] 0 2024-04-22T15:49:36.610 INFO:tasks.workunit.client.0.smithi016.stdout:7/439: truncate d3/dc/d1b/d5a/f17 4097548 0 2024-04-22T15:49:36.617 INFO:tasks.workunit.client.0.smithi016.stdout:0/486: write dc/d13/d14/f18 [1974267,116974] 0 2024-04-22T15:49:36.621 INFO:tasks.workunit.client.0.smithi016.stdout:4/449: creat d7/d3c/f99 x:0 0 0 2024-04-22T15:49:36.622 INFO:tasks.workunit.client.0.smithi016.stdout:0/487: dwrite dc/d13/d28/d65/d57/f8d [0,4194304] 0 2024-04-22T15:49:36.623 INFO:tasks.workunit.client.0.smithi016.stdout:3/484: rename d7/l46 to d7/d17/d18/d32/d42/d63/d86/l99 0 2024-04-22T15:49:36.625 INFO:tasks.workunit.client.0.smithi016.stdout:1/623: creat d3/d4/d15/d38/d63/d67/fd0 x:0 0 0 2024-04-22T15:49:36.633 INFO:tasks.workunit.client.0.smithi016.stdout:6/495: creat d3/d53/d35/fa2 x:0 0 0 2024-04-22T15:49:36.639 INFO:tasks.workunit.client.0.smithi016.stdout:8/416: creat d2/d10/d43/d4a/d28/d44/d66/d1c/f81 x:0 0 0 2024-04-22T15:49:36.648 INFO:tasks.workunit.client.0.smithi016.stdout:3/485: creat d7/d17/d18/d32/d42/f9a x:0 0 0 2024-04-22T15:49:36.650 INFO:tasks.workunit.client.0.smithi016.stdout:1/624: creat d3/d4/d37/fd1 x:0 0 0 2024-04-22T15:49:36.652 INFO:tasks.workunit.client.0.smithi016.stdout:2/397: link d2/de/f5c d2/d23/d27/d67/d88/f95 0 2024-04-22T15:49:36.656 INFO:tasks.workunit.client.0.smithi016.stdout:4/450: symlink d7/d3c/l9a 0 2024-04-22T15:49:36.659 INFO:tasks.workunit.client.0.smithi016.stdout:6/496: creat d3/da/d12/d13/d46/fa3 x:0 0 0 2024-04-22T15:49:36.660 INFO:tasks.workunit.client.0.smithi016.stdout:7/440: mkdir d3/dc/d9f 0 2024-04-22T15:49:36.663 INFO:tasks.workunit.client.0.smithi016.stdout:9/460: getdents d9/de/d23/d24 0 2024-04-22T15:49:36.664 INFO:tasks.workunit.client.0.smithi016.stdout:5/402: creat df/f76 x:0 0 0 2024-04-22T15:49:36.664 INFO:tasks.workunit.client.0.smithi016.stdout:9/461: chown d9/d37/f49 69281985 1 2024-04-22T15:49:36.664 INFO:tasks.workunit.client.0.smithi016.stdout:5/403: fdatasync df/f57 0 2024-04-22T15:49:36.665 INFO:tasks.workunit.client.0.smithi016.stdout:0/488: symlink dc/d13/d14/d26/l95 0 2024-04-22T15:49:36.668 INFO:tasks.workunit.client.0.smithi016.stdout:9/462: fsync d9/de/d26/d5a/f73 0 2024-04-22T15:49:36.669 INFO:tasks.workunit.client.0.smithi016.stdout:1/625: mkdir d3/d4/d3a/d91/db3/dd2 0 2024-04-22T15:49:36.670 INFO:tasks.workunit.client.0.smithi016.stdout:8/417: dwrite d2/d10/d43/d4a/d6a/f74 [0,4194304] 0 2024-04-22T15:49:36.671 INFO:tasks.workunit.client.0.smithi016.stdout:2/398: mknod d2/d23/d27/d67/d88/c96 0 2024-04-22T15:49:36.673 INFO:tasks.workunit.client.0.smithi016.stdout:9/463: write d9/d37/f71 [2828714,90190] 0 2024-04-22T15:49:36.674 INFO:tasks.workunit.client.0.smithi016.stdout:5/404: truncate df/d15/d1a/d28/d33/f4c 522755 0 2024-04-22T15:49:36.675 INFO:tasks.workunit.client.0.smithi016.stdout:6/497: write d3/da/dd/f29 [524120,63342] 0 2024-04-22T15:49:36.678 INFO:tasks.workunit.client.0.smithi016.stdout:2/399: chown d2/d4d/d8e/f91 92846507 1 2024-04-22T15:49:36.678 INFO:tasks.workunit.client.0.smithi016.stdout:0/489: truncate dc/d13/d28/d65/d40/d4a/d59/f71 1458061 0 2024-04-22T15:49:36.679 INFO:tasks.workunit.client.0.smithi016.stdout:6/498: write d3/da/dd/f82 [665134,48105] 0 2024-04-22T15:49:36.681 INFO:tasks.workunit.client.0.smithi016.stdout:0/490: truncate dc/d13/d5d/f6c 827757 0 2024-04-22T15:49:36.681 INFO:tasks.workunit.client.0.smithi016.stdout:6/499: chown d3/d53/d35/l98 0 1 2024-04-22T15:49:36.681 INFO:tasks.workunit.client.0.smithi016.stdout:0/491: stat dc/d13/l2a 0 2024-04-22T15:49:36.681 INFO:tasks.workunit.client.0.smithi016.stdout:2/400: read d2/d23/d27/f31 [2835594,103404] 0 2024-04-22T15:49:36.682 INFO:tasks.workunit.client.0.smithi016.stdout:3/486: creat d7/d17/d18/d32/d8a/f9b x:0 0 0 2024-04-22T15:49:36.682 INFO:tasks.workunit.client.0.smithi016.stdout:6/500: write d3/da/db/d44/f81 [375185,67239] 0 2024-04-22T15:49:36.683 INFO:tasks.workunit.client.0.smithi016.stdout:0/492: chown dc/d13/d28/d65/d30/f84 904008723 1 2024-04-22T15:49:36.684 INFO:tasks.workunit.client.0.smithi016.stdout:8/418: readlink d2/d10/d43/d4a/l57 0 2024-04-22T15:49:36.687 INFO:tasks.workunit.client.0.smithi016.stdout:4/451: truncate d7/d4e/d81/f92 1280255 0 2024-04-22T15:49:36.691 INFO:tasks.workunit.client.0.smithi016.stdout:5/405: chown df/d15/d1a/l1b 0 1 2024-04-22T15:49:36.693 INFO:tasks.workunit.client.0.smithi016.stdout:0/493: dwrite dc/d13/d70/f72 [0,4194304] 0 2024-04-22T15:49:36.696 INFO:tasks.workunit.client.0.smithi016.stdout:2/401: rename d2/d23/d27 to d2/d2c/d97 0 2024-04-22T15:49:36.698 INFO:tasks.workunit.client.0.smithi016.stdout:3/487: creat d7/d17/d18/d32/d8a/f9c x:0 0 0 2024-04-22T15:49:36.698 INFO:tasks.workunit.client.0.smithi016.stdout:2/402: read - d2/d2c/d97/d34/d60/f83 zero size 2024-04-22T15:49:36.699 INFO:tasks.workunit.client.0.smithi016.stdout:1/626: symlink d3/d4/d19/d20/d95/ld3 0 2024-04-22T15:49:36.702 INFO:tasks.workunit.client.0.smithi016.stdout:1/627: dread d3/d4/d3a/d6b/f94 [0,4194304] 0 2024-04-22T15:49:36.703 INFO:tasks.workunit.client.0.smithi016.stdout:8/419: dread d2/d10/f41 [0,4194304] 0 2024-04-22T15:49:36.704 INFO:tasks.workunit.client.0.smithi016.stdout:0/494: mkdir dc/d13/d28/d65/d30/d96 0 2024-04-22T15:49:36.709 INFO:tasks.workunit.client.0.smithi016.stdout:5/406: rename df/d15/d1a/d28/d33 to df/d15/d1a/d50/d77 0 2024-04-22T15:49:36.710 INFO:tasks.workunit.client.0.smithi016.stdout:3/488: creat d7/d17/d18/d32/d42/d63/f9d x:0 0 0 2024-04-22T15:49:36.714 INFO:tasks.workunit.client.0.smithi016.stdout:9/464: write d9/de/d23/d24/f39 [3157532,75737] 0 2024-04-22T15:49:36.715 INFO:tasks.workunit.client.0.smithi016.stdout:9/465: read d9/f1b [3316566,20597] 0 2024-04-22T15:49:36.717 INFO:tasks.workunit.client.0.smithi016.stdout:4/452: mkdir d7/d21/d45/d49/d9b 0 2024-04-22T15:49:36.718 INFO:tasks.workunit.client.0.smithi016.stdout:1/628: creat d3/d4/d3a/d91/db3/fd4 x:0 0 0 2024-04-22T15:49:36.723 INFO:tasks.workunit.client.0.smithi016.stdout:8/420: dread d2/d10/d43/d4a/d28/d44/d66/d1c/f3d [0,4194304] 0 2024-04-22T15:49:36.726 INFO:tasks.workunit.client.0.smithi016.stdout:0/495: dread dc/d13/d28/d65/d30/f44 [0,4194304] 0 2024-04-22T15:49:36.729 INFO:tasks.workunit.client.0.smithi016.stdout:8/421: dwrite d2/d10/d43/d4a/d28/d44/d66/f20 [4194304,4194304] 0 2024-04-22T15:49:36.733 INFO:tasks.workunit.client.0.smithi016.stdout:3/489: rename d7/c61 to d7/d17/d18/d27/c9e 0 2024-04-22T15:49:36.734 INFO:tasks.workunit.client.0.smithi016.stdout:2/403: symlink d2/d4d/l98 0 2024-04-22T15:49:36.735 INFO:tasks.workunit.client.0.smithi016.stdout:3/490: write d7/d17/d18/f4e [185165,49406] 0 2024-04-22T15:49:36.738 INFO:tasks.workunit.client.0.smithi016.stdout:9/466: mkdir d9/de/d23/d24/d25/d36/d87 0 2024-04-22T15:49:36.741 INFO:tasks.workunit.client.0.smithi016.stdout:1/629: readlink d3/d4/d3a/l78 0 2024-04-22T15:49:36.741 INFO:tasks.workunit.client.0.smithi016.stdout:0/496: mkdir dc/d13/d28/d65/d57/d97 0 2024-04-22T15:49:36.743 INFO:tasks.workunit.client.0.smithi016.stdout:6/501: link d3/da/dd/d67/c8a d3/da/db/d14/ca4 0 2024-04-22T15:49:36.745 INFO:tasks.workunit.client.0.smithi016.stdout:0/497: chown dc/d13/d28/d65/d57/f8d 118464212 1 2024-04-22T15:49:36.746 INFO:tasks.workunit.client.0.smithi016.stdout:5/407: creat df/d15/d1a/f78 x:0 0 0 2024-04-22T15:49:36.747 INFO:tasks.workunit.client.0.smithi016.stdout:1/630: mknod d3/d4/d15/d8c/cd5 0 2024-04-22T15:49:36.747 INFO:tasks.workunit.client.0.smithi016.stdout:2/404: mknod d2/de/c99 0 2024-04-22T15:49:36.747 INFO:tasks.workunit.client.0.smithi016.stdout:4/453: dread d7/d24/d41/f5d [0,4194304] 0 2024-04-22T15:49:36.749 INFO:tasks.workunit.client.0.smithi016.stdout:9/467: dwrite d9/de/f28 [0,4194304] 0 2024-04-22T15:49:36.750 INFO:tasks.workunit.client.0.smithi016.stdout:8/422: rename d2/d10/d43/d4a/d28 to d2/d82 0 2024-04-22T15:49:36.756 INFO:tasks.workunit.client.0.smithi016.stdout:9/468: chown d9/de/l6a 156 1 2024-04-22T15:49:36.757 INFO:tasks.workunit.client.0.smithi016.stdout:0/498: mkdir dc/d13/d28/d65/d40/d4a/d59/d98 0 2024-04-22T15:49:36.757 INFO:tasks.workunit.client.0.smithi016.stdout:8/423: chown d2/d82/d56/f80 28 1 2024-04-22T15:49:36.761 INFO:tasks.workunit.client.0.smithi016.stdout:4/454: dread - d7/d3c/f99 zero size 2024-04-22T15:49:36.764 INFO:tasks.workunit.client.0.smithi016.stdout:6/502: dwrite d3/da/dd/f18 [0,4194304] 0 2024-04-22T15:49:36.767 INFO:tasks.workunit.client.0.smithi016.stdout:0/499: read dc/d13/f17 [7652277,41804] 0 2024-04-22T15:49:36.769 INFO:tasks.workunit.client.0.smithi016.stdout:1/631: dwrite d3/d4/d3a/d6b/fc1 [0,4194304] 0 2024-04-22T15:49:36.773 INFO:tasks.workunit.client.0.smithi016.stdout:1/632: chown d3/d4/d45/cad 1 1 2024-04-22T15:49:36.773 INFO:tasks.workunit.client.0.smithi016.stdout:3/491: dread d7/f5e [0,4194304] 0 2024-04-22T15:49:36.776 INFO:tasks.workunit.client.0.smithi016.stdout:2/405: dread d2/d2c/f5b [0,4194304] 0 2024-04-22T15:49:36.777 INFO:tasks.workunit.client.0.smithi016.stdout:8/424: dread d2/d30/f60 [0,4194304] 0 2024-04-22T15:49:36.777 INFO:tasks.workunit.client.0.smithi016.stdout:0/500: mkdir dc/d13/d99 0 2024-04-22T15:49:36.779 INFO:tasks.workunit.client.0.smithi016.stdout:5/408: rename df/d15/d1a/d4d/l52 to df/d15/d1a/d28/l79 0 2024-04-22T15:49:36.782 INFO:tasks.workunit.client.0.smithi016.stdout:4/455: creat d7/d21/d42/d59/d88/f9c x:0 0 0 2024-04-22T15:49:36.784 INFO:tasks.workunit.client.0.smithi016.stdout:1/633: mkdir d3/d4/d19/d20/d2b/dd6 0 2024-04-22T15:49:36.785 INFO:tasks.workunit.client.0.smithi016.stdout:1/634: dread - d3/d4/d15/d38/d63/d67/fd0 zero size 2024-04-22T15:49:36.785 INFO:tasks.workunit.client.0.smithi016.stdout:2/406: readlink d2/d1a/l56 0 2024-04-22T15:49:36.786 INFO:tasks.workunit.client.0.smithi016.stdout:1/635: fsync d3/d4/d37/fb9 0 2024-04-22T15:49:36.791 INFO:tasks.workunit.client.0.smithi016.stdout:6/503: rename d3/f59 to d3/da/db/d44/d68/fa5 0 2024-04-22T15:49:36.793 INFO:tasks.workunit.client.0.smithi016.stdout:4/456: stat d7/d24/l89 0 2024-04-22T15:49:36.794 INFO:tasks.workunit.client.0.smithi016.stdout:5/409: dread df/d22/f24 [0,4194304] 0 2024-04-22T15:49:36.796 INFO:tasks.workunit.client.0.smithi016.stdout:8/425: mknod d2/d77/c83 0 2024-04-22T15:49:36.797 INFO:tasks.workunit.client.0.smithi016.stdout:8/426: chown d2/d82/d44/d66/d7e 12763 1 2024-04-22T15:49:36.799 INFO:tasks.workunit.client.0.smithi016.stdout:3/492: write d7/fc [5642534,56412] 0 2024-04-22T15:49:36.801 INFO:tasks.workunit.client.0.smithi016.stdout:6/504: write d3/da/dd/d1b/f85 [2819754,63491] 0 2024-04-22T15:49:36.802 INFO:tasks.workunit.client.0.smithi016.stdout:4/457: unlink d7/d22/l63 0 2024-04-22T15:49:36.802 INFO:tasks.workunit.client.0.smithi016.stdout:6/505: read - d3/da/db/d14/d4c/d58/f6f zero size 2024-04-22T15:49:36.803 INFO:tasks.workunit.client.0.smithi016.stdout:4/458: stat d7/d24/d41/f5d 0 2024-04-22T15:49:36.804 INFO:tasks.workunit.client.0.smithi016.stdout:6/506: write d3/da/dd/d67/f8f [129711,4448] 0 2024-04-22T15:49:36.807 INFO:tasks.workunit.client.0.smithi016.stdout:5/410: chown df/d15/d1a/l46 6508800 1 2024-04-22T15:49:36.807 INFO:tasks.workunit.client.0.smithi016.stdout:1/636: creat d3/d4/dc4/fd7 x:0 0 0 2024-04-22T15:49:36.807 INFO:tasks.workunit.client.0.smithi016.stdout:6/507: write d3/da/d12/f9c [2843200,100202] 0 2024-04-22T15:49:36.810 INFO:tasks.workunit.client.0.smithi016.stdout:1/637: write d3/d4/da/f26 [4539602,88521] 0 2024-04-22T15:49:36.810 INFO:tasks.workunit.client.0.smithi016.stdout:0/501: dwrite dc/d13/d28/d65/d40/d4a/d59/f87 [0,4194304] 0 2024-04-22T15:49:36.812 INFO:tasks.workunit.client.0.smithi016.stdout:0/502: read - dc/d13/d28/f6a zero size 2024-04-22T15:49:36.813 INFO:tasks.workunit.client.0.smithi016.stdout:4/459: mkdir d7/d21/d42/d59/d88/d9d 0 2024-04-22T15:49:36.815 INFO:tasks.workunit.client.0.smithi016.stdout:4/460: write d7/d21/d42/d59/d88/f9c [24888,111700] 0 2024-04-22T15:49:36.816 INFO:tasks.workunit.client.0.smithi016.stdout:4/461: chown d7/d21/c5f 2 1 2024-04-22T15:49:36.819 INFO:tasks.workunit.client.0.smithi016.stdout:0/503: dread dc/d13/f16 [0,4194304] 0 2024-04-22T15:49:36.819 INFO:tasks.workunit.client.0.smithi016.stdout:8/427: mknod d2/d10/d47/c84 0 2024-04-22T15:49:36.820 INFO:tasks.workunit.client.0.smithi016.stdout:3/493: link d7/d17/d18/d32/d42/d63/f9d d7/d17/d18/d32/f9f 0 2024-04-22T15:49:36.823 INFO:tasks.workunit.client.0.smithi016.stdout:3/494: write d7/f69 [5175546,129696] 0 2024-04-22T15:49:36.824 INFO:tasks.workunit.client.0.smithi016.stdout:5/411: rename df/d15/d1a/d4d to df/d15/d1a/d50/d77/d7a 0 2024-04-22T15:49:36.825 INFO:tasks.workunit.client.0.smithi016.stdout:1/638: creat d3/d4/d3a/d9e/fd8 x:0 0 0 2024-04-22T15:49:36.826 INFO:tasks.workunit.client.0.smithi016.stdout:2/407: dwrite d2/de/d2a/d59/f78 [0,4194304] 0 2024-04-22T15:49:36.829 INFO:tasks.workunit.client.0.smithi016.stdout:2/408: fsync d2/d2c/f82 0 2024-04-22T15:49:36.829 INFO:tasks.workunit.client.0.smithi016.stdout:6/508: creat d3/da/dd/d67/d6c/fa6 x:0 0 0 2024-04-22T15:49:36.830 INFO:tasks.workunit.client.0.smithi016.stdout:4/462: read d7/f8 [3010946,111901] 0 2024-04-22T15:49:36.831 INFO:tasks.workunit.client.0.smithi016.stdout:0/504: dwrite dc/d13/d14/f34 [0,4194304] 0 2024-04-22T15:49:36.832 INFO:tasks.workunit.client.0.smithi016.stdout:8/428: mkdir d2/d82/d44/d66/d7e/d85 0 2024-04-22T15:49:36.833 INFO:tasks.workunit.client.0.smithi016.stdout:8/429: dread - d2/d82/d56/f71 zero size 2024-04-22T15:49:36.836 INFO:tasks.workunit.client.0.smithi016.stdout:0/505: write dc/d13/d28/d65/d1f/d48/f64 [2828214,5494] 0 2024-04-22T15:49:36.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:36 smithi027 ceph-mon[34922]: pgmap v30: 65 pgs: 65 active+clean; 815 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 16 MiB/s rd, 99 MiB/s wr, 299 op/s 2024-04-22T15:49:36.839 INFO:tasks.workunit.client.0.smithi016.stdout:3/495: dread d7/d17/f2e [0,4194304] 0 2024-04-22T15:49:36.842 INFO:tasks.workunit.client.0.smithi016.stdout:2/409: creat d2/d4d/d76/f9a x:0 0 0 2024-04-22T15:49:36.843 INFO:tasks.workunit.client.0.smithi016.stdout:6/509: symlink d3/da/db/d44/d68/d95/la7 0 2024-04-22T15:49:36.845 INFO:tasks.workunit.client.0.smithi016.stdout:5/412: dwrite df/d15/d1a/d50/d77/f40 [0,4194304] 0 2024-04-22T15:49:36.849 INFO:tasks.workunit.client.0.smithi016.stdout:8/430: write d2/d82/d44/f53 [256541,14564] 0 2024-04-22T15:49:36.851 INFO:tasks.workunit.client.0.smithi016.stdout:1/639: creat d3/d2e/d80/dbb/fd9 x:0 0 0 2024-04-22T15:49:36.856 INFO:tasks.workunit.client.0.smithi016.stdout:3/496: mknod d7/d75/ca0 0 2024-04-22T15:49:36.859 INFO:tasks.workunit.client.0.smithi016.stdout:2/410: rename d2/d23/d92/f4a to d2/d51/f9b 0 2024-04-22T15:49:36.859 INFO:tasks.workunit.client.0.smithi016.stdout:2/411: fdatasync d2/d51/f5a 0 2024-04-22T15:49:36.863 INFO:tasks.workunit.client.0.smithi016.stdout:0/506: link dc/d13/d14/f18 dc/d13/d6f/d8b/f9a 0 2024-04-22T15:49:36.864 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:36 smithi016 ceph-mon[26620]: pgmap v30: 65 pgs: 65 active+clean; 815 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 16 MiB/s rd, 99 MiB/s wr, 299 op/s 2024-04-22T15:49:36.868 INFO:tasks.workunit.client.0.smithi016.stdout:8/431: dwrite d2/d10/f61 [0,4194304] 0 2024-04-22T15:49:36.869 INFO:tasks.workunit.client.0.smithi016.stdout:8/432: write d2/d10/d43/f4b [178930,46324] 0 2024-04-22T15:49:36.871 INFO:tasks.workunit.client.0.smithi016.stdout:6/510: mknod d3/da/d12/d13/d38/d84/ca8 0 2024-04-22T15:49:36.873 INFO:tasks.workunit.client.0.smithi016.stdout:2/412: mknod d2/d4d/d64/c9c 0 2024-04-22T15:49:36.874 INFO:tasks.workunit.client.0.smithi016.stdout:2/413: chown d2/d4d/d6d/f7e 702 1 2024-04-22T15:49:36.878 INFO:tasks.workunit.client.0.smithi016.stdout:3/497: dread d7/d17/d18/d32/d3c/d47/f33 [0,4194304] 0 2024-04-22T15:49:36.880 INFO:tasks.workunit.client.0.smithi016.stdout:4/463: getdents d7/d21/d45/d49/d67 0 2024-04-22T15:49:36.880 INFO:tasks.workunit.client.0.smithi016.stdout:3/498: truncate d7/d17/f80 4207533 0 2024-04-22T15:49:36.882 INFO:tasks.workunit.client.0.smithi016.stdout:2/414: creat d2/d23/d92/f9d x:0 0 0 2024-04-22T15:49:36.884 INFO:tasks.workunit.client.0.smithi016.stdout:0/507: mknod dc/d13/d28/d65/d40/d4a/d59/d98/c9b 0 2024-04-22T15:49:36.885 INFO:tasks.workunit.client.0.smithi016.stdout:0/508: write dc/d13/d14/f34 [3897747,61026] 0 2024-04-22T15:49:36.887 INFO:tasks.workunit.client.0.smithi016.stdout:1/640: rename d3/d4/da/f88 to d3/d4/d19/fda 0 2024-04-22T15:49:36.894 INFO:tasks.workunit.client.0.smithi016.stdout:2/415: creat d2/d2c/f9e x:0 0 0 2024-04-22T15:49:36.899 INFO:tasks.workunit.client.0.smithi016.stdout:6/511: rename d3/da/d12/f40 to d3/da/d12/d13/d46/fa9 0 2024-04-22T15:49:36.901 INFO:tasks.workunit.client.0.smithi016.stdout:4/464: mkdir d7/d21/d45/d9e 0 2024-04-22T15:49:36.905 INFO:tasks.workunit.client.0.smithi016.stdout:6/512: dwrite d3/da/dd/f10 [0,4194304] 0 2024-04-22T15:49:36.907 INFO:tasks.workunit.client.0.smithi016.stdout:7/441: sync 2024-04-22T15:49:36.908 INFO:tasks.workunit.client.0.smithi016.stdout:8/433: truncate d2/f22 306135 0 2024-04-22T15:49:36.910 INFO:tasks.workunit.client.0.smithi016.stdout:4/465: fsync f5 0 2024-04-22T15:49:36.912 INFO:tasks.workunit.client.0.smithi016.stdout:3/499: write d7/d17/d18/d32/d3c/d47/f33 [53783,121384] 0 2024-04-22T15:49:36.914 INFO:tasks.workunit.client.0.smithi016.stdout:5/413: dwrite df/d15/d1a/d28/f3a [0,4194304] 0 2024-04-22T15:49:36.916 INFO:tasks.workunit.client.0.smithi016.stdout:7/442: rmdir d3/dc/d1b/d5a 39 2024-04-22T15:49:36.920 INFO:tasks.workunit.client.0.smithi016.stdout:8/434: dread d2/d82/d44/f53 [0,4194304] 0 2024-04-22T15:49:36.920 INFO:tasks.workunit.client.0.smithi016.stdout:6/513: rename d3/da/dd/d1b/d57/fa1 to d3/da/db/d44/d68/faa 0 2024-04-22T15:49:36.922 INFO:tasks.workunit.client.0.smithi016.stdout:5/414: dwrite df/d15/d1a/d50/d77/f61 [0,4194304] 0 2024-04-22T15:49:36.926 INFO:tasks.workunit.client.0.smithi016.stdout:0/509: link dc/d13/d28/d65/c23 dc/d13/d99/c9c 0 2024-04-22T15:49:36.928 INFO:tasks.workunit.client.0.smithi016.stdout:0/510: write dc/d13/d14/f34 [115371,129531] 0 2024-04-22T15:49:36.929 INFO:tasks.workunit.client.0.smithi016.stdout:1/641: dread d3/d4/d37/f41 [0,4194304] 0 2024-04-22T15:49:36.934 INFO:tasks.workunit.client.0.smithi016.stdout:2/416: getdents d2/d2c 0 2024-04-22T15:49:36.940 INFO:tasks.workunit.client.0.smithi016.stdout:3/500: truncate d7/f14 367649 0 2024-04-22T15:49:36.941 INFO:tasks.workunit.client.0.smithi016.stdout:4/466: getdents d7/d21/d45/d49/d9b 0 2024-04-22T15:49:36.942 INFO:tasks.workunit.client.0.smithi016.stdout:4/467: stat d7/c12 0 2024-04-22T15:49:36.942 INFO:tasks.workunit.client.0.smithi016.stdout:2/417: write d2/d2c/d97/f8c [119203,123469] 0 2024-04-22T15:49:36.942 INFO:tasks.workunit.client.0.smithi016.stdout:7/443: creat d3/dc/d1b/d5a/d59/d64/fa0 x:0 0 0 2024-04-22T15:49:36.943 INFO:tasks.workunit.client.0.smithi016.stdout:5/415: unlink df/d15/d1a/d28/l2f 0 2024-04-22T15:49:36.944 INFO:tasks.workunit.client.0.smithi016.stdout:5/416: stat df/l5a 0 2024-04-22T15:49:36.944 INFO:tasks.workunit.client.0.smithi016.stdout:0/511: dread dc/d13/d28/d65/f42 [0,4194304] 0 2024-04-22T15:49:36.945 INFO:tasks.workunit.client.0.smithi016.stdout:3/501: mknod d7/d1b/d21/ca1 0 2024-04-22T15:49:36.945 INFO:tasks.workunit.client.0.smithi016.stdout:5/417: chown df/l18 550694 1 2024-04-22T15:49:36.948 INFO:tasks.workunit.client.0.smithi016.stdout:1/642: link d3/d4/d15/d38/d63/fa9 d3/d4/d3a/d91/db3/fdb 0 2024-04-22T15:49:36.950 INFO:tasks.workunit.client.0.smithi016.stdout:2/418: mkdir d2/d1a/d4c/d9f 0 2024-04-22T15:49:36.951 INFO:tasks.workunit.client.0.smithi016.stdout:2/419: chown d2/d9/l1d 67732566 1 2024-04-22T15:49:36.953 INFO:tasks.workunit.client.0.smithi016.stdout:7/444: rmdir d3/dc/d1b/d5a/d4a/d6e 39 2024-04-22T15:49:36.954 INFO:tasks.workunit.client.0.smithi016.stdout:5/418: readlink df/d15/d1a/d4e/l5e 0 2024-04-22T15:49:36.957 INFO:tasks.workunit.client.0.smithi016.stdout:8/435: dread d2/d82/d3e/f42 [0,4194304] 0 2024-04-22T15:49:36.958 INFO:tasks.workunit.client.0.smithi016.stdout:3/502: rename d7/d1b/d21/d55/f7d to d7/d17/d18/d32/d42/fa2 0 2024-04-22T15:49:36.958 INFO:tasks.workunit.client.0.smithi016.stdout:1/643: rmdir d3/d4/d15/d8c 39 2024-04-22T15:49:36.959 INFO:tasks.workunit.client.0.smithi016.stdout:6/514: dread d3/da/dd/d1b/f85 [0,4194304] 0 2024-04-22T15:49:36.960 INFO:tasks.workunit.client.0.smithi016.stdout:8/436: chown d2/d10/d47/c72 379 1 2024-04-22T15:49:36.960 INFO:tasks.workunit.client.0.smithi016.stdout:9/469: sync 2024-04-22T15:49:36.962 INFO:tasks.workunit.client.0.smithi016.stdout:9/470: chown l3 37 1 2024-04-22T15:49:36.963 INFO:tasks.workunit.client.0.smithi016.stdout:1/644: truncate d3/d84/fcd 853572 0 2024-04-22T15:49:36.963 INFO:tasks.workunit.client.0.smithi016.stdout:2/420: creat d2/d2c/d97/d67/fa0 x:0 0 0 2024-04-22T15:49:36.966 INFO:tasks.workunit.client.0.smithi016.stdout:7/445: rmdir d3/dc/d1b/d5a/d4a 39 2024-04-22T15:49:36.969 INFO:tasks.workunit.client.0.smithi016.stdout:5/419: write df/d22/f3b [1179155,129016] 0 2024-04-22T15:49:36.969 INFO:tasks.workunit.client.0.smithi016.stdout:3/503: dwrite d7/d17/f3e [0,4194304] 0 2024-04-22T15:49:36.971 INFO:tasks.workunit.client.0.smithi016.stdout:7/446: write d3/dc/d33/f4d [413740,10681] 0 2024-04-22T15:49:36.975 INFO:tasks.workunit.client.0.smithi016.stdout:3/504: write d7/d75/f76 [291188,19097] 0 2024-04-22T15:49:36.978 INFO:tasks.workunit.client.0.smithi016.stdout:1/645: mkdir d3/d4/da/ddc 0 2024-04-22T15:49:36.979 INFO:tasks.workunit.client.0.smithi016.stdout:2/421: creat d2/d23/d92/fa1 x:0 0 0 2024-04-22T15:49:36.981 INFO:tasks.workunit.client.0.smithi016.stdout:9/471: dwrite d9/de/d23/d24/d25/d36/f6b [0,4194304] 0 2024-04-22T15:49:36.996 INFO:tasks.workunit.client.0.smithi016.stdout:4/468: write d7/d24/d41/f74 [675808,43872] 0 2024-04-22T15:49:36.996 INFO:tasks.workunit.client.0.smithi016.stdout:0/512: getdents dc/d13/d14 0 2024-04-22T15:49:36.997 INFO:tasks.workunit.client.0.smithi016.stdout:9/472: mknod d9/d20/c88 0 2024-04-22T15:49:37.005 INFO:tasks.workunit.client.0.smithi016.stdout:2/422: mknod d2/d1a/d4c/d9f/ca2 0 2024-04-22T15:49:37.006 INFO:tasks.workunit.client.0.smithi016.stdout:4/469: fsync d7/d21/d42/f55 0 2024-04-22T15:49:37.007 INFO:tasks.workunit.client.0.smithi016.stdout:8/437: dwrite d2/d82/d44/d66/d1c/f3d [0,4194304] 0 2024-04-22T15:49:37.007 INFO:tasks.workunit.client.0.smithi016.stdout:5/420: creat df/d15/f7b x:0 0 0 2024-04-22T15:49:37.009 INFO:tasks.workunit.client.0.smithi016.stdout:0/513: mkdir dc/d13/d28/d65/d40/d4a/d59/d9d 0 2024-04-22T15:49:37.009 INFO:tasks.workunit.client.0.smithi016.stdout:3/505: dread d7/fc [0,4194304] 0 2024-04-22T15:49:37.011 INFO:tasks.workunit.client.0.smithi016.stdout:2/423: write d2/d51/f5a [170440,57278] 0 2024-04-22T15:49:37.012 INFO:tasks.workunit.client.0.smithi016.stdout:6/515: dwrite d3/da/d12/d13/d46/f89 [0,4194304] 0 2024-04-22T15:49:37.012 INFO:tasks.workunit.client.0.smithi016.stdout:9/473: creat d9/de/d23/d24/d25/f89 x:0 0 0 2024-04-22T15:49:37.018 INFO:tasks.workunit.client.0.smithi016.stdout:9/474: write d9/de/d23/d30/f75 [440639,58559] 0 2024-04-22T15:49:37.020 INFO:tasks.workunit.client.0.smithi016.stdout:7/447: getdents d3/dc/d1b/d5a/d69 0 2024-04-22T15:49:37.021 INFO:tasks.workunit.client.0.smithi016.stdout:0/514: mknod dc/d13/d28/c9e 0 2024-04-22T15:49:37.022 INFO:tasks.workunit.client.0.smithi016.stdout:3/506: creat d7/d17/d18/d27/fa3 x:0 0 0 2024-04-22T15:49:37.023 INFO:tasks.workunit.client.0.smithi016.stdout:8/438: write d2/d10/f39 [1708289,48187] 0 2024-04-22T15:49:37.025 INFO:tasks.workunit.client.0.smithi016.stdout:7/448: truncate d3/d14/f85 790720 0 2024-04-22T15:49:37.026 INFO:tasks.workunit.client.0.smithi016.stdout:2/424: mkdir d2/d1a/d4c/d9f/da3 0 2024-04-22T15:49:37.027 INFO:tasks.workunit.client.0.smithi016.stdout:6/516: chown d3/da/db/f61 3 1 2024-04-22T15:49:37.028 INFO:tasks.workunit.client.0.smithi016.stdout:5/421: dwrite df/d15/d1a/d4e/f72 [0,4194304] 0 2024-04-22T15:49:37.028 INFO:tasks.workunit.client.0.smithi016.stdout:1/646: dread d3/d4/d15/f16 [0,4194304] 0 2024-04-22T15:49:37.029 INFO:tasks.workunit.client.0.smithi016.stdout:3/507: read - d7/d17/d18/d32/d42/fa2 zero size 2024-04-22T15:49:37.031 INFO:tasks.workunit.client.0.smithi016.stdout:4/470: rename d7/d4e/d39/l72 to d7/d21/d45/d49/d9b/l9f 0 2024-04-22T15:49:37.035 INFO:tasks.workunit.client.0.smithi016.stdout:6/517: truncate d3/da/dd/d1b/f62 4606653 0 2024-04-22T15:49:37.037 INFO:tasks.workunit.client.0.smithi016.stdout:8/439: mknod d2/d82/d44/d66/d1c/c86 0 2024-04-22T15:49:37.037 INFO:tasks.workunit.client.0.smithi016.stdout:0/515: dwrite dc/d13/d28/d65/d30/d75/f80 [4194304,4194304] 0 2024-04-22T15:49:37.042 INFO:tasks.workunit.client.0.smithi016.stdout:5/422: mknod df/d15/d1a/d50/d77/c7c 0 2024-04-22T15:49:37.043 INFO:tasks.workunit.client.0.smithi016.stdout:2/425: dread d2/d23/f3b [0,4194304] 0 2024-04-22T15:49:37.044 INFO:tasks.workunit.client.0.smithi016.stdout:4/471: rmdir d7/d4e/d2f/d50 39 2024-04-22T15:49:37.044 INFO:tasks.workunit.client.0.smithi016.stdout:3/508: mknod d7/d17/d18/d27/ca4 0 2024-04-22T15:49:37.046 INFO:tasks.workunit.client.0.smithi016.stdout:7/449: mkdir d3/dc/d9f/da1 0 2024-04-22T15:49:37.047 INFO:tasks.workunit.client.0.smithi016.stdout:9/475: creat d9/de/f8a x:0 0 0 2024-04-22T15:49:37.048 INFO:tasks.workunit.client.0.smithi016.stdout:0/516: symlink dc/d13/d14/l9f 0 2024-04-22T15:49:37.049 INFO:tasks.workunit.client.0.smithi016.stdout:3/509: read d7/d17/d18/f2a [2021984,4091] 0 2024-04-22T15:49:37.050 INFO:tasks.workunit.client.0.smithi016.stdout:8/440: rename d2/d82/d44/d66/c11 to d2/d82/d44/d6f/c87 0 2024-04-22T15:49:37.050 INFO:tasks.workunit.client.0.smithi016.stdout:9/476: fsync d9/d37/f49 0 2024-04-22T15:49:37.051 INFO:tasks.workunit.client.0.smithi016.stdout:6/518: mkdir d3/d53/dab 0 2024-04-22T15:49:37.055 INFO:tasks.workunit.client.0.smithi016.stdout:2/426: read d2/d2c/d97/f31 [2464889,32587] 0 2024-04-22T15:49:37.057 INFO:tasks.workunit.client.0.smithi016.stdout:8/441: unlink d2/d30/c3b 0 2024-04-22T15:49:37.057 INFO:tasks.workunit.client.0.smithi016.stdout:5/423: dwrite df/d15/d1a/f6c [0,4194304] 0 2024-04-22T15:49:37.061 INFO:tasks.workunit.client.0.smithi016.stdout:5/424: truncate df/f57 4447068 0 2024-04-22T15:49:37.063 INFO:tasks.workunit.client.0.smithi016.stdout:9/477: creat d9/de/d23/d24/d25/f8b x:0 0 0 2024-04-22T15:49:37.063 INFO:tasks.workunit.client.0.smithi016.stdout:5/425: fdatasync df/f27 0 2024-04-22T15:49:37.066 INFO:tasks.workunit.client.0.smithi016.stdout:0/517: dwrite dc/d13/d28/d65/f63 [0,4194304] 0 2024-04-22T15:49:37.066 INFO:tasks.workunit.client.0.smithi016.stdout:6/519: write d3/da/d12/d13/d46/fa9 [282599,44106] 0 2024-04-22T15:49:37.067 INFO:tasks.workunit.client.0.smithi016.stdout:9/478: dread - d9/de/d23/d30/f86 zero size 2024-04-22T15:49:37.069 INFO:tasks.workunit.client.0.smithi016.stdout:9/479: chown d9/d20/f6f 827744737 1 2024-04-22T15:49:37.070 INFO:tasks.workunit.client.0.smithi016.stdout:1/647: dread d3/d4/d3a/d6b/d77/f76 [0,4194304] 0 2024-04-22T15:49:37.073 INFO:tasks.workunit.client.0.smithi016.stdout:5/426: mkdir df/d15/d1a/d50/d77/d7d 0 2024-04-22T15:49:37.073 INFO:tasks.workunit.client.0.smithi016.stdout:6/520: truncate d3/da/db/d44/d68/faa 347900 0 2024-04-22T15:49:37.073 INFO:tasks.workunit.client.0.smithi016.stdout:3/510: link d7/d17/d18/d27/f53 d7/d1b/d21/fa5 0 2024-04-22T15:49:37.074 INFO:tasks.workunit.client.0.smithi016.stdout:1/648: readlink d3/d4/d3a/l78 0 2024-04-22T15:49:37.076 INFO:tasks.workunit.client.0.smithi016.stdout:6/521: read - d3/da/dd/f9d zero size 2024-04-22T15:49:37.077 INFO:tasks.workunit.client.0.smithi016.stdout:9/480: rename l6 to d9/d37/l8c 0 2024-04-22T15:49:37.077 INFO:tasks.workunit.client.0.smithi016.stdout:5/427: write df/d15/d1a/f78 [264023,20727] 0 2024-04-22T15:49:37.077 INFO:tasks.workunit.client.0.smithi016.stdout:4/472: write d7/d4e/d81/f92 [2004078,23110] 0 2024-04-22T15:49:37.081 INFO:tasks.workunit.client.0.smithi016.stdout:8/442: write d2/d82/d44/f53 [3316358,45887] 0 2024-04-22T15:49:37.083 INFO:tasks.workunit.client.0.smithi016.stdout:6/522: creat d3/da/d12/d13/fac x:0 0 0 2024-04-22T15:49:37.086 INFO:tasks.workunit.client.0.smithi016.stdout:8/443: write d2/d48/f55 [505491,6615] 0 2024-04-22T15:49:37.087 INFO:tasks.workunit.client.0.smithi016.stdout:7/450: dwrite d3/dc/d1b/d5a/d9/df/d94/f72 [4194304,4194304] 0 2024-04-22T15:49:37.088 INFO:tasks.workunit.client.0.smithi016.stdout:5/428: creat df/d15/f7e x:0 0 0 2024-04-22T15:49:37.089 INFO:tasks.workunit.client.0.smithi016.stdout:1/649: link d3/d4/da/fa2 d3/d4/d3a/d96/fdd 0 2024-04-22T15:49:37.090 INFO:tasks.workunit.client.0.smithi016.stdout:9/481: write d9/de/d23/d24/d25/f2c [103598,67105] 0 2024-04-22T15:49:37.092 INFO:tasks.workunit.client.0.smithi016.stdout:1/650: rename d3 to d3/d28/dde 22 2024-04-22T15:49:37.093 INFO:tasks.workunit.client.0.smithi016.stdout:3/511: dread d7/f69 [0,4194304] 0 2024-04-22T15:49:37.095 INFO:tasks.workunit.client.0.smithi016.stdout:9/482: write d9/de/d23/d24/f53 [257795,118356] 0 2024-04-22T15:49:37.095 INFO:tasks.workunit.client.0.smithi016.stdout:3/512: read d7/f5a [9027881,38382] 0 2024-04-22T15:49:37.096 INFO:tasks.workunit.client.0.smithi016.stdout:6/523: dwrite d3/f8 [0,4194304] 0 2024-04-22T15:49:37.100 INFO:tasks.workunit.client.0.smithi016.stdout:0/518: truncate dc/d13/d14/d26/f7b 94433 0 2024-04-22T15:49:37.105 INFO:tasks.workunit.client.0.smithi016.stdout:6/524: dwrite d3/da/dd/f70 [0,4194304] 0 2024-04-22T15:49:37.106 INFO:tasks.workunit.client.0.smithi016.stdout:9/483: read d9/de/d26/f44 [155350,50732] 0 2024-04-22T15:49:37.106 INFO:tasks.workunit.client.0.smithi016.stdout:3/513: creat d7/d1b/d21/d55/fa6 x:0 0 0 2024-04-22T15:49:37.107 INFO:tasks.workunit.client.0.smithi016.stdout:5/429: mknod df/c7f 0 2024-04-22T15:49:37.107 INFO:tasks.workunit.client.0.smithi016.stdout:4/473: getdents d7/d4e/d2f 0 2024-04-22T15:49:37.110 INFO:tasks.workunit.client.0.smithi016.stdout:0/519: symlink dc/d13/d28/d65/d30/la0 0 2024-04-22T15:49:37.113 INFO:tasks.workunit.client.0.smithi016.stdout:6/525: write d3/da/db/d44/d68/faa [85493,71399] 0 2024-04-22T15:49:37.114 INFO:tasks.workunit.client.0.smithi016.stdout:6/526: dread - d3/da/dd/d67/f79 zero size 2024-04-22T15:49:37.118 INFO:tasks.workunit.client.0.smithi016.stdout:3/514: rmdir d7/d17/d18/d32/d3c/d47 39 2024-04-22T15:49:37.123 INFO:tasks.workunit.client.0.smithi016.stdout:4/474: mknod d7/d21/d42/d59/d88/ca0 0 2024-04-22T15:49:37.124 INFO:tasks.workunit.client.0.smithi016.stdout:5/430: dwrite df/d15/d1a/d28/f2c [0,4194304] 0 2024-04-22T15:49:37.125 INFO:tasks.workunit.client.0.smithi016.stdout:7/451: creat d3/dc/d1b/d5a/d4a/d6e/fa2 x:0 0 0 2024-04-22T15:49:37.126 INFO:tasks.workunit.client.0.smithi016.stdout:4/475: dread d7/d21/d42/f55 [0,4194304] 0 2024-04-22T15:49:37.128 INFO:tasks.workunit.client.0.smithi016.stdout:4/476: write d7/f19 [7380342,122146] 0 2024-04-22T15:49:37.129 INFO:tasks.workunit.client.0.smithi016.stdout:9/484: dread d9/f3b [0,4194304] 0 2024-04-22T15:49:37.131 INFO:tasks.workunit.client.0.smithi016.stdout:0/520: symlink dc/d13/la1 0 2024-04-22T15:49:37.131 INFO:tasks.workunit.client.0.smithi016.stdout:7/452: write d3/f34 [4162233,57156] 0 2024-04-22T15:49:37.132 INFO:tasks.workunit.client.0.smithi016.stdout:9/485: chown f7 35976 1 2024-04-22T15:49:37.133 INFO:tasks.workunit.client.0.smithi016.stdout:1/651: dwrite d3/d4/da/f83 [4194304,4194304] 0 2024-04-22T15:49:37.134 INFO:tasks.workunit.client.0.smithi016.stdout:9/486: readlink d9/de/l6a 0 2024-04-22T15:49:37.135 INFO:tasks.workunit.client.0.smithi016.stdout:5/431: fsync df/d15/d1a/d28/f74 0 2024-04-22T15:49:37.138 INFO:tasks.workunit.client.0.smithi016.stdout:5/432: chown df/d15/d1a 140993200 1 2024-04-22T15:49:37.138 INFO:tasks.workunit.client.0.smithi016.stdout:0/521: chown dc/d13/d14/d26/l66 131 1 2024-04-22T15:49:37.138 INFO:tasks.workunit.client.0.smithi016.stdout:3/515: symlink d7/la7 0 2024-04-22T15:49:37.139 INFO:tasks.workunit.client.0.smithi016.stdout:8/444: write d2/d82/d56/f64 [353512,7435] 0 2024-04-22T15:49:37.141 INFO:tasks.workunit.client.0.smithi016.stdout:6/527: link d3/da/dd/d1b/c39 d3/da/db/d14/d4c/d58/cad 0 2024-04-22T15:49:37.143 INFO:tasks.workunit.client.0.smithi016.stdout:1/652: mknod d3/d2e/d80/d82/cdf 0 2024-04-22T15:49:37.145 INFO:tasks.workunit.client.0.smithi016.stdout:4/477: dwrite d7/d22/f54 [0,4194304] 0 2024-04-22T15:49:37.145 INFO:tasks.workunit.client.0.smithi016.stdout:0/522: rmdir dc/d13 39 2024-04-22T15:49:37.146 INFO:tasks.workunit.client.0.smithi016.stdout:8/445: chown d2/d10/d47/c5d 65062486 1 2024-04-22T15:49:37.148 INFO:tasks.workunit.client.0.smithi016.stdout:7/453: unlink d3/c8f 0 2024-04-22T15:49:37.150 INFO:tasks.workunit.client.0.smithi016.stdout:1/653: fdatasync d3/d4/d15/d38/d63/d67/fd0 0 2024-04-22T15:49:37.151 INFO:tasks.workunit.client.0.smithi016.stdout:8/446: write d2/d32/f35 [1780948,49343] 0 2024-04-22T15:49:37.152 INFO:tasks.workunit.client.0.smithi016.stdout:3/516: write d7/d17/d18/d32/d3c/f88 [173854,97575] 0 2024-04-22T15:49:37.155 INFO:tasks.workunit.client.0.smithi016.stdout:1/654: write d3/d4/d19/d20/f48 [1380052,53667] 0 2024-04-22T15:49:37.157 INFO:tasks.workunit.client.0.smithi016.stdout:9/487: dwrite d9/de/d23/d24/d4e/f79 [0,4194304] 0 2024-04-22T15:49:37.160 INFO:tasks.workunit.client.0.smithi016.stdout:3/517: stat d7/d17/d18/d32/d3c/d51/d98 0 2024-04-22T15:49:37.165 INFO:tasks.workunit.client.0.smithi016.stdout:5/433: truncate df/f11 1295621 0 2024-04-22T15:49:37.165 INFO:tasks.workunit.client.0.smithi016.stdout:8/447: rename d2/d32/l40 to d2/d82/d44/d66/d1c/l88 0 2024-04-22T15:49:37.169 INFO:tasks.workunit.client.0.smithi016.stdout:4/478: truncate d7/f16 2621796 0 2024-04-22T15:49:37.170 INFO:tasks.workunit.client.0.smithi016.stdout:0/523: dread dc/d13/d28/d65/d40/d4a/d59/f71 [0,4194304] 0 2024-04-22T15:49:37.171 INFO:tasks.workunit.client.0.smithi016.stdout:9/488: rmdir d9/de/d26 39 2024-04-22T15:49:37.172 INFO:tasks.workunit.client.0.smithi016.stdout:0/524: write dc/d13/f16 [4996784,121055] 0 2024-04-22T15:49:37.173 INFO:tasks.workunit.client.0.smithi016.stdout:7/454: unlink d3/dc/d1b/d5a/d59/d64/f9b 0 2024-04-22T15:49:37.174 INFO:tasks.workunit.client.0.smithi016.stdout:7/455: chown d3/dc/d1b/d5a/d4a/f55 3507405 1 2024-04-22T15:49:37.178 INFO:tasks.workunit.client.0.smithi016.stdout:6/528: dread d3/da/dd/f29 [0,4194304] 0 2024-04-22T15:49:37.178 INFO:tasks.workunit.client.0.smithi016.stdout:8/448: symlink d2/d82/d44/d66/d7e/l89 0 2024-04-22T15:49:37.180 INFO:tasks.workunit.client.0.smithi016.stdout:6/529: fdatasync d3/da/db/f17 0 2024-04-22T15:49:37.184 INFO:tasks.workunit.client.0.smithi016.stdout:5/434: dwrite df/d15/d1a/d28/f6d [0,4194304] 0 2024-04-22T15:49:37.251 INFO:tasks.workunit.client.0.smithi016.stdout:8/449: mknod d2/d82/d44/c8a 0 2024-04-22T15:49:37.251 INFO:tasks.workunit.client.0.smithi016.stdout:1/655: dwrite d3/d4/d19/f66 [0,4194304] 0 2024-04-22T15:49:37.251 INFO:tasks.workunit.client.0.smithi016.stdout:5/435: dwrite df/f57 [0,4194304] 0 2024-04-22T15:49:37.251 INFO:tasks.workunit.client.0.smithi016.stdout:1/656: chown d3/d4/d45/c62 53 1 2024-04-22T15:49:37.252 INFO:tasks.workunit.client.0.smithi016.stdout:6/530: creat d3/da/db/d14/d16/fae x:0 0 0 2024-04-22T15:49:37.252 INFO:tasks.workunit.client.0.smithi016.stdout:4/479: dwrite d7/d24/d51/f6a [0,4194304] 0 2024-04-22T15:49:37.252 INFO:tasks.workunit.client.0.smithi016.stdout:5/436: creat df/d15/d1a/d50/d77/d7a/f80 x:0 0 0 2024-04-22T15:49:37.252 INFO:tasks.workunit.client.0.smithi016.stdout:1/657: mkdir d3/d4/d19/d20/d2b/de0 0 2024-04-22T15:49:37.252 INFO:tasks.workunit.client.0.smithi016.stdout:1/658: chown d3/d2e/d80/dbb/fc2 25 1 2024-04-22T15:49:37.252 INFO:tasks.workunit.client.0.smithi016.stdout:6/531: creat d3/da/db/d44/faf x:0 0 0 2024-04-22T15:49:37.252 INFO:tasks.workunit.client.0.smithi016.stdout:5/437: write df/d15/d1a/d50/d77/f62 [1715329,129104] 0 2024-04-22T15:49:37.252 INFO:tasks.workunit.client.0.smithi016.stdout:6/532: readlink d3/da/db/d14/d16/l6b 0 2024-04-22T15:49:37.253 INFO:tasks.workunit.client.0.smithi016.stdout:7/456: creat d3/dc/d1b/d5a/fa3 x:0 0 0 2024-04-22T15:49:37.253 INFO:tasks.workunit.client.0.smithi016.stdout:3/518: rmdir d7/d17/d18/d32/d3c/d47/d83 0 2024-04-22T15:49:37.253 INFO:tasks.workunit.client.0.smithi016.stdout:3/519: read - d7/d17/d18/d32/d42/f9a zero size 2024-04-22T15:49:37.253 INFO:tasks.workunit.client.0.smithi016.stdout:8/450: creat d2/d82/d44/d66/d79/f8b x:0 0 0 2024-04-22T15:49:37.253 INFO:tasks.workunit.client.0.smithi016.stdout:9/489: getdents d9/d37 0 2024-04-22T15:49:37.253 INFO:tasks.workunit.client.0.smithi016.stdout:4/480: mknod d7/d3c/ca1 0 2024-04-22T15:49:37.253 INFO:tasks.workunit.client.0.smithi016.stdout:8/451: unlink d2/d82/d56/f80 0 2024-04-22T15:49:37.254 INFO:tasks.workunit.client.0.smithi016.stdout:1/659: mknod d3/d4/d15/d38/ce1 0 2024-04-22T15:49:37.254 INFO:tasks.workunit.client.0.smithi016.stdout:9/490: mknod d9/de/d23/d30/c8d 0 2024-04-22T15:49:37.254 INFO:tasks.workunit.client.0.smithi016.stdout:6/533: creat d3/da/dd/d7b/fb0 x:0 0 0 2024-04-22T15:49:37.254 INFO:tasks.workunit.client.0.smithi016.stdout:7/457: mkdir d3/dc/d1b/d5a/da4 0 2024-04-22T15:49:37.254 INFO:tasks.workunit.client.0.smithi016.stdout:4/481: rmdir d7/d21/d45 39 2024-04-22T15:49:37.254 INFO:tasks.workunit.client.0.smithi016.stdout:8/452: dwrite d2/d10/f39 [0,4194304] 0 2024-04-22T15:49:37.254 INFO:tasks.workunit.client.0.smithi016.stdout:1/660: mkdir d3/d4/d3a/d6b/d77/de2 0 2024-04-22T15:49:37.255 INFO:tasks.workunit.client.0.smithi016.stdout:6/534: mkdir d3/da/dd/d67/d6c/db1 0 2024-04-22T15:49:37.255 INFO:tasks.workunit.client.0.smithi016.stdout:4/482: truncate d7/d21/d45/f6d 4763470 0 2024-04-22T15:49:37.255 INFO:tasks.workunit.client.0.smithi016.stdout:1/661: creat d3/d4/d19/d20/d9d/fe3 x:0 0 0 2024-04-22T15:49:37.256 INFO:tasks.workunit.client.0.smithi016.stdout:6/535: write d3/da/db/d14/d4c/f52 [586170,123127] 0 2024-04-22T15:49:37.256 INFO:tasks.workunit.client.0.smithi016.stdout:7/458: mknod d3/dc/d1b/d31/d88/ca5 0 2024-04-22T15:49:37.257 INFO:tasks.workunit.client.0.smithi016.stdout:4/483: dread d7/d4e/d81/f92 [0,4194304] 0 2024-04-22T15:49:37.258 INFO:tasks.workunit.client.0.smithi016.stdout:5/438: truncate df/d15/d1a/d28/f3a 2325324 0 2024-04-22T15:49:37.258 INFO:tasks.workunit.client.0.smithi016.stdout:6/536: truncate d3/da/dd/f51 3764307 0 2024-04-22T15:49:37.259 INFO:tasks.workunit.client.0.smithi016.stdout:5/439: write df/d15/d1a/d28/f2c [4870844,104289] 0 2024-04-22T15:49:37.260 INFO:tasks.workunit.client.0.smithi016.stdout:6/537: stat d3/da/db/d44/d68/fa5 0 2024-04-22T15:49:37.261 INFO:tasks.workunit.client.0.smithi016.stdout:6/538: fsync d3/da/f69 0 2024-04-22T15:49:37.261 INFO:tasks.workunit.client.0.smithi016.stdout:7/459: rename d3/dc/d1b/d5a/d59/l60 to d3/dc/d1b/d5a/d59/la6 0 2024-04-22T15:49:37.263 INFO:tasks.workunit.client.0.smithi016.stdout:6/539: chown d3/da/db/d44/d68/d95 25997492 1 2024-04-22T15:49:37.267 INFO:tasks.workunit.client.0.smithi016.stdout:5/440: dread df/f11 [0,4194304] 0 2024-04-22T15:49:37.267 INFO:tasks.workunit.client.0.smithi016.stdout:2/427: sync 2024-04-22T15:49:37.268 INFO:tasks.workunit.client.0.smithi016.stdout:5/441: chown df/f57 497342 1 2024-04-22T15:49:37.269 INFO:tasks.workunit.client.0.smithi016.stdout:2/428: readlink d2/d1a/l56 0 2024-04-22T15:49:37.270 INFO:tasks.workunit.client.0.smithi016.stdout:1/662: rename d3/d4/d19/d20/d9d to d3/d4/d3a/d96/de4 0 2024-04-22T15:49:37.272 INFO:tasks.workunit.client.0.smithi016.stdout:6/540: creat d3/da/dd/fb2 x:0 0 0 2024-04-22T15:49:37.273 INFO:tasks.workunit.client.0.smithi016.stdout:9/491: dwrite d9/de/f1a [4194304,4194304] 0 2024-04-22T15:49:37.274 INFO:tasks.workunit.client.0.smithi016.stdout:5/442: chown df/d5b 8029 1 2024-04-22T15:49:37.275 INFO:tasks.workunit.client.0.smithi016.stdout:8/453: dwrite d2/d82/f58 [0,4194304] 0 2024-04-22T15:49:37.276 INFO:tasks.workunit.client.0.smithi016.stdout:9/492: readlink d9/de/l42 0 2024-04-22T15:49:37.281 INFO:tasks.workunit.client.0.smithi016.stdout:1/663: symlink d3/d2e/d80/dbb/le5 0 2024-04-22T15:49:37.283 INFO:tasks.workunit.client.0.smithi016.stdout:4/484: dread d7/d21/d45/f6d [0,4194304] 0 2024-04-22T15:49:37.285 INFO:tasks.workunit.client.0.smithi016.stdout:7/460: truncate d3/f34 1602523 0 2024-04-22T15:49:37.286 INFO:tasks.workunit.client.0.smithi016.stdout:8/454: rmdir d2/d82/d56 39 2024-04-22T15:49:37.287 INFO:tasks.workunit.client.0.smithi016.stdout:1/664: write d3/d4/d19/d20/d2b/f4a [5225431,107343] 0 2024-04-22T15:49:37.289 INFO:tasks.workunit.client.0.smithi016.stdout:9/493: dwrite f7 [0,4194304] 0 2024-04-22T15:49:37.290 INFO:tasks.workunit.client.0.smithi016.stdout:7/461: write d3/dc/d1b/d31/f48 [3123152,77105] 0 2024-04-22T15:49:37.297 INFO:tasks.workunit.client.0.smithi016.stdout:9/494: truncate d9/de/d23/d24/f53 1210464 0 2024-04-22T15:49:37.300 INFO:tasks.workunit.client.0.smithi016.stdout:1/665: symlink d3/d4/d3a/d6b/d77/le6 0 2024-04-22T15:49:37.302 INFO:tasks.workunit.client.0.smithi016.stdout:8/455: dwrite d2/d82/d44/d66/f4c [0,4194304] 0 2024-04-22T15:49:37.302 INFO:tasks.workunit.client.0.smithi016.stdout:9/495: write d9/de/d23/d24/d4e/f79 [304764,38624] 0 2024-04-22T15:49:37.310 INFO:tasks.workunit.client.0.smithi016.stdout:9/496: write d9/de/d23/d24/d25/f8b [912643,64682] 0 2024-04-22T15:49:37.311 INFO:tasks.workunit.client.0.smithi016.stdout:4/485: dread d7/d4e/f29 [0,4194304] 0 2024-04-22T15:49:37.312 INFO:tasks.workunit.client.0.smithi016.stdout:7/462: readlink d3/d14/l98 0 2024-04-22T15:49:37.316 INFO:tasks.workunit.client.0.smithi016.stdout:6/541: getdents d3/da/db/d44/d68/d95 0 2024-04-22T15:49:37.316 INFO:tasks.workunit.client.0.smithi016.stdout:6/542: fsync d3/da/d12/d13/f27 0 2024-04-22T15:49:37.317 INFO:tasks.workunit.client.0.smithi016.stdout:1/666: mknod d3/d2e/d80/d82/ce7 0 2024-04-22T15:49:37.319 INFO:tasks.workunit.client.0.smithi016.stdout:9/497: creat d9/de/d23/d24/f8e x:0 0 0 2024-04-22T15:49:37.321 INFO:tasks.workunit.client.0.smithi016.stdout:6/543: write d3/da/db/d14/d16/f55 [552276,123651] 0 2024-04-22T15:49:37.321 INFO:tasks.workunit.client.0.smithi016.stdout:6/544: readlink d3/da/d12/d13/l5e 0 2024-04-22T15:49:37.324 INFO:tasks.workunit.client.0.smithi016.stdout:1/667: mknod d3/d28/ce8 0 2024-04-22T15:49:37.325 INFO:tasks.workunit.client.0.smithi016.stdout:1/668: chown d3/d4/d19/d20/d95/ld3 29708712 1 2024-04-22T15:49:37.329 INFO:tasks.workunit.client.0.smithi016.stdout:9/498: mknod d9/de/d23/d24/d4e/d4f/c8f 0 2024-04-22T15:49:37.331 INFO:tasks.workunit.client.0.smithi016.stdout:5/443: write df/d15/d1a/d28/f3a [1701422,59954] 0 2024-04-22T15:49:37.332 INFO:tasks.workunit.client.0.smithi016.stdout:9/499: truncate d9/de/d23/d24/d25/d36/f5e 248053 0 2024-04-22T15:49:37.332 INFO:tasks.workunit.client.0.smithi016.stdout:6/545: mknod d3/da/db/d14/d16/cb3 0 2024-04-22T15:49:37.334 INFO:tasks.workunit.client.0.smithi016.stdout:1/669: creat d3/d4/d3a/d96/fe9 x:0 0 0 2024-04-22T15:49:37.334 INFO:tasks.workunit.client.0.smithi016.stdout:7/463: creat d3/dc/d1b/d5a/fa7 x:0 0 0 2024-04-22T15:49:37.336 INFO:tasks.workunit.client.0.smithi016.stdout:7/464: chown d3/d14/l98 1417 1 2024-04-22T15:49:37.336 INFO:tasks.workunit.client.0.smithi016.stdout:6/546: creat d3/da/d12/d13/d46/fb4 x:0 0 0 2024-04-22T15:49:37.337 INFO:tasks.workunit.client.0.smithi016.stdout:7/465: fsync d3/d14/f7c 0 2024-04-22T15:49:37.337 INFO:tasks.workunit.client.0.smithi016.stdout:8/456: getdents d2/d82/d56 0 2024-04-22T15:49:37.338 INFO:tasks.workunit.client.0.smithi016.stdout:1/670: mkdir d3/dea 0 2024-04-22T15:49:37.340 INFO:tasks.workunit.client.0.smithi016.stdout:5/444: dwrite df/d15/d1a/f4b [0,4194304] 0 2024-04-22T15:49:37.340 INFO:tasks.workunit.client.0.smithi016.stdout:6/547: chown d3/da/db/d44/d68/d95/la7 59547 1 2024-04-22T15:49:37.346 INFO:tasks.workunit.client.0.smithi016.stdout:1/671: truncate d3/d4/d3a/d91/fa1 1006973 0 2024-04-22T15:49:37.347 INFO:tasks.workunit.client.0.smithi016.stdout:6/548: unlink d3/da/dd/d7b/c94 0 2024-04-22T15:49:37.348 INFO:tasks.workunit.client.0.smithi016.stdout:1/672: write d3/d4/d19/d20/f60 [1712309,50573] 0 2024-04-22T15:49:37.351 INFO:tasks.workunit.client.0.smithi016.stdout:5/445: symlink df/d15/l81 0 2024-04-22T15:49:37.353 INFO:tasks.workunit.client.0.smithi016.stdout:5/446: rename df/d22 to df/d22/d82 22 2024-04-22T15:49:37.353 INFO:tasks.workunit.client.0.smithi016.stdout:8/457: getdents d2/d82/d44/d66/d7e/d85 0 2024-04-22T15:49:37.356 INFO:tasks.workunit.client.0.smithi016.stdout:6/549: rmdir d3/da/db/d14/d16 39 2024-04-22T15:49:37.359 INFO:tasks.workunit.client.0.smithi016.stdout:7/466: link d3/dc/d1b/d5a/f46 d3/dc/d1b/d5a/d4a/d6e/fa8 0 2024-04-22T15:49:37.365 INFO:tasks.workunit.client.0.smithi016.stdout:5/447: symlink df/d15/d1a/l83 0 2024-04-22T15:49:37.367 INFO:tasks.workunit.client.0.smithi016.stdout:7/467: dwrite d3/dc/f6f [0,4194304] 0 2024-04-22T15:49:37.370 INFO:tasks.workunit.client.0.smithi016.stdout:1/673: dread d3/d28/f8b [0,4194304] 0 2024-04-22T15:49:37.370 INFO:tasks.workunit.client.0.smithi016.stdout:7/468: chown d3/dc/d33/f3e 15220 1 2024-04-22T15:49:37.371 INFO:tasks.workunit.client.0.smithi016.stdout:5/448: fsync df/d15/d1a/d28/f2b 0 2024-04-22T15:49:37.371 INFO:tasks.workunit.client.0.smithi016.stdout:8/458: dwrite d2/d82/d44/d66/f3c [4194304,4194304] 0 2024-04-22T15:49:37.373 INFO:tasks.workunit.client.0.smithi016.stdout:8/459: stat d2/d10/l7c 0 2024-04-22T15:49:37.377 INFO:tasks.workunit.client.0.smithi016.stdout:7/469: symlink d3/dc/d1b/d5a/d4a/d6e/la9 0 2024-04-22T15:49:37.379 INFO:tasks.workunit.client.0.smithi016.stdout:1/674: dwrite d3/d4/dc4/fd7 [0,4194304] 0 2024-04-22T15:49:37.382 INFO:tasks.workunit.client.0.smithi016.stdout:7/470: creat d3/dc/d1b/d5a/d9/df/faa x:0 0 0 2024-04-22T15:49:37.384 INFO:tasks.workunit.client.0.smithi016.stdout:1/675: mkdir d3/d2e/d9f/deb 0 2024-04-22T15:49:37.385 INFO:tasks.workunit.client.0.smithi016.stdout:1/676: chown d3/d4/d3a/d96/de4/la6 426925 1 2024-04-22T15:49:37.388 INFO:tasks.workunit.client.0.smithi016.stdout:7/471: rename d3/f34 to d3/dc/d1b/d5a/d69/fab 0 2024-04-22T15:49:37.390 INFO:tasks.workunit.client.0.smithi016.stdout:7/472: chown d3/dc/d1b/d5a/d4a/d6e/f73 2969371 1 2024-04-22T15:49:37.391 INFO:tasks.workunit.client.0.smithi016.stdout:1/677: mknod d3/dea/cec 0 2024-04-22T15:49:37.395 INFO:tasks.workunit.client.0.smithi016.stdout:5/449: write df/d15/d1a/f4b [5019211,112515] 0 2024-04-22T15:49:37.397 INFO:tasks.workunit.client.0.smithi016.stdout:1/678: rename d3/d4/d3a/d6b/cab to d3/d4/d45/ced 0 2024-04-22T15:49:37.398 INFO:tasks.workunit.client.0.smithi016.stdout:5/450: fdatasync df/d15/d1a/d50/d77/f39 0 2024-04-22T15:49:37.398 INFO:tasks.workunit.client.0.smithi016.stdout:7/473: getdents d3/dc/d9f 0 2024-04-22T15:49:37.399 INFO:tasks.workunit.client.0.smithi016.stdout:1/679: chown d3/d4/d3a/d91/db3 10905894 1 2024-04-22T15:49:37.399 INFO:tasks.workunit.client.0.smithi016.stdout:5/451: mknod df/d5b/c84 0 2024-04-22T15:49:37.400 INFO:tasks.workunit.client.0.smithi016.stdout:1/680: chown d3/d4/d19/d20/f60 57 1 2024-04-22T15:49:37.401 INFO:tasks.workunit.client.0.smithi016.stdout:7/474: rename d3/dc/d1b/d5a/fa7 to d3/dc/d1b/d5a/da4/fac 0 2024-04-22T15:49:37.404 INFO:tasks.workunit.client.0.smithi016.stdout:5/452: mkdir df/d5b/d85 0 2024-04-22T15:49:37.405 INFO:tasks.workunit.client.0.smithi016.stdout:1/681: chown d3/d4/d15/d38/lcf 0 1 2024-04-22T15:49:37.406 INFO:tasks.workunit.client.0.smithi016.stdout:7/475: creat d3/d14/fad x:0 0 0 2024-04-22T15:49:37.407 INFO:tasks.workunit.client.0.smithi016.stdout:1/682: mkdir d3/d28/dee 0 2024-04-22T15:49:37.408 INFO:tasks.workunit.client.0.smithi016.stdout:5/453: symlink df/d15/d1a/d50/d77/d7d/l86 0 2024-04-22T15:49:37.412 INFO:tasks.workunit.client.0.smithi016.stdout:0/525: sync 2024-04-22T15:49:37.412 INFO:tasks.workunit.client.0.smithi016.stdout:7/476: creat d3/dc/d9f/fae x:0 0 0 2024-04-22T15:49:37.412 INFO:tasks.workunit.client.0.smithi016.stdout:1/683: readlink d3/d4/l1b 0 2024-04-22T15:49:37.414 INFO:tasks.workunit.client.0.smithi016.stdout:1/684: stat d3/d4/d19/f66 0 2024-04-22T15:49:37.415 INFO:tasks.workunit.client.0.smithi016.stdout:0/526: write dc/d13/d28/d65/d40/d4a/d59/f87 [657957,26393] 0 2024-04-22T15:49:37.415 INFO:tasks.workunit.client.0.smithi016.stdout:7/477: write d3/dc/d1b/d5a/d9/df/f70 [323076,53534] 0 2024-04-22T15:49:37.416 INFO:tasks.workunit.client.0.smithi016.stdout:5/454: dwrite df/d15/d1a/f4b [4194304,4194304] 0 2024-04-22T15:49:37.417 INFO:tasks.workunit.client.0.smithi016.stdout:1/685: fdatasync d3/d4/d19/d20/f60 0 2024-04-22T15:49:37.421 INFO:tasks.workunit.client.0.smithi016.stdout:5/455: write df/d15/d1a/d28/f2c [1431314,93451] 0 2024-04-22T15:49:37.423 INFO:tasks.workunit.client.0.smithi016.stdout:7/478: getdents d3/dc/d1b/d5a/d69/d8c 0 2024-04-22T15:49:37.424 INFO:tasks.workunit.client.0.smithi016.stdout:1/686: fsync d3/d4/d3a/d6b/d77/f76 0 2024-04-22T15:49:37.425 INFO:tasks.workunit.client.0.smithi016.stdout:1/687: chown d3/d4/d15/d38/d63/d67 20632 1 2024-04-22T15:49:37.426 INFO:tasks.workunit.client.0.smithi016.stdout:5/456: mkdir df/d15/d1a/d4e/d58/d87 0 2024-04-22T15:49:37.428 INFO:tasks.workunit.client.0.smithi016.stdout:7/479: mknod d3/dc/d1b/d5a/da4/caf 0 2024-04-22T15:49:37.431 INFO:tasks.workunit.client.0.smithi016.stdout:5/457: rename df/d22/l41 to df/d15/d1a/d4e/d58/d87/l88 0 2024-04-22T15:49:37.432 INFO:tasks.workunit.client.0.smithi016.stdout:5/458: chown df/f11 5 1 2024-04-22T15:49:37.432 INFO:tasks.workunit.client.0.smithi016.stdout:1/688: dwrite d3/d4/d19/d20/f60 [0,4194304] 0 2024-04-22T15:49:37.434 INFO:tasks.workunit.client.0.smithi016.stdout:7/480: symlink d3/dc/d1b/d5a/lb0 0 2024-04-22T15:49:37.436 INFO:tasks.workunit.client.0.smithi016.stdout:5/459: write df/d15/d1a/d28/f32 [4835604,88439] 0 2024-04-22T15:49:37.438 INFO:tasks.workunit.client.0.smithi016.stdout:1/689: dread d3/d4/d19/f66 [0,4194304] 0 2024-04-22T15:49:37.438 INFO:tasks.workunit.client.0.smithi016.stdout:5/460: fsync df/d15/d1a/d50/d77/f39 0 2024-04-22T15:49:37.439 INFO:tasks.workunit.client.0.smithi016.stdout:7/481: creat d3/dc/d9f/fb1 x:0 0 0 2024-04-22T15:49:37.441 INFO:tasks.workunit.client.0.smithi016.stdout:5/461: read df/d15/d1a/d50/d77/f62 [586724,23589] 0 2024-04-22T15:49:37.441 INFO:tasks.workunit.client.0.smithi016.stdout:5/462: chown df/d5b/c84 2 1 2024-04-22T15:49:37.442 INFO:tasks.workunit.client.0.smithi016.stdout:5/463: write df/f10 [3997171,124547] 0 2024-04-22T15:49:37.443 INFO:tasks.workunit.client.0.smithi016.stdout:5/464: fdatasync df/d15/f7e 0 2024-04-22T15:49:37.446 INFO:tasks.workunit.client.0.smithi016.stdout:7/482: rename d3/dc/d33/d84/c97 to d3/dc/d1b/d5a/d69/cb2 0 2024-04-22T15:49:37.450 INFO:tasks.workunit.client.0.smithi016.stdout:7/483: creat d3/dc/d1b/d5a/d59/d64/fb3 x:0 0 0 2024-04-22T15:49:37.451 INFO:tasks.workunit.client.0.smithi016.stdout:7/484: fdatasync d3/dc/d1b/f49 0 2024-04-22T15:49:37.454 INFO:tasks.workunit.client.0.smithi016.stdout:0/527: write dc/d13/d14/d26/f7b [133725,58852] 0 2024-04-22T15:49:37.457 INFO:tasks.workunit.client.0.smithi016.stdout:7/485: symlink d3/dc/d9f/da1/lb4 0 2024-04-22T15:49:37.459 INFO:tasks.workunit.client.0.smithi016.stdout:0/528: creat dc/d13/d28/d65/d57/fa2 x:0 0 0 2024-04-22T15:49:37.464 INFO:tasks.workunit.client.0.smithi016.stdout:1/690: dread d3/d4/d19/f35 [0,4194304] 0 2024-04-22T15:49:37.465 INFO:tasks.workunit.client.0.smithi016.stdout:0/529: dwrite dc/d13/d28/d65/d1f/d48/f64 [0,4194304] 0 2024-04-22T15:49:37.469 INFO:tasks.workunit.client.0.smithi016.stdout:7/486: getdents d3/dc/d1b/d5a/d69 0 2024-04-22T15:49:37.472 INFO:tasks.workunit.client.0.smithi016.stdout:5/465: write df/d15/d1a/d28/f3f [2732904,68130] 0 2024-04-22T15:49:37.475 INFO:tasks.workunit.client.0.smithi016.stdout:1/691: dwrite d3/d4/d3a/d9e/dbf/fc9 [0,4194304] 0 2024-04-22T15:49:37.475 INFO:tasks.workunit.client.0.smithi016.stdout:5/466: fdatasync df/d22/f42 0 2024-04-22T15:49:37.477 INFO:tasks.workunit.client.0.smithi016.stdout:0/530: dwrite dc/d13/d5d/f6c [0,4194304] 0 2024-04-22T15:49:37.478 INFO:tasks.workunit.client.0.smithi016.stdout:0/531: fdatasync dc/d13/d28/d65/d40/f56 0 2024-04-22T15:49:37.486 INFO:tasks.workunit.client.0.smithi016.stdout:5/467: dwrite df/d15/d1a/d28/f2c [4194304,4194304] 0 2024-04-22T15:49:37.492 INFO:tasks.workunit.client.0.smithi016.stdout:5/468: dwrite df/d15/f4a [0,4194304] 0 2024-04-22T15:49:37.493 INFO:tasks.workunit.client.0.smithi016.stdout:1/692: symlink d3/d4/d19/lef 0 2024-04-22T15:49:37.496 INFO:tasks.workunit.client.0.smithi016.stdout:0/532: creat dc/d7d/fa3 x:0 0 0 2024-04-22T15:49:37.498 INFO:tasks.workunit.client.0.smithi016.stdout:7/487: link d3/dc/d1b/d5a/d9/df/d94/f3d d3/dc/d9f/fb5 0 2024-04-22T15:49:37.500 INFO:tasks.workunit.client.0.smithi016.stdout:5/469: readlink df/d15/d1a/l43 0 2024-04-22T15:49:37.502 INFO:tasks.workunit.client.0.smithi016.stdout:0/533: symlink dc/d13/d28/d65/d1f/d48/la4 0 2024-04-22T15:49:37.504 INFO:tasks.workunit.client.0.smithi016.stdout:1/693: creat d3/d4/d3a/d6b/dc3/ff0 x:0 0 0 2024-04-22T15:49:37.505 INFO:tasks.workunit.client.0.smithi016.stdout:7/488: dwrite d3/dc/d1b/d31/d5f/f7b [0,4194304] 0 2024-04-22T15:49:37.506 INFO:tasks.workunit.client.0.smithi016.stdout:7/489: write d3/dc/f9a [921833,63060] 0 2024-04-22T15:49:37.518 INFO:tasks.workunit.client.0.smithi016.stdout:7/490: creat d3/d14/fb6 x:0 0 0 2024-04-22T15:49:37.521 INFO:tasks.workunit.client.0.smithi016.stdout:1/694: symlink d3/d4/d19/d20/d2b/dd6/lf1 0 2024-04-22T15:49:37.522 INFO:tasks.workunit.client.0.smithi016.stdout:5/470: link df/l5a df/d15/d1a/d50/d77/l89 0 2024-04-22T15:49:37.524 INFO:tasks.workunit.client.0.smithi016.stdout:7/491: symlink d3/dc/d1b/d5a/d69/lb7 0 2024-04-22T15:49:37.525 INFO:tasks.workunit.client.0.smithi016.stdout:7/492: write d3/dc/d1b/d5a/d69/f90 [346093,64632] 0 2024-04-22T15:49:37.526 INFO:tasks.workunit.client.0.smithi016.stdout:3/520: sync 2024-04-22T15:49:37.527 INFO:tasks.workunit.client.0.smithi016.stdout:7/493: fsync d3/dc/d1b/d5a/d9/df/faa 0 2024-04-22T15:49:37.532 INFO:tasks.workunit.client.0.smithi016.stdout:5/471: dread df/d15/d1a/d50/d77/f39 [0,4194304] 0 2024-04-22T15:49:37.533 INFO:tasks.workunit.client.0.smithi016.stdout:7/494: creat d3/d14/fb8 x:0 0 0 2024-04-22T15:49:37.534 INFO:tasks.workunit.client.0.smithi016.stdout:5/472: readlink df/d15/d1a/d50/d77/l44 0 2024-04-22T15:49:37.537 INFO:tasks.workunit.client.0.smithi016.stdout:0/534: dwrite dc/d13/d28/d65/d30/f44 [0,4194304] 0 2024-04-22T15:49:37.537 INFO:tasks.workunit.client.0.smithi016.stdout:7/495: dread d3/dc/d1b/d5a/d69/f90 [0,4194304] 0 2024-04-22T15:49:37.538 INFO:tasks.workunit.client.0.smithi016.stdout:0/535: write dc/d13/d28/f50 [180311,120405] 0 2024-04-22T15:49:37.549 INFO:tasks.workunit.client.0.smithi016.stdout:3/521: write d7/d17/d18/d32/f57 [763942,98051] 0 2024-04-22T15:49:37.550 INFO:tasks.workunit.client.0.smithi016.stdout:3/522: write d7/d17/f3e [2244190,110565] 0 2024-04-22T15:49:37.550 INFO:tasks.workunit.client.0.smithi016.stdout:0/536: symlink dc/d13/d22/la5 0 2024-04-22T15:49:37.552 INFO:tasks.workunit.client.0.smithi016.stdout:7/496: mknod d3/cb9 0 2024-04-22T15:49:37.557 INFO:tasks.workunit.client.0.smithi016.stdout:0/537: mkdir dc/d13/d70/da6 0 2024-04-22T15:49:37.560 INFO:tasks.workunit.client.0.smithi016.stdout:3/523: rmdir d7/d17/d18/d32/d42 39 2024-04-22T15:49:37.562 INFO:tasks.workunit.client.0.smithi016.stdout:7/497: unlink d3/dc/d1b/d5a/d4a/d6e/fa2 0 2024-04-22T15:49:37.563 INFO:tasks.workunit.client.0.smithi016.stdout:5/473: getdents df/d5b/d5d/d5f 0 2024-04-22T15:49:37.564 INFO:tasks.workunit.client.0.smithi016.stdout:5/474: stat df/d15/d1a/d4e/l5e 0 2024-04-22T15:49:37.570 INFO:tasks.workunit.client.0.smithi016.stdout:3/524: dwrite d7/f5a [0,4194304] 0 2024-04-22T15:49:37.571 INFO:tasks.workunit.client.0.smithi016.stdout:5/475: mknod df/d15/d1a/d50/d77/c8a 0 2024-04-22T15:49:37.572 INFO:tasks.workunit.client.0.smithi016.stdout:3/525: dread - d7/d17/d18/d32/f9f zero size 2024-04-22T15:49:37.578 INFO:tasks.workunit.client.0.smithi016.stdout:5/476: dwrite df/d15/d1a/d4e/f72 [0,4194304] 0 2024-04-22T15:49:37.579 INFO:tasks.workunit.client.0.smithi016.stdout:0/538: creat dc/d13/d28/d65/fa7 x:0 0 0 2024-04-22T15:49:37.579 INFO:tasks.workunit.client.0.smithi016.stdout:3/526: dwrite d7/f15 [4194304,4194304] 0 2024-04-22T15:49:37.587 INFO:tasks.workunit.client.0.smithi016.stdout:5/477: chown df/d15/d1a/d28/l79 1573 1 2024-04-22T15:49:37.587 INFO:tasks.workunit.client.0.smithi016.stdout:5/478: stat df/d15/d1a/f78 0 2024-04-22T15:49:37.592 INFO:tasks.workunit.client.0.smithi016.stdout:3/527: creat d7/d17/d18/d32/d8a/fa8 x:0 0 0 2024-04-22T15:49:37.593 INFO:tasks.workunit.client.0.smithi016.stdout:7/498: link d3/dc/d33/c81 d3/dc/d1b/d5a/cba 0 2024-04-22T15:49:37.594 INFO:tasks.workunit.client.0.smithi016.stdout:5/479: dwrite df/d15/d1a/d50/d77/f39 [4194304,4194304] 0 2024-04-22T15:49:37.594 INFO:tasks.workunit.client.0.smithi016.stdout:7/499: fdatasync d3/dc/d1b/d5a/d4a/d6e/f73 0 2024-04-22T15:49:37.596 INFO:tasks.workunit.client.0.smithi016.stdout:0/539: creat dc/d13/d14/d55/fa8 x:0 0 0 2024-04-22T15:49:37.597 INFO:tasks.workunit.client.0.smithi016.stdout:7/500: fdatasync d3/dc/d1b/d5a/d9/df/f11 0 2024-04-22T15:49:37.599 INFO:tasks.workunit.client.0.smithi016.stdout:3/528: truncate d7/d17/d18/d32/d42/d68/f7a 952516 0 2024-04-22T15:49:37.599 INFO:tasks.workunit.client.0.smithi016.stdout:5/480: mkdir df/d15/d1a/d50/d77/d7d/d8b 0 2024-04-22T15:49:37.601 INFO:tasks.workunit.client.0.smithi016.stdout:7/501: write d3/dc/d1b/d5a/d59/d64/f6c [349466,112483] 0 2024-04-22T15:49:37.601 INFO:tasks.workunit.client.0.smithi016.stdout:3/529: dread - d7/d17/d18/d32/d8a/f9b zero size 2024-04-22T15:49:37.603 INFO:tasks.workunit.client.0.smithi016.stdout:7/502: write d3/d14/f7c [2340605,99382] 0 2024-04-22T15:49:37.606 INFO:tasks.workunit.client.0.smithi016.stdout:0/540: dwrite dc/d13/d28/d65/d57/fa2 [0,4194304] 0 2024-04-22T15:49:37.607 INFO:tasks.workunit.client.0.smithi016.stdout:0/541: fsync dc/d13/d28/d65/d30/f84 0 2024-04-22T15:49:37.608 INFO:tasks.workunit.client.0.smithi016.stdout:3/530: rmdir d7/d17/d18 39 2024-04-22T15:49:37.613 INFO:tasks.workunit.client.0.smithi016.stdout:5/481: mkdir df/d15/d1a/d50/d77/d7d/d8b/d8c 0 2024-04-22T15:49:37.616 INFO:tasks.workunit.client.0.smithi016.stdout:3/531: fdatasync f1 0 2024-04-22T15:49:37.626 INFO:tasks.workunit.client.0.smithi016.stdout:3/532: truncate d7/d17/d18/d27/fa3 280240 0 2024-04-22T15:49:37.633 INFO:tasks.workunit.client.0.smithi016.stdout:0/542: creat dc/d13/d14/d26/fa9 x:0 0 0 2024-04-22T15:49:37.634 INFO:tasks.workunit.client.0.smithi016.stdout:7/503: getdents d3/dc 0 2024-04-22T15:49:37.636 INFO:tasks.workunit.client.0.smithi016.stdout:3/533: creat d7/d17/d18/d32/d42/d68/fa9 x:0 0 0 2024-04-22T15:49:37.636 INFO:tasks.workunit.client.0.smithi016.stdout:3/534: stat d7/d17/d18/f23 0 2024-04-22T15:49:37.641 INFO:tasks.workunit.client.0.smithi016.stdout:5/482: truncate df/d15/d1a/d28/f2c 7778770 0 2024-04-22T15:49:37.646 INFO:tasks.workunit.client.0.smithi016.stdout:3/535: dread d7/f5a [8388608,4194304] 0 2024-04-22T15:49:37.648 INFO:tasks.workunit.client.0.smithi016.stdout:5/483: creat df/d5b/d70/f8d x:0 0 0 2024-04-22T15:49:37.649 INFO:tasks.workunit.client.0.smithi016.stdout:7/504: getdents d3/dc/d33/d84 0 2024-04-22T15:49:37.653 INFO:tasks.workunit.client.0.smithi016.stdout:7/505: creat d3/dc/d9f/da1/fbb x:0 0 0 2024-04-22T15:49:37.654 INFO:tasks.workunit.client.0.smithi016.stdout:5/484: dwrite df/d15/d1a/f71 [0,4194304] 0 2024-04-22T15:49:37.655 INFO:tasks.workunit.client.0.smithi016.stdout:0/543: dread dc/d13/d14/f18 [0,4194304] 0 2024-04-22T15:49:37.656 INFO:tasks.workunit.client.0.smithi016.stdout:3/536: unlink d7/d17/d18/d32/d3c/f6c 0 2024-04-22T15:49:37.657 INFO:tasks.workunit.client.0.smithi016.stdout:2/429: sync 2024-04-22T15:49:37.662 INFO:tasks.workunit.client.0.smithi016.stdout:5/485: dwrite df/d15/d1a/d50/d77/f61 [0,4194304] 0 2024-04-22T15:49:37.667 INFO:tasks.workunit.client.0.smithi016.stdout:2/430: creat d2/d2c/d97/fa4 x:0 0 0 2024-04-22T15:49:37.675 INFO:tasks.workunit.client.0.smithi016.stdout:5/486: creat df/d5b/d5d/f8e x:0 0 0 2024-04-22T15:49:37.677 INFO:tasks.workunit.client.0.smithi016.stdout:0/544: symlink dc/d13/d70/da6/laa 0 2024-04-22T15:49:37.678 INFO:tasks.workunit.client.0.smithi016.stdout:0/545: fsync dc/d13/d5d/f85 0 2024-04-22T15:49:37.682 INFO:tasks.workunit.client.0.smithi016.stdout:2/431: unlink d2/de/d2a/d59/c69 0 2024-04-22T15:49:37.687 INFO:tasks.workunit.client.0.smithi016.stdout:7/506: rename d3/dc/d1b/d31/d5f/f63 to d3/fbc 0 2024-04-22T15:49:37.692 INFO:tasks.workunit.client.0.smithi016.stdout:2/432: creat d2/d2c/d97/d34/fa5 x:0 0 0 2024-04-22T15:49:37.696 INFO:tasks.workunit.client.0.smithi016.stdout:7/507: dread d3/dc/d1b/d5a/d4a/f55 [0,4194304] 0 2024-04-22T15:49:37.699 INFO:tasks.workunit.client.0.smithi016.stdout:5/487: creat df/d15/d1a/d28/f8f x:0 0 0 2024-04-22T15:49:37.707 INFO:tasks.workunit.client.0.smithi016.stdout:3/537: truncate d7/d1b/f4b 3851526 0 2024-04-22T15:49:37.708 INFO:tasks.workunit.client.0.smithi016.stdout:3/538: stat d7/d1b/d21/d55/f8e 0 2024-04-22T15:49:37.708 INFO:tasks.workunit.client.0.smithi016.stdout:5/488: mkdir df/d90 0 2024-04-22T15:49:37.713 INFO:tasks.workunit.client.0.smithi016.stdout:2/433: getdents d2/d4d/d64 0 2024-04-22T15:49:37.713 INFO:tasks.workunit.client.0.smithi016.stdout:3/539: unlink d7/d1b/d21/d55/f8c 0 2024-04-22T15:49:37.715 INFO:tasks.workunit.client.0.smithi016.stdout:3/540: readlink d7/d17/d18/d32/d3c/d51/l64 0 2024-04-22T15:49:37.719 INFO:tasks.workunit.client.0.smithi016.stdout:5/489: dwrite df/d15/d1a/d28/f8f [0,4194304] 0 2024-04-22T15:49:37.719 INFO:tasks.workunit.client.0.smithi016.stdout:0/546: dwrite dc/d13/f17 [4194304,4194304] 0 2024-04-22T15:49:37.727 INFO:tasks.workunit.client.0.smithi016.stdout:5/490: mknod df/d22/c91 0 2024-04-22T15:49:37.729 INFO:tasks.workunit.client.0.smithi016.stdout:3/541: dwrite d7/d17/d18/f5b [0,4194304] 0 2024-04-22T15:49:37.730 INFO:tasks.workunit.client.0.smithi016.stdout:7/508: dwrite d3/dc/d33/f3e [4194304,4194304] 0 2024-04-22T15:49:37.732 INFO:tasks.workunit.client.0.smithi016.stdout:0/547: unlink dc/d13/d28/d65/d40/d4a/d61/c88 0 2024-04-22T15:49:37.733 INFO:tasks.workunit.client.0.smithi016.stdout:0/548: write dc/d13/d28/d65/d57/f8d [4720223,66772] 0 2024-04-22T15:49:37.737 INFO:tasks.workunit.client.0.smithi016.stdout:7/509: creat d3/dc/d9f/fbd x:0 0 0 2024-04-22T15:49:37.745 INFO:tasks.workunit.client.0.smithi016.stdout:7/510: dwrite d3/dc/d1b/d5a/d9/f2a [0,4194304] 0 2024-04-22T15:49:37.749 INFO:tasks.workunit.client.0.smithi016.stdout:0/549: dread dc/d13/d70/f72 [0,4194304] 0 2024-04-22T15:49:37.754 INFO:tasks.workunit.client.0.smithi016.stdout:0/550: dread dc/d13/d28/d65/d30/f44 [0,4194304] 0 2024-04-22T15:49:37.754 INFO:tasks.workunit.client.0.smithi016.stdout:3/542: creat d7/d17/d18/d32/d3c/faa x:0 0 0 2024-04-22T15:49:37.756 INFO:tasks.workunit.client.0.smithi016.stdout:5/491: link df/d15/d1a/d4e/d58/d87/l88 df/d22/l92 0 2024-04-22T15:49:37.758 INFO:tasks.workunit.client.0.smithi016.stdout:0/551: creat dc/d7d/fab x:0 0 0 2024-04-22T15:49:37.759 INFO:tasks.workunit.client.0.smithi016.stdout:0/552: dread - dc/d13/d14/d55/fa8 zero size 2024-04-22T15:49:37.760 INFO:tasks.workunit.client.0.smithi016.stdout:5/492: mkdir df/d15/d1a/d50/d77/d7d/d8b/d93 0 2024-04-22T15:49:37.761 INFO:tasks.workunit.client.0.smithi016.stdout:7/511: dread d3/dc/d1b/d5a/d9/df/f11 [4194304,4194304] 0 2024-04-22T15:49:37.762 INFO:tasks.workunit.client.0.smithi016.stdout:0/553: creat dc/d13/d28/d65/d40/fac x:0 0 0 2024-04-22T15:49:37.764 INFO:tasks.workunit.client.0.smithi016.stdout:5/493: mkdir df/d15/d1a/d50/d77/d94 0 2024-04-22T15:49:37.766 INFO:tasks.workunit.client.0.smithi016.stdout:7/512: unlink d3/fbc 0 2024-04-22T15:49:37.766 INFO:tasks.workunit.client.0.smithi016.stdout:4/486: sync 2024-04-22T15:49:37.768 INFO:tasks.workunit.client.0.smithi016.stdout:7/513: chown d3/dc/d1b/d5a/d9/df/f11 4 1 2024-04-22T15:49:37.769 INFO:tasks.workunit.client.0.smithi016.stdout:0/554: write dc/d13/d28/d65/f42 [4819322,98346] 0 2024-04-22T15:49:37.769 INFO:tasks.workunit.client.0.smithi016.stdout:5/494: rmdir df/d5b/d5d/d5f 39 2024-04-22T15:49:37.770 INFO:tasks.workunit.client.0.smithi016.stdout:4/487: creat d7/d21/d45/d49/d9b/fa2 x:0 0 0 2024-04-22T15:49:37.771 INFO:tasks.workunit.client.0.smithi016.stdout:2/434: dwrite d2/d2c/d97/d67/d88/f95 [0,4194304] 0 2024-04-22T15:49:37.772 INFO:tasks.workunit.client.0.smithi016.stdout:4/488: fsync d7/d22/f73 0 2024-04-22T15:49:37.772 INFO:tasks.workunit.client.0.smithi016.stdout:5/495: write df/d15/d1a/d50/d77/d7a/f80 [813907,42992] 0 2024-04-22T15:49:37.774 INFO:tasks.workunit.client.0.smithi016.stdout:4/489: chown d7/d21/d45/c7f 330 1 2024-04-22T15:49:37.776 INFO:tasks.workunit.client.0.smithi016.stdout:2/435: write d2/d2c/d97/d67/fa0 [815698,92360] 0 2024-04-22T15:49:37.778 INFO:tasks.workunit.client.0.smithi016.stdout:2/436: fdatasync d2/d2c/f9e 0 2024-04-22T15:49:37.780 INFO:tasks.workunit.client.0.smithi016.stdout:4/490: symlink d7/d24/d51/la3 0 2024-04-22T15:49:37.781 INFO:tasks.workunit.client.0.smithi016.stdout:0/555: dwrite dc/d13/d28/d65/d40/f56 [0,4194304] 0 2024-04-22T15:49:37.781 INFO:tasks.workunit.client.0.smithi016.stdout:5/496: mknod df/d15/d1a/d4e/c95 0 2024-04-22T15:49:37.783 INFO:tasks.workunit.client.0.smithi016.stdout:4/491: stat d7/d21/d42/d59/d88/ca0 0 2024-04-22T15:49:37.784 INFO:tasks.workunit.client.0.smithi016.stdout:0/556: fsync dc/d13/d14/d55/f93 0 2024-04-22T15:49:37.785 INFO:tasks.workunit.client.0.smithi016.stdout:5/497: fsync df/d5b/d5d/d5f/f6f 0 2024-04-22T15:49:37.787 INFO:tasks.workunit.client.0.smithi016.stdout:4/492: creat d7/d4e/d39/fa4 x:0 0 0 2024-04-22T15:49:37.787 INFO:tasks.workunit.client.0.smithi016.stdout:2/437: dwrite d2/d4d/d76/f9a [0,4194304] 0 2024-04-22T15:49:37.789 INFO:tasks.workunit.client.0.smithi016.stdout:0/557: mknod dc/d13/d28/d65/d40/d4a/cad 0 2024-04-22T15:49:37.790 INFO:tasks.workunit.client.0.smithi016.stdout:5/498: mknod df/d5b/d5d/d5f/c96 0 2024-04-22T15:49:37.795 INFO:tasks.workunit.client.0.smithi016.stdout:2/438: link d2/d1a/d4c/f7a d2/d23/d92/d55/fa6 0 2024-04-22T15:49:37.797 INFO:tasks.workunit.client.0.smithi016.stdout:0/558: rename dc/d13/d28/d65/d40/f76 to dc/d13/d28/d65/d30/d96/fae 0 2024-04-22T15:49:37.797 INFO:tasks.workunit.client.0.smithi016.stdout:5/499: dread df/d15/d1a/d4e/f72 [0,4194304] 0 2024-04-22T15:49:37.800 INFO:tasks.workunit.client.0.smithi016.stdout:2/439: creat d2/d23/d92/d55/fa7 x:0 0 0 2024-04-22T15:49:37.801 INFO:tasks.workunit.client.0.smithi016.stdout:0/559: stat dc/d13/d14/c20 0 2024-04-22T15:49:37.804 INFO:tasks.workunit.client.0.smithi016.stdout:2/440: rename d2/d2c/f9e to d2/d9/fa8 0 2024-04-22T15:49:37.805 INFO:tasks.workunit.client.0.smithi016.stdout:5/500: dwrite df/d22/f24 [0,4194304] 0 2024-04-22T15:49:37.808 INFO:tasks.workunit.client.0.smithi016.stdout:4/493: dread d7/f16 [0,4194304] 0 2024-04-22T15:49:37.814 INFO:tasks.workunit.client.0.smithi016.stdout:5/501: dwrite df/d15/d1a/d50/d77/d7a/f80 [0,4194304] 0 2024-04-22T15:49:37.819 INFO:tasks.workunit.client.0.smithi016.stdout:5/502: rmdir df/d15/d1a/d4e 39 2024-04-22T15:49:37.822 INFO:tasks.workunit.client.0.smithi016.stdout:5/503: chown df/d15/d1a/d50/d77/l89 700150557 1 2024-04-22T15:49:37.833 INFO:tasks.workunit.client.0.smithi016.stdout:0/560: truncate dc/d13/d6f/d8b/f9a 2001038 0 2024-04-22T15:49:37.836 INFO:tasks.workunit.client.0.smithi016.stdout:2/441: dwrite d2/d1a/d4c/f3f [0,4194304] 0 2024-04-22T15:49:37.836 INFO:tasks.workunit.client.0.smithi016.stdout:0/561: mkdir dc/d13/d6f/daf 0 2024-04-22T15:49:37.837 INFO:tasks.workunit.client.0.smithi016.stdout:5/504: truncate df/d15/d1a/f6c 1254981 0 2024-04-22T15:49:37.838 INFO:tasks.workunit.client.0.smithi016.stdout:2/442: dread - d2/d2c/d97/f5f zero size 2024-04-22T15:49:37.843 INFO:tasks.workunit.client.0.smithi016.stdout:5/505: dwrite df/f10 [4194304,4194304] 0 2024-04-22T15:49:37.845 INFO:tasks.workunit.client.0.smithi016.stdout:5/506: read df/d15/d1a/d50/d77/f39 [6098249,102361] 0 2024-04-22T15:49:37.846 INFO:tasks.workunit.client.0.smithi016.stdout:2/443: symlink d2/d2c/d97/d34/la9 0 2024-04-22T15:49:37.847 INFO:tasks.workunit.client.0.smithi016.stdout:2/444: write d2/d2c/d97/d67/f86 [437232,85420] 0 2024-04-22T15:49:37.852 INFO:tasks.workunit.client.0.smithi016.stdout:2/445: read d2/d1a/f5e [387500,8423] 0 2024-04-22T15:49:37.853 INFO:tasks.workunit.client.0.smithi016.stdout:2/446: chown d2/d9/fa8 37 1 2024-04-22T15:49:37.854 INFO:tasks.workunit.client.0.smithi016.stdout:0/562: rmdir dc/d13/d28/d65/d57/d97 0 2024-04-22T15:49:37.855 INFO:tasks.workunit.client.0.smithi016.stdout:0/563: write dc/d7d/fab [713946,43873] 0 2024-04-22T15:49:37.858 INFO:tasks.workunit.client.0.smithi016.stdout:5/507: creat df/d15/d1a/d50/d77/d7d/d8b/d93/f97 x:0 0 0 2024-04-22T15:49:37.858 INFO:tasks.workunit.client.0.smithi016.stdout:5/508: stat df/d22/f3b 0 2024-04-22T15:49:37.859 INFO:tasks.workunit.client.0.smithi016.stdout:2/447: mkdir d2/d2c/daa 0 2024-04-22T15:49:37.860 INFO:tasks.workunit.client.0.smithi016.stdout:5/509: write df/f10 [4599221,20616] 0 2024-04-22T15:49:37.862 INFO:tasks.workunit.client.0.smithi016.stdout:0/564: dwrite dc/d13/f16 [0,4194304] 0 2024-04-22T15:49:37.866 INFO:tasks.workunit.client.0.smithi016.stdout:0/565: creat dc/d13/d14/d26/d32/fb0 x:0 0 0 2024-04-22T15:49:37.867 INFO:tasks.workunit.client.0.smithi016.stdout:0/566: write dc/f1e [4159828,6284] 0 2024-04-22T15:49:37.868 INFO:tasks.workunit.client.0.smithi016.stdout:0/567: fsync dc/d13/d28/d65/d30/f84 0 2024-04-22T15:49:37.875 INFO:tasks.workunit.client.0.smithi016.stdout:5/510: rmdir df/d5b/d85 0 2024-04-22T15:49:37.879 INFO:tasks.workunit.client.0.smithi016.stdout:5/511: creat df/d5b/f98 x:0 0 0 2024-04-22T15:49:37.879 INFO:tasks.workunit.client.0.smithi016.stdout:0/568: link dc/d13/d14/l9f dc/d13/d99/lb1 0 2024-04-22T15:49:37.881 INFO:tasks.workunit.client.0.smithi016.stdout:0/569: mkdir dc/d13/d28/d65/d40/d4a/db2 0 2024-04-22T15:49:37.900 INFO:tasks.workunit.client.0.smithi016.stdout:1/695: sync 2024-04-22T15:49:37.900 INFO:tasks.workunit.client.0.smithi016.stdout:8/460: sync 2024-04-22T15:49:37.900 INFO:tasks.workunit.client.0.smithi016.stdout:9/500: sync 2024-04-22T15:49:37.900 INFO:tasks.workunit.client.0.smithi016.stdout:6/550: sync 2024-04-22T15:49:37.914 INFO:tasks.workunit.client.0.smithi016.stdout:5/512: creat df/d15/d1a/f99 x:0 0 0 2024-04-22T15:49:37.918 INFO:tasks.workunit.client.0.smithi016.stdout:6/551: fdatasync d3/da/d12/d13/fac 0 2024-04-22T15:49:37.919 INFO:tasks.workunit.client.0.smithi016.stdout:0/570: dread dc/d13/d14/d26/f4f [0,4194304] 0 2024-04-22T15:49:37.922 INFO:tasks.workunit.client.0.smithi016.stdout:6/552: write d3/da/dd/f18 [614752,129514] 0 2024-04-22T15:49:37.925 INFO:tasks.workunit.client.0.smithi016.stdout:2/448: dwrite d2/d2c/d97/f2e [0,4194304] 0 2024-04-22T15:49:37.926 INFO:tasks.workunit.client.0.smithi016.stdout:1/696: creat d3/d4/d3a/d6b/d77/de2/ff2 x:0 0 0 2024-04-22T15:49:37.927 INFO:tasks.workunit.client.0.smithi016.stdout:9/501: dwrite d9/de/d23/d24/d25/f8b [0,4194304] 0 2024-04-22T15:49:37.929 INFO:tasks.workunit.client.0.smithi016.stdout:9/502: write d9/de/d23/d24/d25/d36/f6b [154779,114863] 0 2024-04-22T15:49:37.933 INFO:tasks.workunit.client.0.smithi016.stdout:0/571: creat dc/d13/d28/d65/d1f/fb3 x:0 0 0 2024-04-22T15:49:37.936 INFO:tasks.workunit.client.0.smithi016.stdout:9/503: dwrite d9/d37/f49 [4194304,4194304] 0 2024-04-22T15:49:37.939 INFO:tasks.workunit.client.0.smithi016.stdout:2/449: unlink d2/d2c/d97/f8c 0 2024-04-22T15:49:37.946 INFO:tasks.workunit.client.0.smithi016.stdout:5/513: rename df/d15/d1a/d28/f6d to df/f9a 0 2024-04-22T15:49:37.948 INFO:tasks.workunit.client.0.smithi016.stdout:0/572: creat dc/d13/d14/d26/d32/fb4 x:0 0 0 2024-04-22T15:49:37.953 INFO:tasks.workunit.client.0.smithi016.stdout:8/461: write d2/f8 [708576,89817] 0 2024-04-22T15:49:37.956 INFO:tasks.workunit.client.0.smithi016.stdout:6/553: creat d3/da/dd/fb5 x:0 0 0 2024-04-22T15:49:37.957 INFO:tasks.workunit.client.0.smithi016.stdout:8/462: dread d2/f5a [0,4194304] 0 2024-04-22T15:49:37.957 INFO:tasks.workunit.client.0.smithi016.stdout:1/697: dread d3/d4/fe [0,4194304] 0 2024-04-22T15:49:37.959 INFO:tasks.workunit.client.0.smithi016.stdout:9/504: rename d9/de/d26/d5a/l84 to d9/de/d26/d5a/l90 0 2024-04-22T15:49:37.960 INFO:tasks.workunit.client.0.smithi016.stdout:6/554: chown d3/da/db/d14/d4c/l66 809 1 2024-04-22T15:49:37.960 INFO:tasks.workunit.client.0.smithi016.stdout:9/505: readlink d9/de/l21 0 2024-04-22T15:49:37.961 INFO:tasks.workunit.client.0.smithi016.stdout:2/450: dwrite d2/d2c/d97/d67/d88/f8a [0,4194304] 0 2024-04-22T15:49:37.962 INFO:tasks.workunit.client.0.smithi016.stdout:0/573: creat dc/d13/d28/d65/d40/d4a/d61/d82/fb5 x:0 0 0 2024-04-22T15:49:37.963 INFO:tasks.workunit.client.0.smithi016.stdout:2/451: chown d2/d4d 140517303 1 2024-04-22T15:49:37.965 INFO:tasks.workunit.client.0.smithi016.stdout:1/698: dread d3/d4/d19/f35 [0,4194304] 0 2024-04-22T15:49:37.966 INFO:tasks.workunit.client.0.smithi016.stdout:1/699: stat d3/d4/d37/f40 0 2024-04-22T15:49:37.967 INFO:tasks.workunit.client.0.smithi016.stdout:5/514: creat df/f9b x:0 0 0 2024-04-22T15:49:37.968 INFO:tasks.workunit.client.0.smithi016.stdout:5/515: chown df/d15/d1a/d28/f74 11867 1 2024-04-22T15:49:37.969 INFO:tasks.workunit.client.0.smithi016.stdout:3/543: sync 2024-04-22T15:49:37.971 INFO:tasks.workunit.client.0.smithi016.stdout:6/555: mknod d3/da/d12/d13/d38/cb6 0 2024-04-22T15:49:37.972 INFO:tasks.workunit.client.0.smithi016.stdout:3/544: dread d7/f5a [8388608,4194304] 0 2024-04-22T15:49:37.976 INFO:tasks.workunit.client.0.smithi016.stdout:2/452: rename d2/d23/d92/l85 to d2/d2c/d97/lab 0 2024-04-22T15:49:37.980 INFO:tasks.workunit.client.0.smithi016.stdout:8/463: link d2/d10/f39 d2/d82/d44/d66/d1c/f8c 0 2024-04-22T15:49:37.983 INFO:tasks.workunit.client.0.smithi016.stdout:0/574: symlink dc/d13/lb6 0 2024-04-22T15:49:37.986 INFO:tasks.workunit.client.0.smithi016.stdout:2/453: mknod d2/d4d/d76/cac 0 2024-04-22T15:49:37.987 INFO:tasks.workunit.client.0.smithi016.stdout:2/454: readlink d2/l3 0 2024-04-22T15:49:37.988 INFO:tasks.workunit.client.0.smithi016.stdout:8/464: mknod d2/d32/c8d 0 2024-04-22T15:49:37.991 INFO:tasks.workunit.client.0.smithi016.stdout:9/506: dwrite d9/de/f16 [0,4194304] 0 2024-04-22T15:49:37.993 INFO:tasks.workunit.client.0.smithi016.stdout:0/575: dwrite dc/f68 [0,4194304] 0 2024-04-22T15:49:37.993 INFO:tasks.workunit.client.0.smithi016.stdout:8/465: chown d2/d82/d44/d66/l1d 441067632 1 2024-04-22T15:49:37.993 INFO:tasks.workunit.client.0.smithi016.stdout:6/556: symlink d3/da/db/d14/lb7 0 2024-04-22T15:49:37.994 INFO:tasks.workunit.client.0.smithi016.stdout:3/545: mknod d7/d17/d18/d32/d42/d63/d8f/cab 0 2024-04-22T15:49:37.998 INFO:tasks.workunit.client.0.smithi016.stdout:8/466: truncate d2/f8 3027518 0 2024-04-22T15:49:37.998 INFO:tasks.workunit.client.0.smithi016.stdout:2/455: creat d2/d4d/d8e/fad x:0 0 0 2024-04-22T15:49:38.001 INFO:tasks.workunit.client.0.smithi016.stdout:9/507: mkdir d9/d57/d91 0 2024-04-22T15:49:38.002 INFO:tasks.workunit.client.0.smithi016.stdout:1/700: dwrite d3/d4/d15/f16 [4194304,4194304] 0 2024-04-22T15:49:38.002 INFO:tasks.workunit.client.0.smithi016.stdout:6/557: symlink d3/d53/d83/lb8 0 2024-04-22T15:49:38.005 INFO:tasks.workunit.client.0.smithi016.stdout:2/456: write d2/d2c/d97/d34/fa5 [112925,111859] 0 2024-04-22T15:49:38.008 INFO:tasks.workunit.client.0.smithi016.stdout:8/467: mkdir d2/d82/d3e/d8e 0 2024-04-22T15:49:38.008 INFO:tasks.workunit.client.0.smithi016.stdout:1/701: dread - d3/d4/d3a/d96/de4/fe3 zero size 2024-04-22T15:49:38.009 INFO:tasks.workunit.client.0.smithi016.stdout:2/457: chown d2/d9/fa8 836 1 2024-04-22T15:49:38.009 INFO:tasks.workunit.client.0.smithi016.stdout:5/516: dwrite df/d15/d1a/f6c [0,4194304] 0 2024-04-22T15:49:38.010 INFO:tasks.workunit.client.0.smithi016.stdout:9/508: rmdir d9/de/d23/d24/d25 39 2024-04-22T15:49:38.011 INFO:tasks.workunit.client.0.smithi016.stdout:1/702: stat d3/d4/d3a/d91/db3/fd4 0 2024-04-22T15:49:38.013 INFO:tasks.workunit.client.0.smithi016.stdout:1/703: dread - d3/d4/d3a/d96/fe9 zero size 2024-04-22T15:49:38.014 INFO:tasks.workunit.client.0.smithi016.stdout:7/514: sync 2024-04-22T15:49:38.014 INFO:tasks.workunit.client.0.smithi016.stdout:4/494: sync 2024-04-22T15:49:38.015 INFO:tasks.workunit.client.0.smithi016.stdout:0/576: rename dc/d13/d28/d65/d40/l6d to dc/d13/d28/d65/d40/d4a/d61/lb7 0 2024-04-22T15:49:38.019 INFO:tasks.workunit.client.0.smithi016.stdout:3/546: write d7/d17/d18/f2a [1687956,61092] 0 2024-04-22T15:49:38.020 INFO:tasks.workunit.client.0.smithi016.stdout:8/468: dread d2/d82/d44/d66/f3c [4194304,4194304] 0 2024-04-22T15:49:38.020 INFO:tasks.workunit.client.0.smithi016.stdout:6/558: mkdir d3/da/dd/d67/d6c/db9 0 2024-04-22T15:49:38.021 INFO:tasks.workunit.client.0.smithi016.stdout:2/458: rmdir d2/d4d/d76 39 2024-04-22T15:49:38.022 INFO:tasks.workunit.client.0.smithi016.stdout:9/509: dwrite d9/d5d/d62/f6e [4194304,4194304] 0 2024-04-22T15:49:38.024 INFO:tasks.workunit.client.0.smithi016.stdout:3/547: write d7/d1b/d21/d55/f79 [1455643,85671] 0 2024-04-22T15:49:38.024 INFO:tasks.workunit.client.0.smithi016.stdout:1/704: dread d3/d4/d19/f66 [0,4194304] 0 2024-04-22T15:49:38.026 INFO:tasks.workunit.client.0.smithi016.stdout:3/548: fsync d7/d17/d18/d32/d42/f9a 0 2024-04-22T15:49:38.027 INFO:tasks.workunit.client.0.smithi016.stdout:5/517: chown df/d15/l6b 906813052 1 2024-04-22T15:49:38.028 INFO:tasks.workunit.client.0.smithi016.stdout:7/515: creat d3/d14/fbe x:0 0 0 2024-04-22T15:49:38.029 INFO:tasks.workunit.client.0.smithi016.stdout:1/705: truncate d3/d4/d3a/d96/fe9 576652 0 2024-04-22T15:49:38.030 INFO:tasks.workunit.client.0.smithi016.stdout:5/518: chown df/d15/d1a/d50/d77/d7d/l86 3895679 1 2024-04-22T15:49:38.030 INFO:tasks.workunit.client.0.smithi016.stdout:3/549: chown d7/d17/d18/d32/d42/d63/f9d 44751079 1 2024-04-22T15:49:38.032 INFO:tasks.workunit.client.0.smithi016.stdout:8/469: rmdir d2/d32 39 2024-04-22T15:49:38.033 INFO:tasks.workunit.client.0.smithi016.stdout:7/516: readlink d3/dc/d1b/d5a/d9/df/d94/l5d 0 2024-04-22T15:49:38.034 INFO:tasks.workunit.client.0.smithi016.stdout:9/510: dwrite d9/de/d23/d24/f8e [0,4194304] 0 2024-04-22T15:49:38.034 INFO:tasks.workunit.client.0.smithi016.stdout:8/470: dread - d2/d82/d56/f71 zero size 2024-04-22T15:49:38.035 INFO:tasks.workunit.client.0.smithi016.stdout:6/559: symlink d3/da/d12/d13/lba 0 2024-04-22T15:49:38.036 INFO:tasks.workunit.client.0.smithi016.stdout:0/577: dread dc/d13/d14/f18 [0,4194304] 0 2024-04-22T15:49:38.038 INFO:tasks.workunit.client.0.smithi016.stdout:0/578: chown dc/d13 93043 1 2024-04-22T15:49:38.038 INFO:tasks.workunit.client.0.smithi016.stdout:2/459: mkdir d2/d1a/d4c/d9f/dae 0 2024-04-22T15:49:38.038 INFO:tasks.workunit.client.0.smithi016.stdout:6/560: truncate d3/da/f3e 505070 0 2024-04-22T15:49:38.040 INFO:tasks.workunit.client.0.smithi016.stdout:4/495: creat d7/d21/d42/d80/fa5 x:0 0 0 2024-04-22T15:49:38.041 INFO:tasks.workunit.client.0.smithi016.stdout:1/706: write d3/d2e/f8a [32582,80184] 0 2024-04-22T15:49:38.041 INFO:tasks.workunit.client.0.smithi016.stdout:3/550: fdatasync f1 0 2024-04-22T15:49:38.046 INFO:tasks.workunit.client.0.smithi016.stdout:1/707: chown d3/d4/dc4/fd7 242 1 2024-04-22T15:49:38.047 INFO:tasks.workunit.client.0.smithi016.stdout:9/511: write d9/de/d23/d24/d25/d36/f5e [1162262,7120] 0 2024-04-22T15:49:38.047 INFO:tasks.workunit.client.0.smithi016.stdout:8/471: dwrite d2/d82/f7b [0,4194304] 0 2024-04-22T15:49:38.049 INFO:tasks.workunit.client.0.smithi016.stdout:1/708: read d3/d4/d19/f1d [783227,44223] 0 2024-04-22T15:49:38.053 INFO:tasks.workunit.client.0.smithi016.stdout:2/460: dwrite d2/d2c/f5b [0,4194304] 0 2024-04-22T15:49:38.056 INFO:tasks.workunit.client.0.smithi016.stdout:5/519: truncate df/d15/d1a/f78 260153 0 2024-04-22T15:49:38.059 INFO:tasks.workunit.client.0.smithi016.stdout:1/709: dwrite d3/f8 [0,4194304] 0 2024-04-22T15:49:38.062 INFO:tasks.workunit.client.0.smithi016.stdout:0/579: rename dc/d13/d28/d65/d1f/f3a to dc/d13/d14/d55/fb8 0 2024-04-22T15:49:38.063 INFO:tasks.workunit.client.0.smithi016.stdout:1/710: truncate d3/d2e/d80/d82/fae 628797 0 2024-04-22T15:49:38.064 INFO:tasks.workunit.client.0.smithi016.stdout:9/512: creat d9/de/d23/d30/f92 x:0 0 0 2024-04-22T15:49:38.068 INFO:tasks.workunit.client.0.smithi016.stdout:6/561: link d3/da/dd/fb2 d3/da/dd/d7b/fbb 0 2024-04-22T15:49:38.070 INFO:tasks.workunit.client.0.smithi016.stdout:2/461: symlink d2/d23/d92/d55/laf 0 2024-04-22T15:49:38.070 INFO:tasks.workunit.client.0.smithi016.stdout:4/496: symlink d7/d21/d42/d59/d88/d9d/la6 0 2024-04-22T15:49:38.070 INFO:tasks.workunit.client.0.smithi016.stdout:5/520: dwrite df/d15/d1a/d28/f74 [0,4194304] 0 2024-04-22T15:49:38.072 INFO:tasks.workunit.client.0.smithi016.stdout:4/497: stat d7/f8 0 2024-04-22T15:49:38.072 INFO:tasks.workunit.client.0.smithi016.stdout:8/472: truncate d2/d10/f39 1840070 0 2024-04-22T15:49:38.073 INFO:tasks.workunit.client.0.smithi016.stdout:5/521: write df/d5b/d5d/f8e [327662,127803] 0 2024-04-22T15:49:38.074 INFO:tasks.workunit.client.0.smithi016.stdout:8/473: fsync d2/d82/f4d 0 2024-04-22T15:49:38.075 INFO:tasks.workunit.client.0.smithi016.stdout:4/498: truncate d7/d24/f76 739633 0 2024-04-22T15:49:38.078 INFO:tasks.workunit.client.0.smithi016.stdout:0/580: creat dc/d13/d28/d65/d30/d75/fb9 x:0 0 0 2024-04-22T15:49:38.084 INFO:tasks.workunit.client.0.smithi016.stdout:9/513: rename d9/de/d23/d24/d25/f8b to d9/d37/f93 0 2024-04-22T15:49:38.085 INFO:tasks.workunit.client.0.smithi016.stdout:6/562: creat d3/da/dd/d1b/d57/fbc x:0 0 0 2024-04-22T15:49:38.091 INFO:tasks.workunit.client.0.smithi016.stdout:3/551: dwrite d7/d17/d18/d27/f53 [0,4194304] 0 2024-04-22T15:49:38.091 INFO:tasks.workunit.client.0.smithi016.stdout:5/522: read df/d15/d1a/f53 [1967124,106503] 0 2024-04-22T15:49:38.093 INFO:tasks.workunit.client.0.smithi016.stdout:0/581: truncate dc/d13/d14/f18 942217 0 2024-04-22T15:49:38.094 INFO:tasks.workunit.client.0.smithi016.stdout:8/474: dread d2/d10/f41 [0,4194304] 0 2024-04-22T15:49:38.098 INFO:tasks.workunit.client.0.smithi016.stdout:2/462: rename d2/d4d/c57 to d2/d4d/d64/cb0 0 2024-04-22T15:49:38.100 INFO:tasks.workunit.client.0.smithi016.stdout:6/563: mkdir d3/da/dd/d67/dbd 0 2024-04-22T15:49:38.100 INFO:tasks.workunit.client.0.smithi016.stdout:4/499: dread d7/d4e/d2f/d6f/f84 [0,4194304] 0 2024-04-22T15:49:38.101 INFO:tasks.workunit.client.0.smithi016.stdout:1/711: dread d3/d4/d19/fb2 [0,4194304] 0 2024-04-22T15:49:38.105 INFO:tasks.workunit.client.0.smithi016.stdout:3/552: dwrite d7/f74 [0,4194304] 0 2024-04-22T15:49:38.105 INFO:tasks.workunit.client.0.smithi016.stdout:5/523: dwrite f2 [0,4194304] 0 2024-04-22T15:49:38.106 INFO:tasks.workunit.client.0.smithi016.stdout:5/524: truncate df/d15/f7e 45926 0 2024-04-22T15:49:38.108 INFO:tasks.workunit.client.0.smithi016.stdout:0/582: mkdir dc/d13/d14/d55/dba 0 2024-04-22T15:49:38.108 INFO:tasks.workunit.client.0.smithi016.stdout:0/583: fsync f4 0 2024-04-22T15:49:38.114 INFO:tasks.workunit.client.0.smithi016.stdout:2/463: readlink d2/d9/l36 0 2024-04-22T15:49:38.116 INFO:tasks.workunit.client.0.smithi016.stdout:2/464: write d2/d2c/d97/d34/fa5 [1058647,123482] 0 2024-04-22T15:49:38.118 INFO:tasks.workunit.client.0.smithi016.stdout:8/475: truncate d2/d30/f73 1217810 0 2024-04-22T15:49:38.120 INFO:tasks.workunit.client.0.smithi016.stdout:4/500: mknod d7/d21/d42/d59/ca7 0 2024-04-22T15:49:38.122 INFO:tasks.workunit.client.0.smithi016.stdout:4/501: truncate d7/d24/d41/f94 986053 0 2024-04-22T15:49:38.125 INFO:tasks.workunit.client.0.smithi016.stdout:3/553: mkdir d7/d1b/d21/d55/dac 0 2024-04-22T15:49:38.126 INFO:tasks.workunit.client.0.smithi016.stdout:5/525: creat df/d22/f9c x:0 0 0 2024-04-22T15:49:38.126 INFO:tasks.workunit.client.0.smithi016.stdout:3/554: dread - d7/d1b/d21/d55/f6f zero size 2024-04-22T15:49:38.129 INFO:tasks.workunit.client.0.smithi016.stdout:0/584: rename dc/d13/d14/d55/d69 to dc/d13/d28/d65/d40/dbb 0 2024-04-22T15:49:38.130 INFO:tasks.workunit.client.0.smithi016.stdout:9/514: write d9/d20/f56 [1273346,83844] 0 2024-04-22T15:49:38.134 INFO:tasks.workunit.client.0.smithi016.stdout:8/476: mkdir d2/d10/d43/d4a/d8f 0 2024-04-22T15:49:38.135 INFO:tasks.workunit.client.0.smithi016.stdout:6/564: mkdir d3/da/dd/d67/d6c/db9/dbe 0 2024-04-22T15:49:38.135 INFO:tasks.workunit.client.0.smithi016.stdout:8/477: write d2/d82/d56/f64 [974183,49681] 0 2024-04-22T15:49:38.136 INFO:tasks.workunit.client.0.smithi016.stdout:1/712: creat d3/d4/d3a/d91/db3/dd2/ff3 x:0 0 0 2024-04-22T15:49:38.137 INFO:tasks.workunit.client.0.smithi016.stdout:1/713: stat d3/d4/da/f26 0 2024-04-22T15:49:38.138 INFO:tasks.workunit.client.0.smithi016.stdout:5/526: rename df/d15/d1a/l1b to df/d15/d1a/d50/l9d 0 2024-04-22T15:49:38.139 INFO:tasks.workunit.client.0.smithi016.stdout:0/585: creat dc/d13/d28/d65/d40/d4a/d59/d78/fbc x:0 0 0 2024-04-22T15:49:38.140 INFO:tasks.workunit.client.0.smithi016.stdout:2/465: symlink d2/d1a/d4c/d77/lb1 0 2024-04-22T15:49:38.142 INFO:tasks.workunit.client.0.smithi016.stdout:3/555: symlink d7/d77/lad 0 2024-04-22T15:49:38.143 INFO:tasks.workunit.client.0.smithi016.stdout:1/714: dwrite d3/d4/d3a/d6b/fc1 [0,4194304] 0 2024-04-22T15:49:38.143 INFO:tasks.workunit.client.0.smithi016.stdout:6/565: creat d3/da/dd/d1b/d57/fbf x:0 0 0 2024-04-22T15:49:38.145 INFO:tasks.workunit.client.0.smithi016.stdout:2/466: truncate d2/d2c/d97/d67/d88/f95 5126292 0 2024-04-22T15:49:38.147 INFO:tasks.workunit.client.0.smithi016.stdout:0/586: mkdir dc/d13/d28/d65/d40/d4a/d59/d78/dbd 0 2024-04-22T15:49:38.150 INFO:tasks.workunit.client.0.smithi016.stdout:6/566: creat d3/da/dd/d7b/fc0 x:0 0 0 2024-04-22T15:49:38.151 INFO:tasks.workunit.client.0.smithi016.stdout:8/478: dread d2/d32/f35 [0,4194304] 0 2024-04-22T15:49:38.153 INFO:tasks.workunit.client.0.smithi016.stdout:6/567: write d3/da/db/d14/f9b [180929,11387] 0 2024-04-22T15:49:38.153 INFO:tasks.workunit.client.0.smithi016.stdout:9/515: rename d9/de/d23/l7b to d9/de/d23/d24/d4e/d4f/l94 0 2024-04-22T15:49:38.155 INFO:tasks.workunit.client.0.smithi016.stdout:2/467: dwrite d2/d2c/d97/f2e [0,4194304] 0 2024-04-22T15:49:38.155 INFO:tasks.workunit.client.0.smithi016.stdout:6/568: fsync d3/da/db/d44/d68/f6a 0 2024-04-22T15:49:38.159 INFO:tasks.workunit.client.0.smithi016.stdout:0/587: creat dc/d13/d5d/fbe x:0 0 0 2024-04-22T15:49:38.160 INFO:tasks.workunit.client.0.smithi016.stdout:0/588: fdatasync dc/d13/d28/d65/d40/fac 0 2024-04-22T15:49:38.160 INFO:tasks.workunit.client.0.smithi016.stdout:3/556: getdents d7/d17/d18/d32/d3c/d51/d98 0 2024-04-22T15:49:38.162 INFO:tasks.workunit.client.0.smithi016.stdout:8/479: creat d2/d82/d56/f90 x:0 0 0 2024-04-22T15:49:38.170 INFO:tasks.workunit.client.0.smithi016.stdout:6/569: rename d3/da/d12/f15 to d3/da/db/d44/d68/d95/d9e/fc1 0 2024-04-22T15:49:38.171 INFO:tasks.workunit.client.0.smithi016.stdout:6/570: read d3/f8 [8678699,27714] 0 2024-04-22T15:49:38.172 INFO:tasks.workunit.client.0.smithi016.stdout:3/557: creat d7/d17/d18/fae x:0 0 0 2024-04-22T15:49:38.173 INFO:tasks.workunit.client.0.smithi016.stdout:3/558: truncate d7/d17/f80 4269819 0 2024-04-22T15:49:38.173 INFO:tasks.workunit.client.0.smithi016.stdout:8/480: mknod d2/d82/d44/d66/d1c/c91 0 2024-04-22T15:49:38.178 INFO:tasks.workunit.client.0.smithi016.stdout:7/517: sync 2024-04-22T15:49:38.183 INFO:tasks.workunit.client.0.smithi016.stdout:2/468: dread d2/f4f [0,4194304] 0 2024-04-22T15:49:38.183 INFO:tasks.workunit.client.0.smithi016.stdout:6/571: rename d3/da/d12/f9c to d3/d53/d35/fc2 0 2024-04-22T15:49:38.184 INFO:tasks.workunit.client.0.smithi016.stdout:2/469: chown d2/d1a/d7b/f8d 9 1 2024-04-22T15:49:38.187 INFO:tasks.workunit.client.0.smithi016.stdout:9/516: link d9/d37/l8c d9/d5d/l95 0 2024-04-22T15:49:38.189 INFO:tasks.workunit.client.0.smithi016.stdout:7/518: creat d3/dc/fbf x:0 0 0 2024-04-22T15:49:38.190 INFO:tasks.workunit.client.0.smithi016.stdout:1/715: dwrite d3/d4/d15/d38/d63/f75 [4194304,4194304] 0 2024-04-22T15:49:38.191 INFO:tasks.workunit.client.0.smithi016.stdout:3/559: symlink d7/laf 0 2024-04-22T15:49:38.195 INFO:tasks.workunit.client.0.smithi016.stdout:2/470: creat d2/de/d2a/d59/fb2 x:0 0 0 2024-04-22T15:49:38.199 INFO:tasks.workunit.client.0.smithi016.stdout:2/471: dread d2/d51/f5a [0,4194304] 0 2024-04-22T15:49:38.200 INFO:tasks.workunit.client.0.smithi016.stdout:1/716: symlink d3/d4/d3a/d91/db3/lf4 0 2024-04-22T15:49:38.201 INFO:tasks.workunit.client.0.smithi016.stdout:2/472: write d2/d4d/d8e/fad [453407,37447] 0 2024-04-22T15:49:38.202 INFO:tasks.workunit.client.0.smithi016.stdout:3/560: creat d7/d17/d18/d32/d42/d63/d8f/fb0 x:0 0 0 2024-04-22T15:49:38.204 INFO:tasks.workunit.client.0.smithi016.stdout:6/572: creat d3/da/fc3 x:0 0 0 2024-04-22T15:49:38.210 INFO:tasks.workunit.client.0.smithi016.stdout:3/561: rmdir d7/d17/d18/d27 39 2024-04-22T15:49:38.211 INFO:tasks.workunit.client.0.smithi016.stdout:8/481: dwrite d2/d48/f49 [0,4194304] 0 2024-04-22T15:49:38.211 INFO:tasks.workunit.client.0.smithi016.stdout:6/573: fdatasync d3/d53/d35/fc2 0 2024-04-22T15:49:38.212 INFO:tasks.workunit.client.0.smithi016.stdout:3/562: write d7/f74 [312675,53480] 0 2024-04-22T15:49:38.213 INFO:tasks.workunit.client.0.smithi016.stdout:6/574: write d3/da/dd/f9d [925580,21641] 0 2024-04-22T15:49:38.213 INFO:tasks.workunit.client.0.smithi016.stdout:3/563: truncate d7/d1b/f85 676594 0 2024-04-22T15:49:38.215 INFO:tasks.workunit.client.0.smithi016.stdout:3/564: dread - d7/d1b/d21/d55/fa6 zero size 2024-04-22T15:49:38.215 INFO:tasks.workunit.client.0.smithi016.stdout:2/473: creat d2/d1a/d4c/d9f/dae/fb3 x:0 0 0 2024-04-22T15:49:38.222 INFO:tasks.workunit.client.0.smithi016.stdout:8/482: mkdir d2/d32/d92 0 2024-04-22T15:49:38.223 INFO:tasks.workunit.client.0.smithi016.stdout:1/717: creat d3/d4/d3a/ff5 x:0 0 0 2024-04-22T15:49:38.224 INFO:tasks.workunit.client.0.smithi016.stdout:6/575: dread d3/da/dd/f5a [0,4194304] 0 2024-04-22T15:49:38.226 INFO:tasks.workunit.client.0.smithi016.stdout:2/474: symlink d2/d9/lb4 0 2024-04-22T15:49:38.226 INFO:tasks.workunit.client.0.smithi016.stdout:3/565: mknod d7/d17/d18/d32/d42/d63/d8f/cb1 0 2024-04-22T15:49:38.228 INFO:tasks.workunit.client.0.smithi016.stdout:7/519: creat d3/dc/d9f/da1/fc0 x:0 0 0 2024-04-22T15:49:38.229 INFO:tasks.workunit.client.0.smithi016.stdout:8/483: creat d2/d10/d43/d4a/d6a/f93 x:0 0 0 2024-04-22T15:49:38.230 INFO:tasks.workunit.client.0.smithi016.stdout:1/718: mknod d3/d4/d19/d20/cf6 0 2024-04-22T15:49:38.232 INFO:tasks.workunit.client.0.smithi016.stdout:6/576: stat d3/da/db/d14/d16/f87 0 2024-04-22T15:49:38.233 INFO:tasks.workunit.client.0.smithi016.stdout:6/577: dread d3/da/d12/d13/f76 [0,4194304] 0 2024-04-22T15:49:38.234 INFO:tasks.workunit.client.0.smithi016.stdout:2/475: creat d2/d1a/d4c/fb5 x:0 0 0 2024-04-22T15:49:38.235 INFO:tasks.workunit.client.0.smithi016.stdout:2/476: write d2/d4d/d8e/fad [1505449,53853] 0 2024-04-22T15:49:38.235 INFO:tasks.workunit.client.0.smithi016.stdout:6/578: truncate d3/da/dd/f82 1369269 0 2024-04-22T15:49:38.235 INFO:tasks.workunit.client.0.smithi016.stdout:3/566: symlink d7/d77/lb2 0 2024-04-22T15:49:38.237 INFO:tasks.workunit.client.0.smithi016.stdout:6/579: fdatasync d3/da/db/d14/d16/f55 0 2024-04-22T15:49:38.243 INFO:tasks.workunit.client.0.smithi016.stdout:2/477: mknod d2/d2c/d97/d67/cb6 0 2024-04-22T15:49:38.244 INFO:tasks.workunit.client.0.smithi016.stdout:7/520: dwrite d3/f8b [0,4194304] 0 2024-04-22T15:49:38.245 INFO:tasks.workunit.client.0.smithi016.stdout:8/484: creat d2/d82/f94 x:0 0 0 2024-04-22T15:49:38.246 INFO:tasks.workunit.client.0.smithi016.stdout:3/567: dread d7/d17/d18/f5b [0,4194304] 0 2024-04-22T15:49:38.251 INFO:tasks.workunit.client.0.smithi016.stdout:6/580: dwrite d3/d53/d35/fa2 [0,4194304] 0 2024-04-22T15:49:38.252 INFO:tasks.workunit.client.0.smithi016.stdout:2/478: symlink d2/d1a/d4c/d9f/dae/lb7 0 2024-04-22T15:49:38.252 INFO:tasks.workunit.client.0.smithi016.stdout:2/479: stat d2/d4d/f7f 0 2024-04-22T15:49:38.253 INFO:tasks.workunit.client.0.smithi016.stdout:6/581: chown d3/da/dd/d1b/d57/fbf 88702 1 2024-04-22T15:49:38.257 INFO:tasks.workunit.client.0.smithi016.stdout:8/485: rename d2/d82/f4d to d2/d10/d47/f95 0 2024-04-22T15:49:38.259 INFO:tasks.workunit.client.0.smithi016.stdout:2/480: dwrite d2/de/f5c [0,4194304] 0 2024-04-22T15:49:38.261 INFO:tasks.workunit.client.0.smithi016.stdout:3/568: symlink d7/d17/d18/d32/d42/d63/d86/lb3 0 2024-04-22T15:49:38.265 INFO:tasks.workunit.client.0.smithi016.stdout:4/502: sync 2024-04-22T15:49:38.265 INFO:tasks.workunit.client.0.smithi016.stdout:9/517: sync 2024-04-22T15:49:38.265 INFO:tasks.workunit.client.0.smithi016.stdout:0/589: sync 2024-04-22T15:49:38.265 INFO:tasks.workunit.client.0.smithi016.stdout:5/527: sync 2024-04-22T15:49:38.269 INFO:tasks.workunit.client.0.smithi016.stdout:7/521: mknod d3/dc/d1b/d5a/cc1 0 2024-04-22T15:49:38.269 INFO:tasks.workunit.client.0.smithi016.stdout:2/481: mkdir d2/d23/db8 0 2024-04-22T15:49:38.271 INFO:tasks.workunit.client.0.smithi016.stdout:8/486: dwrite d2/d10/d43/d4a/f5f [0,4194304] 0 2024-04-22T15:49:38.273 INFO:tasks.workunit.client.0.smithi016.stdout:1/719: dread d3/d4/d15/d38/d63/d67/f86 [0,4194304] 0 2024-04-22T15:49:38.274 INFO:tasks.workunit.client.0.smithi016.stdout:3/569: mkdir d7/d17/d18/d32/d42/d63/db4 0 2024-04-22T15:49:38.274 INFO:tasks.workunit.client.0.smithi016.stdout:5/528: truncate df/d5b/d5d/f8e 1145564 0 2024-04-22T15:49:38.275 INFO:tasks.workunit.client.0.smithi016.stdout:0/590: creat dc/d13/d14/d55/fbf x:0 0 0 2024-04-22T15:49:38.275 INFO:tasks.workunit.client.0.smithi016.stdout:7/522: read - d3/dc/d1b/d5a/d9/df/faa zero size 2024-04-22T15:49:38.279 INFO:tasks.workunit.client.0.smithi016.stdout:4/503: creat d7/d4e/d2f/d6f/fa8 x:0 0 0 2024-04-22T15:49:38.283 INFO:tasks.workunit.client.0.smithi016.stdout:1/720: mkdir d3/d4/d3a/d6b/df7 0 2024-04-22T15:49:38.283 INFO:tasks.workunit.client.0.smithi016.stdout:0/591: write dc/d13/d14/d26/f7b [1170563,13635] 0 2024-04-22T15:49:38.287 INFO:tasks.workunit.client.0.smithi016.stdout:2/482: mknod d2/de/d2a/cb9 0 2024-04-22T15:49:38.287 INFO:tasks.workunit.client.0.smithi016.stdout:6/582: getdents d3/da 0 2024-04-22T15:49:38.289 INFO:tasks.workunit.client.0.smithi016.stdout:9/518: creat d9/de/d23/d24/d25/d36/d87/f96 x:0 0 0 2024-04-22T15:49:38.296 INFO:tasks.workunit.client.0.smithi016.stdout:2/483: unlink d2/d9/l54 0 2024-04-22T15:49:38.296 INFO:tasks.workunit.client.0.smithi016.stdout:7/523: dwrite d3/dc/d9f/fb1 [0,4194304] 0 2024-04-22T15:49:38.296 INFO:tasks.workunit.client.0.smithi016.stdout:3/570: dwrite d7/d17/d18/d32/d3c/faa [0,4194304] 0 2024-04-22T15:49:38.299 INFO:tasks.workunit.client.0.smithi016.stdout:6/583: creat d3/da/dd/d1b/d57/fc4 x:0 0 0 2024-04-22T15:49:38.299 INFO:tasks.workunit.client.0.smithi016.stdout:8/487: dread d2/d82/f58 [0,4194304] 0 2024-04-22T15:49:38.304 INFO:tasks.workunit.client.0.smithi016.stdout:6/584: stat d3/da/db/d14 0 2024-04-22T15:49:38.306 INFO:tasks.workunit.client.0.smithi016.stdout:9/519: mkdir d9/de/d23/d97 0 2024-04-22T15:49:38.306 INFO:tasks.workunit.client.0.smithi016.stdout:8/488: truncate d2/d82/d56/f71 919323 0 2024-04-22T15:49:38.309 INFO:tasks.workunit.client.0.smithi016.stdout:4/504: getdents d7/d21/d42/d59/d88 0 2024-04-22T15:49:38.309 INFO:tasks.workunit.client.0.smithi016.stdout:9/520: dread - d9/de/f43 zero size 2024-04-22T15:49:38.311 INFO:tasks.workunit.client.0.smithi016.stdout:3/571: dread d7/d17/d18/d27/fa3 [0,4194304] 0 2024-04-22T15:49:38.311 INFO:tasks.workunit.client.0.smithi016.stdout:2/484: creat d2/d2c/d97/d67/d88/fba x:0 0 0 2024-04-22T15:49:38.311 INFO:tasks.workunit.client.0.smithi016.stdout:0/592: dwrite dc/d13/d14/d26/f4f [0,4194304] 0 2024-04-22T15:49:38.312 INFO:tasks.workunit.client.0.smithi016.stdout:7/524: symlink d3/dc/d1b/d5a/d9/df/lc2 0 2024-04-22T15:49:38.315 INFO:tasks.workunit.client.0.smithi016.stdout:7/525: dread - d3/dc/d1b/d5a/d59/d64/fb3 zero size 2024-04-22T15:49:38.318 INFO:tasks.workunit.client.0.smithi016.stdout:7/526: readlink d3/dc/d1b/d5a/d69/l8e 0 2024-04-22T15:49:38.321 INFO:tasks.workunit.client.0.smithi016.stdout:1/721: rename d3/d4/d3a/d6b/d77/f81 to d3/d4/d19/ff8 0 2024-04-22T15:49:38.323 INFO:tasks.workunit.client.0.smithi016.stdout:4/505: dwrite d7/d21/d42/d80/fa5 [0,4194304] 0 2024-04-22T15:49:38.325 INFO:tasks.workunit.client.0.smithi016.stdout:9/521: creat d9/de/d23/d24/d4e/f98 x:0 0 0 2024-04-22T15:49:38.326 INFO:tasks.workunit.client.0.smithi016.stdout:3/572: symlink d7/d17/d18/d32/d42/d68/lb5 0 2024-04-22T15:49:38.327 INFO:tasks.workunit.client.0.smithi016.stdout:1/722: truncate d3/d2e/d80/d82/fae 642321 0 2024-04-22T15:49:38.328 INFO:tasks.workunit.client.0.smithi016.stdout:4/506: truncate d7/d24/d41/f94 2030911 0 2024-04-22T15:49:38.329 INFO:tasks.workunit.client.0.smithi016.stdout:7/527: dwrite d3/dc/d1b/d5a/d9/fb [4194304,4194304] 0 2024-04-22T15:49:38.332 INFO:tasks.workunit.client.0.smithi016.stdout:3/573: chown d7/d17/d18/f2a 669285 1 2024-04-22T15:49:38.338 INFO:tasks.workunit.client.0.smithi016.stdout:9/522: dwrite d9/d20/f6f [0,4194304] 0 2024-04-22T15:49:38.341 INFO:tasks.workunit.client.0.smithi016.stdout:4/507: dread d7/f69 [0,4194304] 0 2024-04-22T15:49:38.342 INFO:tasks.workunit.client.0.smithi016.stdout:4/508: dread - d7/d21/d45/d49/d9b/fa2 zero size 2024-04-22T15:49:38.343 INFO:tasks.workunit.client.0.smithi016.stdout:4/509: fsync d7/d22/f54 0 2024-04-22T15:49:38.346 INFO:tasks.workunit.client.0.smithi016.stdout:0/593: link dc/d13/d28/d65/d57/f8d dc/d13/d14/d26/d32/fc0 0 2024-04-22T15:49:38.350 INFO:tasks.workunit.client.0.smithi016.stdout:7/528: truncate d3/dc/d1b/f6d 614315 0 2024-04-22T15:49:38.350 INFO:tasks.workunit.client.0.smithi016.stdout:8/489: dwrite d2/d30/f73 [0,4194304] 0 2024-04-22T15:49:38.352 INFO:tasks.workunit.client.0.smithi016.stdout:4/510: dwrite d7/d22/f73 [0,4194304] 0 2024-04-22T15:49:38.355 INFO:tasks.workunit.client.0.smithi016.stdout:6/585: getdents d3/da/d12/d13/d38/d84 0 2024-04-22T15:49:38.356 INFO:tasks.workunit.client.0.smithi016.stdout:3/574: rmdir d7/d17/d18/d32/d42/d63/d8f 39 2024-04-22T15:49:38.357 INFO:tasks.workunit.client.0.smithi016.stdout:2/485: link d2/d2c/d97/d67/d88/c96 d2/d2c/d97/d34/cbb 0 2024-04-22T15:49:38.358 INFO:tasks.workunit.client.0.smithi016.stdout:1/723: getdents d3/d2e/d9f/deb 0 2024-04-22T15:49:38.364 INFO:tasks.workunit.client.0.smithi016.stdout:9/523: creat d9/d57/d91/f99 x:0 0 0 2024-04-22T15:49:38.366 INFO:tasks.workunit.client.0.smithi016.stdout:0/594: creat dc/d13/d28/d65/d40/d4a/d59/d9d/fc1 x:0 0 0 2024-04-22T15:49:38.367 INFO:tasks.workunit.client.0.smithi016.stdout:6/586: dwrite d3/da/d12/d13/d46/f77 [0,4194304] 0 2024-04-22T15:49:38.368 INFO:tasks.workunit.client.0.smithi016.stdout:1/724: creat d3/d4/d3a/d6b/ff9 x:0 0 0 2024-04-22T15:49:38.371 INFO:tasks.workunit.client.0.smithi016.stdout:8/490: mkdir d2/d10/d43/d4a/d8f/d96 0 2024-04-22T15:49:38.375 INFO:tasks.workunit.client.0.smithi016.stdout:5/529: sync 2024-04-22T15:49:38.377 INFO:tasks.workunit.client.0.smithi016.stdout:0/595: creat dc/d13/d28/d65/d40/dbb/fc2 x:0 0 0 2024-04-22T15:49:38.378 INFO:tasks.workunit.client.0.smithi016.stdout:1/725: chown d3/d4/d15/d8c 0 1 2024-04-22T15:49:38.379 INFO:tasks.workunit.client.0.smithi016.stdout:9/524: dread d9/fc [0,4194304] 0 2024-04-22T15:49:38.380 INFO:tasks.workunit.client.0.smithi016.stdout:6/587: unlink d3/da/d12/d13/c88 0 2024-04-22T15:49:38.388 INFO:tasks.workunit.client.0.smithi016.stdout:3/575: write d7/d17/d18/d32/d3c/f66 [6794537,8036] 0 2024-04-22T15:49:38.389 INFO:tasks.workunit.client.0.smithi016.stdout:7/529: dwrite d3/dc/d1b/d5a/f46 [0,4194304] 0 2024-04-22T15:49:38.389 INFO:tasks.workunit.client.0.smithi016.stdout:1/726: rmdir d3/d4/d3a/d91 39 2024-04-22T15:49:38.391 INFO:tasks.workunit.client.0.smithi016.stdout:2/486: link d2/d23/d92/c70 d2/d4d/d64/cbc 0 2024-04-22T15:49:38.395 INFO:tasks.workunit.client.0.smithi016.stdout:9/525: rename d9/d37/f93 to d9/d5d/f9a 0 2024-04-22T15:49:38.397 INFO:tasks.workunit.client.0.smithi016.stdout:4/511: dread d7/d4e/d2f/d33/f7a [0,4194304] 0 2024-04-22T15:49:38.397 INFO:tasks.workunit.client.0.smithi016.stdout:9/526: chown d9/de/d23/d97 0 1 2024-04-22T15:49:38.398 INFO:tasks.workunit.client.0.smithi016.stdout:4/512: stat d7/d24/d51/f6a 0 2024-04-22T15:49:38.399 INFO:tasks.workunit.client.0.smithi016.stdout:5/530: creat df/d15/d1a/d4e/d58/d87/f9e x:0 0 0 2024-04-22T15:49:38.401 INFO:tasks.workunit.client.0.smithi016.stdout:2/487: creat d2/d51/fbd x:0 0 0 2024-04-22T15:49:38.404 INFO:tasks.workunit.client.0.smithi016.stdout:0/596: dread dc/d13/d28/d65/d57/fa2 [0,4194304] 0 2024-04-22T15:49:38.404 INFO:tasks.workunit.client.0.smithi016.stdout:3/576: fsync d7/f5a 0 2024-04-22T15:49:38.405 INFO:tasks.workunit.client.0.smithi016.stdout:7/530: read d3/dc/d1b/d5a/f1f [359454,125442] 0 2024-04-22T15:49:38.406 INFO:tasks.workunit.client.0.smithi016.stdout:8/491: truncate d2/d82/d56/f71 649795 0 2024-04-22T15:49:38.408 INFO:tasks.workunit.client.0.smithi016.stdout:4/513: stat d7/d4e/d2f/d33/l70 0 2024-04-22T15:49:38.409 INFO:tasks.workunit.client.0.smithi016.stdout:2/488: creat d2/de/d2a/d59/fbe x:0 0 0 2024-04-22T15:49:38.410 INFO:tasks.workunit.client.0.smithi016.stdout:0/597: creat dc/d13/d70/da6/fc3 x:0 0 0 2024-04-22T15:49:38.410 INFO:tasks.workunit.client.0.smithi016.stdout:6/588: dread d3/da/dd/f23 [0,4194304] 0 2024-04-22T15:49:38.412 INFO:tasks.workunit.client.0.smithi016.stdout:8/492: mknod d2/d82/d44/c97 0 2024-04-22T15:49:38.415 INFO:tasks.workunit.client.0.smithi016.stdout:7/531: rename d3/d14/fb8 to d3/dc/d9f/da1/fc3 0 2024-04-22T15:49:38.415 INFO:tasks.workunit.client.0.smithi016.stdout:0/598: dread - dc/d13/d28/d65/d40/d4a/d59/d78/fbc zero size 2024-04-22T15:49:38.415 INFO:tasks.workunit.client.0.smithi016.stdout:4/514: symlink d7/d4e/la9 0 2024-04-22T15:49:38.417 INFO:tasks.workunit.client.0.smithi016.stdout:8/493: dread - d2/d82/d56/f90 zero size 2024-04-22T15:49:38.417 INFO:tasks.workunit.client.0.smithi016.stdout:1/727: creat d3/d4/d15/d38/ffa x:0 0 0 2024-04-22T15:49:38.421 INFO:tasks.workunit.client.0.smithi016.stdout:6/589: rename d3/da/db/d44/d68/c71 to d3/d53/d83/cc5 0 2024-04-22T15:49:38.423 INFO:tasks.workunit.client.0.smithi016.stdout:9/527: dread f7 [0,4194304] 0 2024-04-22T15:49:38.427 INFO:tasks.workunit.client.0.smithi016.stdout:2/489: dwrite d2/d23/d92/f71 [0,4194304] 0 2024-04-22T15:49:38.427 INFO:tasks.workunit.client.0.smithi016.stdout:0/599: creat dc/d13/d99/fc4 x:0 0 0 2024-04-22T15:49:38.428 INFO:tasks.workunit.client.0.smithi016.stdout:5/531: truncate df/d15/d1a/d4e/f72 1746930 0 2024-04-22T15:49:38.436 INFO:tasks.workunit.client.0.smithi016.stdout:6/590: getdents d3/d53/dab 0 2024-04-22T15:49:38.440 INFO:tasks.workunit.client.0.smithi016.stdout:7/532: dwrite d3/dc/d9f/fbd [0,4194304] 0 2024-04-22T15:49:38.443 INFO:tasks.workunit.client.0.smithi016.stdout:4/515: dwrite d7/d24/f3d [4194304,4194304] 0 2024-04-22T15:49:38.446 INFO:tasks.workunit.client.0.smithi016.stdout:9/528: truncate d9/de/d23/d24/d4e/f83 1043131 0 2024-04-22T15:49:38.448 INFO:tasks.workunit.client.0.smithi016.stdout:1/728: dwrite d3/d4/d3a/d6b/dc3/ff0 [0,4194304] 0 2024-04-22T15:49:38.449 INFO:tasks.workunit.client.0.smithi016.stdout:6/591: readlink d3/da/db/d44/d68/l7a 0 2024-04-22T15:49:38.449 INFO:tasks.workunit.client.0.smithi016.stdout:3/577: dwrite d7/d17/d18/d32/d3c/d47/f48 [0,4194304] 0 2024-04-22T15:49:38.452 INFO:tasks.workunit.client.0.smithi016.stdout:6/592: fsync d3/da/dd/f70 0 2024-04-22T15:49:38.455 INFO:tasks.workunit.client.0.smithi016.stdout:0/600: rename dc/d13/d14/c25 to dc/d13/d28/d65/d40/d4a/d59/d78/cc5 0 2024-04-22T15:49:38.455 INFO:tasks.workunit.client.0.smithi016.stdout:7/533: rmdir d3/dc/d1b/d5a/d9/df/d94/d4c/d7a 39 2024-04-22T15:49:38.458 INFO:tasks.workunit.client.0.smithi016.stdout:0/601: read - dc/d13/d28/d65/d30/d96/fae zero size 2024-04-22T15:49:38.458 INFO:tasks.workunit.client.0.smithi016.stdout:6/593: chown d3/da/db/d14/d16/l6b 3 1 2024-04-22T15:49:38.459 INFO:tasks.workunit.client.0.smithi016.stdout:3/578: creat d7/d1b/d21/d55/fb6 x:0 0 0 2024-04-22T15:49:38.460 INFO:tasks.workunit.client.0.smithi016.stdout:6/594: write d3/da/d12/d13/f27 [1022141,16021] 0 2024-04-22T15:49:38.461 INFO:tasks.workunit.client.0.smithi016.stdout:0/602: read dc/d13/f31 [575847,89845] 0 2024-04-22T15:49:38.461 INFO:tasks.workunit.client.0.smithi016.stdout:1/729: symlink d3/d4/lfb 0 2024-04-22T15:49:38.462 INFO:tasks.workunit.client.0.smithi016.stdout:3/579: dread - d7/d17/d18/d32/d42/f9a zero size 2024-04-22T15:49:38.463 INFO:tasks.workunit.client.0.smithi016.stdout:3/580: stat f1 0 2024-04-22T15:49:38.465 INFO:tasks.workunit.client.0.smithi016.stdout:0/603: rename dc/d13/d28 to dc/d13/d28/d65/d40/dbb/dc6 22 2024-04-22T15:49:38.465 INFO:tasks.workunit.client.0.smithi016.stdout:9/529: rmdir d9/de/d23/d97 0 2024-04-22T15:49:38.465 INFO:tasks.workunit.client.0.smithi016.stdout:9/530: readlink d9/l17 0 2024-04-22T15:49:38.466 INFO:tasks.workunit.client.0.smithi016.stdout:9/531: fsync d9/d37/f64 0 2024-04-22T15:49:38.468 INFO:tasks.workunit.client.0.smithi016.stdout:1/730: unlink d3/d2e/d80/fa5 0 2024-04-22T15:49:38.469 INFO:tasks.workunit.client.0.smithi016.stdout:6/595: mknod d3/da/db/d14/d16/cc6 0 2024-04-22T15:49:38.471 INFO:tasks.workunit.client.0.smithi016.stdout:5/532: dread df/d15/d1a/d28/f2c [0,4194304] 0 2024-04-22T15:49:38.471 INFO:tasks.workunit.client.0.smithi016.stdout:6/596: stat d3/da/db/d14/d16/f20 0 2024-04-22T15:49:38.471 INFO:tasks.workunit.client.0.smithi016.stdout:3/581: mkdir d7/d17/d18/d32/d42/d68/db7 0 2024-04-22T15:49:38.472 INFO:tasks.workunit.client.0.smithi016.stdout:4/516: write d7/d21/d45/d49/f4d [539683,14965] 0 2024-04-22T15:49:38.473 INFO:tasks.workunit.client.0.smithi016.stdout:6/597: chown d3/da/d12/d13/d46/c63 354526991 1 2024-04-22T15:49:38.476 INFO:tasks.workunit.client.0.smithi016.stdout:0/604: truncate f4 1066534 0 2024-04-22T15:49:38.476 INFO:tasks.workunit.client.0.smithi016.stdout:7/534: getdents d3/dc/d1b/d5a/da4 0 2024-04-22T15:49:38.478 INFO:tasks.workunit.client.0.smithi016.stdout:7/535: write d3/dc/d1b/d31/d5f/f77 [731433,15417] 0 2024-04-22T15:49:38.478 INFO:tasks.workunit.client.0.smithi016.stdout:0/605: read - dc/d13/d28/d65/d40/d4a/d59/d78/f86 zero size 2024-04-22T15:49:38.480 INFO:tasks.workunit.client.0.smithi016.stdout:7/536: chown d3/dc/d1b/d5a/d69/lb7 15706625 1 2024-04-22T15:49:38.481 INFO:tasks.workunit.client.0.smithi016.stdout:4/517: creat d7/d4e/d2f/faa x:0 0 0 2024-04-22T15:49:38.482 INFO:tasks.workunit.client.0.smithi016.stdout:4/518: chown d7/d21 996498 1 2024-04-22T15:49:38.484 INFO:tasks.workunit.client.0.smithi016.stdout:6/598: mknod d3/da/cc7 0 2024-04-22T15:49:38.496 INFO:tasks.workunit.client.0.smithi016.stdout:5/533: dwrite df/d15/d1a/d50/d77/f39 [4194304,4194304] 0 2024-04-22T15:49:38.496 INFO:tasks.workunit.client.0.smithi016.stdout:7/537: creat d3/dc/d1b/d5a/da4/fc4 x:0 0 0 2024-04-22T15:49:38.497 INFO:tasks.workunit.client.0.smithi016.stdout:7/538: chown d3/dc/d1b/d5a/d69 7596 1 2024-04-22T15:49:38.499 INFO:tasks.workunit.client.0.smithi016.stdout:1/731: link d3/d4/d3a/d91/db3/dd2/ff3 d3/d4/d19/d20/d95/ffc 0 2024-04-22T15:49:38.500 INFO:tasks.workunit.client.0.smithi016.stdout:9/532: getdents d9/d57/d91 0 2024-04-22T15:49:38.504 INFO:tasks.workunit.client.0.smithi016.stdout:0/606: link dc/f7e dc/d13/d28/d65/d30/d96/fc7 0 2024-04-22T15:49:38.513 INFO:tasks.workunit.client.0.smithi016.stdout:4/519: symlink d7/d21/d45/d9e/lab 0 2024-04-22T15:49:38.514 INFO:tasks.workunit.client.0.smithi016.stdout:4/520: write d7/d3c/f99 [799350,112217] 0 2024-04-22T15:49:38.514 INFO:tasks.workunit.client.0.smithi016.stdout:7/539: symlink d3/d14/lc5 0 2024-04-22T15:49:38.514 INFO:tasks.workunit.client.0.smithi016.stdout:5/534: dwrite df/d15/d1a/f1e [0,4194304] 0 2024-04-22T15:49:38.515 INFO:tasks.workunit.client.0.smithi016.stdout:3/582: getdents d7/d17/d18 0 2024-04-22T15:49:38.520 INFO:tasks.workunit.client.0.smithi016.stdout:9/533: rename d9/de/d23/d24/d25/f3d to d9/d57/f9b 0 2024-04-22T15:49:38.521 INFO:tasks.workunit.client.0.smithi016.stdout:0/607: symlink dc/d13/d28/d65/d30/lc8 0 2024-04-22T15:49:38.523 INFO:tasks.workunit.client.0.smithi016.stdout:0/608: truncate dc/d13/d28/d65/d30/d96/fc7 671940 0 2024-04-22T15:49:38.524 INFO:tasks.workunit.client.0.smithi016.stdout:0/609: truncate dc/f7e 834247 0 2024-04-22T15:49:38.526 INFO:tasks.workunit.client.0.smithi016.stdout:3/583: creat d7/d17/d18/d27/fb8 x:0 0 0 2024-04-22T15:49:38.526 INFO:tasks.workunit.client.0.smithi016.stdout:0/610: write dc/d13/d28/d65/d40/fac [118622,115385] 0 2024-04-22T15:49:38.528 INFO:tasks.workunit.client.0.smithi016.stdout:0/611: truncate dc/d13/d28/f50 1686046 0 2024-04-22T15:49:38.531 INFO:tasks.workunit.client.0.smithi016.stdout:7/540: mknod d3/dc/cc6 0 2024-04-22T15:49:38.535 INFO:tasks.workunit.client.0.smithi016.stdout:3/584: rmdir d7/d17/d18/d32/d42/d7e 39 2024-04-22T15:49:38.541 INFO:tasks.workunit.client.0.smithi016.stdout:0/612: mkdir dc/d13/d28/d65/d30/dc9 0 2024-04-22T15:49:38.542 INFO:tasks.workunit.client.0.smithi016.stdout:1/732: getdents d3/d4/d3a/d9e/dbf 0 2024-04-22T15:49:38.544 INFO:tasks.workunit.client.0.smithi016.stdout:3/585: rmdir d7/d1b 39 2024-04-22T15:49:38.544 INFO:tasks.workunit.client.0.smithi016.stdout:1/733: chown d3/d4/d15/d38/d63/d67/fd0 127790 1 2024-04-22T15:49:38.545 INFO:tasks.workunit.client.0.smithi016.stdout:4/521: dwrite d7/d21/d42/f55 [0,4194304] 0 2024-04-22T15:49:38.546 INFO:tasks.workunit.client.0.smithi016.stdout:6/599: rename d3/da/db/d44/f9a to d3/da/db/d14/fc8 0 2024-04-22T15:49:38.547 INFO:tasks.workunit.client.0.smithi016.stdout:0/613: creat dc/d13/d28/d65/d57/fca x:0 0 0 2024-04-22T15:49:38.549 INFO:tasks.workunit.client.0.smithi016.stdout:7/541: unlink d3/dc/d1b/d5a/cc1 0 2024-04-22T15:49:38.550 INFO:tasks.workunit.client.0.smithi016.stdout:3/586: dread d7/d17/d18/d32/d3c/f88 [0,4194304] 0 2024-04-22T15:49:38.550 INFO:tasks.workunit.client.0.smithi016.stdout:9/534: dwrite d9/f1c [0,4194304] 0 2024-04-22T15:49:38.552 INFO:tasks.workunit.client.0.smithi016.stdout:7/542: write d3/dc/d1b/d5a/d69/f79 [427743,70122] 0 2024-04-22T15:49:38.553 INFO:tasks.workunit.client.0.smithi016.stdout:1/734: chown d3/d4/d37/l87 872519 1 2024-04-22T15:49:38.553 INFO:tasks.workunit.client.0.smithi016.stdout:3/587: write d7/d17/d18/d27/f53 [4556430,69672] 0 2024-04-22T15:49:38.565 INFO:tasks.workunit.client.0.smithi016.stdout:0/614: mkdir dc/d13/d28/d65/d57/dcb 0 2024-04-22T15:49:38.594 INFO:tasks.workunit.client.0.smithi016.stdout:5/535: rename df/d15/c19 to df/d22/c9f 0 2024-04-22T15:49:38.595 INFO:tasks.workunit.client.0.smithi016.stdout:6/600: dwrite d3/da/dd/d7b/fc0 [0,4194304] 0 2024-04-22T15:49:38.595 INFO:tasks.workunit.client.0.smithi016.stdout:4/522: write d7/f8 [5945193,38651] 0 2024-04-22T15:49:38.595 INFO:tasks.workunit.client.0.smithi016.stdout:3/588: write d7/d17/d18/f67 [4165601,26269] 0 2024-04-22T15:49:38.596 INFO:tasks.workunit.client.0.smithi016.stdout:0/615: chown dc/d13/d28/d65/c23 6451 1 2024-04-22T15:49:38.598 INFO:tasks.workunit.client.0.smithi016.stdout:7/543: symlink d3/dc/d1b/d5a/d69/d8c/lc7 0 2024-04-22T15:49:38.599 INFO:tasks.workunit.client.0.smithi016.stdout:9/535: creat d9/de/d23/f9c x:0 0 0 2024-04-22T15:49:38.600 INFO:tasks.workunit.client.0.smithi016.stdout:0/616: write dc/d13/d5d/f6c [1118769,83891] 0 2024-04-22T15:49:38.604 INFO:tasks.workunit.client.0.smithi016.stdout:0/617: dread dc/f7e [0,4194304] 0 2024-04-22T15:49:38.604 INFO:tasks.workunit.client.0.smithi016.stdout:4/523: mkdir d7/d21/d45/d9e/dac 0 2024-04-22T15:49:38.605 INFO:tasks.workunit.client.0.smithi016.stdout:6/601: mknod d3/da/db/d44/d68/cc9 0 2024-04-22T15:49:38.607 INFO:tasks.workunit.client.0.smithi016.stdout:3/589: truncate d7/f12 2043037 0 2024-04-22T15:49:38.608 INFO:tasks.workunit.client.0.smithi016.stdout:9/536: dwrite d9/de/d23/d30/f75 [0,4194304] 0 2024-04-22T15:49:38.618 INFO:tasks.workunit.client.0.smithi016.stdout:9/537: dwrite d9/d37/f64 [0,4194304] 0 2024-04-22T15:49:38.620 INFO:tasks.workunit.client.0.smithi016.stdout:0/618: mknod dc/d13/d28/d65/d1f/d48/ccc 0 2024-04-22T15:49:38.623 INFO:tasks.workunit.client.0.smithi016.stdout:5/536: dread df/d15/d1a/d28/f8f [0,4194304] 0 2024-04-22T15:49:38.625 INFO:tasks.workunit.client.0.smithi016.stdout:5/537: write df/d15/d1a/d28/f32 [3747130,37654] 0 2024-04-22T15:49:38.632 INFO:tasks.workunit.client.0.smithi016.stdout:7/544: dread d3/dc/d1b/d31/d5f/f7b [0,4194304] 0 2024-04-22T15:49:38.635 INFO:tasks.workunit.client.0.smithi016.stdout:6/602: mknod d3/da/db/cca 0 2024-04-22T15:49:38.639 INFO:tasks.workunit.client.0.smithi016.stdout:4/524: rename d7/d21/d45/d9e to d7/d21/d42/d59/d88/d9d/dad 0 2024-04-22T15:49:38.641 INFO:tasks.workunit.client.0.smithi016.stdout:9/538: symlink d9/de/d23/l9d 0 2024-04-22T15:49:38.642 INFO:tasks.workunit.client.0.smithi016.stdout:5/538: dread df/d5b/d5d/d5f/f6f [0,4194304] 0 2024-04-22T15:49:38.646 INFO:tasks.workunit.client.0.smithi016.stdout:6/603: mknod d3/da/dd/d1b/ccb 0 2024-04-22T15:49:38.647 INFO:tasks.workunit.client.0.smithi016.stdout:6/604: chown d3/da/dd/f29 13061 1 2024-04-22T15:49:38.652 INFO:tasks.workunit.client.0.smithi016.stdout:6/605: dwrite d3/da/db/f17 [0,4194304] 0 2024-04-22T15:49:38.660 INFO:tasks.workunit.client.0.smithi016.stdout:5/539: rmdir df/d15/d1a 39 2024-04-22T15:49:38.665 INFO:tasks.workunit.client.0.smithi016.stdout:3/590: getdents d7/d77 0 2024-04-22T15:49:38.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:38 smithi016 ceph-mon[26620]: pgmap v31: 65 pgs: 65 active+clean; 1004 MiB data, 3.0 GiB used, 533 GiB / 536 GiB avail; 30 MiB/s rd, 134 MiB/s wr, 368 op/s 2024-04-22T15:49:38.666 INFO:tasks.workunit.client.0.smithi016.stdout:3/591: dread - d7/d17/d18/d32/d8a/fa8 zero size 2024-04-22T15:49:38.669 INFO:tasks.workunit.client.0.smithi016.stdout:6/606: symlink d3/d53/d35/lcc 0 2024-04-22T15:49:38.672 INFO:tasks.workunit.client.0.smithi016.stdout:7/545: creat d3/dc/fc8 x:0 0 0 2024-04-22T15:49:38.673 INFO:tasks.workunit.client.0.smithi016.stdout:7/546: write d3/d14/f2c [5915858,32763] 0 2024-04-22T15:49:38.675 INFO:tasks.workunit.client.0.smithi016.stdout:5/540: stat df/d15/d1a/d28/f74 0 2024-04-22T15:49:38.678 INFO:tasks.workunit.client.0.smithi016.stdout:4/525: creat d7/d4e/d2f/d6f/fae x:0 0 0 2024-04-22T15:49:38.681 INFO:tasks.workunit.client.0.smithi016.stdout:9/539: creat d9/de/f9e x:0 0 0 2024-04-22T15:49:38.685 INFO:tasks.workunit.client.0.smithi016.stdout:7/547: mkdir d3/dc/d1b/d5a/d59/d64/dc9 0 2024-04-22T15:49:38.685 INFO:tasks.workunit.client.0.smithi016.stdout:0/619: truncate dc/d13/d14/f34 4012527 0 2024-04-22T15:49:38.694 INFO:tasks.workunit.client.0.smithi016.stdout:4/526: dread d7/d22/f54 [0,4194304] 0 2024-04-22T15:49:38.696 INFO:tasks.workunit.client.0.smithi016.stdout:4/527: write d7/d24/f3d [2292629,117476] 0 2024-04-22T15:49:38.698 INFO:tasks.workunit.client.0.smithi016.stdout:9/540: unlink d9/de/d26/d5a/f73 0 2024-04-22T15:49:38.700 INFO:tasks.workunit.client.0.smithi016.stdout:8/494: sync 2024-04-22T15:49:38.701 INFO:tasks.workunit.client.0.smithi016.stdout:6/607: dwrite d3/da/d12/d13/f78 [0,4194304] 0 2024-04-22T15:49:38.710 INFO:tasks.workunit.client.0.smithi016.stdout:4/528: creat d7/d21/d45/d49/d9b/faf x:0 0 0 2024-04-22T15:49:38.711 INFO:tasks.workunit.client.0.smithi016.stdout:3/592: dwrite d7/d17/d18/d32/f9f [0,4194304] 0 2024-04-22T15:49:38.719 INFO:tasks.workunit.client.0.smithi016.stdout:9/541: rename d9/de/d26/d5a/l80 to d9/de/d23/d24/d4e/l9f 0 2024-04-22T15:49:38.720 INFO:tasks.workunit.client.0.smithi016.stdout:8/495: dwrite d2/d82/d44/d66/f4c [0,4194304] 0 2024-04-22T15:49:38.720 INFO:tasks.workunit.client.0.smithi016.stdout:6/608: dwrite d3/da/d12/d13/f3f [0,4194304] 0 2024-04-22T15:49:38.724 INFO:tasks.workunit.client.0.smithi016.stdout:6/609: readlink d3/da/db/d14/d16/l6b 0 2024-04-22T15:49:38.724 INFO:tasks.workunit.client.0.smithi016.stdout:5/541: link df/l38 df/d15/d1a/d50/d77/d7a/la0 0 2024-04-22T15:49:38.725 INFO:tasks.workunit.client.0.smithi016.stdout:4/529: dread - d7/d4e/d2f/faa zero size 2024-04-22T15:49:38.725 INFO:tasks.workunit.client.0.smithi016.stdout:3/593: read d7/f69 [4067353,76515] 0 2024-04-22T15:49:38.727 INFO:tasks.workunit.client.0.smithi016.stdout:8/496: write d2/d82/d44/d66/d79/f8b [714880,82301] 0 2024-04-22T15:49:38.729 INFO:tasks.workunit.client.0.smithi016.stdout:8/497: truncate d2/d48/f55 1113950 0 2024-04-22T15:49:38.729 INFO:tasks.workunit.client.0.smithi016.stdout:4/530: truncate d7/d4e/d2f/d6f/fae 636574 0 2024-04-22T15:49:38.738 INFO:tasks.workunit.client.0.smithi016.stdout:7/548: write d3/dc/d1b/d5a/f1f [594678,3424] 0 2024-04-22T15:49:38.742 INFO:tasks.workunit.client.0.smithi016.stdout:9/542: creat d9/de/d23/d24/d25/d36/fa0 x:0 0 0 2024-04-22T15:49:38.742 INFO:tasks.workunit.client.0.smithi016.stdout:1/735: sync 2024-04-22T15:49:38.742 INFO:tasks.workunit.client.0.smithi016.stdout:2/490: sync 2024-04-22T15:49:38.742 INFO:tasks.workunit.client.0.smithi016.stdout:9/543: stat l1 0 2024-04-22T15:49:38.743 INFO:tasks.workunit.client.0.smithi016.stdout:2/491: fdatasync d2/d2c/d97/d34/fa5 0 2024-04-22T15:49:38.752 INFO:tasks.workunit.client.0.smithi016.stdout:0/620: write dc/d13/d6f/d8b/f9a [933953,120653] 0 2024-04-22T15:49:38.753 INFO:tasks.workunit.client.0.smithi016.stdout:0/621: dread - dc/d13/d28/d65/d40/d4a/d61/d82/fb5 zero size 2024-04-22T15:49:38.754 INFO:tasks.workunit.client.0.smithi016.stdout:8/498: mknod d2/d10/d47/d67/c98 0 2024-04-22T15:49:38.754 INFO:tasks.workunit.client.0.smithi016.stdout:2/492: dread d2/d2c/d97/f6a [0,4194304] 0 2024-04-22T15:49:38.755 INFO:tasks.workunit.client.0.smithi016.stdout:4/531: mknod d7/d4e/d2f/d33/cb0 0 2024-04-22T15:49:38.755 INFO:tasks.workunit.client.0.smithi016.stdout:8/499: write d2/d82/d56/f90 [317696,98375] 0 2024-04-22T15:49:38.756 INFO:tasks.workunit.client.0.smithi016.stdout:4/532: readlink d7/d24/l98 0 2024-04-22T15:49:38.756 INFO:tasks.workunit.client.0.smithi016.stdout:8/500: chown d2/d10/d43/d4a 0 1 2024-04-22T15:49:38.759 INFO:tasks.workunit.client.0.smithi016.stdout:1/736: creat d3/d4/d19/d20/d2b/de0/ffd x:0 0 0 2024-04-22T15:49:38.760 INFO:tasks.workunit.client.0.smithi016.stdout:1/737: chown d3/f23 56 1 2024-04-22T15:49:38.760 INFO:tasks.workunit.client.0.smithi016.stdout:3/594: mknod d7/d1b/d21/d55/dac/cb9 0 2024-04-22T15:49:38.762 INFO:tasks.workunit.client.0.smithi016.stdout:3/595: read - d7/d17/d18/d32/d42/f9a zero size 2024-04-22T15:49:38.764 INFO:tasks.workunit.client.0.smithi016.stdout:9/544: creat d9/fa1 x:0 0 0 2024-04-22T15:49:38.768 INFO:tasks.workunit.client.0.smithi016.stdout:2/493: rename d2/d2c/d97/d67/d88 to d2/d2c/d97/d67/dbf 0 2024-04-22T15:49:38.770 INFO:tasks.workunit.client.0.smithi016.stdout:2/494: dread - d2/de/d2a/d59/fbe zero size 2024-04-22T15:49:38.770 INFO:tasks.workunit.client.0.smithi016.stdout:5/542: getdents df/d15/d1a/d50/d77/d7a 0 2024-04-22T15:49:38.774 INFO:tasks.workunit.client.0.smithi016.stdout:5/543: chown df/d15/d1a/d4e/d58/l66 0 1 2024-04-22T15:49:38.775 INFO:tasks.workunit.client.0.smithi016.stdout:6/610: rename d3/da/dd/c5b to d3/da/db/d14/ccd 0 2024-04-22T15:49:38.775 INFO:tasks.workunit.client.0.smithi016.stdout:6/611: dread - d3/da/dd/d7b/fbb zero size 2024-04-22T15:49:38.775 INFO:tasks.workunit.client.0.smithi016.stdout:6/612: write d3/da/dd/f82 [2191224,26184] 0 2024-04-22T15:49:38.785 INFO:tasks.workunit.client.0.smithi016.stdout:6/613: truncate d3/da/db/f17 4851627 0 2024-04-22T15:49:38.787 INFO:tasks.workunit.client.0.smithi016.stdout:3/596: creat d7/d75/fba x:0 0 0 2024-04-22T15:49:38.787 INFO:tasks.workunit.client.0.smithi016.stdout:9/545: mkdir d9/de/d23/d30/da2 0 2024-04-22T15:49:38.806 INFO:tasks.workunit.client.0.smithi016.stdout:1/738: dread d3/d4/d3a/d6b/f94 [0,4194304] 0 2024-04-22T15:49:38.806 INFO:tasks.workunit.client.0.smithi016.stdout:5/544: creat df/d15/d1a/d50/d77/fa1 x:0 0 0 2024-04-22T15:49:38.806 INFO:tasks.workunit.client.0.smithi016.stdout:9/546: read d9/de/d26/f2a [1130819,72089] 0 2024-04-22T15:49:38.807 INFO:tasks.workunit.client.0.smithi016.stdout:3/597: read d7/d17/d18/d27/f3d [605323,74251] 0 2024-04-22T15:49:38.807 INFO:tasks.workunit.client.0.smithi016.stdout:2/495: write d2/d2c/f37 [890550,18334] 0 2024-04-22T15:49:38.807 INFO:tasks.workunit.client.0.smithi016.stdout:7/549: dwrite d3/dc/d1b/d5a/d9/f25 [0,4194304] 0 2024-04-22T15:49:38.807 INFO:tasks.workunit.client.0.smithi016.stdout:4/533: getdents d7/d24 0 2024-04-22T15:49:38.807 INFO:tasks.workunit.client.0.smithi016.stdout:8/501: getdents d2/d82/d3e 0 2024-04-22T15:49:38.807 INFO:tasks.workunit.client.0.smithi016.stdout:6/614: creat d3/da/db/fce x:0 0 0 2024-04-22T15:49:38.807 INFO:tasks.workunit.client.0.smithi016.stdout:2/496: mkdir d2/d4d/d64/dc0 0 2024-04-22T15:49:38.807 INFO:tasks.workunit.client.0.smithi016.stdout:5/545: truncate df/d15/d1a/d50/d77/f4c 744280 0 2024-04-22T15:49:38.807 INFO:tasks.workunit.client.0.smithi016.stdout:6/615: readlink d3/d53/d35/lcc 0 2024-04-22T15:49:38.807 INFO:tasks.workunit.client.0.smithi016.stdout:3/598: truncate d7/d17/d18/d32/d8a/f9c 162198 0 2024-04-22T15:49:38.807 INFO:tasks.workunit.client.0.smithi016.stdout:6/616: dwrite d3/da/d12/d13/d46/fb4 [0,4194304] 0 2024-04-22T15:49:38.807 INFO:tasks.workunit.client.0.smithi016.stdout:4/534: symlink d7/d24/lb1 0 2024-04-22T15:49:38.807 INFO:tasks.workunit.client.0.smithi016.stdout:9/547: write d9/fa [4251734,91109] 0 2024-04-22T15:49:38.808 INFO:tasks.workunit.client.0.smithi016.stdout:9/548: chown d9/l2d 129763390 1 2024-04-22T15:49:38.809 INFO:tasks.workunit.client.0.smithi016.stdout:8/502: chown d2/d82/d44/d66/d1c/d65/l7d 1457949683 1 2024-04-22T15:49:38.810 INFO:tasks.workunit.client.0.smithi016.stdout:5/546: mknod df/d15/d1a/d50/d77/d7d/d8b/ca2 0 2024-04-22T15:49:38.810 INFO:tasks.workunit.client.0.smithi016.stdout:3/599: symlink d7/d17/d18/d32/d42/d68/lbb 0 2024-04-22T15:49:38.810 INFO:tasks.workunit.client.0.smithi016.stdout:7/550: mknod d3/dc/d1b/d5a/cca 0 2024-04-22T15:49:38.810 INFO:tasks.workunit.client.0.smithi016.stdout:9/549: mknod d9/d5d/d62/ca3 0 2024-04-22T15:49:38.812 INFO:tasks.workunit.client.0.smithi016.stdout:9/550: fdatasync d9/de/d23/f35 0 2024-04-22T15:49:38.815 INFO:tasks.workunit.client.0.smithi016.stdout:6/617: creat d3/da/db/d44/d68/d95/fcf x:0 0 0 2024-04-22T15:49:38.818 INFO:tasks.workunit.client.0.smithi016.stdout:5/547: chown df/d15/d1a/d4e/f72 179578 1 2024-04-22T15:49:38.820 INFO:tasks.workunit.client.0.smithi016.stdout:8/503: dwrite d2/d48/f49 [0,4194304] 0 2024-04-22T15:49:38.820 INFO:tasks.workunit.client.0.smithi016.stdout:4/535: fdatasync d7/d4e/d39/f48 0 2024-04-22T15:49:38.821 INFO:tasks.workunit.client.0.smithi016.stdout:5/548: write df/d15/d1a/f4b [2220990,111886] 0 2024-04-22T15:49:38.823 INFO:tasks.workunit.client.0.smithi016.stdout:2/497: link d2/d1a/f5e d2/d2c/d97/fc1 0 2024-04-22T15:49:38.824 INFO:tasks.workunit.client.0.smithi016.stdout:3/600: dwrite d7/d17/d18/d32/d3c/d47/f7f [0,4194304] 0 2024-04-22T15:49:38.825 INFO:tasks.workunit.client.0.smithi016.stdout:6/618: mknod d3/d53/d35/cd0 0 2024-04-22T15:49:38.826 INFO:tasks.workunit.client.0.smithi016.stdout:5/549: dread - df/d15/d1a/d50/d77/d7a/f65 zero size 2024-04-22T15:49:38.826 INFO:tasks.workunit.client.0.smithi016.stdout:6/619: fdatasync d3/da/db/fce 0 2024-04-22T15:49:38.828 INFO:tasks.workunit.client.0.smithi016.stdout:5/550: stat df/d22/f42 0 2024-04-22T15:49:38.828 INFO:tasks.workunit.client.0.smithi016.stdout:3/601: write d7/d17/d18/d32/d3c/d47/f33 [1175154,30943] 0 2024-04-22T15:49:38.831 INFO:tasks.workunit.client.0.smithi016.stdout:5/551: dread - df/d15/d1a/d50/d77/f67 zero size 2024-04-22T15:49:38.832 INFO:tasks.workunit.client.0.smithi016.stdout:6/620: write d3/da/d12/d13/d46/fa9 [1357564,104689] 0 2024-04-22T15:49:38.835 INFO:tasks.workunit.client.0.smithi016.stdout:8/504: mkdir d2/d82/d56/d99 0 2024-04-22T15:49:38.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:38 smithi027 ceph-mon[34922]: pgmap v31: 65 pgs: 65 active+clean; 1004 MiB data, 3.0 GiB used, 533 GiB / 536 GiB avail; 30 MiB/s rd, 134 MiB/s wr, 368 op/s 2024-04-22T15:49:38.836 INFO:tasks.workunit.client.0.smithi016.stdout:2/498: dwrite d2/d4d/d6d/f7e [0,4194304] 0 2024-04-22T15:49:38.837 INFO:tasks.workunit.client.0.smithi016.stdout:7/551: rename d3/dc/d1b/d5a/d9/c58 to d3/dc/d1b/d5a/d69/ccb 0 2024-04-22T15:49:38.839 INFO:tasks.workunit.client.0.smithi016.stdout:7/552: chown d3/dc/d1b/d5a/d69/f90 1016442414 1 2024-04-22T15:49:38.849 INFO:tasks.workunit.client.0.smithi016.stdout:8/505: creat d2/d10/d43/d4a/d6a/f9a x:0 0 0 2024-04-22T15:49:38.851 INFO:tasks.workunit.client.0.smithi016.stdout:4/536: mknod d7/d21/d42/d59/d88/d9d/dad/dac/cb2 0 2024-04-22T15:49:38.858 INFO:tasks.workunit.client.0.smithi016.stdout:4/537: dwrite d7/d22/f27 [0,4194304] 0 2024-04-22T15:49:38.859 INFO:tasks.workunit.client.0.smithi016.stdout:4/538: chown d7/c18 66071 1 2024-04-22T15:49:38.861 INFO:tasks.workunit.client.0.smithi016.stdout:6/621: rename d3/da/dd/f51 to d3/da/dd/d67/d6c/db9/fd1 0 2024-04-22T15:49:38.867 INFO:tasks.workunit.client.0.smithi016.stdout:8/506: write d2/f5a [914562,113001] 0 2024-04-22T15:49:38.874 INFO:tasks.workunit.client.0.smithi016.stdout:8/507: creat d2/d82/d56/d99/f9b x:0 0 0 2024-04-22T15:49:38.875 INFO:tasks.workunit.client.0.smithi016.stdout:7/553: dwrite d3/dc/d1b/d5a/d59/d64/f89 [0,4194304] 0 2024-04-22T15:49:38.882 INFO:tasks.workunit.client.0.smithi016.stdout:3/602: dwrite d7/d1b/f4b [4194304,4194304] 0 2024-04-22T15:49:38.885 INFO:tasks.workunit.client.0.smithi016.stdout:8/508: creat d2/d82/d44/d66/d1c/d65/f9c x:0 0 0 2024-04-22T15:49:38.886 INFO:tasks.workunit.client.0.smithi016.stdout:4/539: dwrite d7/d24/d51/f6a [0,4194304] 0 2024-04-22T15:49:38.887 INFO:tasks.workunit.client.0.smithi016.stdout:7/554: chown d3/dc/d1b/d5a/d9/df/d94/d4c/d7a/f9e 58 1 2024-04-22T15:49:38.888 INFO:tasks.workunit.client.0.smithi016.stdout:2/499: dread d2/d23/d92/d55/fa6 [0,4194304] 0 2024-04-22T15:49:38.889 INFO:tasks.workunit.client.0.smithi016.stdout:6/622: dread d3/da/db/d14/d4c/f52 [0,4194304] 0 2024-04-22T15:49:38.890 INFO:tasks.workunit.client.0.smithi016.stdout:3/603: fsync d7/d17/d18/d32/d42/d63/d8f/fb0 0 2024-04-22T15:49:38.893 INFO:tasks.workunit.client.0.smithi016.stdout:7/555: chown d3/dc/d22 4722188 1 2024-04-22T15:49:38.896 INFO:tasks.workunit.client.0.smithi016.stdout:3/604: creat d7/d1b/fbc x:0 0 0 2024-04-22T15:49:38.897 INFO:tasks.workunit.client.0.smithi016.stdout:8/509: dwrite d2/f5a [0,4194304] 0 2024-04-22T15:49:38.899 INFO:tasks.workunit.client.0.smithi016.stdout:8/510: chown d2/f5c 169 1 2024-04-22T15:49:38.901 INFO:tasks.workunit.client.0.smithi016.stdout:8/511: truncate d2/d82/d56/d99/f9b 6998 0 2024-04-22T15:49:38.901 INFO:tasks.workunit.client.0.smithi016.stdout:3/605: fdatasync d7/d17/d18/d32/d3c/f88 0 2024-04-22T15:49:38.902 INFO:tasks.workunit.client.0.smithi016.stdout:4/540: link d7/d3c/c85 d7/d22/cb3 0 2024-04-22T15:49:38.904 INFO:tasks.workunit.client.0.smithi016.stdout:2/500: rename d2/d2c/d97/d67/dbf/c96 to d2/cc2 0 2024-04-22T15:49:38.907 INFO:tasks.workunit.client.0.smithi016.stdout:7/556: link d3/dc/f9a d3/dc/d1b/d5a/d9/df/fcc 0 2024-04-22T15:49:38.909 INFO:tasks.workunit.client.0.smithi016.stdout:7/557: write d3/dc/d1b/d5a/d9/df/d94/d4c/d7a/f9e [397543,88727] 0 2024-04-22T15:49:38.910 INFO:tasks.workunit.client.0.smithi016.stdout:8/512: rmdir d2/d82/d44/d66/d79 39 2024-04-22T15:49:38.910 INFO:tasks.workunit.client.0.smithi016.stdout:6/623: write d3/da/dd/f5a [1515633,123779] 0 2024-04-22T15:49:38.911 INFO:tasks.workunit.client.0.smithi016.stdout:8/513: chown d2/c63 10 1 2024-04-22T15:49:38.912 INFO:tasks.workunit.client.0.smithi016.stdout:4/541: creat d7/d21/d42/d59/d88/d9d/dad/fb4 x:0 0 0 2024-04-22T15:49:38.913 INFO:tasks.workunit.client.0.smithi016.stdout:2/501: mknod d2/d23/cc3 0 2024-04-22T15:49:38.913 INFO:tasks.workunit.client.0.smithi016.stdout:8/514: chown d2/d82/d44/l76 0 1 2024-04-22T15:49:38.915 INFO:tasks.workunit.client.0.smithi016.stdout:7/558: creat d3/dc/d22/fcd x:0 0 0 2024-04-22T15:49:38.916 INFO:tasks.workunit.client.0.smithi016.stdout:6/624: mkdir d3/da/d12/d13/dd2 0 2024-04-22T15:49:38.917 INFO:tasks.workunit.client.0.smithi016.stdout:6/625: dread - d3/da/dd/d1b/d57/fbc zero size 2024-04-22T15:49:38.918 INFO:tasks.workunit.client.0.smithi016.stdout:4/542: unlink d7/d4e/d39/f48 0 2024-04-22T15:49:38.921 INFO:tasks.workunit.client.0.smithi016.stdout:6/626: dread d3/da/dd/f23 [0,4194304] 0 2024-04-22T15:49:38.925 INFO:tasks.workunit.client.0.smithi016.stdout:4/543: rmdir d7/d21/d45 39 2024-04-22T15:49:38.926 INFO:tasks.workunit.client.0.smithi016.stdout:2/502: fdatasync d2/d1a/f5e 0 2024-04-22T15:49:38.927 INFO:tasks.workunit.client.0.smithi016.stdout:8/515: truncate d2/d82/d44/d66/d1c/f8c 2328383 0 2024-04-22T15:49:38.929 INFO:tasks.workunit.client.0.smithi016.stdout:6/627: dread d3/da/d12/d13/f76 [0,4194304] 0 2024-04-22T15:49:38.930 INFO:tasks.workunit.client.0.smithi016.stdout:3/606: dwrite d7/f22 [0,4194304] 0 2024-04-22T15:49:38.931 INFO:tasks.workunit.client.0.smithi016.stdout:4/544: chown d7/d24/c83 887 1 2024-04-22T15:49:38.932 INFO:tasks.workunit.client.0.smithi016.stdout:6/628: fdatasync d3/da/d12/d13/d46/fa9 0 2024-04-22T15:49:38.935 INFO:tasks.workunit.client.0.smithi016.stdout:8/516: mknod d2/d82/d52/c9d 0 2024-04-22T15:49:38.936 INFO:tasks.workunit.client.0.smithi016.stdout:7/559: truncate d3/dc/d1b/d5a/d9/fb 11897986 0 2024-04-22T15:49:38.941 INFO:tasks.workunit.client.0.smithi016.stdout:2/503: mkdir d2/d4d/dc4 0 2024-04-22T15:49:38.943 INFO:tasks.workunit.client.0.smithi016.stdout:6/629: creat d3/da/db/d44/d68/d95/fd3 x:0 0 0 2024-04-22T15:49:38.944 INFO:tasks.workunit.client.0.smithi016.stdout:8/517: truncate d2/d82/f38 3244093 0 2024-04-22T15:49:38.947 INFO:tasks.workunit.client.0.smithi016.stdout:2/504: dread d2/d51/f9b [0,4194304] 0 2024-04-22T15:49:38.948 INFO:tasks.workunit.client.0.smithi016.stdout:3/607: getdents d7/d17/d18/d32/d3c/d51/d98 0 2024-04-22T15:49:38.948 INFO:tasks.workunit.client.0.smithi016.stdout:2/505: chown d2/d1a/d4c/d9f 4 1 2024-04-22T15:49:38.952 INFO:tasks.workunit.client.0.smithi016.stdout:3/608: fsync d7/d17/d18/d32/d3c/d47/f33 0 2024-04-22T15:49:38.952 INFO:tasks.workunit.client.0.smithi016.stdout:6/630: truncate d3/da/fc3 889299 0 2024-04-22T15:49:38.953 INFO:tasks.workunit.client.0.smithi016.stdout:7/560: mknod d3/dc/d33/d84/cce 0 2024-04-22T15:49:38.955 INFO:tasks.workunit.client.0.smithi016.stdout:8/518: dwrite d2/d82/d44/d66/d1c/f3d [4194304,4194304] 0 2024-04-22T15:49:38.955 INFO:tasks.workunit.client.0.smithi016.stdout:6/631: dread - d3/da/db/d14/fc8 zero size 2024-04-22T15:49:38.956 INFO:tasks.workunit.client.0.smithi016.stdout:2/506: mknod d2/d1a/d53/cc5 0 2024-04-22T15:49:38.957 INFO:tasks.workunit.client.0.smithi016.stdout:2/507: stat d2/c1f 0 2024-04-22T15:49:38.961 INFO:tasks.workunit.client.0.smithi016.stdout:4/545: truncate d7/d24/f76 489180 0 2024-04-22T15:49:38.962 INFO:tasks.workunit.client.0.smithi016.stdout:4/546: fsync d7/d22/f27 0 2024-04-22T15:49:38.962 INFO:tasks.workunit.client.0.smithi016.stdout:3/609: mknod d7/d17/d18/d32/d42/d63/cbd 0 2024-04-22T15:49:38.966 INFO:tasks.workunit.client.0.smithi016.stdout:3/610: dread d7/d17/d18/d32/d42/d63/f9d [0,4194304] 0 2024-04-22T15:49:38.967 INFO:tasks.workunit.client.0.smithi016.stdout:3/611: write d7/d17/d18/f2a [2673498,115703] 0 2024-04-22T15:49:38.968 INFO:tasks.workunit.client.0.smithi016.stdout:6/632: mknod d3/da/dd/d67/d6c/db1/cd4 0 2024-04-22T15:49:38.970 INFO:tasks.workunit.client.0.smithi016.stdout:3/612: dread d7/d17/d18/f4e [0,4194304] 0 2024-04-22T15:49:38.974 INFO:tasks.workunit.client.0.smithi016.stdout:7/561: dwrite d3/dc/d33/f44 [4194304,4194304] 0 2024-04-22T15:49:38.976 INFO:tasks.workunit.client.0.smithi016.stdout:6/633: mknod d3/da/dd/d67/cd5 0 2024-04-22T15:49:38.976 INFO:tasks.workunit.client.0.smithi016.stdout:7/562: fsync d3/dc/d1b/d5a/d59/d64/f6c 0 2024-04-22T15:49:38.979 INFO:tasks.workunit.client.0.smithi016.stdout:7/563: chown d3/dc/d1b/d5a/d9 63837 1 2024-04-22T15:49:38.979 INFO:tasks.workunit.client.0.smithi016.stdout:4/547: read - d7/d4e/d39/fa4 zero size 2024-04-22T15:49:38.980 INFO:tasks.workunit.client.0.smithi016.stdout:8/519: getdents d2/d82 0 2024-04-22T15:49:38.981 INFO:tasks.workunit.client.0.smithi016.stdout:7/564: dread - d3/d14/fbe zero size 2024-04-22T15:49:38.982 INFO:tasks.workunit.client.0.smithi016.stdout:3/613: rename d7/d17/f3e to d7/d17/d18/d27/fbe 0 2024-04-22T15:49:38.983 INFO:tasks.workunit.client.0.smithi016.stdout:7/565: chown d3/dc/d1b/d5a/d69/f9d 111012 1 2024-04-22T15:49:38.983 INFO:tasks.workunit.client.0.smithi016.stdout:3/614: chown d7/d17/d18/d32/d3c/d51/l64 16358956 1 2024-04-22T15:49:38.984 INFO:tasks.workunit.client.0.smithi016.stdout:4/548: read d7/f19 [3670886,125278] 0 2024-04-22T15:49:38.984 INFO:tasks.workunit.client.0.smithi016.stdout:7/566: stat d3/dc/d1b/d31/d88 0 2024-04-22T15:49:38.985 INFO:tasks.workunit.client.0.smithi016.stdout:7/567: write d3/dc/d33/f7f [596196,100517] 0 2024-04-22T15:49:38.988 INFO:tasks.workunit.client.0.smithi016.stdout:7/568: read d3/dc/f9a [360530,106834] 0 2024-04-22T15:49:38.990 INFO:tasks.workunit.client.0.smithi016.stdout:7/569: fsync d3/dc/d1b/d5a/d59/d64/f6c 0 2024-04-22T15:49:38.991 INFO:tasks.workunit.client.0.smithi016.stdout:4/549: dwrite d7/f19 [0,4194304] 0 2024-04-22T15:49:38.993 INFO:tasks.workunit.client.0.smithi016.stdout:3/615: creat d7/d1b/d21/fbf x:0 0 0 2024-04-22T15:49:38.994 INFO:tasks.workunit.client.0.smithi016.stdout:6/634: creat d3/da/dd/d67/dbd/fd6 x:0 0 0 2024-04-22T15:49:38.996 INFO:tasks.workunit.client.0.smithi016.stdout:8/520: dread d2/d82/d44/f53 [0,4194304] 0 2024-04-22T15:49:38.997 INFO:tasks.workunit.client.0.smithi016.stdout:8/521: chown d2/d32 6 1 2024-04-22T15:49:39.004 INFO:tasks.workunit.client.0.smithi016.stdout:6/635: symlink d3/da/dd/d1b/ld7 0 2024-04-22T15:49:39.005 INFO:tasks.workunit.client.0.smithi016.stdout:6/636: write d3/da/db/d14/f9b [473704,25756] 0 2024-04-22T15:49:39.006 INFO:tasks.workunit.client.0.smithi016.stdout:4/550: write d7/d4e/d2f/d33/f7a [4044994,128616] 0 2024-04-22T15:49:39.007 INFO:tasks.workunit.client.0.smithi016.stdout:8/522: unlink d2/d82/d44/d66/f4c 0 2024-04-22T15:49:39.010 INFO:tasks.workunit.client.0.smithi016.stdout:7/570: link d3/dc/d1b/d5a/d9/df/fcc d3/dc/d1b/d5a/d9/df/d94/d4c/d7a/fcf 0 2024-04-22T15:49:39.015 INFO:tasks.workunit.client.0.smithi016.stdout:7/571: dwrite d3/dc/d1b/d5a/d69/f9d [0,4194304] 0 2024-04-22T15:49:39.018 INFO:tasks.workunit.client.0.smithi016.stdout:8/523: read d2/d82/f38 [1632873,116756] 0 2024-04-22T15:49:39.021 INFO:tasks.workunit.client.0.smithi016.stdout:4/551: symlink d7/d4e/d2f/lb5 0 2024-04-22T15:49:39.023 INFO:tasks.workunit.client.0.smithi016.stdout:7/572: creat d3/d14/fd0 x:0 0 0 2024-04-22T15:49:39.023 INFO:tasks.workunit.client.0.smithi016.stdout:3/616: dwrite d7/d17/d18/d32/d3c/f88 [0,4194304] 0 2024-04-22T15:49:39.027 INFO:tasks.workunit.client.0.smithi016.stdout:8/524: symlink d2/d10/d43/l9e 0 2024-04-22T15:49:39.029 INFO:tasks.workunit.client.0.smithi016.stdout:6/637: rmdir d3/da/dd/d67/d6c/db9/dbe 0 2024-04-22T15:49:39.030 INFO:tasks.workunit.client.0.smithi016.stdout:3/617: dwrite d7/d1b/d21/fa5 [0,4194304] 0 2024-04-22T15:49:39.032 INFO:tasks.workunit.client.0.smithi016.stdout:3/618: fdatasync d7/d17/d18/d32/d3c/faa 0 2024-04-22T15:49:39.035 INFO:tasks.workunit.client.0.smithi016.stdout:7/573: read d3/dc/d9f/fbd [295955,45770] 0 2024-04-22T15:49:39.041 INFO:tasks.workunit.client.0.smithi016.stdout:0/622: sync 2024-04-22T15:49:39.043 INFO:tasks.workunit.client.0.smithi016.stdout:7/574: dwrite d3/d14/fd0 [0,4194304] 0 2024-04-22T15:49:39.044 INFO:tasks.workunit.client.0.smithi016.stdout:0/623: dread - dc/d13/d14/d26/d32/fb4 zero size 2024-04-22T15:49:39.044 INFO:tasks.workunit.client.0.smithi016.stdout:7/575: dread - d3/dc/d1b/d5a/d59/d64/fb3 zero size 2024-04-22T15:49:39.045 INFO:tasks.workunit.client.0.smithi016.stdout:3/619: unlink f1 0 2024-04-22T15:49:39.048 INFO:tasks.workunit.client.0.smithi016.stdout:4/552: mkdir d7/d21/d45/d49/d9b/db6 0 2024-04-22T15:49:39.053 INFO:tasks.workunit.client.0.smithi016.stdout:7/576: mknod d3/dc/d1b/d5a/da4/cd1 0 2024-04-22T15:49:39.061 INFO:tasks.workunit.client.0.smithi016.stdout:8/525: dread d2/d82/d44/d66/d1c/f3d [0,4194304] 0 2024-04-22T15:49:39.061 INFO:tasks.workunit.client.0.smithi016.stdout:0/624: dwrite dc/d13/d14/d26/d32/fb4 [0,4194304] 0 2024-04-22T15:49:39.063 INFO:tasks.workunit.client.0.smithi016.stdout:6/638: dwrite d3/da/db/d14/d16/f20 [0,4194304] 0 2024-04-22T15:49:39.064 INFO:tasks.workunit.client.0.smithi016.stdout:4/553: dread d7/d4e/f60 [0,4194304] 0 2024-04-22T15:49:39.066 INFO:tasks.workunit.client.0.smithi016.stdout:3/620: dwrite d7/d17/d18/d32/d42/d68/f7a [0,4194304] 0 2024-04-22T15:49:39.074 INFO:tasks.workunit.client.0.smithi016.stdout:0/625: symlink dc/d13/d22/lcd 0 2024-04-22T15:49:39.074 INFO:tasks.workunit.client.0.smithi016.stdout:7/577: mkdir d3/dc/d1b/d5a/d59/d64/dc9/dd2 0 2024-04-22T15:49:39.076 INFO:tasks.workunit.client.0.smithi016.stdout:3/621: mknod d7/d17/d18/d32/d42/d68/cc0 0 2024-04-22T15:49:39.077 INFO:tasks.workunit.client.0.smithi016.stdout:8/526: dwrite d2/f24 [0,4194304] 0 2024-04-22T15:49:39.078 INFO:tasks.workunit.client.0.smithi016.stdout:7/578: truncate d3/dc/d1b/d5a/fa3 294425 0 2024-04-22T15:49:39.079 INFO:tasks.workunit.client.0.smithi016.stdout:7/579: dread - d3/dc/d1b/d5a/da4/fc4 zero size 2024-04-22T15:49:39.080 INFO:tasks.workunit.client.0.smithi016.stdout:6/639: mknod d3/da/db/cd8 0 2024-04-22T15:49:39.082 INFO:tasks.workunit.client.0.smithi016.stdout:3/622: symlink d7/lc1 0 2024-04-22T15:49:39.082 INFO:tasks.workunit.client.0.smithi016.stdout:8/527: symlink d2/d10/d47/d67/l9f 0 2024-04-22T15:49:39.085 INFO:tasks.workunit.client.0.smithi016.stdout:6/640: mknod d3/da/db/d44/d68/d95/d9e/cd9 0 2024-04-22T15:49:39.085 INFO:tasks.workunit.client.0.smithi016.stdout:3/623: creat d7/d1b/fc2 x:0 0 0 2024-04-22T15:49:39.086 INFO:tasks.workunit.client.0.smithi016.stdout:6/641: chown d3/da/db/cca 228 1 2024-04-22T15:49:39.089 INFO:tasks.workunit.client.0.smithi016.stdout:4/554: dread d7/f16 [0,4194304] 0 2024-04-22T15:49:39.092 INFO:tasks.workunit.client.0.smithi016.stdout:6/642: rmdir d3/da/db/d44/d68 39 2024-04-22T15:49:39.093 INFO:tasks.workunit.client.0.smithi016.stdout:6/643: chown d3/da/db/d14/d16/f3c 652172579 1 2024-04-22T15:49:39.104 INFO:tasks.workunit.client.0.smithi016.stdout:7/580: truncate d3/d14/f7c 460823 0 2024-04-22T15:49:39.105 INFO:tasks.workunit.client.0.smithi016.stdout:7/581: chown d3/dc/d1b/d31/f48 59 1 2024-04-22T15:49:39.106 INFO:tasks.workunit.client.0.smithi016.stdout:4/555: dwrite d7/d4e/d2f/d6f/fa8 [0,4194304] 0 2024-04-22T15:49:39.108 INFO:tasks.workunit.client.0.smithi016.stdout:7/582: mknod d3/dc/d9f/cd3 0 2024-04-22T15:49:39.111 INFO:tasks.workunit.client.0.smithi016.stdout:3/624: getdents d7/d17/d18/d32/d3c/d47 0 2024-04-22T15:49:39.112 INFO:tasks.workunit.client.0.smithi016.stdout:4/556: dread d7/d4e/d2f/d6f/f97 [0,4194304] 0 2024-04-22T15:49:39.114 INFO:tasks.workunit.client.0.smithi016.stdout:4/557: stat d7/d21/d45/c7f 0 2024-04-22T15:49:39.116 INFO:tasks.workunit.client.0.smithi016.stdout:3/625: fsync d7/d1b/d21/d55/f8e 0 2024-04-22T15:49:39.118 INFO:tasks.workunit.client.0.smithi016.stdout:8/528: dwrite d2/f22 [0,4194304] 0 2024-04-22T15:49:39.119 INFO:tasks.workunit.client.0.smithi016.stdout:6/644: dwrite d3/f8 [0,4194304] 0 2024-04-22T15:49:39.120 INFO:tasks.workunit.client.0.smithi016.stdout:4/558: write d7/d24/f3d [4641783,21862] 0 2024-04-22T15:49:39.124 INFO:tasks.workunit.client.0.smithi016.stdout:7/583: creat d3/dc/d1b/d5a/d9/df/d94/d4c/fd4 x:0 0 0 2024-04-22T15:49:39.124 INFO:tasks.workunit.client.0.smithi016.stdout:3/626: mkdir d7/d17/d18/d32/d3c/d47/dc3 0 2024-04-22T15:49:39.126 INFO:tasks.workunit.client.0.smithi016.stdout:6/645: creat d3/da/d12/d13/d46/fda x:0 0 0 2024-04-22T15:49:39.128 INFO:tasks.workunit.client.0.smithi016.stdout:4/559: write d7/d24/d41/f94 [1296608,54184] 0 2024-04-22T15:49:39.131 INFO:tasks.workunit.client.0.smithi016.stdout:7/584: creat d3/dc/d1b/d5a/d59/d64/fd5 x:0 0 0 2024-04-22T15:49:39.132 INFO:tasks.workunit.client.0.smithi016.stdout:8/529: symlink d2/d10/la0 0 2024-04-22T15:49:39.132 INFO:tasks.workunit.client.0.smithi016.stdout:7/585: chown d3/dc/d1b/d31/l82 524923 1 2024-04-22T15:49:39.133 INFO:tasks.workunit.client.0.smithi016.stdout:6/646: symlink d3/d53/d83/ldb 0 2024-04-22T15:49:39.138 INFO:tasks.workunit.client.0.smithi016.stdout:7/586: rename d3/dc/d1b/d5a/d59/d64/fd5 to d3/dc/d1b/d5a/d59/d64/fd6 0 2024-04-22T15:49:39.139 INFO:tasks.workunit.client.0.smithi016.stdout:8/530: dwrite d2/f27 [4194304,4194304] 0 2024-04-22T15:49:39.140 INFO:tasks.workunit.client.0.smithi016.stdout:7/587: write d3/dc/d9f/da1/fbb [71093,41904] 0 2024-04-22T15:49:39.141 INFO:tasks.workunit.client.0.smithi016.stdout:8/531: write d2/f7a [191421,4349] 0 2024-04-22T15:49:39.144 INFO:tasks.workunit.client.0.smithi016.stdout:8/532: write d2/f5a [2083599,17613] 0 2024-04-22T15:49:39.145 INFO:tasks.workunit.client.0.smithi016.stdout:4/560: creat d7/d24/fb7 x:0 0 0 2024-04-22T15:49:39.149 INFO:tasks.workunit.client.0.smithi016.stdout:3/627: dwrite d7/d17/d18/f23 [0,4194304] 0 2024-04-22T15:49:39.150 INFO:tasks.workunit.client.0.smithi016.stdout:3/628: chown d7/c45 390 1 2024-04-22T15:49:39.152 INFO:tasks.workunit.client.0.smithi016.stdout:3/629: dread - d7/d1b/d21/d55/f6f zero size 2024-04-22T15:49:39.153 INFO:tasks.workunit.client.0.smithi016.stdout:6/647: link d3/da/db/d14/d16/c2d d3/da/db/d44/d68/cdc 0 2024-04-22T15:49:39.155 INFO:tasks.workunit.client.0.smithi016.stdout:4/561: dread d7/d4e/d2f/d6f/fa8 [0,4194304] 0 2024-04-22T15:49:39.156 INFO:tasks.workunit.client.0.smithi016.stdout:3/630: truncate d7/d17/d18/f5b 8518713 0 2024-04-22T15:49:39.160 INFO:tasks.workunit.client.0.smithi016.stdout:7/588: write d3/d14/f7c [17030,112996] 0 2024-04-22T15:49:39.163 INFO:tasks.workunit.client.0.smithi016.stdout:4/562: rename d7/f1d to d7/d3c/d96/fb8 0 2024-04-22T15:49:39.166 INFO:tasks.workunit.client.0.smithi016.stdout:6/648: dwrite d3/da/d12/d13/f3f [0,4194304] 0 2024-04-22T15:49:39.169 INFO:tasks.workunit.client.0.smithi016.stdout:4/563: chown d7/d24/d51/f6a 8 1 2024-04-22T15:49:39.170 INFO:tasks.workunit.client.0.smithi016.stdout:6/649: chown d3/da/d12/d13 995 1 2024-04-22T15:49:39.171 INFO:tasks.workunit.client.0.smithi016.stdout:3/631: dwrite d7/d17/d18/d27/fb8 [0,4194304] 0 2024-04-22T15:49:39.173 INFO:tasks.workunit.client.0.smithi016.stdout:4/564: fsync d7/d24/d41/f74 0 2024-04-22T15:49:39.174 INFO:tasks.workunit.client.0.smithi016.stdout:8/533: dwrite d2/d82/d44/d66/f3c [0,4194304] 0 2024-04-22T15:49:39.180 INFO:tasks.workunit.client.0.smithi016.stdout:4/565: mkdir d7/d21/d42/d59/d88/d9d/dad/db9 0 2024-04-22T15:49:39.181 INFO:tasks.workunit.client.0.smithi016.stdout:8/534: mkdir d2/d10/d43/d4a/d8f/da1 0 2024-04-22T15:49:39.181 INFO:tasks.workunit.client.0.smithi016.stdout:6/650: mkdir d3/da/ddd 0 2024-04-22T15:49:39.182 INFO:tasks.workunit.client.0.smithi016.stdout:6/651: read - d3/da/db/d44/d68/f6a zero size 2024-04-22T15:49:39.184 INFO:tasks.workunit.client.0.smithi016.stdout:4/566: mknod d7/d21/d45/d49/d67/cba 0 2024-04-22T15:49:39.184 INFO:tasks.workunit.client.0.smithi016.stdout:6/652: truncate d3/da/dd/d1b/d57/fc4 391862 0 2024-04-22T15:49:39.188 INFO:tasks.workunit.client.0.smithi016.stdout:6/653: symlink d3/lde 0 2024-04-22T15:49:39.189 INFO:tasks.workunit.client.0.smithi016.stdout:7/589: dread d3/dc/d1b/d5a/d9/f13 [4194304,4194304] 0 2024-04-22T15:49:39.190 INFO:tasks.workunit.client.0.smithi016.stdout:8/535: rename d2/d82/d56/d99/f9b to d2/fa2 0 2024-04-22T15:49:39.190 INFO:tasks.workunit.client.0.smithi016.stdout:7/590: write d3/dc/d33/f7f [1364183,59370] 0 2024-04-22T15:49:39.191 INFO:tasks.workunit.client.0.smithi016.stdout:4/567: dread f5 [0,4194304] 0 2024-04-22T15:49:39.192 INFO:tasks.workunit.client.0.smithi016.stdout:7/591: fsync d3/dc/d1b/d5a/d59/d64/f6c 0 2024-04-22T15:49:39.193 INFO:tasks.workunit.client.0.smithi016.stdout:7/592: write d3/dc/d33/f7f [1133683,101892] 0 2024-04-22T15:49:39.194 INFO:tasks.workunit.client.0.smithi016.stdout:6/654: unlink d3/da/dd/f9d 0 2024-04-22T15:49:39.195 INFO:tasks.workunit.client.0.smithi016.stdout:5/552: sync 2024-04-22T15:49:39.195 INFO:tasks.workunit.client.0.smithi016.stdout:1/739: sync 2024-04-22T15:49:39.195 INFO:tasks.workunit.client.0.smithi016.stdout:2/508: sync 2024-04-22T15:49:39.196 INFO:tasks.workunit.client.0.smithi016.stdout:9/551: sync 2024-04-22T15:49:39.198 INFO:tasks.workunit.client.0.smithi016.stdout:4/568: rmdir d7/d21/d42/d59/d88 39 2024-04-22T15:49:39.253 INFO:tasks.workunit.client.0.smithi016.stdout:0/626: sync 2024-04-22T15:49:39.273 INFO:tasks.workunit.client.0.smithi016.stdout:2/509: mkdir d2/d1a/d7b/dc6 0 2024-04-22T15:49:39.276 INFO:tasks.workunit.client.0.smithi016.stdout:4/569: creat d7/d24/d51/fbb x:0 0 0 2024-04-22T15:49:39.278 INFO:tasks.workunit.client.0.smithi016.stdout:5/553: dwrite df/d15/f4a [4194304,4194304] 0 2024-04-22T15:49:39.278 INFO:tasks.workunit.client.0.smithi016.stdout:1/740: dwrite d3/d84/d97/fb6 [0,4194304] 0 2024-04-22T15:49:39.279 INFO:tasks.workunit.client.0.smithi016.stdout:6/655: symlink d3/d53/dab/ldf 0 2024-04-22T15:49:39.279 INFO:tasks.workunit.client.0.smithi016.stdout:3/632: dwrite d7/f8 [0,4194304] 0 2024-04-22T15:49:39.282 INFO:tasks.workunit.client.0.smithi016.stdout:9/552: dread d9/de/d26/f2a [0,4194304] 0 2024-04-22T15:49:39.286 INFO:tasks.workunit.client.0.smithi016.stdout:9/553: readlink d9/l17 0 2024-04-22T15:49:39.286 INFO:tasks.workunit.client.0.smithi016.stdout:0/627: symlink dc/d13/d28/d65/d40/d4a/d61/d82/lce 0 2024-04-22T15:49:39.290 INFO:tasks.workunit.client.0.smithi016.stdout:9/554: write d9/d20/f59 [2217147,58143] 0 2024-04-22T15:49:39.291 INFO:tasks.workunit.client.0.smithi016.stdout:3/633: read d7/f12 [1084210,121232] 0 2024-04-22T15:49:39.293 INFO:tasks.workunit.client.0.smithi016.stdout:2/510: dwrite d2/d23/d92/d55/fa7 [0,4194304] 0 2024-04-22T15:49:39.294 INFO:tasks.workunit.client.0.smithi016.stdout:5/554: mknod df/d15/d1a/d4e/d58/ca3 0 2024-04-22T15:49:39.298 INFO:tasks.workunit.client.0.smithi016.stdout:1/741: unlink d3/d4/d15/d8c/cd5 0 2024-04-22T15:49:39.300 INFO:tasks.workunit.client.0.smithi016.stdout:6/656: mkdir d3/da/d12/d13/d97/de0 0 2024-04-22T15:49:39.302 INFO:tasks.workunit.client.0.smithi016.stdout:0/628: rename dc/d13/f17 to dc/d13/d28/d65/d1f/fcf 0 2024-04-22T15:49:39.303 INFO:tasks.workunit.client.0.smithi016.stdout:6/657: fsync d3/da/db/d44/d68/d95/fcf 0 2024-04-22T15:49:39.304 INFO:tasks.workunit.client.0.smithi016.stdout:2/511: dwrite d2/d2c/d97/d34/fa5 [0,4194304] 0 2024-04-22T15:49:39.306 INFO:tasks.workunit.client.0.smithi016.stdout:3/634: creat d7/d17/d18/d32/d3c/d47/fc4 x:0 0 0 2024-04-22T15:49:39.309 INFO:tasks.workunit.client.0.smithi016.stdout:5/555: creat df/d5b/d70/fa4 x:0 0 0 2024-04-22T15:49:39.309 INFO:tasks.workunit.client.0.smithi016.stdout:7/593: write d3/dc/d1b/d5a/d9/df/d94/f3d [6873533,11105] 0 2024-04-22T15:49:39.311 INFO:tasks.workunit.client.0.smithi016.stdout:1/742: creat d3/d4/da/ffe x:0 0 0 2024-04-22T15:49:39.312 INFO:tasks.workunit.client.0.smithi016.stdout:2/512: dread d2/d23/d92/d55/fa6 [0,4194304] 0 2024-04-22T15:49:39.318 INFO:tasks.workunit.client.0.smithi016.stdout:6/658: mknod d3/da/db/d44/d68/d95/d9e/ce1 0 2024-04-22T15:49:39.319 INFO:tasks.workunit.client.0.smithi016.stdout:0/629: dread dc/d13/d28/f50 [0,4194304] 0 2024-04-22T15:49:39.320 INFO:tasks.workunit.client.0.smithi016.stdout:3/635: symlink d7/d1b/d21/lc5 0 2024-04-22T15:49:39.321 INFO:tasks.workunit.client.0.smithi016.stdout:3/636: chown d7/d17/l29 26993 1 2024-04-22T15:49:39.322 INFO:tasks.workunit.client.0.smithi016.stdout:7/594: creat d3/dc/d1b/d5a/da4/fd7 x:0 0 0 2024-04-22T15:49:39.324 INFO:tasks.workunit.client.0.smithi016.stdout:0/630: dwrite dc/d13/d28/d65/d1f/d48/f64 [0,4194304] 0 2024-04-22T15:49:39.328 INFO:tasks.workunit.client.0.smithi016.stdout:9/555: creat d9/de/d23/d24/fa4 x:0 0 0 2024-04-22T15:49:39.331 INFO:tasks.workunit.client.0.smithi016.stdout:6/659: write d3/da/f3e [283595,62391] 0 2024-04-22T15:49:39.336 INFO:tasks.workunit.client.0.smithi016.stdout:5/556: creat df/d15/d1a/fa5 x:0 0 0 2024-04-22T15:49:39.337 INFO:tasks.workunit.client.0.smithi016.stdout:0/631: dread dc/d13/d28/d65/d1f/f39 [0,4194304] 0 2024-04-22T15:49:39.339 INFO:tasks.workunit.client.0.smithi016.stdout:3/637: symlink d7/d17/d18/d32/d3c/lc6 0 2024-04-22T15:49:39.345 INFO:tasks.workunit.client.0.smithi016.stdout:3/638: read d7/d1b/d21/d55/f79 [1783613,87714] 0 2024-04-22T15:49:39.345 INFO:tasks.workunit.client.0.smithi016.stdout:5/557: creat df/d15/d1a/d50/d77/fa6 x:0 0 0 2024-04-22T15:49:39.345 INFO:tasks.workunit.client.0.smithi016.stdout:7/595: write d3/dc/d9f/fbd [4405921,33005] 0 2024-04-22T15:49:39.346 INFO:tasks.workunit.client.0.smithi016.stdout:0/632: mknod dc/d13/d28/d65/d57/cd0 0 2024-04-22T15:49:39.346 INFO:tasks.workunit.client.0.smithi016.stdout:5/558: stat df/d22/l29 0 2024-04-22T15:49:39.346 INFO:tasks.workunit.client.0.smithi016.stdout:2/513: truncate d2/d4d/d8e/fad 1042687 0 2024-04-22T15:49:39.350 INFO:tasks.workunit.client.0.smithi016.stdout:9/556: creat d9/de/d23/fa5 x:0 0 0 2024-04-22T15:49:39.352 INFO:tasks.workunit.client.0.smithi016.stdout:7/596: fdatasync d3/dc/f6f 0 2024-04-22T15:49:39.353 INFO:tasks.workunit.client.0.smithi016.stdout:0/633: mkdir dc/d13/d28/dd1 0 2024-04-22T15:49:39.356 INFO:tasks.workunit.client.0.smithi016.stdout:5/559: creat df/d15/d1a/d50/d77/d7d/d8b/d93/fa7 x:0 0 0 2024-04-22T15:49:39.356 INFO:tasks.workunit.client.0.smithi016.stdout:0/634: dread - dc/d13/d14/d26/fa9 zero size 2024-04-22T15:49:39.358 INFO:tasks.workunit.client.0.smithi016.stdout:9/557: dwrite d9/d57/d91/f99 [0,4194304] 0 2024-04-22T15:49:39.361 INFO:tasks.workunit.client.0.smithi016.stdout:0/635: chown dc/d13/d28/d65/f63 2430 1 2024-04-22T15:49:39.362 INFO:tasks.workunit.client.0.smithi016.stdout:9/558: chown d9/de/d23/d24/c46 5 1 2024-04-22T15:49:39.362 INFO:tasks.workunit.client.0.smithi016.stdout:7/597: creat d3/dc/d1b/d5a/d9/df/fd8 x:0 0 0 2024-04-22T15:49:39.363 INFO:tasks.workunit.client.0.smithi016.stdout:5/560: write df/d15/d1a/d50/d77/d7a/f65 [477000,103722] 0 2024-04-22T15:49:39.363 INFO:tasks.workunit.client.0.smithi016.stdout:6/660: link d3/da/c2e d3/da/d12/d13/d46/ce2 0 2024-04-22T15:49:39.364 INFO:tasks.workunit.client.0.smithi016.stdout:2/514: dwrite d2/de/f5d [0,4194304] 0 2024-04-22T15:49:39.367 INFO:tasks.workunit.client.0.smithi016.stdout:9/559: write d9/de/d23/d24/d4e/f79 [1796634,62282] 0 2024-04-22T15:49:39.371 INFO:tasks.workunit.client.0.smithi016.stdout:6/661: write d3/da/db/d14/d4c/d58/f6f [679531,113440] 0 2024-04-22T15:49:39.371 INFO:tasks.workunit.client.0.smithi016.stdout:9/560: fdatasync d9/de/d23/f9c 0 2024-04-22T15:49:39.379 INFO:tasks.workunit.client.0.smithi016.stdout:2/515: dwrite d2/d1a/d7b/f8d [0,4194304] 0 2024-04-22T15:49:39.384 INFO:tasks.workunit.client.0.smithi016.stdout:9/561: rename d9/de/d23/d24/d4e to d9/de/d23/d24/d25/da6 0 2024-04-22T15:49:39.387 INFO:tasks.workunit.client.0.smithi016.stdout:6/662: dwrite d3/da/db/d44/d68/fa5 [0,4194304] 0 2024-04-22T15:49:39.388 INFO:tasks.workunit.client.0.smithi016.stdout:5/561: rmdir df/d15/d1a/d50/d77/d7a/d75 0 2024-04-22T15:49:39.389 INFO:tasks.workunit.client.0.smithi016.stdout:2/516: dread d2/d23/d92/f66 [0,4194304] 0 2024-04-22T15:49:39.393 INFO:tasks.workunit.client.0.smithi016.stdout:9/562: mkdir d9/d57/da7 0 2024-04-22T15:49:39.394 INFO:tasks.workunit.client.0.smithi016.stdout:3/639: truncate d7/fc 1682572 0 2024-04-22T15:49:39.395 INFO:tasks.workunit.client.0.smithi016.stdout:3/640: readlink d7/d17/l29 0 2024-04-22T15:49:39.397 INFO:tasks.workunit.client.0.smithi016.stdout:2/517: fsync d2/d2c/f3c 0 2024-04-22T15:49:39.397 INFO:tasks.workunit.client.0.smithi016.stdout:2/518: readlink d2/de/l29 0 2024-04-22T15:49:39.401 INFO:tasks.workunit.client.0.smithi016.stdout:9/563: rename d9/de/l42 to d9/de/d23/d24/d25/d36/la8 0 2024-04-22T15:49:39.403 INFO:tasks.workunit.client.0.smithi016.stdout:0/636: write dc/d13/f16 [995897,118657] 0 2024-04-22T15:49:39.405 INFO:tasks.workunit.client.0.smithi016.stdout:2/519: mknod d2/d23/d92/d55/cc7 0 2024-04-22T15:49:39.405 INFO:tasks.workunit.client.0.smithi016.stdout:7/598: dread d3/dc/d9f/fb1 [0,4194304] 0 2024-04-22T15:49:39.406 INFO:tasks.workunit.client.0.smithi016.stdout:7/599: fsync d3/dc/d1b/d5a/f1f 0 2024-04-22T15:49:39.407 INFO:tasks.workunit.client.0.smithi016.stdout:5/562: dread df/d15/d1a/f4b [0,4194304] 0 2024-04-22T15:49:39.410 INFO:tasks.workunit.client.0.smithi016.stdout:0/637: symlink dc/d13/d70/ld2 0 2024-04-22T15:49:39.411 INFO:tasks.workunit.client.0.smithi016.stdout:7/600: dread d3/dc/d1b/d31/d5f/f7e [0,4194304] 0 2024-04-22T15:49:39.415 INFO:tasks.workunit.client.0.smithi016.stdout:2/520: dwrite d2/d2c/d97/d67/dbf/fba [0,4194304] 0 2024-04-22T15:49:39.417 INFO:tasks.workunit.client.0.smithi016.stdout:2/521: readlink d2/d2c/d97/d34/la9 0 2024-04-22T15:49:39.424 INFO:tasks.workunit.client.0.smithi016.stdout:7/601: chown d3/c8 4 1 2024-04-22T15:49:39.426 INFO:tasks.workunit.client.0.smithi016.stdout:3/641: rmdir d7/d17/d18/d32/d3c/d51/d98 0 2024-04-22T15:49:39.430 INFO:tasks.workunit.client.0.smithi016.stdout:0/638: dread dc/d13/d28/d65/d30/d75/f80 [4194304,4194304] 0 2024-04-22T15:49:39.434 INFO:tasks.workunit.client.0.smithi016.stdout:9/564: truncate f7 1725658 0 2024-04-22T15:49:39.437 INFO:tasks.workunit.client.0.smithi016.stdout:7/602: unlink d3/dc/f16 0 2024-04-22T15:49:39.438 INFO:tasks.workunit.client.0.smithi016.stdout:7/603: chown d3/dc/d1b/d5a/d59/d64 233 1 2024-04-22T15:49:39.440 INFO:tasks.workunit.client.0.smithi016.stdout:2/522: write d2/f2b [5947621,74490] 0 2024-04-22T15:49:39.441 INFO:tasks.workunit.client.0.smithi016.stdout:9/565: rmdir d9/de/d23/d30 39 2024-04-22T15:49:39.441 INFO:tasks.workunit.client.0.smithi016.stdout:2/523: chown d2/d1a/f20 1 1 2024-04-22T15:49:39.443 INFO:tasks.workunit.client.0.smithi016.stdout:5/563: getdents df/d15/d1a/d50/d77/d7d/d8b 0 2024-04-22T15:49:39.445 INFO:tasks.workunit.client.0.smithi016.stdout:5/564: fdatasync df/f57 0 2024-04-22T15:49:39.445 INFO:tasks.workunit.client.0.smithi016.stdout:0/639: symlink dc/d13/d28/d65/d30/dc9/ld3 0 2024-04-22T15:49:39.450 INFO:tasks.workunit.client.0.smithi016.stdout:3/642: getdents d7/d17 0 2024-04-22T15:49:39.450 INFO:tasks.workunit.client.0.smithi016.stdout:0/640: write dc/d13/d14/d26/d32/fb4 [2025886,72577] 0 2024-04-22T15:49:39.451 INFO:tasks.workunit.client.0.smithi016.stdout:9/566: dwrite d9/d37/f49 [0,4194304] 0 2024-04-22T15:49:39.460 INFO:tasks.workunit.client.0.smithi016.stdout:5/565: creat df/d5b/d5d/fa8 x:0 0 0 2024-04-22T15:49:39.461 INFO:tasks.workunit.client.0.smithi016.stdout:9/567: rmdir d9/de/d23/d24/d25/da6/d4f 39 2024-04-22T15:49:39.467 INFO:tasks.workunit.client.0.smithi016.stdout:2/524: dwrite d2/de/d2a/f49 [4194304,4194304] 0 2024-04-22T15:49:39.467 INFO:tasks.workunit.client.0.smithi016.stdout:7/604: link d3/dc/d1b/d5a/cba d3/dc/d1b/d5a/d59/d64/cd9 0 2024-04-22T15:49:39.470 INFO:tasks.workunit.client.0.smithi016.stdout:7/605: truncate d3/dc/d1b/d5a/d9/df/fcc 1231403 0 2024-04-22T15:49:39.470 INFO:tasks.workunit.client.0.smithi016.stdout:2/525: readlink d2/d2c/d97/d34/d60/l89 0 2024-04-22T15:49:39.471 INFO:tasks.workunit.client.0.smithi016.stdout:2/526: chown d2/d2c/d97/d34/c63 154218 1 2024-04-22T15:49:39.472 INFO:tasks.workunit.client.0.smithi016.stdout:2/527: chown d2/d1a/d4c/d77 8 1 2024-04-22T15:49:39.476 INFO:tasks.workunit.client.0.smithi016.stdout:0/641: truncate dc/d13/d14/d26/d32/fb4 724376 0 2024-04-22T15:49:39.477 INFO:tasks.workunit.client.0.smithi016.stdout:3/643: creat d7/d17/d18/d32/fc7 x:0 0 0 2024-04-22T15:49:39.479 INFO:tasks.workunit.client.0.smithi016.stdout:3/644: readlink d7/d1b/l34 0 2024-04-22T15:49:39.480 INFO:tasks.workunit.client.0.smithi016.stdout:2/528: dwrite d2/de/f5c [0,4194304] 0 2024-04-22T15:49:39.481 INFO:tasks.workunit.client.0.smithi016.stdout:3/645: unlink d7/d75/fba 0 2024-04-22T15:49:39.483 INFO:tasks.workunit.client.0.smithi016.stdout:9/568: creat d9/fa9 x:0 0 0 2024-04-22T15:49:39.485 INFO:tasks.workunit.client.0.smithi016.stdout:7/606: link d3/dc/d33/f7f d3/dc/d1b/d5a/d69/d8c/fda 0 2024-04-22T15:49:39.487 INFO:tasks.workunit.client.0.smithi016.stdout:7/607: chown d3/dc/d1b/d5a/d9/df/d94/d4c/d7a/fcf 7 1 2024-04-22T15:49:39.489 INFO:tasks.workunit.client.0.smithi016.stdout:7/608: write d3/dc/d1b/d5a/da4/fd7 [971129,15033] 0 2024-04-22T15:49:39.489 INFO:tasks.workunit.client.0.smithi016.stdout:5/566: truncate df/d15/d1a/d28/f8f 1902899 0 2024-04-22T15:49:39.490 INFO:tasks.workunit.client.0.smithi016.stdout:7/609: read d3/dc/d1b/d5a/fa3 [175282,28032] 0 2024-04-22T15:49:39.491 INFO:tasks.workunit.client.0.smithi016.stdout:2/529: creat d2/d4d/d6d/fc8 x:0 0 0 2024-04-22T15:49:39.504 INFO:tasks.workunit.client.0.smithi016.stdout:9/569: mkdir d9/d57/da7/daa 0 2024-04-22T15:49:39.508 INFO:tasks.workunit.client.0.smithi016.stdout:0/642: getdents dc/d13/d28/d65/d40 0 2024-04-22T15:49:39.515 INFO:tasks.workunit.client.0.smithi016.stdout:9/570: truncate d9/de/d23/d30/f86 236562 0 2024-04-22T15:49:39.516 INFO:tasks.workunit.client.0.smithi016.stdout:8/536: sync 2024-04-22T15:49:39.516 INFO:tasks.workunit.client.0.smithi016.stdout:0/643: mkdir dc/d13/d28/d65/d1f/dd4 0 2024-04-22T15:49:39.517 INFO:tasks.workunit.client.0.smithi016.stdout:0/644: fsync dc/d13/d28/f6a 0 2024-04-22T15:49:39.519 INFO:tasks.workunit.client.0.smithi016.stdout:2/530: creat d2/de/fc9 x:0 0 0 2024-04-22T15:49:39.525 INFO:tasks.workunit.client.0.smithi016.stdout:5/567: getdents df/d22 0 2024-04-22T15:49:39.526 INFO:tasks.workunit.client.0.smithi016.stdout:7/610: creat d3/d14/fdb x:0 0 0 2024-04-22T15:49:39.527 INFO:tasks.workunit.client.0.smithi016.stdout:9/571: dread d9/de/d23/d24/d25/f2c [0,4194304] 0 2024-04-22T15:49:39.528 INFO:tasks.workunit.client.0.smithi016.stdout:0/645: mkdir dc/d13/d28/d65/d40/dd5 0 2024-04-22T15:49:39.530 INFO:tasks.workunit.client.0.smithi016.stdout:3/646: getdents d7/d17/d18/d32/d42/d7e 0 2024-04-22T15:49:39.532 INFO:tasks.workunit.client.0.smithi016.stdout:5/568: dwrite df/d5b/d5d/f8e [0,4194304] 0 2024-04-22T15:49:39.534 INFO:tasks.workunit.client.0.smithi016.stdout:9/572: creat d9/d20/fab x:0 0 0 2024-04-22T15:49:39.536 INFO:tasks.workunit.client.0.smithi016.stdout:0/646: mkdir dc/d13/d28/d65/d40/dd6 0 2024-04-22T15:49:39.538 INFO:tasks.workunit.client.0.smithi016.stdout:8/537: link d2/d82/d44/c8a d2/d32/d92/ca3 0 2024-04-22T15:49:39.538 INFO:tasks.workunit.client.0.smithi016.stdout:0/647: dread - dc/d13/d28/d65/d30/f84 zero size 2024-04-22T15:49:39.541 INFO:tasks.workunit.client.0.smithi016.stdout:7/611: symlink d3/dc/d1b/d5a/d9/df/d94/d4c/ldc 0 2024-04-22T15:49:39.546 INFO:tasks.workunit.client.0.smithi016.stdout:2/531: dread f0 [0,4194304] 0 2024-04-22T15:49:39.547 INFO:tasks.workunit.client.0.smithi016.stdout:5/569: rename df/d5b/d5d/fa8 to df/d15/fa9 0 2024-04-22T15:49:39.549 INFO:tasks.workunit.client.0.smithi016.stdout:0/648: creat dc/d13/d28/d65/d57/dcb/fd7 x:0 0 0 2024-04-22T15:49:39.553 INFO:tasks.workunit.client.0.smithi016.stdout:8/538: dread d2/d82/d44/d66/f3c [4194304,4194304] 0 2024-04-22T15:49:39.560 INFO:tasks.workunit.client.0.smithi016.stdout:2/532: link d2/d2c/d97/f32 d2/d2c/d97/d67/fca 0 2024-04-22T15:49:39.562 INFO:tasks.workunit.client.0.smithi016.stdout:0/649: dread dc/d13/d28/d65/f63 [0,4194304] 0 2024-04-22T15:49:39.563 INFO:tasks.workunit.client.0.smithi016.stdout:2/533: write d2/d2c/d97/d67/f86 [256002,26420] 0 2024-04-22T15:49:39.566 INFO:tasks.workunit.client.0.smithi016.stdout:7/612: rename d3/dc/d1b/d5a/d69/f79 to d3/dc/d1b/d5a/d9/df/d94/d4c/fdd 0 2024-04-22T15:49:39.566 INFO:tasks.workunit.client.0.smithi016.stdout:3/647: write d7/d17/d18/d27/fa3 [1041317,125462] 0 2024-04-22T15:49:39.572 INFO:tasks.workunit.client.0.smithi016.stdout:9/573: write d9/de/d23/d24/f2e [2480461,80582] 0 2024-04-22T15:49:39.572 INFO:tasks.workunit.client.0.smithi016.stdout:0/650: mkdir dc/d13/d28/dd8 0 2024-04-22T15:49:39.572 INFO:tasks.workunit.client.0.smithi016.stdout:3/648: dwrite d7/d1b/f4b [0,4194304] 0 2024-04-22T15:49:39.574 INFO:tasks.workunit.client.0.smithi016.stdout:0/651: dread - dc/d13/d14/d55/fa8 zero size 2024-04-22T15:49:39.574 INFO:tasks.workunit.client.0.smithi016.stdout:9/574: dread - d9/de/d23/d24/d25/da6/f98 zero size 2024-04-22T15:49:39.576 INFO:tasks.workunit.client.0.smithi016.stdout:0/652: dread - dc/d13/d28/f6a zero size 2024-04-22T15:49:39.577 INFO:tasks.workunit.client.0.smithi016.stdout:9/575: chown d9/de/d23/d24/d25/f41 60 1 2024-04-22T15:49:39.578 INFO:tasks.workunit.client.0.smithi016.stdout:0/653: chown dc/d13/d6f/d8b/f9a 229 1 2024-04-22T15:49:39.578 INFO:tasks.workunit.client.0.smithi016.stdout:9/576: chown d9/de/d23/d24/d25/d36/fa0 155342 1 2024-04-22T15:49:39.579 INFO:tasks.workunit.client.0.smithi016.stdout:5/570: getdents df/d15/d1a/d50/d77/d7d 0 2024-04-22T15:49:39.584 INFO:tasks.workunit.client.0.smithi016.stdout:9/577: dwrite d9/de/f1a [4194304,4194304] 0 2024-04-22T15:49:39.585 INFO:tasks.workunit.client.0.smithi016.stdout:2/534: creat d2/d1a/d4c/d9f/da3/fcb x:0 0 0 2024-04-22T15:49:39.586 INFO:tasks.workunit.client.0.smithi016.stdout:2/535: stat d2/d1a/d7b 0 2024-04-22T15:49:39.587 INFO:tasks.workunit.client.0.smithi016.stdout:3/649: rename d7/d17/d18/d32/c91 to d7/d1b/d21/d55/dac/cc8 0 2024-04-22T15:49:39.587 INFO:tasks.workunit.client.0.smithi016.stdout:1/743: sync 2024-04-22T15:49:39.587 INFO:tasks.workunit.client.0.smithi016.stdout:4/570: sync 2024-04-22T15:49:39.590 INFO:tasks.workunit.client.0.smithi016.stdout:2/536: chown d2/de/d2a/f93 7211145 1 2024-04-22T15:49:39.592 INFO:tasks.workunit.client.0.smithi016.stdout:2/537: fdatasync d2/d2c/f5b 0 2024-04-22T15:49:39.593 INFO:tasks.workunit.client.0.smithi016.stdout:2/538: readlink d2/d4d/l87 0 2024-04-22T15:49:39.594 INFO:tasks.workunit.client.0.smithi016.stdout:5/571: unlink df/d15/f7e 0 2024-04-22T15:49:39.597 INFO:tasks.workunit.client.0.smithi016.stdout:5/572: fdatasync df/d15/d1a/f71 0 2024-04-22T15:49:39.598 INFO:tasks.workunit.client.0.smithi016.stdout:4/571: dread d7/f19 [0,4194304] 0 2024-04-22T15:49:39.598 INFO:tasks.workunit.client.0.smithi016.stdout:9/578: write d9/d57/f9b [77718,35630] 0 2024-04-22T15:49:39.601 INFO:tasks.workunit.client.0.smithi016.stdout:8/539: write d2/d82/d52/f54 [99913,97765] 0 2024-04-22T15:49:39.601 INFO:tasks.workunit.client.0.smithi016.stdout:1/744: fsync d3/d4/fe 0 2024-04-22T15:49:39.609 INFO:tasks.workunit.client.0.smithi016.stdout:3/650: dread d7/f69 [0,4194304] 0 2024-04-22T15:49:39.610 INFO:tasks.workunit.client.0.smithi016.stdout:2/539: rename d2/d4d/d6d to d2/d1a/d7b/dcc 0 2024-04-22T15:49:39.613 INFO:tasks.workunit.client.0.smithi016.stdout:5/573: chown df/l38 1 1 2024-04-22T15:49:39.621 INFO:tasks.workunit.client.0.smithi016.stdout:7/613: dwrite d3/f74 [0,4194304] 0 2024-04-22T15:49:39.624 INFO:tasks.workunit.client.0.smithi016.stdout:7/614: truncate d3/dc/d9f/da1/fc0 671637 0 2024-04-22T15:49:39.625 INFO:tasks.workunit.client.0.smithi016.stdout:1/745: symlink d3/d4/d3a/d91/lff 0 2024-04-22T15:49:39.626 INFO:tasks.workunit.client.0.smithi016.stdout:7/615: write d3/dc/d1b/d5a/f46 [5022892,31133] 0 2024-04-22T15:49:39.632 INFO:tasks.workunit.client.0.smithi016.stdout:0/654: write dc/d13/d14/d26/f46 [664967,108459] 0 2024-04-22T15:49:39.633 INFO:tasks.workunit.client.0.smithi016.stdout:5/574: mknod df/d15/d1a/d50/d77/d7d/caa 0 2024-04-22T15:49:39.634 INFO:tasks.workunit.client.0.smithi016.stdout:3/651: dread d7/d17/d18/d27/f6e [0,4194304] 0 2024-04-22T15:49:39.636 INFO:tasks.workunit.client.0.smithi016.stdout:5/575: fdatasync df/d22/f24 0 2024-04-22T15:49:39.636 INFO:tasks.workunit.client.0.smithi016.stdout:1/746: dwrite d3/d4/d3a/d6b/fc1 [0,4194304] 0 2024-04-22T15:49:39.637 INFO:tasks.workunit.client.0.smithi016.stdout:4/572: mkdir d7/d21/d7c/dbc 0 2024-04-22T15:49:39.637 INFO:tasks.workunit.client.0.smithi016.stdout:6/663: sync 2024-04-22T15:49:39.638 INFO:tasks.workunit.client.0.smithi016.stdout:7/616: dwrite d3/dc/d1b/d5a/d59/d64/f89 [4194304,4194304] 0 2024-04-22T15:49:39.642 INFO:tasks.workunit.client.0.smithi016.stdout:8/540: symlink d2/la4 0 2024-04-22T15:49:39.644 INFO:tasks.workunit.client.0.smithi016.stdout:5/576: write df/d15/d1a/d50/d77/d7a/f80 [3487818,107742] 0 2024-04-22T15:49:39.644 INFO:tasks.workunit.client.0.smithi016.stdout:1/747: write d3/d4/d3a/d91/fb5 [4156178,102144] 0 2024-04-22T15:49:39.647 INFO:tasks.workunit.client.0.smithi016.stdout:9/579: write d9/de/d26/f44 [1848327,128371] 0 2024-04-22T15:49:39.648 INFO:tasks.workunit.client.0.smithi016.stdout:2/540: mknod d2/d4d/ccd 0 2024-04-22T15:49:39.651 INFO:tasks.workunit.client.0.smithi016.stdout:4/573: creat d7/d21/d45/d49/d67/fbd x:0 0 0 2024-04-22T15:49:39.652 INFO:tasks.workunit.client.0.smithi016.stdout:2/541: chown d2/d4d 355557 1 2024-04-22T15:49:39.653 INFO:tasks.workunit.client.0.smithi016.stdout:0/655: rename dc/d13/d28/d65/d40/d4a/d59/f71 to dc/d13/d28/d65/d40/fd9 0 2024-04-22T15:49:39.654 INFO:tasks.workunit.client.0.smithi016.stdout:8/541: rmdir d2/d82 39 2024-04-22T15:49:39.657 INFO:tasks.workunit.client.0.smithi016.stdout:7/617: mknod d3/dc/d1b/d31/cde 0 2024-04-22T15:49:39.660 INFO:tasks.workunit.client.0.smithi016.stdout:6/664: dwrite d3/da/db/d14/f86 [0,4194304] 0 2024-04-22T15:49:39.663 INFO:tasks.workunit.client.0.smithi016.stdout:4/574: write d7/f1e [1595074,77177] 0 2024-04-22T15:49:39.663 INFO:tasks.workunit.client.0.smithi016.stdout:1/748: dwrite d3/d2e/d80/dbb/fd9 [0,4194304] 0 2024-04-22T15:49:39.670 INFO:tasks.workunit.client.0.smithi016.stdout:0/656: unlink dc/d13/d14/d26/d32/c51 0 2024-04-22T15:49:39.672 INFO:tasks.workunit.client.0.smithi016.stdout:2/542: mknod d2/d1a/d7b/dcc/cce 0 2024-04-22T15:49:39.685 INFO:tasks.workunit.client.0.smithi016.stdout:4/575: fdatasync f5 0 2024-04-22T15:49:39.685 INFO:tasks.workunit.client.0.smithi016.stdout:3/652: dwrite d7/d17/d18/d32/f9f [0,4194304] 0 2024-04-22T15:49:39.686 INFO:tasks.workunit.client.0.smithi016.stdout:4/576: readlink d7/d24/d41/l62 0 2024-04-22T15:49:39.692 INFO:tasks.workunit.client.0.smithi016.stdout:8/542: write d2/d82/d3e/f42 [11985511,35399] 0 2024-04-22T15:49:39.696 INFO:tasks.workunit.client.0.smithi016.stdout:5/577: truncate df/d15/d1a/f53 690237 0 2024-04-22T15:49:39.697 INFO:tasks.workunit.client.0.smithi016.stdout:3/653: dwrite d7/d17/d18/d32/d42/d68/fa9 [0,4194304] 0 2024-04-22T15:49:39.700 INFO:tasks.workunit.client.0.smithi016.stdout:3/654: write d7/d17/d18/d27/f4f [4292804,25098] 0 2024-04-22T15:49:39.702 INFO:tasks.workunit.client.0.smithi016.stdout:6/665: dread d3/da/dd/d67/d6c/db9/fd1 [0,4194304] 0 2024-04-22T15:49:39.704 INFO:tasks.workunit.client.0.smithi016.stdout:6/666: chown d3/da/dd/d7b/fbb 11387 1 2024-04-22T15:49:39.709 INFO:tasks.workunit.client.0.smithi016.stdout:3/655: dread d7/d17/d18/d27/f3d [0,4194304] 0 2024-04-22T15:49:39.713 INFO:tasks.workunit.client.0.smithi016.stdout:0/657: creat dc/d13/d28/d65/d40/d4a/d59/d78/dbd/fda x:0 0 0 2024-04-22T15:49:39.719 INFO:tasks.workunit.client.0.smithi016.stdout:7/618: dread d3/d14/fd0 [0,4194304] 0 2024-04-22T15:49:39.727 INFO:tasks.workunit.client.0.smithi016.stdout:9/580: getdents d9/d57 0 2024-04-22T15:49:39.728 INFO:tasks.workunit.client.0.smithi016.stdout:8/543: dread d2/d10/d47/f95 [0,4194304] 0 2024-04-22T15:49:39.728 INFO:tasks.workunit.client.0.smithi016.stdout:7/619: write d3/dc/d1b/d5a/f46 [5488581,72035] 0 2024-04-22T15:49:39.729 INFO:tasks.workunit.client.0.smithi016.stdout:8/544: stat d2/d10/l69 0 2024-04-22T15:49:39.732 INFO:tasks.workunit.client.0.smithi016.stdout:3/656: truncate d7/d17/d18/f4e 280520 0 2024-04-22T15:49:39.733 INFO:tasks.workunit.client.0.smithi016.stdout:0/658: mkdir dc/d13/d70/da6/ddb 0 2024-04-22T15:49:39.735 INFO:tasks.workunit.client.0.smithi016.stdout:0/659: dread - dc/d13/d28/d65/d40/d4a/d61/d82/fb5 zero size 2024-04-22T15:49:39.738 INFO:tasks.workunit.client.0.smithi016.stdout:2/543: truncate d2/d2c/d97/d34/fa5 171698 0 2024-04-22T15:49:39.740 INFO:tasks.workunit.client.0.smithi016.stdout:2/544: fsync d2/d1a/d4c/d9f/da3/fcb 0 2024-04-22T15:49:39.742 INFO:tasks.workunit.client.0.smithi016.stdout:2/545: chown d2/de/l28 0 1 2024-04-22T15:49:39.742 INFO:tasks.workunit.client.0.smithi016.stdout:7/620: dread d3/dc/d33/f3e [0,4194304] 0 2024-04-22T15:49:39.743 INFO:tasks.workunit.client.0.smithi016.stdout:0/660: dwrite dc/d13/d28/d65/d40/d4a/d61/d82/fb5 [0,4194304] 0 2024-04-22T15:49:39.744 INFO:tasks.workunit.client.0.smithi016.stdout:2/546: write d2/d23/f6c [4367529,104953] 0 2024-04-22T15:49:39.751 INFO:tasks.workunit.client.0.smithi016.stdout:9/581: mkdir d9/d20/dac 0 2024-04-22T15:49:39.752 INFO:tasks.workunit.client.0.smithi016.stdout:6/667: link d3/da/dd/fb5 d3/da/d12/d13/d97/de0/fe3 0 2024-04-22T15:49:39.752 INFO:tasks.workunit.client.0.smithi016.stdout:7/621: dwrite d3/dc/d1b/d5a/d9/df/faa [0,4194304] 0 2024-04-22T15:49:39.753 INFO:tasks.workunit.client.0.smithi016.stdout:9/582: write d9/f1c [2690673,9237] 0 2024-04-22T15:49:39.755 INFO:tasks.workunit.client.0.smithi016.stdout:5/578: creat df/d5b/d5d/fab x:0 0 0 2024-04-22T15:49:39.757 INFO:tasks.workunit.client.0.smithi016.stdout:1/749: getdents d3/d4/d19/d20/d95 0 2024-04-22T15:49:39.759 INFO:tasks.workunit.client.0.smithi016.stdout:7/622: chown d3/dc/d1b/d5a/c1c 124659443 1 2024-04-22T15:49:39.762 INFO:tasks.workunit.client.0.smithi016.stdout:0/661: truncate dc/d13/d28/f50 250084 0 2024-04-22T15:49:39.762 INFO:tasks.workunit.client.0.smithi016.stdout:4/577: rmdir d7/d4e/d2f/d50/d93 0 2024-04-22T15:49:39.763 INFO:tasks.workunit.client.0.smithi016.stdout:8/545: mkdir d2/d10/d43/d4a/d8f/d96/da5 0 2024-04-22T15:49:39.765 INFO:tasks.workunit.client.0.smithi016.stdout:9/583: mknod d9/de/d23/d30/cad 0 2024-04-22T15:49:39.765 INFO:tasks.workunit.client.0.smithi016.stdout:7/623: mkdir d3/dc/d1b/d5a/d9/df/d94/ddf 0 2024-04-22T15:49:39.766 INFO:tasks.workunit.client.0.smithi016.stdout:5/579: creat df/d15/d1a/d50/fac x:0 0 0 2024-04-22T15:49:39.769 INFO:tasks.workunit.client.0.smithi016.stdout:8/546: creat d2/d82/d44/d6f/fa6 x:0 0 0 2024-04-22T15:49:39.773 INFO:tasks.workunit.client.0.smithi016.stdout:6/668: creat d3/da/d12/d13/d97/de0/fe4 x:0 0 0 2024-04-22T15:49:39.773 INFO:tasks.workunit.client.0.smithi016.stdout:3/657: link d7/d17/d18/d32/d42/d63/d8f/cab d7/d17/d18/d32/d42/d63/d8f/cc9 0 2024-04-22T15:49:39.774 INFO:tasks.workunit.client.0.smithi016.stdout:0/662: mkdir dc/d13/d28/dd1/ddc 0 2024-04-22T15:49:39.777 INFO:tasks.workunit.client.0.smithi016.stdout:4/578: creat d7/fbe x:0 0 0 2024-04-22T15:49:39.780 INFO:tasks.workunit.client.0.smithi016.stdout:4/579: write d7/d21/d42/f55 [4863637,112525] 0 2024-04-22T15:49:39.780 INFO:tasks.workunit.client.0.smithi016.stdout:6/669: dwrite d3/da/db/d44/d68/d95/fcf [0,4194304] 0 2024-04-22T15:49:39.785 INFO:tasks.workunit.client.0.smithi016.stdout:9/584: mkdir d9/d20/dac/dae 0 2024-04-22T15:49:39.786 INFO:tasks.workunit.client.0.smithi016.stdout:1/750: dread d3/d4/d19/d20/f33 [0,4194304] 0 2024-04-22T15:49:39.787 INFO:tasks.workunit.client.0.smithi016.stdout:0/663: symlink dc/d13/d5d/ldd 0 2024-04-22T15:49:39.789 INFO:tasks.workunit.client.0.smithi016.stdout:5/580: mkdir df/d90/dad 0 2024-04-22T15:49:39.790 INFO:tasks.workunit.client.0.smithi016.stdout:7/624: dread d3/dc/d1b/d5a/d9/f50 [0,4194304] 0 2024-04-22T15:49:39.792 INFO:tasks.workunit.client.0.smithi016.stdout:3/658: unlink d7/d17/d18/d32/d42/d63/d86/l99 0 2024-04-22T15:49:39.793 INFO:tasks.workunit.client.0.smithi016.stdout:3/659: stat d7/f12 0 2024-04-22T15:49:39.793 INFO:tasks.workunit.client.0.smithi016.stdout:7/625: chown d3/dc/d9f/cd3 8347 1 2024-04-22T15:49:39.795 INFO:tasks.workunit.client.0.smithi016.stdout:6/670: fdatasync d3/da/db/d14/d16/f87 0 2024-04-22T15:49:39.796 INFO:tasks.workunit.client.0.smithi016.stdout:4/580: rmdir d7/d21/d42/d80 39 2024-04-22T15:49:39.796 INFO:tasks.workunit.client.0.smithi016.stdout:5/581: chown df/d15/d1a/d50/d77/f62 315529098 1 2024-04-22T15:49:39.797 INFO:tasks.workunit.client.0.smithi016.stdout:7/626: write d3/dc/d1b/d5a/d9/df/f11 [1104577,13417] 0 2024-04-22T15:49:39.797 INFO:tasks.workunit.client.0.smithi016.stdout:0/664: rmdir dc/d13/d22 39 2024-04-22T15:49:39.797 INFO:tasks.workunit.client.0.smithi016.stdout:4/581: fsync d7/f8 0 2024-04-22T15:49:39.800 INFO:tasks.workunit.client.0.smithi016.stdout:9/585: mknod d9/de/d26/caf 0 2024-04-22T15:49:39.801 INFO:tasks.workunit.client.0.smithi016.stdout:6/671: write d3/da/dd/f10 [3841925,2959] 0 2024-04-22T15:49:39.802 INFO:tasks.workunit.client.0.smithi016.stdout:8/547: creat d2/d82/d44/d66/fa7 x:0 0 0 2024-04-22T15:49:39.804 INFO:tasks.workunit.client.0.smithi016.stdout:2/547: write d2/d2c/d97/d67/fca [789297,97213] 0 2024-04-22T15:49:39.806 INFO:tasks.workunit.client.0.smithi016.stdout:2/548: write d2/d2c/d97/f2e [1949580,11449] 0 2024-04-22T15:49:39.806 INFO:tasks.workunit.client.0.smithi016.stdout:3/660: rename d7/d1b/d21/fa5 to d7/d17/fca 0 2024-04-22T15:49:39.807 INFO:tasks.workunit.client.0.smithi016.stdout:2/549: fdatasync d2/d1a/d7b/f8d 0 2024-04-22T15:49:39.825 INFO:tasks.workunit.client.0.smithi016.stdout:3/661: creat d7/d1b/d21/fcb x:0 0 0 2024-04-22T15:49:39.827 INFO:tasks.workunit.client.0.smithi016.stdout:2/550: rename f0 to d2/d2c/d97/d67/fcf 0 2024-04-22T15:49:39.827 INFO:tasks.workunit.client.0.smithi016.stdout:1/751: link d3/d4/da/cb0 d3/d4/d19/d20/c100 0 2024-04-22T15:49:39.829 INFO:tasks.workunit.client.0.smithi016.stdout:9/586: getdents d9/d20/dac/dae 0 2024-04-22T15:49:39.832 INFO:tasks.workunit.client.0.smithi016.stdout:6/672: mkdir d3/de5 0 2024-04-22T15:49:39.833 INFO:tasks.workunit.client.0.smithi016.stdout:6/673: fdatasync d3/da/db/d14/f86 0 2024-04-22T15:49:39.834 INFO:tasks.workunit.client.0.smithi016.stdout:4/582: mknod d7/d21/d42/d59/d88/d9d/cbf 0 2024-04-22T15:49:39.835 INFO:tasks.workunit.client.0.smithi016.stdout:4/583: dread - d7/d21/d45/d49/d67/fbd zero size 2024-04-22T15:49:39.838 INFO:tasks.workunit.client.0.smithi016.stdout:1/752: symlink d3/d2e/d80/d82/l101 0 2024-04-22T15:49:39.839 INFO:tasks.workunit.client.0.smithi016.stdout:1/753: dread - d3/d4/d15/fce zero size 2024-04-22T15:49:39.843 INFO:tasks.workunit.client.0.smithi016.stdout:4/584: chown d7/d4e/c8f 160186 1 2024-04-22T15:49:39.844 INFO:tasks.workunit.client.0.smithi016.stdout:4/585: stat d7/d21/d42/d59/d88/ca0 0 2024-04-22T15:49:39.844 INFO:tasks.workunit.client.0.smithi016.stdout:2/551: symlink d2/ld0 0 2024-04-22T15:49:39.908 INFO:tasks.workunit.client.0.smithi016.stdout:1/754: creat d3/d4/d3a/d6b/d77/f102 x:0 0 0 2024-04-22T15:49:39.910 INFO:tasks.workunit.client.0.smithi016.stdout:7/627: truncate d3/dc/d22/f6b 865134 0 2024-04-22T15:49:39.911 INFO:tasks.workunit.client.0.smithi016.stdout:2/552: symlink d2/d23/d92/d55/ld1 0 2024-04-22T15:49:39.913 INFO:tasks.workunit.client.0.smithi016.stdout:9/587: creat d9/de/d23/d24/fb0 x:0 0 0 2024-04-22T15:49:39.915 INFO:tasks.workunit.client.0.smithi016.stdout:7/628: write d3/f99 [547665,46763] 0 2024-04-22T15:49:39.916 INFO:tasks.workunit.client.0.smithi016.stdout:3/662: fsync d7/d17/d18/d27/fa3 0 2024-04-22T15:49:39.916 INFO:tasks.workunit.client.0.smithi016.stdout:1/755: fdatasync d3/d4/d37/f40 0 2024-04-22T15:49:39.917 INFO:tasks.workunit.client.0.smithi016.stdout:9/588: fsync d9/d5d/f9a 0 2024-04-22T15:49:39.919 INFO:tasks.workunit.client.0.smithi016.stdout:5/582: dwrite df/d15/d1a/d28/f2b [0,4194304] 0 2024-04-22T15:49:39.920 INFO:tasks.workunit.client.0.smithi016.stdout:2/553: write d2/de/f84 [941512,7093] 0 2024-04-22T15:49:39.922 INFO:tasks.workunit.client.0.smithi016.stdout:4/586: rename d7/d21/d45/d49/d9b/l9f to d7/d21/d42/d59/d88/d9d/dad/db9/lc0 0 2024-04-22T15:49:39.925 INFO:tasks.workunit.client.0.smithi016.stdout:0/665: dwrite dc/d13/d28/d65/d1f/f53 [0,4194304] 0 2024-04-22T15:49:39.925 INFO:tasks.workunit.client.0.smithi016.stdout:1/756: chown d3/d4/d19/d20/d2b/f4a 781316 1 2024-04-22T15:49:39.926 INFO:tasks.workunit.client.0.smithi016.stdout:5/583: creat df/d15/d1a/d50/fae x:0 0 0 2024-04-22T15:49:39.929 INFO:tasks.workunit.client.0.smithi016.stdout:3/663: write d7/d17/fca [3117572,45977] 0 2024-04-22T15:49:39.932 INFO:tasks.workunit.client.0.smithi016.stdout:0/666: symlink dc/d7d/lde 0 2024-04-22T15:49:39.933 INFO:tasks.workunit.client.0.smithi016.stdout:0/667: chown dc/d13/d14/c20 5 1 2024-04-22T15:49:39.933 INFO:tasks.workunit.client.0.smithi016.stdout:1/757: mknod d3/d4/dc4/c103 0 2024-04-22T15:49:39.939 INFO:tasks.workunit.client.0.smithi016.stdout:1/758: stat d3/d4/d19/l1a 0 2024-04-22T15:49:39.945 INFO:tasks.workunit.client.0.smithi016.stdout:2/554: getdents d2/d2c/d97/d34/d60 0 2024-04-22T15:49:39.946 INFO:tasks.workunit.client.0.smithi016.stdout:0/668: symlink dc/d13/d28/d65/d40/d4a/db2/ldf 0 2024-04-22T15:49:39.947 INFO:tasks.workunit.client.0.smithi016.stdout:9/589: link d9/d5d/l95 d9/de/lb1 0 2024-04-22T15:49:39.948 INFO:tasks.workunit.client.0.smithi016.stdout:0/669: chown dc/d13/d28/d65/d1f/f53 25828 1 2024-04-22T15:49:39.949 INFO:tasks.workunit.client.0.smithi016.stdout:1/759: mkdir d3/d4/da/d7e/d104 0 2024-04-22T15:49:39.950 INFO:tasks.workunit.client.0.smithi016.stdout:5/584: getdents df/d5b/d5d 0 2024-04-22T15:49:39.953 INFO:tasks.workunit.client.0.smithi016.stdout:2/555: rename d2/d4d/d64/c9c to d2/d2c/d97/d34/cd2 0 2024-04-22T15:49:39.958 INFO:tasks.workunit.client.0.smithi016.stdout:5/585: rename df/d22/c91 to df/d15/d1a/d50/d77/d7d/d8b/d8c/caf 0 2024-04-22T15:49:39.961 INFO:tasks.workunit.client.0.smithi016.stdout:9/590: creat d9/de/d23/d24/d25/da6/d4f/fb2 x:0 0 0 2024-04-22T15:49:39.962 INFO:tasks.workunit.client.0.smithi016.stdout:3/664: write d7/d17/d18/f67 [38305,119881] 0 2024-04-22T15:49:39.963 INFO:tasks.workunit.client.0.smithi016.stdout:0/670: mknod dc/d13/d22/ce0 0 2024-04-22T15:49:39.965 INFO:tasks.workunit.client.0.smithi016.stdout:5/586: readlink df/d15/d1a/l1c 0 2024-04-22T15:49:39.967 INFO:tasks.workunit.client.0.smithi016.stdout:2/556: mknod d2/d1a/d7b/dc6/cd3 0 2024-04-22T15:49:39.967 INFO:tasks.workunit.client.0.smithi016.stdout:5/587: dread - df/d15/d1a/d50/d77/fa1 zero size 2024-04-22T15:49:39.969 INFO:tasks.workunit.client.0.smithi016.stdout:9/591: symlink d9/d20/dac/lb3 0 2024-04-22T15:49:39.970 INFO:tasks.workunit.client.0.smithi016.stdout:5/588: dread df/d15/d1a/d50/d77/d7a/f65 [0,4194304] 0 2024-04-22T15:49:39.971 INFO:tasks.workunit.client.0.smithi016.stdout:9/592: write d9/de/d23/d24/fa4 [566560,103517] 0 2024-04-22T15:49:39.972 INFO:tasks.workunit.client.0.smithi016.stdout:0/671: unlink dc/d13/l2a 0 2024-04-22T15:49:39.977 INFO:tasks.workunit.client.0.smithi016.stdout:2/557: mkdir d2/d23/db8/dd4 0 2024-04-22T15:49:39.977 INFO:tasks.workunit.client.0.smithi016.stdout:0/672: mkdir dc/d13/d28/d65/d57/dcb/de1 0 2024-04-22T15:49:39.978 INFO:tasks.workunit.client.0.smithi016.stdout:2/558: write d2/d2c/f82 [657903,111722] 0 2024-04-22T15:49:39.978 INFO:tasks.workunit.client.0.smithi016.stdout:3/665: creat d7/d17/d18/d32/d3c/fcc x:0 0 0 2024-04-22T15:49:39.980 INFO:tasks.workunit.client.0.smithi016.stdout:5/589: rename df/d15/d1a/d50/d77/c36 to df/d15/d1a/d50/d77/d7d/cb0 0 2024-04-22T15:49:39.981 INFO:tasks.workunit.client.0.smithi016.stdout:9/593: symlink d9/de/d23/d24/d25/da6/lb4 0 2024-04-22T15:49:39.983 INFO:tasks.workunit.client.0.smithi016.stdout:2/559: creat d2/d1a/d7b/dc6/fd5 x:0 0 0 2024-04-22T15:49:39.986 INFO:tasks.workunit.client.0.smithi016.stdout:3/666: link d7/d1b/d21/f73 d7/d1b/d21/fcd 0 2024-04-22T15:49:39.987 INFO:tasks.workunit.client.0.smithi016.stdout:2/560: symlink d2/d1a/d7b/dcc/ld6 0 2024-04-22T15:49:39.989 INFO:tasks.workunit.client.0.smithi016.stdout:2/561: stat d2/d23/db8 0 2024-04-22T15:49:39.991 INFO:tasks.workunit.client.0.smithi016.stdout:2/562: symlink d2/d1a/ld7 0 2024-04-22T15:49:39.995 INFO:tasks.workunit.client.0.smithi016.stdout:0/673: dread dc/d13/d28/d65/d30/f44 [0,4194304] 0 2024-04-22T15:49:39.997 INFO:tasks.workunit.client.0.smithi016.stdout:9/594: dread d9/d37/f4a [0,4194304] 0 2024-04-22T15:49:39.997 INFO:tasks.workunit.client.0.smithi016.stdout:0/674: dread - dc/d13/d28/d65/d40/dbb/fc2 zero size 2024-04-22T15:49:39.997 INFO:tasks.workunit.client.0.smithi016.stdout:2/563: mknod d2/d23/db8/dd4/cd8 0 2024-04-22T15:49:40.000 INFO:tasks.workunit.client.0.smithi016.stdout:2/564: write d2/d2c/d97/d34/d60/f83 [636637,108108] 0 2024-04-22T15:49:40.000 INFO:tasks.workunit.client.0.smithi016.stdout:0/675: write dc/d13/d28/d65/d40/f56 [5821919,33020] 0 2024-04-22T15:49:40.000 INFO:tasks.workunit.client.0.smithi016.stdout:9/595: truncate d9/de/d23/d24/d25/f41 309212 0 2024-04-22T15:49:40.001 INFO:tasks.workunit.client.0.smithi016.stdout:3/667: link d7/d17/d18/d32/d42/d63/d86/lb3 d7/d17/d18/d32/d3c/lce 0 2024-04-22T15:49:40.002 INFO:tasks.workunit.client.0.smithi016.stdout:2/565: write d2/d9/f80 [471955,77636] 0 2024-04-22T15:49:40.004 INFO:tasks.workunit.client.0.smithi016.stdout:9/596: write d9/de/d23/d24/f53 [1960266,87665] 0 2024-04-22T15:49:40.004 INFO:tasks.workunit.client.0.smithi016.stdout:2/566: write d2/d2c/d97/fa4 [43536,57110] 0 2024-04-22T15:49:40.006 INFO:tasks.workunit.client.0.smithi016.stdout:2/567: dread - d2/d1a/d7b/dcc/fc8 zero size 2024-04-22T15:49:40.007 INFO:tasks.workunit.client.0.smithi016.stdout:3/668: unlink d7/f5e 0 2024-04-22T15:49:40.011 INFO:tasks.workunit.client.0.smithi016.stdout:0/676: creat dc/d13/d14/d55/fe2 x:0 0 0 2024-04-22T15:49:40.014 INFO:tasks.workunit.client.0.smithi016.stdout:3/669: dwrite d7/d17/d18/d27/f54 [0,4194304] 0 2024-04-22T15:49:40.016 INFO:tasks.workunit.client.0.smithi016.stdout:2/568: rename d2/de/d2a/f93 to d2/d23/db8/fd9 0 2024-04-22T15:49:40.016 INFO:tasks.workunit.client.0.smithi016.stdout:2/569: chown d2/de/d2a/f42 2754759 1 2024-04-22T15:49:40.017 INFO:tasks.workunit.client.0.smithi016.stdout:0/677: creat dc/d13/d14/d55/fe3 x:0 0 0 2024-04-22T15:49:40.020 INFO:tasks.workunit.client.0.smithi016.stdout:9/597: symlink d9/de/d23/d24/d25/da6/lb5 0 2024-04-22T15:49:40.022 INFO:tasks.workunit.client.0.smithi016.stdout:9/598: write d9/de/d23/d24/f39 [4127877,128205] 0 2024-04-22T15:49:40.025 INFO:tasks.workunit.client.0.smithi016.stdout:2/570: symlink d2/d1a/d4c/lda 0 2024-04-22T15:49:40.025 INFO:tasks.workunit.client.0.smithi016.stdout:0/678: creat dc/d13/d28/d65/d40/d4a/d61/fe4 x:0 0 0 2024-04-22T15:49:40.027 INFO:tasks.workunit.client.0.smithi016.stdout:3/670: symlink d7/d17/d18/d32/d3c/d47/dc3/lcf 0 2024-04-22T15:49:40.029 INFO:tasks.workunit.client.0.smithi016.stdout:5/590: dwrite df/d15/d1a/f53 [0,4194304] 0 2024-04-22T15:49:40.030 INFO:tasks.workunit.client.0.smithi016.stdout:3/671: chown d7/d17/d18/d32/d42/d63 498 1 2024-04-22T15:49:40.031 INFO:tasks.workunit.client.0.smithi016.stdout:9/599: read f7 [1006895,93146] 0 2024-04-22T15:49:40.033 INFO:tasks.workunit.client.0.smithi016.stdout:9/600: chown d9/de/d23/f9c 19 1 2024-04-22T15:49:40.034 INFO:tasks.workunit.client.0.smithi016.stdout:5/591: mkdir df/d5b/db1 0 2024-04-22T15:49:40.042 INFO:tasks.workunit.client.0.smithi016.stdout:9/601: dwrite d9/de/d23/d24/f2e [4194304,4194304] 0 2024-04-22T15:49:40.043 INFO:tasks.workunit.client.0.smithi016.stdout:3/672: rename d7/d17/d18/d27/f54 to d7/d17/fd0 0 2024-04-22T15:49:40.047 INFO:tasks.workunit.client.0.smithi016.stdout:5/592: dwrite df/d15/d1a/d50/d77/d7d/d8b/d93/fa7 [0,4194304] 0 2024-04-22T15:49:40.052 INFO:tasks.workunit.client.0.smithi016.stdout:9/602: dread d9/d37/f4a [0,4194304] 0 2024-04-22T15:49:40.055 INFO:tasks.workunit.client.0.smithi016.stdout:3/673: chown d7/d1b/d21/d55/dac/cc8 32882734 1 2024-04-22T15:49:40.058 INFO:tasks.workunit.client.0.smithi016.stdout:2/571: getdents d2/d1a/d4c 0 2024-04-22T15:49:40.059 INFO:tasks.workunit.client.0.smithi016.stdout:5/593: dwrite f2 [0,4194304] 0 2024-04-22T15:49:40.063 INFO:tasks.workunit.client.0.smithi016.stdout:5/594: write df/d22/f24 [391658,77200] 0 2024-04-22T15:49:40.066 INFO:tasks.workunit.client.0.smithi016.stdout:9/603: dwrite d9/de/d23/d24/f76 [0,4194304] 0 2024-04-22T15:49:40.077 INFO:tasks.workunit.client.0.smithi016.stdout:5/595: rename df/d15/d1a/d50/d77/f3c to df/d90/dad/fb2 0 2024-04-22T15:49:40.077 INFO:tasks.workunit.client.0.smithi016.stdout:9/604: write d9/de/d23/d30/f75 [2961581,75544] 0 2024-04-22T15:49:40.080 INFO:tasks.workunit.client.0.smithi016.stdout:3/674: dwrite d7/d17/d18/fae [0,4194304] 0 2024-04-22T15:49:40.083 INFO:tasks.workunit.client.0.smithi016.stdout:5/596: dread df/d15/d1a/d50/d77/d7a/f65 [0,4194304] 0 2024-04-22T15:49:40.085 INFO:tasks.workunit.client.0.smithi016.stdout:9/605: stat d9/de/d23/d24/d25/da6/d4f/l94 0 2024-04-22T15:49:40.086 INFO:tasks.workunit.client.0.smithi016.stdout:9/606: chown d9/l2d 0 1 2024-04-22T15:49:40.086 INFO:tasks.workunit.client.0.smithi016.stdout:5/597: mkdir df/d5b/db3 0 2024-04-22T15:49:40.088 INFO:tasks.workunit.client.0.smithi016.stdout:5/598: dread - df/d5b/d5d/fab zero size 2024-04-22T15:49:40.090 INFO:tasks.workunit.client.0.smithi016.stdout:5/599: chown df/d15/d1a/d50/c51 86056043 1 2024-04-22T15:49:40.090 INFO:tasks.workunit.client.0.smithi016.stdout:5/600: fdatasync df/f9b 0 2024-04-22T15:49:40.091 INFO:tasks.workunit.client.0.smithi016.stdout:9/607: dread d9/de/d23/d24/d25/f41 [0,4194304] 0 2024-04-22T15:49:40.091 INFO:tasks.workunit.client.0.smithi016.stdout:3/675: getdents d7/d75 0 2024-04-22T15:49:40.094 INFO:tasks.workunit.client.0.smithi016.stdout:2/572: dread d2/d4d/d76/f9a [0,4194304] 0 2024-04-22T15:49:40.100 INFO:tasks.workunit.client.0.smithi016.stdout:3/676: dread d7/d17/d18/d27/fa3 [0,4194304] 0 2024-04-22T15:49:40.100 INFO:tasks.workunit.client.0.smithi016.stdout:5/601: getdents df/d15/d1a 0 2024-04-22T15:49:40.100 INFO:tasks.workunit.client.0.smithi016.stdout:2/573: truncate d2/d1a/f5e 1990106 0 2024-04-22T15:49:40.103 INFO:tasks.workunit.client.0.smithi016.stdout:9/608: dwrite d9/de/d23/fa5 [0,4194304] 0 2024-04-22T15:49:40.105 INFO:tasks.workunit.client.0.smithi016.stdout:2/574: fdatasync d2/de/d2a/f49 0 2024-04-22T15:49:40.106 INFO:tasks.workunit.client.0.smithi016.stdout:3/677: dread d7/d17/d18/d27/fa3 [0,4194304] 0 2024-04-22T15:49:40.107 INFO:tasks.workunit.client.0.smithi016.stdout:5/602: creat df/d15/d1a/d50/d77/fb4 x:0 0 0 2024-04-22T15:49:40.107 INFO:tasks.workunit.client.0.smithi016.stdout:3/678: fdatasync d7/d1b/d21/fcb 0 2024-04-22T15:49:40.109 INFO:tasks.workunit.client.0.smithi016.stdout:9/609: symlink d9/de/d23/lb6 0 2024-04-22T15:49:40.112 INFO:tasks.workunit.client.0.smithi016.stdout:2/575: link d2/f2b d2/d2c/d97/d34/fdb 0 2024-04-22T15:49:40.119 INFO:tasks.workunit.client.0.smithi016.stdout:2/576: dwrite d2/d23/d92/d55/fa7 [0,4194304] 0 2024-04-22T15:49:40.119 INFO:tasks.workunit.client.0.smithi016.stdout:5/603: mknod df/d5b/db1/cb5 0 2024-04-22T15:49:40.123 INFO:tasks.workunit.client.0.smithi016.stdout:2/577: creat d2/d1a/d7b/fdc x:0 0 0 2024-04-22T15:49:40.124 INFO:tasks.workunit.client.0.smithi016.stdout:5/604: creat df/fb6 x:0 0 0 2024-04-22T15:49:40.125 INFO:tasks.workunit.client.0.smithi016.stdout:5/605: fsync df/f76 0 2024-04-22T15:49:40.131 INFO:tasks.workunit.client.0.smithi016.stdout:2/578: mkdir d2/d2c/daa/ddd 0 2024-04-22T15:49:40.131 INFO:tasks.workunit.client.0.smithi016.stdout:3/679: dwrite d7/d1b/d21/f73 [0,4194304] 0 2024-04-22T15:49:40.137 INFO:tasks.workunit.client.0.smithi016.stdout:9/610: dread d9/d20/f59 [0,4194304] 0 2024-04-22T15:49:40.140 INFO:tasks.workunit.client.0.smithi016.stdout:9/611: stat d9/de/d26/c45 0 2024-04-22T15:49:40.141 INFO:tasks.workunit.client.0.smithi016.stdout:2/579: rename d2/de/f5d to d2/d1a/d7b/dc6/fde 0 2024-04-22T15:49:40.144 INFO:tasks.workunit.client.0.smithi016.stdout:9/612: dread - d9/de/d23/f9c zero size 2024-04-22T15:49:40.144 INFO:tasks.workunit.client.0.smithi016.stdout:5/606: dwrite df/d15/d1a/d28/f2c [4194304,4194304] 0 2024-04-22T15:49:40.145 INFO:tasks.workunit.client.0.smithi016.stdout:2/580: fsync d2/d9/f80 0 2024-04-22T15:49:40.148 INFO:tasks.workunit.client.0.smithi016.stdout:3/680: dwrite d7/d17/d18/d32/d42/d68/f7a [0,4194304] 0 2024-04-22T15:49:40.154 INFO:tasks.workunit.client.0.smithi016.stdout:5/607: dwrite df/d15/d1a/d28/f3f [0,4194304] 0 2024-04-22T15:49:40.159 INFO:tasks.workunit.client.0.smithi016.stdout:9/613: getdents d9/de/d23/d30 0 2024-04-22T15:49:40.159 INFO:tasks.workunit.client.0.smithi016.stdout:2/581: link d2/d1a/c2d d2/d4d/d8e/cdf 0 2024-04-22T15:49:40.159 INFO:tasks.workunit.client.0.smithi016.stdout:5/608: write df/d5b/d70/f8d [283464,5219] 0 2024-04-22T15:49:40.163 INFO:tasks.workunit.client.0.smithi016.stdout:9/614: creat d9/d37/fb7 x:0 0 0 2024-04-22T15:49:40.164 INFO:tasks.workunit.client.0.smithi016.stdout:9/615: truncate d9/d5d/f9a 4764441 0 2024-04-22T15:49:40.164 INFO:tasks.workunit.client.0.smithi016.stdout:2/582: getdents d2/de/d2a 0 2024-04-22T15:49:40.166 INFO:tasks.workunit.client.0.smithi016.stdout:2/583: chown d2/d4d/d64/cb0 0 1 2024-04-22T15:49:40.167 INFO:tasks.workunit.client.0.smithi016.stdout:9/616: mkdir d9/d5d/db8 0 2024-04-22T15:49:40.169 INFO:tasks.workunit.client.0.smithi016.stdout:2/584: mknod d2/d1a/d7b/dcc/ce0 0 2024-04-22T15:49:40.172 INFO:tasks.workunit.client.0.smithi016.stdout:2/585: creat d2/d23/d92/fe1 x:0 0 0 2024-04-22T15:49:40.176 INFO:tasks.workunit.client.0.smithi016.stdout:2/586: unlink d2/d2c/d97/d67/f86 0 2024-04-22T15:49:40.182 INFO:tasks.workunit.client.0.smithi016.stdout:2/587: rename d2/d23/c30 to d2/ce2 0 2024-04-22T15:49:40.184 INFO:tasks.workunit.client.0.smithi016.stdout:2/588: truncate d2/d2c/f5b 5211535 0 2024-04-22T15:49:40.186 INFO:tasks.workunit.client.0.smithi016.stdout:9/617: dwrite f7 [0,4194304] 0 2024-04-22T15:49:40.186 INFO:tasks.workunit.client.0.smithi016.stdout:8/548: sync 2024-04-22T15:49:40.189 INFO:tasks.workunit.client.0.smithi016.stdout:9/618: stat d9/de/d23/d24/d25/d36/d87 0 2024-04-22T15:49:40.190 INFO:tasks.workunit.client.0.smithi016.stdout:2/589: mknod d2/d4d/ce3 0 2024-04-22T15:49:40.194 INFO:tasks.workunit.client.0.smithi016.stdout:2/590: symlink d2/d4d/d64/le4 0 2024-04-22T15:49:40.195 INFO:tasks.workunit.client.0.smithi016.stdout:9/619: creat d9/de/d26/d5a/fb9 x:0 0 0 2024-04-22T15:49:40.195 INFO:tasks.workunit.client.0.smithi016.stdout:8/549: dwrite d2/f7a [0,4194304] 0 2024-04-22T15:49:40.197 INFO:tasks.workunit.client.0.smithi016.stdout:2/591: unlink d2/d2c/d97/d34/f4b 0 2024-04-22T15:49:40.199 INFO:tasks.workunit.client.0.smithi016.stdout:9/620: write d9/d37/f4a [4111561,13138] 0 2024-04-22T15:49:40.201 INFO:tasks.workunit.client.0.smithi016.stdout:9/621: chown d9/de/d23/d24/d25/da6/d4f/f69 1 1 2024-04-22T15:49:40.202 INFO:tasks.workunit.client.0.smithi016.stdout:2/592: mkdir d2/d1a/d7b/dcc/de5 0 2024-04-22T15:49:40.205 INFO:tasks.workunit.client.0.smithi016.stdout:8/550: rename d2/d10/l46 to d2/d32/la8 0 2024-04-22T15:49:40.208 INFO:tasks.workunit.client.0.smithi016.stdout:8/551: write d2/d82/d44/d66/f18 [2354565,110023] 0 2024-04-22T15:49:40.209 INFO:tasks.workunit.client.0.smithi016.stdout:8/552: stat d2/l2b 0 2024-04-22T15:49:40.209 INFO:tasks.workunit.client.0.smithi016.stdout:2/593: dwrite d2/d1a/d4c/d9f/dae/fb3 [0,4194304] 0 2024-04-22T15:49:40.212 INFO:tasks.workunit.client.0.smithi016.stdout:8/553: write d2/d82/d44/d66/d1c/d65/f9c [724811,79797] 0 2024-04-22T15:49:40.212 INFO:tasks.workunit.client.0.smithi016.stdout:9/622: rmdir d9/de/d26/d5a 39 2024-04-22T15:49:40.218 INFO:tasks.workunit.client.0.smithi016.stdout:8/554: write d2/f8 [24736,57112] 0 2024-04-22T15:49:40.218 INFO:tasks.workunit.client.0.smithi016.stdout:2/594: creat d2/d4d/d64/fe6 x:0 0 0 2024-04-22T15:49:40.218 INFO:tasks.workunit.client.0.smithi016.stdout:8/555: readlink d2/d82/d44/d66/d7e/l89 0 2024-04-22T15:49:40.222 INFO:tasks.workunit.client.0.smithi016.stdout:8/556: unlink d2/d10/f61 0 2024-04-22T15:49:40.227 INFO:tasks.workunit.client.0.smithi016.stdout:9/623: unlink d9/de/d23/d24/d25/d36/la8 0 2024-04-22T15:49:40.237 INFO:tasks.workunit.client.0.smithi016.stdout:8/557: unlink d2/d82/d44/d66/f18 0 2024-04-22T15:49:40.238 INFO:tasks.workunit.client.0.smithi016.stdout:9/624: creat d9/de/d23/d30/da2/fba x:0 0 0 2024-04-22T15:49:40.239 INFO:tasks.workunit.client.0.smithi016.stdout:9/625: stat d9/de/d23/d24/d25/f41 0 2024-04-22T15:49:40.241 INFO:tasks.workunit.client.0.smithi016.stdout:8/558: mkdir d2/d10/d43/da9 0 2024-04-22T15:49:40.242 INFO:tasks.workunit.client.0.smithi016.stdout:2/595: truncate d2/d23/d92/d55/fa7 3394661 0 2024-04-22T15:49:40.243 INFO:tasks.workunit.client.0.smithi016.stdout:9/626: dread d9/d20/f59 [0,4194304] 0 2024-04-22T15:49:40.243 INFO:tasks.workunit.client.0.smithi016.stdout:8/559: write d2/d82/d44/d6f/fa6 [608577,125407] 0 2024-04-22T15:49:40.245 INFO:tasks.workunit.client.0.smithi016.stdout:9/627: chown d9/de/f16 615003245 1 2024-04-22T15:49:40.245 INFO:tasks.workunit.client.0.smithi016.stdout:2/596: mkdir d2/d2c/de7 0 2024-04-22T15:49:40.246 INFO:tasks.workunit.client.0.smithi016.stdout:8/560: creat d2/d82/d56/d99/faa x:0 0 0 2024-04-22T15:49:40.248 INFO:tasks.workunit.client.0.smithi016.stdout:8/561: write d2/d82/d44/d66/d1c/f81 [843841,103845] 0 2024-04-22T15:49:40.250 INFO:tasks.workunit.client.0.smithi016.stdout:2/597: rename d2/d2c/f5b to d2/d2c/d97/d67/dbf/fe8 0 2024-04-22T15:49:40.251 INFO:tasks.workunit.client.0.smithi016.stdout:2/598: readlink d2/d1a/d4c/l8b 0 2024-04-22T15:49:40.251 INFO:tasks.workunit.client.0.smithi016.stdout:7/629: sync 2024-04-22T15:49:40.251 INFO:tasks.workunit.client.0.smithi016.stdout:1/760: sync 2024-04-22T15:49:40.252 INFO:tasks.workunit.client.0.smithi016.stdout:6/674: sync 2024-04-22T15:49:40.252 INFO:tasks.workunit.client.0.smithi016.stdout:4/587: sync 2024-04-22T15:49:40.253 INFO:tasks.workunit.client.0.smithi016.stdout:8/562: mknod d2/d32/cab 0 2024-04-22T15:49:40.253 INFO:tasks.workunit.client.0.smithi016.stdout:1/761: chown d3/d4/d3a/l4e 0 1 2024-04-22T15:49:40.255 INFO:tasks.workunit.client.0.smithi016.stdout:8/563: write d2/d82/d52/f54 [1839314,58113] 0 2024-04-22T15:49:40.256 INFO:tasks.workunit.client.0.smithi016.stdout:6/675: write d3/da/d12/d13/d46/fa3 [939549,81539] 0 2024-04-22T15:49:40.259 INFO:tasks.workunit.client.0.smithi016.stdout:9/628: rename d9/de/d23/d24/d25/d36/d87/f96 to d9/de/d26/fbb 0 2024-04-22T15:49:40.260 INFO:tasks.workunit.client.0.smithi016.stdout:4/588: symlink d7/d21/lc1 0 2024-04-22T15:49:40.264 INFO:tasks.workunit.client.0.smithi016.stdout:2/599: creat d2/d1a/d7b/dcc/de5/fe9 x:0 0 0 2024-04-22T15:49:40.265 INFO:tasks.workunit.client.0.smithi016.stdout:7/630: mknod d3/dc/d1b/d5a/d59/d64/dc9/ce0 0 2024-04-22T15:49:40.266 INFO:tasks.workunit.client.0.smithi016.stdout:1/762: fsync d3/d4/d15/d38/d63/fa9 0 2024-04-22T15:49:40.269 INFO:tasks.workunit.client.0.smithi016.stdout:6/676: creat d3/da/db/d44/d68/fe6 x:0 0 0 2024-04-22T15:49:40.275 INFO:tasks.workunit.client.0.smithi016.stdout:2/600: mkdir d2/d1a/d4c/d77/dea 0 2024-04-22T15:49:40.276 INFO:tasks.workunit.client.0.smithi016.stdout:7/631: chown d3/dc/d1b/d5a/cba 0 1 2024-04-22T15:49:40.281 INFO:tasks.workunit.client.0.smithi016.stdout:1/763: mknod d3/d4/d3a/d91/db3/dd2/c105 0 2024-04-22T15:49:40.284 INFO:tasks.workunit.client.0.smithi016.stdout:6/677: unlink d3/da/d12/d13/d38/d84/ca8 0 2024-04-22T15:49:40.285 INFO:tasks.workunit.client.0.smithi016.stdout:6/678: dread d3/da/fc3 [0,4194304] 0 2024-04-22T15:49:40.286 INFO:tasks.workunit.client.0.smithi016.stdout:6/679: rename d3 to d3/da/ddd/de7 22 2024-04-22T15:49:40.288 INFO:tasks.workunit.client.0.smithi016.stdout:6/680: fsync d3/da/dd/d1b/d57/fbc 0 2024-04-22T15:49:40.288 INFO:tasks.workunit.client.0.smithi016.stdout:9/629: mkdir d9/dbc 0 2024-04-22T15:49:40.289 INFO:tasks.workunit.client.0.smithi016.stdout:0/679: sync 2024-04-22T15:49:40.291 INFO:tasks.workunit.client.0.smithi016.stdout:2/601: read d2/f3e [888441,116629] 0 2024-04-22T15:49:40.293 INFO:tasks.workunit.client.0.smithi016.stdout:1/764: creat d3/d4/da/d7e/f106 x:0 0 0 2024-04-22T15:49:40.293 INFO:tasks.workunit.client.0.smithi016.stdout:6/681: dread d3/da/db/d14/f9b [0,4194304] 0 2024-04-22T15:49:40.294 INFO:tasks.workunit.client.0.smithi016.stdout:1/765: fdatasync d3/d4/d15/d38/d63/d67/fd0 0 2024-04-22T15:49:40.296 INFO:tasks.workunit.client.0.smithi016.stdout:6/682: read d3/da/db/f17 [4414605,116926] 0 2024-04-22T15:49:40.302 INFO:tasks.workunit.client.0.smithi016.stdout:2/602: symlink d2/d1a/leb 0 2024-04-22T15:49:40.304 INFO:tasks.workunit.client.0.smithi016.stdout:4/589: getdents d7/d4e/d39 0 2024-04-22T15:49:40.304 INFO:tasks.workunit.client.0.smithi016.stdout:8/564: dwrite d2/d82/d44/d66/d1c/f8c [0,4194304] 0 2024-04-22T15:49:40.307 INFO:tasks.workunit.client.0.smithi016.stdout:1/766: truncate d3/d4/d37/f40 1232531 0 2024-04-22T15:49:40.309 INFO:tasks.workunit.client.0.smithi016.stdout:7/632: mkdir d3/dc/d1b/d31/d88/de1 0 2024-04-22T15:49:40.312 INFO:tasks.workunit.client.0.smithi016.stdout:2/603: mknod d2/d4d/d64/cec 0 2024-04-22T15:49:40.313 INFO:tasks.workunit.client.0.smithi016.stdout:9/630: dread d9/d20/f40 [0,4194304] 0 2024-04-22T15:49:40.314 INFO:tasks.workunit.client.0.smithi016.stdout:9/631: chown d9/de/d23/d30/f86 3 1 2024-04-22T15:49:40.315 INFO:tasks.workunit.client.0.smithi016.stdout:4/590: symlink d7/d21/d42/d59/d88/d9d/lc2 0 2024-04-22T15:49:40.316 INFO:tasks.workunit.client.0.smithi016.stdout:0/680: write dc/d13/d28/d65/d30/d96/fc7 [1472,118509] 0 2024-04-22T15:49:40.318 INFO:tasks.workunit.client.0.smithi016.stdout:9/632: write d9/d37/fb7 [1032013,127759] 0 2024-04-22T15:49:40.318 INFO:tasks.workunit.client.0.smithi016.stdout:8/565: creat d2/d10/d43/da9/fac x:0 0 0 2024-04-22T15:49:40.319 INFO:tasks.workunit.client.0.smithi016.stdout:6/683: creat d3/da/db/fe8 x:0 0 0 2024-04-22T15:49:40.327 INFO:tasks.workunit.client.0.smithi016.stdout:2/604: dwrite d2/d2c/d97/d34/d60/f83 [0,4194304] 0 2024-04-22T15:49:40.328 INFO:tasks.workunit.client.0.smithi016.stdout:8/566: rename d2/d10/d43/d4a to d2/d48/dad 0 2024-04-22T15:49:40.335 INFO:tasks.workunit.client.0.smithi016.stdout:7/633: getdents d3/dc/d1b/d5a/d9/df 0 2024-04-22T15:49:40.336 INFO:tasks.workunit.client.0.smithi016.stdout:8/567: write d2/f27 [6942596,16918] 0 2024-04-22T15:49:40.337 INFO:tasks.workunit.client.0.smithi016.stdout:4/591: dwrite d7/d21/d42/d80/fa5 [4194304,4194304] 0 2024-04-22T15:49:40.339 INFO:tasks.workunit.client.0.smithi016.stdout:8/568: chown d2/d82/d44/d6f/fa6 1553578 1 2024-04-22T15:49:40.340 INFO:tasks.workunit.client.0.smithi016.stdout:2/605: creat d2/d1a/d4c/d9f/dae/fed x:0 0 0 2024-04-22T15:49:40.346 INFO:tasks.workunit.client.0.smithi016.stdout:2/606: creat d2/d2c/d97/d67/dbf/fee x:0 0 0 2024-04-22T15:49:40.347 INFO:tasks.workunit.client.0.smithi016.stdout:9/633: truncate d9/de/d23/f78 2959720 0 2024-04-22T15:49:40.348 INFO:tasks.workunit.client.0.smithi016.stdout:0/681: getdents dc/d13/d28/d65/d40/d4a 0 2024-04-22T15:49:40.351 INFO:tasks.workunit.client.0.smithi016.stdout:7/634: link d3/dc/d9f/fb1 d3/dc/d1b/d5a/d9/df/d94/ddf/fe2 0 2024-04-22T15:49:40.351 INFO:tasks.workunit.client.0.smithi016.stdout:2/607: write d2/d23/f6c [4500457,104841] 0 2024-04-22T15:49:40.351 INFO:tasks.workunit.client.0.smithi016.stdout:8/569: dread d2/d82/d44/d66/d1c/f8c [0,4194304] 0 2024-04-22T15:49:40.352 INFO:tasks.workunit.client.0.smithi016.stdout:0/682: chown dc/d13/d28/d65/d30/d75 0 1 2024-04-22T15:49:40.357 INFO:tasks.workunit.client.0.smithi016.stdout:1/767: dread d3/d4/dc4/fd7 [0,4194304] 0 2024-04-22T15:49:40.358 INFO:tasks.workunit.client.0.smithi016.stdout:0/683: dread - dc/d13/d28/d65/d57/dcb/fd7 zero size 2024-04-22T15:49:40.358 INFO:tasks.workunit.client.0.smithi016.stdout:8/570: mknod d2/d10/d43/cae 0 2024-04-22T15:49:40.359 INFO:tasks.workunit.client.0.smithi016.stdout:7/635: mkdir d3/dc/d33/d84/de3 0 2024-04-22T15:49:40.359 INFO:tasks.workunit.client.0.smithi016.stdout:2/608: write d2/d1a/d7b/fdc [142331,25841] 0 2024-04-22T15:49:40.362 INFO:tasks.workunit.client.0.smithi016.stdout:4/592: dwrite d7/d21/d45/d49/f56 [0,4194304] 0 2024-04-22T15:49:40.365 INFO:tasks.workunit.client.0.smithi016.stdout:9/634: dwrite d9/d37/f3c [0,4194304] 0 2024-04-22T15:49:40.366 INFO:tasks.workunit.client.0.smithi016.stdout:7/636: symlink d3/dc/d1b/d5a/d69/le4 0 2024-04-22T15:49:40.370 INFO:tasks.workunit.client.0.smithi016.stdout:0/684: dread - dc/d13/d28/d65/d40/d4a/d59/d78/dbd/fda zero size 2024-04-22T15:49:40.371 INFO:tasks.workunit.client.0.smithi016.stdout:8/571: mknod d2/d82/d44/d66/d79/caf 0 2024-04-22T15:49:40.374 INFO:tasks.workunit.client.0.smithi016.stdout:8/572: write d2/f5a [3935666,103647] 0 2024-04-22T15:49:40.374 INFO:tasks.workunit.client.0.smithi016.stdout:1/768: creat d3/f107 x:0 0 0 2024-04-22T15:49:40.374 INFO:tasks.workunit.client.0.smithi016.stdout:9/635: rename d9/dbc to d9/d20/dac/dbd 0 2024-04-22T15:49:40.375 INFO:tasks.workunit.client.0.smithi016.stdout:0/685: chown dc/d13/d28/d65/d40/d4a/d61/d82/fb5 19045613 1 2024-04-22T15:49:40.376 INFO:tasks.workunit.client.0.smithi016.stdout:9/636: chown d9/de/d23/d24/d25/da6/lb4 2 1 2024-04-22T15:49:40.378 INFO:tasks.workunit.client.0.smithi016.stdout:7/637: mknod d3/dc/d9f/ce5 0 2024-04-22T15:49:40.379 INFO:tasks.workunit.client.0.smithi016.stdout:2/609: symlink d2/d1a/d7b/dcc/de5/lef 0 2024-04-22T15:49:40.383 INFO:tasks.workunit.client.0.smithi016.stdout:8/573: mkdir d2/d82/d44/db0 0 2024-04-22T15:49:40.384 INFO:tasks.workunit.client.0.smithi016.stdout:7/638: dwrite d3/dc/f6f [0,4194304] 0 2024-04-22T15:49:40.384 INFO:tasks.workunit.client.0.smithi016.stdout:1/769: rmdir d3/d4/d3a/d96 39 2024-04-22T15:49:40.386 INFO:tasks.workunit.client.0.smithi016.stdout:9/637: mknod d9/d20/cbe 0 2024-04-22T15:49:40.387 INFO:tasks.workunit.client.0.smithi016.stdout:9/638: readlink d9/de/d23/d24/d25/da6/d4f/l94 0 2024-04-22T15:49:40.389 INFO:tasks.workunit.client.0.smithi016.stdout:4/593: truncate d7/f8 4948915 0 2024-04-22T15:49:40.390 INFO:tasks.workunit.client.0.smithi016.stdout:2/610: symlink d2/d1a/d7b/dcc/lf0 0 2024-04-22T15:49:40.390 INFO:tasks.workunit.client.0.smithi016.stdout:2/611: chown d2/de/l29 1 1 2024-04-22T15:49:40.391 INFO:tasks.workunit.client.0.smithi016.stdout:2/612: write d2/de/d2a/f49 [1952444,8469] 0 2024-04-22T15:49:40.392 INFO:tasks.workunit.client.0.smithi016.stdout:2/613: write d2/de/d2a/f49 [4274762,22583] 0 2024-04-22T15:49:40.393 INFO:tasks.workunit.client.0.smithi016.stdout:4/594: dread d7/d21/d45/d49/f56 [0,4194304] 0 2024-04-22T15:49:40.401 INFO:tasks.workunit.client.0.smithi016.stdout:9/639: truncate d9/d20/f40 1069073 0 2024-04-22T15:49:40.401 INFO:tasks.workunit.client.0.smithi016.stdout:2/614: dwrite d2/d1a/d7b/f8d [4194304,4194304] 0 2024-04-22T15:49:40.401 INFO:tasks.workunit.client.0.smithi016.stdout:7/639: fdatasync d3/d14/f2c 0 2024-04-22T15:49:40.403 INFO:tasks.workunit.client.0.smithi016.stdout:3/681: sync 2024-04-22T15:49:40.403 INFO:tasks.workunit.client.0.smithi016.stdout:5/609: sync 2024-04-22T15:49:40.404 INFO:tasks.workunit.client.0.smithi016.stdout:2/615: dread - d2/d2c/d97/d34/d60/f6f zero size 2024-04-22T15:49:40.408 INFO:tasks.workunit.client.0.smithi016.stdout:1/770: dwrite d3/d4/dc4/fd7 [0,4194304] 0 2024-04-22T15:49:40.411 INFO:tasks.workunit.client.0.smithi016.stdout:5/610: dread df/d15/d1a/d28/f3a [0,4194304] 0 2024-04-22T15:49:40.412 INFO:tasks.workunit.client.0.smithi016.stdout:1/771: dread d3/d2e/d80/d82/fae [0,4194304] 0 2024-04-22T15:49:40.415 INFO:tasks.workunit.client.0.smithi016.stdout:9/640: symlink d9/d37/lbf 0 2024-04-22T15:49:40.420 INFO:tasks.workunit.client.0.smithi016.stdout:3/682: mknod d7/d17/d18/d32/d42/d68/cd1 0 2024-04-22T15:49:40.426 INFO:tasks.workunit.client.0.smithi016.stdout:8/574: getdents d2/d82/d56 0 2024-04-22T15:49:40.428 INFO:tasks.workunit.client.0.smithi016.stdout:8/575: read d2/d82/d56/f90 [208582,88348] 0 2024-04-22T15:49:40.430 INFO:tasks.workunit.client.0.smithi016.stdout:5/611: unlink df/d15/d1a/d50/l68 0 2024-04-22T15:49:40.430 INFO:tasks.workunit.client.0.smithi016.stdout:3/683: dwrite d7/d17/d18/d32/d3c/d47/f33 [0,4194304] 0 2024-04-22T15:49:40.437 INFO:tasks.workunit.client.0.smithi016.stdout:9/641: rename d9/de/d23/d24/d25/da6/f83 to d9/de/d23/d30/fc0 0 2024-04-22T15:49:40.438 INFO:tasks.workunit.client.0.smithi016.stdout:4/595: getdents d7/d21/d42/d59/d88 0 2024-04-22T15:49:40.444 INFO:tasks.workunit.client.0.smithi016.stdout:8/576: creat d2/d82/d3e/fb1 x:0 0 0 2024-04-22T15:49:40.445 INFO:tasks.workunit.client.0.smithi016.stdout:1/772: creat d3/d28/dee/f108 x:0 0 0 2024-04-22T15:49:40.447 INFO:tasks.workunit.client.0.smithi016.stdout:5/612: rmdir df/d15/d1a/d50/d77/d7a 39 2024-04-22T15:49:40.447 INFO:tasks.workunit.client.0.smithi016.stdout:7/640: rename d3/d14 to d3/de6 0 2024-04-22T15:49:40.448 INFO:tasks.workunit.client.0.smithi016.stdout:2/616: dwrite d2/f4 [4194304,4194304] 0 2024-04-22T15:49:40.450 INFO:tasks.workunit.client.0.smithi016.stdout:3/684: dread d7/d17/d18/f2a [0,4194304] 0 2024-04-22T15:49:40.453 INFO:tasks.workunit.client.0.smithi016.stdout:3/685: dread d7/d1b/f85 [0,4194304] 0 2024-04-22T15:49:40.453 INFO:tasks.workunit.client.0.smithi016.stdout:9/642: rename d9/de/d23/d30 to d9/d57/d91/dc1 0 2024-04-22T15:49:40.455 INFO:tasks.workunit.client.0.smithi016.stdout:9/643: chown d9/de/d26/f2a 1791 1 2024-04-22T15:49:40.457 INFO:tasks.workunit.client.0.smithi016.stdout:9/644: truncate d9/de/d23/d24/d25/da6/f79 4517353 0 2024-04-22T15:49:40.457 INFO:tasks.workunit.client.0.smithi016.stdout:9/645: readlink d9/d20/dac/lb3 0 2024-04-22T15:49:40.459 INFO:tasks.workunit.client.0.smithi016.stdout:1/773: unlink d3/d4/d19/ff8 0 2024-04-22T15:49:40.459 INFO:tasks.workunit.client.0.smithi016.stdout:1/774: stat d3/d4/d15/d38/ffa 0 2024-04-22T15:49:40.460 INFO:tasks.workunit.client.0.smithi016.stdout:8/577: fdatasync d2/d82/f38 0 2024-04-22T15:49:40.464 INFO:tasks.workunit.client.0.smithi016.stdout:5/613: mkdir df/d15/d1a/d50/d77/d7d/d8b/d93/db7 0 2024-04-22T15:49:40.466 INFO:tasks.workunit.client.0.smithi016.stdout:2/617: unlink d2/d1a/d7b/dcc/l74 0 2024-04-22T15:49:40.466 INFO:tasks.workunit.client.0.smithi016.stdout:4/596: write d7/f16 [3334952,25904] 0 2024-04-22T15:49:40.470 INFO:tasks.workunit.client.0.smithi016.stdout:3/686: unlink d7/d1b/d21/f73 0 2024-04-22T15:49:40.472 INFO:tasks.workunit.client.0.smithi016.stdout:6/684: sync 2024-04-22T15:49:40.475 INFO:tasks.workunit.client.0.smithi016.stdout:6/685: read d3/da/d12/d13/f3f [827990,26760] 0 2024-04-22T15:49:40.476 INFO:tasks.workunit.client.0.smithi016.stdout:7/641: dread d3/dc/d1b/d5a/d9/fb [0,4194304] 0 2024-04-22T15:49:40.480 INFO:tasks.workunit.client.0.smithi016.stdout:6/686: truncate d3/da/db/d14/fc8 941884 0 2024-04-22T15:49:40.480 INFO:tasks.workunit.client.0.smithi016.stdout:4/597: rename d7/d21/d42/d80 to d7/d4e/d39/dc3 0 2024-04-22T15:49:40.483 INFO:tasks.workunit.client.0.smithi016.stdout:5/614: dwrite df/d15/d1a/d28/f3a [0,4194304] 0 2024-04-22T15:49:40.489 INFO:tasks.workunit.client.0.smithi016.stdout:7/642: dread d3/dc/d1b/d5a/d9/df/d94/d4c/d7a/f9e [0,4194304] 0 2024-04-22T15:49:40.491 INFO:tasks.workunit.client.0.smithi016.stdout:5/615: creat df/d15/d1a/d4e/d58/fb8 x:0 0 0 2024-04-22T15:49:40.491 INFO:tasks.workunit.client.0.smithi016.stdout:3/687: dwrite d7/d17/fca [4194304,4194304] 0 2024-04-22T15:49:40.498 INFO:tasks.workunit.client.0.smithi016.stdout:6/687: mkdir d3/d53/de9 0 2024-04-22T15:49:40.504 INFO:tasks.workunit.client.0.smithi016.stdout:9/646: write f2 [1835005,80060] 0 2024-04-22T15:49:40.506 INFO:tasks.workunit.client.0.smithi016.stdout:3/688: symlink d7/d17/d18/d32/d42/d7e/ld2 0 2024-04-22T15:49:40.508 INFO:tasks.workunit.client.0.smithi016.stdout:8/578: truncate d2/d82/d44/d66/d1c/f8c 1918053 0 2024-04-22T15:49:40.509 INFO:tasks.workunit.client.0.smithi016.stdout:8/579: write d2/f27 [530943,28214] 0 2024-04-22T15:49:40.510 INFO:tasks.workunit.client.0.smithi016.stdout:5/616: mknod df/d15/d1a/d50/d77/d7d/d8b/cb9 0 2024-04-22T15:49:40.510 INFO:tasks.workunit.client.0.smithi016.stdout:2/618: write d2/d2c/d97/d34/fa5 [481532,89264] 0 2024-04-22T15:49:40.512 INFO:tasks.workunit.client.0.smithi016.stdout:6/688: creat d3/da/db/d14/d4c/fea x:0 0 0 2024-04-22T15:49:40.515 INFO:tasks.workunit.client.0.smithi016.stdout:6/689: chown d3/da/db/d14/d16/l6b 1 1 2024-04-22T15:49:40.517 INFO:tasks.workunit.client.0.smithi016.stdout:5/617: dwrite f2 [0,4194304] 0 2024-04-22T15:49:40.521 INFO:tasks.workunit.client.0.smithi016.stdout:9/647: mknod d9/de/d23/d24/d25/da6/d4f/cc2 0 2024-04-22T15:49:40.522 INFO:tasks.workunit.client.0.smithi016.stdout:3/689: creat d7/d75/fd3 x:0 0 0 2024-04-22T15:49:40.525 INFO:tasks.workunit.client.0.smithi016.stdout:8/580: creat d2/d10/d47/fb2 x:0 0 0 2024-04-22T15:49:40.534 INFO:tasks.workunit.client.0.smithi016.stdout:8/581: dwrite d2/d82/d56/d99/faa [0,4194304] 0 2024-04-22T15:49:40.537 INFO:tasks.workunit.client.0.smithi016.stdout:9/648: rename d9/de/d23/d24/d25/d36/fa0 to d9/d5d/fc3 0 2024-04-22T15:49:40.544 INFO:tasks.workunit.client.0.smithi016.stdout:5/618: symlink df/d15/d1a/d50/d77/d94/lba 0 2024-04-22T15:49:40.546 INFO:tasks.workunit.client.0.smithi016.stdout:8/582: dwrite d2/d82/d44/d66/f20 [8388608,4194304] 0 2024-04-22T15:49:40.549 INFO:tasks.workunit.client.0.smithi016.stdout:3/690: rename d7/d17/d18/d32/d42/d7e to d7/d17/d18/d32/d42/d63/d86/dd4 0 2024-04-22T15:49:40.550 INFO:tasks.workunit.client.0.smithi016.stdout:3/691: stat d7/d75 0 2024-04-22T15:49:40.552 INFO:tasks.workunit.client.0.smithi016.stdout:9/649: chown d9/de/d23/d24/d25/c81 190 1 2024-04-22T15:49:40.553 INFO:tasks.workunit.client.0.smithi016.stdout:2/619: write d2/de/d2a/f42 [1471139,80841] 0 2024-04-22T15:49:40.555 INFO:tasks.workunit.client.0.smithi016.stdout:7/643: link d3/dc/l27 d3/dc/d1b/d31/le7 0 2024-04-22T15:49:40.556 INFO:tasks.workunit.client.0.smithi016.stdout:7/644: dread - d3/dc/d1b/d5a/d9/df/d94/d4c/fd4 zero size 2024-04-22T15:49:40.557 INFO:tasks.workunit.client.0.smithi016.stdout:5/619: mkdir df/d15/d1a/d50/d77/d7d/d8b/d93/dbb 0 2024-04-22T15:49:40.560 INFO:tasks.workunit.client.0.smithi016.stdout:8/583: creat d2/d48/dad/d8f/fb3 x:0 0 0 2024-04-22T15:49:40.565 INFO:tasks.workunit.client.0.smithi016.stdout:9/650: creat d9/d57/d91/dc1/fc4 x:0 0 0 2024-04-22T15:49:40.565 INFO:tasks.workunit.client.0.smithi016.stdout:6/690: write d3/da/f45 [1374037,37142] 0 2024-04-22T15:49:40.565 INFO:tasks.workunit.client.0.smithi016.stdout:7/645: read d3/dc/d1b/d5a/d9/df/d94/d4c/fdd [393259,115341] 0 2024-04-22T15:49:40.566 INFO:tasks.workunit.client.0.smithi016.stdout:2/620: fdatasync d2/d1a/d4c/f7a 0 2024-04-22T15:49:40.569 INFO:tasks.workunit.client.0.smithi016.stdout:6/691: chown d3/da/dd/d1b/ld7 993 1 2024-04-22T15:49:40.569 INFO:tasks.workunit.client.0.smithi016.stdout:2/621: fdatasync d2/d2c/f6b 0 2024-04-22T15:49:40.569 INFO:tasks.workunit.client.0.smithi016.stdout:3/692: fdatasync d7/d17/f80 0 2024-04-22T15:49:40.570 INFO:tasks.workunit.client.0.smithi016.stdout:8/584: symlink d2/d82/d44/db0/lb4 0 2024-04-22T15:49:40.571 INFO:tasks.workunit.client.0.smithi016.stdout:9/651: mknod d9/de/d23/d24/d25/da6/d4f/cc5 0 2024-04-22T15:49:40.573 INFO:tasks.workunit.client.0.smithi016.stdout:7/646: mknod d3/dc/d1b/d5a/d69/ce8 0 2024-04-22T15:49:40.578 INFO:tasks.workunit.client.0.smithi016.stdout:3/693: mknod d7/d17/d18/d32/d3c/d47/dc3/cd5 0 2024-04-22T15:49:40.578 INFO:tasks.workunit.client.0.smithi016.stdout:9/652: fdatasync d9/de/d23/d24/f76 0 2024-04-22T15:49:40.580 INFO:tasks.workunit.client.0.smithi016.stdout:5/620: dwrite df/d90/dad/fb2 [0,4194304] 0 2024-04-22T15:49:40.584 INFO:tasks.workunit.client.0.smithi016.stdout:8/585: symlink d2/d82/d3e/d8e/lb5 0 2024-04-22T15:49:40.585 INFO:tasks.workunit.client.0.smithi016.stdout:0/686: sync 2024-04-22T15:49:40.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:40 smithi027 ceph-mon[34922]: pgmap v32: 65 pgs: 65 active+clean; 1.2 GiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 47 MiB/s rd, 167 MiB/s wr, 460 op/s 2024-04-22T15:49:40.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:40 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:40.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:40 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:40.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:40 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:49:40.591 INFO:tasks.workunit.client.0.smithi016.stdout:9/653: creat d9/de/d23/d24/d25/d36/d87/fc6 x:0 0 0 2024-04-22T15:49:40.593 INFO:tasks.workunit.client.0.smithi016.stdout:2/622: dwrite d2/d23/d92/f71 [0,4194304] 0 2024-04-22T15:49:40.595 INFO:tasks.workunit.client.0.smithi016.stdout:8/586: truncate d2/d32/f35 2381856 0 2024-04-22T15:49:40.598 INFO:tasks.workunit.client.0.smithi016.stdout:3/694: rename d7/d17/d18/d32/d42/d68/cd1 to d7/d17/d18/cd6 0 2024-04-22T15:49:40.599 INFO:tasks.workunit.client.0.smithi016.stdout:7/647: dwrite d3/dc/d1b/d5a/da4/fc4 [0,4194304] 0 2024-04-22T15:49:40.600 INFO:tasks.workunit.client.0.smithi016.stdout:2/623: write d2/d1a/d4c/d9f/dae/fb3 [4589176,73030] 0 2024-04-22T15:49:40.645 INFO:tasks.workunit.client.0.smithi016.stdout:8/587: symlink d2/d82/d44/db0/lb6 0 2024-04-22T15:49:40.647 INFO:tasks.workunit.client.0.smithi016.stdout:7/648: symlink d3/dc/d9f/le9 0 2024-04-22T15:49:40.648 INFO:tasks.workunit.client.0.smithi016.stdout:8/588: dread d2/d82/d56/f71 [0,4194304] 0 2024-04-22T15:49:40.649 INFO:tasks.workunit.client.0.smithi016.stdout:2/624: symlink d2/d1a/d4c/d9f/dae/lf1 0 2024-04-22T15:49:40.654 INFO:tasks.workunit.client.0.smithi016.stdout:8/589: symlink d2/d82/d52/lb7 0 2024-04-22T15:49:40.659 INFO:tasks.workunit.client.0.smithi016.stdout:5/621: dwrite df/d15/d1a/d4e/f72 [0,4194304] 0 2024-04-22T15:49:40.666 INFO:tasks.workunit.client.0.smithi016.stdout:0/687: rename dc/d13/d28 to dc/d13/d14/de5 0 2024-04-22T15:49:40.668 INFO:tasks.workunit.client.0.smithi016.stdout:0/688: fsync dc/d13/d14/d26/f4f 0 2024-04-22T15:49:40.670 INFO:tasks.workunit.client.0.smithi016.stdout:0/689: write dc/d13/d14/d26/d32/f33 [4195855,32952] 0 2024-04-22T15:49:40.685 INFO:tasks.workunit.client.0.smithi016.stdout:9/654: getdents d9/de/d23/d24 0 2024-04-22T15:49:40.689 INFO:tasks.workunit.client.0.smithi016.stdout:0/690: mknod dc/d13/d14/de5/d65/d57/ce6 0 2024-04-22T15:49:40.696 INFO:tasks.workunit.client.0.smithi016.stdout:5/622: dread df/d15/d1a/d50/d77/f40 [0,4194304] 0 2024-04-22T15:49:40.698 INFO:tasks.workunit.client.0.smithi016.stdout:9/655: creat d9/de/d26/d5a/fc7 x:0 0 0 2024-04-22T15:49:40.702 INFO:tasks.workunit.client.0.smithi016.stdout:0/691: dwrite dc/d13/d14/de5/d65/d40/fac [0,4194304] 0 2024-04-22T15:49:40.706 INFO:tasks.workunit.client.0.smithi016.stdout:9/656: rmdir d9/de/d23 39 2024-04-22T15:49:40.707 INFO:tasks.workunit.client.0.smithi016.stdout:0/692: write dc/d13/d14/d26/d32/fb0 [1020761,118845] 0 2024-04-22T15:49:40.710 INFO:tasks.workunit.client.0.smithi016.stdout:5/623: dread f2 [0,4194304] 0 2024-04-22T15:49:40.712 INFO:tasks.workunit.client.0.smithi016.stdout:9/657: dread d9/d57/d91/dc1/f86 [0,4194304] 0 2024-04-22T15:49:40.719 INFO:tasks.workunit.client.0.smithi016.stdout:3/695: creat d7/d17/d18/d32/d3c/fd7 x:0 0 0 2024-04-22T15:49:40.720 INFO:tasks.workunit.client.0.smithi016.stdout:5/624: dwrite df/f1f [0,4194304] 0 2024-04-22T15:49:40.721 INFO:tasks.workunit.client.0.smithi016.stdout:9/658: unlink d9/d5d/f9a 0 2024-04-22T15:49:40.722 INFO:tasks.workunit.client.0.smithi016.stdout:3/696: write d7/fc [906143,112102] 0 2024-04-22T15:49:40.726 INFO:tasks.workunit.client.0.smithi016.stdout:0/693: mkdir dc/d13/de7 0 2024-04-22T15:49:40.726 INFO:tasks.workunit.client.0.smithi016.stdout:3/697: write d7/d17/d18/d32/d3c/fd7 [891602,45251] 0 2024-04-22T15:49:40.734 INFO:tasks.workunit.client.0.smithi016.stdout:0/694: readlink dc/d13/d70/da6/laa 0 2024-04-22T15:49:40.742 INFO:tasks.workunit.client.0.smithi016.stdout:0/695: dread dc/d13/d14/de5/d65/d30/f44 [0,4194304] 0 2024-04-22T15:49:40.743 INFO:tasks.workunit.client.0.smithi016.stdout:5/625: dwrite df/d15/f4a [8388608,4194304] 0 2024-04-22T15:49:40.758 INFO:tasks.workunit.client.0.smithi016.stdout:2/625: truncate d2/f5 3618907 0 2024-04-22T15:49:40.769 INFO:tasks.workunit.client.0.smithi016.stdout:3/698: readlink d7/d17/d18/d32/d3c/lce 0 2024-04-22T15:49:40.771 INFO:tasks.workunit.client.0.smithi016.stdout:1/775: sync 2024-04-22T15:49:40.774 INFO:tasks.workunit.client.0.smithi016.stdout:0/696: mkdir dc/d13/d70/de8 0 2024-04-22T15:49:40.776 INFO:tasks.workunit.client.0.smithi016.stdout:0/697: fdatasync dc/d13/d14/de5/d65/d1f/f53 0 2024-04-22T15:49:40.779 INFO:tasks.workunit.client.0.smithi016.stdout:1/776: dwrite d3/d4/d3a/d6b/d77/f102 [0,4194304] 0 2024-04-22T15:49:40.779 INFO:tasks.workunit.client.0.smithi016.stdout:3/699: rename d7/d75 to d7/d17/d18/d32/d42/d63/d86/dd8 0 2024-04-22T15:49:40.792 INFO:tasks.workunit.client.0.smithi016.stdout:5/626: creat df/d15/d1a/d50/d77/fbc x:0 0 0 2024-04-22T15:49:40.793 INFO:tasks.workunit.client.0.smithi016.stdout:1/777: creat d3/d4/d15/d38/d63/f109 x:0 0 0 2024-04-22T15:49:40.794 INFO:tasks.workunit.client.0.smithi016.stdout:1/778: stat d3/d4/d45/cad 0 2024-04-22T15:49:40.795 INFO:tasks.workunit.client.0.smithi016.stdout:1/779: dread - d3/f107 zero size 2024-04-22T15:49:40.795 INFO:tasks.workunit.client.0.smithi016.stdout:2/626: rmdir d2/d4d/d64/dc0 0 2024-04-22T15:49:40.798 INFO:tasks.workunit.client.0.smithi016.stdout:5/627: mknod df/d15/d1a/d4e/d58/cbd 0 2024-04-22T15:49:40.802 INFO:tasks.workunit.client.0.smithi016.stdout:5/628: fdatasync df/d15/d1a/d50/d77/d7d/d8b/d93/f97 0 2024-04-22T15:49:40.803 INFO:tasks.workunit.client.0.smithi016.stdout:1/780: rename d3/d4/d19/d20/l5f to d3/d4/d19/d20/d95/l10a 0 2024-04-22T15:49:40.804 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:40 smithi016 ceph-mon[26620]: pgmap v32: 65 pgs: 65 active+clean; 1.2 GiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 47 MiB/s rd, 167 MiB/s wr, 460 op/s 2024-04-22T15:49:40.804 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:40 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:40.804 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:40 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:40.804 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:40 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:49:40.806 INFO:tasks.workunit.client.0.smithi016.stdout:1/781: stat d3/d4/d15/d38/d63/fa9 0 2024-04-22T15:49:40.808 INFO:tasks.workunit.client.0.smithi016.stdout:0/698: dread dc/d13/d14/d26/f4f [0,4194304] 0 2024-04-22T15:49:40.809 INFO:tasks.workunit.client.0.smithi016.stdout:5/629: dread df/d15/d1a/d28/f3a [0,4194304] 0 2024-04-22T15:49:40.811 INFO:tasks.workunit.client.0.smithi016.stdout:5/630: dread - df/d15/d1a/d50/d77/fa6 zero size 2024-04-22T15:49:40.815 INFO:tasks.workunit.client.0.smithi016.stdout:1/782: creat d3/d4/d3a/d91/db3/dd2/f10b x:0 0 0 2024-04-22T15:49:40.817 INFO:tasks.workunit.client.0.smithi016.stdout:1/783: dread - d3/d4/d3a/d6b/d77/de2/ff2 zero size 2024-04-22T15:49:40.818 INFO:tasks.workunit.client.0.smithi016.stdout:0/699: fsync dc/d13/d14/de5/d65/d40/f73 0 2024-04-22T15:49:40.818 INFO:tasks.workunit.client.0.smithi016.stdout:1/784: dread - d3/d4/d15/d38/d63/d67/fd0 zero size 2024-04-22T15:49:40.820 INFO:tasks.workunit.client.0.smithi016.stdout:3/700: link d7/d1b/d21/fcd d7/d17/d18/d32/d42/d63/d71/fd9 0 2024-04-22T15:49:40.821 INFO:tasks.workunit.client.0.smithi016.stdout:0/700: stat dc/d13/d14/de5/d65/d57/dcb/fd7 0 2024-04-22T15:49:40.823 INFO:tasks.workunit.client.0.smithi016.stdout:1/785: creat d3/d4/d19/d20/d2b/de0/f10c x:0 0 0 2024-04-22T15:49:40.828 INFO:tasks.workunit.client.0.smithi016.stdout:5/631: link df/f1f df/d15/d1a/fbe 0 2024-04-22T15:49:40.830 INFO:tasks.workunit.client.0.smithi016.stdout:1/786: mkdir d3/d4/d19/d20/d2b/d10d 0 2024-04-22T15:49:40.830 INFO:tasks.workunit.client.0.smithi016.stdout:3/701: creat d7/d17/d18/d32/d42/d63/db4/fda x:0 0 0 2024-04-22T15:49:40.833 INFO:tasks.workunit.client.0.smithi016.stdout:0/701: symlink dc/d13/d14/de5/d65/le9 0 2024-04-22T15:49:40.834 INFO:tasks.workunit.client.0.smithi016.stdout:1/787: symlink d3/d4/d45/l10e 0 2024-04-22T15:49:40.834 INFO:tasks.workunit.client.0.smithi016.stdout:1/788: read - d3/d4/da/ffe zero size 2024-04-22T15:49:40.837 INFO:tasks.workunit.client.0.smithi016.stdout:0/702: creat dc/d13/d6f/fea x:0 0 0 2024-04-22T15:49:40.841 INFO:tasks.workunit.client.0.smithi016.stdout:6/692: sync 2024-04-22T15:49:40.841 INFO:tasks.workunit.client.0.smithi016.stdout:4/598: sync 2024-04-22T15:49:40.843 INFO:tasks.workunit.client.0.smithi016.stdout:4/599: fsync d7/d22/f73 0 2024-04-22T15:49:40.843 INFO:tasks.workunit.client.0.smithi016.stdout:1/789: dwrite d3/d4/d15/d38/d63/d67/fd0 [0,4194304] 0 2024-04-22T15:49:40.843 INFO:tasks.workunit.client.0.smithi016.stdout:0/703: rename dc/d13/d14/de5/d65/l4d to dc/d13/d6f/d8b/leb 0 2024-04-22T15:49:40.845 INFO:tasks.workunit.client.0.smithi016.stdout:0/704: dread - dc/d7d/fa3 zero size 2024-04-22T15:49:40.851 INFO:tasks.workunit.client.0.smithi016.stdout:1/790: dwrite d3/d4/d19/d20/f48 [0,4194304] 0 2024-04-22T15:49:40.853 INFO:tasks.workunit.client.0.smithi016.stdout:4/600: creat d7/d21/d45/d49/d9b/db6/fc4 x:0 0 0 2024-04-22T15:49:40.861 INFO:tasks.workunit.client.0.smithi016.stdout:6/693: getdents d3/da/db/d14/d4c 0 2024-04-22T15:49:40.862 INFO:tasks.workunit.client.0.smithi016.stdout:4/601: symlink d7/d4e/d2f/d50/lc5 0 2024-04-22T15:49:40.865 INFO:tasks.workunit.client.0.smithi016.stdout:0/705: rename dc/d13/d70/f72 to dc/d13/d14/de5/d65/fec 0 2024-04-22T15:49:40.866 INFO:tasks.workunit.client.0.smithi016.stdout:1/791: mknod d3/d4/d19/c10f 0 2024-04-22T15:49:40.869 INFO:tasks.workunit.client.0.smithi016.stdout:0/706: truncate dc/d13/d14/de5/d65/d40/f73 4962249 0 2024-04-22T15:49:40.871 INFO:tasks.workunit.client.0.smithi016.stdout:1/792: fsync d3/d4/d19/fda 0 2024-04-22T15:49:40.872 INFO:tasks.workunit.client.0.smithi016.stdout:4/602: creat d7/d21/fc6 x:0 0 0 2024-04-22T15:49:40.875 INFO:tasks.workunit.client.0.smithi016.stdout:1/793: fdatasync d3/d4/d15/d38/d63/d67/fd0 0 2024-04-22T15:49:40.876 INFO:tasks.workunit.client.0.smithi016.stdout:4/603: stat d7/d4e/f29 0 2024-04-22T15:49:40.876 INFO:tasks.workunit.client.0.smithi016.stdout:2/627: dread d2/d4d/f7f [0,4194304] 0 2024-04-22T15:49:40.879 INFO:tasks.workunit.client.0.smithi016.stdout:9/659: truncate d9/de/f1a 1233947 0 2024-04-22T15:49:40.880 INFO:tasks.workunit.client.0.smithi016.stdout:2/628: write d2/d23/d92/d55/fa6 [811680,47084] 0 2024-04-22T15:49:40.880 INFO:tasks.workunit.client.0.smithi016.stdout:1/794: write d3/d4/d3a/d6b/d77/f99 [627316,10762] 0 2024-04-22T15:49:40.884 INFO:tasks.workunit.client.0.smithi016.stdout:0/707: dwrite dc/d13/f52 [0,4194304] 0 2024-04-22T15:49:40.890 INFO:tasks.workunit.client.0.smithi016.stdout:1/795: creat d3/d4/d45/f110 x:0 0 0 2024-04-22T15:49:40.891 INFO:tasks.workunit.client.0.smithi016.stdout:9/660: mknod d9/de/d23/d24/d25/da6/d4f/cc8 0 2024-04-22T15:49:40.892 INFO:tasks.workunit.client.0.smithi016.stdout:0/708: write dc/d13/d14/d26/d32/fb0 [517024,1864] 0 2024-04-22T15:49:40.896 INFO:tasks.workunit.client.0.smithi016.stdout:6/694: dread d3/da/dd/f70 [0,4194304] 0 2024-04-22T15:49:40.897 INFO:tasks.workunit.client.0.smithi016.stdout:4/604: dwrite d7/d21/d45/d49/f4d [0,4194304] 0 2024-04-22T15:49:40.898 INFO:tasks.workunit.client.0.smithi016.stdout:0/709: fdatasync dc/d13/d14/d55/fe3 0 2024-04-22T15:49:40.899 INFO:tasks.workunit.client.0.smithi016.stdout:8/590: sync 2024-04-22T15:49:40.900 INFO:tasks.workunit.client.0.smithi016.stdout:7/649: sync 2024-04-22T15:49:40.904 INFO:tasks.workunit.client.0.smithi016.stdout:4/605: write d7/d24/d51/fbb [137124,96751] 0 2024-04-22T15:49:40.905 INFO:tasks.workunit.client.0.smithi016.stdout:6/695: creat d3/da/db/d44/d68/feb x:0 0 0 2024-04-22T15:49:40.909 INFO:tasks.workunit.client.0.smithi016.stdout:9/661: dread - d9/de/d23/d24/d25/f7d zero size 2024-04-22T15:49:40.909 INFO:tasks.workunit.client.0.smithi016.stdout:7/650: write d3/de6/fdb [580002,116789] 0 2024-04-22T15:49:40.909 INFO:tasks.workunit.client.0.smithi016.stdout:8/591: creat d2/d48/dad/d8f/fb8 x:0 0 0 2024-04-22T15:49:40.911 INFO:tasks.workunit.client.0.smithi016.stdout:2/629: dwrite d2/d2c/d97/d67/dbf/fee [0,4194304] 0 2024-04-22T15:49:40.915 INFO:tasks.workunit.client.0.smithi016.stdout:6/696: chown d3/da/db/d44/c54 342776951 1 2024-04-22T15:49:40.917 INFO:tasks.workunit.client.0.smithi016.stdout:3/702: write d7/d17/d18/d32/d3c/d47/f52 [402843,60256] 0 2024-04-22T15:49:40.917 INFO:tasks.workunit.client.0.smithi016.stdout:1/796: dwrite d3/d4/d3a/ff5 [0,4194304] 0 2024-04-22T15:49:40.918 INFO:tasks.workunit.client.0.smithi016.stdout:4/606: read d7/d22/f73 [2823954,378] 0 2024-04-22T15:49:40.921 INFO:tasks.workunit.client.0.smithi016.stdout:2/630: mknod d2/d51/cf2 0 2024-04-22T15:49:40.924 INFO:tasks.workunit.client.0.smithi016.stdout:8/592: chown d2/d82/d44/d66/d7e/l7f 21936 1 2024-04-22T15:49:40.928 INFO:tasks.workunit.client.0.smithi016.stdout:2/631: chown d2/d1a/d7b/dcc/ce0 3025108 1 2024-04-22T15:49:40.928 INFO:tasks.workunit.client.0.smithi016.stdout:4/607: mkdir d7/d22/dc7 0 2024-04-22T15:49:40.928 INFO:tasks.workunit.client.0.smithi016.stdout:7/651: dread d3/dc/d1b/f49 [0,4194304] 0 2024-04-22T15:49:40.929 INFO:tasks.workunit.client.0.smithi016.stdout:1/797: rename d3/d2e/cbc to d3/d4/d19/d20/d2b/c111 0 2024-04-22T15:49:40.931 INFO:tasks.workunit.client.0.smithi016.stdout:3/703: dread - d7/d1b/d21/d55/f8e zero size 2024-04-22T15:49:40.932 INFO:tasks.workunit.client.0.smithi016.stdout:8/593: mkdir d2/d48/db9 0 2024-04-22T15:49:40.932 INFO:tasks.workunit.client.0.smithi016.stdout:1/798: chown d3/d4/d3a/d9e/dbf/fc9 4577 1 2024-04-22T15:49:40.933 INFO:tasks.workunit.client.0.smithi016.stdout:9/662: creat d9/d57/da7/daa/fc9 x:0 0 0 2024-04-22T15:49:40.937 INFO:tasks.workunit.client.0.smithi016.stdout:4/608: write d7/d4e/d2f/d33/f7a [3373794,104428] 0 2024-04-22T15:49:40.938 INFO:tasks.workunit.client.0.smithi016.stdout:7/652: mknod d3/dc/d1b/d5a/d9/df/cea 0 2024-04-22T15:49:40.944 INFO:tasks.workunit.client.0.smithi016.stdout:9/663: mknod d9/d57/cca 0 2024-04-22T15:49:40.951 INFO:tasks.workunit.client.0.smithi016.stdout:7/653: dread d3/dc/d1b/d5a/fa3 [0,4194304] 0 2024-04-22T15:49:40.955 INFO:tasks.workunit.client.0.smithi016.stdout:9/664: dread d9/fa [0,4194304] 0 2024-04-22T15:49:40.957 INFO:tasks.workunit.client.0.smithi016.stdout:8/594: link d2/d82/d52/f54 d2/d82/d44/fba 0 2024-04-22T15:49:40.958 INFO:tasks.workunit.client.0.smithi016.stdout:8/595: write d2/d82/d44/d66/d79/f8b [475163,95688] 0 2024-04-22T15:49:40.962 INFO:tasks.workunit.client.0.smithi016.stdout:5/632: link df/d15/d1a/l83 df/d15/d1a/d50/d77/d7d/d8b/d93/dbb/lbf 0 2024-04-22T15:49:40.965 INFO:tasks.workunit.client.0.smithi016.stdout:2/632: link d2/l21 d2/d1a/d4c/d9f/dae/lf3 0 2024-04-22T15:49:40.969 INFO:tasks.workunit.client.0.smithi016.stdout:0/710: truncate dc/d13/d5d/f6c 4193272 0 2024-04-22T15:49:40.971 INFO:tasks.workunit.client.0.smithi016.stdout:0/711: write dc/d13/d14/d55/fbf [952149,96781] 0 2024-04-22T15:49:41.014 INFO:tasks.workunit.client.0.smithi016.stdout:6/697: write d3/da/d12/d13/d97/de0/fe3 [815240,64083] 0 2024-04-22T15:49:41.015 INFO:tasks.workunit.client.0.smithi016.stdout:3/704: getdents d7/d17/d18/d27 0 2024-04-22T15:49:41.016 INFO:tasks.workunit.client.0.smithi016.stdout:9/665: dwrite d9/d5d/fc3 [0,4194304] 0 2024-04-22T15:49:41.016 INFO:tasks.workunit.client.0.smithi016.stdout:1/799: getdents d3/d2e/d80/d82 0 2024-04-22T15:49:41.022 INFO:tasks.workunit.client.0.smithi016.stdout:6/698: dread - d3/da/dd/fb2 zero size 2024-04-22T15:49:41.024 INFO:tasks.workunit.client.0.smithi016.stdout:4/609: link d7/c12 d7/d21/d42/cc8 0 2024-04-22T15:49:41.024 INFO:tasks.workunit.client.0.smithi016.stdout:7/654: creat d3/dc/d1b/feb x:0 0 0 2024-04-22T15:49:41.024 INFO:tasks.workunit.client.0.smithi016.stdout:2/633: creat d2/d2c/de7/ff4 x:0 0 0 2024-04-22T15:49:41.025 INFO:tasks.workunit.client.0.smithi016.stdout:1/800: unlink d3/d4/d19/d20/d2b/l61 0 2024-04-22T15:49:41.026 INFO:tasks.workunit.client.0.smithi016.stdout:2/634: chown d2/de/d2a/l38 79196 1 2024-04-22T15:49:41.027 INFO:tasks.workunit.client.0.smithi016.stdout:1/801: readlink d3/d4/l6c 0 2024-04-22T15:49:41.028 INFO:tasks.workunit.client.0.smithi016.stdout:9/666: unlink d9/de/d23/d24/d25/da6/lb4 0 2024-04-22T15:49:41.029 INFO:tasks.workunit.client.0.smithi016.stdout:9/667: write d9/de/d23/d24/d25/f89 [925164,72790] 0 2024-04-22T15:49:41.030 INFO:tasks.workunit.client.0.smithi016.stdout:9/668: dread - d9/d57/d91/dc1/da2/fba zero size 2024-04-22T15:49:41.031 INFO:tasks.workunit.client.0.smithi016.stdout:3/705: unlink d7/d17/d18/d32/d3c/f66 0 2024-04-22T15:49:41.032 INFO:tasks.workunit.client.0.smithi016.stdout:3/706: truncate d7/d1b/fbc 85475 0 2024-04-22T15:49:41.033 INFO:tasks.workunit.client.0.smithi016.stdout:7/655: write d3/dc/d1b/d31/f48 [1045321,75601] 0 2024-04-22T15:49:41.033 INFO:tasks.workunit.client.0.smithi016.stdout:4/610: read d7/d4e/d2f/d6f/f84 [4320915,15557] 0 2024-04-22T15:49:41.034 INFO:tasks.workunit.client.0.smithi016.stdout:3/707: write d7/d17/d18/d27/fa3 [2022996,72275] 0 2024-04-22T15:49:41.040 INFO:tasks.workunit.client.0.smithi016.stdout:4/611: creat d7/d22/fc9 x:0 0 0 2024-04-22T15:49:41.041 INFO:tasks.workunit.client.0.smithi016.stdout:5/633: truncate df/d15/d1a/d4e/f69 82813 0 2024-04-22T15:49:41.044 INFO:tasks.workunit.client.0.smithi016.stdout:0/712: truncate dc/d13/d14/de5/d65/f63 1927990 0 2024-04-22T15:49:41.045 INFO:tasks.workunit.client.0.smithi016.stdout:1/802: write d3/d4/d15/d38/fa8 [586406,75047] 0 2024-04-22T15:49:41.049 INFO:tasks.workunit.client.0.smithi016.stdout:2/635: creat d2/d4d/ff5 x:0 0 0 2024-04-22T15:49:41.052 INFO:tasks.workunit.client.0.smithi016.stdout:2/636: dread d2/d2c/d97/d67/dbf/f95 [4194304,4194304] 0 2024-04-22T15:49:41.052 INFO:tasks.workunit.client.0.smithi016.stdout:3/708: unlink d7/d17/c43 0 2024-04-22T15:49:41.053 INFO:tasks.workunit.client.0.smithi016.stdout:9/669: truncate d9/de/d23/d24/fa4 470698 0 2024-04-22T15:49:41.054 INFO:tasks.workunit.client.0.smithi016.stdout:5/634: unlink df/d15/d1a/d50/d77/fbc 0 2024-04-22T15:49:41.055 INFO:tasks.workunit.client.0.smithi016.stdout:5/635: chown df/d15/d1a/d50/d77/d7d/d8b/cb9 12 1 2024-04-22T15:49:41.058 INFO:tasks.workunit.client.0.smithi016.stdout:7/656: creat d3/de6/fec x:0 0 0 2024-04-22T15:49:41.059 INFO:tasks.workunit.client.0.smithi016.stdout:1/803: unlink d3/d4/d3a/d9e/dbf/fc9 0 2024-04-22T15:49:41.064 INFO:tasks.workunit.client.0.smithi016.stdout:4/612: dwrite d7/d24/d41/f57 [0,4194304] 0 2024-04-22T15:49:41.068 INFO:tasks.workunit.client.0.smithi016.stdout:4/613: read d7/d21/d42/d59/d88/f9c [75327,105299] 0 2024-04-22T15:49:41.070 INFO:tasks.workunit.client.0.smithi016.stdout:3/709: rename d7/d17/d18/d32/d42/d68 to d7/d1b/d21/d55/ddb 0 2024-04-22T15:49:41.083 INFO:tasks.workunit.client.0.smithi016.stdout:2/637: dwrite d2/d23/d92/f66 [0,4194304] 0 2024-04-22T15:49:41.084 INFO:tasks.workunit.client.0.smithi016.stdout:7/657: write d3/dc/d1b/d5a/d69/f90 [173971,96884] 0 2024-04-22T15:49:41.087 INFO:tasks.workunit.client.0.smithi016.stdout:3/710: symlink d7/d17/d18/d32/d42/d63/d8f/ldc 0 2024-04-22T15:49:41.087 INFO:tasks.workunit.client.0.smithi016.stdout:1/804: creat d3/d4/d3a/d96/f112 x:0 0 0 2024-04-22T15:49:41.087 INFO:tasks.workunit.client.0.smithi016.stdout:3/711: write d7/d17/d18/d32/d42/d63/db4/fda [275160,87455] 0 2024-04-22T15:49:41.088 INFO:tasks.workunit.client.0.smithi016.stdout:0/713: link dc/d13/d14/de5/d65/d40/d4a/d59/d78/c81 dc/d13/d14/de5/dd8/ced 0 2024-04-22T15:49:41.089 INFO:tasks.workunit.client.0.smithi016.stdout:1/805: write d3/f8 [2790370,81107] 0 2024-04-22T15:49:41.090 INFO:tasks.workunit.client.0.smithi016.stdout:1/806: chown d3/d4/l13 12 1 2024-04-22T15:49:41.092 INFO:tasks.workunit.client.0.smithi016.stdout:0/714: dread - dc/d13/d14/de5/d65/d30/f84 zero size 2024-04-22T15:49:41.094 INFO:tasks.workunit.client.0.smithi016.stdout:2/638: truncate d2/d2c/d97/f31 5912040 0 2024-04-22T15:49:41.096 INFO:tasks.workunit.client.0.smithi016.stdout:5/636: link df/d15/d1a/d28/f74 df/d15/d1a/d50/d77/d7d/fc0 0 2024-04-22T15:49:41.099 INFO:tasks.workunit.client.0.smithi016.stdout:3/712: creat d7/d1b/d21/d55/dac/fdd x:0 0 0 2024-04-22T15:49:41.099 INFO:tasks.workunit.client.0.smithi016.stdout:7/658: dread d3/dc/d1b/d5a/f46 [4194304,4194304] 0 2024-04-22T15:49:41.100 INFO:tasks.workunit.client.0.smithi016.stdout:9/670: rename d9/d57/d91/dc1/fc0 to d9/de/fcb 0 2024-04-22T15:49:41.102 INFO:tasks.workunit.client.0.smithi016.stdout:1/807: dwrite d3/d4/d15/d38/d63/d67/fd0 [0,4194304] 0 2024-04-22T15:49:41.104 INFO:tasks.workunit.client.0.smithi016.stdout:0/715: dwrite dc/d13/d99/fc4 [0,4194304] 0 2024-04-22T15:49:41.109 INFO:tasks.workunit.client.0.smithi016.stdout:7/659: dread d3/dc/f65 [0,4194304] 0 2024-04-22T15:49:41.109 INFO:tasks.workunit.client.0.smithi016.stdout:0/716: fdatasync dc/d13/d14/de5/d65/d40/d4a/d61/fe4 0 2024-04-22T15:49:41.109 INFO:tasks.workunit.client.0.smithi016.stdout:9/671: fdatasync d9/de/d26/f2a 0 2024-04-22T15:49:41.109 INFO:tasks.workunit.client.0.smithi016.stdout:2/639: getdents d2/d4d/dc4 0 2024-04-22T15:49:41.110 INFO:tasks.workunit.client.0.smithi016.stdout:5/637: dwrite df/d5b/d70/fa4 [0,4194304] 0 2024-04-22T15:49:41.111 INFO:tasks.workunit.client.0.smithi016.stdout:9/672: stat d9/de/f28 0 2024-04-22T15:49:41.111 INFO:tasks.workunit.client.0.smithi016.stdout:3/713: creat d7/d17/d18/d32/d3c/d47/dc3/fde x:0 0 0 2024-04-22T15:49:41.117 INFO:tasks.workunit.client.0.smithi016.stdout:1/808: symlink d3/d4/d19/d20/d95/l113 0 2024-04-22T15:49:41.119 INFO:tasks.workunit.client.0.smithi016.stdout:1/809: chown d3/dea/cec 973 1 2024-04-22T15:49:41.120 INFO:tasks.workunit.client.0.smithi016.stdout:4/614: truncate d7/f16 1217340 0 2024-04-22T15:49:41.122 INFO:tasks.workunit.client.0.smithi016.stdout:2/640: rmdir d2/d1a/d7b/dcc/de5 39 2024-04-22T15:49:41.124 INFO:tasks.workunit.client.0.smithi016.stdout:7/660: read d3/dc/f1e [8614,69964] 0 2024-04-22T15:49:41.128 INFO:tasks.workunit.client.0.smithi016.stdout:3/714: creat d7/d17/d18/d32/d42/d63/d86/fdf x:0 0 0 2024-04-22T15:49:41.131 INFO:tasks.workunit.client.0.smithi016.stdout:1/810: rename d3/d4/d19/d20/d2b/c52 to d3/d28/c114 0 2024-04-22T15:49:41.131 INFO:tasks.workunit.client.0.smithi016.stdout:7/661: dwrite d3/dc/d1b/d31/f48 [0,4194304] 0 2024-04-22T15:49:41.133 INFO:tasks.workunit.client.0.smithi016.stdout:4/615: mkdir d7/d3c/d96/dca 0 2024-04-22T15:49:41.134 INFO:tasks.workunit.client.0.smithi016.stdout:2/641: creat d2/d23/ff6 x:0 0 0 2024-04-22T15:49:41.135 INFO:tasks.workunit.client.0.smithi016.stdout:5/638: mkdir df/d5b/db3/dc1 0 2024-04-22T15:49:41.141 INFO:tasks.workunit.client.0.smithi016.stdout:3/715: chown d7/d17/d18/d32/d42/d63/d8f/cc9 23694979 1 2024-04-22T15:49:41.141 INFO:tasks.workunit.client.0.smithi016.stdout:5/639: dwrite df/d15/d1a/d50/fac [0,4194304] 0 2024-04-22T15:49:41.145 INFO:tasks.workunit.client.0.smithi016.stdout:1/811: creat d3/d4/d3a/d6b/dc3/f115 x:0 0 0 2024-04-22T15:49:41.145 INFO:tasks.workunit.client.0.smithi016.stdout:7/662: mknod d3/dc/d1b/d5a/d69/ced 0 2024-04-22T15:49:41.147 INFO:tasks.workunit.client.0.smithi016.stdout:9/673: write d9/fa [3301078,118085] 0 2024-04-22T15:49:41.153 INFO:tasks.workunit.client.0.smithi016.stdout:3/716: fsync d7/d17/d18/d32/d3c/faa 0 2024-04-22T15:49:41.156 INFO:tasks.workunit.client.0.smithi016.stdout:7/663: mknod d3/dc/d33/d84/cee 0 2024-04-22T15:49:41.158 INFO:tasks.workunit.client.0.smithi016.stdout:1/812: mkdir d3/d4/d3a/d6b/d77/de2/d116 0 2024-04-22T15:49:41.159 INFO:tasks.workunit.client.0.smithi016.stdout:0/717: link dc/d13/d22/l2b dc/d13/d14/de5/d65/lee 0 2024-04-22T15:49:41.163 INFO:tasks.workunit.client.0.smithi016.stdout:3/717: unlink d7/d17/d18/c4d 0 2024-04-22T15:49:41.164 INFO:tasks.workunit.client.0.smithi016.stdout:7/664: mkdir d3/dc/d1b/d5a/d69/def 0 2024-04-22T15:49:41.165 INFO:tasks.workunit.client.0.smithi016.stdout:1/813: dwrite d3/d4/d15/d38/d63/f75 [4194304,4194304] 0 2024-04-22T15:49:41.165 INFO:tasks.workunit.client.0.smithi016.stdout:3/718: read d7/d17/d18/d32/d3c/d47/f33 [356961,122060] 0 2024-04-22T15:49:41.167 INFO:tasks.workunit.client.0.smithi016.stdout:0/718: read dc/d13/f31 [1074801,86669] 0 2024-04-22T15:49:41.169 INFO:tasks.workunit.client.0.smithi016.stdout:0/719: stat dc/d13/d70/da6/laa 0 2024-04-22T15:49:41.170 INFO:tasks.workunit.client.0.smithi016.stdout:0/720: stat dc/d13/d70/l79 0 2024-04-22T15:49:41.171 INFO:tasks.workunit.client.0.smithi016.stdout:7/665: fdatasync d3/de6/fad 0 2024-04-22T15:49:41.171 INFO:tasks.workunit.client.0.smithi016.stdout:2/642: write d2/f2b [9383151,34413] 0 2024-04-22T15:49:41.172 INFO:tasks.workunit.client.0.smithi016.stdout:3/719: symlink d7/d1b/d21/d55/dac/le0 0 2024-04-22T15:49:41.175 INFO:tasks.workunit.client.0.smithi016.stdout:9/674: rename d9/de/d23/d24/c46 to d9/de/ccc 0 2024-04-22T15:49:41.176 INFO:tasks.workunit.client.0.smithi016.stdout:3/720: fsync d7/d17/d18/d32/fc7 0 2024-04-22T15:49:41.176 INFO:tasks.workunit.client.0.smithi016.stdout:0/721: chown dc/d13/d70/da6/fc3 111146351 1 2024-04-22T15:49:41.177 INFO:tasks.workunit.client.0.smithi016.stdout:6/699: sync 2024-04-22T15:49:41.177 INFO:tasks.workunit.client.0.smithi016.stdout:8/596: sync 2024-04-22T15:49:41.179 INFO:tasks.workunit.client.0.smithi016.stdout:1/814: dwrite d3/d4/da/d7e/f106 [0,4194304] 0 2024-04-22T15:49:41.179 INFO:tasks.workunit.client.0.smithi016.stdout:7/666: mkdir d3/dc/d1b/d31/d88/df0 0 2024-04-22T15:49:41.181 INFO:tasks.workunit.client.0.smithi016.stdout:0/722: write dc/f7e [1164233,16734] 0 2024-04-22T15:49:41.181 INFO:tasks.workunit.client.0.smithi016.stdout:2/643: read - d2/d1a/d7b/dcc/de5/fe9 zero size 2024-04-22T15:49:41.182 INFO:tasks.workunit.client.0.smithi016.stdout:9/675: truncate d9/de/fcb 1859290 0 2024-04-22T15:49:41.183 INFO:tasks.workunit.client.0.smithi016.stdout:4/616: write d7/d3c/f44 [874174,82780] 0 2024-04-22T15:49:41.187 INFO:tasks.workunit.client.0.smithi016.stdout:6/700: dread - d3/da/f72 zero size 2024-04-22T15:49:41.187 INFO:tasks.workunit.client.0.smithi016.stdout:2/644: fsync d2/d1a/d4c/f7a 0 2024-04-22T15:49:41.188 INFO:tasks.workunit.client.0.smithi016.stdout:1/815: stat d3/d4/d15/d38/d63/d67/f79 0 2024-04-22T15:49:41.188 INFO:tasks.workunit.client.0.smithi016.stdout:5/640: dwrite df/d15/d1a/fbe [4194304,4194304] 0 2024-04-22T15:49:41.189 INFO:tasks.workunit.client.0.smithi016.stdout:9/676: fdatasync d9/de/d23/f35 0 2024-04-22T15:49:41.189 INFO:tasks.workunit.client.0.smithi016.stdout:4/617: fdatasync d7/d4e/d2f/faa 0 2024-04-22T15:49:41.191 INFO:tasks.workunit.client.0.smithi016.stdout:3/721: creat d7/d1b/d21/d55/ddb/fe1 x:0 0 0 2024-04-22T15:49:41.191 INFO:tasks.workunit.client.0.smithi016.stdout:6/701: readlink d3/da/d12/d13/l5e 0 2024-04-22T15:49:41.191 INFO:tasks.workunit.client.0.smithi016.stdout:4/618: read - d7/d21/fc6 zero size 2024-04-22T15:49:41.199 INFO:tasks.workunit.client.0.smithi016.stdout:8/597: truncate d2/d82/f38 3231754 0 2024-04-22T15:49:41.199 INFO:tasks.workunit.client.0.smithi016.stdout:1/816: fsync d3/d2e/d80/d82/fae 0 2024-04-22T15:49:41.201 INFO:tasks.workunit.client.0.smithi016.stdout:1/817: write d3/f8 [2705771,112604] 0 2024-04-22T15:49:41.203 INFO:tasks.workunit.client.0.smithi016.stdout:5/641: creat df/d15/d1a/d50/d77/d94/fc2 x:0 0 0 2024-04-22T15:49:41.204 INFO:tasks.workunit.client.0.smithi016.stdout:7/667: mknod d3/dc/d1b/d31/d88/de1/cf1 0 2024-04-22T15:49:41.204 INFO:tasks.workunit.client.0.smithi016.stdout:8/598: dread d2/d48/f55 [0,4194304] 0 2024-04-22T15:49:41.205 INFO:tasks.workunit.client.0.smithi016.stdout:4/619: fsync d7/d4e/f29 0 2024-04-22T15:49:41.207 INFO:tasks.workunit.client.0.smithi016.stdout:4/620: readlink d7/l1b 0 2024-04-22T15:49:41.208 INFO:tasks.workunit.client.0.smithi016.stdout:7/668: truncate d3/dc/d1b/d5a/d9/df/f70 586546 0 2024-04-22T15:49:41.212 INFO:tasks.workunit.client.0.smithi016.stdout:6/702: truncate d3/da/d12/d13/f76 1538198 0 2024-04-22T15:49:41.214 INFO:tasks.workunit.client.0.smithi016.stdout:2/645: dread d2/d1a/d7b/dcc/f7e [0,4194304] 0 2024-04-22T15:49:41.216 INFO:tasks.workunit.client.0.smithi016.stdout:9/677: readlink d9/de/d23/d24/d25/d36/l51 0 2024-04-22T15:49:41.218 INFO:tasks.workunit.client.0.smithi016.stdout:5/642: creat df/d5b/d70/fc3 x:0 0 0 2024-04-22T15:49:41.218 INFO:tasks.workunit.client.0.smithi016.stdout:9/678: fsync d9/de/d23/d24/d25/f89 0 2024-04-22T15:49:41.220 INFO:tasks.workunit.client.0.smithi016.stdout:3/722: creat d7/d17/d18/d32/d42/d63/d71/fe2 x:0 0 0 2024-04-22T15:49:41.221 INFO:tasks.workunit.client.0.smithi016.stdout:0/723: truncate dc/f1e 3153918 0 2024-04-22T15:49:41.222 INFO:tasks.workunit.client.0.smithi016.stdout:2/646: dwrite d2/d1a/d4c/fb5 [0,4194304] 0 2024-04-22T15:49:41.224 INFO:tasks.workunit.client.0.smithi016.stdout:0/724: fdatasync dc/d13/d14/d26/fa9 0 2024-04-22T15:49:41.225 INFO:tasks.workunit.client.0.smithi016.stdout:6/703: fdatasync d3/da/f72 0 2024-04-22T15:49:41.226 INFO:tasks.workunit.client.0.smithi016.stdout:6/704: write d3/da/dd/d1b/d57/fbc [706788,9114] 0 2024-04-22T15:49:41.229 INFO:tasks.workunit.client.0.smithi016.stdout:5/643: rename df/f27 to df/d5b/fc4 0 2024-04-22T15:49:41.231 INFO:tasks.workunit.client.0.smithi016.stdout:4/621: creat d7/d4e/d81/fcb x:0 0 0 2024-04-22T15:49:41.238 INFO:tasks.workunit.client.0.smithi016.stdout:5/644: dwrite df/d15/d1a/d50/d77/f61 [0,4194304] 0 2024-04-22T15:49:41.239 INFO:tasks.workunit.client.0.smithi016.stdout:3/723: creat d7/d17/d18/d32/d3c/d47/fe3 x:0 0 0 2024-04-22T15:49:41.239 INFO:tasks.workunit.client.0.smithi016.stdout:0/725: rmdir dc 39 2024-04-22T15:49:41.239 INFO:tasks.workunit.client.0.smithi016.stdout:2/647: creat d2/d2c/d97/d67/dbf/ff7 x:0 0 0 2024-04-22T15:49:41.247 INFO:tasks.workunit.client.0.smithi016.stdout:8/599: truncate d2/d48/dad/d6a/f74 1796497 0 2024-04-22T15:49:41.248 INFO:tasks.workunit.client.0.smithi016.stdout:8/600: stat d2/d82/d44/d66/d1c 0 2024-04-22T15:49:41.251 INFO:tasks.workunit.client.0.smithi016.stdout:9/679: rename d9/d20/cbe to d9/de/d23/d24/d25/da6/d4f/ccd 0 2024-04-22T15:49:41.253 INFO:tasks.workunit.client.0.smithi016.stdout:4/622: fdatasync d7/d21/d45/d49/f56 0 2024-04-22T15:49:41.254 INFO:tasks.workunit.client.0.smithi016.stdout:4/623: stat d7/d24/d51/la3 0 2024-04-22T15:49:41.256 INFO:tasks.workunit.client.0.smithi016.stdout:6/705: mknod d3/da/db/cec 0 2024-04-22T15:49:41.262 INFO:tasks.workunit.client.0.smithi016.stdout:0/726: mkdir dc/d13/d14/de5/def 0 2024-04-22T15:49:41.264 INFO:tasks.workunit.client.0.smithi016.stdout:2/648: dread d2/d1a/d4c/f3f [0,4194304] 0 2024-04-22T15:49:41.265 INFO:tasks.workunit.client.0.smithi016.stdout:4/624: creat d7/d21/d42/d59/d88/d9d/dad/dac/fcc x:0 0 0 2024-04-22T15:49:41.269 INFO:tasks.workunit.client.0.smithi016.stdout:9/680: symlink d9/d5d/db8/lce 0 2024-04-22T15:49:41.270 INFO:tasks.workunit.client.0.smithi016.stdout:0/727: dwrite dc/d13/d99/fc4 [0,4194304] 0 2024-04-22T15:49:41.274 INFO:tasks.workunit.client.0.smithi016.stdout:2/649: unlink d2/d1a/d7b/f8d 0 2024-04-22T15:49:41.274 INFO:tasks.workunit.client.0.smithi016.stdout:4/625: creat d7/d22/fcd x:0 0 0 2024-04-22T15:49:41.279 INFO:tasks.workunit.client.0.smithi016.stdout:3/724: rename d7/f74 to d7/d17/d18/d32/fe4 0 2024-04-22T15:49:41.281 INFO:tasks.workunit.client.0.smithi016.stdout:5/645: write df/d15/d1a/d50/d77/d7a/f65 [1251270,6021] 0 2024-04-22T15:49:41.283 INFO:tasks.workunit.client.0.smithi016.stdout:0/728: creat dc/d13/d14/de5/d65/d40/d4a/d59/ff0 x:0 0 0 2024-04-22T15:49:41.291 INFO:tasks.workunit.client.0.smithi016.stdout:5/646: creat df/d5b/d5d/fc5 x:0 0 0 2024-04-22T15:49:41.291 INFO:tasks.workunit.client.0.smithi016.stdout:8/601: dwrite d2/d10/f39 [0,4194304] 0 2024-04-22T15:49:41.292 INFO:tasks.workunit.client.0.smithi016.stdout:2/650: getdents d2/d2c/d97/d34/d60 0 2024-04-22T15:49:41.293 INFO:tasks.workunit.client.0.smithi016.stdout:4/626: mknod d7/d21/d42/cce 0 2024-04-22T15:49:41.295 INFO:tasks.workunit.client.0.smithi016.stdout:4/627: write d7/d24/d41/f5d [3872657,39660] 0 2024-04-22T15:49:41.297 INFO:tasks.workunit.client.0.smithi016.stdout:0/729: mknod dc/d13/d14/de5/d65/d1f/dd4/cf1 0 2024-04-22T15:49:41.297 INFO:tasks.workunit.client.0.smithi016.stdout:5/647: dread df/d15/d1a/d4e/f72 [0,4194304] 0 2024-04-22T15:49:41.298 INFO:tasks.workunit.client.0.smithi016.stdout:4/628: fsync d7/d21/d42/d59/d88/d9d/dad/dac/fcc 0 2024-04-22T15:49:41.298 INFO:tasks.workunit.client.0.smithi016.stdout:8/602: symlink d2/d77/lbb 0 2024-04-22T15:49:41.300 INFO:tasks.workunit.client.0.smithi016.stdout:4/629: read d7/d21/d45/d49/f56 [2722230,34541] 0 2024-04-22T15:49:41.302 INFO:tasks.workunit.client.0.smithi016.stdout:2/651: creat d2/d1a/d53/ff8 x:0 0 0 2024-04-22T15:49:41.303 INFO:tasks.workunit.client.0.smithi016.stdout:9/681: getdents d9/de/d26/d5a 0 2024-04-22T15:49:41.308 INFO:tasks.workunit.client.0.smithi016.stdout:8/603: dwrite d2/d82/d44/d66/d79/f8b [0,4194304] 0 2024-04-22T15:49:41.310 INFO:tasks.workunit.client.0.smithi016.stdout:0/730: rename dc/d13/d14/d55/fe3 to dc/d13/d14/de5/d65/d40/d4a/d61/ff2 0 2024-04-22T15:49:41.312 INFO:tasks.workunit.client.0.smithi016.stdout:2/652: dwrite d2/d51/f62 [0,4194304] 0 2024-04-22T15:49:41.315 INFO:tasks.workunit.client.0.smithi016.stdout:3/725: getdents d7/d17/d18/d32/d42/d63/d86 0 2024-04-22T15:49:41.315 INFO:tasks.workunit.client.0.smithi016.stdout:4/630: write d7/d21/d42/f4c [4525066,59067] 0 2024-04-22T15:49:41.317 INFO:tasks.workunit.client.0.smithi016.stdout:4/631: chown d7/f1e 307428175 1 2024-04-22T15:49:41.318 INFO:tasks.workunit.client.0.smithi016.stdout:5/648: link df/f1f df/d15/d1a/d50/d77/d7d/fc6 0 2024-04-22T15:49:41.319 INFO:tasks.workunit.client.0.smithi016.stdout:3/726: write d7/d17/d18/d32/d3c/d47/fc4 [777488,84008] 0 2024-04-22T15:49:41.320 INFO:tasks.workunit.client.0.smithi016.stdout:4/632: dread d7/d4e/d2f/d6f/fae [0,4194304] 0 2024-04-22T15:49:41.321 INFO:tasks.workunit.client.0.smithi016.stdout:8/604: mknod d2/d82/d44/d66/d1c/cbc 0 2024-04-22T15:49:41.321 INFO:tasks.workunit.client.0.smithi016.stdout:3/727: chown d7/d17/d18/f23 663733 1 2024-04-22T15:49:41.322 INFO:tasks.workunit.client.0.smithi016.stdout:8/605: read d2/d82/d56/f71 [61704,119060] 0 2024-04-22T15:49:41.323 INFO:tasks.workunit.client.0.smithi016.stdout:0/731: fsync dc/d13/f31 0 2024-04-22T15:49:41.323 INFO:tasks.workunit.client.0.smithi016.stdout:3/728: write d7/d1b/d21/d55/fa6 [170698,125735] 0 2024-04-22T15:49:41.329 INFO:tasks.workunit.client.0.smithi016.stdout:4/633: creat d7/d24/d51/fcf x:0 0 0 2024-04-22T15:49:41.330 INFO:tasks.workunit.client.0.smithi016.stdout:8/606: dread d2/d82/d56/f64 [0,4194304] 0 2024-04-22T15:49:41.332 INFO:tasks.workunit.client.0.smithi016.stdout:9/682: creat d9/de/d23/fcf x:0 0 0 2024-04-22T15:49:41.336 INFO:tasks.workunit.client.0.smithi016.stdout:0/732: rename dc/d13/d14/de5/f6a to dc/d13/d14/de5/d65/d30/ff3 0 2024-04-22T15:49:41.339 INFO:tasks.workunit.client.0.smithi016.stdout:5/649: read df/d5b/d5d/d5f/f6f [110066,60997] 0 2024-04-22T15:49:41.341 INFO:tasks.workunit.client.0.smithi016.stdout:4/634: mknod d7/d21/d42/d59/d88/d9d/dad/dac/cd0 0 2024-04-22T15:49:41.342 INFO:tasks.workunit.client.0.smithi016.stdout:8/607: creat d2/d82/d44/d66/d1c/d65/fbd x:0 0 0 2024-04-22T15:49:41.344 INFO:tasks.workunit.client.0.smithi016.stdout:3/729: truncate d7/f5a 2886177 0 2024-04-22T15:49:41.345 INFO:tasks.workunit.client.0.smithi016.stdout:3/730: fdatasync d7/f15 0 2024-04-22T15:49:41.346 INFO:tasks.workunit.client.0.smithi016.stdout:0/733: rmdir dc/d13/d14/de5/d65/d40/d4a/d59 39 2024-04-22T15:49:41.347 INFO:tasks.workunit.client.0.smithi016.stdout:8/608: write d2/d10/d43/da9/fac [161335,45405] 0 2024-04-22T15:49:41.347 INFO:tasks.workunit.client.0.smithi016.stdout:2/653: dwrite d2/d2c/d97/d67/f75 [0,4194304] 0 2024-04-22T15:49:41.350 INFO:tasks.workunit.client.0.smithi016.stdout:9/683: creat d9/d20/dac/dae/fd0 x:0 0 0 2024-04-22T15:49:41.352 INFO:tasks.workunit.client.0.smithi016.stdout:9/684: dread - d9/de/d23/f9c zero size 2024-04-22T15:49:41.353 INFO:tasks.workunit.client.0.smithi016.stdout:9/685: write d9/f1c [3234161,52660] 0 2024-04-22T15:49:41.355 INFO:tasks.workunit.client.0.smithi016.stdout:8/609: dread d2/d82/d44/d66/d1c/f8c [0,4194304] 0 2024-04-22T15:49:41.357 INFO:tasks.workunit.client.0.smithi016.stdout:5/650: mknod df/d15/d1a/d50/d77/cc7 0 2024-04-22T15:49:41.367 INFO:tasks.workunit.client.0.smithi016.stdout:4/635: rename d7/d4e/d2f/d50/f61 to d7/d21/d42/d59/d88/d9d/dad/dac/fd1 0 2024-04-22T15:49:41.367 INFO:tasks.workunit.client.0.smithi016.stdout:9/686: fsync d9/de/d23/d24/d25/f2c 0 2024-04-22T15:49:41.367 INFO:tasks.workunit.client.0.smithi016.stdout:3/731: dwrite d7/d17/d18/d32/fe4 [4194304,4194304] 0 2024-04-22T15:49:41.368 INFO:tasks.workunit.client.0.smithi016.stdout:8/610: fsync d2/d82/f58 0 2024-04-22T15:49:41.378 INFO:tasks.workunit.client.0.smithi016.stdout:8/611: mkdir d2/d10/d43/dbe 0 2024-04-22T15:49:41.379 INFO:tasks.workunit.client.0.smithi016.stdout:2/654: dwrite d2/de/f40 [0,4194304] 0 2024-04-22T15:49:41.380 INFO:tasks.workunit.client.0.smithi016.stdout:5/651: dread df/d15/d1a/d50/d77/f39 [4194304,4194304] 0 2024-04-22T15:49:41.381 INFO:tasks.workunit.client.0.smithi016.stdout:3/732: mknod d7/d17/ce5 0 2024-04-22T15:49:41.381 INFO:tasks.workunit.client.0.smithi016.stdout:8/612: write d2/d82/d3e/f42 [6262697,29067] 0 2024-04-22T15:49:41.388 INFO:tasks.workunit.client.0.smithi016.stdout:5/652: mknod df/d15/d1a/d4e/d58/d87/cc8 0 2024-04-22T15:49:41.390 INFO:tasks.workunit.client.0.smithi016.stdout:8/613: dwrite d2/f24 [0,4194304] 0 2024-04-22T15:49:41.392 INFO:tasks.workunit.client.0.smithi016.stdout:0/734: truncate dc/d13/d14/de5/d65/d1f/d48/f64 1679880 0 2024-04-22T15:49:41.395 INFO:tasks.workunit.client.0.smithi016.stdout:5/653: mknod df/d15/d1a/d50/d77/cc9 0 2024-04-22T15:49:41.396 INFO:tasks.workunit.client.0.smithi016.stdout:4/636: dread d7/d21/d45/f6d [0,4194304] 0 2024-04-22T15:49:41.402 INFO:tasks.workunit.client.0.smithi016.stdout:8/614: dwrite d2/d82/d44/d66/fa7 [0,4194304] 0 2024-04-22T15:49:41.403 INFO:tasks.workunit.client.0.smithi016.stdout:3/733: dread d7/d17/d18/d32/d3c/d47/f48 [0,4194304] 0 2024-04-22T15:49:41.405 INFO:tasks.workunit.client.0.smithi016.stdout:4/637: mknod d7/d21/d45/d49/d9b/db6/cd2 0 2024-04-22T15:49:41.405 INFO:tasks.workunit.client.0.smithi016.stdout:5/654: fsync df/d15/d1a/d4e/f72 0 2024-04-22T15:49:41.408 INFO:tasks.workunit.client.0.smithi016.stdout:3/734: mknod d7/d17/d18/ce6 0 2024-04-22T15:49:41.408 INFO:tasks.workunit.client.0.smithi016.stdout:3/735: stat d7/d17/d18/d27/f3d 0 2024-04-22T15:49:41.409 INFO:tasks.workunit.client.0.smithi016.stdout:8/615: creat d2/d48/dad/d6a/fbf x:0 0 0 2024-04-22T15:49:41.415 INFO:tasks.workunit.client.0.smithi016.stdout:5/655: creat df/d22/fca x:0 0 0 2024-04-22T15:49:41.415 INFO:tasks.workunit.client.0.smithi016.stdout:0/735: getdents dc/d13/d14/de5/dd8 0 2024-04-22T15:49:41.419 INFO:tasks.workunit.client.0.smithi016.stdout:4/638: mknod d7/d21/d7c/dbc/cd3 0 2024-04-22T15:49:41.424 INFO:tasks.workunit.client.0.smithi016.stdout:3/736: mknod d7/d1b/d21/d55/ddb/db7/ce7 0 2024-04-22T15:49:41.427 INFO:tasks.workunit.client.0.smithi016.stdout:5/656: rename df/d15/d1a/d28/f8f to df/d15/d1a/d50/d77/d7a/fcb 0 2024-04-22T15:49:41.427 INFO:tasks.workunit.client.0.smithi016.stdout:3/737: dread - d7/d17/d18/d32/d42/d63/d86/dd8/fd3 zero size 2024-04-22T15:49:41.429 INFO:tasks.workunit.client.0.smithi016.stdout:0/736: dwrite dc/d13/d14/de5/d65/d30/d96/fae [0,4194304] 0 2024-04-22T15:49:41.430 INFO:tasks.workunit.client.0.smithi016.stdout:3/738: dread - d7/d1b/d21/d55/fb6 zero size 2024-04-22T15:49:41.432 INFO:tasks.workunit.client.1.smithi027.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-22T15:49:41.432 INFO:tasks.workunit.client.0.smithi016.stdout:4/639: link d7/d21/d45/d49/f56 d7/d21/d45/d49/d67/fd4 0 2024-04-22T15:49:41.434 INFO:tasks.workunit.client.0.smithi016.stdout:3/739: read - d7/d17/d18/d32/d42/f9a zero size 2024-04-22T15:49:41.435 INFO:tasks.workunit.client.0.smithi016.stdout:5/657: rename df/d15/d1a/d50/d77/d7a/la0 to df/d5b/lcc 0 2024-04-22T15:49:41.435 INFO:tasks.workunit.client.0.smithi016.stdout:8/616: dwrite d2/f5c [0,4194304] 0 2024-04-22T15:49:41.436 INFO:tasks.workunit.client.0.smithi016.stdout:0/737: mkdir dc/d13/d14/de5/d65/d1f/dd4/df4 0 2024-04-22T15:49:41.439 INFO:tasks.workunit.client.0.smithi016.stdout:8/617: truncate d2/d48/dad/d6a/fbf 641156 0 2024-04-22T15:49:41.440 INFO:tasks.workunit.client.0.smithi016.stdout:3/740: creat d7/d17/d18/d32/d42/d63/d8f/fe8 x:0 0 0 2024-04-22T15:49:41.441 INFO:tasks.workunit.client.1.smithi027.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-22T15:49:41.441 INFO:tasks.workunit.client.1.smithi027.stderr:+ make 2024-04-22T15:49:41.441 INFO:tasks.workunit.client.0.smithi016.stdout:0/738: rmdir dc/d13/d14/de5/d65/d57 39 2024-04-22T15:49:41.445 INFO:tasks.workunit.client.0.smithi016.stdout:8/618: mkdir d2/d82/d44/d66/d7e/dc0 0 2024-04-22T15:49:41.449 INFO:tasks.workunit.client.0.smithi016.stdout:0/739: fsync dc/d13/d14/de5/d65/d40/d4a/d59/f87 0 2024-04-22T15:49:41.450 INFO:tasks.workunit.client.0.smithi016.stdout:0/740: readlink dc/d13/d5d/ldd 0 2024-04-22T15:49:41.451 INFO:tasks.workunit.client.0.smithi016.stdout:0/741: write dc/d13/d14/de5/d65/d1f/fb3 [805228,79841] 0 2024-04-22T15:49:41.451 INFO:tasks.workunit.client.0.smithi016.stdout:8/619: mkdir d2/d82/d44/dc1 0 2024-04-22T15:49:41.452 INFO:tasks.workunit.client.0.smithi016.stdout:0/742: stat dc/d13/d6f/d8b/f9a 0 2024-04-22T15:49:41.453 INFO:tasks.workunit.client.0.smithi016.stdout:5/658: chown df/d5b/lcc 22767855 1 2024-04-22T15:49:41.454 INFO:tasks.workunit.client.0.smithi016.stdout:4/640: write d7/d4e/d2f/d6f/f84 [1800298,87254] 0 2024-04-22T15:49:41.456 INFO:tasks.workunit.client.0.smithi016.stdout:8/620: fsync d2/d82/d44/f53 0 2024-04-22T15:49:41.457 INFO:tasks.workunit.client.0.smithi016.stdout:3/741: rmdir d7/d17/d18/d32/d42/d63/d8f 39 2024-04-22T15:49:41.459 INFO:tasks.workunit.client.0.smithi016.stdout:4/641: rmdir d7/d21/d42/d59/d88 39 2024-04-22T15:49:41.460 INFO:tasks.workunit.client.0.smithi016.stdout:0/743: dwrite dc/d13/d14/de5/d65/d30/d96/fae [0,4194304] 0 2024-04-22T15:49:41.461 INFO:tasks.workunit.client.0.smithi016.stdout:4/642: fdatasync d7/f8 0 2024-04-22T15:49:41.463 INFO:tasks.workunit.client.0.smithi016.stdout:3/742: fsync d7/d17/d18/d27/f6e 0 2024-04-22T15:49:41.465 INFO:tasks.workunit.client.0.smithi016.stdout:3/743: chown d7/d17/d18/d32/d42/d63/d86/fdf 1003838673 1 2024-04-22T15:49:41.466 INFO:tasks.workunit.client.0.smithi016.stdout:0/744: truncate dc/d13/d14/de5/d65/fa7 608201 0 2024-04-22T15:49:41.466 INFO:tasks.workunit.client.0.smithi016.stdout:4/643: symlink d7/d4e/d2f/d50/ld5 0 2024-04-22T15:49:41.468 INFO:tasks.workunit.client.0.smithi016.stdout:4/644: truncate d7/fbe 357785 0 2024-04-22T15:49:41.470 INFO:tasks.workunit.client.0.smithi016.stdout:4/645: chown d7/d21/d42/c5b 91563283 1 2024-04-22T15:49:41.471 INFO:tasks.workunit.client.0.smithi016.stdout:5/659: dread df/d22/f25 [0,4194304] 0 2024-04-22T15:49:41.472 INFO:tasks.workunit.client.0.smithi016.stdout:3/744: mknod d7/d17/d18/d32/d3c/ce9 0 2024-04-22T15:49:41.473 INFO:tasks.workunit.client.0.smithi016.stdout:3/745: fsync d7/d1b/d21/d55/dac/fdd 0 2024-04-22T15:49:41.476 INFO:tasks.workunit.client.0.smithi016.stdout:5/660: mknod df/d5b/db1/ccd 0 2024-04-22T15:49:41.481 INFO:tasks.workunit.client.0.smithi016.stdout:3/746: creat d7/fea x:0 0 0 2024-04-22T15:49:41.483 INFO:tasks.workunit.client.0.smithi016.stdout:5/661: dwrite df/d15/d1a/d50/d77/f39 [4194304,4194304] 0 2024-04-22T15:49:41.486 INFO:tasks.workunit.client.0.smithi016.stdout:0/745: truncate dc/d13/d14/de5/d65/f42 7403087 0 2024-04-22T15:49:41.487 INFO:tasks.workunit.client.0.smithi016.stdout:4/646: write d7/d21/d45/d49/f56 [4024384,58] 0 2024-04-22T15:49:41.493 INFO:tasks.workunit.client.0.smithi016.stdout:4/647: fdatasync d7/d4e/d2f/d6f/fa8 0 2024-04-22T15:49:41.494 INFO:tasks.workunit.client.0.smithi016.stdout:5/662: dread df/d22/f42 [0,4194304] 0 2024-04-22T15:49:41.495 INFO:tasks.workunit.client.1.smithi027.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-22T15:49:41.496 INFO:tasks.workunit.client.0.smithi016.stdout:3/747: dwrite d7/d17/d18/d32/d3c/d47/dc3/fde [0,4194304] 0 2024-04-22T15:49:41.498 INFO:tasks.workunit.client.0.smithi016.stdout:0/746: dwrite dc/d7d/fab [0,4194304] 0 2024-04-22T15:49:41.502 INFO:tasks.workunit.client.0.smithi016.stdout:5/663: write df/d15/d1a/d50/d77/d7a/f80 [3552737,59175] 0 2024-04-22T15:49:41.502 INFO:tasks.workunit.client.0.smithi016.stdout:0/747: write dc/d13/d5d/f85 [119001,23297] 0 2024-04-22T15:49:41.502 INFO:tasks.workunit.client.0.smithi016.stdout:5/664: chown df/d5b 523388356 1 2024-04-22T15:49:41.510 INFO:tasks.workunit.client.0.smithi016.stdout:0/748: mkdir dc/d13/d14/de5/d65/df5 0 2024-04-22T15:49:41.512 INFO:tasks.workunit.client.0.smithi016.stdout:5/665: dwrite df/d15/d1a/d50/d77/d7d/d8b/d93/fa7 [0,4194304] 0 2024-04-22T15:49:41.513 INFO:tasks.workunit.client.0.smithi016.stdout:0/749: dread - dc/d13/d70/da6/fc3 zero size 2024-04-22T15:49:41.515 INFO:tasks.workunit.client.0.smithi016.stdout:3/748: dwrite d7/d17/d18/d32/d42/d63/d8f/fb0 [0,4194304] 0 2024-04-22T15:49:41.519 INFO:tasks.workunit.client.0.smithi016.stdout:3/749: write d7/d17/d18/d32/d42/d63/d86/dd8/f76 [2013541,71390] 0 2024-04-22T15:49:41.521 INFO:tasks.workunit.client.0.smithi016.stdout:3/750: dread d7/d1b/fbc [0,4194304] 0 2024-04-22T15:49:41.522 INFO:tasks.workunit.client.0.smithi016.stdout:3/751: dread - d7/d17/d18/d32/d8a/f9b zero size 2024-04-22T15:49:41.527 INFO:tasks.workunit.client.0.smithi016.stdout:4/648: truncate d7/d24/d41/f57 2851136 0 2024-04-22T15:49:41.534 INFO:tasks.workunit.client.0.smithi016.stdout:5/666: rename df/d15/d1a/f6c to df/d15/d1a/d50/d77/fce 0 2024-04-22T15:49:41.536 INFO:tasks.workunit.client.0.smithi016.stdout:4/649: mkdir d7/d21/d45/dd6 0 2024-04-22T15:49:41.543 INFO:tasks.workunit.client.0.smithi016.stdout:4/650: mknod d7/d21/d45/d49/d9b/db6/cd7 0 2024-04-22T15:49:41.549 INFO:tasks.workunit.client.0.smithi016.stdout:3/752: rename d7/d1b to d7/d17/d18/d32/d42/d63/deb 0 2024-04-22T15:49:41.552 INFO:tasks.workunit.client.0.smithi016.stdout:4/651: creat d7/d21/d45/d49/fd8 x:0 0 0 2024-04-22T15:49:41.557 INFO:tasks.workunit.client.0.smithi016.stdout:4/652: unlink d7/c18 0 2024-04-22T15:49:41.565 INFO:tasks.workunit.client.0.smithi016.stdout:1/818: sync 2024-04-22T15:49:41.567 INFO:tasks.workunit.client.0.smithi016.stdout:4/653: creat d7/d21/d42/d59/d88/d9d/dad/db9/fd9 x:0 0 0 2024-04-22T15:49:41.570 INFO:tasks.workunit.client.0.smithi016.stdout:3/753: dread d7/d17/d18/d32/d3c/d47/f7f [0,4194304] 0 2024-04-22T15:49:41.572 INFO:tasks.workunit.client.0.smithi016.stdout:1/819: unlink d3/d4/d3a/d96/fe9 0 2024-04-22T15:49:41.579 INFO:tasks.workunit.client.0.smithi016.stdout:5/667: creat df/d15/d1a/d50/d77/d7d/d8b/fcf x:0 0 0 2024-04-22T15:49:41.584 INFO:tasks.workunit.client.0.smithi016.stdout:4/654: mkdir d7/d21/d45/dd6/dda 0 2024-04-22T15:49:41.591 INFO:tasks.workunit.client.0.smithi016.stdout:1/820: unlink d3/d4/da/f83 0 2024-04-22T15:49:41.591 INFO:tasks.workunit.client.0.smithi016.stdout:5/668: write f2 [4137356,2838] 0 2024-04-22T15:49:41.618 INFO:tasks.workunit.client.0.smithi016.stdout:3/754: truncate d7/d17/d18/d27/f53 2826012 0 2024-04-22T15:49:41.618 INFO:tasks.workunit.client.0.smithi016.stdout:5/669: dread df/d15/d1a/d28/f74 [0,4194304] 0 2024-04-22T15:49:41.620 INFO:tasks.workunit.client.0.smithi016.stdout:3/755: truncate d7/d17/d18/d32/d42/d63/deb/d21/d55/fa6 389239 0 2024-04-22T15:49:41.621 INFO:tasks.workunit.client.0.smithi016.stdout:4/655: write d7/d4e/f60 [1170276,76470] 0 2024-04-22T15:49:41.622 INFO:tasks.workunit.client.0.smithi016.stdout:4/656: chown d7/d24/d41/l95 96830001 1 2024-04-22T15:49:41.626 INFO:tasks.workunit.client.0.smithi016.stdout:5/670: rename df/d15/d1a/d28/f74 to df/d5b/fd0 0 2024-04-22T15:49:41.626 INFO:tasks.workunit.client.0.smithi016.stdout:4/657: dread d7/d21/d45/f6d [0,4194304] 0 2024-04-22T15:49:41.626 INFO:tasks.workunit.client.0.smithi016.stdout:3/756: dread - d7/d17/d18/d32/d42/d63/deb/f7c zero size 2024-04-22T15:49:41.629 INFO:tasks.workunit.client.0.smithi016.stdout:4/658: dread f5 [0,4194304] 0 2024-04-22T15:49:41.630 INFO:tasks.workunit.client.0.smithi016.stdout:4/659: fdatasync d7/d4e/d39/dc3/fa5 0 2024-04-22T15:49:41.630 INFO:tasks.workunit.client.0.smithi016.stdout:4/660: stat d7/d24/fb7 0 2024-04-22T15:49:41.630 INFO:tasks.workunit.client.0.smithi016.stdout:4/661: read - d7/d4e/d39/fa4 zero size 2024-04-22T15:49:41.632 INFO:tasks.workunit.client.0.smithi016.stdout:4/662: write d7/d21/d45/d49/fd8 [159611,91284] 0 2024-04-22T15:49:41.635 INFO:tasks.workunit.client.0.smithi016.stdout:5/671: truncate df/f57 1400220 0 2024-04-22T15:49:41.636 INFO:tasks.workunit.client.0.smithi016.stdout:5/672: dread - df/d15/d1a/d50/d77/fa6 zero size 2024-04-22T15:49:41.636 INFO:tasks.workunit.client.0.smithi016.stdout:5/673: stat df/l38 0 2024-04-22T15:49:41.642 INFO:tasks.workunit.client.0.smithi016.stdout:4/663: unlink d7/d21/d42/d59/d88/d9d/dad/db9/fd9 0 2024-04-22T15:49:41.649 INFO:tasks.workunit.client.0.smithi016.stdout:4/664: readlink d7/d24/l2a 0 2024-04-22T15:49:41.655 INFO:tasks.workunit.client.0.smithi016.stdout:5/674: getdents df/d5b/d5d 0 2024-04-22T15:49:41.658 INFO:tasks.workunit.client.0.smithi016.stdout:3/757: dwrite d7/d17/d18/d32/d3c/d47/f33 [0,4194304] 0 2024-04-22T15:49:41.662 INFO:tasks.workunit.client.0.smithi016.stdout:4/665: rename d7/d21/d42/cc8 to d7/d3c/cdb 0 2024-04-22T15:49:41.666 INFO:tasks.workunit.client.0.smithi016.stdout:5/675: dwrite df/d15/d1a/f1e [0,4194304] 0 2024-04-22T15:49:41.671 INFO:tasks.workunit.client.0.smithi016.stdout:4/666: mknod d7/d22/cdc 0 2024-04-22T15:49:41.677 INFO:tasks.workunit.client.0.smithi016.stdout:4/667: symlink d7/d21/d45/d49/d9b/ldd 0 2024-04-22T15:49:41.682 INFO:tasks.workunit.client.0.smithi016.stdout:5/676: link df/d5b/d70/fc3 df/d5b/db1/fd1 0 2024-04-22T15:49:41.683 INFO:tasks.workunit.client.0.smithi016.stdout:6/706: sync 2024-04-22T15:49:41.683 INFO:tasks.workunit.client.0.smithi016.stdout:7/669: sync 2024-04-22T15:49:41.685 INFO:tasks.workunit.client.0.smithi016.stdout:7/670: write d3/dc/d1b/d5a/d9/f9c [924020,105620] 0 2024-04-22T15:49:41.685 INFO:tasks.workunit.client.0.smithi016.stdout:3/758: getdents d7/d17/d18/d32/d42/d63/d86/dd8 0 2024-04-22T15:49:41.695 INFO:tasks.workunit.client.0.smithi016.stdout:7/671: stat d3/dc/d1b/d31/c37 0 2024-04-22T15:49:41.704 INFO:tasks.workunit.client.0.smithi016.stdout:7/672: write d3/dc/d1b/d5a/f46 [144018,6932] 0 2024-04-22T15:49:41.706 INFO:tasks.workunit.client.0.smithi016.stdout:5/677: creat df/d15/d1a/fd2 x:0 0 0 2024-04-22T15:49:41.712 INFO:tasks.workunit.client.0.smithi016.stdout:7/673: readlink d3/dc/d1b/d31/l82 0 2024-04-22T15:49:41.718 INFO:tasks.workunit.client.0.smithi016.stdout:5/678: creat df/d15/d1a/d4e/d58/d87/fd3 x:0 0 0 2024-04-22T15:49:41.720 INFO:tasks.workunit.client.0.smithi016.stdout:4/668: write f5 [773927,25431] 0 2024-04-22T15:49:41.725 INFO:tasks.workunit.client.0.smithi016.stdout:5/679: chown df/d15/d1a/d50/d77/f4c 20980428 1 2024-04-22T15:49:41.727 INFO:tasks.workunit.client.0.smithi016.stdout:2/655: sync 2024-04-22T15:49:41.727 INFO:tasks.workunit.client.0.smithi016.stdout:8/621: sync 2024-04-22T15:49:41.727 INFO:tasks.workunit.client.0.smithi016.stdout:9/687: sync 2024-04-22T15:49:41.730 INFO:tasks.workunit.client.0.smithi016.stdout:6/707: write d3/da/dd/d7b/fbb [806005,89867] 0 2024-04-22T15:49:41.741 INFO:tasks.workunit.client.0.smithi016.stdout:9/688: creat d9/d20/fd1 x:0 0 0 2024-04-22T15:49:41.741 INFO:tasks.workunit.client.0.smithi016.stdout:6/708: rmdir d3/da/dd/d1b/d57 39 2024-04-22T15:49:41.741 INFO:tasks.workunit.client.0.smithi016.stdout:3/759: truncate d7/d17/d18/d27/fa3 431212 0 2024-04-22T15:49:41.748 INFO:tasks.workunit.client.0.smithi016.stdout:8/622: dwrite d2/d82/d44/d66/d1c/f3d [4194304,4194304] 0 2024-04-22T15:49:41.749 INFO:tasks.workunit.client.0.smithi016.stdout:7/674: truncate d3/dc/d1b/d5a/d69/f9d 2485645 0 2024-04-22T15:49:41.753 INFO:tasks.workunit.client.0.smithi016.stdout:6/709: dwrite d3/da/db/d44/d68/fe6 [0,4194304] 0 2024-04-22T15:49:41.757 INFO:tasks.workunit.client.0.smithi016.stdout:4/669: truncate d7/d22/f64 1457426 0 2024-04-22T15:49:41.761 INFO:tasks.workunit.client.0.smithi016.stdout:5/680: dwrite df/d22/f25 [4194304,4194304] 0 2024-04-22T15:49:41.768 INFO:tasks.workunit.client.0.smithi016.stdout:0/750: sync 2024-04-22T15:49:41.768 INFO:tasks.workunit.client.0.smithi016.stdout:1/821: sync 2024-04-22T15:49:41.771 INFO:tasks.workunit.client.0.smithi016.stdout:2/656: write d2/d4d/d8e/fad [1607958,83641] 0 2024-04-22T15:49:41.782 INFO:tasks.workunit.client.0.smithi016.stdout:9/689: mkdir d9/d5d/d62/dd2 0 2024-04-22T15:49:41.786 INFO:tasks.workunit.client.0.smithi016.stdout:3/760: mknod d7/d17/d18/d32/d42/d63/d86/cec 0 2024-04-22T15:49:41.790 INFO:tasks.workunit.client.0.smithi016.stdout:8/623: symlink d2/d10/d43/lc2 0 2024-04-22T15:49:41.794 INFO:tasks.workunit.client.0.smithi016.stdout:8/624: dread d2/d82/d44/d66/d79/f8b [0,4194304] 0 2024-04-22T15:49:41.796 INFO:tasks.workunit.client.0.smithi016.stdout:6/710: rename d3/da/db/c60 to d3/da/d12/d13/d97/ced 0 2024-04-22T15:49:41.815 INFO:tasks.workunit.client.0.smithi016.stdout:7/675: creat d3/dc/d33/d84/de3/ff2 x:0 0 0 2024-04-22T15:49:41.817 INFO:tasks.workunit.client.0.smithi016.stdout:8/625: rename d2/d48/dad/l2e to d2/d10/d47/lc3 0 2024-04-22T15:49:41.817 INFO:tasks.workunit.client.0.smithi016.stdout:3/761: dwrite d7/d17/d18/d32/d3c/d47/f48 [0,4194304] 0 2024-04-22T15:49:41.818 INFO:tasks.workunit.client.0.smithi016.stdout:4/670: mknod d7/d3c/cde 0 2024-04-22T15:49:41.819 INFO:tasks.workunit.client.0.smithi016.stdout:8/626: dread - d2/d48/dad/d6a/f9a zero size 2024-04-22T15:49:41.821 INFO:tasks.workunit.client.0.smithi016.stdout:4/671: write d7/d3c/f99 [1714758,81368] 0 2024-04-22T15:49:41.825 INFO:tasks.workunit.client.0.smithi016.stdout:0/751: mkdir dc/d13/d14/de5/dd1/ddc/df6 0 2024-04-22T15:49:41.826 INFO:tasks.workunit.client.0.smithi016.stdout:3/762: dwrite d7/d17/d18/d32/d3c/fcc [0,4194304] 0 2024-04-22T15:49:41.832 INFO:tasks.workunit.client.0.smithi016.stdout:3/763: fdatasync d7/d17/d18/d27/fb8 0 2024-04-22T15:49:41.832 INFO:tasks.workunit.client.0.smithi016.stdout:2/657: link d2/d2c/d97/d67/dbf/f8a d2/d4d/dc4/ff9 0 2024-04-22T15:49:41.833 INFO:tasks.workunit.client.0.smithi016.stdout:9/690: unlink d9/de/d23/d24/d25/da6/l9f 0 2024-04-22T15:49:41.837 INFO:tasks.workunit.client.0.smithi016.stdout:7/676: chown d3/dc/d1b/d31/c93 3082362 1 2024-04-22T15:49:41.839 INFO:tasks.workunit.client.0.smithi016.stdout:6/711: rename d3/da/d12/d13/d46 to d3/da/d12/d13/dd2/dee 0 2024-04-22T15:49:41.880 INFO:tasks.workunit.client.0.smithi016.stdout:4/672: creat d7/d21/d45/d49/d9b/fdf x:0 0 0 2024-04-22T15:49:41.881 INFO:tasks.workunit.client.0.smithi016.stdout:9/691: mknod d9/d20/dac/cd3 0 2024-04-22T15:49:41.883 INFO:tasks.workunit.client.0.smithi016.stdout:7/677: symlink d3/dc/d1b/d31/d88/lf3 0 2024-04-22T15:49:41.884 INFO:tasks.workunit.client.0.smithi016.stdout:5/681: dwrite df/d5b/d70/fc3 [0,4194304] 0 2024-04-22T15:49:41.885 INFO:tasks.workunit.client.0.smithi016.stdout:0/752: chown dc/d13/d6f/d8b/leb 50 1 2024-04-22T15:49:41.889 INFO:tasks.workunit.client.0.smithi016.stdout:6/712: dwrite d3/da/db/d44/d68/faa [0,4194304] 0 2024-04-22T15:49:41.889 INFO:tasks.workunit.client.0.smithi016.stdout:1/822: getdents d3/d4/d15/d38/d63 0 2024-04-22T15:49:41.891 INFO:tasks.workunit.client.0.smithi016.stdout:0/753: write dc/f68 [65456,90557] 0 2024-04-22T15:49:41.892 INFO:tasks.workunit.client.0.smithi016.stdout:4/673: symlink d7/d21/d42/d59/d88/d9d/le0 0 2024-04-22T15:49:41.894 INFO:tasks.workunit.client.0.smithi016.stdout:3/764: dread d7/f69 [0,4194304] 0 2024-04-22T15:49:41.895 INFO:tasks.workunit.client.0.smithi016.stdout:1/823: mkdir d3/d2e/d117 0 2024-04-22T15:49:41.896 INFO:tasks.workunit.client.0.smithi016.stdout:4/674: fdatasync d7/d3c/f99 0 2024-04-22T15:49:41.901 INFO:tasks.workunit.client.0.smithi016.stdout:0/754: creat dc/d13/d99/ff7 x:0 0 0 2024-04-22T15:49:41.905 INFO:tasks.workunit.client.0.smithi016.stdout:7/678: rename d3/de6/fb6 to d3/dc/d33/ff4 0 2024-04-22T15:49:41.905 INFO:tasks.workunit.client.0.smithi016.stdout:4/675: chown d7/d21/lc1 1026438527 1 2024-04-22T15:49:41.907 INFO:tasks.workunit.client.0.smithi016.stdout:9/692: dwrite d9/de/d26/d5a/fc7 [0,4194304] 0 2024-04-22T15:49:41.908 INFO:tasks.workunit.client.0.smithi016.stdout:1/824: mkdir d3/d4/d15/d38/d63/d118 0 2024-04-22T15:49:41.911 INFO:tasks.workunit.client.0.smithi016.stdout:5/682: dwrite f2 [0,4194304] 0 2024-04-22T15:49:41.915 INFO:tasks.workunit.client.0.smithi016.stdout:1/825: dread - d3/d4/d15/fce zero size 2024-04-22T15:49:41.918 INFO:tasks.workunit.client.0.smithi016.stdout:7/679: chown d3/dc/d1b/d5a/f4b 85692 1 2024-04-22T15:49:41.919 INFO:tasks.workunit.client.0.smithi016.stdout:4/676: chown d7/d21/d7c/dbc/cd3 7986 1 2024-04-22T15:49:41.920 INFO:tasks.workunit.client.0.smithi016.stdout:3/765: truncate d7/d17/d18/d27/fa3 623254 0 2024-04-22T15:49:41.920 INFO:tasks.workunit.client.0.smithi016.stdout:2/658: dwrite d2/d4d/dc4/ff9 [0,4194304] 0 2024-04-22T15:49:41.923 INFO:tasks.workunit.client.0.smithi016.stdout:9/693: creat d9/d57/d91/dc1/da2/fd4 x:0 0 0 2024-04-22T15:49:41.928 INFO:tasks.workunit.client.0.smithi016.stdout:7/680: chown d3/de6/l98 5 1 2024-04-22T15:49:41.930 INFO:tasks.workunit.client.0.smithi016.stdout:0/755: dwrite dc/d13/d14/d55/fbf [0,4194304] 0 2024-04-22T15:49:41.931 INFO:tasks.workunit.client.0.smithi016.stdout:4/677: rmdir d7/d4e 39 2024-04-22T15:49:41.933 INFO:tasks.workunit.client.0.smithi016.stdout:2/659: rmdir d2/d4d/d76 39 2024-04-22T15:49:41.934 INFO:tasks.workunit.client.0.smithi016.stdout:6/713: dread d3/da/db/d14/d16/f20 [0,4194304] 0 2024-04-22T15:49:41.935 INFO:tasks.workunit.client.0.smithi016.stdout:9/694: creat d9/de/d23/d24/d25/da6/d4f/fd5 x:0 0 0 2024-04-22T15:49:41.936 INFO:tasks.workunit.client.0.smithi016.stdout:4/678: readlink d7/d21/d42/d59/d88/d9d/le0 0 2024-04-22T15:49:41.938 INFO:tasks.workunit.client.0.smithi016.stdout:5/683: link df/d5b/d70/fa4 df/d15/d1a/d4e/d58/fd4 0 2024-04-22T15:49:41.938 INFO:tasks.workunit.client.0.smithi016.stdout:1/826: write d3/d28/f8b [1293193,55328] 0 2024-04-22T15:49:41.942 INFO:tasks.workunit.client.0.smithi016.stdout:2/660: symlink d2/d2c/d97/lfa 0 2024-04-22T15:49:41.944 INFO:tasks.workunit.client.0.smithi016.stdout:3/766: dwrite d7/d17/d18/d32/d3c/f88 [0,4194304] 0 2024-04-22T15:49:41.945 INFO:tasks.workunit.client.0.smithi016.stdout:7/681: creat d3/dc/d1b/d31/d88/de1/ff5 x:0 0 0 2024-04-22T15:49:41.945 INFO:tasks.workunit.client.0.smithi016.stdout:1/827: dread - d3/d4/d19/d20/d2b/de0/ffd zero size 2024-04-22T15:49:41.946 INFO:tasks.workunit.client.0.smithi016.stdout:9/695: creat d9/d57/da7/fd6 x:0 0 0 2024-04-22T15:49:41.950 INFO:tasks.workunit.client.0.smithi016.stdout:2/661: chown d2/d1a/d7b/dcc/l72 485541 1 2024-04-22T15:49:41.951 INFO:tasks.workunit.client.0.smithi016.stdout:3/767: chown d7/d17/d18/d32/d3c/d47/f52 4 1 2024-04-22T15:49:41.951 INFO:tasks.workunit.client.0.smithi016.stdout:9/696: mkdir d9/d20/dd7 0 2024-04-22T15:49:41.953 INFO:tasks.workunit.client.0.smithi016.stdout:1/828: symlink d3/d4/d3a/d91/db3/l119 0 2024-04-22T15:49:41.953 INFO:tasks.workunit.client.0.smithi016.stdout:4/679: mkdir d7/d4e/de1 0 2024-04-22T15:49:41.954 INFO:tasks.workunit.client.0.smithi016.stdout:2/662: chown d2/d1a/d7b/dcc/de5 133626586 1 2024-04-22T15:49:41.958 INFO:tasks.workunit.client.0.smithi016.stdout:2/663: chown d2/d23/d92/f66 59285958 1 2024-04-22T15:49:41.958 INFO:tasks.workunit.client.0.smithi016.stdout:0/756: rename dc/d13/d70/da6/fc3 to dc/d13/d14/de5/d65/d40/d4a/ff8 0 2024-04-22T15:49:41.959 INFO:tasks.workunit.client.0.smithi016.stdout:9/697: write d9/de/d23/d24/d25/da6/f98 [995511,114965] 0 2024-04-22T15:49:41.959 INFO:tasks.workunit.client.0.smithi016.stdout:1/829: rmdir d3/d28/dee 39 2024-04-22T15:49:41.961 INFO:tasks.workunit.client.0.smithi016.stdout:3/768: truncate d7/d17/d18/d32/d3c/d51/f5f 1423281 0 2024-04-22T15:49:41.962 INFO:tasks.workunit.client.0.smithi016.stdout:2/664: chown d2/d1a/d4c/d9f/ca2 15060 1 2024-04-22T15:49:41.964 INFO:tasks.workunit.client.0.smithi016.stdout:1/830: write d3/d4/d19/d20/f60 [5190046,90373] 0 2024-04-22T15:49:41.964 INFO:tasks.workunit.client.0.smithi016.stdout:6/714: rename d3/d53/d83/ldb to d3/da/dd/d67/d6c/db1/lef 0 2024-04-22T15:49:41.964 INFO:tasks.workunit.client.0.smithi016.stdout:9/698: write d9/d57/d91/dc1/f92 [684875,1976] 0 2024-04-22T15:49:41.965 INFO:tasks.workunit.client.0.smithi016.stdout:5/684: dread df/d15/d1a/fbe [0,4194304] 0 2024-04-22T15:49:41.967 INFO:tasks.workunit.client.0.smithi016.stdout:7/682: rename d3/dc/d1b/d5a/d59 to d3/dc/d1b/d5a/d59/d64/df6 22 2024-04-22T15:49:41.969 INFO:tasks.workunit.client.0.smithi016.stdout:5/685: rmdir df/d5b/db3 39 2024-04-22T15:49:41.970 INFO:tasks.workunit.client.0.smithi016.stdout:6/715: mknod d3/d53/dab/cf0 0 2024-04-22T15:49:41.970 INFO:tasks.workunit.client.0.smithi016.stdout:2/665: truncate d2/d51/f5a 403379 0 2024-04-22T15:49:41.973 INFO:tasks.workunit.client.0.smithi016.stdout:5/686: truncate df/d15/d1a/d50/d77/fa1 801089 0 2024-04-22T15:49:41.974 INFO:tasks.workunit.client.0.smithi016.stdout:5/687: write df/d15/d1a/d50/d77/fa1 [1373055,8479] 0 2024-04-22T15:49:41.976 INFO:tasks.workunit.client.0.smithi016.stdout:0/757: creat dc/d13/d14/ff9 x:0 0 0 2024-04-22T15:49:41.977 INFO:tasks.workunit.client.0.smithi016.stdout:2/666: symlink d2/d1a/d4c/d77/lfb 0 2024-04-22T15:49:41.978 INFO:tasks.workunit.client.0.smithi016.stdout:3/769: link d7/d17/ce5 d7/d17/d18/d27/ced 0 2024-04-22T15:49:41.979 INFO:tasks.workunit.client.0.smithi016.stdout:0/758: dread - dc/d13/d14/de5/d65/d40/d4a/d59/d78/f86 zero size 2024-04-22T15:49:41.979 INFO:tasks.workunit.client.0.smithi016.stdout:1/831: mkdir d3/d4/d15/d38/d11a 0 2024-04-22T15:49:41.982 INFO:tasks.workunit.client.0.smithi016.stdout:9/699: mknod d9/de/d23/d24/d25/da6/cd8 0 2024-04-22T15:49:41.983 INFO:tasks.workunit.client.0.smithi016.stdout:5/688: creat df/d15/d1a/d50/d77/d7d/d8b/d93/fd5 x:0 0 0 2024-04-22T15:49:41.987 INFO:tasks.workunit.client.0.smithi016.stdout:4/680: rename d7/d21/d7c/dbc to d7/d21/d42/d59/de2 0 2024-04-22T15:49:41.987 INFO:tasks.workunit.client.0.smithi016.stdout:2/667: dwrite d2/d1a/d4c/d9f/da3/fcb [0,4194304] 0 2024-04-22T15:49:41.987 INFO:tasks.workunit.client.0.smithi016.stdout:6/716: rename d3/da/dd/d67/d6c to d3/da/dd/d67/d6c/db1/df1 22 2024-04-22T15:49:41.998 INFO:tasks.workunit.client.1.smithi027.stderr:++ readlink -f fsstress 2024-04-22T15:49:42.000 INFO:tasks.workunit.client.0.smithi016.stdout:6/717: dwrite d3/da/db/fce [0,4194304] 0 2024-04-22T15:49:42.000 INFO:tasks.workunit.client.0.smithi016.stdout:2/668: chown d2/d1a/d4c/c4e 225785660 1 2024-04-22T15:49:42.001 INFO:tasks.workunit.client.1.smithi027.stderr:+ BIN=/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-04-22T15:49:42.001 INFO:tasks.workunit.client.1.smithi027.stderr:+ popd 2024-04-22T15:49:42.003 INFO:tasks.workunit.client.1.smithi027.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-22T15:49:42.003 INFO:tasks.workunit.client.1.smithi027.stderr:+ popd 2024-04-22T15:49:42.004 INFO:tasks.workunit.client.1.smithi027.stdout:~/cephtest/mnt.1/client.1/tmp 2024-04-22T15:49:42.004 INFO:tasks.workunit.client.1.smithi027.stderr:++ mktemp -d -p . 2024-04-22T15:49:42.007 INFO:tasks.workunit.client.0.smithi016.stdout:3/770: rename d7/d17/d18/d32/d42/d63/d8f/cc9 to d7/d17/d18/d32/d3c/cee 0 2024-04-22T15:49:42.010 INFO:tasks.workunit.client.0.smithi016.stdout:7/683: dwrite d3/de6/f7c [0,4194304] 0 2024-04-22T15:49:42.011 INFO:tasks.workunit.client.0.smithi016.stdout:2/669: creat d2/d1a/d4c/d77/dea/ffc x:0 0 0 2024-04-22T15:49:42.012 INFO:tasks.workunit.client.1.smithi027.stderr:+ T=./tmp.RI33Yxg0i3 2024-04-22T15:49:42.012 INFO:tasks.workunit.client.1.smithi027.stderr:+ /home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.RI33Yxg0i3 -l 1 -n 1000 -p 10 -v 2024-04-22T15:49:42.015 INFO:tasks.workunit.client.0.smithi016.stdout:0/759: dread dc/d13/d14/de5/d65/f42 [0,4194304] 0 2024-04-22T15:49:42.017 INFO:tasks.workunit.client.0.smithi016.stdout:9/700: write d9/d20/f40 [1288664,110453] 0 2024-04-22T15:49:42.019 INFO:tasks.workunit.client.0.smithi016.stdout:1/832: rename d3/l2f to d3/d4/d19/d20/d2b/l11b 0 2024-04-22T15:49:42.019 INFO:tasks.workunit.client.1.smithi027.stdout:seed = 1713816494 2024-04-22T15:49:42.020 INFO:tasks.workunit.client.0.smithi016.stdout:2/670: unlink d2/d1a/l58 0 2024-04-22T15:49:42.023 INFO:tasks.workunit.client.0.smithi016.stdout:3/771: symlink d7/d17/d18/d32/d42/d63/deb/d21/d55/ddb/lef 0 2024-04-22T15:49:42.026 INFO:tasks.workunit.client.0.smithi016.stdout:9/701: mkdir d9/d57/da7/dd9 0 2024-04-22T15:49:42.027 INFO:tasks.workunit.client.0.smithi016.stdout:4/681: dread d7/d4e/d39/dc3/fa5 [0,4194304] 0 2024-04-22T15:49:42.029 INFO:tasks.workunit.client.0.smithi016.stdout:6/718: link d3/da/db/d44/d68/d95/d9e/cd9 d3/da/dd/d67/dbd/cf2 0 2024-04-22T15:49:42.030 INFO:tasks.workunit.client.0.smithi016.stdout:7/684: dwrite d3/dc/d1b/d5a/da4/fd7 [0,4194304] 0 2024-04-22T15:49:42.030 INFO:tasks.workunit.client.0.smithi016.stdout:5/689: dwrite df/d15/d1a/f78 [0,4194304] 0 2024-04-22T15:49:42.030 INFO:tasks.workunit.client.1.smithi027.stdout:1/0: write - no filename 2024-04-22T15:49:42.034 INFO:tasks.workunit.client.0.smithi016.stdout:3/772: chown d7/d17/d18/d32/d42/d63/deb/fc2 75900 1 2024-04-22T15:49:42.034 INFO:tasks.workunit.client.0.smithi016.stdout:2/671: mknod d2/d9/cfd 0 2024-04-22T15:49:42.034 INFO:tasks.workunit.client.0.smithi016.stdout:4/682: fsync d7/d4e/f29 0 2024-04-22T15:49:42.034 INFO:tasks.workunit.client.0.smithi016.stdout:0/760: unlink dc/d13/d14/de5/d65/d57/l58 0 2024-04-22T15:49:42.034 INFO:tasks.workunit.client.0.smithi016.stdout:2/672: chown d2/d2c/de7 26 1 2024-04-22T15:49:42.034 INFO:tasks.workunit.client.0.smithi016.stdout:6/719: write d3/da/dd/fb2 [436516,58555] 0 2024-04-22T15:49:42.035 INFO:tasks.workunit.client.0.smithi016.stdout:2/673: write d2/d4d/d64/fe6 [317392,60300] 0 2024-04-22T15:49:42.036 INFO:tasks.workunit.client.0.smithi016.stdout:0/761: fsync dc/d13/d14/d26/d32/fb0 0 2024-04-22T15:49:42.036 INFO:tasks.workunit.client.0.smithi016.stdout:3/773: fsync d7/d17/d18/d32/d3c/d47/fe3 0 2024-04-22T15:49:42.037 INFO:tasks.workunit.client.1.smithi027.stdout:2/0: unlink - no file 2024-04-22T15:49:42.044 INFO:tasks.workunit.client.1.smithi027.stdout:0/0: mkdir d0 0 2024-04-22T15:49:42.049 INFO:tasks.workunit.client.1.smithi027.stdout:3/0: fsync - no filename 2024-04-22T15:49:42.049 INFO:tasks.workunit.client.1.smithi027.stdout:3/1: dread - no filename 2024-04-22T15:49:42.049 INFO:tasks.workunit.client.1.smithi027.stdout:3/2: dwrite - no filename 2024-04-22T15:49:42.049 INFO:tasks.workunit.client.1.smithi027.stdout:3/3: unlink - no file 2024-04-22T15:49:42.049 INFO:tasks.workunit.client.1.smithi027.stdout:3/4: readlink - no filename 2024-04-22T15:49:42.050 INFO:tasks.workunit.client.1.smithi027.stdout:3/5: write - no filename 2024-04-22T15:49:42.050 INFO:tasks.workunit.client.1.smithi027.stdout:3/6: write - no filename 2024-04-22T15:49:42.050 INFO:tasks.workunit.client.1.smithi027.stdout:3/7: write - no filename 2024-04-22T15:49:42.050 INFO:tasks.workunit.client.0.smithi016.stdout:7/685: mknod d3/dc/d1b/d5a/d69/cf7 0 2024-04-22T15:49:42.051 INFO:tasks.workunit.client.0.smithi016.stdout:7/686: write d3/dc/d1b/d5a/d59/d64/fb3 [363105,93993] 0 2024-04-22T15:49:42.052 INFO:tasks.workunit.client.1.smithi027.stdout:1/1: creat f0 x:0 0 0 2024-04-22T15:49:42.053 INFO:tasks.workunit.client.1.smithi027.stdout:1/2: chown f0 0 1 2024-04-22T15:49:42.053 INFO:tasks.workunit.client.0.smithi016.stdout:2/674: creat d2/d23/d92/d55/ffe x:0 0 0 2024-04-22T15:49:42.054 INFO:tasks.workunit.client.1.smithi027.stdout:1/3: write f0 [173260,113179] 0 2024-04-22T15:49:42.055 INFO:tasks.workunit.client.0.smithi016.stdout:6/720: mknod d3/da/d12/d13/d38/d84/cf3 0 2024-04-22T15:49:42.056 INFO:tasks.workunit.client.1.smithi027.stdout:4/0: dwrite - no filename 2024-04-22T15:49:42.057 INFO:tasks.workunit.client.0.smithi016.stdout:3/774: mknod d7/d17/d18/d32/d42/d63/deb/d21/d55/ddb/db7/cf0 0 2024-04-22T15:49:42.057 INFO:tasks.workunit.client.1.smithi027.stdout:2/1: getdents . 0 2024-04-22T15:49:42.058 INFO:tasks.workunit.client.1.smithi027.stdout:2/2: write - no filename 2024-04-22T15:49:42.058 INFO:tasks.workunit.client.1.smithi027.stdout:2/3: fsync - no filename 2024-04-22T15:49:42.058 INFO:tasks.workunit.client.1.smithi027.stdout:2/4: write - no filename 2024-04-22T15:49:42.058 INFO:tasks.workunit.client.1.smithi027.stdout:2/5: dread - no filename 2024-04-22T15:49:42.058 INFO:tasks.workunit.client.1.smithi027.stdout:2/6: dwrite - no filename 2024-04-22T15:49:42.058 INFO:tasks.workunit.client.1.smithi027.stdout:2/7: chown . 47 1 2024-04-22T15:49:42.058 INFO:tasks.workunit.client.1.smithi027.stdout:2/8: fsync - no filename 2024-04-22T15:49:42.058 INFO:tasks.workunit.client.1.smithi027.stdout:2/9: rename - no filename 2024-04-22T15:49:42.059 INFO:tasks.workunit.client.1.smithi027.stdout:0/1: creat d0/f1 x:0 0 0 2024-04-22T15:49:42.059 INFO:tasks.workunit.client.0.smithi016.stdout:5/690: symlink df/ld6 0 2024-04-22T15:49:42.059 INFO:tasks.workunit.client.0.smithi016.stdout:3/775: write d7/d17/d18/d32/d42/d63/deb/d21/d55/fa6 [523151,108281] 0 2024-04-22T15:49:42.060 INFO:tasks.workunit.client.1.smithi027.stdout:0/2: dread - d0/f1 zero size 2024-04-22T15:49:42.060 INFO:tasks.workunit.client.0.smithi016.stdout:9/702: creat d9/d20/dd7/fda x:0 0 0 2024-04-22T15:49:42.060 INFO:tasks.workunit.client.0.smithi016.stdout:2/675: dread d2/d2c/d97/f6a [0,4194304] 0 2024-04-22T15:49:42.061 INFO:tasks.workunit.client.1.smithi027.stdout:1/4: mkdir d1 0 2024-04-22T15:49:42.062 INFO:tasks.workunit.client.0.smithi016.stdout:7/687: creat d3/dc/d33/d84/de3/ff8 x:0 0 0 2024-04-22T15:49:42.063 INFO:tasks.workunit.client.1.smithi027.stdout:5/0: write - no filename 2024-04-22T15:49:42.063 INFO:tasks.workunit.client.1.smithi027.stdout:5/1: link - no file 2024-04-22T15:49:42.063 INFO:tasks.workunit.client.1.smithi027.stdout:5/2: readlink - no filename 2024-04-22T15:49:42.063 INFO:tasks.workunit.client.1.smithi027.stdout:5/3: write - no filename 2024-04-22T15:49:42.063 INFO:tasks.workunit.client.1.smithi027.stdout:3/8: symlink l0 0 2024-04-22T15:49:42.064 INFO:tasks.workunit.client.1.smithi027.stdout:1/5: dread f0 [0,4194304] 0 2024-04-22T15:49:42.064 INFO:tasks.workunit.client.0.smithi016.stdout:7/688: write d3/de6/f7c [4050353,114874] 0 2024-04-22T15:49:42.065 INFO:tasks.workunit.client.0.smithi016.stdout:1/833: rename d3/d4/d3a/d6b/dc3/f115 to d3/d4/d15/f11c 0 2024-04-22T15:49:42.065 INFO:tasks.workunit.client.0.smithi016.stdout:6/721: rename d3 to d3/da/dd/d67/d6c/df4 22 2024-04-22T15:49:42.067 INFO:tasks.workunit.client.0.smithi016.stdout:6/722: readlink d3/da/db/d14/d16/l6b 0 2024-04-22T15:49:42.069 INFO:tasks.workunit.client.0.smithi016.stdout:9/703: unlink d9/d20/dac/cd3 0 2024-04-22T15:49:42.070 INFO:tasks.workunit.client.0.smithi016.stdout:0/762: dread dc/d13/d14/d26/f4f [0,4194304] 0 2024-04-22T15:49:42.070 INFO:tasks.workunit.client.1.smithi027.stdout:6/0: truncate - no filename 2024-04-22T15:49:42.071 INFO:tasks.workunit.client.1.smithi027.stdout:6/1: write - no filename 2024-04-22T15:49:42.071 INFO:tasks.workunit.client.1.smithi027.stdout:6/2: rename - no filename 2024-04-22T15:49:42.071 INFO:tasks.workunit.client.0.smithi016.stdout:2/676: mknod d2/d23/db8/cff 0 2024-04-22T15:49:42.072 INFO:tasks.workunit.client.1.smithi027.stdout:3/9: symlink l1 0 2024-04-22T15:49:42.072 INFO:tasks.workunit.client.1.smithi027.stdout:3/10: dread - no filename 2024-04-22T15:49:42.072 INFO:tasks.workunit.client.1.smithi027.stdout:3/11: write - no filename 2024-04-22T15:49:42.072 INFO:tasks.workunit.client.1.smithi027.stdout:3/12: fsync - no filename 2024-04-22T15:49:42.078 INFO:tasks.workunit.client.1.smithi027.stdout:4/1: creat f0 x:0 0 0 2024-04-22T15:49:42.081 INFO:tasks.workunit.client.1.smithi027.stdout:2/10: creat f0 x:0 0 0 2024-04-22T15:49:42.081 INFO:tasks.workunit.client.0.smithi016.stdout:1/834: truncate d3/d4/d19/f1d 6695701 0 2024-04-22T15:49:42.084 INFO:tasks.workunit.client.0.smithi016.stdout:5/691: dread df/d15/d1a/d28/f2c [4194304,4194304] 0 2024-04-22T15:49:42.086 INFO:tasks.workunit.client.0.smithi016.stdout:7/689: dwrite d3/de6/fbe [0,4194304] 0 2024-04-22T15:49:42.086 INFO:tasks.workunit.client.0.smithi016.stdout:5/692: chown df/d15/d1a/d4e/l5e 1194795 1 2024-04-22T15:49:42.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:41 smithi027 ceph-mon[34922]: pgmap v33: 65 pgs: 65 active+clean; 1.2 GiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 46 MiB/s rd, 155 MiB/s wr, 386 op/s 2024-04-22T15:49:42.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:41 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:49:42.087 INFO:tasks.workunit.client.0.smithi016.stdout:6/723: rmdir d3/da/db/d44/d68/d95/d9e 39 2024-04-22T15:49:42.087 INFO:tasks.workunit.client.1.smithi027.stdout:4/2: dwrite f0 [0,4194304] 0 2024-04-22T15:49:42.087 INFO:tasks.workunit.client.0.smithi016.stdout:6/724: chown d3/da/d12 25836 1 2024-04-22T15:49:42.088 INFO:tasks.workunit.client.1.smithi027.stdout:4/3: read f0 [358804,114607] 0 2024-04-22T15:49:42.088 INFO:tasks.workunit.client.0.smithi016.stdout:7/690: dread - d3/dc/d1b/d5a/d59/d64/fa0 zero size 2024-04-22T15:49:42.088 INFO:tasks.workunit.client.0.smithi016.stdout:5/693: stat df/d15/d1a/d50/d77/d7d/d8b/d93/dbb 0 2024-04-22T15:49:42.088 INFO:tasks.workunit.client.0.smithi016.stdout:7/691: fdatasync d3/de6/fdb 0 2024-04-22T15:49:42.089 INFO:tasks.workunit.client.1.smithi027.stdout:4/4: chown f0 63 1 2024-04-22T15:49:42.089 INFO:tasks.workunit.client.1.smithi027.stdout:0/3: rmdir d0 39 2024-04-22T15:49:42.093 INFO:tasks.workunit.client.1.smithi027.stdout:3/13: rename l0 to l2 0 2024-04-22T15:49:42.093 INFO:tasks.workunit.client.1.smithi027.stdout:3/14: truncate - no filename 2024-04-22T15:49:42.095 INFO:tasks.workunit.client.1.smithi027.stdout:5/4: getdents . 0 2024-04-22T15:49:42.095 INFO:tasks.workunit.client.1.smithi027.stdout:5/5: chown . 87049752 1 2024-04-22T15:49:42.104 INFO:tasks.workunit.client.1.smithi027.stdout:8/0: link - no file 2024-04-22T15:49:42.104 INFO:tasks.workunit.client.1.smithi027.stdout:8/1: unlink - no file 2024-04-22T15:49:42.104 INFO:tasks.workunit.client.1.smithi027.stdout:2/11: fsync f0 0 2024-04-22T15:49:42.105 INFO:tasks.workunit.client.0.smithi016.stdout:9/704: mkdir d9/de/d23/ddb 0 2024-04-22T15:49:42.107 INFO:tasks.workunit.client.1.smithi027.stdout:6/3: creat f0 x:0 0 0 2024-04-22T15:49:42.109 INFO:tasks.workunit.client.1.smithi027.stdout:6/4: write f0 [682094,6325] 0 2024-04-22T15:49:42.111 INFO:tasks.workunit.client.0.smithi016.stdout:5/694: creat df/d5b/d5d/d5f/fd7 x:0 0 0 2024-04-22T15:49:42.112 INFO:tasks.workunit.client.0.smithi016.stdout:3/776: truncate d7/d17/d18/d32/d42/d63/d86/dd8/f76 1075485 0 2024-04-22T15:49:42.113 INFO:tasks.workunit.client.0.smithi016.stdout:5/695: stat df/d15/d1a/d50/d77/fb4 0 2024-04-22T15:49:42.113 INFO:tasks.workunit.client.1.smithi027.stdout:4/5: creat f1 x:0 0 0 2024-04-22T15:49:42.114 INFO:tasks.workunit.client.1.smithi027.stdout:2/12: dwrite f0 [0,4194304] 0 2024-04-22T15:49:42.114 INFO:tasks.workunit.client.1.smithi027.stdout:2/13: rmdir - no directory 2024-04-22T15:49:42.115 INFO:tasks.workunit.client.0.smithi016.stdout:7/692: symlink d3/dc/d1b/d5a/d9/df/d94/lf9 0 2024-04-22T15:49:42.115 INFO:tasks.workunit.client.0.smithi016.stdout:9/705: dwrite d9/d20/f6f [0,4194304] 0 2024-04-22T15:49:42.116 INFO:tasks.workunit.client.0.smithi016.stdout:3/777: stat d7/d17/d18/d32/d42/d63/deb/d21/d55/dac/le0 0 2024-04-22T15:49:42.117 INFO:tasks.workunit.client.1.smithi027.stdout:6/5: dread f0 [0,4194304] 0 2024-04-22T15:49:42.117 INFO:tasks.workunit.client.0.smithi016.stdout:0/763: rename dc/d13/d14/de5/d65/d30/l3e to dc/d13/d14/de5/d65/d40/dd6/lfa 0 2024-04-22T15:49:42.117 INFO:tasks.workunit.client.0.smithi016.stdout:5/696: dread - df/d15/d1a/d4e/d58/d87/f9e zero size 2024-04-22T15:49:42.118 INFO:tasks.workunit.client.1.smithi027.stdout:6/6: chown f0 13942428 1 2024-04-22T15:49:42.119 INFO:tasks.workunit.client.1.smithi027.stdout:3/15: symlink l3 0 2024-04-22T15:49:42.119 INFO:tasks.workunit.client.1.smithi027.stdout:6/7: fdatasync f0 0 2024-04-22T15:49:42.120 INFO:tasks.workunit.client.0.smithi016.stdout:0/764: stat dc/d13/d14/de5/d65/d40/dbb/fc2 0 2024-04-22T15:49:42.120 INFO:tasks.workunit.client.0.smithi016.stdout:1/835: dwrite d3/f4f [0,4194304] 0 2024-04-22T15:49:42.122 INFO:tasks.workunit.client.1.smithi027.stdout:1/6: rmdir d1 0 2024-04-22T15:49:42.122 INFO:tasks.workunit.client.0.smithi016.stdout:0/765: write dc/d13/d14/d55/fa8 [977525,10520] 0 2024-04-22T15:49:42.123 INFO:tasks.workunit.client.1.smithi027.stdout:2/14: dwrite f0 [0,4194304] 0 2024-04-22T15:49:42.123 INFO:tasks.workunit.client.1.smithi027.stdout:1/7: truncate f0 912879 0 2024-04-22T15:49:42.128 INFO:tasks.workunit.client.1.smithi027.stdout:7/0: mkdir d0 0 2024-04-22T15:49:42.128 INFO:tasks.workunit.client.1.smithi027.stdout:7/1: truncate - no filename 2024-04-22T15:49:42.129 INFO:tasks.workunit.client.1.smithi027.stdout:7/2: chown d0 7 1 2024-04-22T15:49:42.130 INFO:tasks.workunit.client.0.smithi016.stdout:0/766: dwrite dc/d13/d14/d55/fbf [0,4194304] 0 2024-04-22T15:49:42.131 INFO:tasks.workunit.client.1.smithi027.stdout:9/0: dread - no filename 2024-04-22T15:49:42.132 INFO:tasks.workunit.client.0.smithi016.stdout:6/725: rename d3/da/dd/d67/d6c/db1 to d3/da/d12/d13/d38/df5 0 2024-04-22T15:49:42.135 INFO:tasks.workunit.client.1.smithi027.stdout:2/15: dwrite f0 [0,4194304] 0 2024-04-22T15:49:42.145 INFO:tasks.workunit.client.0.smithi016.stdout:3/778: mkdir d7/d17/d18/d32/d3c/d51/df1 0 2024-04-22T15:49:42.149 INFO:tasks.workunit.client.0.smithi016.stdout:3/779: truncate d7/d17/d18/d32/d42/d63/d86/dd8/fd3 333114 0 2024-04-22T15:49:42.149 INFO:tasks.workunit.client.1.smithi027.stdout:0/4: mknod d0/c2 0 2024-04-22T15:49:42.149 INFO:tasks.workunit.client.1.smithi027.stdout:0/5: dread - d0/f1 zero size 2024-04-22T15:49:42.150 INFO:tasks.workunit.client.1.smithi027.stdout:3/16: mknod c4 0 2024-04-22T15:49:42.153 INFO:tasks.workunit.client.1.smithi027.stdout:6/8: mknod c1 0 2024-04-22T15:49:42.154 INFO:tasks.workunit.client.1.smithi027.stdout:5/6: mkdir d0 0 2024-04-22T15:49:42.155 INFO:tasks.workunit.client.1.smithi027.stdout:5/7: rename d0 to d0/d1 22 2024-04-22T15:49:42.157 INFO:tasks.workunit.client.1.smithi027.stdout:8/2: creat f0 x:0 0 0 2024-04-22T15:49:42.158 INFO:tasks.workunit.client.0.smithi016.stdout:5/697: mkdir df/d15/d1a/dd8 0 2024-04-22T15:49:42.159 INFO:tasks.workunit.client.0.smithi016.stdout:2/677: getdents d2/d1a/d7b/dcc/de5 0 2024-04-22T15:49:42.160 INFO:tasks.workunit.client.0.smithi016.stdout:3/780: creat d7/d17/d18/d32/d3c/d47/ff2 x:0 0 0 2024-04-22T15:49:42.161 INFO:tasks.workunit.client.1.smithi027.stdout:2/16: creat f1 x:0 0 0 2024-04-22T15:49:42.161 INFO:tasks.workunit.client.0.smithi016.stdout:2/678: dread - d2/d23/ff6 zero size 2024-04-22T15:49:42.162 INFO:tasks.workunit.client.1.smithi027.stdout:2/17: write f1 [800935,27367] 0 2024-04-22T15:49:42.162 INFO:tasks.workunit.client.0.smithi016.stdout:9/706: creat d9/de/d23/fdc x:0 0 0 2024-04-22T15:49:42.163 INFO:tasks.workunit.client.0.smithi016.stdout:2/679: write d2/d2c/f6b [788135,68952] 0 2024-04-22T15:49:42.164 INFO:tasks.workunit.client.0.smithi016.stdout:7/693: creat d3/dc/d1b/d5a/d4a/ffa x:0 0 0 2024-04-22T15:49:42.165 INFO:tasks.workunit.client.0.smithi016.stdout:0/767: mknod dc/d13/d14/de5/d65/d1f/dd4/df4/cfb 0 2024-04-22T15:49:42.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:41 smithi016 ceph-mon[26620]: pgmap v33: 65 pgs: 65 active+clean; 1.2 GiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 46 MiB/s rd, 155 MiB/s wr, 386 op/s 2024-04-22T15:49:42.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:41 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:49:42.166 INFO:tasks.workunit.client.1.smithi027.stdout:4/6: rename f1 to f2 0 2024-04-22T15:49:42.168 INFO:tasks.workunit.client.0.smithi016.stdout:5/698: dwrite df/d15/d1a/d50/d77/d7a/f80 [0,4194304] 0 2024-04-22T15:49:42.168 INFO:tasks.workunit.client.1.smithi027.stdout:0/6: symlink d0/l3 0 2024-04-22T15:49:42.169 INFO:tasks.workunit.client.1.smithi027.stdout:3/17: stat l2 0 2024-04-22T15:49:42.169 INFO:tasks.workunit.client.1.smithi027.stdout:3/18: truncate - no filename 2024-04-22T15:49:42.170 INFO:tasks.workunit.client.1.smithi027.stdout:3/19: chown l3 0 1 2024-04-22T15:49:42.170 INFO:tasks.workunit.client.1.smithi027.stdout:3/20: truncate - no filename 2024-04-22T15:49:42.171 INFO:tasks.workunit.client.0.smithi016.stdout:8/627: sync 2024-04-22T15:49:42.172 INFO:tasks.workunit.client.0.smithi016.stdout:3/781: rmdir d7/d17/d18/d27 39 2024-04-22T15:49:42.174 INFO:tasks.workunit.client.0.smithi016.stdout:9/707: creat d9/d20/fdd x:0 0 0 2024-04-22T15:49:42.175 INFO:tasks.workunit.client.1.smithi027.stdout:8/3: mkdir d1 0 2024-04-22T15:49:42.175 INFO:tasks.workunit.client.0.smithi016.stdout:7/694: creat d3/dc/d1b/d5a/d4a/d6e/ffb x:0 0 0 2024-04-22T15:49:42.176 INFO:tasks.workunit.client.0.smithi016.stdout:2/680: dwrite d2/de/f40 [0,4194304] 0 2024-04-22T15:49:42.177 INFO:tasks.workunit.client.1.smithi027.stdout:7/3: symlink d0/l1 0 2024-04-22T15:49:42.178 INFO:tasks.workunit.client.1.smithi027.stdout:9/1: getdents . 0 2024-04-22T15:49:42.178 INFO:tasks.workunit.client.1.smithi027.stdout:9/2: dwrite - no filename 2024-04-22T15:49:42.179 INFO:tasks.workunit.client.1.smithi027.stdout:9/3: rmdir - no directory 2024-04-22T15:49:42.179 INFO:tasks.workunit.client.1.smithi027.stdout:9/4: dwrite - no filename 2024-04-22T15:49:42.179 INFO:tasks.workunit.client.0.smithi016.stdout:8/628: chown d2/d82/d44/d66/d1c/f81 18195 1 2024-04-22T15:49:42.181 INFO:tasks.workunit.client.0.smithi016.stdout:6/726: dread d3/da/dd/d1b/f85 [0,4194304] 0 2024-04-22T15:49:42.182 INFO:tasks.workunit.client.1.smithi027.stdout:2/18: unlink f1 0 2024-04-22T15:49:42.183 INFO:tasks.workunit.client.1.smithi027.stdout:4/7: creat f3 x:0 0 0 2024-04-22T15:49:42.183 INFO:tasks.workunit.client.0.smithi016.stdout:7/695: write d3/dc/d1b/d5a/d4a/d6e/ffb [930928,33782] 0 2024-04-22T15:49:42.184 INFO:tasks.workunit.client.0.smithi016.stdout:6/727: fdatasync d3/da/dd/d67/f8f 0 2024-04-22T15:49:42.186 INFO:tasks.workunit.client.1.smithi027.stdout:2/19: dread f0 [0,4194304] 0 2024-04-22T15:49:42.188 INFO:tasks.workunit.client.1.smithi027.stdout:0/7: symlink d0/l4 0 2024-04-22T15:49:42.188 INFO:tasks.workunit.client.0.smithi016.stdout:5/699: dread df/d5b/d5d/f8e [0,4194304] 0 2024-04-22T15:49:42.188 INFO:tasks.workunit.client.1.smithi027.stdout:0/8: fsync d0/f1 0 2024-04-22T15:49:42.189 INFO:tasks.workunit.client.1.smithi027.stdout:0/9: write d0/f1 [81838,2286] 0 2024-04-22T15:49:42.190 INFO:tasks.workunit.client.1.smithi027.stdout:3/21: creat f5 x:0 0 0 2024-04-22T15:49:42.191 INFO:tasks.workunit.client.1.smithi027.stdout:3/22: chown l3 228 1 2024-04-22T15:49:42.193 INFO:tasks.workunit.client.1.smithi027.stdout:5/8: symlink d0/l2 0 2024-04-22T15:49:42.194 INFO:tasks.workunit.client.1.smithi027.stdout:5/9: write - no filename 2024-04-22T15:49:42.194 INFO:tasks.workunit.client.1.smithi027.stdout:5/10: dwrite - no filename 2024-04-22T15:49:42.195 INFO:tasks.workunit.client.1.smithi027.stdout:8/4: fsync f0 0 2024-04-22T15:49:42.195 INFO:tasks.workunit.client.0.smithi016.stdout:3/782: unlink d7/d17/d18/d32/d3c/d47/l90 0 2024-04-22T15:49:42.196 INFO:tasks.workunit.client.0.smithi016.stdout:5/700: dwrite df/d15/f45 [0,4194304] 0 2024-04-22T15:49:42.196 INFO:tasks.workunit.client.1.smithi027.stdout:8/5: dread - f0 zero size 2024-04-22T15:49:42.196 INFO:tasks.workunit.client.0.smithi016.stdout:8/629: mknod d2/d82/d44/d66/d7e/cc4 0 2024-04-22T15:49:42.198 INFO:tasks.workunit.client.1.smithi027.stdout:0/10: dwrite d0/f1 [0,4194304] 0 2024-04-22T15:49:42.198 INFO:tasks.workunit.client.1.smithi027.stdout:7/4: creat d0/f2 x:0 0 0 2024-04-22T15:49:42.198 INFO:tasks.workunit.client.0.smithi016.stdout:8/630: write d2/d48/dad/d6a/f9a [722801,108411] 0 2024-04-22T15:49:42.198 INFO:tasks.workunit.client.0.smithi016.stdout:3/783: truncate d7/d17/d18/d32/d3c/fcc 4403189 0 2024-04-22T15:49:42.199 INFO:tasks.workunit.client.1.smithi027.stdout:8/6: write f0 [313678,108636] 0 2024-04-22T15:49:42.199 INFO:tasks.workunit.client.0.smithi016.stdout:9/708: symlink d9/de/d23/d24/d25/d36/d87/lde 0 2024-04-22T15:49:42.199 INFO:tasks.workunit.client.0.smithi016.stdout:8/631: stat d2/d48/dad/d8f/d96/da5 0 2024-04-22T15:49:42.201 INFO:tasks.workunit.client.1.smithi027.stdout:6/9: truncate f0 1655 0 2024-04-22T15:49:42.205 INFO:tasks.workunit.client.0.smithi016.stdout:6/728: creat d3/da/dd/d67/d6c/ff6 x:0 0 0 2024-04-22T15:49:42.206 INFO:tasks.workunit.client.0.smithi016.stdout:6/729: stat d3/d53/d35/fc2 0 2024-04-22T15:49:42.208 INFO:tasks.workunit.client.0.smithi016.stdout:6/730: read - d3/da/db/d14/d4c/fea zero size 2024-04-22T15:49:42.209 INFO:tasks.workunit.client.1.smithi027.stdout:8/7: dread f0 [0,4194304] 0 2024-04-22T15:49:42.209 INFO:tasks.workunit.client.0.smithi016.stdout:1/836: rename d3/d4/d19/f1d to d3/d4/d3a/d91/db3/f11d 0 2024-04-22T15:49:42.210 INFO:tasks.workunit.client.1.smithi027.stdout:4/8: creat f4 x:0 0 0 2024-04-22T15:49:42.211 INFO:tasks.workunit.client.1.smithi027.stdout:4/9: dread - f2 zero size 2024-04-22T15:49:42.212 INFO:tasks.workunit.client.1.smithi027.stdout:2/20: creat f2 x:0 0 0 2024-04-22T15:49:42.212 INFO:tasks.workunit.client.1.smithi027.stdout:2/21: chown f2 31205304 1 2024-04-22T15:49:42.213 INFO:tasks.workunit.client.1.smithi027.stdout:2/22: chown f0 950612 1 2024-04-22T15:49:42.213 INFO:tasks.workunit.client.1.smithi027.stdout:2/23: chown f0 139914638 1 2024-04-22T15:49:42.214 INFO:tasks.workunit.client.1.smithi027.stdout:1/8: fsync f0 0 2024-04-22T15:49:42.214 INFO:tasks.workunit.client.1.smithi027.stdout:1/9: readlink - no filename 2024-04-22T15:49:42.217 INFO:tasks.workunit.client.1.smithi027.stdout:3/23: symlink l6 0 2024-04-22T15:49:42.217 INFO:tasks.workunit.client.1.smithi027.stdout:3/24: write f5 [697336,118265] 0 2024-04-22T15:49:42.219 INFO:tasks.workunit.client.1.smithi027.stdout:5/11: creat d0/f3 x:0 0 0 2024-04-22T15:49:42.223 INFO:tasks.workunit.client.0.smithi016.stdout:0/768: write dc/d13/d14/de5/d65/f63 [1401253,36296] 0 2024-04-22T15:49:42.225 INFO:tasks.workunit.client.1.smithi027.stdout:3/25: fsync f5 0 2024-04-22T15:49:42.227 INFO:tasks.workunit.client.1.smithi027.stdout:9/5: mkdir d0 0 2024-04-22T15:49:42.227 INFO:tasks.workunit.client.1.smithi027.stdout:9/6: dwrite - no filename 2024-04-22T15:49:42.228 INFO:tasks.workunit.client.0.smithi016.stdout:6/731: rmdir d3/da/db/d44/d68/d95 39 2024-04-22T15:49:42.230 INFO:tasks.workunit.client.0.smithi016.stdout:3/784: dread d7/d17/d18/d32/d3c/d51/f5f [0,4194304] 0 2024-04-22T15:49:42.231 INFO:tasks.workunit.client.0.smithi016.stdout:1/837: mkdir d3/d4/d19/d20/d2b/dd6/d11e 0 2024-04-22T15:49:42.231 INFO:tasks.workunit.client.1.smithi027.stdout:7/5: rmdir d0 39 2024-04-22T15:49:42.232 INFO:tasks.workunit.client.1.smithi027.stdout:3/26: dwrite f5 [0,4194304] 0 2024-04-22T15:49:42.233 INFO:tasks.workunit.client.0.smithi016.stdout:6/732: dread d3/da/dd/fb2 [0,4194304] 0 2024-04-22T15:49:42.238 INFO:tasks.workunit.client.0.smithi016.stdout:5/701: unlink df/d15/d1a/fa5 0 2024-04-22T15:49:42.239 INFO:tasks.workunit.client.0.smithi016.stdout:4/683: sync 2024-04-22T15:49:42.239 INFO:tasks.workunit.client.0.smithi016.stdout:4/684: dread - d7/d22/fcd zero size 2024-04-22T15:49:42.241 INFO:tasks.workunit.client.0.smithi016.stdout:4/685: fdatasync d7/d21/d45/d49/d9b/faf 0 2024-04-22T15:49:42.242 INFO:tasks.workunit.client.0.smithi016.stdout:6/733: dwrite d3/da/f3e [0,4194304] 0 2024-04-22T15:49:42.243 INFO:tasks.workunit.client.0.smithi016.stdout:0/769: mknod dc/d13/d14/de5/d65/d40/d4a/d59/d78/cfc 0 2024-04-22T15:49:42.243 INFO:tasks.workunit.client.0.smithi016.stdout:2/681: write d2/d2c/d97/fc1 [1846528,77591] 0 2024-04-22T15:49:42.244 INFO:tasks.workunit.client.1.smithi027.stdout:2/24: write f0 [2542780,85461] 0 2024-04-22T15:49:42.247 INFO:tasks.workunit.client.0.smithi016.stdout:0/770: fsync dc/d13/d14/de5/d65/d40/f73 0 2024-04-22T15:49:42.247 INFO:tasks.workunit.client.0.smithi016.stdout:6/734: truncate d3/da/d12/d13/dd2/dee/fda 920190 0 2024-04-22T15:49:42.247 INFO:tasks.workunit.client.1.smithi027.stdout:1/10: mkdir d2 0 2024-04-22T15:49:42.254 INFO:tasks.workunit.client.1.smithi027.stdout:0/11: dwrite d0/f1 [4194304,4194304] 0 2024-04-22T15:49:42.256 INFO:tasks.workunit.client.0.smithi016.stdout:1/838: creat d3/d4/d15/d8c/f11f x:0 0 0 2024-04-22T15:49:42.258 INFO:tasks.workunit.client.0.smithi016.stdout:4/686: dread d7/d21/d45/d49/f4d [0,4194304] 0 2024-04-22T15:49:42.263 INFO:tasks.workunit.client.1.smithi027.stdout:0/12: dread d0/f1 [4194304,4194304] 0 2024-04-22T15:49:42.264 INFO:tasks.workunit.client.1.smithi027.stdout:0/13: stat d0/l3 0 2024-04-22T15:49:42.264 INFO:tasks.workunit.client.0.smithi016.stdout:9/709: creat d9/de/fdf x:0 0 0 2024-04-22T15:49:42.265 INFO:tasks.workunit.client.1.smithi027.stdout:0/14: read d0/f1 [2315903,61452] 0 2024-04-22T15:49:42.266 INFO:tasks.workunit.client.0.smithi016.stdout:9/710: write d9/de/d23/fa5 [5111996,109027] 0 2024-04-22T15:49:42.267 INFO:tasks.workunit.client.1.smithi027.stdout:4/10: dwrite f2 [0,4194304] 0 2024-04-22T15:49:42.268 INFO:tasks.workunit.client.1.smithi027.stdout:3/27: mknod c7 0 2024-04-22T15:49:42.269 INFO:tasks.workunit.client.0.smithi016.stdout:8/632: link d2/d82/d44/d66/d79/f8b d2/d82/d44/d66/d7e/fc5 0 2024-04-22T15:49:42.270 INFO:tasks.workunit.client.1.smithi027.stdout:8/8: mknod d1/c2 0 2024-04-22T15:49:42.270 INFO:tasks.workunit.client.1.smithi027.stdout:8/9: stat d1 0 2024-04-22T15:49:42.272 INFO:tasks.workunit.client.0.smithi016.stdout:2/682: creat d2/d23/db8/f100 x:0 0 0 2024-04-22T15:49:42.273 INFO:tasks.workunit.client.1.smithi027.stdout:2/25: unlink f2 0 2024-04-22T15:49:42.275 INFO:tasks.workunit.client.0.smithi016.stdout:7/696: rename d3/dc/d1b/d5a/d9/df/d94/d4c/fd4 to d3/de6/ffc 0 2024-04-22T15:49:42.281 INFO:tasks.workunit.client.0.smithi016.stdout:3/785: creat d7/d17/d18/d27/ff3 x:0 0 0 2024-04-22T15:49:42.282 INFO:tasks.workunit.client.0.smithi016.stdout:1/839: symlink d3/d2e/d9f/l120 0 2024-04-22T15:49:42.284 INFO:tasks.workunit.client.1.smithi027.stdout:9/7: symlink d0/l1 0 2024-04-22T15:49:42.284 INFO:tasks.workunit.client.1.smithi027.stdout:9/8: write - no filename 2024-04-22T15:49:42.287 INFO:tasks.workunit.client.0.smithi016.stdout:9/711: truncate d9/de/d26/f2a 439902 0 2024-04-22T15:49:42.287 INFO:tasks.workunit.client.1.smithi027.stdout:6/10: link c1 c2 0 2024-04-22T15:49:42.288 INFO:tasks.workunit.client.0.smithi016.stdout:5/702: creat df/d5b/db3/fd9 x:0 0 0 2024-04-22T15:49:42.290 INFO:tasks.workunit.client.0.smithi016.stdout:5/703: write df/d5b/d5d/d5f/fd7 [272940,87159] 0 2024-04-22T15:49:42.293 INFO:tasks.workunit.client.1.smithi027.stdout:3/28: unlink c7 0 2024-04-22T15:49:42.295 INFO:tasks.workunit.client.0.smithi016.stdout:2/683: rename d2/d2c/d97/fa4 to d2/d9/f101 0 2024-04-22T15:49:42.296 INFO:tasks.workunit.client.0.smithi016.stdout:2/684: readlink d2/d1a/l56 0 2024-04-22T15:49:42.296 INFO:tasks.workunit.client.1.smithi027.stdout:2/26: symlink l3 0 2024-04-22T15:49:42.296 INFO:tasks.workunit.client.0.smithi016.stdout:2/685: chown d2/d23/d92 33009915 1 2024-04-22T15:49:42.297 INFO:tasks.workunit.client.1.smithi027.stdout:2/27: fdatasync f0 0 2024-04-22T15:49:42.297 INFO:tasks.workunit.client.1.smithi027.stdout:2/28: chown l3 1156919738 1 2024-04-22T15:49:42.298 INFO:tasks.workunit.client.0.smithi016.stdout:7/697: mkdir d3/dc/d9f/da1/dfd 0 2024-04-22T15:49:42.298 INFO:tasks.workunit.client.0.smithi016.stdout:8/633: write d2/d82/d44/d66/d1c/f8c [4009213,43501] 0 2024-04-22T15:49:42.299 INFO:tasks.workunit.client.1.smithi027.stdout:1/11: creat d2/f3 x:0 0 0 2024-04-22T15:49:42.300 INFO:tasks.workunit.client.0.smithi016.stdout:8/634: write d2/d82/d44/d66/d1c/f3d [5594297,88503] 0 2024-04-22T15:49:42.301 INFO:tasks.workunit.client.1.smithi027.stdout:1/12: write d2/f3 [53935,84058] 0 2024-04-22T15:49:42.301 INFO:tasks.workunit.client.1.smithi027.stdout:1/13: chown f0 6265604 1 2024-04-22T15:49:42.303 INFO:tasks.workunit.client.0.smithi016.stdout:3/786: fdatasync d7/d17/fd0 0 2024-04-22T15:49:42.304 INFO:tasks.workunit.client.0.smithi016.stdout:3/787: write d7/d17/d18/d32/d42/d63/deb/d21/d55/f8e [138482,1846] 0 2024-04-22T15:49:42.306 INFO:tasks.workunit.client.0.smithi016.stdout:3/788: dread - d7/d17/d18/d32/d3c/d47/fe3 zero size 2024-04-22T15:49:42.307 INFO:tasks.workunit.client.1.smithi027.stdout:9/9: rename d0/l1 to d0/l2 0 2024-04-22T15:49:42.307 INFO:tasks.workunit.client.1.smithi027.stdout:9/10: dread - no filename 2024-04-22T15:49:42.307 INFO:tasks.workunit.client.1.smithi027.stdout:9/11: dread - no filename 2024-04-22T15:49:42.310 INFO:tasks.workunit.client.1.smithi027.stdout:7/6: link d0/f2 d0/f3 0 2024-04-22T15:49:42.310 INFO:tasks.workunit.client.0.smithi016.stdout:4/687: write d7/d22/f64 [1787128,106059] 0 2024-04-22T15:49:42.310 INFO:tasks.workunit.client.1.smithi027.stdout:1/14: dwrite d2/f3 [0,4194304] 0 2024-04-22T15:49:42.312 INFO:tasks.workunit.client.1.smithi027.stdout:6/11: creat f3 x:0 0 0 2024-04-22T15:49:42.313 INFO:tasks.workunit.client.1.smithi027.stdout:6/12: write f3 [268316,40134] 0 2024-04-22T15:49:42.316 INFO:tasks.workunit.client.0.smithi016.stdout:5/704: rename df/d5b/c84 to df/d5b/d5d/cda 0 2024-04-22T15:49:42.317 INFO:tasks.workunit.client.1.smithi027.stdout:1/15: dread d2/f3 [0,4194304] 0 2024-04-22T15:49:42.317 INFO:tasks.workunit.client.0.smithi016.stdout:2/686: symlink d2/d1a/d4c/d9f/dae/l102 0 2024-04-22T15:49:42.319 INFO:tasks.workunit.client.0.smithi016.stdout:2/687: write d2/d23/d92/d55/ffe [312780,80950] 0 2024-04-22T15:49:42.319 INFO:tasks.workunit.client.0.smithi016.stdout:7/698: mkdir d3/dc/d1b/d5a/d59/d64/dfe 0 2024-04-22T15:49:42.320 INFO:tasks.workunit.client.1.smithi027.stdout:5/12: getdents d0 0 2024-04-22T15:49:42.320 INFO:tasks.workunit.client.0.smithi016.stdout:2/688: dread - d2/d23/d92/fe1 zero size 2024-04-22T15:49:42.320 INFO:tasks.workunit.client.0.smithi016.stdout:7/699: dread - d3/dc/d1b/d5a/d4a/ffa zero size 2024-04-22T15:49:42.322 INFO:tasks.workunit.client.1.smithi027.stdout:5/13: truncate d0/f3 522025 0 2024-04-22T15:49:42.322 INFO:tasks.workunit.client.0.smithi016.stdout:7/700: truncate d3/dc/d9f/da1/fc3 641468 0 2024-04-22T15:49:42.323 INFO:tasks.workunit.client.1.smithi027.stdout:9/12: symlink d0/l3 0 2024-04-22T15:49:42.324 INFO:tasks.workunit.client.1.smithi027.stdout:9/13: rename d0 to d0/d4 22 2024-04-22T15:49:42.324 INFO:tasks.workunit.client.1.smithi027.stdout:9/14: dwrite - no filename 2024-04-22T15:49:42.324 INFO:tasks.workunit.client.1.smithi027.stdout:7/7: rmdir d0 39 2024-04-22T15:49:42.325 INFO:tasks.workunit.client.1.smithi027.stdout:5/14: truncate d0/f3 576009 0 2024-04-22T15:49:42.326 INFO:tasks.workunit.client.1.smithi027.stdout:1/16: dwrite f0 [0,4194304] 0 2024-04-22T15:49:42.328 INFO:tasks.workunit.client.0.smithi016.stdout:8/635: dwrite d2/fa2 [0,4194304] 0 2024-04-22T15:49:42.330 INFO:tasks.workunit.client.0.smithi016.stdout:3/789: rmdir d7/d17 39 2024-04-22T15:49:42.331 INFO:tasks.workunit.client.0.smithi016.stdout:8/636: read d2/d10/f41 [28278,101344] 0 2024-04-22T15:49:42.332 INFO:tasks.workunit.client.0.smithi016.stdout:1/840: mknod d3/d4/d3a/d6b/d77/de2/d116/c121 0 2024-04-22T15:49:42.333 INFO:tasks.workunit.client.1.smithi027.stdout:6/13: fsync f3 0 2024-04-22T15:49:42.338 INFO:tasks.workunit.client.1.smithi027.stdout:6/14: dread f3 [0,4194304] 0 2024-04-22T15:49:42.339 INFO:tasks.workunit.client.0.smithi016.stdout:2/689: mknod d2/d23/d92/d55/c103 0 2024-04-22T15:49:42.342 INFO:tasks.workunit.client.0.smithi016.stdout:6/735: creat d3/da/db/d44/d68/d95/ff7 x:0 0 0 2024-04-22T15:49:42.344 INFO:tasks.workunit.client.0.smithi016.stdout:7/701: mknod d3/dc/d1b/d5a/d4a/d6e/cff 0 2024-04-22T15:49:42.345 INFO:tasks.workunit.client.0.smithi016.stdout:7/702: dread d3/dc/d1b/d5a/fa3 [0,4194304] 0 2024-04-22T15:49:42.346 INFO:tasks.workunit.client.1.smithi027.stdout:0/15: getdents d0 0 2024-04-22T15:49:42.347 INFO:tasks.workunit.client.1.smithi027.stdout:9/15: symlink d0/l5 0 2024-04-22T15:49:42.347 INFO:tasks.workunit.client.1.smithi027.stdout:9/16: dread - no filename 2024-04-22T15:49:42.347 INFO:tasks.workunit.client.0.smithi016.stdout:7/703: fdatasync d3/dc/d1b/d5a/d59/d64/fa0 0 2024-04-22T15:49:42.349 INFO:tasks.workunit.client.0.smithi016.stdout:7/704: readlink d3/dc/d1b/d5a/d69/l8e 0 2024-04-22T15:49:42.349 INFO:tasks.workunit.client.1.smithi027.stdout:1/17: creat d2/f4 x:0 0 0 2024-04-22T15:49:42.349 INFO:tasks.workunit.client.1.smithi027.stdout:1/18: truncate d2/f4 709063 0 2024-04-22T15:49:42.350 INFO:tasks.workunit.client.1.smithi027.stdout:4/11: getdents . 0 2024-04-22T15:49:42.350 INFO:tasks.workunit.client.0.smithi016.stdout:7/705: truncate d3/dc/d33/f7f 1674755 0 2024-04-22T15:49:42.351 INFO:tasks.workunit.client.0.smithi016.stdout:8/637: dread d2/d30/f73 [0,4194304] 0 2024-04-22T15:49:42.352 INFO:tasks.workunit.client.0.smithi016.stdout:7/706: chown d3/dc/d1b/d31/cde 9 1 2024-04-22T15:49:42.352 INFO:tasks.workunit.client.0.smithi016.stdout:1/841: read - d3/d4/d45/f58 zero size 2024-04-22T15:49:42.353 INFO:tasks.workunit.client.0.smithi016.stdout:7/707: write d3/dc/d9f/fae [606554,2071] 0 2024-04-22T15:49:42.354 INFO:tasks.workunit.client.1.smithi027.stdout:6/15: rename f3 to f4 0 2024-04-22T15:49:42.357 INFO:tasks.workunit.client.1.smithi027.stdout:6/16: dread f4 [0,4194304] 0 2024-04-22T15:49:42.358 INFO:tasks.workunit.client.1.smithi027.stdout:1/19: dwrite d2/f4 [0,4194304] 0 2024-04-22T15:49:42.359 INFO:tasks.workunit.client.1.smithi027.stdout:1/20: readlink - no filename 2024-04-22T15:49:42.360 INFO:tasks.workunit.client.1.smithi027.stdout:0/16: fsync d0/f1 0 2024-04-22T15:49:42.361 INFO:tasks.workunit.client.1.smithi027.stdout:0/17: stat d0/c2 0 2024-04-22T15:49:42.364 INFO:tasks.workunit.client.1.smithi027.stdout:9/17: symlink d0/l6 0 2024-04-22T15:49:42.364 INFO:tasks.workunit.client.1.smithi027.stdout:9/18: read - no filename 2024-04-22T15:49:42.364 INFO:tasks.workunit.client.1.smithi027.stdout:1/21: dread d2/f4 [0,4194304] 0 2024-04-22T15:49:42.364 INFO:tasks.workunit.client.1.smithi027.stdout:9/19: rename d0 to d0/d7 22 2024-04-22T15:49:42.368 INFO:tasks.workunit.client.0.smithi016.stdout:5/705: write df/d15/d1a/d4e/f69 [506183,96986] 0 2024-04-22T15:49:42.368 INFO:tasks.workunit.client.0.smithi016.stdout:5/706: fdatasync df/d15/d1a/d28/f2b 0 2024-04-22T15:49:42.369 INFO:tasks.workunit.client.1.smithi027.stdout:4/12: mkdir d5 0 2024-04-22T15:49:42.371 INFO:tasks.workunit.client.1.smithi027.stdout:3/29: getdents . 0 2024-04-22T15:49:42.373 INFO:tasks.workunit.client.0.smithi016.stdout:1/842: mknod d3/d2e/d80/dbb/c122 0 2024-04-22T15:49:42.374 INFO:tasks.workunit.client.1.smithi027.stdout:7/8: dwrite d0/f2 [0,4194304] 0 2024-04-22T15:49:42.375 INFO:tasks.workunit.client.0.smithi016.stdout:2/690: dwrite d2/f5 [0,4194304] 0 2024-04-22T15:49:42.376 INFO:tasks.workunit.client.0.smithi016.stdout:9/712: getdents d9/de/d23/d24/d25/da6/d4f 0 2024-04-22T15:49:42.377 INFO:tasks.workunit.client.0.smithi016.stdout:1/843: write d3/d4/d19/d20/f48 [5051567,95968] 0 2024-04-22T15:49:42.378 INFO:tasks.workunit.client.1.smithi027.stdout:0/18: rename d0/c2 to d0/c5 0 2024-04-22T15:49:42.380 INFO:tasks.workunit.client.1.smithi027.stdout:1/22: symlink d2/l5 0 2024-04-22T15:49:42.380 INFO:tasks.workunit.client.1.smithi027.stdout:4/13: dwrite f2 [0,4194304] 0 2024-04-22T15:49:42.381 INFO:tasks.workunit.client.0.smithi016.stdout:3/790: mknod d7/d17/d18/cf4 0 2024-04-22T15:49:42.383 INFO:tasks.workunit.client.1.smithi027.stdout:9/20: mkdir d0/d8 0 2024-04-22T15:49:42.383 INFO:tasks.workunit.client.1.smithi027.stdout:9/21: write - no filename 2024-04-22T15:49:42.389 INFO:tasks.workunit.client.0.smithi016.stdout:5/707: symlink df/d90/dad/ldb 0 2024-04-22T15:49:42.391 INFO:tasks.workunit.client.1.smithi027.stdout:4/14: dwrite f4 [0,4194304] 0 2024-04-22T15:49:42.392 INFO:tasks.workunit.client.1.smithi027.stdout:4/15: chown f2 1 1 2024-04-22T15:49:42.393 INFO:tasks.workunit.client.0.smithi016.stdout:3/791: dwrite d7/d17/d18/d32/d8a/fa8 [0,4194304] 0 2024-04-22T15:49:42.393 INFO:tasks.workunit.client.1.smithi027.stdout:3/30: symlink l8 0 2024-04-22T15:49:42.395 INFO:tasks.workunit.client.0.smithi016.stdout:7/708: symlink d3/dc/d1b/d5a/d69/def/l100 0 2024-04-22T15:49:42.397 INFO:tasks.workunit.client.1.smithi027.stdout:7/9: symlink d0/l4 0 2024-04-22T15:49:42.399 INFO:tasks.workunit.client.0.smithi016.stdout:0/771: sync 2024-04-22T15:49:42.399 INFO:tasks.workunit.client.0.smithi016.stdout:4/688: sync 2024-04-22T15:49:42.399 INFO:tasks.workunit.client.0.smithi016.stdout:2/691: symlink d2/d2c/l104 0 2024-04-22T15:49:42.402 INFO:tasks.workunit.client.0.smithi016.stdout:9/713: unlink d9/d57/d91/dc1/da2/fd4 0 2024-04-22T15:49:42.403 INFO:tasks.workunit.client.0.smithi016.stdout:4/689: write d7/d24/d51/fcf [871863,61908] 0 2024-04-22T15:49:42.404 INFO:tasks.workunit.client.1.smithi027.stdout:0/19: chown d0/c5 93049968 1 2024-04-22T15:49:42.405 INFO:tasks.workunit.client.0.smithi016.stdout:1/844: dread d3/d4/d3a/d6b/fc1 [0,4194304] 0 2024-04-22T15:49:42.405 INFO:tasks.workunit.client.1.smithi027.stdout:4/16: write f2 [4338478,52597] 0 2024-04-22T15:49:42.406 INFO:tasks.workunit.client.0.smithi016.stdout:5/708: dwrite df/d5b/d70/f8d [0,4194304] 0 2024-04-22T15:49:42.406 INFO:tasks.workunit.client.1.smithi027.stdout:1/23: write d2/f4 [3677650,20319] 0 2024-04-22T15:49:42.407 INFO:tasks.workunit.client.0.smithi016.stdout:0/772: write dc/d13/d14/de5/d65/d30/d96/fae [3573758,47324] 0 2024-04-22T15:49:42.407 INFO:tasks.workunit.client.0.smithi016.stdout:8/638: write d2/d82/f38 [326348,70727] 0 2024-04-22T15:49:42.408 INFO:tasks.workunit.client.1.smithi027.stdout:9/22: readlink d0/l2 0 2024-04-22T15:49:42.408 INFO:tasks.workunit.client.1.smithi027.stdout:9/23: truncate - no filename 2024-04-22T15:49:42.408 INFO:tasks.workunit.client.0.smithi016.stdout:5/709: chown df/d15 166 1 2024-04-22T15:49:42.408 INFO:tasks.workunit.client.0.smithi016.stdout:6/736: getdents d3/da/db/d14/d4c 0 2024-04-22T15:49:42.409 INFO:tasks.workunit.client.0.smithi016.stdout:4/690: stat d7/d21/d45/l82 0 2024-04-22T15:49:42.410 INFO:tasks.workunit.client.0.smithi016.stdout:5/710: stat df/f11 0 2024-04-22T15:49:42.410 INFO:tasks.workunit.client.0.smithi016.stdout:8/639: stat d2/d48/f55 0 2024-04-22T15:49:42.414 INFO:tasks.workunit.client.1.smithi027.stdout:3/31: symlink l9 0 2024-04-22T15:49:42.414 INFO:tasks.workunit.client.0.smithi016.stdout:3/792: unlink d7/d17/d18/d32/d42/d63/d8f/ldc 0 2024-04-22T15:49:42.416 INFO:tasks.workunit.client.0.smithi016.stdout:3/793: chown d7/d17/d18/ce6 104339275 1 2024-04-22T15:49:42.419 INFO:tasks.workunit.client.1.smithi027.stdout:7/10: creat d0/f5 x:0 0 0 2024-04-22T15:49:42.420 INFO:tasks.workunit.client.0.smithi016.stdout:7/709: read d3/dc/d1b/d5a/f17 [3783226,81119] 0 2024-04-22T15:49:42.420 INFO:tasks.workunit.client.1.smithi027.stdout:0/20: mkdir d0/d6 0 2024-04-22T15:49:42.422 INFO:tasks.workunit.client.0.smithi016.stdout:4/691: dwrite d7/d4e/d2f/d6f/fa8 [0,4194304] 0 2024-04-22T15:49:42.423 INFO:tasks.workunit.client.0.smithi016.stdout:9/714: unlink d9/d20/dac/dae/fd0 0 2024-04-22T15:49:42.426 INFO:tasks.workunit.client.0.smithi016.stdout:2/692: rename d2/d1a/d4c/l8b to d2/d23/db8/dd4/l105 0 2024-04-22T15:49:42.427 INFO:tasks.workunit.client.1.smithi027.stdout:1/24: read d2/f3 [3766596,56874] 0 2024-04-22T15:49:42.428 INFO:tasks.workunit.client.0.smithi016.stdout:2/693: write d2/d23/ff6 [801215,14204] 0 2024-04-22T15:49:42.430 INFO:tasks.workunit.client.0.smithi016.stdout:1/845: readlink d3/d4/d3a/l56 0 2024-04-22T15:49:42.432 INFO:tasks.workunit.client.0.smithi016.stdout:9/715: dwrite d9/d5d/fc3 [0,4194304] 0 2024-04-22T15:49:42.432 INFO:tasks.workunit.client.0.smithi016.stdout:8/640: fsync d2/d82/d56/f90 0 2024-04-22T15:49:42.434 INFO:tasks.workunit.client.0.smithi016.stdout:9/716: write f2 [252765,69643] 0 2024-04-22T15:49:42.434 INFO:tasks.workunit.client.1.smithi027.stdout:3/32: symlink la 0 2024-04-22T15:49:42.434 INFO:tasks.workunit.client.1.smithi027.stdout:3/33: stat l6 0 2024-04-22T15:49:42.435 INFO:tasks.workunit.client.0.smithi016.stdout:2/694: dread d2/d1a/d7b/fdc [0,4194304] 0 2024-04-22T15:49:42.437 INFO:tasks.workunit.client.1.smithi027.stdout:6/17: getdents . 0 2024-04-22T15:49:42.442 INFO:tasks.workunit.client.0.smithi016.stdout:0/773: creat dc/d13/d14/de5/d65/d40/d4a/db2/ffd x:0 0 0 2024-04-22T15:49:42.442 INFO:tasks.workunit.client.1.smithi027.stdout:7/11: creat d0/f6 x:0 0 0 2024-04-22T15:49:42.442 INFO:tasks.workunit.client.1.smithi027.stdout:3/34: dread f5 [0,4194304] 0 2024-04-22T15:49:42.442 INFO:tasks.workunit.client.1.smithi027.stdout:7/12: stat d0/f5 0 2024-04-22T15:49:42.442 INFO:tasks.workunit.client.1.smithi027.stdout:7/13: write d0/f2 [4041364,123908] 0 2024-04-22T15:49:42.443 INFO:tasks.workunit.client.0.smithi016.stdout:2/695: read d2/de/d2a/f42 [3635882,62808] 0 2024-04-22T15:49:42.443 INFO:tasks.workunit.client.0.smithi016.stdout:5/711: dwrite df/d15/d1a/f1e [0,4194304] 0 2024-04-22T15:49:42.444 INFO:tasks.workunit.client.1.smithi027.stdout:4/17: getdents d5 0 2024-04-22T15:49:42.450 INFO:tasks.workunit.client.0.smithi016.stdout:5/712: dread df/d5b/d70/fc3 [0,4194304] 0 2024-04-22T15:49:42.451 INFO:tasks.workunit.client.0.smithi016.stdout:2/696: dwrite d2/d1a/d4c/d77/dea/ffc [0,4194304] 0 2024-04-22T15:49:42.451 INFO:tasks.workunit.client.0.smithi016.stdout:3/794: read d7/f5a [122566,37595] 0 2024-04-22T15:49:42.452 INFO:tasks.workunit.client.1.smithi027.stdout:1/25: rmdir d2 39 2024-04-22T15:49:42.452 INFO:tasks.workunit.client.0.smithi016.stdout:2/697: chown d2/d4d/d64/cb0 11347375 1 2024-04-22T15:49:42.453 INFO:tasks.workunit.client.0.smithi016.stdout:5/713: chown df/d15/d1a/d50/d77/d7d/d8b/fcf 3671 1 2024-04-22T15:49:42.454 INFO:tasks.workunit.client.0.smithi016.stdout:2/698: write d2/de/f84 [662441,96121] 0 2024-04-22T15:49:42.455 INFO:tasks.workunit.client.1.smithi027.stdout:9/24: creat d0/d8/f9 x:0 0 0 2024-04-22T15:49:42.461 INFO:tasks.workunit.client.1.smithi027.stdout:6/18: mkdir d5 0 2024-04-22T15:49:42.462 INFO:tasks.workunit.client.1.smithi027.stdout:3/35: symlink lb 0 2024-04-22T15:49:42.463 INFO:tasks.workunit.client.0.smithi016.stdout:1/846: truncate d3/d28/f98 4875022 0 2024-04-22T15:49:42.466 INFO:tasks.workunit.client.1.smithi027.stdout:1/26: fsync d2/f4 0 2024-04-22T15:49:42.471 INFO:tasks.workunit.client.0.smithi016.stdout:0/774: creat dc/d13/d6f/d8b/ffe x:0 0 0 2024-04-22T15:49:42.471 INFO:tasks.workunit.client.1.smithi027.stdout:0/21: link d0/c5 d0/c7 0 2024-04-22T15:49:42.471 INFO:tasks.workunit.client.1.smithi027.stdout:0/22: readlink d0/l4 0 2024-04-22T15:49:42.471 INFO:tasks.workunit.client.1.smithi027.stdout:0/23: write d0/f1 [4468080,113837] 0 2024-04-22T15:49:42.472 INFO:tasks.workunit.client.1.smithi027.stdout:4/18: link f0 d5/f6 0 2024-04-22T15:49:42.472 INFO:tasks.workunit.client.1.smithi027.stdout:0/24: write d0/f1 [6500587,73655] 0 2024-04-22T15:49:42.476 INFO:tasks.workunit.client.1.smithi027.stdout:1/27: creat d2/f6 x:0 0 0 2024-04-22T15:49:42.477 INFO:tasks.workunit.client.0.smithi016.stdout:0/775: dwrite dc/d13/d14/de5/d65/d1f/f53 [4194304,4194304] 0 2024-04-22T15:49:42.477 INFO:tasks.workunit.client.1.smithi027.stdout:6/19: symlink d5/l6 0 2024-04-22T15:49:42.477 INFO:tasks.workunit.client.1.smithi027.stdout:9/25: fsync d0/d8/f9 0 2024-04-22T15:49:42.479 INFO:tasks.workunit.client.1.smithi027.stdout:6/20: dread f4 [0,4194304] 0 2024-04-22T15:49:42.480 INFO:tasks.workunit.client.1.smithi027.stdout:9/26: dread - d0/d8/f9 zero size 2024-04-22T15:49:42.483 INFO:tasks.workunit.client.0.smithi016.stdout:5/714: fdatasync df/f9a 0 2024-04-22T15:49:42.486 INFO:tasks.workunit.client.1.smithi027.stdout:0/25: creat d0/f8 x:0 0 0 2024-04-22T15:49:42.487 INFO:tasks.workunit.client.1.smithi027.stdout:1/28: dwrite d2/f6 [0,4194304] 0 2024-04-22T15:49:42.491 INFO:tasks.workunit.client.1.smithi027.stdout:1/29: readlink d2/l5 0 2024-04-22T15:49:42.493 INFO:tasks.workunit.client.0.smithi016.stdout:8/641: write d2/d82/d44/fba [624446,78035] 0 2024-04-22T15:49:42.497 INFO:tasks.workunit.client.1.smithi027.stdout:9/27: dwrite d0/d8/f9 [0,4194304] 0 2024-04-22T15:49:42.497 INFO:tasks.workunit.client.1.smithi027.stdout:4/19: mkdir d5/d7 0 2024-04-22T15:49:42.499 INFO:tasks.workunit.client.1.smithi027.stdout:4/20: write f3 [476418,71111] 0 2024-04-22T15:49:42.499 INFO:tasks.workunit.client.0.smithi016.stdout:6/737: truncate d3/da/dd/d1b/f85 4673753 0 2024-04-22T15:49:42.502 INFO:tasks.workunit.client.1.smithi027.stdout:0/26: dwrite d0/f1 [0,4194304] 0 2024-04-22T15:49:42.504 INFO:tasks.workunit.client.0.smithi016.stdout:9/717: mknod d9/de/d23/d24/d25/da6/ce0 0 2024-04-22T15:49:42.507 INFO:tasks.workunit.client.0.smithi016.stdout:6/738: dread d3/da/d12/d13/f27 [0,4194304] 0 2024-04-22T15:49:42.510 INFO:tasks.workunit.client.1.smithi027.stdout:4/21: dwrite f2 [4194304,4194304] 0 2024-04-22T15:49:42.510 INFO:tasks.workunit.client.1.smithi027.stdout:1/30: unlink d2/l5 0 2024-04-22T15:49:42.511 INFO:tasks.workunit.client.1.smithi027.stdout:4/22: chown f0 61600665 1 2024-04-22T15:49:42.512 INFO:tasks.workunit.client.1.smithi027.stdout:9/28: mknod d0/d8/ca 0 2024-04-22T15:49:42.515 INFO:tasks.workunit.client.0.smithi016.stdout:7/710: creat d3/dc/d1b/d5a/d9/df/d94/d4c/f101 x:0 0 0 2024-04-22T15:49:42.516 INFO:tasks.workunit.client.0.smithi016.stdout:6/739: dwrite d3/da/db/d44/d68/f6a [0,4194304] 0 2024-04-22T15:49:42.521 INFO:tasks.workunit.client.1.smithi027.stdout:1/31: creat d2/f7 x:0 0 0 2024-04-22T15:49:42.522 INFO:tasks.workunit.client.1.smithi027.stdout:6/21: write f0 [1026702,68571] 0 2024-04-22T15:49:42.523 INFO:tasks.workunit.client.0.smithi016.stdout:7/711: dwrite d3/dc/d33/f44 [4194304,4194304] 0 2024-04-22T15:49:42.524 INFO:tasks.workunit.client.0.smithi016.stdout:7/712: stat d3/dc/d1b/d31/f48 0 2024-04-22T15:49:42.529 INFO:tasks.workunit.client.0.smithi016.stdout:5/715: symlink df/d15/d1a/d50/d77/d7d/d8b/d8c/ldc 0 2024-04-22T15:49:42.532 INFO:tasks.workunit.client.0.smithi016.stdout:2/699: rename d2/cc2 to d2/d23/c106 0 2024-04-22T15:49:42.534 INFO:tasks.workunit.client.1.smithi027.stdout:9/29: creat d0/fb x:0 0 0 2024-04-22T15:49:42.536 INFO:tasks.workunit.client.1.smithi027.stdout:9/30: fdatasync d0/fb 0 2024-04-22T15:49:42.536 INFO:tasks.workunit.client.0.smithi016.stdout:8/642: symlink d2/d10/d47/d67/lc6 0 2024-04-22T15:49:42.537 INFO:tasks.workunit.client.0.smithi016.stdout:8/643: chown d2/d10/d47/c84 41354370 1 2024-04-22T15:49:42.538 INFO:tasks.workunit.client.1.smithi027.stdout:4/23: mknod d5/d7/c8 0 2024-04-22T15:49:42.538 INFO:tasks.workunit.client.0.smithi016.stdout:8/644: write d2/f8 [3643682,55220] 0 2024-04-22T15:49:42.538 INFO:tasks.workunit.client.1.smithi027.stdout:1/32: dwrite f0 [0,4194304] 0 2024-04-22T15:49:42.543 INFO:tasks.workunit.client.0.smithi016.stdout:9/718: symlink d9/d20/dd7/le1 0 2024-04-22T15:49:42.547 INFO:tasks.workunit.client.0.smithi016.stdout:8/645: dwrite d2/d10/d43/f4b [0,4194304] 0 2024-04-22T15:49:42.547 INFO:tasks.workunit.client.1.smithi027.stdout:1/33: fsync d2/f7 0 2024-04-22T15:49:42.548 INFO:tasks.workunit.client.1.smithi027.stdout:1/34: write d2/f6 [756314,113559] 0 2024-04-22T15:49:42.553 INFO:tasks.workunit.client.0.smithi016.stdout:6/740: mknod d3/da/db/d14/d16/cf8 0 2024-04-22T15:49:42.554 INFO:tasks.workunit.client.0.smithi016.stdout:6/741: chown d3/da/db/d44/d68/d95/ff7 2652275 1 2024-04-22T15:49:42.558 INFO:tasks.workunit.client.0.smithi016.stdout:4/692: getdents d7/d21/d42/d59/d88/d9d/dad/dac 0 2024-04-22T15:49:42.559 INFO:tasks.workunit.client.0.smithi016.stdout:4/693: write d7/d22/f73 [4882556,53085] 0 2024-04-22T15:49:42.560 INFO:tasks.workunit.client.0.smithi016.stdout:4/694: readlink d7/d4e/d2f/d50/ld5 0 2024-04-22T15:49:42.561 INFO:tasks.workunit.client.0.smithi016.stdout:5/716: unlink df/f10 0 2024-04-22T15:49:42.562 INFO:tasks.workunit.client.0.smithi016.stdout:5/717: truncate df/d15/fa9 639261 0 2024-04-22T15:49:42.563 INFO:tasks.workunit.client.0.smithi016.stdout:5/718: fdatasync df/d22/f3b 0 2024-04-22T15:49:42.563 INFO:tasks.workunit.client.0.smithi016.stdout:5/719: readlink df/d15/d1a/d28/l79 0 2024-04-22T15:49:42.567 INFO:tasks.workunit.client.0.smithi016.stdout:5/720: dread df/d15/f45 [0,4194304] 0 2024-04-22T15:49:42.571 INFO:tasks.workunit.client.0.smithi016.stdout:9/719: rename d9/d57/da7/fd6 to d9/de/d23/d24/d25/fe2 0 2024-04-22T15:49:42.571 INFO:tasks.workunit.client.0.smithi016.stdout:0/776: truncate dc/d13/d14/de5/d65/d1f/fcf 9177853 0 2024-04-22T15:49:42.574 INFO:tasks.workunit.client.0.smithi016.stdout:3/795: getdents d7/d17/d18/d32 0 2024-04-22T15:49:42.575 INFO:tasks.workunit.client.0.smithi016.stdout:3/796: chown d7/d17/d18/d27/c81 267753 1 2024-04-22T15:49:42.576 INFO:tasks.workunit.client.0.smithi016.stdout:6/742: creat d3/da/d12/d13/d38/d84/ff9 x:0 0 0 2024-04-22T15:49:42.580 INFO:tasks.workunit.client.0.smithi016.stdout:1/847: link d3/d4/d19/d20/d2b/l11b d3/d4/d3a/d6b/d77/de2/d116/l123 0 2024-04-22T15:49:42.582 INFO:tasks.workunit.client.0.smithi016.stdout:9/720: symlink d9/d57/da7/daa/le3 0 2024-04-22T15:49:42.586 INFO:tasks.workunit.client.0.smithi016.stdout:3/797: fsync d7/d17/d18/d27/f6e 0 2024-04-22T15:49:42.588 INFO:tasks.workunit.client.0.smithi016.stdout:4/695: symlink d7/d21/d42/d59/le3 0 2024-04-22T15:49:42.589 INFO:tasks.workunit.client.0.smithi016.stdout:4/696: stat d7/d21/d42/cce 0 2024-04-22T15:49:42.590 INFO:tasks.workunit.client.0.smithi016.stdout:4/697: read - d7/d21/d45/d49/d9b/fdf zero size 2024-04-22T15:49:42.591 INFO:tasks.workunit.client.0.smithi016.stdout:6/743: dread d3/da/d12/d13/dd2/dee/f77 [0,4194304] 0 2024-04-22T15:49:42.592 INFO:tasks.workunit.client.0.smithi016.stdout:6/744: write d3/da/dd/d1b/f62 [4559401,51346] 0 2024-04-22T15:49:42.595 INFO:tasks.workunit.client.0.smithi016.stdout:9/721: read d9/de/d23/d24/f53 [322168,72895] 0 2024-04-22T15:49:42.595 INFO:tasks.workunit.client.0.smithi016.stdout:8/646: link d2/d10/d47/d67/lc6 d2/d77/lc7 0 2024-04-22T15:49:42.596 INFO:tasks.workunit.client.0.smithi016.stdout:8/647: read d2/d10/f41 [78896,102857] 0 2024-04-22T15:49:42.605 INFO:tasks.workunit.client.0.smithi016.stdout:3/798: symlink d7/d17/d18/d32/d3c/d51/lf5 0 2024-04-22T15:49:42.606 INFO:tasks.workunit.client.0.smithi016.stdout:3/799: dread - d7/d17/d18/d32/d42/d63/d86/fdf zero size 2024-04-22T15:49:42.607 INFO:tasks.workunit.client.0.smithi016.stdout:3/800: chown d7/d17/d18/d32/l40 8053 1 2024-04-22T15:49:42.609 INFO:tasks.workunit.client.0.smithi016.stdout:5/721: creat df/d5b/fdd x:0 0 0 2024-04-22T15:49:42.611 INFO:tasks.workunit.client.0.smithi016.stdout:8/648: dread d2/d82/f7b [0,4194304] 0 2024-04-22T15:49:42.613 INFO:tasks.workunit.client.0.smithi016.stdout:5/722: dread df/d15/d1a/f78 [0,4194304] 0 2024-04-22T15:49:42.613 INFO:tasks.workunit.client.0.smithi016.stdout:6/745: mkdir d3/da/dd/d67/d6c/db9/dfa 0 2024-04-22T15:49:42.617 INFO:tasks.workunit.client.0.smithi016.stdout:5/723: dread df/d22/f3b [0,4194304] 0 2024-04-22T15:49:42.617 INFO:tasks.workunit.client.0.smithi016.stdout:5/724: fdatasync df/d15/d1a/f78 0 2024-04-22T15:49:42.618 INFO:tasks.workunit.client.0.smithi016.stdout:9/722: rename d9/de/d23/d24 to d9/d57/d91/de4 0 2024-04-22T15:49:42.625 INFO:tasks.workunit.client.0.smithi016.stdout:8/649: unlink d2/d82/d44/fba 0 2024-04-22T15:49:42.626 INFO:tasks.workunit.client.0.smithi016.stdout:8/650: write d2/d82/d44/d66/d1c/f81 [700695,27845] 0 2024-04-22T15:49:42.628 INFO:tasks.workunit.client.0.smithi016.stdout:6/746: stat d3/da/d12/d13/f76 0 2024-04-22T15:49:42.630 INFO:tasks.workunit.client.0.smithi016.stdout:6/747: read d3/da/dd/d1b/f62 [164759,4337] 0 2024-04-22T15:49:42.632 INFO:tasks.workunit.client.0.smithi016.stdout:3/801: rename d7/d17/d18/d27/f6e to d7/d17/d18/d32/d42/d63/d8f/ff6 0 2024-04-22T15:49:42.633 INFO:tasks.workunit.client.0.smithi016.stdout:0/777: truncate dc/d13/d14/d26/f46 250475 0 2024-04-22T15:49:42.634 INFO:tasks.workunit.client.0.smithi016.stdout:9/723: mknod d9/d57/da7/daa/ce5 0 2024-04-22T15:49:42.635 INFO:tasks.workunit.client.0.smithi016.stdout:9/724: read - d9/de/d23/f9c zero size 2024-04-22T15:49:42.635 INFO:tasks.workunit.client.0.smithi016.stdout:0/778: truncate dc/d13/d14/de5/d65/d30/d75/fb9 214035 0 2024-04-22T15:49:42.649 INFO:tasks.workunit.client.0.smithi016.stdout:8/651: dwrite d2/d82/d44/d66/f3c [0,4194304] 0 2024-04-22T15:49:42.650 INFO:tasks.workunit.client.0.smithi016.stdout:6/748: mknod d3/da/db/d14/d4c/cfb 0 2024-04-22T15:49:42.651 INFO:tasks.workunit.client.0.smithi016.stdout:6/749: fdatasync d3/d53/d35/fa2 0 2024-04-22T15:49:42.654 INFO:tasks.workunit.client.0.smithi016.stdout:3/802: dread d7/d17/d18/d32/d42/d63/deb/f85 [0,4194304] 0 2024-04-22T15:49:42.656 INFO:tasks.workunit.client.0.smithi016.stdout:9/725: truncate d9/f3b 41651 0 2024-04-22T15:49:42.657 INFO:tasks.workunit.client.0.smithi016.stdout:4/698: write d7/f16 [324526,14414] 0 2024-04-22T15:49:42.659 INFO:tasks.workunit.client.0.smithi016.stdout:0/779: mknod dc/d13/d14/de5/d65/d1f/cff 0 2024-04-22T15:49:42.662 INFO:tasks.workunit.client.0.smithi016.stdout:0/780: read dc/d13/d14/de5/d65/fec [34677,10038] 0 2024-04-22T15:49:42.663 INFO:tasks.workunit.client.0.smithi016.stdout:5/725: symlink df/d5b/d5d/d5f/lde 0 2024-04-22T15:49:42.664 INFO:tasks.workunit.client.0.smithi016.stdout:4/699: dwrite d7/d3c/f99 [0,4194304] 0 2024-04-22T15:49:42.665 INFO:tasks.workunit.client.0.smithi016.stdout:0/781: write dc/d13/d14/d55/f93 [302653,65317] 0 2024-04-22T15:49:42.666 INFO:tasks.workunit.client.0.smithi016.stdout:8/652: write d2/d82/f7b [884270,21716] 0 2024-04-22T15:49:42.666 INFO:tasks.workunit.client.0.smithi016.stdout:5/726: chown df/d15/d1a/d50/d77/f61 168 1 2024-04-22T15:49:42.667 INFO:tasks.workunit.client.0.smithi016.stdout:3/803: rmdir d7/d17/d18/d32/d42 39 2024-04-22T15:49:42.668 INFO:tasks.workunit.client.0.smithi016.stdout:9/726: rmdir d9/de/d23 39 2024-04-22T15:49:42.671 INFO:tasks.workunit.client.0.smithi016.stdout:9/727: chown d9/d57/d91/de4/d25/da6/f5b 4643136 1 2024-04-22T15:49:42.671 INFO:tasks.workunit.client.0.smithi016.stdout:8/653: read d2/d48/f49 [2408857,106476] 0 2024-04-22T15:49:42.674 INFO:tasks.workunit.client.0.smithi016.stdout:4/700: mkdir d7/d21/d45/d49/de4 0 2024-04-22T15:49:42.677 INFO:tasks.workunit.client.0.smithi016.stdout:4/701: chown d7/d21/d42/d59/d88/d9d/dad/dac/cb2 4632458 1 2024-04-22T15:49:42.677 INFO:tasks.workunit.client.0.smithi016.stdout:5/727: dwrite df/d15/d1a/d50/d77/d7d/d8b/fcf [0,4194304] 0 2024-04-22T15:49:42.698 INFO:tasks.workunit.client.0.smithi016.stdout:5/728: mkdir df/d15/d1a/d50/d77/d7d/d8b/d8c/ddf 0 2024-04-22T15:49:42.701 INFO:tasks.workunit.client.0.smithi016.stdout:0/782: symlink dc/d13/d14/de5/d65/l100 0 2024-04-22T15:49:42.702 INFO:tasks.workunit.client.0.smithi016.stdout:0/783: truncate dc/d13/d6f/d8b/ffe 729275 0 2024-04-22T15:49:42.706 INFO:tasks.workunit.client.0.smithi016.stdout:8/654: link d2/f7a d2/d48/dad/d8f/da1/fc8 0 2024-04-22T15:49:42.710 INFO:tasks.workunit.client.0.smithi016.stdout:5/729: creat df/d15/d1a/d50/d77/d7d/d8b/d93/fe0 x:0 0 0 2024-04-22T15:49:42.712 INFO:tasks.workunit.client.0.smithi016.stdout:9/728: creat d9/de/fe6 x:0 0 0 2024-04-22T15:49:42.713 INFO:tasks.workunit.client.0.smithi016.stdout:0/784: truncate dc/d13/d14/de5/d65/d30/f44 1259236 0 2024-04-22T15:49:42.714 INFO:tasks.workunit.client.0.smithi016.stdout:5/730: dread df/d15/d1a/d28/f2c [4194304,4194304] 0 2024-04-22T15:49:42.715 INFO:tasks.workunit.client.0.smithi016.stdout:3/804: creat d7/d17/d18/d32/d3c/d51/df1/ff7 x:0 0 0 2024-04-22T15:49:42.716 INFO:tasks.workunit.client.0.smithi016.stdout:3/805: write d7/f8 [4362651,124811] 0 2024-04-22T15:49:42.717 INFO:tasks.workunit.client.0.smithi016.stdout:3/806: chown d7/d17/d18/d27/c81 5 1 2024-04-22T15:49:42.718 INFO:tasks.workunit.client.0.smithi016.stdout:8/655: rename d2/d48/f49 to d2/d82/d44/d66/d1c/fc9 0 2024-04-22T15:49:42.719 INFO:tasks.workunit.client.0.smithi016.stdout:8/656: chown d2/l6e 0 1 2024-04-22T15:49:42.720 INFO:tasks.workunit.client.0.smithi016.stdout:8/657: write d2/d82/f58 [1103594,97167] 0 2024-04-22T15:49:42.721 INFO:tasks.workunit.client.0.smithi016.stdout:8/658: chown d2/d10/l7c 0 1 2024-04-22T15:49:42.726 INFO:tasks.workunit.client.0.smithi016.stdout:5/731: creat df/d15/d1a/d4e/d58/d87/fe1 x:0 0 0 2024-04-22T15:49:42.727 INFO:tasks.workunit.client.0.smithi016.stdout:3/807: write d7/d17/d18/d27/fbe [4651833,101628] 0 2024-04-22T15:49:42.730 INFO:tasks.workunit.client.0.smithi016.stdout:0/785: rename dc/d13/d14/de5/d65/d30/f84 to dc/d13/d14/de5/d65/d1f/f101 0 2024-04-22T15:49:42.732 INFO:tasks.workunit.client.0.smithi016.stdout:4/702: getdents d7/d21/d45 0 2024-04-22T15:49:42.738 INFO:tasks.workunit.client.0.smithi016.stdout:3/808: creat d7/d17/d18/d32/d3c/d47/dc3/ff8 x:0 0 0 2024-04-22T15:49:42.739 INFO:tasks.workunit.client.0.smithi016.stdout:3/809: write d7/d17/d18/d32/d3c/d47/f52 [1084864,30544] 0 2024-04-22T15:49:42.739 INFO:tasks.workunit.client.0.smithi016.stdout:8/659: rename d2/d10/d43/l9e to d2/d82/d44/d66/d79/lca 0 2024-04-22T15:49:42.745 INFO:tasks.workunit.client.0.smithi016.stdout:4/703: unlink d7/d4e/d2f/d33/f7a 0 2024-04-22T15:49:42.745 INFO:tasks.workunit.client.0.smithi016.stdout:9/729: getdents d9/de/d23/ddb 0 2024-04-22T15:49:42.745 INFO:tasks.workunit.client.0.smithi016.stdout:8/660: fsync d2/d10/d43/da9/fac 0 2024-04-22T15:49:42.746 INFO:tasks.workunit.client.0.smithi016.stdout:0/786: read dc/d13/d22/f2e [3848623,84978] 0 2024-04-22T15:49:42.746 INFO:tasks.workunit.client.0.smithi016.stdout:9/730: fsync d9/d5d/d62/f6e 0 2024-04-22T15:49:42.746 INFO:tasks.workunit.client.0.smithi016.stdout:8/661: chown d2/d82/d44 1737492 1 2024-04-22T15:49:42.754 INFO:tasks.workunit.client.0.smithi016.stdout:9/731: dwrite d9/de/d23/fcf [0,4194304] 0 2024-04-22T15:49:42.755 INFO:tasks.workunit.client.0.smithi016.stdout:8/662: dwrite d2/d48/dad/d8f/fb8 [0,4194304] 0 2024-04-22T15:49:42.764 INFO:tasks.workunit.client.0.smithi016.stdout:7/713: sync 2024-04-22T15:49:42.775 INFO:tasks.workunit.client.0.smithi016.stdout:5/732: write df/d15/d1a/fbe [2405401,68227] 0 2024-04-22T15:49:42.783 INFO:tasks.workunit.client.0.smithi016.stdout:9/732: dread f2 [0,4194304] 0 2024-04-22T15:49:42.784 INFO:tasks.workunit.client.0.smithi016.stdout:9/733: read d9/fa [1184110,51263] 0 2024-04-22T15:49:42.785 INFO:tasks.workunit.client.0.smithi016.stdout:9/734: readlink d9/d57/d91/de4/d25/da6/lb5 0 2024-04-22T15:49:42.787 INFO:tasks.workunit.client.1.smithi027.stdout:0/27: rmdir d0/d6 0 2024-04-22T15:49:42.788 INFO:tasks.workunit.client.1.smithi027.stdout:9/31: mkdir d0/dc 0 2024-04-22T15:49:42.791 INFO:tasks.workunit.client.1.smithi027.stdout:4/24: creat d5/f9 x:0 0 0 2024-04-22T15:49:42.795 INFO:tasks.workunit.client.1.smithi027.stdout:0/28: symlink d0/l9 0 2024-04-22T15:49:42.795 INFO:tasks.workunit.client.1.smithi027.stdout:0/29: stat d0/f1 0 2024-04-22T15:49:42.796 INFO:tasks.workunit.client.1.smithi027.stdout:9/32: symlink d0/d8/ld 0 2024-04-22T15:49:42.799 INFO:tasks.workunit.client.1.smithi027.stdout:4/25: creat d5/fa x:0 0 0 2024-04-22T15:49:42.800 INFO:tasks.workunit.client.1.smithi027.stdout:0/30: symlink d0/la 0 2024-04-22T15:49:42.803 INFO:tasks.workunit.client.1.smithi027.stdout:9/33: dwrite d0/d8/f9 [0,4194304] 0 2024-04-22T15:49:42.805 INFO:tasks.workunit.client.1.smithi027.stdout:4/26: creat d5/d7/fb x:0 0 0 2024-04-22T15:49:42.806 INFO:tasks.workunit.client.1.smithi027.stdout:9/34: truncate d0/fb 874653 0 2024-04-22T15:49:42.806 INFO:tasks.workunit.client.0.smithi016.stdout:4/704: mknod d7/d21/d42/d59/de2/ce5 0 2024-04-22T15:49:42.806 INFO:tasks.workunit.client.1.smithi027.stdout:4/27: stat d5/d7 0 2024-04-22T15:49:42.808 INFO:tasks.workunit.client.1.smithi027.stdout:4/28: chown f0 14 1 2024-04-22T15:49:42.811 INFO:tasks.workunit.client.1.smithi027.stdout:0/31: dwrite d0/f1 [4194304,4194304] 0 2024-04-22T15:49:42.813 INFO:tasks.workunit.client.1.smithi027.stdout:9/35: unlink d0/l5 0 2024-04-22T15:49:42.815 INFO:tasks.workunit.client.0.smithi016.stdout:2/700: sync 2024-04-22T15:49:42.815 INFO:tasks.workunit.client.0.smithi016.stdout:1/848: sync 2024-04-22T15:49:42.815 INFO:tasks.workunit.client.0.smithi016.stdout:6/750: sync 2024-04-22T15:49:42.815 INFO:tasks.workunit.client.1.smithi027.stdout:9/36: write d0/d8/f9 [3548448,60351] 0 2024-04-22T15:49:42.818 INFO:tasks.workunit.client.1.smithi027.stdout:0/32: mknod d0/cb 0 2024-04-22T15:49:42.819 INFO:tasks.workunit.client.1.smithi027.stdout:4/29: dwrite f3 [0,4194304] 0 2024-04-22T15:49:42.819 INFO:tasks.workunit.client.0.smithi016.stdout:8/663: readlink d2/d77/lc7 0 2024-04-22T15:49:42.828 INFO:tasks.workunit.client.1.smithi027.stdout:0/33: rename d0/f8 to d0/fc 0 2024-04-22T15:49:42.831 INFO:tasks.workunit.client.0.smithi016.stdout:9/735: symlink d9/d57/d91/de4/d25/le7 0 2024-04-22T15:49:42.831 INFO:tasks.workunit.client.1.smithi027.stdout:9/37: mkdir d0/dc/de 0 2024-04-22T15:49:42.832 INFO:tasks.workunit.client.1.smithi027.stdout:9/38: truncate d0/fb 1014923 0 2024-04-22T15:49:42.834 INFO:tasks.workunit.client.1.smithi027.stdout:0/34: symlink d0/ld 0 2024-04-22T15:49:42.835 INFO:tasks.workunit.client.1.smithi027.stdout:0/35: chown d0/la 528692409 1 2024-04-22T15:49:42.836 INFO:tasks.workunit.client.1.smithi027.stdout:4/30: fdatasync f3 0 2024-04-22T15:49:42.836 INFO:tasks.workunit.client.1.smithi027.stdout:9/39: mkdir d0/dc/df 0 2024-04-22T15:49:42.837 INFO:tasks.workunit.client.1.smithi027.stdout:9/40: read d0/d8/f9 [2548606,86865] 0 2024-04-22T15:49:42.837 INFO:tasks.workunit.client.1.smithi027.stdout:9/41: fsync d0/fb 0 2024-04-22T15:49:42.838 INFO:tasks.workunit.client.0.smithi016.stdout:0/787: rename dc/d13/d22/l2b to dc/d13/d14/de5/d65/d57/dcb/l102 0 2024-04-22T15:49:42.839 INFO:tasks.workunit.client.1.smithi027.stdout:0/36: creat d0/fe x:0 0 0 2024-04-22T15:49:42.840 INFO:tasks.workunit.client.1.smithi027.stdout:0/37: truncate d0/fc 222398 0 2024-04-22T15:49:42.841 INFO:tasks.workunit.client.1.smithi027.stdout:4/31: mknod d5/cc 0 2024-04-22T15:49:42.841 INFO:tasks.workunit.client.1.smithi027.stdout:0/38: truncate d0/fc 1193812 0 2024-04-22T15:49:42.841 INFO:tasks.workunit.client.1.smithi027.stdout:9/42: dread d0/d8/f9 [0,4194304] 0 2024-04-22T15:49:42.844 INFO:tasks.workunit.client.1.smithi027.stdout:0/39: dread d0/fc [0,4194304] 0 2024-04-22T15:49:42.844 INFO:tasks.workunit.client.0.smithi016.stdout:6/751: dread - d3/da/dd/d1b/f8b zero size 2024-04-22T15:49:42.845 INFO:tasks.workunit.client.0.smithi016.stdout:2/701: dwrite d2/d1a/d4c/f3f [0,4194304] 0 2024-04-22T15:49:42.845 INFO:tasks.workunit.client.0.smithi016.stdout:6/752: readlink d3/da/db/d14/d4c/d58/l8d 0 2024-04-22T15:49:42.846 INFO:tasks.workunit.client.0.smithi016.stdout:1/849: truncate d3/d4/d19/d20/d2b/fc8 917845 0 2024-04-22T15:49:42.850 INFO:tasks.workunit.client.1.smithi027.stdout:9/43: dwrite d0/fb [0,4194304] 0 2024-04-22T15:49:42.850 INFO:tasks.workunit.client.0.smithi016.stdout:8/664: unlink d2/d48/f55 0 2024-04-22T15:49:42.855 INFO:tasks.workunit.client.1.smithi027.stdout:0/40: symlink d0/lf 0 2024-04-22T15:49:42.856 INFO:tasks.workunit.client.1.smithi027.stdout:0/41: read - d0/fe zero size 2024-04-22T15:49:42.857 INFO:tasks.workunit.client.0.smithi016.stdout:5/733: mknod df/d15/d1a/d50/d77/d7d/d8b/d93/dbb/ce2 0 2024-04-22T15:49:42.857 INFO:tasks.workunit.client.1.smithi027.stdout:0/42: truncate d0/f1 8476007 0 2024-04-22T15:49:42.860 INFO:tasks.workunit.client.1.smithi027.stdout:9/44: unlink d0/d8/f9 0 2024-04-22T15:49:42.865 INFO:tasks.workunit.client.0.smithi016.stdout:5/734: dwrite df/d15/d1a/d4e/d58/fb8 [0,4194304] 0 2024-04-22T15:49:42.871 INFO:tasks.workunit.client.0.smithi016.stdout:1/850: truncate d3/d4/d19/fda 3707790 0 2024-04-22T15:49:42.873 INFO:tasks.workunit.client.0.smithi016.stdout:1/851: chown d3/d2e/d9f/cac 5015120 1 2024-04-22T15:49:42.878 INFO:tasks.workunit.client.0.smithi016.stdout:3/810: getdents d7/d17/d18/d32/d3c 0 2024-04-22T15:49:42.879 INFO:tasks.workunit.client.0.smithi016.stdout:4/705: creat d7/d3c/fe6 x:0 0 0 2024-04-22T15:49:42.881 INFO:tasks.workunit.client.0.smithi016.stdout:6/753: dwrite d3/da/dd/d1b/d57/fbf [0,4194304] 0 2024-04-22T15:49:42.884 INFO:tasks.workunit.client.0.smithi016.stdout:1/852: mkdir d3/d4/d3a/d9e/d124 0 2024-04-22T15:49:42.884 INFO:tasks.workunit.client.0.smithi016.stdout:0/788: rmdir dc/d13/d6f/daf 0 2024-04-22T15:49:42.885 INFO:tasks.workunit.client.0.smithi016.stdout:6/754: stat d3/da/db/d44/d68/fa5 0 2024-04-22T15:49:42.886 INFO:tasks.workunit.client.0.smithi016.stdout:7/714: dwrite d3/dc/d33/f4d [0,4194304] 0 2024-04-22T15:49:42.888 INFO:tasks.workunit.client.0.smithi016.stdout:9/736: getdents d9/d5d/d62 0 2024-04-22T15:49:42.889 INFO:tasks.workunit.client.0.smithi016.stdout:6/755: write d3/da/dd/f70 [4269555,44484] 0 2024-04-22T15:49:42.890 INFO:tasks.workunit.client.0.smithi016.stdout:4/706: rmdir d7/d21 39 2024-04-22T15:49:42.896 INFO:tasks.workunit.client.0.smithi016.stdout:7/715: rmdir d3/dc/d1b/d31 39 2024-04-22T15:49:42.897 INFO:tasks.workunit.client.1.smithi027.stdout:0/43: rmdir d0 39 2024-04-22T15:49:42.898 INFO:tasks.workunit.client.0.smithi016.stdout:0/789: dread dc/d13/d14/d26/d32/fc0 [4194304,4194304] 0 2024-04-22T15:49:42.898 INFO:tasks.workunit.client.0.smithi016.stdout:9/737: creat d9/d5d/d62/fe8 x:0 0 0 2024-04-22T15:49:42.904 INFO:tasks.workunit.client.1.smithi027.stdout:0/44: dread d0/fc [0,4194304] 0 2024-04-22T15:49:42.904 INFO:tasks.workunit.client.0.smithi016.stdout:6/756: mkdir d3/da/dd/d67/d6c/dfc 0 2024-04-22T15:49:42.906 INFO:tasks.workunit.client.1.smithi027.stdout:9/45: dwrite d0/fb [4194304,4194304] 0 2024-04-22T15:49:42.907 INFO:tasks.workunit.client.0.smithi016.stdout:8/665: write d2/d82/d44/d66/d79/f8b [2250661,94641] 0 2024-04-22T15:49:42.908 INFO:tasks.workunit.client.0.smithi016.stdout:7/716: rmdir d3/dc/d1b/d5a/d59/d64/dc9 39 2024-04-22T15:49:42.908 INFO:tasks.workunit.client.0.smithi016.stdout:6/757: readlink d3/da/db/d14/d4c/l66 0 2024-04-22T15:49:42.909 INFO:tasks.workunit.client.1.smithi027.stdout:0/45: rmdir d0 39 2024-04-22T15:49:42.911 INFO:tasks.workunit.client.1.smithi027.stdout:9/46: dread d0/fb [4194304,4194304] 0 2024-04-22T15:49:42.914 INFO:tasks.workunit.client.0.smithi016.stdout:6/758: write d3/da/db/d14/d16/f55 [360921,116063] 0 2024-04-22T15:49:42.914 INFO:tasks.workunit.client.0.smithi016.stdout:2/702: dwrite d2/d2c/d97/d34/d60/f83 [4194304,4194304] 0 2024-04-22T15:49:42.917 INFO:tasks.workunit.client.0.smithi016.stdout:9/738: dread d9/d57/d91/de4/d25/f2c [0,4194304] 0 2024-04-22T15:49:42.918 INFO:tasks.workunit.client.0.smithi016.stdout:5/735: truncate df/d15/d1a/d50/d77/fce 2917485 0 2024-04-22T15:49:42.919 INFO:tasks.workunit.client.0.smithi016.stdout:4/707: symlink d7/d21/d45/le7 0 2024-04-22T15:49:42.922 INFO:tasks.workunit.client.0.smithi016.stdout:0/790: dread dc/d13/d14/d26/f7b [0,4194304] 0 2024-04-22T15:49:42.924 INFO:tasks.workunit.client.1.smithi027.stdout:0/46: rename d0/fe to d0/f10 0 2024-04-22T15:49:42.925 INFO:tasks.workunit.client.1.smithi027.stdout:0/47: truncate d0/fc 1805151 0 2024-04-22T15:49:42.927 INFO:tasks.workunit.client.0.smithi016.stdout:6/759: dread - d3/d53/d83/f9f zero size 2024-04-22T15:49:42.928 INFO:tasks.workunit.client.0.smithi016.stdout:1/853: dwrite d3/d4/d3a/d9e/fd8 [0,4194304] 0 2024-04-22T15:49:42.931 INFO:tasks.workunit.client.0.smithi016.stdout:2/703: mkdir d2/d1a/d7b/dcc/de5/d107 0 2024-04-22T15:49:42.932 INFO:tasks.workunit.client.0.smithi016.stdout:5/736: chown df/d15/d1a/l1c 0 1 2024-04-22T15:49:42.932 INFO:tasks.workunit.client.0.smithi016.stdout:4/708: creat d7/d21/d7c/fe8 x:0 0 0 2024-04-22T15:49:42.933 INFO:tasks.workunit.client.0.smithi016.stdout:9/739: truncate d9/d57/d91/de4/d25/d36/f6b 4380760 0 2024-04-22T15:49:42.933 INFO:tasks.workunit.client.0.smithi016.stdout:3/811: rename d7/d17/d18/d32/d42/d63/deb/d21/d55/fa6 to d7/d17/d18/d32/d42/ff9 0 2024-04-22T15:49:42.934 INFO:tasks.workunit.client.1.smithi027.stdout:9/47: truncate d0/fb 6849715 0 2024-04-22T15:49:42.934 INFO:tasks.workunit.client.0.smithi016.stdout:9/740: readlink d9/l1e 0 2024-04-22T15:49:42.934 INFO:tasks.workunit.client.0.smithi016.stdout:6/760: write d3/da/db/d44/d68/feb [553167,56924] 0 2024-04-22T15:49:42.935 INFO:tasks.workunit.client.1.smithi027.stdout:9/48: rename d0/dc/de to d0/dc/de/d10 22 2024-04-22T15:49:42.936 INFO:tasks.workunit.client.1.smithi027.stdout:9/49: chown d0/fb 1071243899 1 2024-04-22T15:49:42.937 INFO:tasks.workunit.client.0.smithi016.stdout:7/717: creat d3/dc/d1b/d5a/d9/df/d94/ddf/f102 x:0 0 0 2024-04-22T15:49:42.941 INFO:tasks.workunit.client.0.smithi016.stdout:3/812: truncate d7/d17/d18/d27/f3d 1985314 0 2024-04-22T15:49:42.943 INFO:tasks.workunit.client.0.smithi016.stdout:7/718: stat d3/dc/d1b/d5a/d9/df/faa 0 2024-04-22T15:49:42.943 INFO:tasks.workunit.client.0.smithi016.stdout:0/791: chown dc/d13/d14/de5/dd8/ced 0 1 2024-04-22T15:49:42.944 INFO:tasks.workunit.client.1.smithi027.stdout:0/48: truncate d0/f1 1318492 0 2024-04-22T15:49:42.944 INFO:tasks.workunit.client.1.smithi027.stdout:0/49: rename d0 to d0/d11 22 2024-04-22T15:49:42.946 INFO:tasks.workunit.client.0.smithi016.stdout:5/737: dwrite df/d15/d1a/d50/d77/d7d/d8b/d93/fa7 [0,4194304] 0 2024-04-22T15:49:42.947 INFO:tasks.workunit.client.0.smithi016.stdout:2/704: dwrite d2/d1a/d7b/dcc/fc8 [0,4194304] 0 2024-04-22T15:49:42.949 INFO:tasks.workunit.client.1.smithi027.stdout:9/50: dread d0/fb [4194304,4194304] 0 2024-04-22T15:49:42.950 INFO:tasks.workunit.client.0.smithi016.stdout:2/705: chown d2/d2c/de7 94 1 2024-04-22T15:49:42.954 INFO:tasks.workunit.client.0.smithi016.stdout:6/761: rename d3/c99 to d3/da/db/d14/d4c/cfd 0 2024-04-22T15:49:42.955 INFO:tasks.workunit.client.1.smithi027.stdout:0/50: dwrite d0/f10 [0,4194304] 0 2024-04-22T15:49:42.957 INFO:tasks.workunit.client.1.smithi027.stdout:9/51: dread d0/fb [0,4194304] 0 2024-04-22T15:49:42.957 INFO:tasks.workunit.client.0.smithi016.stdout:4/709: dread - d7/d21/d45/d49/d9b/fa2 zero size 2024-04-22T15:49:42.960 INFO:tasks.workunit.client.0.smithi016.stdout:8/666: getdents d2/d82/d44/d66/d1c/d65 0 2024-04-22T15:49:42.960 INFO:tasks.workunit.client.0.smithi016.stdout:0/792: readlink dc/d13/d14/de5/d65/d57/dcb/l102 0 2024-04-22T15:49:42.960 INFO:tasks.workunit.client.0.smithi016.stdout:8/667: chown d2/d82/d52 49 1 2024-04-22T15:49:42.963 INFO:tasks.workunit.client.1.smithi027.stdout:0/51: creat d0/f12 x:0 0 0 2024-04-22T15:49:42.963 INFO:tasks.workunit.client.1.smithi027.stdout:9/52: rmdir d0/dc 39 2024-04-22T15:49:42.963 INFO:tasks.workunit.client.0.smithi016.stdout:0/793: fdatasync dc/d13/d14/d55/f93 0 2024-04-22T15:49:42.963 INFO:tasks.workunit.client.0.smithi016.stdout:0/794: dread - dc/d13/d14/de5/d65/d40/d4a/d59/d78/fbc zero size 2024-04-22T15:49:42.965 INFO:tasks.workunit.client.0.smithi016.stdout:3/813: mkdir d7/d17/d18/d32/d42/d63/deb/d21/d55/dfa 0 2024-04-22T15:49:42.967 INFO:tasks.workunit.client.0.smithi016.stdout:3/814: chown d7/d17/d18/d32/d42 4 1 2024-04-22T15:49:42.967 INFO:tasks.workunit.client.1.smithi027.stdout:9/53: dread d0/fb [0,4194304] 0 2024-04-22T15:49:42.968 INFO:tasks.workunit.client.0.smithi016.stdout:3/815: chown d7/d17/d18/d32/d42 72657441 1 2024-04-22T15:49:42.971 INFO:tasks.workunit.client.1.smithi027.stdout:9/54: creat d0/d8/f11 x:0 0 0 2024-04-22T15:49:42.972 INFO:tasks.workunit.client.1.smithi027.stdout:0/52: dwrite d0/fc [0,4194304] 0 2024-04-22T15:49:42.973 INFO:tasks.workunit.client.0.smithi016.stdout:6/762: write d3/da/db/d14/fc8 [1010116,33586] 0 2024-04-22T15:49:42.976 INFO:tasks.workunit.client.0.smithi016.stdout:1/854: write d3/d4/d3a/d91/db3/f11d [402066,17644] 0 2024-04-22T15:49:42.979 INFO:tasks.workunit.client.0.smithi016.stdout:4/710: creat d7/d24/d41/fe9 x:0 0 0 2024-04-22T15:49:42.980 INFO:tasks.workunit.client.0.smithi016.stdout:6/763: write d3/da/db/d14/d4c/d58/f6f [1718299,57698] 0 2024-04-22T15:49:42.981 INFO:tasks.workunit.client.1.smithi027.stdout:0/53: dwrite d0/fc [0,4194304] 0 2024-04-22T15:49:42.984 INFO:tasks.workunit.client.0.smithi016.stdout:9/741: dread d9/de/f31 [0,4194304] 0 2024-04-22T15:49:42.984 INFO:tasks.workunit.client.1.smithi027.stdout:9/55: mkdir d0/dc/d12 0 2024-04-22T15:49:42.985 INFO:tasks.workunit.client.1.smithi027.stdout:9/56: write d0/d8/f11 [125453,102016] 0 2024-04-22T15:49:42.985 INFO:tasks.workunit.client.0.smithi016.stdout:4/711: chown d7/d21/d42/d59/d88/d9d/la6 967997185 1 2024-04-22T15:49:42.985 INFO:tasks.workunit.client.0.smithi016.stdout:6/764: chown d3/da/db/d14/d16/f20 7311609 1 2024-04-22T15:49:42.988 INFO:tasks.workunit.client.0.smithi016.stdout:8/668: dread d2/d82/d3e/f42 [4194304,4194304] 0 2024-04-22T15:49:42.988 INFO:tasks.workunit.client.0.smithi016.stdout:3/816: symlink d7/d17/d18/d27/lfb 0 2024-04-22T15:49:42.990 INFO:tasks.workunit.client.0.smithi016.stdout:8/669: chown d2/d30 0 1 2024-04-22T15:49:42.990 INFO:tasks.workunit.client.0.smithi016.stdout:8/670: symlink d2/d48/dad/d8f/d96/lcb 2 2024-04-22T15:49:42.990 INFO:tasks.workunit.client.0.smithi016.stdout:8/671: stat d2/d48/db9 0 2024-04-22T15:49:42.992 INFO:tasks.workunit.client.0.smithi016.stdout:8/672: dread - d2/d48/dad/d8f/fb3 zero size 2024-04-22T15:49:42.994 INFO:tasks.workunit.client.1.smithi027.stdout:9/57: symlink d0/l13 0 2024-04-22T15:49:42.994 INFO:tasks.workunit.client.0.smithi016.stdout:3/817: dread - d7/d17/d18/d32/d42/d63/d86/fdf zero size 2024-04-22T15:49:42.997 INFO:tasks.workunit.client.1.smithi027.stdout:0/54: getdents d0 0 2024-04-22T15:49:42.997 INFO:tasks.workunit.client.0.smithi016.stdout:3/818: write d7/d17/d18/d32/d42/d63/db4/fda [732064,75857] 0 2024-04-22T15:49:42.997 INFO:tasks.workunit.client.0.smithi016.stdout:1/855: mknod d3/d2e/d80/c125 0 2024-04-22T15:49:42.998 INFO:tasks.workunit.client.0.smithi016.stdout:1/856: chown d3/d4/d19/d20/d2b/dd6/lf1 7938332 1 2024-04-22T15:49:42.998 INFO:tasks.workunit.client.0.smithi016.stdout:6/765: mknod d3/da/d12/d13/dd2/cfe 0 2024-04-22T15:49:43.001 INFO:tasks.workunit.client.0.smithi016.stdout:1/857: fsync d3/d4/d15/f16 0 2024-04-22T15:49:43.001 INFO:tasks.workunit.client.0.smithi016.stdout:8/673: symlink d2/d48/lcc 0 2024-04-22T15:49:43.002 INFO:tasks.workunit.client.0.smithi016.stdout:8/674: fsync d2/f5c 0 2024-04-22T15:49:43.003 INFO:tasks.workunit.client.1.smithi027.stdout:0/55: dwrite d0/f12 [0,4194304] 0 2024-04-22T15:49:43.005 INFO:tasks.workunit.client.0.smithi016.stdout:3/819: creat d7/d17/d18/d32/d42/d63/d86/dd8/ffc x:0 0 0 2024-04-22T15:49:43.007 INFO:tasks.workunit.client.1.smithi027.stdout:0/56: unlink d0/l9 0 2024-04-22T15:49:43.008 INFO:tasks.workunit.client.0.smithi016.stdout:6/766: dwrite d3/da/dd/f29 [0,4194304] 0 2024-04-22T15:49:43.009 INFO:tasks.workunit.client.0.smithi016.stdout:8/675: dread d2/d48/dad/d6a/f9a [0,4194304] 0 2024-04-22T15:49:43.011 INFO:tasks.workunit.client.0.smithi016.stdout:8/676: write d2/f27 [2516093,57456] 0 2024-04-22T15:49:43.013 INFO:tasks.workunit.client.0.smithi016.stdout:7/719: rename d3/dc/d1b/d5a/d59/la6 to d3/dc/d1b/d5a/l103 0 2024-04-22T15:49:43.016 INFO:tasks.workunit.client.0.smithi016.stdout:0/795: write dc/d13/d14/de5/d65/d1f/d48/f64 [1982954,68871] 0 2024-04-22T15:49:43.017 INFO:tasks.workunit.client.0.smithi016.stdout:2/706: link d2/d23/d92/d55/cc7 d2/d4d/d76/c108 0 2024-04-22T15:49:43.019 INFO:tasks.workunit.client.0.smithi016.stdout:7/720: chown d3/dc/d1b/d5a/d9/df/d94/d4c/d7a/fcf 315 1 2024-04-22T15:49:43.019 INFO:tasks.workunit.client.0.smithi016.stdout:1/858: creat d3/d84/f126 x:0 0 0 2024-04-22T15:49:43.020 INFO:tasks.workunit.client.0.smithi016.stdout:7/721: stat d3/dc/d33/ff4 0 2024-04-22T15:49:43.025 INFO:tasks.workunit.client.1.smithi027.stdout:0/57: getdents d0 0 2024-04-22T15:49:43.027 INFO:tasks.workunit.client.0.smithi016.stdout:0/796: dwrite dc/d13/d5d/f85 [0,4194304] 0 2024-04-22T15:49:43.027 INFO:tasks.workunit.client.0.smithi016.stdout:6/767: dread d3/da/db/d14/d4c/d58/f75 [0,4194304] 0 2024-04-22T15:49:43.029 INFO:tasks.workunit.client.1.smithi027.stdout:0/58: mknod d0/c13 0 2024-04-22T15:49:43.029 INFO:tasks.workunit.client.1.smithi027.stdout:0/59: readlink d0/la 0 2024-04-22T15:49:43.032 INFO:tasks.workunit.client.0.smithi016.stdout:5/738: rename df/f11 to df/d15/d1a/d50/fe3 0 2024-04-22T15:49:43.033 INFO:tasks.workunit.client.1.smithi027.stdout:0/60: creat d0/f14 x:0 0 0 2024-04-22T15:49:43.033 INFO:tasks.workunit.client.0.smithi016.stdout:4/712: dwrite d7/d24/d41/f57 [0,4194304] 0 2024-04-22T15:49:43.034 INFO:tasks.workunit.client.0.smithi016.stdout:7/722: symlink d3/dc/d1b/d5a/d9/df/d94/l104 0 2024-04-22T15:49:43.034 INFO:tasks.workunit.client.0.smithi016.stdout:6/768: unlink d3/da/dd/d67/f79 0 2024-04-22T15:49:43.035 INFO:tasks.workunit.client.0.smithi016.stdout:1/859: chown d3/d4/d45/ced 24295726 1 2024-04-22T15:49:43.036 INFO:tasks.workunit.client.0.smithi016.stdout:4/713: chown d7/d22/f27 1023113 1 2024-04-22T15:49:43.038 INFO:tasks.workunit.client.1.smithi027.stdout:0/61: creat d0/f15 x:0 0 0 2024-04-22T15:49:43.038 INFO:tasks.workunit.client.0.smithi016.stdout:8/677: creat d2/d82/d44/d66/d7e/dc0/fcd x:0 0 0 2024-04-22T15:49:43.039 INFO:tasks.workunit.client.0.smithi016.stdout:0/797: read dc/d13/d6f/d8b/f9a [768709,111663] 0 2024-04-22T15:49:43.042 INFO:tasks.workunit.client.0.smithi016.stdout:3/820: rename d7/d17/d18/d32/d42/d63/d86/dd8/fd3 to d7/d17/d18/d32/d8a/ffd 0 2024-04-22T15:49:43.043 INFO:tasks.workunit.client.1.smithi027.stdout:0/62: rename d0/l4 to d0/l16 0 2024-04-22T15:49:43.044 INFO:tasks.workunit.client.1.smithi027.stdout:0/63: dread d0/f1 [0,4194304] 0 2024-04-22T15:49:43.045 INFO:tasks.workunit.client.1.smithi027.stdout:0/64: read d0/f1 [804182,124794] 0 2024-04-22T15:49:43.049 INFO:tasks.workunit.client.1.smithi027.stdout:0/65: dread d0/f12 [0,4194304] 0 2024-04-22T15:49:43.051 INFO:tasks.workunit.client.0.smithi016.stdout:6/769: dwrite d3/da/d12/d13/dd2/dee/fb4 [0,4194304] 0 2024-04-22T15:49:43.051 INFO:tasks.workunit.client.0.smithi016.stdout:7/723: unlink d3/dc/d1b/d5a/d9/df/d94/d4c/d7a/fcf 0 2024-04-22T15:49:43.052 INFO:tasks.workunit.client.0.smithi016.stdout:2/707: dwrite d2/d9/f12 [4194304,4194304] 0 2024-04-22T15:49:43.054 INFO:tasks.workunit.client.1.smithi027.stdout:0/66: symlink d0/l17 0 2024-04-22T15:49:43.055 INFO:tasks.workunit.client.1.smithi027.stdout:0/67: dread d0/f1 [0,4194304] 0 2024-04-22T15:49:43.058 INFO:tasks.workunit.client.1.smithi027.stdout:0/68: readlink d0/l3 0 2024-04-22T15:49:43.058 INFO:tasks.workunit.client.0.smithi016.stdout:4/714: rename d7/d24/d41/l95 to d7/d24/d41/lea 0 2024-04-22T15:49:43.059 INFO:tasks.workunit.client.1.smithi027.stdout:0/69: creat d0/f18 x:0 0 0 2024-04-22T15:49:43.060 INFO:tasks.workunit.client.0.smithi016.stdout:1/860: mknod d3/d4/d3a/d91/db3/c127 0 2024-04-22T15:49:43.061 INFO:tasks.workunit.client.0.smithi016.stdout:1/861: readlink d3/d4/lfb 0 2024-04-22T15:49:43.063 INFO:tasks.workunit.client.1.smithi027.stdout:0/70: rename d0/l17 to d0/l19 0 2024-04-22T15:49:43.065 INFO:tasks.workunit.client.0.smithi016.stdout:8/678: write d2/d10/f41 [952043,93227] 0 2024-04-22T15:49:43.066 INFO:tasks.workunit.client.0.smithi016.stdout:4/715: dread d7/d4e/d2f/d6f/fa8 [0,4194304] 0 2024-04-22T15:49:43.066 INFO:tasks.workunit.client.1.smithi027.stdout:0/71: fsync d0/f12 0 2024-04-22T15:49:43.067 INFO:tasks.workunit.client.1.smithi027.stdout:0/72: chown d0/ld 400 1 2024-04-22T15:49:43.069 INFO:tasks.workunit.client.0.smithi016.stdout:0/798: creat dc/d13/d14/d26/f103 x:0 0 0 2024-04-22T15:49:43.069 INFO:tasks.workunit.client.0.smithi016.stdout:5/739: getdents df/d5b 0 2024-04-22T15:49:43.070 INFO:tasks.workunit.client.0.smithi016.stdout:0/799: fdatasync dc/d13/d14/d55/fa8 0 2024-04-22T15:49:43.071 INFO:tasks.workunit.client.0.smithi016.stdout:5/740: chown df/d5b/db1/cb5 32164234 1 2024-04-22T15:49:43.072 INFO:tasks.workunit.client.0.smithi016.stdout:2/708: creat d2/d1a/d4c/d9f/dae/f109 x:0 0 0 2024-04-22T15:49:43.074 INFO:tasks.workunit.client.0.smithi016.stdout:7/724: link d3/dc/d33/f7f d3/dc/d1b/d5a/d9/df/d94/f105 0 2024-04-22T15:49:43.076 INFO:tasks.workunit.client.0.smithi016.stdout:6/770: mkdir d3/da/db/d14/dff 0 2024-04-22T15:49:43.077 INFO:tasks.workunit.client.1.smithi027.stdout:0/73: rmdir d0 39 2024-04-22T15:49:43.079 INFO:tasks.workunit.client.0.smithi016.stdout:4/716: mkdir d7/d21/d42/d59/d88/deb 0 2024-04-22T15:49:43.080 INFO:tasks.workunit.client.0.smithi016.stdout:0/800: dwrite dc/d13/d14/de5/d65/d40/d4a/d61/fe4 [0,4194304] 0 2024-04-22T15:49:43.081 INFO:tasks.workunit.client.1.smithi027.stdout:0/74: dread d0/f1 [0,4194304] 0 2024-04-22T15:49:43.082 INFO:tasks.workunit.client.0.smithi016.stdout:1/862: creat d3/d4/d19/d20/d2b/d10d/f128 x:0 0 0 2024-04-22T15:49:43.082 INFO:tasks.workunit.client.0.smithi016.stdout:2/709: symlink d2/d2c/daa/l10a 0 2024-04-22T15:49:43.085 INFO:tasks.workunit.client.0.smithi016.stdout:3/821: link d7/d17/d18/d32/d42/d63/deb/d21/fbf d7/d17/d18/d32/d42/d63/d86/ffe 0 2024-04-22T15:49:43.086 INFO:tasks.workunit.client.1.smithi027.stdout:0/75: mkdir d0/d1a 0 2024-04-22T15:49:43.087 INFO:tasks.workunit.client.0.smithi016.stdout:1/863: write d3/d4/d3a/d6b/d77/de2/ff2 [747774,100515] 0 2024-04-22T15:49:43.087 INFO:tasks.workunit.client.0.smithi016.stdout:8/679: write d2/d48/dad/d6a/f9a [1581474,125166] 0 2024-04-22T15:49:43.090 INFO:tasks.workunit.client.0.smithi016.stdout:5/741: rename df/d5b/db1 to df/d15/d1a/d50/d77/d7d/d8b/d8c/de4 0 2024-04-22T15:49:43.090 INFO:tasks.workunit.client.0.smithi016.stdout:8/680: write d2/d82/d44/d66/f3c [2942203,47650] 0 2024-04-22T15:49:43.091 INFO:tasks.workunit.client.0.smithi016.stdout:5/742: chown df/d90/dad 445597 1 2024-04-22T15:49:43.091 INFO:tasks.workunit.client.0.smithi016.stdout:7/725: chown d3/fa 0 1 2024-04-22T15:49:43.092 INFO:tasks.workunit.client.0.smithi016.stdout:7/726: fsync d3/dc/fbf 0 2024-04-22T15:49:43.093 INFO:tasks.workunit.client.1.smithi027.stdout:0/76: unlink d0/c5 0 2024-04-22T15:49:43.094 INFO:tasks.workunit.client.0.smithi016.stdout:7/727: dread - d3/de6/fec zero size 2024-04-22T15:49:43.094 INFO:tasks.workunit.client.0.smithi016.stdout:6/771: rmdir d3/da/dd/d1b 39 2024-04-22T15:49:43.096 INFO:tasks.workunit.client.0.smithi016.stdout:2/710: mknod d2/de/d2a/d59/c10b 0 2024-04-22T15:49:43.100 INFO:tasks.workunit.client.1.smithi027.stdout:0/77: dwrite d0/f12 [0,4194304] 0 2024-04-22T15:49:43.101 INFO:tasks.workunit.client.1.smithi027.stdout:0/78: fsync d0/fc 0 2024-04-22T15:49:43.101 INFO:tasks.workunit.client.0.smithi016.stdout:3/822: rename d7/d17/d18/f23 to d7/d17/d18/d32/d3c/d47/dc3/fff 0 2024-04-22T15:49:43.104 INFO:tasks.workunit.client.0.smithi016.stdout:0/801: dread dc/d13/f16 [0,4194304] 0 2024-04-22T15:49:43.105 INFO:tasks.workunit.client.1.smithi027.stdout:0/79: fdatasync d0/f1 0 2024-04-22T15:49:43.110 INFO:tasks.workunit.client.0.smithi016.stdout:6/772: fdatasync d3/da/db/f61 0 2024-04-22T15:49:43.112 INFO:tasks.workunit.client.0.smithi016.stdout:1/864: mknod d3/d4/d3a/d9e/d124/c129 0 2024-04-22T15:49:43.112 INFO:tasks.workunit.client.1.smithi027.stdout:0/80: dwrite d0/fc [0,4194304] 0 2024-04-22T15:49:43.114 INFO:tasks.workunit.client.0.smithi016.stdout:1/865: stat d3/d4/d3a/d91/fb5 0 2024-04-22T15:49:43.119 INFO:tasks.workunit.client.0.smithi016.stdout:1/866: truncate d3/d4/d3a/d91/db3/dd2/ff3 447736 0 2024-04-22T15:49:43.120 INFO:tasks.workunit.client.0.smithi016.stdout:1/867: readlink d3/d4/d15/l30 0 2024-04-22T15:49:43.121 INFO:tasks.workunit.client.0.smithi016.stdout:1/868: readlink d3/d4/d15/d38/lcf 0 2024-04-22T15:49:43.122 INFO:tasks.workunit.client.0.smithi016.stdout:0/802: fdatasync dc/d13/d14/d26/f5e 0 2024-04-22T15:49:43.126 INFO:tasks.workunit.client.0.smithi016.stdout:6/773: mkdir d3/da/dd/d67/d6c/db9/d100 0 2024-04-22T15:49:43.128 INFO:tasks.workunit.client.0.smithi016.stdout:5/743: creat df/d15/d1a/fe5 x:0 0 0 2024-04-22T15:49:43.128 INFO:tasks.workunit.client.0.smithi016.stdout:2/711: fsync d2/d23/d92/d55/fa7 0 2024-04-22T15:49:43.129 INFO:tasks.workunit.client.0.smithi016.stdout:4/717: getdents d7/d24 0 2024-04-22T15:49:43.131 INFO:tasks.workunit.client.0.smithi016.stdout:3/823: dread d7/d17/d18/d32/d42/d63/deb/d21/d55/f79 [0,4194304] 0 2024-04-22T15:49:43.135 INFO:tasks.workunit.client.0.smithi016.stdout:2/712: write d2/d1a/d4c/d9f/dae/fb3 [5407816,39196] 0 2024-04-22T15:49:43.135 INFO:tasks.workunit.client.0.smithi016.stdout:1/869: creat d3/d28/dee/f12a x:0 0 0 2024-04-22T15:49:43.135 INFO:tasks.workunit.client.0.smithi016.stdout:9/742: sync 2024-04-22T15:49:43.135 INFO:tasks.workunit.client.0.smithi016.stdout:8/681: truncate d2/f24 1206345 0 2024-04-22T15:49:43.136 INFO:tasks.workunit.client.0.smithi016.stdout:7/728: dread d3/f28 [0,4194304] 0 2024-04-22T15:49:43.137 INFO:tasks.workunit.client.0.smithi016.stdout:1/870: chown d3/d4/d3a/d9e/d124 3237 1 2024-04-22T15:49:43.139 INFO:tasks.workunit.client.0.smithi016.stdout:3/824: fdatasync d7/d17/d18/f70 0 2024-04-22T15:49:43.140 INFO:tasks.workunit.client.0.smithi016.stdout:8/682: write d2/d48/dad/d6a/f9a [643050,108032] 0 2024-04-22T15:49:43.140 INFO:tasks.workunit.client.0.smithi016.stdout:0/803: rename dc/d13/d14/de5/d65/d40/d4a/d61/d82 to dc/d13/d14/de5/d65/d40/dd6/d104 0 2024-04-22T15:49:43.142 INFO:tasks.workunit.client.1.smithi027.stdout:0/81: dwrite d0/f10 [4194304,4194304] 0 2024-04-22T15:49:43.142 INFO:tasks.workunit.client.0.smithi016.stdout:4/718: mknod d7/d4e/d2f/cec 0 2024-04-22T15:49:43.144 INFO:tasks.workunit.client.0.smithi016.stdout:2/713: mknod d2/d4d/d76/c10c 0 2024-04-22T15:49:43.145 INFO:tasks.workunit.client.0.smithi016.stdout:4/719: stat d7/d21/d45/d49/d9b/fdf 0 2024-04-22T15:49:43.145 INFO:tasks.workunit.client.0.smithi016.stdout:2/714: fdatasync d2/d23/d92/d55/fa6 0 2024-04-22T15:49:43.150 INFO:tasks.workunit.client.0.smithi016.stdout:4/720: dread d7/d4e/d2f/d6f/fa8 [0,4194304] 0 2024-04-22T15:49:43.150 INFO:tasks.workunit.client.0.smithi016.stdout:5/744: creat df/d15/d1a/d50/d77/d7d/d8b/d8c/de4/fe6 x:0 0 0 2024-04-22T15:49:43.154 INFO:tasks.workunit.client.0.smithi016.stdout:6/774: dwrite d3/da/db/d44/faf [0,4194304] 0 2024-04-22T15:49:43.156 INFO:tasks.workunit.client.0.smithi016.stdout:8/683: symlink d2/d82/d3e/lce 0 2024-04-22T15:49:43.157 INFO:tasks.workunit.client.0.smithi016.stdout:1/871: symlink d3/d2e/d80/d82/l12b 0 2024-04-22T15:49:43.160 INFO:tasks.workunit.client.0.smithi016.stdout:2/715: symlink d2/d2c/d97/d67/dbf/l10d 0 2024-04-22T15:49:43.163 INFO:tasks.workunit.client.0.smithi016.stdout:4/721: creat d7/d24/fed x:0 0 0 2024-04-22T15:49:43.163 INFO:tasks.workunit.client.0.smithi016.stdout:5/745: rmdir df/d15/d1a/d50/d77/d94 39 2024-04-22T15:49:43.164 INFO:tasks.workunit.client.0.smithi016.stdout:0/804: mkdir dc/d13/d14/d105 0 2024-04-22T15:49:43.165 INFO:tasks.workunit.client.0.smithi016.stdout:0/805: chown dc/d13/d99/c9c 75071857 1 2024-04-22T15:49:43.167 INFO:tasks.workunit.client.1.smithi027.stdout:0/82: dwrite d0/f1 [0,4194304] 0 2024-04-22T15:49:43.170 INFO:tasks.workunit.client.1.smithi027.stdout:0/83: write d0/fc [4739175,116882] 0 2024-04-22T15:49:43.170 INFO:tasks.workunit.client.0.smithi016.stdout:2/716: truncate d2/d2c/d97/f5f 1003014 0 2024-04-22T15:49:43.171 INFO:tasks.workunit.client.0.smithi016.stdout:2/717: stat d2/d2c/d97/c3a 0 2024-04-22T15:49:43.173 INFO:tasks.workunit.client.0.smithi016.stdout:9/743: rename d9/d57/d91/de4/d25/d36/f6b to d9/de/d26/fe9 0 2024-04-22T15:49:43.173 INFO:tasks.workunit.client.0.smithi016.stdout:7/729: dwrite d3/dc/d1b/d5a/d9/df/d94/ddf/fe2 [0,4194304] 0 2024-04-22T15:49:43.176 INFO:tasks.workunit.client.0.smithi016.stdout:2/718: chown d2/de/d2a/d59/fbe 10644592 1 2024-04-22T15:49:43.177 INFO:tasks.workunit.client.0.smithi016.stdout:6/775: mkdir d3/da/ddd/d101 0 2024-04-22T15:49:43.178 INFO:tasks.workunit.client.1.smithi027.stdout:0/84: mknod d0/c1b 0 2024-04-22T15:49:43.179 INFO:tasks.workunit.client.0.smithi016.stdout:0/806: unlink dc/d13/d14/d26/f4f 0 2024-04-22T15:49:43.180 INFO:tasks.workunit.client.0.smithi016.stdout:1/872: mknod d3/d4/d3a/d6b/df7/c12c 0 2024-04-22T15:49:43.181 INFO:tasks.workunit.client.1.smithi027.stdout:0/85: chown d0/f12 3799 1 2024-04-22T15:49:43.187 INFO:tasks.workunit.client.0.smithi016.stdout:2/719: creat d2/d9/f10e x:0 0 0 2024-04-22T15:49:43.190 INFO:tasks.workunit.client.0.smithi016.stdout:7/730: creat d3/dc/d33/d84/f106 x:0 0 0 2024-04-22T15:49:43.190 INFO:tasks.workunit.client.1.smithi027.stdout:0/86: creat d0/d1a/f1c x:0 0 0 2024-04-22T15:49:43.191 INFO:tasks.workunit.client.0.smithi016.stdout:3/825: getdents d7/d17/d18/d32/d42/d63/d71 0 2024-04-22T15:49:43.193 INFO:tasks.workunit.client.0.smithi016.stdout:4/722: dwrite d7/d24/d51/fbb [0,4194304] 0 2024-04-22T15:49:43.194 INFO:tasks.workunit.client.1.smithi027.stdout:0/87: symlink d0/d1a/l1d 0 2024-04-22T15:49:43.195 INFO:tasks.workunit.client.0.smithi016.stdout:9/744: dread d9/d57/d91/dc1/f75 [0,4194304] 0 2024-04-22T15:49:43.197 INFO:tasks.workunit.client.0.smithi016.stdout:3/826: dread d7/d17/d18/d32/d42/d63/deb/f85 [0,4194304] 0 2024-04-22T15:49:43.199 INFO:tasks.workunit.client.0.smithi016.stdout:5/746: rename df/d22/f24 to df/d5b/d70/fe7 0 2024-04-22T15:49:43.200 INFO:tasks.workunit.client.0.smithi016.stdout:5/747: chown df/d15/d1a/d4e/d58/d87/fd3 1769458 1 2024-04-22T15:49:43.200 INFO:tasks.workunit.client.0.smithi016.stdout:9/745: read d9/d57/d91/de4/d25/da6/f79 [2711373,98203] 0 2024-04-22T15:49:43.202 INFO:tasks.workunit.client.0.smithi016.stdout:6/776: creat d3/da/dd/d1b/d57/f102 x:0 0 0 2024-04-22T15:49:43.202 INFO:tasks.workunit.client.0.smithi016.stdout:5/748: write df/f76 [932757,39110] 0 2024-04-22T15:49:43.204 INFO:tasks.workunit.client.0.smithi016.stdout:5/749: dread - df/d22/fca zero size 2024-04-22T15:49:43.207 INFO:tasks.workunit.client.0.smithi016.stdout:7/731: mknod d3/dc/d1b/d31/d5f/c107 0 2024-04-22T15:49:43.209 INFO:tasks.workunit.client.0.smithi016.stdout:1/873: getdents d3/d4/d15 0 2024-04-22T15:49:43.210 INFO:tasks.workunit.client.0.smithi016.stdout:2/720: write d2/de/d2a/d59/f78 [3207636,89716] 0 2024-04-22T15:49:43.210 INFO:tasks.workunit.client.0.smithi016.stdout:6/777: mknod d3/d53/de9/c103 0 2024-04-22T15:49:43.212 INFO:tasks.workunit.client.0.smithi016.stdout:5/750: mknod df/d15/d1a/d50/d77/ce8 0 2024-04-22T15:49:43.214 INFO:tasks.workunit.client.0.smithi016.stdout:0/807: dwrite dc/d13/d14/de5/f50 [0,4194304] 0 2024-04-22T15:49:43.215 INFO:tasks.workunit.client.0.smithi016.stdout:7/732: symlink d3/dc/d1b/d5a/d4a/l108 0 2024-04-22T15:49:43.219 INFO:tasks.workunit.client.0.smithi016.stdout:0/808: chown dc/d13/d14/de5/dd8/ced 1068581883 1 2024-04-22T15:49:43.221 INFO:tasks.workunit.client.0.smithi016.stdout:5/751: write df/d15/f45 [3494082,20185] 0 2024-04-22T15:49:43.223 INFO:tasks.workunit.client.0.smithi016.stdout:1/874: rename d3/d4/d3a/d6b/d77/de2 to d3/d4/d19/d20/d2b/dd6/d12d 0 2024-04-22T15:49:43.224 INFO:tasks.workunit.client.0.smithi016.stdout:9/746: dread d9/d5d/d62/f6e [4194304,4194304] 0 2024-04-22T15:49:43.226 INFO:tasks.workunit.client.0.smithi016.stdout:2/721: dwrite d2/d23/ff6 [0,4194304] 0 2024-04-22T15:49:43.230 INFO:tasks.workunit.client.0.smithi016.stdout:7/733: dread d3/dc/f9a [0,4194304] 0 2024-04-22T15:49:43.230 INFO:tasks.workunit.client.0.smithi016.stdout:1/875: rmdir d3/d4/d3a/d91/db3 39 2024-04-22T15:49:43.231 INFO:tasks.workunit.client.0.smithi016.stdout:5/752: truncate df/d15/d1a/d50/d77/d7d/d8b/d93/fe0 982558 0 2024-04-22T15:49:43.231 INFO:tasks.workunit.client.0.smithi016.stdout:2/722: symlink d2/d23/db8/l10f 0 2024-04-22T15:49:43.233 INFO:tasks.workunit.client.0.smithi016.stdout:0/809: link dc/d13/d6f/d8b/ffe dc/d13/d14/de5/d65/d30/d75/f106 0 2024-04-22T15:49:43.235 INFO:tasks.workunit.client.0.smithi016.stdout:6/778: truncate d3/da/dd/f10 3707088 0 2024-04-22T15:49:43.237 INFO:tasks.workunit.client.0.smithi016.stdout:6/779: chown d3/da/dd 250320 1 2024-04-22T15:49:43.238 INFO:tasks.workunit.client.0.smithi016.stdout:2/723: mkdir d2/d2c/daa/d110 0 2024-04-22T15:49:43.241 INFO:tasks.workunit.client.0.smithi016.stdout:7/734: dwrite d3/dc/d9f/fb5 [0,4194304] 0 2024-04-22T15:49:43.243 INFO:tasks.workunit.client.0.smithi016.stdout:2/724: creat d2/d4d/d64/f111 x:0 0 0 2024-04-22T15:49:43.247 INFO:tasks.workunit.client.0.smithi016.stdout:5/753: creat df/d15/d1a/d50/d77/d7d/d8b/d8c/ddf/fe9 x:0 0 0 2024-04-22T15:49:43.247 INFO:tasks.workunit.client.0.smithi016.stdout:2/725: stat d2/d2c/d97/d67 0 2024-04-22T15:49:43.252 INFO:tasks.workunit.client.0.smithi016.stdout:6/780: read d3/da/dd/d1b/f85 [4049939,23212] 0 2024-04-22T15:49:43.254 INFO:tasks.workunit.client.0.smithi016.stdout:0/810: dwrite dc/d13/f16 [0,4194304] 0 2024-04-22T15:49:43.255 INFO:tasks.workunit.client.0.smithi016.stdout:9/747: dread d9/d57/d91/de4/f2e [0,4194304] 0 2024-04-22T15:49:43.255 INFO:tasks.workunit.client.0.smithi016.stdout:5/754: dread - df/d15/d1a/d4e/d58/d87/f9e zero size 2024-04-22T15:49:43.258 INFO:tasks.workunit.client.0.smithi016.stdout:7/735: mkdir d3/dc/d1b/d5a/d9/d109 0 2024-04-22T15:49:43.260 INFO:tasks.workunit.client.0.smithi016.stdout:9/748: fsync d9/de/d23/f34 0 2024-04-22T15:49:43.260 INFO:tasks.workunit.client.0.smithi016.stdout:1/876: dwrite d3/d4/d3a/d91/fb7 [0,4194304] 0 2024-04-22T15:49:43.260 INFO:tasks.workunit.client.0.smithi016.stdout:2/726: creat d2/d4d/dc4/f112 x:0 0 0 2024-04-22T15:49:43.261 INFO:tasks.workunit.client.0.smithi016.stdout:6/781: truncate d3/da/f45 4216916 0 2024-04-22T15:49:43.263 INFO:tasks.workunit.client.0.smithi016.stdout:7/736: truncate d3/dc/d33/d84/f106 762684 0 2024-04-22T15:49:43.265 INFO:tasks.workunit.client.0.smithi016.stdout:1/877: read d3/d4/d15/f16 [3588247,37181] 0 2024-04-22T15:49:43.265 INFO:tasks.workunit.client.0.smithi016.stdout:9/749: mkdir d9/d37/dea 0 2024-04-22T15:49:43.266 INFO:tasks.workunit.client.0.smithi016.stdout:6/782: chown d3/d53/d35/fa2 284447612 1 2024-04-22T15:49:43.269 INFO:tasks.workunit.client.0.smithi016.stdout:5/755: symlink df/d5b/db3/dc1/lea 0 2024-04-22T15:49:43.270 INFO:tasks.workunit.client.0.smithi016.stdout:1/878: dread - d3/d4/d15/f11c zero size 2024-04-22T15:49:43.270 INFO:tasks.workunit.client.0.smithi016.stdout:2/727: dread d2/d4d/d64/fe6 [0,4194304] 0 2024-04-22T15:49:43.270 INFO:tasks.workunit.client.0.smithi016.stdout:6/783: stat d3/da/dd/d67 0 2024-04-22T15:49:43.274 INFO:tasks.workunit.client.0.smithi016.stdout:0/811: dwrite dc/d13/d14/d55/fb8 [0,4194304] 0 2024-04-22T15:49:43.274 INFO:tasks.workunit.client.0.smithi016.stdout:7/737: unlink d3/dc/d1b/d5a/d9/df/d94/lf9 0 2024-04-22T15:49:43.277 INFO:tasks.workunit.client.0.smithi016.stdout:1/879: readlink d3/d4/d3a/d91/db3/lf4 0 2024-04-22T15:49:43.279 INFO:tasks.workunit.client.0.smithi016.stdout:9/750: getdents d9/d57/d91/de4/d25/d36 0 2024-04-22T15:49:43.280 INFO:tasks.workunit.client.0.smithi016.stdout:1/880: stat d3/d4/d3a/d91/db3/dd2 0 2024-04-22T15:49:43.284 INFO:tasks.workunit.client.0.smithi016.stdout:1/881: chown d3/d4/d19/lef 617 1 2024-04-22T15:49:43.284 INFO:tasks.workunit.client.0.smithi016.stdout:7/738: mknod d3/de6/c10a 0 2024-04-22T15:49:43.286 INFO:tasks.workunit.client.0.smithi016.stdout:2/728: dwrite d2/d2c/d97/d67/f75 [0,4194304] 0 2024-04-22T15:49:43.286 INFO:tasks.workunit.client.0.smithi016.stdout:1/882: fdatasync d3/d4/d19/d20/f60 0 2024-04-22T15:49:43.286 INFO:tasks.workunit.client.0.smithi016.stdout:9/751: mknod d9/d57/da7/daa/ceb 0 2024-04-22T15:49:43.287 INFO:tasks.workunit.client.0.smithi016.stdout:6/784: truncate d3/da/db/d14/d4c/f52 3240739 0 2024-04-22T15:49:43.290 INFO:tasks.workunit.client.0.smithi016.stdout:1/883: creat d3/d4/dc4/f12e x:0 0 0 2024-04-22T15:49:43.291 INFO:tasks.workunit.client.0.smithi016.stdout:9/752: fsync d9/de/d26/fe9 0 2024-04-22T15:49:43.291 INFO:tasks.workunit.client.0.smithi016.stdout:2/729: creat d2/d1a/d4c/d77/dea/f113 x:0 0 0 2024-04-22T15:49:43.292 INFO:tasks.workunit.client.0.smithi016.stdout:5/756: dread df/d5b/d70/fe7 [0,4194304] 0 2024-04-22T15:49:43.292 INFO:tasks.workunit.client.0.smithi016.stdout:7/739: mkdir d3/d10b 0 2024-04-22T15:49:43.293 INFO:tasks.workunit.client.0.smithi016.stdout:1/884: write d3/d4/d3a/d91/fb5 [903375,19286] 0 2024-04-22T15:49:43.294 INFO:tasks.workunit.client.0.smithi016.stdout:6/785: mknod d3/da/d12/d13/d97/de0/c104 0 2024-04-22T15:49:43.296 INFO:tasks.workunit.client.0.smithi016.stdout:6/786: write d3/da/dd/f18 [5083040,61183] 0 2024-04-22T15:49:43.299 INFO:tasks.workunit.client.0.smithi016.stdout:0/812: truncate dc/d13/d14/de5/d65/d40/d4a/d61/fe4 199357 0 2024-04-22T15:49:43.300 INFO:tasks.workunit.client.0.smithi016.stdout:9/753: chown d9/d37/l8c 30434 1 2024-04-22T15:49:43.301 INFO:tasks.workunit.client.0.smithi016.stdout:5/757: unlink df/d15/d1a/d50/d77/d7a/f80 0 2024-04-22T15:49:43.302 INFO:tasks.workunit.client.0.smithi016.stdout:7/740: mknod d3/c10c 0 2024-04-22T15:49:43.302 INFO:tasks.workunit.client.0.smithi016.stdout:0/813: dread - dc/d13/d14/de5/d65/d57/dcb/fd7 zero size 2024-04-22T15:49:43.305 INFO:tasks.workunit.client.0.smithi016.stdout:6/787: creat d3/da/d12/d13/dd2/dee/f105 x:0 0 0 2024-04-22T15:49:43.306 INFO:tasks.workunit.client.0.smithi016.stdout:6/788: fsync d3/da/d12/d13/f78 0 2024-04-22T15:49:43.309 INFO:tasks.workunit.client.0.smithi016.stdout:1/885: mknod d3/d28/dbd/c12f 0 2024-04-22T15:49:43.311 INFO:tasks.workunit.client.0.smithi016.stdout:0/814: rename dc/d13/d14/de5/d65/d57/fa2 to dc/d13/d14/de5/d65/d40/d4a/d59/d78/dbd/f107 0 2024-04-22T15:49:43.316 INFO:tasks.workunit.client.0.smithi016.stdout:1/886: dread d3/d4/d19/d20/f60 [4194304,4194304] 0 2024-04-22T15:49:43.317 INFO:tasks.workunit.client.0.smithi016.stdout:0/815: write dc/d13/d14/de5/d65/d40/d4a/d59/d78/dbd/f107 [1278132,114968] 0 2024-04-22T15:49:43.317 INFO:tasks.workunit.client.0.smithi016.stdout:2/730: truncate d2/f7 600244 0 2024-04-22T15:49:43.317 INFO:tasks.workunit.client.0.smithi016.stdout:5/758: mkdir df/d5b/d5d/deb 0 2024-04-22T15:49:43.318 INFO:tasks.workunit.client.0.smithi016.stdout:0/816: stat dc/d13/d14/de5/d65/d1f/d48 0 2024-04-22T15:49:43.319 INFO:tasks.workunit.client.0.smithi016.stdout:6/789: link d3/da/db/d14/d16/f20 d3/da/dd/d67/d6c/db9/f106 0 2024-04-22T15:49:43.320 INFO:tasks.workunit.client.0.smithi016.stdout:9/754: write d9/d57/d91/dc1/f58 [609115,4908] 0 2024-04-22T15:49:43.323 INFO:tasks.workunit.client.0.smithi016.stdout:1/887: creat d3/d4/f130 x:0 0 0 2024-04-22T15:49:43.325 INFO:tasks.workunit.client.0.smithi016.stdout:6/790: stat d3/da/db/d44/d68/d95/d9e 0 2024-04-22T15:49:43.327 INFO:tasks.workunit.client.0.smithi016.stdout:9/755: write d9/d57/d91/de4/d25/fe2 [179107,58602] 0 2024-04-22T15:49:43.332 INFO:tasks.workunit.client.0.smithi016.stdout:6/791: fdatasync d3/da/db/d44/d68/d95/fcf 0 2024-04-22T15:49:43.335 INFO:tasks.workunit.client.0.smithi016.stdout:5/759: dwrite df/d15/d1a/d50/d77/d7d/d8b/d93/fe0 [0,4194304] 0 2024-04-22T15:49:43.337 INFO:tasks.workunit.client.0.smithi016.stdout:7/741: dwrite d3/dc/d1b/d31/d5f/f7e [0,4194304] 0 2024-04-22T15:49:43.340 INFO:tasks.workunit.client.0.smithi016.stdout:0/817: dwrite dc/d13/d14/de5/d65/d1f/f53 [0,4194304] 0 2024-04-22T15:49:43.340 INFO:tasks.workunit.client.0.smithi016.stdout:1/888: write d3/d4/d19/d20/f21 [2100937,71515] 0 2024-04-22T15:49:43.346 INFO:tasks.workunit.client.0.smithi016.stdout:1/889: fsync d3/d4/d3a/d6b/dc3/ff0 0 2024-04-22T15:49:43.346 INFO:tasks.workunit.client.0.smithi016.stdout:5/760: symlink df/d15/d1a/d4e/d58/lec 0 2024-04-22T15:49:43.346 INFO:tasks.workunit.client.0.smithi016.stdout:6/792: mkdir d3/d53/d107 0 2024-04-22T15:49:43.350 INFO:tasks.workunit.client.0.smithi016.stdout:9/756: rename d9/d57/d91/dc1/f58 to d9/de/fec 0 2024-04-22T15:49:43.351 INFO:tasks.workunit.client.0.smithi016.stdout:1/890: mknod d3/d4/da/d7e/c131 0 2024-04-22T15:49:43.354 INFO:tasks.workunit.client.0.smithi016.stdout:9/757: write d9/d57/d91/de4/d25/da6/f98 [743970,42440] 0 2024-04-22T15:49:43.357 INFO:tasks.workunit.client.0.smithi016.stdout:0/818: rename dc/d13/d5d/f85 to dc/d13/d14/d55/f108 0 2024-04-22T15:49:43.358 INFO:tasks.workunit.client.0.smithi016.stdout:6/793: link d3/da/db/d14/d16/cc6 d3/da/db/d44/d68/d95/d9e/c108 0 2024-04-22T15:49:43.366 INFO:tasks.workunit.client.0.smithi016.stdout:0/819: creat dc/d13/d14/d55/dba/f109 x:0 0 0 2024-04-22T15:49:43.369 INFO:tasks.workunit.client.0.smithi016.stdout:6/794: dwrite d3/da/f3e [0,4194304] 0 2024-04-22T15:49:43.371 INFO:tasks.workunit.client.0.smithi016.stdout:1/891: dread d3/d4/fe [0,4194304] 0 2024-04-22T15:49:43.373 INFO:tasks.workunit.client.0.smithi016.stdout:9/758: dwrite d9/d57/d91/de4/f8e [0,4194304] 0 2024-04-22T15:49:43.378 INFO:tasks.workunit.client.0.smithi016.stdout:5/761: dread df/d5b/d5d/d5f/f6f [0,4194304] 0 2024-04-22T15:49:43.378 INFO:tasks.workunit.client.0.smithi016.stdout:1/892: write d3/d4/d15/d38/d63/f75 [4556007,27016] 0 2024-04-22T15:49:43.378 INFO:tasks.workunit.client.0.smithi016.stdout:6/795: truncate d3/da/f69 541282 0 2024-04-22T15:49:43.379 INFO:tasks.workunit.client.0.smithi016.stdout:7/742: dwrite d3/dc/d1b/d5a/d69/f9d [0,4194304] 0 2024-04-22T15:49:43.382 INFO:tasks.workunit.client.0.smithi016.stdout:9/759: fdatasync d9/f1b 0 2024-04-22T15:49:43.383 INFO:tasks.workunit.client.0.smithi016.stdout:1/893: fdatasync d3/d4/d19/d20/f21 0 2024-04-22T15:49:43.385 INFO:tasks.workunit.client.0.smithi016.stdout:6/796: write d3/da/d12/d13/d97/de0/fe3 [1449593,23907] 0 2024-04-22T15:49:43.386 INFO:tasks.workunit.client.0.smithi016.stdout:7/743: creat d3/dc/d1b/d5a/d69/def/f10d x:0 0 0 2024-04-22T15:49:43.388 INFO:tasks.workunit.client.0.smithi016.stdout:5/762: mknod df/d5b/d5d/ced 0 2024-04-22T15:49:43.389 INFO:tasks.workunit.client.0.smithi016.stdout:0/820: dwrite dc/d13/d14/de5/d65/d40/fd9 [0,4194304] 0 2024-04-22T15:49:43.392 INFO:tasks.workunit.client.0.smithi016.stdout:0/821: fdatasync dc/d13/d14/de5/d65/d57/dcb/fd7 0 2024-04-22T15:49:43.392 INFO:tasks.workunit.client.0.smithi016.stdout:1/894: creat d3/d4/d19/d20/d2b/de0/f132 x:0 0 0 2024-04-22T15:49:43.393 INFO:tasks.workunit.client.0.smithi016.stdout:6/797: creat d3/da/d12/d13/d38/df5/f109 x:0 0 0 2024-04-22T15:49:43.393 INFO:tasks.workunit.client.0.smithi016.stdout:7/744: mknod d3/dc/d1b/d31/d88/c10e 0 2024-04-22T15:49:43.394 INFO:tasks.workunit.client.0.smithi016.stdout:5/763: rmdir df/d5b/d5d/d5f 39 2024-04-22T15:49:43.396 INFO:tasks.workunit.client.0.smithi016.stdout:9/760: getdents d9/d5d/d62/dd2 0 2024-04-22T15:49:43.397 INFO:tasks.workunit.client.0.smithi016.stdout:0/822: creat dc/d13/de7/f10a x:0 0 0 2024-04-22T15:49:43.400 INFO:tasks.workunit.client.0.smithi016.stdout:5/764: creat df/d5b/d5d/d5f/fee x:0 0 0 2024-04-22T15:49:43.401 INFO:tasks.workunit.client.0.smithi016.stdout:9/761: truncate d9/d57/d91/de4/d25/d36/d87/fc6 344749 0 2024-04-22T15:49:43.401 INFO:tasks.workunit.client.0.smithi016.stdout:5/765: fdatasync df/d5b/d5d/fc5 0 2024-04-22T15:49:43.403 INFO:tasks.workunit.client.0.smithi016.stdout:1/895: getdents d3/d4/d19/d20/d2b/dd6/d12d 0 2024-04-22T15:49:43.404 INFO:tasks.workunit.client.0.smithi016.stdout:9/762: rmdir d9/d20/dd7 39 2024-04-22T15:49:43.406 INFO:tasks.workunit.client.0.smithi016.stdout:9/763: write d9/d20/fdd [180378,59061] 0 2024-04-22T15:49:43.408 INFO:tasks.workunit.client.0.smithi016.stdout:1/896: mkdir d3/d4/d3a/d96/de4/d133 0 2024-04-22T15:49:43.410 INFO:tasks.workunit.client.0.smithi016.stdout:5/766: write df/d15/d1a/d50/d77/d7d/fc0 [535006,46303] 0 2024-04-22T15:49:43.410 INFO:tasks.workunit.client.0.smithi016.stdout:9/764: creat d9/d37/fed x:0 0 0 2024-04-22T15:49:43.412 INFO:tasks.workunit.client.0.smithi016.stdout:5/767: fdatasync df/d5b/d5d/d5f/fd7 0 2024-04-22T15:49:43.413 INFO:tasks.workunit.client.0.smithi016.stdout:5/768: chown df/ld6 6277298 1 2024-04-22T15:49:43.414 INFO:tasks.workunit.client.0.smithi016.stdout:9/765: creat d9/d37/dea/fee x:0 0 0 2024-04-22T15:49:43.414 INFO:tasks.workunit.client.0.smithi016.stdout:0/823: dread dc/d13/d14/de5/d65/d40/fac [0,4194304] 0 2024-04-22T15:49:43.422 INFO:tasks.workunit.client.0.smithi016.stdout:9/766: rename d9/d5d/d62 to d9/de/d23/ddb/def 0 2024-04-22T15:49:43.423 INFO:tasks.workunit.client.0.smithi016.stdout:0/824: dwrite dc/d13/d14/de5/d65/d1f/f53 [0,4194304] 0 2024-04-22T15:49:43.427 INFO:tasks.workunit.client.0.smithi016.stdout:6/798: dread d3/da/db/d44/d68/f6a [0,4194304] 0 2024-04-22T15:49:43.427 INFO:tasks.workunit.client.0.smithi016.stdout:5/769: dwrite df/d15/d1a/f4b [4194304,4194304] 0 2024-04-22T15:49:43.434 INFO:tasks.workunit.client.0.smithi016.stdout:1/897: dwrite d3/d4/d19/d20/d2b/fc8 [0,4194304] 0 2024-04-22T15:49:43.437 INFO:tasks.workunit.client.0.smithi016.stdout:6/799: unlink d3/da/db/d44/d68/d95/d9e/c108 0 2024-04-22T15:49:43.443 INFO:tasks.workunit.client.0.smithi016.stdout:0/825: getdents dc/d13/d14/de5/d65/d40/dbb 0 2024-04-22T15:49:43.446 INFO:tasks.workunit.client.0.smithi016.stdout:9/767: dwrite d9/d57/d91/dc1/f86 [0,4194304] 0 2024-04-22T15:49:43.447 INFO:tasks.workunit.client.0.smithi016.stdout:6/800: symlink d3/da/dd/d67/d6c/dfc/l10a 0 2024-04-22T15:49:43.447 INFO:tasks.workunit.client.0.smithi016.stdout:0/826: read dc/d13/d14/d26/f7b [567443,42344] 0 2024-04-22T15:49:43.449 INFO:tasks.workunit.client.0.smithi016.stdout:0/827: write dc/d13/d14/d55/fe2 [961068,80090] 0 2024-04-22T15:49:43.452 INFO:tasks.workunit.client.0.smithi016.stdout:6/801: dread d3/da/d12/d13/f76 [0,4194304] 0 2024-04-22T15:49:43.457 INFO:tasks.workunit.client.0.smithi016.stdout:9/768: dwrite d9/de/fdf [0,4194304] 0 2024-04-22T15:49:43.457 INFO:tasks.workunit.client.0.smithi016.stdout:1/898: link d3/c27 d3/d4/c134 0 2024-04-22T15:49:43.458 INFO:tasks.workunit.client.0.smithi016.stdout:0/828: creat dc/d13/d14/de5/d65/d40/d4a/d61/f10b x:0 0 0 2024-04-22T15:49:43.460 INFO:tasks.workunit.client.0.smithi016.stdout:5/770: truncate df/d15/d1a/d50/d77/d7d/d8b/d93/fe0 2510751 0 2024-04-22T15:49:43.461 INFO:tasks.workunit.client.0.smithi016.stdout:9/769: readlink l0 0 2024-04-22T15:49:43.462 INFO:tasks.workunit.client.0.smithi016.stdout:6/802: rename d3/d53/dab/cf0 to d3/da/db/d44/d68/d95/d9e/c10b 0 2024-04-22T15:49:43.464 INFO:tasks.workunit.client.0.smithi016.stdout:6/803: readlink d3/da/dd/l4e 0 2024-04-22T15:49:43.464 INFO:tasks.workunit.client.0.smithi016.stdout:0/829: rename dc/d13 to dc/d13/d99/d10c 22 2024-04-22T15:49:43.469 INFO:tasks.workunit.client.0.smithi016.stdout:0/830: symlink dc/d13/d14/de5/d65/d57/l10d 0 2024-04-22T15:49:43.470 INFO:tasks.workunit.client.0.smithi016.stdout:5/771: dwrite df/d15/d1a/d50/d77/fb4 [0,4194304] 0 2024-04-22T15:49:43.475 INFO:tasks.workunit.client.0.smithi016.stdout:5/772: dread - df/d15/d1a/d50/d77/f67 zero size 2024-04-22T15:49:43.475 INFO:tasks.workunit.client.0.smithi016.stdout:9/770: creat d9/d57/d91/de4/d25/da6/ff0 x:0 0 0 2024-04-22T15:49:43.478 INFO:tasks.workunit.client.0.smithi016.stdout:6/804: dwrite d3/da/db/d44/d68/faa [0,4194304] 0 2024-04-22T15:49:43.481 INFO:tasks.workunit.client.0.smithi016.stdout:9/771: mknod d9/de/d23/ddb/def/cf1 0 2024-04-22T15:49:43.481 INFO:tasks.workunit.client.0.smithi016.stdout:1/899: dwrite d3/d4/d37/f41 [0,4194304] 0 2024-04-22T15:49:43.484 INFO:tasks.workunit.client.0.smithi016.stdout:6/805: mkdir d3/da/db/d44/d68/d95/d10c 0 2024-04-22T15:49:43.484 INFO:tasks.workunit.client.0.smithi016.stdout:9/772: write d9/d37/fed [63316,17731] 0 2024-04-22T15:49:43.485 INFO:tasks.workunit.client.0.smithi016.stdout:9/773: fdatasync d9/d57/f9b 0 2024-04-22T15:49:43.487 INFO:tasks.workunit.client.0.smithi016.stdout:5/773: truncate df/d15/d1a/d50/d77/d7a/fcb 1813665 0 2024-04-22T15:49:43.488 INFO:tasks.workunit.client.0.smithi016.stdout:9/774: chown d9/d57/da7/daa/fc9 1651755 1 2024-04-22T15:49:43.491 INFO:tasks.workunit.client.0.smithi016.stdout:5/774: creat df/d15/d1a/d50/d77/d7a/fef x:0 0 0 2024-04-22T15:49:43.492 INFO:tasks.workunit.client.0.smithi016.stdout:6/806: mkdir d3/da/db/d14/d10d 0 2024-04-22T15:49:43.494 INFO:tasks.workunit.client.0.smithi016.stdout:5/775: chown df/d15/d1a/d50/d77/d7a/fef 512 1 2024-04-22T15:49:43.496 INFO:tasks.workunit.client.0.smithi016.stdout:5/776: write df/d15/d1a/fd2 [391150,55848] 0 2024-04-22T15:49:43.497 INFO:tasks.workunit.client.0.smithi016.stdout:6/807: mkdir d3/da/ddd/d10e 0 2024-04-22T15:49:43.501 INFO:tasks.workunit.client.0.smithi016.stdout:5/777: creat df/d15/d1a/d50/d77/d7a/ff0 x:0 0 0 2024-04-22T15:49:43.502 INFO:tasks.workunit.client.0.smithi016.stdout:5/778: stat f2 0 2024-04-22T15:49:43.503 INFO:tasks.workunit.client.0.smithi016.stdout:6/808: chown d3/da/db/d44/d68/d95/d10c 13151 1 2024-04-22T15:49:43.505 INFO:tasks.workunit.client.0.smithi016.stdout:1/900: dwrite d3/d4/d3a/d6b/f94 [0,4194304] 0 2024-04-22T15:49:43.507 INFO:tasks.workunit.client.0.smithi016.stdout:1/901: readlink d3/d4/d15/d38/d63/lc7 0 2024-04-22T15:49:43.511 INFO:tasks.workunit.client.0.smithi016.stdout:1/902: dread d3/d4/faa [0,4194304] 0 2024-04-22T15:49:43.517 INFO:tasks.workunit.client.0.smithi016.stdout:5/779: dwrite df/d15/d1a/d50/d77/d7d/d8b/d93/fd5 [0,4194304] 0 2024-04-22T15:49:43.518 INFO:tasks.workunit.client.0.smithi016.stdout:9/775: dread d9/de/d26/d5a/fc7 [0,4194304] 0 2024-04-22T15:49:43.519 INFO:tasks.workunit.client.0.smithi016.stdout:1/903: link d3/d4/d19/d20/cf6 d3/d2e/d9f/deb/c135 0 2024-04-22T15:49:43.523 INFO:tasks.workunit.client.0.smithi016.stdout:1/904: mkdir d3/d4/d3a/d96/de4/d133/d136 0 2024-04-22T15:49:43.524 INFO:tasks.workunit.client.0.smithi016.stdout:1/905: fsync d3/f4f 0 2024-04-22T15:49:43.527 INFO:tasks.workunit.client.0.smithi016.stdout:9/776: dread d9/fc [0,4194304] 0 2024-04-22T15:49:43.528 INFO:tasks.workunit.client.0.smithi016.stdout:5/780: dwrite df/d5b/fc4 [0,4194304] 0 2024-04-22T15:49:43.532 INFO:tasks.workunit.client.0.smithi016.stdout:1/906: creat d3/d4/d15/d38/d63/d118/f137 x:0 0 0 2024-04-22T15:49:43.537 INFO:tasks.workunit.client.0.smithi016.stdout:5/781: creat df/ff1 x:0 0 0 2024-04-22T15:49:43.538 INFO:tasks.workunit.client.0.smithi016.stdout:9/777: fdatasync d9/de/d23/f78 0 2024-04-22T15:49:43.538 INFO:tasks.workunit.client.0.smithi016.stdout:5/782: readlink df/d5b/d5d/d5f/lde 0 2024-04-22T15:49:43.541 INFO:tasks.workunit.client.0.smithi016.stdout:1/907: creat d3/d4/d3a/d6b/d77/f138 x:0 0 0 2024-04-22T15:49:43.549 INFO:tasks.workunit.client.0.smithi016.stdout:9/778: mknod d9/d20/dd7/cf2 0 2024-04-22T15:49:43.550 INFO:tasks.workunit.client.0.smithi016.stdout:1/908: symlink d3/d2e/d9f/deb/l139 0 2024-04-22T15:49:43.551 INFO:tasks.workunit.client.0.smithi016.stdout:5/783: dread df/d15/d1a/d28/f3f [0,4194304] 0 2024-04-22T15:49:43.551 INFO:tasks.workunit.client.0.smithi016.stdout:9/779: dread d9/d20/fdd [0,4194304] 0 2024-04-22T15:49:43.552 INFO:tasks.workunit.client.0.smithi016.stdout:9/780: fsync d9/d5d/fc3 0 2024-04-22T15:49:43.553 INFO:tasks.workunit.client.0.smithi016.stdout:9/781: write d9/d37/f49 [4447524,7943] 0 2024-04-22T15:49:43.556 INFO:tasks.workunit.client.0.smithi016.stdout:9/782: creat d9/d20/dac/ff3 x:0 0 0 2024-04-22T15:49:43.558 INFO:tasks.workunit.client.0.smithi016.stdout:5/784: dwrite df/d15/d1a/d50/d77/f4c [0,4194304] 0 2024-04-22T15:49:43.565 INFO:tasks.workunit.client.0.smithi016.stdout:5/785: creat df/d5b/d5d/ff2 x:0 0 0 2024-04-22T15:49:43.566 INFO:tasks.workunit.client.0.smithi016.stdout:5/786: stat df/d15/d1a/d50/d77/d7d/d8b/d8c/ddf/fe9 0 2024-04-22T15:49:43.571 INFO:tasks.workunit.client.0.smithi016.stdout:5/787: dread df/d15/d1a/d50/d77/d7d/d8b/d93/fe0 [0,4194304] 0 2024-04-22T15:49:43.576 INFO:tasks.workunit.client.0.smithi016.stdout:8/684: sync 2024-04-22T15:49:43.576 INFO:tasks.workunit.client.0.smithi016.stdout:4/723: sync 2024-04-22T15:49:43.577 INFO:tasks.workunit.client.0.smithi016.stdout:4/724: dread - d7/d21/d42/d59/d88/d9d/dad/dac/fcc zero size 2024-04-22T15:49:43.580 INFO:tasks.workunit.client.0.smithi016.stdout:5/788: getdents df/d15/d1a/d50 0 2024-04-22T15:49:43.580 INFO:tasks.workunit.client.0.smithi016.stdout:8/685: dread d2/d82/d56/f90 [0,4194304] 0 2024-04-22T15:49:43.581 INFO:tasks.workunit.client.0.smithi016.stdout:4/725: rmdir d7/d4e/d39 39 2024-04-22T15:49:43.582 INFO:tasks.workunit.client.0.smithi016.stdout:4/726: fdatasync d7/d3c/f99 0 2024-04-22T15:49:43.583 INFO:tasks.workunit.client.0.smithi016.stdout:5/789: chown df/d15/d1a/d50/d77/d94 16 1 2024-04-22T15:49:43.585 INFO:tasks.workunit.client.0.smithi016.stdout:9/783: dwrite d9/fa [0,4194304] 0 2024-04-22T15:49:43.586 INFO:tasks.workunit.client.0.smithi016.stdout:8/686: creat d2/d82/d44/d66/d7e/dc0/fcf x:0 0 0 2024-04-22T15:49:43.589 INFO:tasks.workunit.client.0.smithi016.stdout:5/790: creat df/d15/d1a/d50/d77/d7d/d8b/d8c/de4/ff3 x:0 0 0 2024-04-22T15:49:43.592 INFO:tasks.workunit.client.0.smithi016.stdout:5/791: chown df/d15/d1a/d50/d77/d7a/ff0 64503 1 2024-04-22T15:49:43.593 INFO:tasks.workunit.client.0.smithi016.stdout:4/727: dwrite d7/d3c/f44 [0,4194304] 0 2024-04-22T15:49:43.594 INFO:tasks.workunit.client.0.smithi016.stdout:9/784: truncate d9/d57/d91/de4/d25/da6/d4f/fd5 219288 0 2024-04-22T15:49:43.597 INFO:tasks.workunit.client.0.smithi016.stdout:8/687: truncate d2/f24 1499181 0 2024-04-22T15:49:43.601 INFO:tasks.workunit.client.0.smithi016.stdout:4/728: chown d7/l58 228150 1 2024-04-22T15:49:43.603 INFO:tasks.workunit.client.0.smithi016.stdout:9/785: symlink d9/d57/da7/daa/lf4 0 2024-04-22T15:49:43.605 INFO:tasks.workunit.client.0.smithi016.stdout:5/792: creat df/d15/d1a/d50/d77/d7d/d8b/d93/dbb/ff4 x:0 0 0 2024-04-22T15:49:43.606 INFO:tasks.workunit.client.0.smithi016.stdout:5/793: fdatasync df/d15/f45 0 2024-04-22T15:49:43.606 INFO:tasks.workunit.client.0.smithi016.stdout:8/688: chown d2/d10/c5e 9 1 2024-04-22T15:49:43.607 INFO:tasks.workunit.client.0.smithi016.stdout:8/689: dread - d2/d82/d44/d66/d7e/dc0/fcd zero size 2024-04-22T15:49:43.608 INFO:tasks.workunit.client.0.smithi016.stdout:8/690: chown d2/d48/dad/d8f 38509075 1 2024-04-22T15:49:43.609 INFO:tasks.workunit.client.0.smithi016.stdout:4/729: unlink d7/d21/d42/d59/d88/d9d/dad/dac/fd1 0 2024-04-22T15:49:43.611 INFO:tasks.workunit.client.0.smithi016.stdout:4/730: truncate d7/d21/d42/d59/d88/d9d/dad/dac/fcc 31675 0 2024-04-22T15:49:43.612 INFO:tasks.workunit.client.0.smithi016.stdout:5/794: unlink df/d5b/d5d/d5f/c96 0 2024-04-22T15:49:43.613 INFO:tasks.workunit.client.0.smithi016.stdout:8/691: mknod d2/d82/d52/cd0 0 2024-04-22T15:49:43.614 INFO:tasks.workunit.client.0.smithi016.stdout:4/731: rmdir d7/d21/d42/d59/de2 39 2024-04-22T15:49:43.616 INFO:tasks.workunit.client.0.smithi016.stdout:9/786: rename d9/d20/dac/lb3 to d9/de/d26/lf5 0 2024-04-22T15:49:43.618 INFO:tasks.workunit.client.0.smithi016.stdout:8/692: creat d2/d82/fd1 x:0 0 0 2024-04-22T15:49:43.618 INFO:tasks.workunit.client.0.smithi016.stdout:9/787: dread - d9/de/d23/fdc zero size 2024-04-22T15:49:43.620 INFO:tasks.workunit.client.0.smithi016.stdout:5/795: dwrite df/d15/d1a/d50/d77/d7d/d8b/d8c/ddf/fe9 [0,4194304] 0 2024-04-22T15:49:43.620 INFO:tasks.workunit.client.0.smithi016.stdout:4/732: mkdir d7/d21/d42/d59/dee 0 2024-04-22T15:49:43.622 INFO:tasks.workunit.client.0.smithi016.stdout:4/733: write d7/d21/d7c/fe8 [784049,52803] 0 2024-04-22T15:49:43.625 INFO:tasks.workunit.client.0.smithi016.stdout:9/788: dread d9/d20/f56 [0,4194304] 0 2024-04-22T15:49:43.625 INFO:tasks.workunit.client.0.smithi016.stdout:4/734: write d7/d21/d45/d49/d9b/fa2 [487157,43798] 0 2024-04-22T15:49:43.625 INFO:tasks.workunit.client.0.smithi016.stdout:8/693: mknod d2/d10/cd2 0 2024-04-22T15:49:43.629 INFO:tasks.workunit.client.0.smithi016.stdout:9/789: fdatasync d9/d57/d91/de4/f39 0 2024-04-22T15:49:43.634 INFO:tasks.workunit.client.0.smithi016.stdout:8/694: rename d2/d82/d3e/c26 to d2/d82/d44/db0/cd3 0 2024-04-22T15:49:43.640 INFO:tasks.workunit.client.0.smithi016.stdout:9/790: write d9/d57/d91/dc1/f75 [4489386,8929] 0 2024-04-22T15:49:43.641 INFO:tasks.workunit.client.0.smithi016.stdout:9/791: fsync d9/d57/d91/de4/d25/da6/f98 0 2024-04-22T15:49:43.643 INFO:tasks.workunit.client.0.smithi016.stdout:4/735: link d7/d21/d45/d49/fd8 d7/d22/fef 0 2024-04-22T15:49:43.647 INFO:tasks.workunit.client.0.smithi016.stdout:8/695: creat d2/d48/dad/d8f/d96/fd4 x:0 0 0 2024-04-22T15:49:43.647 INFO:tasks.workunit.client.0.smithi016.stdout:8/696: dread - d2/d82/d44/d66/d7e/dc0/fcd zero size 2024-04-22T15:49:43.651 INFO:tasks.workunit.client.0.smithi016.stdout:4/736: mknod d7/d21/d45/d49/cf0 0 2024-04-22T15:49:43.654 INFO:tasks.workunit.client.0.smithi016.stdout:8/697: mkdir d2/d82/d3e/dd5 0 2024-04-22T15:49:43.656 INFO:tasks.workunit.client.0.smithi016.stdout:9/792: link d9/de/fec d9/d57/d91/de4/d25/d36/ff6 0 2024-04-22T15:49:43.659 INFO:tasks.workunit.client.0.smithi016.stdout:9/793: symlink d9/d20/dac/dae/lf7 0 2024-04-22T15:49:43.659 INFO:tasks.workunit.client.0.smithi016.stdout:8/698: dwrite d2/d82/d44/d66/d1c/d65/fbd [0,4194304] 0 2024-04-22T15:49:43.661 INFO:tasks.workunit.client.0.smithi016.stdout:8/699: write d2/d82/d44/d66/f3c [6299337,114712] 0 2024-04-22T15:49:43.665 INFO:tasks.workunit.client.0.smithi016.stdout:9/794: creat d9/d20/dac/dbd/ff8 x:0 0 0 2024-04-22T15:49:43.666 INFO:tasks.workunit.client.0.smithi016.stdout:4/737: link d7/d3c/l9a d7/d21/d42/d59/d88/d9d/lf1 0 2024-04-22T15:49:43.670 INFO:tasks.workunit.client.0.smithi016.stdout:8/700: unlink d2/d82/d56/d99/faa 0 2024-04-22T15:49:43.671 INFO:tasks.workunit.client.0.smithi016.stdout:5/796: dwrite df/d15/d1a/d4e/f72 [0,4194304] 0 2024-04-22T15:49:43.674 INFO:tasks.workunit.client.0.smithi016.stdout:9/795: creat d9/ff9 x:0 0 0 2024-04-22T15:49:43.676 INFO:tasks.workunit.client.0.smithi016.stdout:9/796: write d9/de/fdf [4269535,6023] 0 2024-04-22T15:49:43.676 INFO:tasks.workunit.client.0.smithi016.stdout:4/738: dwrite d7/d21/d45/d49/d9b/fdf [0,4194304] 0 2024-04-22T15:49:43.677 INFO:tasks.workunit.client.0.smithi016.stdout:4/739: chown d7/d4e/d81/l8b 14 1 2024-04-22T15:49:43.678 INFO:tasks.workunit.client.0.smithi016.stdout:4/740: fsync d7/d22/fc9 0 2024-04-22T15:49:43.688 INFO:tasks.workunit.client.0.smithi016.stdout:8/701: rmdir d2/d48/dad/d8f/d96 39 2024-04-22T15:49:43.689 INFO:tasks.workunit.client.0.smithi016.stdout:9/797: mknod d9/d57/d91/de4/cfa 0 2024-04-22T15:49:43.691 INFO:tasks.workunit.client.0.smithi016.stdout:4/741: mkdir d7/d4e/d2f/df2 0 2024-04-22T15:49:43.693 INFO:tasks.workunit.client.0.smithi016.stdout:8/702: rmdir d2/d82/d52 39 2024-04-22T15:49:43.695 INFO:tasks.workunit.client.0.smithi016.stdout:9/798: write d9/d20/f59 [1417198,13721] 0 2024-04-22T15:49:43.701 INFO:tasks.workunit.client.0.smithi016.stdout:9/799: creat d9/d57/d91/de4/d25/d36/d87/ffb x:0 0 0 2024-04-22T15:49:43.704 INFO:tasks.workunit.client.0.smithi016.stdout:3/827: sync 2024-04-22T15:49:43.706 INFO:tasks.workunit.client.0.smithi016.stdout:9/800: symlink d9/d20/dac/lfc 0 2024-04-22T15:49:43.708 INFO:tasks.workunit.client.0.smithi016.stdout:4/742: dwrite d7/d21/d45/f6d [0,4194304] 0 2024-04-22T15:49:43.710 INFO:tasks.workunit.client.0.smithi016.stdout:9/801: dread d9/de/fcb [0,4194304] 0 2024-04-22T15:49:43.713 INFO:tasks.workunit.client.0.smithi016.stdout:3/828: mkdir d7/d17/d18/d32/d42/d63/d86/dd4/d100 0 2024-04-22T15:49:43.721 INFO:tasks.workunit.client.0.smithi016.stdout:3/829: dwrite d7/d17/d18/d32/d8a/f9b [0,4194304] 0 2024-04-22T15:49:43.723 INFO:tasks.workunit.client.0.smithi016.stdout:9/802: fsync d9/de/f43 0 2024-04-22T15:49:43.728 INFO:tasks.workunit.client.0.smithi016.stdout:4/743: mkdir d7/d4e/d39/df3 0 2024-04-22T15:49:43.732 INFO:tasks.workunit.client.0.smithi016.stdout:9/803: symlink d9/lfd 0 2024-04-22T15:49:43.734 INFO:tasks.workunit.client.0.smithi016.stdout:3/830: link d7/d17/d18/d32/d42/fa2 d7/d17/d18/d32/d42/f101 0 2024-04-22T15:49:43.735 INFO:tasks.workunit.client.0.smithi016.stdout:4/744: dwrite d7/f16 [0,4194304] 0 2024-04-22T15:49:43.735 INFO:tasks.workunit.client.0.smithi016.stdout:9/804: write d9/d57/d91/de4/d25/d36/d87/ffb [670710,79786] 0 2024-04-22T15:49:43.737 INFO:tasks.workunit.client.0.smithi016.stdout:3/831: creat d7/d77/f102 x:0 0 0 2024-04-22T15:49:43.737 INFO:tasks.workunit.client.0.smithi016.stdout:4/745: mkdir d7/d21/d45/d49/d9b/df4 0 2024-04-22T15:49:43.738 INFO:tasks.workunit.client.0.smithi016.stdout:3/832: stat d7/d17/d18/d27/c9e 0 2024-04-22T15:49:43.740 INFO:tasks.workunit.client.0.smithi016.stdout:4/746: chown d7/d21/d42/d59/d88/d9d/lc2 1194015 1 2024-04-22T15:49:43.740 INFO:tasks.workunit.client.0.smithi016.stdout:4/747: chown d7/f1e 108047335 1 2024-04-22T15:49:43.743 INFO:tasks.workunit.client.0.smithi016.stdout:3/833: mknod d7/d17/d18/d32/d42/d63/deb/d21/d55/ddb/db7/c103 0 2024-04-22T15:49:43.744 INFO:tasks.workunit.client.0.smithi016.stdout:4/748: mkdir d7/d24/df5 0 2024-04-22T15:49:43.787 INFO:tasks.workunit.client.0.smithi016.stdout:4/749: symlink d7/d4e/d2f/lf6 0 2024-04-22T15:49:43.788 INFO:tasks.workunit.client.0.smithi016.stdout:3/834: dread d7/d17/d18/d32/d3c/fd7 [0,4194304] 0 2024-04-22T15:49:43.790 INFO:tasks.workunit.client.0.smithi016.stdout:8/703: dwrite d2/d82/d52/f54 [0,4194304] 0 2024-04-22T15:49:43.791 INFO:tasks.workunit.client.0.smithi016.stdout:8/704: chown d2/d82/d44/dc1 213244598 1 2024-04-22T15:49:43.793 INFO:tasks.workunit.client.0.smithi016.stdout:4/750: creat d7/d21/d42/d59/de2/ff7 x:0 0 0 2024-04-22T15:49:43.793 INFO:tasks.workunit.client.0.smithi016.stdout:3/835: mknod d7/d17/d18/d32/d3c/d51/df1/c104 0 2024-04-22T15:49:43.798 INFO:tasks.workunit.client.0.smithi016.stdout:8/705: creat d2/d10/d47/d67/fd6 x:0 0 0 2024-04-22T15:49:43.799 INFO:tasks.workunit.client.0.smithi016.stdout:4/751: symlink d7/lf8 0 2024-04-22T15:49:43.799 INFO:tasks.workunit.client.0.smithi016.stdout:8/706: fsync d2/d82/d44/d66/d7e/fc5 0 2024-04-22T15:49:43.802 INFO:tasks.workunit.client.0.smithi016.stdout:8/707: mknod d2/d10/d43/da9/cd7 0 2024-04-22T15:49:43.806 INFO:tasks.workunit.client.0.smithi016.stdout:8/708: rename d2/d82/d3e/fb1 to d2/d82/d3e/dd5/fd8 0 2024-04-22T15:49:43.807 INFO:tasks.workunit.client.0.smithi016.stdout:4/752: dwrite d7/d21/fc6 [0,4194304] 0 2024-04-22T15:49:43.809 INFO:tasks.workunit.client.0.smithi016.stdout:4/753: rmdir d7/d4e/d39/dc3 39 2024-04-22T15:49:43.810 INFO:tasks.workunit.client.0.smithi016.stdout:4/754: chown d7/d21/d45/d49/d67/c7b 19785004 1 2024-04-22T15:49:43.814 INFO:tasks.workunit.client.0.smithi016.stdout:4/755: unlink d7/d4e/c53 0 2024-04-22T15:49:43.815 INFO:tasks.workunit.client.0.smithi016.stdout:4/756: mknod d7/d21/d42/d59/d88/d9d/dad/cf9 0 2024-04-22T15:49:43.821 INFO:tasks.workunit.client.0.smithi016.stdout:6/809: sync 2024-04-22T15:49:43.821 INFO:tasks.workunit.client.0.smithi016.stdout:0/831: sync 2024-04-22T15:49:43.821 INFO:tasks.workunit.client.0.smithi016.stdout:7/745: sync 2024-04-22T15:49:43.821 INFO:tasks.workunit.client.0.smithi016.stdout:2/731: sync 2024-04-22T15:49:43.821 INFO:tasks.workunit.client.0.smithi016.stdout:1/909: sync 2024-04-22T15:49:43.823 INFO:tasks.workunit.client.0.smithi016.stdout:7/746: readlink d3/l12 0 2024-04-22T15:49:43.823 INFO:tasks.workunit.client.0.smithi016.stdout:7/747: readlink d3/de6/l98 0 2024-04-22T15:49:43.825 INFO:tasks.workunit.client.0.smithi016.stdout:2/732: mkdir d2/d23/db8/dd4/d114 0 2024-04-22T15:49:43.825 INFO:tasks.workunit.client.0.smithi016.stdout:1/910: unlink d3/d4/d3a/d9e/d124/c129 0 2024-04-22T15:49:43.826 INFO:tasks.workunit.client.0.smithi016.stdout:0/832: rename dc/d13/la1 to dc/d13/d14/de5/d65/d40/d4a/d59/d78/dbd/l10e 0 2024-04-22T15:49:43.827 INFO:tasks.workunit.client.0.smithi016.stdout:8/709: dread d2/d48/dad/f5f [0,4194304] 0 2024-04-22T15:49:43.828 INFO:tasks.workunit.client.0.smithi016.stdout:2/733: rmdir d2/d1a/d7b 39 2024-04-22T15:49:43.833 INFO:tasks.workunit.client.0.smithi016.stdout:7/748: creat d3/dc/d1b/d5a/d9/df/d94/d4c/f10f x:0 0 0 2024-04-22T15:49:43.833 INFO:tasks.workunit.client.0.smithi016.stdout:8/710: mknod d2/d77/cd9 0 2024-04-22T15:49:43.834 INFO:tasks.workunit.client.0.smithi016.stdout:6/810: dwrite d3/da/db/d14/f86 [0,4194304] 0 2024-04-22T15:49:43.838 INFO:tasks.workunit.client.0.smithi016.stdout:6/811: chown d3/da/dd/f91 32 1 2024-04-22T15:49:43.839 INFO:tasks.workunit.client.0.smithi016.stdout:1/911: dwrite d3/d4/d37/f41 [0,4194304] 0 2024-04-22T15:49:43.839 INFO:tasks.workunit.client.0.smithi016.stdout:0/833: rename dc/d13/d14/de5/d65/d40/fac to dc/d13/d14/de5/def/f10f 0 2024-04-22T15:49:43.840 INFO:tasks.workunit.client.0.smithi016.stdout:4/757: dread d7/f19 [0,4194304] 0 2024-04-22T15:49:43.841 INFO:tasks.workunit.client.0.smithi016.stdout:0/834: fdatasync dc/d13/de7/f10a 0 2024-04-22T15:49:43.841 INFO:tasks.workunit.client.0.smithi016.stdout:1/912: readlink d3/d4/d3a/d91/db3/l119 0 2024-04-22T15:49:43.842 INFO:tasks.workunit.client.0.smithi016.stdout:4/758: fdatasync d7/d4e/f29 0 2024-04-22T15:49:43.843 INFO:tasks.workunit.client.0.smithi016.stdout:1/913: chown d3/d4/d19/d20/d2b/dd6/d12d/ff2 7266468 1 2024-04-22T15:49:43.844 INFO:tasks.workunit.client.0.smithi016.stdout:6/812: fsync d3/da/db/d14/d4c/f90 0 2024-04-22T15:49:43.844 INFO:tasks.workunit.client.0.smithi016.stdout:2/734: link d2/d2c/d97/d67/dbf/l10d d2/d4d/dc4/l115 0 2024-04-22T15:49:43.848 INFO:tasks.workunit.client.0.smithi016.stdout:8/711: mkdir d2/d48/dad/d8f/d96/dda 0 2024-04-22T15:49:43.855 INFO:tasks.workunit.client.0.smithi016.stdout:7/749: creat d3/dc/d1b/d5a/d4a/f110 x:0 0 0 2024-04-22T15:49:43.856 INFO:tasks.workunit.client.0.smithi016.stdout:8/712: symlink d2/d32/d92/ldb 0 2024-04-22T15:49:43.858 INFO:tasks.workunit.client.0.smithi016.stdout:7/750: readlink d3/dc/d1b/d5a/l103 0 2024-04-22T15:49:43.859 INFO:tasks.workunit.client.0.smithi016.stdout:8/713: symlink d2/d82/d3e/d8e/ldc 0 2024-04-22T15:49:43.860 INFO:tasks.workunit.client.0.smithi016.stdout:4/759: write d7/d3c/d96/fb8 [3706231,83982] 0 2024-04-22T15:49:43.861 INFO:tasks.workunit.client.0.smithi016.stdout:7/751: stat d3/dc/d1b/d5a/d69/l8e 0 2024-04-22T15:49:43.862 INFO:tasks.workunit.client.0.smithi016.stdout:7/752: readlink d3/dc/d9f/le9 0 2024-04-22T15:49:43.863 INFO:tasks.workunit.client.0.smithi016.stdout:8/714: unlink d2/d48/lcc 0 2024-04-22T15:49:43.864 INFO:tasks.workunit.client.0.smithi016.stdout:2/735: dwrite d2/d1a/d7b/dcc/fc8 [0,4194304] 0 2024-04-22T15:49:43.866 INFO:tasks.workunit.client.0.smithi016.stdout:4/760: dread d7/d21/d45/f66 [0,4194304] 0 2024-04-22T15:49:43.869 INFO:tasks.workunit.client.0.smithi016.stdout:7/753: mknod d3/dc/d1b/d5a/d69/def/c111 0 2024-04-22T15:49:43.869 INFO:tasks.workunit.client.0.smithi016.stdout:2/736: truncate d2/d2c/d97/f6a 192885 0 2024-04-22T15:49:43.869 INFO:tasks.workunit.client.0.smithi016.stdout:4/761: write d7/f1e [3459729,68985] 0 2024-04-22T15:49:43.873 INFO:tasks.workunit.client.0.smithi016.stdout:7/754: readlink d3/de6/lc5 0 2024-04-22T15:49:43.876 INFO:tasks.workunit.client.0.smithi016.stdout:4/762: unlink d7/d4e/d2f/l3f 0 2024-04-22T15:49:43.879 INFO:tasks.workunit.client.0.smithi016.stdout:8/715: rmdir d2/d48 39 2024-04-22T15:49:43.885 INFO:tasks.workunit.client.0.smithi016.stdout:0/835: dread dc/d13/d14/de5/d65/d1f/fcf [0,4194304] 0 2024-04-22T15:49:43.888 INFO:tasks.workunit.client.0.smithi016.stdout:2/737: write d2/d1a/f5e [218073,2555] 0 2024-04-22T15:49:43.890 INFO:tasks.workunit.client.0.smithi016.stdout:8/716: creat d2/d82/d44/d66/d1c/d65/fdd x:0 0 0 2024-04-22T15:49:43.891 INFO:tasks.workunit.client.0.smithi016.stdout:4/763: dwrite d7/d4e/f60 [0,4194304] 0 2024-04-22T15:49:43.893 INFO:tasks.workunit.client.0.smithi016.stdout:7/755: dwrite d3/dc/d1b/d31/f48 [4194304,4194304] 0 2024-04-22T15:49:43.894 INFO:tasks.workunit.client.0.smithi016.stdout:0/836: chown dc/d13/d14/d55/f93 0 1 2024-04-22T15:49:43.894 INFO:tasks.workunit.client.0.smithi016.stdout:6/813: dread d3/da/dd/d7b/fc0 [0,4194304] 0 2024-04-22T15:49:43.895 INFO:tasks.workunit.client.0.smithi016.stdout:4/764: readlink d7/d3c/l86 0 2024-04-22T15:49:43.896 INFO:tasks.workunit.client.0.smithi016.stdout:2/738: write d2/d1a/d53/ff8 [860946,114525] 0 2024-04-22T15:49:43.899 INFO:tasks.workunit.client.0.smithi016.stdout:8/717: mknod d2/d82/d56/cde 0 2024-04-22T15:49:43.903 INFO:tasks.workunit.client.0.smithi016.stdout:4/765: chown d7/d4e/d81/fcb 26190204 1 2024-04-22T15:49:43.903 INFO:tasks.workunit.client.0.smithi016.stdout:2/739: write d2/d2c/f6b [853372,88493] 0 2024-04-22T15:49:43.903 INFO:tasks.workunit.client.0.smithi016.stdout:6/814: write d3/da/d12/d13/d97/de0/fe4 [260141,50134] 0 2024-04-22T15:49:43.909 INFO:tasks.workunit.client.0.smithi016.stdout:9/805: sync 2024-04-22T15:49:43.910 INFO:tasks.workunit.client.0.smithi016.stdout:5/797: sync 2024-04-22T15:49:43.910 INFO:tasks.workunit.client.0.smithi016.stdout:2/740: unlink d2/d4d/l98 0 2024-04-22T15:49:43.913 INFO:tasks.workunit.client.0.smithi016.stdout:6/815: mknod d3/da/d12/d13/dd2/dee/c10f 0 2024-04-22T15:49:43.915 INFO:tasks.workunit.client.0.smithi016.stdout:7/756: dwrite d3/dc/d1b/feb [0,4194304] 0 2024-04-22T15:49:43.916 INFO:tasks.workunit.client.0.smithi016.stdout:8/718: link d2/d10/c5e d2/d10/d43/dbe/cdf 0 2024-04-22T15:49:43.917 INFO:tasks.workunit.client.0.smithi016.stdout:5/798: stat df/d15/d1a/fd2 0 2024-04-22T15:49:43.918 INFO:tasks.workunit.client.0.smithi016.stdout:9/806: rmdir d9/d20/dac/dbd 39 2024-04-22T15:49:43.920 INFO:tasks.workunit.client.0.smithi016.stdout:2/741: creat d2/d1a/d4c/d77/dea/f116 x:0 0 0 2024-04-22T15:49:43.922 INFO:tasks.workunit.client.0.smithi016.stdout:5/799: symlink df/d5b/d70/lf5 0 2024-04-22T15:49:43.923 INFO:tasks.workunit.client.0.smithi016.stdout:4/766: rename d7/d4e/c3e to d7/d21/d42/d59/d88/d9d/dad/cfa 0 2024-04-22T15:49:43.928 INFO:tasks.workunit.client.0.smithi016.stdout:8/719: symlink d2/d48/le0 0 2024-04-22T15:49:43.931 INFO:tasks.workunit.client.0.smithi016.stdout:0/837: dwrite dc/d13/d14/de5/d65/d40/d4a/d59/d9d/fc1 [0,4194304] 0 2024-04-22T15:49:43.931 INFO:tasks.workunit.client.0.smithi016.stdout:5/800: creat df/d5b/db3/dc1/ff6 x:0 0 0 2024-04-22T15:49:43.932 INFO:tasks.workunit.client.0.smithi016.stdout:0/838: readlink dc/d13/d70/l79 0 2024-04-22T15:49:43.932 INFO:tasks.workunit.client.0.smithi016.stdout:9/807: creat d9/d57/d91/de4/ffe x:0 0 0 2024-04-22T15:49:43.933 INFO:tasks.workunit.client.0.smithi016.stdout:5/801: chown df/d15/d1a/l1c 13037 1 2024-04-22T15:49:43.933 INFO:tasks.workunit.client.0.smithi016.stdout:2/742: truncate d2/d23/d92/fa1 640696 0 2024-04-22T15:49:43.936 INFO:tasks.workunit.client.0.smithi016.stdout:9/808: fdatasync d9/d20/fab 0 2024-04-22T15:49:43.937 INFO:tasks.workunit.client.0.smithi016.stdout:6/816: dread d3/da/d12/d13/d97/de0/fe3 [0,4194304] 0 2024-04-22T15:49:43.938 INFO:tasks.workunit.client.0.smithi016.stdout:0/839: creat dc/d13/d14/de5/d65/d30/d75/f110 x:0 0 0 2024-04-22T15:49:43.943 INFO:tasks.workunit.client.0.smithi016.stdout:9/809: creat d9/d20/dac/dae/fff x:0 0 0 2024-04-22T15:49:43.943 INFO:tasks.workunit.client.0.smithi016.stdout:2/743: write d2/d9/fa8 [138266,96668] 0 2024-04-22T15:49:43.944 INFO:tasks.workunit.client.0.smithi016.stdout:5/802: rename df/d5b/d5d/d5f/f6f to df/d15/d1a/d50/d77/d7d/d8b/d8c/ff7 0 2024-04-22T15:49:43.946 INFO:tasks.workunit.client.0.smithi016.stdout:6/817: fsync d3/da/dd/d1b/f8b 0 2024-04-22T15:49:43.947 INFO:tasks.workunit.client.0.smithi016.stdout:0/840: mknod dc/d13/d22/c111 0 2024-04-22T15:49:43.948 INFO:tasks.workunit.client.0.smithi016.stdout:5/803: read - df/d15/d1a/fe5 zero size 2024-04-22T15:49:43.948 INFO:tasks.workunit.client.0.smithi016.stdout:7/757: dwrite d3/dc/d1b/d5a/d9/f26 [0,4194304] 0 2024-04-22T15:49:43.950 INFO:tasks.workunit.client.0.smithi016.stdout:2/744: truncate d2/d51/f9b 2108488 0 2024-04-22T15:49:43.954 INFO:tasks.workunit.client.0.smithi016.stdout:4/767: dread d7/d4e/d2f/d6f/f84 [0,4194304] 0 2024-04-22T15:49:43.957 INFO:tasks.workunit.client.0.smithi016.stdout:6/818: dread d3/da/fc3 [0,4194304] 0 2024-04-22T15:49:43.958 INFO:tasks.workunit.client.0.smithi016.stdout:5/804: rename df/d15/d1a/f4b to df/d15/d1a/d4e/d58/d87/ff8 0 2024-04-22T15:49:43.960 INFO:tasks.workunit.client.0.smithi016.stdout:5/805: chown df/d15/d1a/d50/fe3 43619 1 2024-04-22T15:49:43.962 INFO:tasks.workunit.client.0.smithi016.stdout:7/758: symlink d3/dc/d1b/d5a/d9/df/d94/d4c/d7a/l112 0 2024-04-22T15:49:43.962 INFO:tasks.workunit.client.0.smithi016.stdout:4/768: symlink d7/d21/d42/d59/d88/lfb 0 2024-04-22T15:49:43.964 INFO:tasks.workunit.client.0.smithi016.stdout:2/745: getdents d2/d23/d92 0 2024-04-22T15:49:43.967 INFO:tasks.workunit.client.0.smithi016.stdout:6/819: mknod d3/da/dd/c110 0 2024-04-22T15:49:43.968 INFO:tasks.workunit.client.0.smithi016.stdout:2/746: mknod d2/d2c/d97/d34/d60/c117 0 2024-04-22T15:49:43.968 INFO:tasks.workunit.client.0.smithi016.stdout:7/759: write d3/dc/d1b/d5a/d9/df/fcc [797602,99162] 0 2024-04-22T15:49:43.969 INFO:tasks.workunit.client.0.smithi016.stdout:5/806: symlink df/d15/d1a/d28/lf9 0 2024-04-22T15:49:43.969 INFO:tasks.workunit.client.0.smithi016.stdout:4/769: mkdir d7/d21/d45/dfc 0 2024-04-22T15:49:43.971 INFO:tasks.workunit.client.0.smithi016.stdout:6/820: unlink d3/da/dd/d7b/fc0 0 2024-04-22T15:49:43.974 INFO:tasks.workunit.client.0.smithi016.stdout:7/760: dread d3/dc/d9f/da1/fbb [0,4194304] 0 2024-04-22T15:49:43.975 INFO:tasks.workunit.client.0.smithi016.stdout:0/841: dwrite dc/d13/d14/de5/d65/d57/fca [0,4194304] 0 2024-04-22T15:49:43.975 INFO:tasks.workunit.client.0.smithi016.stdout:2/747: rmdir d2/d1a/d4c/d77/dea 39 2024-04-22T15:49:43.977 INFO:tasks.workunit.client.0.smithi016.stdout:5/807: fdatasync df/d15/d1a/f71 0 2024-04-22T15:49:43.984 INFO:tasks.workunit.client.0.smithi016.stdout:4/770: link d7/d21/d45/f6d d7/d4e/d2f/d33/ffd 0 2024-04-22T15:49:43.985 INFO:tasks.workunit.client.0.smithi016.stdout:5/808: mkdir df/d15/d1a/d50/d77/d7d/d8b/d93/dfa 0 2024-04-22T15:49:43.986 INFO:tasks.workunit.client.0.smithi016.stdout:0/842: dwrite dc/d13/d14/de5/d65/fa7 [0,4194304] 0 2024-04-22T15:49:43.986 INFO:tasks.workunit.client.0.smithi016.stdout:7/761: symlink d3/dc/d1b/d5a/d4a/l113 0 2024-04-22T15:49:43.987 INFO:tasks.workunit.client.0.smithi016.stdout:2/748: creat d2/d4d/f118 x:0 0 0 2024-04-22T15:49:43.991 INFO:tasks.workunit.client.0.smithi016.stdout:6/821: getdents d3/da/db/d14/d16 0 2024-04-22T15:49:43.992 INFO:tasks.workunit.client.0.smithi016.stdout:4/771: dread d7/d4e/d2f/d6f/f84 [4194304,4194304] 0 2024-04-22T15:49:43.993 INFO:tasks.workunit.client.0.smithi016.stdout:4/772: readlink d7/d21/lc1 0 2024-04-22T15:49:43.994 INFO:tasks.workunit.client.0.smithi016.stdout:0/843: mkdir dc/d13/d14/de5/d65/d57/dcb/de1/d112 0 2024-04-22T15:49:43.995 INFO:tasks.workunit.client.0.smithi016.stdout:5/809: symlink df/d15/d1a/d50/d77/d7d/lfb 0 2024-04-22T15:49:43.997 INFO:tasks.workunit.client.0.smithi016.stdout:7/762: creat d3/dc/d1b/d31/d5f/f114 x:0 0 0 2024-04-22T15:49:43.998 INFO:tasks.workunit.client.0.smithi016.stdout:2/749: symlink d2/d1a/l119 0 2024-04-22T15:49:43.999 INFO:tasks.workunit.client.0.smithi016.stdout:6/822: mkdir d3/da/dd/d67/d111 0 2024-04-22T15:49:44.000 INFO:tasks.workunit.client.0.smithi016.stdout:2/750: fdatasync d2/d4d/dc4/f112 0 2024-04-22T15:49:44.000 INFO:tasks.workunit.client.0.smithi016.stdout:4/773: dread d7/d21/d42/f4c [4194304,4194304] 0 2024-04-22T15:49:44.000 INFO:tasks.workunit.client.0.smithi016.stdout:0/844: stat dc/d13/d14/d26/d32/fb4 0 2024-04-22T15:49:44.000 INFO:tasks.workunit.client.0.smithi016.stdout:7/763: rmdir d3/dc/d1b/d5a/d59 39 2024-04-22T15:49:44.001 INFO:tasks.workunit.client.0.smithi016.stdout:6/823: chown d3/da/dd/d7b/fb0 1155665 1 2024-04-22T15:49:44.003 INFO:tasks.workunit.client.0.smithi016.stdout:2/751: creat d2/d2c/d97/d34/d60/f11a x:0 0 0 2024-04-22T15:49:44.004 INFO:tasks.workunit.client.0.smithi016.stdout:6/824: chown d3/da/d12/d13/d38/d84/ff9 2 1 2024-04-22T15:49:44.006 INFO:tasks.workunit.client.0.smithi016.stdout:6/825: chown d3/da/dd/d7b 33 1 2024-04-22T15:49:44.006 INFO:tasks.workunit.client.0.smithi016.stdout:0/845: getdents dc/d13/d70 0 2024-04-22T15:49:44.008 INFO:tasks.workunit.client.0.smithi016.stdout:6/826: stat d3/da/dd/f82 0 2024-04-22T15:49:44.009 INFO:tasks.workunit.client.0.smithi016.stdout:0/846: write dc/d13/d14/de5/d65/d40/fd9 [1087931,11613] 0 2024-04-22T15:49:44.010 INFO:tasks.workunit.client.0.smithi016.stdout:2/752: rename d2/d4d/d76 to d2/d23/db8/dd4/d114/d11b 0 2024-04-22T15:49:44.011 INFO:tasks.workunit.client.0.smithi016.stdout:4/774: link d7/d21/d45/d49/d9b/db6/cd2 d7/d4e/d39/cfe 0 2024-04-22T15:49:44.013 INFO:tasks.workunit.client.0.smithi016.stdout:6/827: symlink d3/da/d12/d13/d38/d84/l112 0 2024-04-22T15:49:44.013 INFO:tasks.workunit.client.0.smithi016.stdout:5/810: dread df/f1f [4194304,4194304] 0 2024-04-22T15:49:44.017 INFO:tasks.workunit.client.0.smithi016.stdout:2/753: dread - d2/d1a/d4c/d77/dea/f116 zero size 2024-04-22T15:49:44.017 INFO:tasks.workunit.client.0.smithi016.stdout:4/775: creat d7/d4e/de1/fff x:0 0 0 2024-04-22T15:49:44.018 INFO:tasks.workunit.client.0.smithi016.stdout:6/828: symlink d3/d53/d35/l113 0 2024-04-22T15:49:44.019 INFO:tasks.workunit.client.0.smithi016.stdout:6/829: chown d3/da/d12/d13/dd2 0 1 2024-04-22T15:49:44.020 INFO:tasks.workunit.client.0.smithi016.stdout:4/776: chown d7/d24/d41/f57 412 1 2024-04-22T15:49:44.021 INFO:tasks.workunit.client.0.smithi016.stdout:2/754: creat d2/d2c/d97/d34/f11c x:0 0 0 2024-04-22T15:49:44.021 INFO:tasks.workunit.client.0.smithi016.stdout:3/836: sync 2024-04-22T15:49:44.021 INFO:tasks.workunit.client.0.smithi016.stdout:1/914: sync 2024-04-22T15:49:44.022 INFO:tasks.workunit.client.0.smithi016.stdout:6/830: rmdir d3/da/ddd 39 2024-04-22T15:49:44.023 INFO:tasks.workunit.client.0.smithi016.stdout:5/811: link df/d15/d1a/d4e/d58/ca3 df/d90/cfc 0 2024-04-22T15:49:44.025 INFO:tasks.workunit.client.0.smithi016.stdout:7/764: dread d3/dc/d1b/d31/d5f/f7b [0,4194304] 0 2024-04-22T15:49:44.025 INFO:tasks.workunit.client.0.smithi016.stdout:2/755: chown d2/d23/d92/c70 19 1 2024-04-22T15:49:44.025 INFO:tasks.workunit.client.0.smithi016.stdout:5/812: chown df/d15/d1a/d28/f3a 0 1 2024-04-22T15:49:44.029 INFO:tasks.workunit.client.0.smithi016.stdout:1/915: truncate d3/d4/fe 5108252 0 2024-04-22T15:49:44.031 INFO:tasks.workunit.client.0.smithi016.stdout:3/837: write d7/d17/d18/d32/d42/f101 [373502,68402] 0 2024-04-22T15:49:44.032 INFO:tasks.workunit.client.0.smithi016.stdout:1/916: chown d3/d2e/d80/dbb/le5 20 1 2024-04-22T15:49:44.033 INFO:tasks.workunit.client.0.smithi016.stdout:4/777: dread d7/d3c/f44 [0,4194304] 0 2024-04-22T15:49:44.034 INFO:tasks.workunit.client.0.smithi016.stdout:3/838: chown d7/fea 311469 1 2024-04-22T15:49:44.035 INFO:tasks.workunit.client.0.smithi016.stdout:2/756: link d2/d9/f12 d2/d23/d92/d55/f11d 0 2024-04-22T15:49:44.037 INFO:tasks.workunit.client.0.smithi016.stdout:1/917: write d3/d4/d3a/d6b/d77/f102 [2454167,58745] 0 2024-04-22T15:49:44.037 INFO:tasks.workunit.client.0.smithi016.stdout:4/778: readlink d7/d4e/l3b 0 2024-04-22T15:49:44.039 INFO:tasks.workunit.client.0.smithi016.stdout:0/847: dread dc/d13/d14/de5/d65/d40/f56 [4194304,4194304] 0 2024-04-22T15:49:44.041 INFO:tasks.workunit.client.0.smithi016.stdout:5/813: dread df/f57 [0,4194304] 0 2024-04-22T15:49:44.042 INFO:tasks.workunit.client.0.smithi016.stdout:2/757: fdatasync d2/d1a/f20 0 2024-04-22T15:49:44.043 INFO:tasks.workunit.client.0.smithi016.stdout:3/839: stat d7/d17/d18/d32/d42/d63/deb/d21/d55/ddb 0 2024-04-22T15:49:44.043 INFO:tasks.workunit.client.0.smithi016.stdout:6/831: dwrite d3/da/dd/d1b/f62 [0,4194304] 0 2024-04-22T15:49:44.044 INFO:tasks.workunit.client.0.smithi016.stdout:1/918: readlink d3/d4/d19/d20/d95/l10a 0 2024-04-22T15:49:44.046 INFO:tasks.workunit.client.0.smithi016.stdout:0/848: mkdir dc/d13/d14/de5/dd8/d113 0 2024-04-22T15:49:44.049 INFO:tasks.workunit.client.0.smithi016.stdout:6/832: rmdir d3/da/d12/d13 39 2024-04-22T15:49:44.050 INFO:tasks.workunit.client.0.smithi016.stdout:5/814: creat df/d5b/ffd x:0 0 0 2024-04-22T15:49:44.052 INFO:tasks.workunit.client.0.smithi016.stdout:3/840: creat d7/d17/d18/d32/d3c/f105 x:0 0 0 2024-04-22T15:49:44.052 INFO:tasks.workunit.client.0.smithi016.stdout:0/849: chown dc/d13/d14/de5/d65/d57/dcb/de1/d112 815959 1 2024-04-22T15:49:44.055 INFO:tasks.workunit.client.0.smithi016.stdout:7/765: dread d3/dc/d1b/d5a/d9/df/d94/f3d [0,4194304] 0 2024-04-22T15:49:44.055 INFO:tasks.workunit.client.0.smithi016.stdout:5/815: dread - df/d5b/d5d/ff2 zero size 2024-04-22T15:49:44.055 INFO:tasks.workunit.client.0.smithi016.stdout:4/779: dread d7/d21/d45/f6d [0,4194304] 0 2024-04-22T15:49:44.057 INFO:tasks.workunit.client.0.smithi016.stdout:6/833: creat d3/da/db/d14/d4c/d58/f114 x:0 0 0 2024-04-22T15:49:44.057 INFO:tasks.workunit.client.0.smithi016.stdout:3/841: write d7/fc [1522725,16398] 0 2024-04-22T15:49:44.060 INFO:tasks.workunit.client.0.smithi016.stdout:1/919: dwrite d3/d4/d3a/d96/f112 [0,4194304] 0 2024-04-22T15:49:44.060 INFO:tasks.workunit.client.0.smithi016.stdout:5/816: readlink df/d15/d1a/d50/l9d 0 2024-04-22T15:49:44.061 INFO:tasks.workunit.client.0.smithi016.stdout:0/850: readlink dc/d13/d14/de5/d65/d40/dd6/d104/lce 0 2024-04-22T15:49:44.062 INFO:tasks.workunit.client.0.smithi016.stdout:3/842: readlink d7/d17/l29 0 2024-04-22T15:49:44.063 INFO:tasks.workunit.client.0.smithi016.stdout:7/766: mkdir d3/dc/d1b/d5a/d4a/d6e/d115 0 2024-04-22T15:49:44.066 INFO:tasks.workunit.client.0.smithi016.stdout:6/834: symlink d3/da/dd/d67/d111/l115 0 2024-04-22T15:49:44.067 INFO:tasks.workunit.client.0.smithi016.stdout:1/920: mknod d3/d4/d3a/d9e/d124/c13a 0 2024-04-22T15:49:44.068 INFO:tasks.workunit.client.0.smithi016.stdout:4/780: rename d7/d21/d45/d49/cf0 to d7/d21/d45/dd6/dda/c100 0 2024-04-22T15:49:44.070 INFO:tasks.workunit.client.0.smithi016.stdout:3/843: fsync d7/d17/d18/d32/d3c/d51/df1/ff7 0 2024-04-22T15:49:44.070 INFO:tasks.workunit.client.0.smithi016.stdout:6/835: rmdir d3/da/dd/d67/d6c/dfc 39 2024-04-22T15:49:44.075 INFO:tasks.workunit.client.0.smithi016.stdout:1/921: rename d3/d4/d15/d8c/fc5 to d3/d4/d3a/d9e/d124/f13b 0 2024-04-22T15:49:44.078 INFO:tasks.workunit.client.0.smithi016.stdout:1/922: stat d3/d2e/d80/d82/ce7 0 2024-04-22T15:49:44.078 INFO:tasks.workunit.client.0.smithi016.stdout:3/844: creat d7/d17/d18/d32/d42/d63/deb/d21/d55/f106 x:0 0 0 2024-04-22T15:49:44.078 INFO:tasks.workunit.client.0.smithi016.stdout:4/781: write d7/d22/fef [379959,101975] 0 2024-04-22T15:49:44.079 INFO:tasks.workunit.client.0.smithi016.stdout:7/767: creat d3/dc/d1b/d31/f116 x:0 0 0 2024-04-22T15:49:44.079 INFO:tasks.workunit.client.0.smithi016.stdout:5/817: link df/c7f df/d15/d1a/d50/d77/cfe 0 2024-04-22T15:49:44.081 INFO:tasks.workunit.client.0.smithi016.stdout:0/851: getdents dc/d13/d14/de5/d65/d57 0 2024-04-22T15:49:44.081 INFO:tasks.workunit.client.0.smithi016.stdout:1/923: mknod d3/d4/d3a/d9e/dbf/c13c 0 2024-04-22T15:49:44.085 INFO:tasks.workunit.client.0.smithi016.stdout:5/818: rmdir df/d15/d1a 39 2024-04-22T15:49:44.085 INFO:tasks.workunit.client.0.smithi016.stdout:7/768: chown d3/dc/d1b/d5a/d9/df/d94/d4c/ldc 1493429 1 2024-04-22T15:49:44.087 INFO:tasks.workunit.client.0.smithi016.stdout:3/845: mknod d7/d17/d18/d32/d42/c107 0 2024-04-22T15:49:44.088 INFO:tasks.workunit.client.0.smithi016.stdout:1/924: mkdir d3/d4/d37/d13d 0 2024-04-22T15:49:44.089 INFO:tasks.workunit.client.0.smithi016.stdout:4/782: dread d7/d4e/f60 [0,4194304] 0 2024-04-22T15:49:44.090 INFO:tasks.workunit.client.0.smithi016.stdout:1/925: write d3/f107 [1031072,99886] 0 2024-04-22T15:49:44.091 INFO:tasks.workunit.client.0.smithi016.stdout:0/852: dread dc/d13/d14/de5/d65/d1f/fb3 [0,4194304] 0 2024-04-22T15:49:44.091 INFO:tasks.workunit.client.0.smithi016.stdout:6/836: dread d3/da/d12/ff [0,4194304] 0 2024-04-22T15:49:44.093 INFO:tasks.workunit.client.0.smithi016.stdout:5/819: write df/d15/d1a/fbe [1923275,109194] 0 2024-04-22T15:49:44.094 INFO:tasks.workunit.client.0.smithi016.stdout:7/769: fsync d3/dc/d1b/f6d 0 2024-04-22T15:49:44.095 INFO:tasks.workunit.client.0.smithi016.stdout:3/846: chown d7/d17/d18/d32/d3c/d47/dc3/fde 180345610 1 2024-04-22T15:49:44.101 INFO:tasks.workunit.client.0.smithi016.stdout:1/926: creat d3/d4/d19/f13e x:0 0 0 2024-04-22T15:49:44.101 INFO:tasks.workunit.client.0.smithi016.stdout:5/820: chown df/d15/d1a/d50/d77/d7d/d8b/d93/db7 2221213 1 2024-04-22T15:49:44.103 INFO:tasks.workunit.client.0.smithi016.stdout:4/783: dwrite d7/d24/fed [0,4194304] 0 2024-04-22T15:49:44.104 INFO:tasks.workunit.client.0.smithi016.stdout:7/770: read d3/dc/d1b/d5a/d9/f2a [1047325,123867] 0 2024-04-22T15:49:44.107 INFO:tasks.workunit.client.0.smithi016.stdout:1/927: creat d3/d2e/f13f x:0 0 0 2024-04-22T15:49:44.111 INFO:tasks.workunit.client.0.smithi016.stdout:5/821: creat df/d15/d1a/d50/fff x:0 0 0 2024-04-22T15:49:44.114 INFO:tasks.workunit.client.0.smithi016.stdout:6/837: dwrite d3/da/d12/d13/d38/df5/f109 [0,4194304] 0 2024-04-22T15:49:44.117 INFO:tasks.workunit.client.0.smithi016.stdout:1/928: write d3/f4f [3164076,7032] 0 2024-04-22T15:49:44.117 INFO:tasks.workunit.client.0.smithi016.stdout:0/853: dwrite dc/d13/d14/de5/d65/d40/d4a/d61/f10b [0,4194304] 0 2024-04-22T15:49:44.117 INFO:tasks.workunit.client.0.smithi016.stdout:7/771: fsync d3/dc/d1b/d5a/d9/df/faa 0 2024-04-22T15:49:44.125 INFO:tasks.workunit.client.0.smithi016.stdout:3/847: dwrite d7/f12 [0,4194304] 0 2024-04-22T15:49:44.125 INFO:tasks.workunit.client.0.smithi016.stdout:4/784: getdents d7/d21/d42/d59/dee 0 2024-04-22T15:49:44.127 INFO:tasks.workunit.client.0.smithi016.stdout:7/772: creat d3/dc/d9f/f117 x:0 0 0 2024-04-22T15:49:44.127 INFO:tasks.workunit.client.0.smithi016.stdout:1/929: stat d3/d4/d15/l8e 0 2024-04-22T15:49:44.132 INFO:tasks.workunit.client.0.smithi016.stdout:0/854: symlink dc/d13/d14/de5/d65/d57/dcb/l114 0 2024-04-22T15:49:44.132 INFO:tasks.workunit.client.0.smithi016.stdout:7/773: creat d3/dc/d33/f118 x:0 0 0 2024-04-22T15:49:44.133 INFO:tasks.workunit.client.0.smithi016.stdout:6/838: fsync d3/da/db/d44/f81 0 2024-04-22T15:49:44.135 INFO:tasks.workunit.client.0.smithi016.stdout:1/930: rename d3/d4/d3a/lb1 to d3/d4/d15/d38/d11a/l140 0 2024-04-22T15:49:44.136 INFO:tasks.workunit.client.0.smithi016.stdout:4/785: mkdir d7/d21/d42/d59/d88/deb/d101 0 2024-04-22T15:49:44.136 INFO:tasks.workunit.client.0.smithi016.stdout:5/822: dwrite df/d15/d1a/d28/f32 [4194304,4194304] 0 2024-04-22T15:49:44.140 INFO:tasks.workunit.client.0.smithi016.stdout:0/855: creat dc/d13/d14/de5/d65/d40/d4a/d59/d98/f115 x:0 0 0 2024-04-22T15:49:44.144 INFO:tasks.workunit.client.0.smithi016.stdout:1/931: truncate d3/d84/fcd 1160083 0 2024-04-22T15:49:44.146 INFO:tasks.workunit.client.0.smithi016.stdout:4/786: mkdir d7/d4e/d102 0 2024-04-22T15:49:44.147 INFO:tasks.workunit.client.0.smithi016.stdout:0/856: chown dc/f1e 20508361 1 2024-04-22T15:49:44.149 INFO:tasks.workunit.client.0.smithi016.stdout:5/823: dwrite df/d15/d1a/d50/d77/d7d/d8b/fcf [0,4194304] 0 2024-04-22T15:49:44.150 INFO:tasks.workunit.client.0.smithi016.stdout:0/857: fdatasync dc/d13/d14/de5/d65/d40/dd6/d104/fb5 0 2024-04-22T15:49:44.153 INFO:tasks.workunit.client.0.smithi016.stdout:0/858: chown dc/d13/d14/d26/l95 4357416 1 2024-04-22T15:49:44.154 INFO:tasks.workunit.client.0.smithi016.stdout:5/824: mknod df/d15/d1a/d50/d77/d7d/d8b/d93/dbb/c100 0 2024-04-22T15:49:44.158 INFO:tasks.workunit.client.0.smithi016.stdout:0/859: creat dc/d13/d14/de5/d65/d57/dcb/de1/f116 x:0 0 0 2024-04-22T15:49:44.164 INFO:tasks.workunit.client.0.smithi016.stdout:7/774: dwrite d3/de6/f85 [0,4194304] 0 2024-04-22T15:49:44.165 INFO:tasks.workunit.client.0.smithi016.stdout:0/860: dread - dc/d13/d14/de5/d65/d40/d4a/d61/ff2 zero size 2024-04-22T15:49:44.168 INFO:tasks.workunit.client.0.smithi016.stdout:8/720: sync 2024-04-22T15:49:44.169 INFO:tasks.workunit.client.0.smithi016.stdout:5/825: dwrite df/d15/d1a/f71 [0,4194304] 0 2024-04-22T15:49:44.170 INFO:tasks.workunit.client.0.smithi016.stdout:4/787: dread d7/f69 [0,4194304] 0 2024-04-22T15:49:44.171 INFO:tasks.workunit.client.0.smithi016.stdout:6/839: dread d3/da/db/d14/d16/f3d [0,4194304] 0 2024-04-22T15:49:44.172 INFO:tasks.workunit.client.0.smithi016.stdout:5/826: stat df/f1f 0 2024-04-22T15:49:44.172 INFO:tasks.workunit.client.0.smithi016.stdout:0/861: dread - dc/d13/d14/d26/f103 zero size 2024-04-22T15:49:44.178 INFO:tasks.workunit.client.0.smithi016.stdout:8/721: truncate d2/d82/d56/f64 688780 0 2024-04-22T15:49:44.179 INFO:tasks.workunit.client.0.smithi016.stdout:7/775: dread d3/dc/d1b/d31/d5f/f77 [0,4194304] 0 2024-04-22T15:49:44.179 INFO:tasks.workunit.client.0.smithi016.stdout:4/788: chown d7/d24/d51/f6a 0 1 2024-04-22T15:49:44.180 INFO:tasks.workunit.client.0.smithi016.stdout:8/722: stat d2/l2b 0 2024-04-22T15:49:44.181 INFO:tasks.workunit.client.0.smithi016.stdout:5/827: rename df/d15/d1a/d50/d77/d7d/d8b/d8c/de4/fe6 to df/d90/dad/f101 0 2024-04-22T15:49:44.182 INFO:tasks.workunit.client.0.smithi016.stdout:0/862: link dc/f7e dc/d13/d14/de5/f117 0 2024-04-22T15:49:44.182 INFO:tasks.workunit.client.0.smithi016.stdout:4/789: dread - d7/d21/d42/d59/d88/d9d/dad/fb4 zero size 2024-04-22T15:49:44.182 INFO:tasks.workunit.client.0.smithi016.stdout:5/828: dread - df/d5b/d5d/ff2 zero size 2024-04-22T15:49:44.185 INFO:tasks.workunit.client.0.smithi016.stdout:4/790: read d7/d24/d51/fbb [3097493,105562] 0 2024-04-22T15:49:44.186 INFO:tasks.workunit.client.0.smithi016.stdout:8/723: symlink d2/d48/dad/d8f/da1/le1 0 2024-04-22T15:49:44.187 INFO:tasks.workunit.client.0.smithi016.stdout:7/776: dread d3/dc/d1b/d31/f48 [4194304,4194304] 0 2024-04-22T15:49:44.188 INFO:tasks.workunit.client.0.smithi016.stdout:7/777: readlink d3/dc/d9f/da1/lb4 0 2024-04-22T15:49:44.188 INFO:tasks.workunit.client.0.smithi016.stdout:6/840: link d3/da/d12/d13/dd2/dee/c10f d3/da/db/d44/d68/d95/d9e/c116 0 2024-04-22T15:49:44.190 INFO:tasks.workunit.client.0.smithi016.stdout:0/863: fdatasync dc/d13/d5d/f6c 0 2024-04-22T15:49:44.191 INFO:tasks.workunit.client.0.smithi016.stdout:4/791: mkdir d7/d21/d7c/d103 0 2024-04-22T15:49:44.193 INFO:tasks.workunit.client.0.smithi016.stdout:4/792: dread - d7/d21/d45/d49/d9b/faf zero size 2024-04-22T15:49:44.195 INFO:tasks.workunit.client.0.smithi016.stdout:6/841: creat d3/da/db/d14/d4c/d58/f117 x:0 0 0 2024-04-22T15:49:44.198 INFO:tasks.workunit.client.0.smithi016.stdout:7/778: mkdir d3/dc/d1b/d5a/d59/d119 0 2024-04-22T15:49:44.202 INFO:tasks.workunit.client.0.smithi016.stdout:6/842: mkdir d3/da/dd/d67/d6c/db9/dfa/d118 0 2024-04-22T15:49:44.202 INFO:tasks.workunit.client.0.smithi016.stdout:7/779: write d3/dc/d22/fcd [265745,85188] 0 2024-04-22T15:49:44.204 INFO:tasks.workunit.client.0.smithi016.stdout:4/793: dwrite d7/d21/d42/d59/d88/d9d/dad/fb4 [0,4194304] 0 2024-04-22T15:49:44.206 INFO:tasks.workunit.client.0.smithi016.stdout:7/780: chown d3/de6/f85 33 1 2024-04-22T15:49:44.208 INFO:tasks.workunit.client.0.smithi016.stdout:8/724: dread d2/fa2 [0,4194304] 0 2024-04-22T15:49:44.208 INFO:tasks.workunit.client.0.smithi016.stdout:6/843: unlink d3/da/dd/f5a 0 2024-04-22T15:49:44.209 INFO:tasks.workunit.client.0.smithi016.stdout:0/864: dwrite dc/d13/d14/d26/d32/fc0 [0,4194304] 0 2024-04-22T15:49:44.211 INFO:tasks.workunit.client.0.smithi016.stdout:4/794: creat d7/d21/d42/d59/d88/d9d/dad/dac/f104 x:0 0 0 2024-04-22T15:49:44.211 INFO:tasks.workunit.client.0.smithi016.stdout:8/725: fsync d2/d82/d44/d66/d1c/f81 0 2024-04-22T15:49:44.214 INFO:tasks.workunit.client.0.smithi016.stdout:4/795: rename d7 to d7/d21/d45/d105 22 2024-04-22T15:49:44.221 INFO:tasks.workunit.client.0.smithi016.stdout:8/726: dread d2/d82/d44/d66/d1c/d65/f9c [0,4194304] 0 2024-04-22T15:49:44.224 INFO:tasks.workunit.client.0.smithi016.stdout:6/844: dwrite d3/da/db/d14/fc8 [0,4194304] 0 2024-04-22T15:49:44.225 INFO:tasks.workunit.client.0.smithi016.stdout:0/865: dread dc/d13/d14/de5/f117 [0,4194304] 0 2024-04-22T15:49:44.227 INFO:tasks.workunit.client.0.smithi016.stdout:8/727: mkdir d2/d82/d44/d66/d7e/dc0/de2 0 2024-04-22T15:49:44.230 INFO:tasks.workunit.client.0.smithi016.stdout:4/796: dwrite d7/d24/f3d [0,4194304] 0 2024-04-22T15:49:44.230 INFO:tasks.workunit.client.0.smithi016.stdout:6/845: write d3/da/dd/d67/f8f [727911,86689] 0 2024-04-22T15:49:44.232 INFO:tasks.workunit.client.0.smithi016.stdout:4/797: truncate d7/d4e/d81/fcb 791137 0 2024-04-22T15:49:44.235 INFO:tasks.workunit.client.0.smithi016.stdout:8/728: symlink d2/d82/d44/d66/d7e/le3 0 2024-04-22T15:49:44.237 INFO:tasks.workunit.client.0.smithi016.stdout:6/846: symlink d3/da/dd/d67/d6c/db9/dfa/l119 0 2024-04-22T15:49:44.237 INFO:tasks.workunit.client.0.smithi016.stdout:0/866: chown dc/d13/d14/de5/d65/d40/d4a/d59/d78/dbd/l10e 11836196 1 2024-04-22T15:49:44.239 INFO:tasks.workunit.client.0.smithi016.stdout:6/847: truncate d3/da/db/d14/d4c/d58/f114 193781 0 2024-04-22T15:49:44.239 INFO:tasks.workunit.client.0.smithi016.stdout:8/729: mkdir d2/d48/dad/d8f/d96/de4 0 2024-04-22T15:49:44.242 INFO:tasks.workunit.client.0.smithi016.stdout:6/848: fdatasync d3/da/db/f61 0 2024-04-22T15:49:44.243 INFO:tasks.workunit.client.0.smithi016.stdout:4/798: write d7/d24/d51/fbb [75314,28779] 0 2024-04-22T15:49:44.245 INFO:tasks.workunit.client.0.smithi016.stdout:4/799: write d7/d24/fed [2508449,32018] 0 2024-04-22T15:49:44.247 INFO:tasks.workunit.client.0.smithi016.stdout:0/867: dwrite dc/d13/d14/de5/d65/d1f/f53 [4194304,4194304] 0 2024-04-22T15:49:44.249 INFO:tasks.workunit.client.0.smithi016.stdout:6/849: stat d3/da/dd/d67/c8a 0 2024-04-22T15:49:44.252 INFO:tasks.workunit.client.0.smithi016.stdout:6/850: write d3/da/dd/f82 [1784457,74533] 0 2024-04-22T15:49:44.253 INFO:tasks.workunit.client.0.smithi016.stdout:4/800: rename d7/d21/d42/d59 to d7/d4e/d102/d106 0 2024-04-22T15:49:44.254 INFO:tasks.workunit.client.0.smithi016.stdout:6/851: dread - d3/da/d12/d13/dd2/dee/f105 zero size 2024-04-22T15:49:44.258 INFO:tasks.workunit.client.0.smithi016.stdout:4/801: dread d7/d22/fef [0,4194304] 0 2024-04-22T15:49:44.258 INFO:tasks.workunit.client.0.smithi016.stdout:8/730: truncate d2/d10/d47/f95 6661473 0 2024-04-22T15:49:44.260 INFO:tasks.workunit.client.0.smithi016.stdout:0/868: link dc/d13/d14/de5/d65/d1f/c6b dc/d13/d14/d55/dba/c118 0 2024-04-22T15:49:44.261 INFO:tasks.workunit.client.0.smithi016.stdout:8/731: creat d2/d10/d47/d67/fe5 x:0 0 0 2024-04-22T15:49:44.262 INFO:tasks.workunit.client.0.smithi016.stdout:0/869: fdatasync dc/d13/d14/de5/d65/d1f/d48/f64 0 2024-04-22T15:49:44.262 INFO:tasks.workunit.client.0.smithi016.stdout:8/732: chown d2/d10/d43 544470656 1 2024-04-22T15:49:44.263 INFO:tasks.workunit.client.0.smithi016.stdout:0/870: stat dc/d13/d14/d55/dba/f109 0 2024-04-22T15:49:44.263 INFO:tasks.workunit.client.0.smithi016.stdout:6/852: mkdir d3/da/ddd/d101/d11a 0 2024-04-22T15:49:44.266 INFO:tasks.workunit.client.0.smithi016.stdout:8/733: mkdir d2/d82/d3e/d8e/de6 0 2024-04-22T15:49:44.267 INFO:tasks.workunit.client.0.smithi016.stdout:0/871: read dc/d13/d14/f34 [381078,124261] 0 2024-04-22T15:49:44.269 INFO:tasks.workunit.client.0.smithi016.stdout:6/853: rename d3/d53 to d3/da/ddd/d101/d11b 0 2024-04-22T15:49:44.270 INFO:tasks.workunit.client.0.smithi016.stdout:4/802: truncate d7/d4e/d2f/d6f/f97 989148 0 2024-04-22T15:49:44.271 INFO:tasks.workunit.client.0.smithi016.stdout:4/803: chown d7/d24/lb1 7387 1 2024-04-22T15:49:44.271 INFO:tasks.workunit.client.0.smithi016.stdout:8/734: symlink d2/d82/d44/d66/d79/le7 0 2024-04-22T15:49:44.273 INFO:tasks.workunit.client.0.smithi016.stdout:4/804: write d7/d4e/d102/d106/d88/d9d/dad/fb4 [1938614,98956] 0 2024-04-22T15:49:44.275 INFO:tasks.workunit.client.0.smithi016.stdout:4/805: fsync d7/d21/fc6 0 2024-04-22T15:49:44.277 INFO:tasks.workunit.client.0.smithi016.stdout:8/735: mkdir d2/d82/d44/d66/d7e/d85/de8 0 2024-04-22T15:49:44.281 INFO:tasks.workunit.client.0.smithi016.stdout:0/872: dwrite dc/d13/d14/de5/d65/d40/f56 [0,4194304] 0 2024-04-22T15:49:44.285 INFO:tasks.workunit.client.0.smithi016.stdout:0/873: chown dc/l62 2256988 1 2024-04-22T15:49:44.290 INFO:tasks.workunit.client.0.smithi016.stdout:4/806: dwrite d7/d21/d42/f4c [8388608,4194304] 0 2024-04-22T15:49:44.291 INFO:tasks.workunit.client.0.smithi016.stdout:8/736: dwrite d2/d82/d44/d66/d7e/dc0/fcd [0,4194304] 0 2024-04-22T15:49:44.293 INFO:tasks.workunit.client.0.smithi016.stdout:6/854: dwrite d3/da/d12/d13/fac [0,4194304] 0 2024-04-22T15:49:44.298 INFO:tasks.workunit.client.0.smithi016.stdout:6/855: mkdir d3/da/d12/d13/dd2/dee/d11c 0 2024-04-22T15:49:44.299 INFO:tasks.workunit.client.0.smithi016.stdout:8/737: unlink d2/d10/d43/da9/cd7 0 2024-04-22T15:49:44.300 INFO:tasks.workunit.client.0.smithi016.stdout:4/807: creat d7/d4e/d2f/df2/f107 x:0 0 0 2024-04-22T15:49:44.301 INFO:tasks.workunit.client.0.smithi016.stdout:8/738: chown d2/d48/dad/l57 6 1 2024-04-22T15:49:44.302 INFO:tasks.workunit.client.0.smithi016.stdout:4/808: stat d7/d21/d45/d49/d67/c7b 0 2024-04-22T15:49:44.303 INFO:tasks.workunit.client.0.smithi016.stdout:6/856: rename d3/da/db/d14/d4c/d58/c7e to d3/da/ddd/d101/d11b/de9/c11d 0 2024-04-22T15:49:44.307 INFO:tasks.workunit.client.0.smithi016.stdout:8/739: rename d2/d82/d44/c97 to d2/d10/d43/ce9 0 2024-04-22T15:49:44.308 INFO:tasks.workunit.client.0.smithi016.stdout:6/857: symlink d3/da/dd/d67/d6c/db9/dfa/d118/l11e 0 2024-04-22T15:49:44.309 INFO:tasks.workunit.client.0.smithi016.stdout:4/809: dwrite d7/d21/d42/f55 [0,4194304] 0 2024-04-22T15:49:44.313 INFO:tasks.workunit.client.0.smithi016.stdout:4/810: creat d7/d4e/d102/d106/d88/d9d/dad/f108 x:0 0 0 2024-04-22T15:49:44.316 INFO:tasks.workunit.client.0.smithi016.stdout:4/811: read d7/d24/d41/f57 [1692293,65896] 0 2024-04-22T15:49:44.320 INFO:tasks.workunit.client.0.smithi016.stdout:4/812: truncate d7/d24/f76 84487 0 2024-04-22T15:49:44.327 INFO:tasks.workunit.client.0.smithi016.stdout:4/813: write d7/d21/d45/f66 [757473,70127] 0 2024-04-22T15:49:44.330 INFO:tasks.workunit.client.0.smithi016.stdout:8/740: dread d2/d48/dad/d6a/f9a [0,4194304] 0 2024-04-22T15:49:44.330 INFO:tasks.workunit.client.0.smithi016.stdout:6/858: dwrite d3/da/dd/d7b/fb0 [0,4194304] 0 2024-04-22T15:49:44.330 INFO:tasks.workunit.client.0.smithi016.stdout:4/814: read - d7/d4e/d102/d106/d88/d9d/dad/f108 zero size 2024-04-22T15:49:44.351 INFO:tasks.workunit.client.0.smithi016.stdout:2/758: sync 2024-04-22T15:49:44.351 INFO:tasks.workunit.client.0.smithi016.stdout:9/810: sync 2024-04-22T15:49:44.352 INFO:tasks.workunit.client.0.smithi016.stdout:6/859: truncate d3/da/db/fe8 490812 0 2024-04-22T15:49:44.353 INFO:tasks.workunit.client.0.smithi016.stdout:8/741: unlink d2/d82/d44/d66/d1c/l88 0 2024-04-22T15:49:44.362 INFO:tasks.workunit.client.0.smithi016.stdout:4/815: dwrite d7/d4e/d102/d106/d88/d9d/dad/f108 [0,4194304] 0 2024-04-22T15:49:44.362 INFO:tasks.workunit.client.0.smithi016.stdout:9/811: write d9/d57/d91/de4/fa4 [269223,90052] 0 2024-04-22T15:49:44.364 INFO:tasks.workunit.client.0.smithi016.stdout:6/860: truncate d3/da/dd/fb5 458145 0 2024-04-22T15:49:44.368 INFO:tasks.workunit.client.0.smithi016.stdout:6/861: chown d3/da/dd/d67/d6c/db9/dfa 6259 1 2024-04-22T15:49:44.369 INFO:tasks.workunit.client.0.smithi016.stdout:4/816: dread - d7/d4e/d102/d106/d88/d9d/dad/dac/f104 zero size 2024-04-22T15:49:44.369 INFO:tasks.workunit.client.0.smithi016.stdout:2/759: dread d2/d1a/d7b/dc6/fde [0,4194304] 0 2024-04-22T15:49:44.370 INFO:tasks.workunit.client.0.smithi016.stdout:6/862: readlink d3/l7 0 2024-04-22T15:49:44.372 INFO:tasks.workunit.client.0.smithi016.stdout:8/742: dwrite d2/d48/dad/d6a/f93 [0,4194304] 0 2024-04-22T15:49:44.373 INFO:tasks.workunit.client.0.smithi016.stdout:9/812: dread d9/d57/d91/de4/d25/f41 [0,4194304] 0 2024-04-22T15:49:44.376 INFO:tasks.workunit.client.0.smithi016.stdout:2/760: fdatasync d2/d2c/d97/d67/dbf/fee 0 2024-04-22T15:49:44.376 INFO:tasks.workunit.client.0.smithi016.stdout:9/813: chown d9/l12 1 1 2024-04-22T15:49:44.376 INFO:tasks.workunit.client.0.smithi016.stdout:4/817: mkdir d7/d4e/d102/d106/d88/d9d/dad/db9/d109 0 2024-04-22T15:49:44.378 INFO:tasks.workunit.client.0.smithi016.stdout:8/743: rename d2/d48/dad/l57 to d2/d82/d44/lea 0 2024-04-22T15:49:44.380 INFO:tasks.workunit.client.0.smithi016.stdout:4/818: symlink d7/d21/d45/dd6/dda/l10a 0 2024-04-22T15:49:44.382 INFO:tasks.workunit.client.0.smithi016.stdout:6/863: creat d3/da/dd/f11f x:0 0 0 2024-04-22T15:49:44.383 INFO:tasks.workunit.client.0.smithi016.stdout:2/761: symlink d2/l11e 0 2024-04-22T15:49:44.384 INFO:tasks.workunit.client.0.smithi016.stdout:2/762: chown d2/d23/d92/d55/c103 140548422 1 2024-04-22T15:49:44.387 INFO:tasks.workunit.client.0.smithi016.stdout:9/814: rename d9/d57/d91/de4/c54 to d9/d5d/db8/c100 0 2024-04-22T15:49:44.388 INFO:tasks.workunit.client.0.smithi016.stdout:8/744: creat d2/d48/dad/d8f/d96/da5/feb x:0 0 0 2024-04-22T15:49:44.389 INFO:tasks.workunit.client.0.smithi016.stdout:2/763: chown d2/d23/db8/dd4/l105 36 1 2024-04-22T15:49:44.389 INFO:tasks.workunit.client.0.smithi016.stdout:4/819: mkdir d7/d22/dc7/d10b 0 2024-04-22T15:49:44.390 INFO:tasks.workunit.client.0.smithi016.stdout:8/745: fdatasync d2/d82/d3e/f42 0 2024-04-22T15:49:44.390 INFO:tasks.workunit.client.0.smithi016.stdout:2/764: dread - d2/d4d/dc4/f112 zero size 2024-04-22T15:49:44.394 INFO:tasks.workunit.client.0.smithi016.stdout:2/765: symlink d2/d1a/d4c/d77/dea/l11f 0 2024-04-22T15:49:44.395 INFO:tasks.workunit.client.0.smithi016.stdout:4/820: rename d7/d4e/d81/l8b to d7/d4e/l10c 0 2024-04-22T15:49:44.397 INFO:tasks.workunit.client.0.smithi016.stdout:2/766: mkdir d2/d23/d92/d120 0 2024-04-22T15:49:44.399 INFO:tasks.workunit.client.0.smithi016.stdout:2/767: creat d2/d23/f121 x:0 0 0 2024-04-22T15:49:44.400 INFO:tasks.workunit.client.0.smithi016.stdout:2/768: dread - d2/d2c/de7/ff4 zero size 2024-04-22T15:49:44.400 INFO:tasks.workunit.client.0.smithi016.stdout:4/821: creat d7/d4e/d39/dc3/f10d x:0 0 0 2024-04-22T15:49:44.402 INFO:tasks.workunit.client.0.smithi016.stdout:8/746: link d2/d82/d44/d66/d7e/l7f d2/d82/d44/d66/d7e/dc0/lec 0 2024-04-22T15:49:44.402 INFO:tasks.workunit.client.0.smithi016.stdout:9/815: write d9/d57/d91/de4/f2f [3482844,1218] 0 2024-04-22T15:49:44.402 INFO:tasks.workunit.client.0.smithi016.stdout:4/822: rmdir d7/d3c/d96 39 2024-04-22T15:49:44.404 INFO:tasks.workunit.client.0.smithi016.stdout:9/816: truncate d9/d57/d91/de4/ffe 651282 0 2024-04-22T15:49:44.404 INFO:tasks.workunit.client.0.smithi016.stdout:2/769: creat d2/d4d/f122 x:0 0 0 2024-04-22T15:49:44.405 INFO:tasks.workunit.client.0.smithi016.stdout:4/823: mknod d7/d21/d45/c10e 0 2024-04-22T15:49:44.407 INFO:tasks.workunit.client.0.smithi016.stdout:9/817: mknod d9/d20/dd7/c101 0 2024-04-22T15:49:44.408 INFO:tasks.workunit.client.0.smithi016.stdout:2/770: symlink d2/d1a/d4c/d77/dea/l123 0 2024-04-22T15:49:44.409 INFO:tasks.workunit.client.0.smithi016.stdout:8/747: dwrite d2/d82/d44/d66/d1c/f3d [8388608,4194304] 0 2024-04-22T15:49:44.411 INFO:tasks.workunit.client.0.smithi016.stdout:4/824: getdents d7/d4e/d39 0 2024-04-22T15:49:44.411 INFO:tasks.workunit.client.0.smithi016.stdout:9/818: read d9/d37/f3c [3838763,72784] 0 2024-04-22T15:49:44.411 INFO:tasks.workunit.client.0.smithi016.stdout:2/771: write d2/d1a/d4c/d9f/dae/fb3 [3911646,32992] 0 2024-04-22T15:49:44.415 INFO:tasks.workunit.client.0.smithi016.stdout:2/772: write d2/d23/d92/f71 [1673106,95694] 0 2024-04-22T15:49:44.418 INFO:tasks.workunit.client.0.smithi016.stdout:9/819: fsync d9/d57/d91/de4/d25/d36/f5e 0 2024-04-22T15:49:44.419 INFO:tasks.workunit.client.0.smithi016.stdout:2/773: mknod d2/d2c/d97/d34/d60/c124 0 2024-04-22T15:49:44.422 INFO:tasks.workunit.client.0.smithi016.stdout:2/774: mknod d2/d1a/d7b/dcc/de5/d107/c125 0 2024-04-22T15:49:44.425 INFO:tasks.workunit.client.0.smithi016.stdout:9/820: write d9/de/fec [338361,39569] 0 2024-04-22T15:49:44.428 INFO:tasks.workunit.client.0.smithi016.stdout:9/821: creat d9/d37/dea/f102 x:0 0 0 2024-04-22T15:49:44.431 INFO:tasks.workunit.client.0.smithi016.stdout:2/775: dwrite d2/de/f40 [4194304,4194304] 0 2024-04-22T15:49:44.433 INFO:tasks.workunit.client.0.smithi016.stdout:8/748: dwrite d2/f24 [0,4194304] 0 2024-04-22T15:49:44.439 INFO:tasks.workunit.client.0.smithi016.stdout:2/776: creat d2/d23/f126 x:0 0 0 2024-04-22T15:49:44.440 INFO:tasks.workunit.client.0.smithi016.stdout:9/822: dwrite d9/d37/f4a [0,4194304] 0 2024-04-22T15:49:44.441 INFO:tasks.workunit.client.0.smithi016.stdout:8/749: mkdir d2/d48/dad/d6a/ded 0 2024-04-22T15:49:44.446 INFO:tasks.workunit.client.0.smithi016.stdout:9/823: rmdir d9/d57/d91/dc1 39 2024-04-22T15:49:44.447 INFO:tasks.workunit.client.0.smithi016.stdout:2/777: dwrite d2/f5 [0,4194304] 0 2024-04-22T15:49:44.452 INFO:tasks.workunit.client.0.smithi016.stdout:2/778: fsync d2/d1a/d4c/d9f/da3/fcb 0 2024-04-22T15:49:44.452 INFO:tasks.workunit.client.0.smithi016.stdout:9/824: mkdir d9/d57/d91/d103 0 2024-04-22T15:49:44.454 INFO:tasks.workunit.client.0.smithi016.stdout:3/848: sync 2024-04-22T15:49:44.458 INFO:tasks.workunit.client.0.smithi016.stdout:3/849: fsync d7/f8 0 2024-04-22T15:49:44.458 INFO:tasks.workunit.client.0.smithi016.stdout:8/750: dwrite d2/d48/dad/d8f/d96/da5/feb [0,4194304] 0 2024-04-22T15:49:44.460 INFO:tasks.workunit.client.0.smithi016.stdout:8/751: chown d2/d82/d44/d66/d7e/dc0/de2 29717074 1 2024-04-22T15:49:44.461 INFO:tasks.workunit.client.0.smithi016.stdout:9/825: creat d9/de/d23/ddb/def/f104 x:0 0 0 2024-04-22T15:49:44.461 INFO:tasks.workunit.client.0.smithi016.stdout:3/850: fdatasync d7/d17/d18/d32/d42/d63/d86/dd8/ffc 0 2024-04-22T15:49:44.463 INFO:tasks.workunit.client.0.smithi016.stdout:9/826: write d9/d20/dac/ff3 [509697,80153] 0 2024-04-22T15:49:44.464 INFO:tasks.workunit.client.0.smithi016.stdout:8/752: rmdir d2/d82/d44/d66/d79 39 2024-04-22T15:49:44.465 INFO:tasks.workunit.client.0.smithi016.stdout:2/779: dread d2/d1a/d53/ff8 [0,4194304] 0 2024-04-22T15:49:44.467 INFO:tasks.workunit.client.0.smithi016.stdout:3/851: unlink d7/d17/d18/d32/d42/d63/deb/f4b 0 2024-04-22T15:49:44.468 INFO:tasks.workunit.client.0.smithi016.stdout:3/852: stat d7/d17/d18/d27 0 2024-04-22T15:49:44.470 INFO:tasks.workunit.client.0.smithi016.stdout:8/753: mknod d2/d82/d56/d99/cee 0 2024-04-22T15:49:44.471 INFO:tasks.workunit.client.0.smithi016.stdout:3/853: dread - d7/d17/d18/d32/d42/d63/deb/d21/fbf zero size 2024-04-22T15:49:44.471 INFO:tasks.workunit.client.0.smithi016.stdout:2/780: symlink d2/d2c/d97/d67/l127 0 2024-04-22T15:49:44.474 INFO:tasks.workunit.client.0.smithi016.stdout:9/827: dwrite d9/d57/d91/f99 [0,4194304] 0 2024-04-22T15:49:44.480 INFO:tasks.workunit.client.0.smithi016.stdout:2/781: link d2/d1a/d7b/dcc/de5/d107/c125 d2/d2c/d97/d67/dbf/c128 0 2024-04-22T15:49:44.480 INFO:tasks.workunit.client.0.smithi016.stdout:8/754: truncate d2/f24 1209834 0 2024-04-22T15:49:44.485 INFO:tasks.workunit.client.0.smithi016.stdout:8/755: write d2/d82/d44/d66/d7e/dc0/fcf [770879,70251] 0 2024-04-22T15:49:44.486 INFO:tasks.workunit.client.0.smithi016.stdout:9/828: dwrite d9/d57/d91/de4/d25/d36/d87/ffb [0,4194304] 0 2024-04-22T15:49:44.487 INFO:tasks.workunit.client.0.smithi016.stdout:8/756: dread - d2/d48/dad/d8f/fb3 zero size 2024-04-22T15:49:44.490 INFO:tasks.workunit.client.0.smithi016.stdout:2/782: dwrite d2/d2c/d97/d34/d60/f83 [4194304,4194304] 0 2024-04-22T15:49:44.492 INFO:tasks.workunit.client.0.smithi016.stdout:9/829: rename d9/de/fe6 to d9/de/d23/f105 0 2024-04-22T15:49:44.492 INFO:tasks.workunit.client.0.smithi016.stdout:2/783: chown d2/d1a/d4c/d9f/dae/lf1 2793 1 2024-04-22T15:49:44.494 INFO:tasks.workunit.client.0.smithi016.stdout:2/784: chown d2/d9/c19 0 1 2024-04-22T15:49:44.497 INFO:tasks.workunit.client.0.smithi016.stdout:9/830: mkdir d9/d20/dac/d106 0 2024-04-22T15:49:44.498 INFO:tasks.workunit.client.0.smithi016.stdout:8/757: dwrite d2/d32/f35 [0,4194304] 0 2024-04-22T15:49:44.502 INFO:tasks.workunit.client.0.smithi016.stdout:9/831: write d9/d20/f6f [2308152,112736] 0 2024-04-22T15:49:44.506 INFO:tasks.workunit.client.0.smithi016.stdout:2/785: dwrite d2/d1a/d4c/d9f/dae/fed [0,4194304] 0 2024-04-22T15:49:44.509 INFO:tasks.workunit.client.0.smithi016.stdout:2/786: chown d2/d4d/d64/f111 3 1 2024-04-22T15:49:44.515 INFO:tasks.workunit.client.0.smithi016.stdout:2/787: mknod d2/d1a/d7b/dcc/c129 0 2024-04-22T15:49:44.519 INFO:tasks.workunit.client.0.smithi016.stdout:2/788: dread d2/d1a/d7b/dc6/fde [0,4194304] 0 2024-04-22T15:49:44.522 INFO:tasks.workunit.client.0.smithi016.stdout:2/789: dread d2/f5 [0,4194304] 0 2024-04-22T15:49:44.528 INFO:tasks.workunit.client.0.smithi016.stdout:9/832: dread d9/de/d23/f78 [0,4194304] 0 2024-04-22T15:49:44.529 INFO:tasks.workunit.client.0.smithi016.stdout:2/790: dwrite d2/d23/d92/f66 [0,4194304] 0 2024-04-22T15:49:44.529 INFO:tasks.workunit.client.0.smithi016.stdout:2/791: chown d2/d1a/d4c/d9f/dae 59043 1 2024-04-22T15:49:44.532 INFO:tasks.workunit.client.0.smithi016.stdout:9/833: stat d9/d57/d91/dc1/da2/fba 0 2024-04-22T15:49:44.533 INFO:tasks.workunit.client.0.smithi016.stdout:2/792: mknod d2/de/c12a 0 2024-04-22T15:49:44.533 INFO:tasks.workunit.client.0.smithi016.stdout:2/793: fsync d2/d2c/f6b 0 2024-04-22T15:49:44.534 INFO:tasks.workunit.client.0.smithi016.stdout:9/834: symlink d9/d57/d91/de4/d25/d36/d87/l107 0 2024-04-22T15:49:44.534 INFO:tasks.workunit.client.0.smithi016.stdout:9/835: fsync d9/fa9 0 2024-04-22T15:49:44.536 INFO:tasks.workunit.client.0.smithi016.stdout:2/794: symlink d2/d1a/d4c/d77/l12b 0 2024-04-22T15:49:44.537 INFO:tasks.workunit.client.0.smithi016.stdout:9/836: rmdir d9/de/d26/d5a 39 2024-04-22T15:49:44.537 INFO:tasks.workunit.client.0.smithi016.stdout:2/795: dread - d2/de/d2a/d59/fbe zero size 2024-04-22T15:49:44.538 INFO:tasks.workunit.client.0.smithi016.stdout:2/796: truncate d2/d9/f101 216851 0 2024-04-22T15:49:44.540 INFO:tasks.workunit.client.0.smithi016.stdout:9/837: rmdir d9/de/d23/ddb/def 39 2024-04-22T15:49:44.543 INFO:tasks.workunit.client.0.smithi016.stdout:9/838: mkdir d9/d5d/d108 0 2024-04-22T15:49:44.545 INFO:tasks.workunit.client.0.smithi016.stdout:2/797: mkdir d2/d1a/d4c/d77/d12c 0 2024-04-22T15:49:44.545 INFO:tasks.workunit.client.0.smithi016.stdout:2/798: chown d2/d1a/l119 1929812 1 2024-04-22T15:49:44.549 INFO:tasks.workunit.client.0.smithi016.stdout:9/839: rename d9/d57/da7/dd9 to d9/d57/d91/de4/d25/d36/d87/d109 0 2024-04-22T15:49:44.550 INFO:tasks.workunit.client.0.smithi016.stdout:9/840: chown d9/d57/d91/de4/d25/da6/d4f/c8f 94305439 1 2024-04-22T15:49:44.551 INFO:tasks.workunit.client.0.smithi016.stdout:9/841: chown d9/fa 6199 1 2024-04-22T15:49:44.552 INFO:tasks.workunit.client.0.smithi016.stdout:2/799: mknod d2/d2c/daa/ddd/c12d 0 2024-04-22T15:49:44.555 INFO:tasks.workunit.client.0.smithi016.stdout:9/842: mknod d9/d20/dac/dbd/c10a 0 2024-04-22T15:49:44.564 INFO:tasks.workunit.client.0.smithi016.stdout:2/800: write d2/d1a/d7b/fdc [1100824,75217] 0 2024-04-22T15:49:44.568 INFO:tasks.workunit.client.0.smithi016.stdout:2/801: mkdir d2/d2c/d97/d12e 0 2024-04-22T15:49:44.569 INFO:tasks.workunit.client.0.smithi016.stdout:2/802: rmdir d2/d2c/d97/d67/dbf 39 2024-04-22T15:49:44.576 INFO:tasks.workunit.client.0.smithi016.stdout:2/803: rename d2/d4d/l68 to d2/d1a/d4c/d77/l12f 0 2024-04-22T15:49:44.579 INFO:tasks.workunit.client.0.smithi016.stdout:2/804: creat d2/d1a/d4c/d77/dea/f130 x:0 0 0 2024-04-22T15:49:44.582 INFO:tasks.workunit.client.0.smithi016.stdout:2/805: creat d2/d2c/daa/f131 x:0 0 0 2024-04-22T15:49:44.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:44 smithi027 ceph-mon[34922]: mgrmap e25: smithi027.ukdgzs(active, since 61s), standbys: smithi016.rnsnnr 2024-04-22T15:49:44.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:44 smithi027 ceph-mon[34922]: pgmap v34: 65 pgs: 65 active+clean; 1.8 GiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 78 MiB/s rd, 242 MiB/s wr, 589 op/s 2024-04-22T15:49:44.589 INFO:tasks.workunit.client.0.smithi016.stdout:2/806: dwrite d2/d2c/f82 [0,4194304] 0 2024-04-22T15:49:44.597 INFO:tasks.workunit.client.0.smithi016.stdout:2/807: dwrite d2/de/d2a/d59/fbe [0,4194304] 0 2024-04-22T15:49:44.598 INFO:tasks.workunit.client.0.smithi016.stdout:2/808: chown d2/d2c/d97/d34/fdb 0 1 2024-04-22T15:49:44.602 INFO:tasks.workunit.client.0.smithi016.stdout:2/809: creat d2/d1a/d7b/dc6/f132 x:0 0 0 2024-04-22T15:49:44.603 INFO:tasks.workunit.client.0.smithi016.stdout:2/810: dread - d2/d1a/d7b/dc6/fd5 zero size 2024-04-22T15:49:44.609 INFO:tasks.workunit.client.0.smithi016.stdout:2/811: mknod d2/d2c/d97/d67/dbf/c133 0 2024-04-22T15:49:44.612 INFO:tasks.workunit.client.0.smithi016.stdout:2/812: dread d2/f5 [0,4194304] 0 2024-04-22T15:49:44.616 INFO:tasks.workunit.client.0.smithi016.stdout:1/932: sync 2024-04-22T15:49:44.616 INFO:tasks.workunit.client.0.smithi016.stdout:5/829: sync 2024-04-22T15:49:44.617 INFO:tasks.workunit.client.0.smithi016.stdout:1/933: write d3/d4/d19/d20/d2b/de0/f10c [905282,39] 0 2024-04-22T15:49:44.618 INFO:tasks.workunit.client.0.smithi016.stdout:2/813: link d2/d1a/f5e d2/d1a/d4c/d9f/da3/f134 0 2024-04-22T15:49:44.625 INFO:tasks.workunit.client.0.smithi016.stdout:2/814: creat d2/d1a/d7b/dcc/f135 x:0 0 0 2024-04-22T15:49:44.632 INFO:tasks.workunit.client.0.smithi016.stdout:2/815: creat d2/f136 x:0 0 0 2024-04-22T15:49:44.656 INFO:tasks.workunit.client.0.smithi016.stdout:5/830: dwrite df/f57 [0,4194304] 0 2024-04-22T15:49:44.660 INFO:tasks.workunit.client.0.smithi016.stdout:5/831: mknod df/d15/d1a/d50/d77/d7d/d8b/d8c/ddf/c102 0 2024-04-22T15:49:44.661 INFO:tasks.workunit.client.0.smithi016.stdout:5/832: dread - df/d5b/d5d/d5f/fee zero size 2024-04-22T15:49:44.665 INFO:tasks.workunit.client.0.smithi016.stdout:5/833: rmdir df/d15/d1a/d50/d77/d94 39 2024-04-22T15:49:44.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:44 smithi016 ceph-mon[26620]: mgrmap e25: smithi027.ukdgzs(active, since 61s), standbys: smithi016.rnsnnr 2024-04-22T15:49:44.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:44 smithi016 ceph-mon[26620]: pgmap v34: 65 pgs: 65 active+clean; 1.8 GiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 78 MiB/s rd, 242 MiB/s wr, 589 op/s 2024-04-22T15:49:44.773 INFO:tasks.workunit.client.0.smithi016.stdout:7/781: sync 2024-04-22T15:49:44.773 INFO:tasks.workunit.client.0.smithi016.stdout:6/864: sync 2024-04-22T15:49:44.773 INFO:tasks.workunit.client.0.smithi016.stdout:0/874: sync 2024-04-22T15:49:44.774 INFO:tasks.workunit.client.0.smithi016.stdout:6/865: readlink d3/da/d12/d13/lba 0 2024-04-22T15:49:44.774 INFO:tasks.workunit.client.0.smithi016.stdout:6/866: readlink l1 0 2024-04-22T15:49:44.778 INFO:tasks.workunit.client.0.smithi016.stdout:7/782: rename d3/dc/d1b/d5a/d69 to d3/d11a 0 2024-04-22T15:49:44.779 INFO:tasks.workunit.client.0.smithi016.stdout:7/783: stat d3/d11a/d8c/fda 0 2024-04-22T15:49:44.781 INFO:tasks.workunit.client.0.smithi016.stdout:0/875: dwrite dc/d13/d14/de5/def/f10f [0,4194304] 0 2024-04-22T15:49:44.783 INFO:tasks.workunit.client.0.smithi016.stdout:7/784: creat d3/d11a/f11b x:0 0 0 2024-04-22T15:49:44.810 INFO:tasks.workunit.client.0.smithi016.stdout:0/876: getdents dc/d13/d14/de5/d65/d40/d4a 0 2024-04-22T15:49:44.812 INFO:tasks.workunit.client.0.smithi016.stdout:7/785: write d3/dc/f1e [941258,76841] 0 2024-04-22T15:49:44.814 INFO:tasks.workunit.client.0.smithi016.stdout:0/877: getdents dc/d13/d14/de5/d65/d57/dcb/de1 0 2024-04-22T15:49:44.818 INFO:tasks.workunit.client.0.smithi016.stdout:6/867: dwrite d3/da/db/d14/d4c/d58/f75 [0,4194304] 0 2024-04-22T15:49:44.826 INFO:tasks.workunit.client.0.smithi016.stdout:7/786: dwrite d3/dc/d1b/d5a/d59/d64/fa0 [0,4194304] 0 2024-04-22T15:49:44.826 INFO:tasks.workunit.client.0.smithi016.stdout:0/878: dwrite dc/d13/d14/d55/fbf [4194304,4194304] 0 2024-04-22T15:49:44.827 INFO:tasks.workunit.client.0.smithi016.stdout:7/787: write d3/dc/f1e [477224,101406] 0 2024-04-22T15:49:44.829 INFO:tasks.workunit.client.0.smithi016.stdout:6/868: rename d3/da/db/d44/d68/faa to d3/da/db/d44/d68/d95/d10c/f120 0 2024-04-22T15:49:44.830 INFO:tasks.workunit.client.0.smithi016.stdout:8/758: sync 2024-04-22T15:49:44.830 INFO:tasks.workunit.client.0.smithi016.stdout:4/825: sync 2024-04-22T15:49:44.830 INFO:tasks.workunit.client.0.smithi016.stdout:3/854: sync 2024-04-22T15:49:44.834 INFO:tasks.workunit.client.0.smithi016.stdout:0/879: rmdir dc/d13/d14/de5/d65/d1f/dd4/df4 39 2024-04-22T15:49:44.840 INFO:tasks.workunit.client.0.smithi016.stdout:4/826: write d7/d3c/f44 [5190532,35432] 0 2024-04-22T15:49:44.849 INFO:tasks.workunit.client.0.smithi016.stdout:4/827: mkdir d7/d21/d7c/d103/d10f 0 2024-04-22T15:49:44.849 INFO:tasks.workunit.client.0.smithi016.stdout:6/869: dwrite d3/da/f45 [4194304,4194304] 0 2024-04-22T15:49:44.851 INFO:tasks.workunit.client.0.smithi016.stdout:3/855: rename d7/d17/d18/d32/d3c/d47/fc4 to d7/d17/d18/d32/d42/d63/deb/d21/d55/dac/f108 0 2024-04-22T15:49:44.852 INFO:tasks.workunit.client.0.smithi016.stdout:8/759: truncate d2/d48/dad/d6a/fbf 471632 0 2024-04-22T15:49:44.853 INFO:tasks.workunit.client.0.smithi016.stdout:8/760: fsync d2/d10/f39 0 2024-04-22T15:49:44.858 INFO:tasks.workunit.client.0.smithi016.stdout:0/880: dwrite dc/d13/d14/de5/d65/d30/d96/fc7 [0,4194304] 0 2024-04-22T15:49:44.862 INFO:tasks.workunit.client.0.smithi016.stdout:8/761: read d2/d82/d44/d66/d1c/f81 [258348,25214] 0 2024-04-22T15:49:44.863 INFO:tasks.workunit.client.0.smithi016.stdout:4/828: creat d7/d21/d45/d49/de4/f110 x:0 0 0 2024-04-22T15:49:44.864 INFO:tasks.workunit.client.0.smithi016.stdout:7/788: dwrite d3/dc/d1b/d5a/d9/df/fd8 [0,4194304] 0 2024-04-22T15:49:44.866 INFO:tasks.workunit.client.0.smithi016.stdout:8/762: stat d2/d82/d44/d66/d1c/d65/f9c 0 2024-04-22T15:49:44.867 INFO:tasks.workunit.client.0.smithi016.stdout:3/856: rename d7/d17/d18/d32/d3c/l82 to d7/d77/l109 0 2024-04-22T15:49:44.868 INFO:tasks.workunit.client.0.smithi016.stdout:8/763: chown d2/d10/d47/fb2 3182886 1 2024-04-22T15:49:44.871 INFO:tasks.workunit.client.0.smithi016.stdout:4/829: fsync d7/d22/fef 0 2024-04-22T15:49:44.871 INFO:tasks.workunit.client.0.smithi016.stdout:3/857: dread - d7/d17/d18/d32/d42/d63/deb/d21/fbf zero size 2024-04-22T15:49:44.872 INFO:tasks.workunit.client.0.smithi016.stdout:6/870: write d3/da/dd/d1b/f8b [44310,93876] 0 2024-04-22T15:49:44.872 INFO:tasks.workunit.client.0.smithi016.stdout:0/881: mknod dc/d13/d14/de5/d65/d40/d4a/db2/c119 0 2024-04-22T15:49:44.873 INFO:tasks.workunit.client.0.smithi016.stdout:7/789: write d3/de6/f85 [4773711,123358] 0 2024-04-22T15:49:44.876 INFO:tasks.workunit.client.0.smithi016.stdout:7/790: write d3/de6/f7c [3428845,95509] 0 2024-04-22T15:49:44.879 INFO:tasks.workunit.client.0.smithi016.stdout:0/882: mkdir dc/d13/d14/de5/d65/d40/dd5/d11a 0 2024-04-22T15:49:44.885 INFO:tasks.workunit.client.0.smithi016.stdout:6/871: getdents d3/da/ddd/d101/d11b/d35 0 2024-04-22T15:49:44.885 INFO:tasks.workunit.client.0.smithi016.stdout:8/764: dwrite d2/d82/d44/d66/d1c/d65/f9c [0,4194304] 0 2024-04-22T15:49:44.887 INFO:tasks.workunit.client.0.smithi016.stdout:1/934: sync 2024-04-22T15:49:44.887 INFO:tasks.workunit.client.0.smithi016.stdout:9/843: sync 2024-04-22T15:49:44.893 INFO:tasks.workunit.client.0.smithi016.stdout:1/935: chown d3/d2e/d80 1381 1 2024-04-22T15:49:44.893 INFO:tasks.workunit.client.0.smithi016.stdout:0/883: truncate dc/d13/d14/d26/f5e 1665231 0 2024-04-22T15:49:44.895 INFO:tasks.workunit.client.0.smithi016.stdout:8/765: truncate d2/d82/d44/d6f/fa6 989226 0 2024-04-22T15:49:44.895 INFO:tasks.workunit.client.0.smithi016.stdout:7/791: dwrite d3/dc/d1b/d5a/d4a/ffa [0,4194304] 0 2024-04-22T15:49:44.896 INFO:tasks.workunit.client.0.smithi016.stdout:6/872: readlink d3/da/dd/d67/d6c/dfc/l10a 0 2024-04-22T15:49:44.896 INFO:tasks.workunit.client.0.smithi016.stdout:9/844: rmdir d9/de/d26/d5a 39 2024-04-22T15:49:44.898 INFO:tasks.workunit.client.0.smithi016.stdout:0/884: creat dc/d13/d14/de5/d65/d40/d4a/d59/f11b x:0 0 0 2024-04-22T15:49:44.899 INFO:tasks.workunit.client.0.smithi016.stdout:1/936: fsync d3/d4/faa 0 2024-04-22T15:49:44.900 INFO:tasks.workunit.client.0.smithi016.stdout:8/766: mkdir d2/d82/d44/d66/d1c/d65/def 0 2024-04-22T15:49:44.904 INFO:tasks.workunit.client.0.smithi016.stdout:1/937: dread - d3/d2e/f13f zero size 2024-04-22T15:49:44.905 INFO:tasks.workunit.client.0.smithi016.stdout:9/845: rename d9/d20/c88 to d9/d20/dac/dae/c10b 0 2024-04-22T15:49:44.907 INFO:tasks.workunit.client.0.smithi016.stdout:0/885: write dc/d13/d14/d26/d32/f33 [1187399,93025] 0 2024-04-22T15:49:44.910 INFO:tasks.workunit.client.0.smithi016.stdout:8/767: fsync d2/d48/dad/d6a/fbf 0 2024-04-22T15:49:44.911 INFO:tasks.workunit.client.0.smithi016.stdout:2/816: dread d2/d1a/d4c/d9f/da3/fcb [0,4194304] 0 2024-04-22T15:49:44.914 INFO:tasks.workunit.client.0.smithi016.stdout:6/873: write d3/da/d12/d13/f76 [1880283,88797] 0 2024-04-22T15:49:44.916 INFO:tasks.workunit.client.0.smithi016.stdout:0/886: creat dc/d13/d14/de5/d65/d30/d96/f11c x:0 0 0 2024-04-22T15:49:44.917 INFO:tasks.workunit.client.0.smithi016.stdout:9/846: creat d9/d57/d91/d103/f10c x:0 0 0 2024-04-22T15:49:44.921 INFO:tasks.workunit.client.0.smithi016.stdout:7/792: link d3/dc/l27 d3/dc/d9f/da1/l11c 0 2024-04-22T15:49:44.924 INFO:tasks.workunit.client.0.smithi016.stdout:0/887: dwrite dc/d13/d14/de5/def/f10f [0,4194304] 0 2024-04-22T15:49:44.926 INFO:tasks.workunit.client.0.smithi016.stdout:1/938: rename d3/d4/d19/f35 to d3/d4/d15/d38/f141 0 2024-04-22T15:49:44.927 INFO:tasks.workunit.client.0.smithi016.stdout:6/874: symlink d3/da/dd/d67/l121 0 2024-04-22T15:49:44.929 INFO:tasks.workunit.client.0.smithi016.stdout:5/834: dread df/d15/d1a/d4e/f69 [0,4194304] 0 2024-04-22T15:49:44.931 INFO:tasks.workunit.client.0.smithi016.stdout:0/888: readlink dc/d13/d14/l4b 0 2024-04-22T15:49:44.933 INFO:tasks.workunit.client.0.smithi016.stdout:8/768: rename d2/d82/d44 to d2/d82/d52/df0 0 2024-04-22T15:49:44.935 INFO:tasks.workunit.client.0.smithi016.stdout:6/875: fsync d3/da/db/d44/d68/fa5 0 2024-04-22T15:49:44.939 INFO:tasks.workunit.client.0.smithi016.stdout:0/889: write dc/d13/d14/f34 [4097517,108877] 0 2024-04-22T15:49:44.941 INFO:tasks.workunit.client.0.smithi016.stdout:8/769: mknod d2/d48/dad/cf1 0 2024-04-22T15:49:44.941 INFO:tasks.workunit.client.0.smithi016.stdout:2/817: write d2/d2c/d97/f5f [879474,82005] 0 2024-04-22T15:49:44.941 INFO:tasks.workunit.client.0.smithi016.stdout:8/770: stat d2/d82/d3e/lce 0 2024-04-22T15:49:44.943 INFO:tasks.workunit.client.0.smithi016.stdout:6/876: creat d3/da/dd/d67/f122 x:0 0 0 2024-04-22T15:49:44.943 INFO:tasks.workunit.client.0.smithi016.stdout:9/847: write d9/f3b [628501,95008] 0 2024-04-22T15:49:44.945 INFO:tasks.workunit.client.0.smithi016.stdout:7/793: truncate d3/d11a/f90 211170 0 2024-04-22T15:49:44.950 INFO:tasks.workunit.client.0.smithi016.stdout:8/771: unlink d2/d48/dad/d8f/d96/da5/feb 0 2024-04-22T15:49:44.950 INFO:tasks.workunit.client.0.smithi016.stdout:8/772: stat d2/d32 0 2024-04-22T15:49:44.951 INFO:tasks.workunit.client.0.smithi016.stdout:1/939: rename d3/d4/d3a/d6b/f94 to d3/f142 0 2024-04-22T15:49:44.951 INFO:tasks.workunit.client.0.smithi016.stdout:6/877: rmdir d3/da/d12/d13/d38/df5 39 2024-04-22T15:49:44.954 INFO:tasks.workunit.client.0.smithi016.stdout:8/773: dread d2/d82/d56/f90 [0,4194304] 0 2024-04-22T15:49:44.955 INFO:tasks.workunit.client.0.smithi016.stdout:0/890: link dc/d13/d14/d26/d32/f33 dc/d13/d70/d8c/f11d 0 2024-04-22T15:49:44.958 INFO:tasks.workunit.client.0.smithi016.stdout:1/940: rmdir d3/d4/d3a/d6b/dc3 39 2024-04-22T15:49:44.960 INFO:tasks.workunit.client.0.smithi016.stdout:9/848: symlink d9/d57/d91/l10d 0 2024-04-22T15:49:44.962 INFO:tasks.workunit.client.0.smithi016.stdout:0/891: chown dc/d13/d14/de5/d65/d30 306448 1 2024-04-22T15:49:44.964 INFO:tasks.workunit.client.0.smithi016.stdout:8/774: rmdir d2/d48/dad/d8f 39 2024-04-22T15:49:44.965 INFO:tasks.workunit.client.0.smithi016.stdout:2/818: dwrite d2/d51/fbd [0,4194304] 0 2024-04-22T15:49:44.970 INFO:tasks.workunit.client.0.smithi016.stdout:6/878: dread d3/da/db/d14/d4c/d58/f75 [0,4194304] 0 2024-04-22T15:49:44.970 INFO:tasks.workunit.client.0.smithi016.stdout:8/775: write d2/d82/f7b [2828064,32247] 0 2024-04-22T15:49:44.974 INFO:tasks.workunit.client.0.smithi016.stdout:1/941: link d3/d4/dc4/f12e d3/d4/d3a/d91/db3/f143 0 2024-04-22T15:49:44.974 INFO:tasks.workunit.client.0.smithi016.stdout:0/892: fdatasync dc/d13/f31 0 2024-04-22T15:49:44.979 INFO:tasks.workunit.client.0.smithi016.stdout:6/879: chown d3/da/dd/d67/l121 130719416 1 2024-04-22T15:49:44.980 INFO:tasks.workunit.client.0.smithi016.stdout:7/794: dwrite d3/dc/d1b/d31/d5f/f77 [0,4194304] 0 2024-04-22T15:49:44.983 INFO:tasks.workunit.client.0.smithi016.stdout:1/942: creat d3/d2e/f144 x:0 0 0 2024-04-22T15:49:44.984 INFO:tasks.workunit.client.0.smithi016.stdout:9/849: dwrite d9/d20/dac/ff3 [0,4194304] 0 2024-04-22T15:49:44.987 INFO:tasks.workunit.client.0.smithi016.stdout:9/850: chown d9/d57/d91/de4/d25/f89 67495821 1 2024-04-22T15:49:44.987 INFO:tasks.workunit.client.0.smithi016.stdout:2/819: getdents d2/d1a/d4c/d9f/dae 0 2024-04-22T15:49:44.988 INFO:tasks.workunit.client.0.smithi016.stdout:8/776: creat d2/d82/d52/df0/d66/d1c/d65/def/ff2 x:0 0 0 2024-04-22T15:49:44.991 INFO:tasks.workunit.client.0.smithi016.stdout:6/880: rmdir d3/da/dd/d67/d6c/db9/dfa/d118 39 2024-04-22T15:49:44.992 INFO:tasks.workunit.client.0.smithi016.stdout:1/943: creat d3/d2e/d9f/f145 x:0 0 0 2024-04-22T15:49:44.994 INFO:tasks.workunit.client.0.smithi016.stdout:9/851: fsync d9/de/d23/f78 0 2024-04-22T15:49:44.995 INFO:tasks.workunit.client.0.smithi016.stdout:8/777: rmdir d2/d82/d3e 39 2024-04-22T15:49:44.995 INFO:tasks.workunit.client.0.smithi016.stdout:2/820: mknod d2/d2c/c137 0 2024-04-22T15:49:44.995 INFO:tasks.workunit.client.0.smithi016.stdout:0/893: link dc/d13/d14/de5/d65/d30/d96/f11c dc/d13/d14/de5/d65/d40/d4a/db2/f11e 0 2024-04-22T15:49:44.998 INFO:tasks.workunit.client.0.smithi016.stdout:9/852: fsync d9/d37/f3c 0 2024-04-22T15:49:44.998 INFO:tasks.workunit.client.0.smithi016.stdout:6/881: mknod d3/da/d12/d13/dd2/dee/d11c/c123 0 2024-04-22T15:49:44.999 INFO:tasks.workunit.client.0.smithi016.stdout:8/778: mknod d2/d82/d56/d99/cf3 0 2024-04-22T15:49:45.003 INFO:tasks.workunit.client.0.smithi016.stdout:0/894: creat dc/d13/d14/de5/d65/d40/dd6/f11f x:0 0 0 2024-04-22T15:49:45.007 INFO:tasks.workunit.client.0.smithi016.stdout:9/853: symlink d9/de/d23/ddb/def/l10e 0 2024-04-22T15:49:45.013 INFO:tasks.workunit.client.0.smithi016.stdout:9/854: dread - d9/d57/da7/daa/fc9 zero size 2024-04-22T15:49:45.014 INFO:tasks.workunit.client.0.smithi016.stdout:6/882: dwrite d3/da/db/d44/d68/d95/fcf [4194304,4194304] 0 2024-04-22T15:49:45.021 INFO:tasks.workunit.client.0.smithi016.stdout:9/855: read d9/d37/fb7 [56591,19165] 0 2024-04-22T15:49:45.024 INFO:tasks.workunit.client.0.smithi016.stdout:1/944: dwrite d3/d4/d19/d20/d95/ffc [0,4194304] 0 2024-04-22T15:49:45.032 INFO:tasks.workunit.client.0.smithi016.stdout:6/883: mknod d3/da/db/d44/c124 0 2024-04-22T15:49:45.035 INFO:tasks.workunit.client.0.smithi016.stdout:9/856: symlink d9/d57/d91/de4/d25/d36/d87/l10f 0 2024-04-22T15:49:45.035 INFO:tasks.workunit.client.0.smithi016.stdout:2/821: dread d2/f2b [0,4194304] 0 2024-04-22T15:49:45.035 INFO:tasks.workunit.client.0.smithi016.stdout:0/895: dwrite dc/d13/d14/de5/d65/d1f/fcf [8388608,4194304] 0 2024-04-22T15:49:45.038 INFO:tasks.workunit.client.0.smithi016.stdout:8/779: dread d2/d10/d47/f95 [0,4194304] 0 2024-04-22T15:49:45.038 INFO:tasks.workunit.client.0.smithi016.stdout:1/945: truncate d3/d4/d3a/d91/db3/dd2/f10b 895319 0 2024-04-22T15:49:45.042 INFO:tasks.workunit.client.0.smithi016.stdout:6/884: fsync d3/da/f69 0 2024-04-22T15:49:45.043 INFO:tasks.workunit.client.0.smithi016.stdout:6/885: stat d3/da/db/d14/d16/f55 0 2024-04-22T15:49:45.045 INFO:tasks.workunit.client.0.smithi016.stdout:1/946: mknod d3/d4/d15/c146 0 2024-04-22T15:49:45.047 INFO:tasks.workunit.client.0.smithi016.stdout:8/780: fdatasync d2/d82/d52/df0/d66/fc 0 2024-04-22T15:49:45.048 INFO:tasks.workunit.client.0.smithi016.stdout:2/822: mkdir d2/d2c/d97/d67/d138 0 2024-04-22T15:49:45.049 INFO:tasks.workunit.client.0.smithi016.stdout:8/781: chown d2/d32/f35 28752 1 2024-04-22T15:49:45.051 INFO:tasks.workunit.client.0.smithi016.stdout:9/857: creat d9/d57/d91/f110 x:0 0 0 2024-04-22T15:49:45.053 INFO:tasks.workunit.client.0.smithi016.stdout:1/947: read d3/d4/d19/fb2 [2141407,24768] 0 2024-04-22T15:49:45.054 INFO:tasks.workunit.client.0.smithi016.stdout:2/823: truncate d2/d4d/d64/f111 888756 0 2024-04-22T15:49:45.058 INFO:tasks.workunit.client.0.smithi016.stdout:6/886: link d3/da/ddd/d101/d11b/d83/cc5 d3/da/c125 0 2024-04-22T15:49:45.058 INFO:tasks.workunit.client.0.smithi016.stdout:0/896: dwrite dc/d13/d6f/d8b/ffe [0,4194304] 0 2024-04-22T15:49:45.063 INFO:tasks.workunit.client.0.smithi016.stdout:0/897: write dc/d13/d14/de5/d65/d40/d4a/d59/f11b [123897,79909] 0 2024-04-22T15:49:45.064 INFO:tasks.workunit.client.0.smithi016.stdout:9/858: write d9/d57/d91/de4/f53 [1779631,74939] 0 2024-04-22T15:49:45.067 INFO:tasks.workunit.client.0.smithi016.stdout:6/887: rename d3/da/dd/d1b/d57/fc4 to d3/da/ddd/d101/d11a/f126 0 2024-04-22T15:49:45.068 INFO:tasks.workunit.client.0.smithi016.stdout:0/898: mkdir dc/d13/d22/d120 0 2024-04-22T15:49:45.071 INFO:tasks.workunit.client.0.smithi016.stdout:3/858: sync 2024-04-22T15:49:45.071 INFO:tasks.workunit.client.0.smithi016.stdout:4/830: sync 2024-04-22T15:49:45.071 INFO:tasks.workunit.client.0.smithi016.stdout:5/835: sync 2024-04-22T15:49:45.072 INFO:tasks.workunit.client.0.smithi016.stdout:2/824: rename d2/d4d/dc4/l115 to d2/d1a/d7b/dcc/de5/d107/l139 0 2024-04-22T15:49:45.075 INFO:tasks.workunit.client.0.smithi016.stdout:8/782: dwrite d2/d82/d3e/dd5/fd8 [0,4194304] 0 2024-04-22T15:49:45.077 INFO:tasks.workunit.client.0.smithi016.stdout:3/859: rmdir d7/d17/d18/d27 39 2024-04-22T15:49:45.080 INFO:tasks.workunit.client.0.smithi016.stdout:6/888: rmdir d3/da/dd/d67/d6c/db9/d100 0 2024-04-22T15:49:45.081 INFO:tasks.workunit.client.0.smithi016.stdout:6/889: chown d3/da/dd/d1b/d57/fbc 1771 1 2024-04-22T15:49:45.084 INFO:tasks.workunit.client.0.smithi016.stdout:9/859: rename d9/f1c to d9/d57/da7/f111 0 2024-04-22T15:49:45.084 INFO:tasks.workunit.client.0.smithi016.stdout:9/860: stat d9/de/d26/caf 0 2024-04-22T15:49:45.085 INFO:tasks.workunit.client.0.smithi016.stdout:0/899: creat dc/d13/d14/de5/d65/d40/d4a/d92/f121 x:0 0 0 2024-04-22T15:49:45.085 INFO:tasks.workunit.client.0.smithi016.stdout:9/861: fsync d9/d37/dea/f102 0 2024-04-22T15:49:45.086 INFO:tasks.workunit.client.0.smithi016.stdout:0/900: chown dc/d13/d14/d26/d32/fb0 11033516 1 2024-04-22T15:49:45.087 INFO:tasks.workunit.client.0.smithi016.stdout:2/825: unlink d2/d1a/d4c/f3f 0 2024-04-22T15:49:45.087 INFO:tasks.workunit.client.0.smithi016.stdout:4/831: truncate d7/d24/d41/f74 1337946 0 2024-04-22T15:49:45.089 INFO:tasks.workunit.client.0.smithi016.stdout:8/783: dread - d2/d48/dad/d8f/fb3 zero size 2024-04-22T15:49:45.089 INFO:tasks.workunit.client.0.smithi016.stdout:3/860: creat d7/d17/d18/d32/d42/d63/db4/f10a x:0 0 0 2024-04-22T15:49:45.091 INFO:tasks.workunit.client.0.smithi016.stdout:0/901: rmdir dc/d13/d6f 39 2024-04-22T15:49:45.092 INFO:tasks.workunit.client.0.smithi016.stdout:5/836: mkdir df/d15/d1a/d50/d77/d7d/d8b/d93/dfa/d103 0 2024-04-22T15:49:45.094 INFO:tasks.workunit.client.0.smithi016.stdout:9/862: creat d9/d57/d91/de4/d25/d36/d87/f112 x:0 0 0 2024-04-22T15:49:45.098 INFO:tasks.workunit.client.0.smithi016.stdout:2/826: rename d2/d23/d92/d55/f11d to d2/d23/db8/dd4/d114/d11b/f13a 0 2024-04-22T15:49:45.099 INFO:tasks.workunit.client.0.smithi016.stdout:6/890: fdatasync d3/da/db/d14/d4c/f52 0 2024-04-22T15:49:45.102 INFO:tasks.workunit.client.0.smithi016.stdout:9/863: rmdir d9/d20/dac 39 2024-04-22T15:49:45.102 INFO:tasks.workunit.client.0.smithi016.stdout:5/837: rmdir df/d15/d1a/d4e/d58/d87 39 2024-04-22T15:49:45.105 INFO:tasks.workunit.client.0.smithi016.stdout:0/902: dwrite dc/d13/d14/de5/d65/d40/d4a/d59/d78/fbc [0,4194304] 0 2024-04-22T15:49:45.106 INFO:tasks.workunit.client.0.smithi016.stdout:8/784: dread d2/d82/d3e/dd5/fd8 [0,4194304] 0 2024-04-22T15:49:45.107 INFO:tasks.workunit.client.0.smithi016.stdout:2/827: truncate d2/d2c/de7/ff4 363100 0 2024-04-22T15:49:45.107 INFO:tasks.workunit.client.0.smithi016.stdout:9/864: stat d9/d57/d91/de4/d25/da6/cd8 0 2024-04-22T15:49:45.110 INFO:tasks.workunit.client.0.smithi016.stdout:4/832: dread d7/d4e/d39/dc3/fa5 [0,4194304] 0 2024-04-22T15:49:45.111 INFO:tasks.workunit.client.0.smithi016.stdout:3/861: creat d7/d17/d18/d32/f10b x:0 0 0 2024-04-22T15:49:45.112 INFO:tasks.workunit.client.0.smithi016.stdout:2/828: creat d2/d51/f13b x:0 0 0 2024-04-22T15:49:45.114 INFO:tasks.workunit.client.0.smithi016.stdout:6/891: write d3/da/dd/d67/dbd/fd6 [410034,121466] 0 2024-04-22T15:49:45.115 INFO:tasks.workunit.client.0.smithi016.stdout:9/865: symlink d9/d57/d91/de4/d25/d36/d87/l113 0 2024-04-22T15:49:45.116 INFO:tasks.workunit.client.0.smithi016.stdout:6/892: chown d3/da/d12/d13/dd2/dee/d11c 1 1 2024-04-22T15:49:45.117 INFO:tasks.workunit.client.0.smithi016.stdout:9/866: chown d9/d57/d91/de4/d25/f89 586 1 2024-04-22T15:49:45.117 INFO:tasks.workunit.client.0.smithi016.stdout:4/833: creat d7/d4e/d102/d106/d88/deb/f111 x:0 0 0 2024-04-22T15:49:45.118 INFO:tasks.workunit.client.0.smithi016.stdout:9/867: fdatasync d9/de/d23/ddb/def/fe8 0 2024-04-22T15:49:45.120 INFO:tasks.workunit.client.0.smithi016.stdout:3/862: unlink d7/d17/d18/d32/d42/d63/deb/d21/l2b 0 2024-04-22T15:49:45.122 INFO:tasks.workunit.client.0.smithi016.stdout:9/868: write d9/d20/f56 [999001,77188] 0 2024-04-22T15:49:45.122 INFO:tasks.workunit.client.0.smithi016.stdout:2/829: creat d2/d2c/d97/d67/f13c x:0 0 0 2024-04-22T15:49:45.123 INFO:tasks.workunit.client.0.smithi016.stdout:0/903: getdents dc/d13/d14/de5/d65/d30/dc9 0 2024-04-22T15:49:45.124 INFO:tasks.workunit.client.0.smithi016.stdout:3/863: mknod d7/d17/d18/c10c 0 2024-04-22T15:49:45.125 INFO:tasks.workunit.client.0.smithi016.stdout:9/869: rename d9/d20/fd1 to d9/d5d/f114 0 2024-04-22T15:49:45.126 INFO:tasks.workunit.client.0.smithi016.stdout:2/830: creat d2/d23/db8/dd4/d114/d11b/f13d x:0 0 0 2024-04-22T15:49:45.127 INFO:tasks.workunit.client.0.smithi016.stdout:0/904: write dc/d13/d14/de5/d65/d40/d4a/ff8 [349642,63621] 0 2024-04-22T15:49:45.128 INFO:tasks.workunit.client.0.smithi016.stdout:3/864: chown d7/d17/d18/d32/l6a 643906 1 2024-04-22T15:49:45.128 INFO:tasks.workunit.client.0.smithi016.stdout:9/870: fsync d9/d20/f38 0 2024-04-22T15:49:45.128 INFO:tasks.workunit.client.0.smithi016.stdout:2/831: fdatasync d2/f7 0 2024-04-22T15:49:45.133 INFO:tasks.workunit.client.0.smithi016.stdout:2/832: write d2/d2c/d97/d67/f13c [10375,109030] 0 2024-04-22T15:49:45.135 INFO:tasks.workunit.client.0.smithi016.stdout:3/865: creat d7/d17/d18/d32/d3c/d47/f10d x:0 0 0 2024-04-22T15:49:45.137 INFO:tasks.workunit.client.0.smithi016.stdout:9/871: rename d9/l2d to d9/d57/d91/de4/d25/d36/d87/d109/l115 0 2024-04-22T15:49:45.137 INFO:tasks.workunit.client.0.smithi016.stdout:0/905: dwrite dc/d13/d14/ff9 [0,4194304] 0 2024-04-22T15:49:45.139 INFO:tasks.workunit.client.0.smithi016.stdout:0/906: readlink dc/d13/d14/d26/l95 0 2024-04-22T15:49:45.142 INFO:tasks.workunit.client.0.smithi016.stdout:3/866: fsync d7/d17/d18/d32/d3c/fd7 0 2024-04-22T15:49:45.146 INFO:tasks.workunit.client.0.smithi016.stdout:9/872: rename d9/d5d/d108 to d9/d57/d91/de4/d25/d36/d116 0 2024-04-22T15:49:45.146 INFO:tasks.workunit.client.0.smithi016.stdout:9/873: stat d9/d37/f49 0 2024-04-22T15:49:45.148 INFO:tasks.workunit.client.0.smithi016.stdout:2/833: truncate d2/d2c/d97/d34/fdb 988472 0 2024-04-22T15:49:45.149 INFO:tasks.workunit.client.0.smithi016.stdout:0/907: creat dc/d13/d14/de5/d65/d1f/dd4/df4/f122 x:0 0 0 2024-04-22T15:49:45.151 INFO:tasks.workunit.client.0.smithi016.stdout:3/867: rename d7/d17/d18/d32/d42/d63/deb/d21/d55/fb6 to d7/d17/d18/d32/d42/d63/f10e 0 2024-04-22T15:49:45.152 INFO:tasks.workunit.client.0.smithi016.stdout:3/868: chown d7/d17/d18/d32/d42/d63/deb/d21/d55 764443 1 2024-04-22T15:49:45.153 INFO:tasks.workunit.client.0.smithi016.stdout:9/874: rename d9/lfd to d9/d57/d91/de4/d25/d36/d116/l117 0 2024-04-22T15:49:45.155 INFO:tasks.workunit.client.0.smithi016.stdout:0/908: getdents dc/d7d 0 2024-04-22T15:49:45.158 INFO:tasks.workunit.client.0.smithi016.stdout:3/869: write d7/d17/d18/d27/fb8 [1477368,81155] 0 2024-04-22T15:49:45.159 INFO:tasks.workunit.client.0.smithi016.stdout:0/909: fsync dc/d13/d14/de5/d65/d57/f8d 0 2024-04-22T15:49:45.162 INFO:tasks.workunit.client.0.smithi016.stdout:9/875: dwrite d9/d37/dea/f102 [0,4194304] 0 2024-04-22T15:49:45.168 INFO:tasks.workunit.client.0.smithi016.stdout:9/876: mkdir d9/d5d/d118 0 2024-04-22T15:49:45.168 INFO:tasks.workunit.client.0.smithi016.stdout:9/877: write d9/d57/f9b [855404,23964] 0 2024-04-22T15:49:45.168 INFO:tasks.workunit.client.0.smithi016.stdout:0/910: rename dc/d13/d14/de5/d65/d40/c43 to dc/d13/d14/de5/dd1/ddc/df6/c123 0 2024-04-22T15:49:45.169 INFO:tasks.workunit.client.0.smithi016.stdout:9/878: chown d9/d5d 1542943 1 2024-04-22T15:49:45.174 INFO:tasks.workunit.client.0.smithi016.stdout:0/911: rename dc/f7e to dc/d13/d14/de5/d65/d57/dcb/de1/f124 0 2024-04-22T15:49:45.175 INFO:tasks.workunit.client.0.smithi016.stdout:9/879: mknod d9/de/c119 0 2024-04-22T15:49:45.175 INFO:tasks.workunit.client.0.smithi016.stdout:0/912: write dc/d13/d14/de5/d65/d40/dbb/fc2 [375840,33958] 0 2024-04-22T15:49:45.179 INFO:tasks.workunit.client.0.smithi016.stdout:9/880: mknod d9/d57/d91/de4/d25/da6/d4f/c11a 0 2024-04-22T15:49:45.179 INFO:tasks.workunit.client.0.smithi016.stdout:0/913: fsync dc/d13/d14/de5/d65/d40/d4a/d59/d78/dbd/fda 0 2024-04-22T15:49:45.182 INFO:tasks.workunit.client.0.smithi016.stdout:3/870: dwrite d7/d17/d18/d32/d42/d63/deb/d21/d55/f6f [0,4194304] 0 2024-04-22T15:49:45.190 INFO:tasks.workunit.client.0.smithi016.stdout:3/871: mknod d7/d17/d18/d32/d42/d63/deb/d21/d55/ddb/db7/c10f 0 2024-04-22T15:49:45.190 INFO:tasks.workunit.client.0.smithi016.stdout:9/881: dwrite d9/d57/d91/de4/d25/da6/d4f/fd5 [0,4194304] 0 2024-04-22T15:49:45.194 INFO:tasks.workunit.client.0.smithi016.stdout:9/882: mkdir d9/d37/dea/d11b 0 2024-04-22T15:49:45.197 INFO:tasks.workunit.client.0.smithi016.stdout:9/883: readlink d9/l12 0 2024-04-22T15:49:45.199 INFO:tasks.workunit.client.0.smithi016.stdout:3/872: dwrite d7/d17/d18/d32/d3c/fd7 [0,4194304] 0 2024-04-22T15:49:45.202 INFO:tasks.workunit.client.0.smithi016.stdout:3/873: fdatasync d7/d17/d18/d32/d8a/f9b 0 2024-04-22T15:49:45.204 INFO:tasks.workunit.client.1.smithi027.stdout:2/29: sync 2024-04-22T15:49:45.204 INFO:tasks.workunit.client.1.smithi027.stdout:8/10: sync 2024-04-22T15:49:45.205 INFO:tasks.workunit.client.1.smithi027.stdout:2/30: write f0 [4027934,115901] 0 2024-04-22T15:49:45.206 INFO:tasks.workunit.client.1.smithi027.stdout:5/15: sync 2024-04-22T15:49:45.206 INFO:tasks.workunit.client.1.smithi027.stdout:3/36: sync 2024-04-22T15:49:45.206 INFO:tasks.workunit.client.1.smithi027.stdout:7/14: sync 2024-04-22T15:49:45.206 INFO:tasks.workunit.client.1.smithi027.stdout:2/31: stat l3 0 2024-04-22T15:49:45.207 INFO:tasks.workunit.client.1.smithi027.stdout:2/32: readlink l3 0 2024-04-22T15:49:45.207 INFO:tasks.workunit.client.1.smithi027.stdout:7/15: chown d0/f3 6372785 1 2024-04-22T15:49:45.208 INFO:tasks.workunit.client.0.smithi016.stdout:0/914: dread dc/d13/d14/de5/d65/d40/dd6/d104/fb5 [0,4194304] 0 2024-04-22T15:49:45.208 INFO:tasks.workunit.client.1.smithi027.stdout:5/16: truncate d0/f3 1478636 0 2024-04-22T15:49:45.209 INFO:tasks.workunit.client.0.smithi016.stdout:9/884: dwrite d9/fa9 [0,4194304] 0 2024-04-22T15:49:45.212 INFO:tasks.workunit.client.1.smithi027.stdout:5/17: creat d0/f4 x:0 0 0 2024-04-22T15:49:45.214 INFO:tasks.workunit.client.0.smithi016.stdout:3/874: mknod d7/d17/d18/d32/d42/d63/deb/d21/c110 0 2024-04-22T15:49:45.215 INFO:tasks.workunit.client.1.smithi027.stdout:3/37: link c4 cc 0 2024-04-22T15:49:45.216 INFO:tasks.workunit.client.0.smithi016.stdout:0/915: mkdir dc/d13/d14/de5/d65/d40/d4a/d59/d9d/d125 0 2024-04-22T15:49:45.217 INFO:tasks.workunit.client.0.smithi016.stdout:9/885: dread d9/d57/d91/de4/d25/da6/f5b [0,4194304] 0 2024-04-22T15:49:45.220 INFO:tasks.workunit.client.0.smithi016.stdout:0/916: truncate dc/d13/d14/de5/d65/d40/d4a/d59/d78/fbc 4243029 0 2024-04-22T15:49:45.221 INFO:tasks.workunit.client.1.smithi027.stdout:5/18: rename d0/f4 to d0/f5 0 2024-04-22T15:49:45.222 INFO:tasks.workunit.client.0.smithi016.stdout:0/917: write dc/d13/d14/d26/d32/fc0 [1680210,122590] 0 2024-04-22T15:49:45.223 INFO:tasks.workunit.client.1.smithi027.stdout:2/33: link l3 l4 0 2024-04-22T15:49:45.223 INFO:tasks.workunit.client.1.smithi027.stdout:2/34: fsync f0 0 2024-04-22T15:49:45.227 INFO:tasks.workunit.client.1.smithi027.stdout:3/38: dread f5 [0,4194304] 0 2024-04-22T15:49:45.228 INFO:tasks.workunit.client.1.smithi027.stdout:7/16: truncate d0/f3 2970740 0 2024-04-22T15:49:45.229 INFO:tasks.workunit.client.1.smithi027.stdout:5/19: creat d0/f6 x:0 0 0 2024-04-22T15:49:45.230 INFO:tasks.workunit.client.0.smithi016.stdout:7/795: sync 2024-04-22T15:49:45.231 INFO:tasks.workunit.client.0.smithi016.stdout:9/886: rmdir d9/de/d23/ddb/def/dd2 0 2024-04-22T15:49:45.232 INFO:tasks.workunit.client.0.smithi016.stdout:3/875: dread d7/d17/d18/d32/d42/d63/deb/d21/fcd [0,4194304] 0 2024-04-22T15:49:45.232 INFO:tasks.workunit.client.1.smithi027.stdout:2/35: symlink l5 0 2024-04-22T15:49:45.232 INFO:tasks.workunit.client.1.smithi027.stdout:3/39: creat fd x:0 0 0 2024-04-22T15:49:45.233 INFO:tasks.workunit.client.0.smithi016.stdout:7/796: read d3/dc/d1b/d5a/d9/df/fd8 [693144,3743] 0 2024-04-22T15:49:45.233 INFO:tasks.workunit.client.0.smithi016.stdout:0/918: dwrite dc/d13/d14/de5/d65/d30/d96/fc7 [0,4194304] 0 2024-04-22T15:49:45.234 INFO:tasks.workunit.client.1.smithi027.stdout:5/20: write d0/f5 [440744,35110] 0 2024-04-22T15:49:45.234 INFO:tasks.workunit.client.1.smithi027.stdout:2/36: symlink l6 0 2024-04-22T15:49:45.236 INFO:tasks.workunit.client.1.smithi027.stdout:3/40: creat fe x:0 0 0 2024-04-22T15:49:45.237 INFO:tasks.workunit.client.1.smithi027.stdout:7/17: dwrite d0/f5 [0,4194304] 0 2024-04-22T15:49:45.237 INFO:tasks.workunit.client.0.smithi016.stdout:7/797: write d3/dc/d1b/d5a/d4a/f110 [764688,124136] 0 2024-04-22T15:49:45.237 INFO:tasks.workunit.client.1.smithi027.stdout:3/41: dread - fe zero size 2024-04-22T15:49:45.238 INFO:tasks.workunit.client.0.smithi016.stdout:9/887: unlink d9/d57/d91/de4/d25/l4d 0 2024-04-22T15:49:45.239 INFO:tasks.workunit.client.0.smithi016.stdout:9/888: fsync d9/d20/f40 0 2024-04-22T15:49:45.240 INFO:tasks.workunit.client.0.smithi016.stdout:9/889: chown d9/c19 230368933 1 2024-04-22T15:49:45.241 INFO:tasks.workunit.client.1.smithi027.stdout:2/37: dread f0 [0,4194304] 0 2024-04-22T15:49:45.241 INFO:tasks.workunit.client.0.smithi016.stdout:9/890: fdatasync d9/de/d23/f35 0 2024-04-22T15:49:45.245 INFO:tasks.workunit.client.0.smithi016.stdout:3/876: rename d7/c45 to d7/d17/d18/d32/d42/d63/deb/d21/d55/ddb/db7/c111 0 2024-04-22T15:49:45.247 INFO:tasks.workunit.client.1.smithi027.stdout:7/18: rename d0/f5 to d0/f7 0 2024-04-22T15:49:45.250 INFO:tasks.workunit.client.0.smithi016.stdout:0/919: dread dc/f68 [0,4194304] 0 2024-04-22T15:49:45.250 INFO:tasks.workunit.client.1.smithi027.stdout:5/21: link d0/f3 d0/f7 0 2024-04-22T15:49:45.252 INFO:tasks.workunit.client.1.smithi027.stdout:5/22: readlink d0/l2 0 2024-04-22T15:49:45.254 INFO:tasks.workunit.client.0.smithi016.stdout:9/891: symlink d9/d57/da7/l11c 0 2024-04-22T15:49:45.255 INFO:tasks.workunit.client.1.smithi027.stdout:7/19: dwrite d0/f6 [0,4194304] 0 2024-04-22T15:49:45.256 INFO:tasks.workunit.client.1.smithi027.stdout:7/20: truncate d0/f7 4829339 0 2024-04-22T15:49:45.263 INFO:tasks.workunit.client.1.smithi027.stdout:2/38: link f0 f7 0 2024-04-22T15:49:45.264 INFO:tasks.workunit.client.0.smithi016.stdout:0/920: symlink dc/d13/d14/de5/d65/d40/d4a/d59/d98/l126 0 2024-04-22T15:49:45.269 INFO:tasks.workunit.client.1.smithi027.stdout:5/23: creat d0/f8 x:0 0 0 2024-04-22T15:49:45.271 INFO:tasks.workunit.client.1.smithi027.stdout:7/21: write d0/f7 [1076131,122248] 0 2024-04-22T15:49:45.272 INFO:tasks.workunit.client.0.smithi016.stdout:7/798: creat d3/dc/d1b/d31/f11d x:0 0 0 2024-04-22T15:49:45.273 INFO:tasks.workunit.client.1.smithi027.stdout:2/39: mknod c8 0 2024-04-22T15:49:45.273 INFO:tasks.workunit.client.1.smithi027.stdout:2/40: readlink l6 0 2024-04-22T15:49:45.277 INFO:tasks.workunit.client.0.smithi016.stdout:9/892: creat d9/de/d26/d5a/f11d x:0 0 0 2024-04-22T15:49:45.278 INFO:tasks.workunit.client.1.smithi027.stdout:3/42: getdents . 0 2024-04-22T15:49:45.279 INFO:tasks.workunit.client.1.smithi027.stdout:3/43: read - fd zero size 2024-04-22T15:49:45.280 INFO:tasks.workunit.client.1.smithi027.stdout:5/24: mkdir d0/d9 0 2024-04-22T15:49:45.282 INFO:tasks.workunit.client.0.smithi016.stdout:0/921: unlink dc/d13/d14/de5/d65/f42 0 2024-04-22T15:49:45.283 INFO:tasks.workunit.client.1.smithi027.stdout:7/22: mkdir d0/d8 0 2024-04-22T15:49:45.285 INFO:tasks.workunit.client.1.smithi027.stdout:2/41: unlink l6 0 2024-04-22T15:49:45.288 INFO:tasks.workunit.client.1.smithi027.stdout:3/44: creat ff x:0 0 0 2024-04-22T15:49:45.288 INFO:tasks.workunit.client.0.smithi016.stdout:7/799: mknod d3/dc/d1b/d5a/d59/d64/dc9/c11e 0 2024-04-22T15:49:45.289 INFO:tasks.workunit.client.0.smithi016.stdout:7/800: dread - d3/dc/d33/d84/de3/ff8 zero size 2024-04-22T15:49:45.290 INFO:tasks.workunit.client.1.smithi027.stdout:5/25: mknod d0/ca 0 2024-04-22T15:49:45.290 INFO:tasks.workunit.client.0.smithi016.stdout:0/922: creat dc/d13/d14/de5/def/f127 x:0 0 0 2024-04-22T15:49:45.292 INFO:tasks.workunit.client.0.smithi016.stdout:3/877: link d7/c25 d7/d17/c112 0 2024-04-22T15:49:45.293 INFO:tasks.workunit.client.1.smithi027.stdout:2/42: readlink l4 0 2024-04-22T15:49:45.293 INFO:tasks.workunit.client.1.smithi027.stdout:2/43: chown f0 205368144 1 2024-04-22T15:49:45.294 INFO:tasks.workunit.client.0.smithi016.stdout:7/801: truncate d3/f28 2190208 0 2024-04-22T15:49:45.295 INFO:tasks.workunit.client.0.smithi016.stdout:9/893: creat d9/de/f11e x:0 0 0 2024-04-22T15:49:45.296 INFO:tasks.workunit.client.1.smithi027.stdout:3/45: dwrite ff [0,4194304] 0 2024-04-22T15:49:45.297 INFO:tasks.workunit.client.0.smithi016.stdout:0/923: unlink dc/d13/d14/de5/d65/d1f/dd4/df4/cfb 0 2024-04-22T15:49:45.299 INFO:tasks.workunit.client.1.smithi027.stdout:5/26: symlink d0/d9/lb 0 2024-04-22T15:49:45.299 INFO:tasks.workunit.client.0.smithi016.stdout:3/878: chown d7/d17/d18/d32/d42/d63/deb/d21/d55/ddb/db7/c111 977317 1 2024-04-22T15:49:45.301 INFO:tasks.workunit.client.1.smithi027.stdout:2/44: mknod c9 0 2024-04-22T15:49:45.301 INFO:tasks.workunit.client.1.smithi027.stdout:3/46: dread ff [0,4194304] 0 2024-04-22T15:49:45.302 INFO:tasks.workunit.client.1.smithi027.stdout:7/23: rmdir d0/d8 0 2024-04-22T15:49:45.305 INFO:tasks.workunit.client.1.smithi027.stdout:7/24: readlink d0/l4 0 2024-04-22T15:49:45.306 INFO:tasks.workunit.client.0.smithi016.stdout:9/894: dwrite d9/d57/d91/de4/f2f [0,4194304] 0 2024-04-22T15:49:45.306 INFO:tasks.workunit.client.0.smithi016.stdout:3/879: read d7/d17/d18/d32/d8a/fa8 [4137691,86897] 0 2024-04-22T15:49:45.306 INFO:tasks.workunit.client.0.smithi016.stdout:0/924: rmdir dc/d13/d14/de5/d65/d1f 39 2024-04-22T15:49:45.306 INFO:tasks.workunit.client.0.smithi016.stdout:0/925: chown dc/d13/d14/de5/def 544328738 1 2024-04-22T15:49:45.308 INFO:tasks.workunit.client.0.smithi016.stdout:7/802: mkdir d3/d11f 0 2024-04-22T15:49:45.309 INFO:tasks.workunit.client.1.smithi027.stdout:5/27: symlink d0/lc 0 2024-04-22T15:49:45.313 INFO:tasks.workunit.client.1.smithi027.stdout:2/45: symlink la 0 2024-04-22T15:49:45.313 INFO:tasks.workunit.client.0.smithi016.stdout:0/926: dread dc/d13/d14/de5/d65/d30/d96/fc7 [0,4194304] 0 2024-04-22T15:49:45.318 INFO:tasks.workunit.client.1.smithi027.stdout:3/47: creat f10 x:0 0 0 2024-04-22T15:49:45.319 INFO:tasks.workunit.client.1.smithi027.stdout:3/48: write fd [60771,81485] 0 2024-04-22T15:49:45.321 INFO:tasks.workunit.client.0.smithi016.stdout:3/880: fdatasync d7/d17/d18/d32/d42/d63/d86/ffe 0 2024-04-22T15:49:45.323 INFO:tasks.workunit.client.0.smithi016.stdout:3/881: fdatasync d7/d17/d18/d32/d42/d63/d71/fe2 0 2024-04-22T15:49:45.325 INFO:tasks.workunit.client.1.smithi027.stdout:7/25: dread d0/f3 [0,4194304] 0 2024-04-22T15:49:45.328 INFO:tasks.workunit.client.1.smithi027.stdout:5/28: fsync d0/f7 0 2024-04-22T15:49:45.331 INFO:tasks.workunit.client.1.smithi027.stdout:2/46: mknod cb 0 2024-04-22T15:49:45.332 INFO:tasks.workunit.client.0.smithi016.stdout:0/927: mkdir dc/d13/d14/de5/d65/d57/dcb/d128 0 2024-04-22T15:49:45.334 INFO:tasks.workunit.client.1.smithi027.stdout:3/49: creat f11 x:0 0 0 2024-04-22T15:49:45.335 INFO:tasks.workunit.client.0.smithi016.stdout:9/895: creat d9/d20/dac/dae/f11f x:0 0 0 2024-04-22T15:49:45.336 INFO:tasks.workunit.client.1.smithi027.stdout:5/29: dwrite d0/f6 [0,4194304] 0 2024-04-22T15:49:45.337 INFO:tasks.workunit.client.1.smithi027.stdout:7/26: unlink d0/f6 0 2024-04-22T15:49:45.337 INFO:tasks.workunit.client.1.smithi027.stdout:3/50: dread fd [0,4194304] 0 2024-04-22T15:49:45.346 INFO:tasks.workunit.client.1.smithi027.stdout:2/47: fsync f7 0 2024-04-22T15:49:45.348 INFO:tasks.workunit.client.0.smithi016.stdout:3/882: mknod d7/d17/d18/d32/d42/d63/deb/d21/d55/ddb/db7/c113 0 2024-04-22T15:49:45.352 INFO:tasks.workunit.client.1.smithi027.stdout:5/30: symlink d0/d9/ld 0 2024-04-22T15:49:45.353 INFO:tasks.workunit.client.0.smithi016.stdout:9/896: mknod d9/d57/d91/de4/d25/d36/c120 0 2024-04-22T15:49:45.353 INFO:tasks.workunit.client.1.smithi027.stdout:7/27: creat d0/f9 x:0 0 0 2024-04-22T15:49:45.354 INFO:tasks.workunit.client.1.smithi027.stdout:7/28: chown d0/f2 1543887 1 2024-04-22T15:49:45.354 INFO:tasks.workunit.client.0.smithi016.stdout:7/803: creat d3/de6/f120 x:0 0 0 2024-04-22T15:49:45.355 INFO:tasks.workunit.client.1.smithi027.stdout:3/51: creat f12 x:0 0 0 2024-04-22T15:49:45.360 INFO:tasks.workunit.client.1.smithi027.stdout:2/48: write f0 [1105767,105523] 0 2024-04-22T15:49:45.363 INFO:tasks.workunit.client.0.smithi016.stdout:7/804: creat d3/dc/d1b/d5a/d59/d64/dc9/f121 x:0 0 0 2024-04-22T15:49:45.363 INFO:tasks.workunit.client.0.smithi016.stdout:7/805: readlink d3/d11a/def/l100 0 2024-04-22T15:49:45.366 INFO:tasks.workunit.client.1.smithi027.stdout:7/29: unlink d0/f7 0 2024-04-22T15:49:45.368 INFO:tasks.workunit.client.0.smithi016.stdout:7/806: dread d3/dc/d1b/d5a/f5b [0,4194304] 0 2024-04-22T15:49:45.369 INFO:tasks.workunit.client.1.smithi027.stdout:3/52: rename fd to f13 0 2024-04-22T15:49:45.370 INFO:tasks.workunit.client.0.smithi016.stdout:3/883: creat d7/d17/d18/d32/d3c/d51/f114 x:0 0 0 2024-04-22T15:49:45.371 INFO:tasks.workunit.client.0.smithi016.stdout:3/884: chown d7/d17/d18/d32/d3c/d47/f10d 4 1 2024-04-22T15:49:45.371 INFO:tasks.workunit.client.1.smithi027.stdout:2/49: rename c8 to cc 0 2024-04-22T15:49:45.372 INFO:tasks.workunit.client.0.smithi016.stdout:9/897: creat d9/d20/dac/d106/f121 x:0 0 0 2024-04-22T15:49:45.376 INFO:tasks.workunit.client.1.smithi027.stdout:7/30: creat d0/fa x:0 0 0 2024-04-22T15:49:45.376 INFO:tasks.workunit.client.0.smithi016.stdout:1/948: sync 2024-04-22T15:49:45.376 INFO:tasks.workunit.client.1.smithi027.stdout:3/53: mknod c14 0 2024-04-22T15:49:45.377 INFO:tasks.workunit.client.1.smithi027.stdout:3/54: chown la 91887 1 2024-04-22T15:49:45.378 INFO:tasks.workunit.client.0.smithi016.stdout:9/898: dwrite d9/d20/dac/dae/fff [0,4194304] 0 2024-04-22T15:49:45.379 INFO:tasks.workunit.client.0.smithi016.stdout:3/885: creat d7/d17/d18/d32/d3c/d47/dc3/f115 x:0 0 0 2024-04-22T15:49:45.382 INFO:tasks.workunit.client.1.smithi027.stdout:7/31: fdatasync d0/f3 0 2024-04-22T15:49:45.385 INFO:tasks.workunit.client.1.smithi027.stdout:3/55: rename f12 to f15 0 2024-04-22T15:49:45.388 INFO:tasks.workunit.client.0.smithi016.stdout:0/928: write dc/d13/d14/d26/f7b [2087292,59795] 0 2024-04-22T15:49:45.391 INFO:tasks.workunit.client.1.smithi027.stdout:2/50: unlink f0 0 2024-04-22T15:49:45.392 INFO:tasks.workunit.client.0.smithi016.stdout:3/886: mkdir d7/d17/d18/d32/d8a/d116 0 2024-04-22T15:49:45.392 INFO:tasks.workunit.client.1.smithi027.stdout:5/31: getdents d0/d9 0 2024-04-22T15:49:45.393 INFO:tasks.workunit.client.0.smithi016.stdout:3/887: stat d7/d17/d18/d32/d42/d63/deb/d21/d55/ddb/db7/cf0 0 2024-04-22T15:49:45.394 INFO:tasks.workunit.client.0.smithi016.stdout:1/949: mkdir d3/d4/d147 0 2024-04-22T15:49:45.394 INFO:tasks.workunit.client.1.smithi027.stdout:4/32: sync 2024-04-22T15:49:45.394 INFO:tasks.workunit.client.1.smithi027.stdout:1/35: sync 2024-04-22T15:49:45.394 INFO:tasks.workunit.client.1.smithi027.stdout:0/88: sync 2024-04-22T15:49:45.394 INFO:tasks.workunit.client.1.smithi027.stdout:9/58: sync 2024-04-22T15:49:45.394 INFO:tasks.workunit.client.1.smithi027.stdout:6/22: sync 2024-04-22T15:49:45.395 INFO:tasks.workunit.client.1.smithi027.stdout:5/32: dread d0/f3 [0,4194304] 0 2024-04-22T15:49:45.396 INFO:tasks.workunit.client.0.smithi016.stdout:0/929: creat dc/d13/d14/de5/dd8/f129 x:0 0 0 2024-04-22T15:49:45.397 INFO:tasks.workunit.client.0.smithi016.stdout:7/807: write d3/f28 [796489,109613] 0 2024-04-22T15:49:45.398 INFO:tasks.workunit.client.1.smithi027.stdout:7/32: write d0/f2 [1501220,63148] 0 2024-04-22T15:49:45.399 INFO:tasks.workunit.client.0.smithi016.stdout:1/950: creat d3/d4/d3a/d9e/f148 x:0 0 0 2024-04-22T15:49:45.399 INFO:tasks.workunit.client.1.smithi027.stdout:7/33: stat d0/fa 0 2024-04-22T15:49:45.400 INFO:tasks.workunit.client.1.smithi027.stdout:3/56: creat f16 x:0 0 0 2024-04-22T15:49:45.401 INFO:tasks.workunit.client.0.smithi016.stdout:0/930: stat dc/d13/c7c 0 2024-04-22T15:49:45.401 INFO:tasks.workunit.client.1.smithi027.stdout:7/34: chown d0/fa 6 1 2024-04-22T15:49:45.402 INFO:tasks.workunit.client.1.smithi027.stdout:1/36: dwrite f0 [0,4194304] 0 2024-04-22T15:49:45.402 INFO:tasks.workunit.client.1.smithi027.stdout:3/57: write f11 [196155,94555] 0 2024-04-22T15:49:45.402 INFO:tasks.workunit.client.0.smithi016.stdout:7/808: write d3/dc/d1b/d5a/d9/df/d94/d4c/fdd [269254,81662] 0 2024-04-22T15:49:45.403 INFO:tasks.workunit.client.1.smithi027.stdout:7/35: fsync d0/f2 0 2024-04-22T15:49:45.403 INFO:tasks.workunit.client.1.smithi027.stdout:1/37: stat d2 0 2024-04-22T15:49:45.405 INFO:tasks.workunit.client.1.smithi027.stdout:0/89: readlink d0/lf 0 2024-04-22T15:49:45.407 INFO:tasks.workunit.client.1.smithi027.stdout:6/23: symlink d5/l7 0 2024-04-22T15:49:45.410 INFO:tasks.workunit.client.0.smithi016.stdout:1/951: unlink d3/d4/f1e 0 2024-04-22T15:49:45.411 INFO:tasks.workunit.client.1.smithi027.stdout:1/38: dwrite f0 [0,4194304] 0 2024-04-22T15:49:45.411 INFO:tasks.workunit.client.1.smithi027.stdout:4/33: unlink f4 0 2024-04-22T15:49:45.412 INFO:tasks.workunit.client.1.smithi027.stdout:1/39: chown f0 100707361 1 2024-04-22T15:49:45.413 INFO:tasks.workunit.client.1.smithi027.stdout:1/40: chown d2 1242888 1 2024-04-22T15:49:45.416 INFO:tasks.workunit.client.0.smithi016.stdout:7/809: dread d3/dc/d1b/feb [0,4194304] 0 2024-04-22T15:49:45.418 INFO:tasks.workunit.client.0.smithi016.stdout:1/952: dwrite d3/d4/d3a/d91/db3/dd2/ff3 [0,4194304] 0 2024-04-22T15:49:45.420 INFO:tasks.workunit.client.1.smithi027.stdout:5/33: unlink d0/d9/ld 0 2024-04-22T15:49:45.421 INFO:tasks.workunit.client.1.smithi027.stdout:4/34: dwrite d5/fa [0,4194304] 0 2024-04-22T15:49:45.421 INFO:tasks.workunit.client.1.smithi027.stdout:3/58: symlink l17 0 2024-04-22T15:49:45.424 INFO:tasks.workunit.client.1.smithi027.stdout:4/35: write d5/fa [137307,117685] 0 2024-04-22T15:49:45.424 INFO:tasks.workunit.client.0.smithi016.stdout:7/810: rename d3/dc/d1b/d5a/d9/f13 to d3/dc/d1b/d5a/d4a/d6e/d115/f122 0 2024-04-22T15:49:45.427 INFO:tasks.workunit.client.1.smithi027.stdout:8/11: sync 2024-04-22T15:49:45.429 INFO:tasks.workunit.client.1.smithi027.stdout:9/59: dwrite d0/fb [4194304,4194304] 0 2024-04-22T15:49:45.429 INFO:tasks.workunit.client.0.smithi016.stdout:1/953: mknod d3/d4/d19/d20/c149 0 2024-04-22T15:49:45.437 INFO:tasks.workunit.client.1.smithi027.stdout:4/36: dread f0 [0,4194304] 0 2024-04-22T15:49:45.438 INFO:tasks.workunit.client.0.smithi016.stdout:7/811: unlink d3/fa 0 2024-04-22T15:49:45.440 INFO:tasks.workunit.client.0.smithi016.stdout:7/812: chown d3/dc/d1b/d5a/d9/df/d94/d4c/d7a 3743518 1 2024-04-22T15:49:45.444 INFO:tasks.workunit.client.0.smithi016.stdout:7/813: dread d3/dc/d1b/feb [0,4194304] 0 2024-04-22T15:49:45.445 INFO:tasks.workunit.client.1.smithi027.stdout:6/24: dwrite f4 [0,4194304] 0 2024-04-22T15:49:45.452 INFO:tasks.workunit.client.1.smithi027.stdout:3/59: mknod c18 0 2024-04-22T15:49:45.455 INFO:tasks.workunit.client.0.smithi016.stdout:7/814: creat d3/d10b/f123 x:0 0 0 2024-04-22T15:49:45.456 INFO:tasks.workunit.client.1.smithi027.stdout:6/25: dwrite f0 [0,4194304] 0 2024-04-22T15:49:45.458 INFO:tasks.workunit.client.1.smithi027.stdout:8/12: mkdir d1/d3 0 2024-04-22T15:49:45.460 INFO:tasks.workunit.client.0.smithi016.stdout:1/954: getdents d3/d4/d15/d38 0 2024-04-22T15:49:45.462 INFO:tasks.workunit.client.0.smithi016.stdout:7/815: creat d3/dc/d1b/d5a/d9/df/d94/f124 x:0 0 0 2024-04-22T15:49:45.464 INFO:tasks.workunit.client.1.smithi027.stdout:5/34: dwrite d0/f3 [0,4194304] 0 2024-04-22T15:49:45.465 INFO:tasks.workunit.client.0.smithi016.stdout:1/955: unlink d3/d4/d19/d20/d95/ffc 0 2024-04-22T15:49:45.471 INFO:tasks.workunit.client.0.smithi016.stdout:7/816: rename d3/dc/d1b/d31/d5f/c107 to d3/d11a/def/c125 0 2024-04-22T15:49:45.472 INFO:tasks.workunit.client.1.smithi027.stdout:0/90: truncate d0/f1 1506257 0 2024-04-22T15:49:45.473 INFO:tasks.workunit.client.1.smithi027.stdout:4/37: mknod d5/cd 0 2024-04-22T15:49:45.474 INFO:tasks.workunit.client.1.smithi027.stdout:4/38: chown d5/f6 15616 1 2024-04-22T15:49:45.474 INFO:tasks.workunit.client.1.smithi027.stdout:5/35: dwrite d0/f3 [0,4194304] 0 2024-04-22T15:49:45.475 INFO:tasks.workunit.client.1.smithi027.stdout:5/36: chown d0/f3 482 1 2024-04-22T15:49:45.479 INFO:tasks.workunit.client.1.smithi027.stdout:1/41: link d2/f7 d2/f8 0 2024-04-22T15:49:45.482 INFO:tasks.workunit.client.0.smithi016.stdout:7/817: mknod d3/de6/c126 0 2024-04-22T15:49:45.483 INFO:tasks.workunit.client.0.smithi016.stdout:7/818: dread d3/dc/d1b/d5a/fa3 [0,4194304] 0 2024-04-22T15:49:45.486 INFO:tasks.workunit.client.1.smithi027.stdout:6/26: mkdir d5/d8 0 2024-04-22T15:49:45.492 INFO:tasks.workunit.client.0.smithi016.stdout:7/819: rename d3/dc/d1b/d5a/d59/d64/dc9/dd2 to d3/d11a/d8c/d127 0 2024-04-22T15:49:45.493 INFO:tasks.workunit.client.0.smithi016.stdout:1/956: truncate d3/d4/d3a/d91/db3/f11d 5866924 0 2024-04-22T15:49:45.493 INFO:tasks.workunit.client.0.smithi016.stdout:7/820: chown d3/de6/fbe 16026318 1 2024-04-22T15:49:45.498 INFO:tasks.workunit.client.0.smithi016.stdout:7/821: mkdir d3/dc/d1b/d5a/d4a/d6e/d115/d128 0 2024-04-22T15:49:45.499 INFO:tasks.workunit.client.0.smithi016.stdout:1/957: dwrite d3/d28/f8b [0,4194304] 0 2024-04-22T15:49:45.499 INFO:tasks.workunit.client.1.smithi027.stdout:0/91: creat d0/d1a/f1e x:0 0 0 2024-04-22T15:49:45.500 INFO:tasks.workunit.client.1.smithi027.stdout:9/60: truncate d0/d8/f11 169244 0 2024-04-22T15:49:45.500 INFO:tasks.workunit.client.1.smithi027.stdout:9/61: chown d0/l6 659666 1 2024-04-22T15:49:45.501 INFO:tasks.workunit.client.1.smithi027.stdout:4/39: fdatasync f0 0 2024-04-22T15:49:45.502 INFO:tasks.workunit.client.1.smithi027.stdout:4/40: write d5/fa [2871294,16287] 0 2024-04-22T15:49:45.503 INFO:tasks.workunit.client.1.smithi027.stdout:5/37: mkdir d0/d9/de 0 2024-04-22T15:49:45.506 INFO:tasks.workunit.client.0.smithi016.stdout:5/838: sync 2024-04-22T15:49:45.506 INFO:tasks.workunit.client.0.smithi016.stdout:6/893: sync 2024-04-22T15:49:45.506 INFO:tasks.workunit.client.0.smithi016.stdout:8/785: sync 2024-04-22T15:49:45.508 INFO:tasks.workunit.client.0.smithi016.stdout:8/786: fsync d2/d82/d52/df0/d66/d1c/d65/f9c 0 2024-04-22T15:49:45.512 INFO:tasks.workunit.client.1.smithi027.stdout:7/36: getdents d0 0 2024-04-22T15:49:45.513 INFO:tasks.workunit.client.1.smithi027.stdout:8/13: dwrite f0 [0,4194304] 0 2024-04-22T15:49:45.514 INFO:tasks.workunit.client.0.smithi016.stdout:5/839: dread - df/d15/d1a/f99 zero size 2024-04-22T15:49:45.515 INFO:tasks.workunit.client.1.smithi027.stdout:7/37: chown d0/f9 40030 1 2024-04-22T15:49:45.515 INFO:tasks.workunit.client.0.smithi016.stdout:6/894: dwrite d3/da/dd/d1b/f62 [0,4194304] 0 2024-04-22T15:49:45.518 INFO:tasks.workunit.client.1.smithi027.stdout:2/51: truncate f7 1343036 0 2024-04-22T15:49:45.519 INFO:tasks.workunit.client.1.smithi027.stdout:8/14: dread f0 [0,4194304] 0 2024-04-22T15:49:45.526 INFO:tasks.workunit.client.0.smithi016.stdout:1/958: link d3/d4/d15/d38/d63/f75 d3/d28/dbd/f14a 0 2024-04-22T15:49:45.526 INFO:tasks.workunit.client.1.smithi027.stdout:1/42: write d2/f7 [741169,97117] 0 2024-04-22T15:49:45.531 INFO:tasks.workunit.client.1.smithi027.stdout:0/92: rmdir d0/d1a 39 2024-04-22T15:49:45.539 INFO:tasks.workunit.client.1.smithi027.stdout:4/41: mkdir d5/de 0 2024-04-22T15:49:45.542 INFO:tasks.workunit.client.0.smithi016.stdout:5/840: write df/d15/d1a/d50/d77/f62 [2464467,99465] 0 2024-04-22T15:49:45.542 INFO:tasks.workunit.client.0.smithi016.stdout:6/895: mknod d3/da/db/d14/d10d/c127 0 2024-04-22T15:49:45.543 INFO:tasks.workunit.client.0.smithi016.stdout:6/896: chown d3 13 1 2024-04-22T15:49:45.543 INFO:tasks.workunit.client.1.smithi027.stdout:3/60: link lb l19 0 2024-04-22T15:49:45.544 INFO:tasks.workunit.client.0.smithi016.stdout:7/822: dread d3/dc/d33/f4d [0,4194304] 0 2024-04-22T15:49:45.545 INFO:tasks.workunit.client.0.smithi016.stdout:1/959: write d3/d4/d3a/d91/db3/fdb [253536,102040] 0 2024-04-22T15:49:45.548 INFO:tasks.workunit.client.1.smithi027.stdout:8/15: creat d1/f4 x:0 0 0 2024-04-22T15:49:45.549 INFO:tasks.workunit.client.1.smithi027.stdout:8/16: truncate d1/f4 247816 0 2024-04-22T15:49:45.549 INFO:tasks.workunit.client.1.smithi027.stdout:8/17: rename d1 to d1/d3/d5 22 2024-04-22T15:49:45.550 INFO:tasks.workunit.client.0.smithi016.stdout:8/787: dread d2/d82/d52/df0/d66/fa7 [0,4194304] 0 2024-04-22T15:49:45.550 INFO:tasks.workunit.client.0.smithi016.stdout:1/960: symlink d3/d4/d3a/d6b/d77/l14b 0 2024-04-22T15:49:45.555 INFO:tasks.workunit.client.0.smithi016.stdout:6/897: creat d3/da/dd/d67/d6c/db9/dfa/f128 x:0 0 0 2024-04-22T15:49:45.557 INFO:tasks.workunit.client.0.smithi016.stdout:2/834: sync 2024-04-22T15:49:45.557 INFO:tasks.workunit.client.0.smithi016.stdout:4/834: sync 2024-04-22T15:49:45.559 INFO:tasks.workunit.client.1.smithi027.stdout:4/42: creat d5/d7/ff x:0 0 0 2024-04-22T15:49:45.559 INFO:tasks.workunit.client.1.smithi027.stdout:6/27: truncate f4 3361722 0 2024-04-22T15:49:45.561 INFO:tasks.workunit.client.0.smithi016.stdout:5/841: rename df/d90/cfc to df/c104 0 2024-04-22T15:49:45.561 INFO:tasks.workunit.client.1.smithi027.stdout:1/43: write d2/f3 [3738467,42818] 0 2024-04-22T15:49:45.562 INFO:tasks.workunit.client.1.smithi027.stdout:3/61: rename c18 to c1a 0 2024-04-22T15:49:45.563 INFO:tasks.workunit.client.1.smithi027.stdout:3/62: chown l2 79995 1 2024-04-22T15:49:45.565 INFO:tasks.workunit.client.1.smithi027.stdout:2/52: mkdir dd 0 2024-04-22T15:49:45.565 INFO:tasks.workunit.client.1.smithi027.stdout:2/53: stat dd 0 2024-04-22T15:49:45.566 INFO:tasks.workunit.client.0.smithi016.stdout:8/788: fsync d2/d82/d56/f64 0 2024-04-22T15:49:45.569 INFO:tasks.workunit.client.1.smithi027.stdout:0/93: creat d0/d1a/f1f x:0 0 0 2024-04-22T15:49:45.569 INFO:tasks.workunit.client.1.smithi027.stdout:0/94: fsync d0/f10 0 2024-04-22T15:49:45.569 INFO:tasks.workunit.client.0.smithi016.stdout:7/823: rename d3/d11a/ced to d3/dc/d1b/d5a/d9/df/d94/d4c/d7a/c129 0 2024-04-22T15:49:45.570 INFO:tasks.workunit.client.0.smithi016.stdout:6/898: rmdir d3/da/dd/d67 39 2024-04-22T15:49:45.580 INFO:tasks.workunit.client.0.smithi016.stdout:1/961: truncate d3/d4/d15/d38/d63/d67/fd0 748860 0 2024-04-22T15:49:45.584 INFO:tasks.workunit.client.0.smithi016.stdout:4/835: write d7/d24/f76 [1054688,124708] 0 2024-04-22T15:49:45.589 INFO:tasks.workunit.client.0.smithi016.stdout:6/899: write d3/da/db/d14/d16/f3d [5224221,4653] 0 2024-04-22T15:49:45.595 INFO:tasks.workunit.client.0.smithi016.stdout:8/789: link d2/d82/l51 d2/d48/dad/d8f/da1/lf4 0 2024-04-22T15:49:45.596 INFO:tasks.workunit.client.0.smithi016.stdout:1/962: mkdir d3/d4/d19/d20/d2b/dd6/d11e/d14c 0 2024-04-22T15:49:45.597 INFO:tasks.workunit.client.0.smithi016.stdout:7/824: truncate d3/dc/d1b/d5a/d9/df/d94/ddf/fe2 3489861 0 2024-04-22T15:49:45.600 INFO:tasks.workunit.client.0.smithi016.stdout:8/790: rmdir d2/d48/dad/d8f 39 2024-04-22T15:49:45.600 INFO:tasks.workunit.client.0.smithi016.stdout:8/791: chown d2/d82 1 1 2024-04-22T15:49:45.603 INFO:tasks.workunit.client.0.smithi016.stdout:7/825: creat d3/dc/d9f/f12a x:0 0 0 2024-04-22T15:49:45.603 INFO:tasks.workunit.client.0.smithi016.stdout:7/826: read - d3/dc/d9f/f12a zero size 2024-04-22T15:49:45.605 INFO:tasks.workunit.client.0.smithi016.stdout:1/963: symlink d3/d2e/d117/l14d 0 2024-04-22T15:49:45.612 INFO:tasks.workunit.client.0.smithi016.stdout:1/964: dwrite d3/d4/d3a/d6b/d77/f138 [0,4194304] 0 2024-04-22T15:49:45.665 INFO:tasks.workunit.client.0.smithi016.stdout:3/888: sync 2024-04-22T15:49:45.666 INFO:tasks.workunit.client.0.smithi016.stdout:9/899: sync 2024-04-22T15:49:45.666 INFO:tasks.workunit.client.0.smithi016.stdout:0/931: sync 2024-04-22T15:49:45.695 INFO:tasks.workunit.client.0.smithi016.stdout:8/792: creat d2/d82/d52/df0/d66/d7e/ff5 x:0 0 0 2024-04-22T15:49:45.699 INFO:tasks.workunit.client.0.smithi016.stdout:4/836: dwrite d7/d4e/d2f/d6f/f97 [0,4194304] 0 2024-04-22T15:49:45.701 INFO:tasks.workunit.client.0.smithi016.stdout:4/837: write d7/d22/f73 [5150443,121601] 0 2024-04-22T15:49:45.702 INFO:tasks.workunit.client.0.smithi016.stdout:9/900: mkdir d9/d5d/db8/d122 0 2024-04-22T15:49:45.702 INFO:tasks.workunit.client.0.smithi016.stdout:9/901: stat d9/fc 0 2024-04-22T15:49:45.705 INFO:tasks.workunit.client.0.smithi016.stdout:3/889: mkdir d7/d17/d18/d32/d3c/d51/d117 0 2024-04-22T15:49:45.710 INFO:tasks.workunit.client.0.smithi016.stdout:3/890: dread d7/d17/d18/d32/d42/d63/deb/d21/fcd [0,4194304] 0 2024-04-22T15:49:45.713 INFO:tasks.workunit.client.0.smithi016.stdout:8/793: readlink d2/d82/d52/df0/d66/d7e/l7f 0 2024-04-22T15:49:45.721 INFO:tasks.workunit.client.0.smithi016.stdout:8/794: dread d2/d10/d43/da9/fac [0,4194304] 0 2024-04-22T15:49:45.733 INFO:tasks.workunit.client.0.smithi016.stdout:8/795: creat d2/d82/d3e/d8e/ff6 x:0 0 0 2024-04-22T15:49:45.735 INFO:tasks.workunit.client.0.smithi016.stdout:8/796: write d2/d82/d52/df0/d66/d7e/dc0/fcf [666123,91410] 0 2024-04-22T15:49:45.737 INFO:tasks.workunit.client.0.smithi016.stdout:8/797: chown d2/d82/d52/df0/d66/d1c/f3d 211556090 1 2024-04-22T15:49:45.739 INFO:tasks.workunit.client.0.smithi016.stdout:9/902: link d9/de/d26/d5a/f11d d9/d57/d91/de4/d25/d36/d116/f123 0 2024-04-22T15:49:45.741 INFO:tasks.workunit.client.0.smithi016.stdout:9/903: dread d9/d20/f56 [0,4194304] 0 2024-04-22T15:49:45.742 INFO:tasks.workunit.client.0.smithi016.stdout:3/891: unlink d7/d17/d18/d32/d42/d63/deb/d21/lc5 0 2024-04-22T15:49:45.746 INFO:tasks.workunit.client.0.smithi016.stdout:8/798: symlink d2/d10/d43/lf7 0 2024-04-22T15:49:45.748 INFO:tasks.workunit.client.0.smithi016.stdout:3/892: mkdir d7/d17/d18/d32/d3c/d51/d118 0 2024-04-22T15:49:45.750 INFO:tasks.workunit.client.0.smithi016.stdout:4/838: getdents d7/d21/d7c/d103 0 2024-04-22T15:49:45.752 INFO:tasks.workunit.client.0.smithi016.stdout:8/799: mkdir d2/d82/d3e/dd5/df8 0 2024-04-22T15:49:45.756 INFO:tasks.workunit.client.0.smithi016.stdout:3/893: dread d7/d17/d18/d32/f57 [0,4194304] 0 2024-04-22T15:49:45.758 INFO:tasks.workunit.client.0.smithi016.stdout:8/800: creat d2/d82/d3e/dd5/df8/ff9 x:0 0 0 2024-04-22T15:49:45.762 INFO:tasks.workunit.client.0.smithi016.stdout:8/801: mkdir d2/d77/dfa 0 2024-04-22T15:49:45.764 INFO:tasks.workunit.client.0.smithi016.stdout:7/827: rename d3/de6/f120 to d3/d11a/f12b 0 2024-04-22T15:49:45.764 INFO:tasks.workunit.client.0.smithi016.stdout:8/802: read d2/d82/d52/df0/d66/d1c/f8c [860167,106622] 0 2024-04-22T15:49:45.767 INFO:tasks.workunit.client.0.smithi016.stdout:0/932: rename dc/d13/d14/de5/d65/d40/d4a/d92/f121 to dc/d13/d70/de8/f12a 0 2024-04-22T15:49:45.771 INFO:tasks.workunit.client.0.smithi016.stdout:8/803: dread d2/d10/f41 [0,4194304] 0 2024-04-22T15:49:45.773 INFO:tasks.workunit.client.0.smithi016.stdout:3/894: dread d7/d17/d18/d27/fbe [0,4194304] 0 2024-04-22T15:49:45.780 INFO:tasks.workunit.client.0.smithi016.stdout:0/933: mknod dc/d13/d14/de5/c12b 0 2024-04-22T15:49:45.782 INFO:tasks.workunit.client.0.smithi016.stdout:3/895: dwrite d7/d17/d18/d32/d42/d63/d86/dd8/ffc [0,4194304] 0 2024-04-22T15:49:45.786 INFO:tasks.workunit.client.1.smithi027.stdout:5/38: sync 2024-04-22T15:49:45.794 INFO:tasks.workunit.client.1.smithi027.stdout:5/39: dwrite d0/f3 [4194304,4194304] 0 2024-04-22T15:49:45.799 INFO:tasks.workunit.client.1.smithi027.stdout:5/40: dread d0/f6 [0,4194304] 0 2024-04-22T15:49:45.799 INFO:tasks.workunit.client.1.smithi027.stdout:5/41: stat d0/f7 0 2024-04-22T15:49:45.800 INFO:tasks.workunit.client.0.smithi016.stdout:0/934: rmdir dc/d13/d14/de5/d65/d40/d4a 39 2024-04-22T15:49:45.801 INFO:tasks.workunit.client.0.smithi016.stdout:8/804: dread d2/f5a [0,4194304] 0 2024-04-22T15:49:45.808 INFO:tasks.workunit.client.0.smithi016.stdout:8/805: dread d2/d82/d52/df0/d66/d7e/dc0/fcf [0,4194304] 0 2024-04-22T15:49:45.823 INFO:tasks.workunit.client.0.smithi016.stdout:8/806: dread d2/d82/d52/df0/d66/d1c/f8c [0,4194304] 0 2024-04-22T15:49:45.825 INFO:tasks.workunit.client.0.smithi016.stdout:8/807: write d2/d82/d52/df0/d66/d7e/fc5 [2671474,129042] 0 2024-04-22T15:49:45.829 INFO:tasks.workunit.client.0.smithi016.stdout:7/828: creat d3/dc/d1b/d5a/d59/d64/f12c x:0 0 0 2024-04-22T15:49:45.830 INFO:tasks.workunit.client.0.smithi016.stdout:7/829: dread - d3/de6/ffc zero size 2024-04-22T15:49:45.833 INFO:tasks.workunit.client.0.smithi016.stdout:7/830: dread d3/dc/d1b/d5a/d9/df/f70 [0,4194304] 0 2024-04-22T15:49:45.861 INFO:tasks.workunit.client.0.smithi016.stdout:1/965: truncate d3/f4f 2637424 0 2024-04-22T15:49:45.867 INFO:tasks.workunit.client.0.smithi016.stdout:0/935: dwrite dc/d13/d14/de5/d65/d40/d4a/ff8 [0,4194304] 0 2024-04-22T15:49:45.868 INFO:tasks.workunit.client.1.smithi027.stdout:7/38: sync 2024-04-22T15:49:45.871 INFO:tasks.workunit.client.1.smithi027.stdout:7/39: dread d0/f3 [0,4194304] 0 2024-04-22T15:49:45.871 INFO:tasks.workunit.client.1.smithi027.stdout:7/40: chown d0 10409239 1 2024-04-22T15:49:45.874 INFO:tasks.workunit.client.0.smithi016.stdout:3/896: rmdir d7/d17/d18/d32/d3c/d51/d117 0 2024-04-22T15:49:45.876 INFO:tasks.workunit.client.0.smithi016.stdout:3/897: read d7/f12 [1253979,344] 0 2024-04-22T15:49:45.882 INFO:tasks.workunit.client.0.smithi016.stdout:0/936: creat dc/d13/d14/de5/d65/d57/dcb/de1/d112/f12c x:0 0 0 2024-04-22T15:49:45.889 INFO:tasks.workunit.client.0.smithi016.stdout:3/898: creat d7/d17/d18/d32/d42/d63/d86/dd4/d100/f119 x:0 0 0 2024-04-22T15:49:45.890 INFO:tasks.workunit.client.0.smithi016.stdout:7/831: link d3/dc/d1b/d31/le7 d3/de6/l12d 0 2024-04-22T15:49:45.890 INFO:tasks.workunit.client.0.smithi016.stdout:3/899: chown d7/d17/d18/d32/d42/d63/d86/dd4/ld2 8 1 2024-04-22T15:49:45.891 INFO:tasks.workunit.client.0.smithi016.stdout:7/832: stat d3/d11a/d8c/d127 0 2024-04-22T15:49:45.892 INFO:tasks.workunit.client.0.smithi016.stdout:0/937: symlink dc/d13/d6f/d8b/l12d 0 2024-04-22T15:49:45.892 INFO:tasks.workunit.client.0.smithi016.stdout:0/938: chown dc/d13/d22/c8f 16 1 2024-04-22T15:49:45.893 INFO:tasks.workunit.client.0.smithi016.stdout:3/900: write d7/d17/d18/d27/fbe [881459,3519] 0 2024-04-22T15:49:45.894 INFO:tasks.workunit.client.0.smithi016.stdout:0/939: stat dc/d13/d14/de5/d65/d40/dd5/d11a 0 2024-04-22T15:49:45.896 INFO:tasks.workunit.client.0.smithi016.stdout:3/901: fsync d7/d17/d18/d32/d3c/faa 0 2024-04-22T15:49:45.897 INFO:tasks.workunit.client.0.smithi016.stdout:3/902: stat d7/d17/d18/d32/d3c/fd7 0 2024-04-22T15:49:45.901 INFO:tasks.workunit.client.0.smithi016.stdout:0/940: mkdir dc/d13/d14/de5/d65/d40/dd5/d11a/d12e 0 2024-04-22T15:49:45.902 INFO:tasks.workunit.client.0.smithi016.stdout:3/903: dread d7/d17/d18/f70 [0,4194304] 0 2024-04-22T15:49:45.905 INFO:tasks.workunit.client.0.smithi016.stdout:0/941: mknod dc/d13/d14/de5/d65/d40/d4a/db2/c12f 0 2024-04-22T15:49:45.909 INFO:tasks.workunit.client.0.smithi016.stdout:0/942: write dc/d13/d14/de5/d65/d1f/f53 [4039247,127808] 0 2024-04-22T15:49:45.910 INFO:tasks.workunit.client.0.smithi016.stdout:0/943: chown dc/d13/d22/f2e 7 1 2024-04-22T15:49:45.910 INFO:tasks.workunit.client.1.smithi027.stdout:1/44: mknod d2/c9 0 2024-04-22T15:49:45.911 INFO:tasks.workunit.client.1.smithi027.stdout:1/45: truncate d2/f6 4769479 0 2024-04-22T15:49:45.911 INFO:tasks.workunit.client.1.smithi027.stdout:1/46: stat d2/f7 0 2024-04-22T15:49:45.913 INFO:tasks.workunit.client.1.smithi027.stdout:3/63: unlink fe 0 2024-04-22T15:49:45.913 INFO:tasks.workunit.client.1.smithi027.stdout:3/64: fsync f11 0 2024-04-22T15:49:45.916 INFO:tasks.workunit.client.0.smithi016.stdout:0/944: read dc/d13/d6f/d8b/f9a [83346,59956] 0 2024-04-22T15:49:45.917 INFO:tasks.workunit.client.1.smithi027.stdout:0/95: truncate d0/f14 915520 0 2024-04-22T15:49:45.917 INFO:tasks.workunit.client.1.smithi027.stdout:8/18: creat d1/d3/f6 x:0 0 0 2024-04-22T15:49:45.918 INFO:tasks.workunit.client.0.smithi016.stdout:5/842: sync 2024-04-22T15:49:45.918 INFO:tasks.workunit.client.0.smithi016.stdout:2/835: sync 2024-04-22T15:49:45.919 INFO:tasks.workunit.client.0.smithi016.stdout:5/843: chown df 285361 1 2024-04-22T15:49:45.919 INFO:tasks.workunit.client.1.smithi027.stdout:8/19: dread d1/f4 [0,4194304] 0 2024-04-22T15:49:45.919 INFO:tasks.workunit.client.0.smithi016.stdout:2/836: write d2/d51/fbd [1276751,107923] 0 2024-04-22T15:49:45.921 INFO:tasks.workunit.client.0.smithi016.stdout:0/945: creat dc/d13/d14/de5/f130 x:0 0 0 2024-04-22T15:49:45.921 INFO:tasks.workunit.client.0.smithi016.stdout:2/837: fdatasync d2/d1a/d4c/d9f/dae/fb3 0 2024-04-22T15:49:45.923 INFO:tasks.workunit.client.1.smithi027.stdout:5/42: rename d0/d9 to d0/df 0 2024-04-22T15:49:45.924 INFO:tasks.workunit.client.0.smithi016.stdout:2/838: chown d2/d4d/d64/fe6 1 1 2024-04-22T15:49:45.925 INFO:tasks.workunit.client.1.smithi027.stdout:7/41: creat d0/fb x:0 0 0 2024-04-22T15:49:45.926 INFO:tasks.workunit.client.1.smithi027.stdout:7/42: dread - d0/fa zero size 2024-04-22T15:49:45.926 INFO:tasks.workunit.client.1.smithi027.stdout:9/62: link d0/d8/f11 d0/dc/de/f14 0 2024-04-22T15:49:45.926 INFO:tasks.workunit.client.0.smithi016.stdout:3/904: dread d7/d17/d18/d32/d42/d63/f9d [0,4194304] 0 2024-04-22T15:49:45.927 INFO:tasks.workunit.client.1.smithi027.stdout:7/43: chown d0/fa 0 1 2024-04-22T15:49:45.928 INFO:tasks.workunit.client.1.smithi027.stdout:7/44: truncate d0/fb 348905 0 2024-04-22T15:49:45.928 INFO:tasks.workunit.client.0.smithi016.stdout:5/844: dwrite df/d15/d1a/d50/d77/f61 [0,4194304] 0 2024-04-22T15:49:45.930 INFO:tasks.workunit.client.0.smithi016.stdout:0/946: dread dc/d13/d14/de5/d65/d40/dd6/d104/fb5 [0,4194304] 0 2024-04-22T15:49:45.931 INFO:tasks.workunit.client.1.smithi027.stdout:1/47: creat d2/fa x:0 0 0 2024-04-22T15:49:45.932 INFO:tasks.workunit.client.1.smithi027.stdout:1/48: write d2/f7 [1471435,54836] 0 2024-04-22T15:49:45.933 INFO:tasks.workunit.client.1.smithi027.stdout:1/49: fdatasync d2/f3 0 2024-04-22T15:49:45.935 INFO:tasks.workunit.client.0.smithi016.stdout:5/845: dwrite df/d15/d1a/fbe [4194304,4194304] 0 2024-04-22T15:49:45.936 INFO:tasks.workunit.client.0.smithi016.stdout:2/839: link d2/d1a/d4c/f7a d2/d2c/f13e 0 2024-04-22T15:49:45.939 INFO:tasks.workunit.client.1.smithi027.stdout:3/65: write f15 [494146,88617] 0 2024-04-22T15:49:45.941 INFO:tasks.workunit.client.0.smithi016.stdout:6/900: creat d3/da/dd/f129 x:0 0 0 2024-04-22T15:49:45.942 INFO:tasks.workunit.client.1.smithi027.stdout:2/54: symlink dd/le 0 2024-04-22T15:49:45.946 INFO:tasks.workunit.client.1.smithi027.stdout:8/20: creat d1/f7 x:0 0 0 2024-04-22T15:49:45.946 INFO:tasks.workunit.client.0.smithi016.stdout:0/947: mkdir dc/d13/d14/d105/d131 0 2024-04-22T15:49:45.948 INFO:tasks.workunit.client.0.smithi016.stdout:2/840: mknod d2/d1a/d7b/dcc/c13f 0 2024-04-22T15:49:45.949 INFO:tasks.workunit.client.0.smithi016.stdout:2/841: fdatasync d2/d2c/d97/d67/f13c 0 2024-04-22T15:49:45.950 INFO:tasks.workunit.client.1.smithi027.stdout:9/63: symlink d0/d8/l15 0 2024-04-22T15:49:45.955 INFO:tasks.workunit.client.1.smithi027.stdout:7/45: mkdir d0/dc 0 2024-04-22T15:49:45.957 INFO:tasks.workunit.client.0.smithi016.stdout:2/842: fdatasync d2/d1a/d7b/dcc/f7e 0 2024-04-22T15:49:45.960 INFO:tasks.workunit.client.1.smithi027.stdout:1/50: symlink d2/lb 0 2024-04-22T15:49:45.961 INFO:tasks.workunit.client.0.smithi016.stdout:5/846: getdents df/d15/d1a/d50 0 2024-04-22T15:49:45.962 INFO:tasks.workunit.client.1.smithi027.stdout:3/66: chown c4 50574008 1 2024-04-22T15:49:45.962 INFO:tasks.workunit.client.1.smithi027.stdout:3/67: truncate f15 1577898 0 2024-04-22T15:49:45.963 INFO:tasks.workunit.client.1.smithi027.stdout:3/68: write f16 [902098,22942] 0 2024-04-22T15:49:45.965 INFO:tasks.workunit.client.0.smithi016.stdout:5/847: creat df/d5b/db3/f105 x:0 0 0 2024-04-22T15:49:45.967 INFO:tasks.workunit.client.0.smithi016.stdout:2/843: mknod d2/de/c140 0 2024-04-22T15:49:45.968 INFO:tasks.workunit.client.1.smithi027.stdout:3/69: dread f16 [0,4194304] 0 2024-04-22T15:49:45.969 INFO:tasks.workunit.client.0.smithi016.stdout:5/848: read - df/d5b/fdd zero size 2024-04-22T15:49:45.970 INFO:tasks.workunit.client.1.smithi027.stdout:8/21: symlink d1/d3/l8 0 2024-04-22T15:49:45.970 INFO:tasks.workunit.client.1.smithi027.stdout:3/70: dread f11 [0,4194304] 0 2024-04-22T15:49:45.971 INFO:tasks.workunit.client.1.smithi027.stdout:3/71: readlink l8 0 2024-04-22T15:49:45.971 INFO:tasks.workunit.client.1.smithi027.stdout:3/72: readlink l17 0 2024-04-22T15:49:45.972 INFO:tasks.workunit.client.0.smithi016.stdout:5/849: truncate df/d15/d1a/d50/d77/d7d/d8b/d8c/de4/ff3 855562 0 2024-04-22T15:49:45.972 INFO:tasks.workunit.client.1.smithi027.stdout:3/73: dread f16 [0,4194304] 0 2024-04-22T15:49:45.973 INFO:tasks.workunit.client.1.smithi027.stdout:3/74: fsync f16 0 2024-04-22T15:49:45.973 INFO:tasks.workunit.client.0.smithi016.stdout:2/844: dread d2/d2c/d97/f6a [0,4194304] 0 2024-04-22T15:49:45.976 INFO:tasks.workunit.client.0.smithi016.stdout:5/850: rmdir df/d15/d1a/d50/d77/d7d 39 2024-04-22T15:49:45.977 INFO:tasks.workunit.client.1.smithi027.stdout:3/75: dread f15 [0,4194304] 0 2024-04-22T15:49:45.977 INFO:tasks.workunit.client.1.smithi027.stdout:7/46: rename d0/f9 to d0/fd 0 2024-04-22T15:49:45.985 INFO:tasks.workunit.client.0.smithi016.stdout:5/851: mkdir df/d15/d1a/d50/d77/d7d/d8b/d93/dfa/d103/d106 0 2024-04-22T15:49:45.987 INFO:tasks.workunit.client.1.smithi027.stdout:3/76: dwrite f16 [0,4194304] 0 2024-04-22T15:49:45.987 INFO:tasks.workunit.client.1.smithi027.stdout:7/47: dwrite d0/fb [0,4194304] 0 2024-04-22T15:49:45.987 INFO:tasks.workunit.client.1.smithi027.stdout:8/22: symlink d1/d3/l9 0 2024-04-22T15:49:45.987 INFO:tasks.workunit.client.1.smithi027.stdout:8/23: read - d1/f7 zero size 2024-04-22T15:49:45.988 INFO:tasks.workunit.client.1.smithi027.stdout:8/24: chown d1/c2 222889 1 2024-04-22T15:49:45.988 INFO:tasks.workunit.client.1.smithi027.stdout:8/25: readlink d1/d3/l8 0 2024-04-22T15:49:45.989 INFO:tasks.workunit.client.0.smithi016.stdout:5/852: unlink df/d5b/d70/fe7 0 2024-04-22T15:49:45.999 INFO:tasks.workunit.client.1.smithi027.stdout:7/48: rename d0/f2 to d0/dc/fe 0 2024-04-22T15:49:46.001 INFO:tasks.workunit.client.1.smithi027.stdout:8/26: dread d1/f4 [0,4194304] 0 2024-04-22T15:49:46.006 INFO:tasks.workunit.client.0.smithi016.stdout:4/839: rename d7/l25 to d7/d4e/d102/l112 0 2024-04-22T15:49:46.008 INFO:tasks.workunit.client.1.smithi027.stdout:7/49: dwrite d0/fa [0,4194304] 0 2024-04-22T15:49:46.008 INFO:tasks.workunit.client.0.smithi016.stdout:4/840: write d7/d21/d45/d49/d9b/db6/fc4 [994079,68061] 0 2024-04-22T15:49:46.009 INFO:tasks.workunit.client.1.smithi027.stdout:8/27: dread f0 [0,4194304] 0 2024-04-22T15:49:46.010 INFO:tasks.workunit.client.1.smithi027.stdout:8/28: fdatasync d1/f7 0 2024-04-22T15:49:46.012 INFO:tasks.workunit.client.0.smithi016.stdout:8/808: rename d2/d10/d47/d67/fd6 to d2/d82/d52/df0/d66/ffb 0 2024-04-22T15:49:46.015 INFO:tasks.workunit.client.0.smithi016.stdout:5/853: dread f2 [0,4194304] 0 2024-04-22T15:49:46.015 INFO:tasks.workunit.client.0.smithi016.stdout:4/841: mkdir d7/d4e/d102/d106/d88/d9d/dad/d113 0 2024-04-22T15:49:46.015 INFO:tasks.workunit.client.1.smithi027.stdout:8/29: creat d1/d3/fa x:0 0 0 2024-04-22T15:49:46.016 INFO:tasks.workunit.client.1.smithi027.stdout:8/30: chown f0 166 1 2024-04-22T15:49:46.018 INFO:tasks.workunit.client.0.smithi016.stdout:1/966: rename d3/d2e/d80/d82/l101 to d3/d4/d3a/d9e/dbf/l14e 0 2024-04-22T15:49:46.019 INFO:tasks.workunit.client.0.smithi016.stdout:8/809: dwrite d2/d82/f58 [0,4194304] 0 2024-04-22T15:49:46.020 INFO:tasks.workunit.client.1.smithi027.stdout:7/50: link d0/fa d0/ff 0 2024-04-22T15:49:46.022 INFO:tasks.workunit.client.1.smithi027.stdout:8/31: creat d1/fb x:0 0 0 2024-04-22T15:49:46.023 INFO:tasks.workunit.client.1.smithi027.stdout:8/32: chown d1/d3/f6 3 1 2024-04-22T15:49:46.026 INFO:tasks.workunit.client.1.smithi027.stdout:7/51: fdatasync d0/dc/fe 0 2024-04-22T15:49:46.027 INFO:tasks.workunit.client.1.smithi027.stdout:7/52: chown d0/ff 0 1 2024-04-22T15:49:46.027 INFO:tasks.workunit.client.0.smithi016.stdout:1/967: symlink d3/d4/d3a/d96/l14f 0 2024-04-22T15:49:46.029 INFO:tasks.workunit.client.1.smithi027.stdout:8/33: symlink d1/d3/lc 0 2024-04-22T15:49:46.031 INFO:tasks.workunit.client.0.smithi016.stdout:8/810: rename d2/d48/dad/cf1 to d2/d10/d43/cfc 0 2024-04-22T15:49:46.033 INFO:tasks.workunit.client.0.smithi016.stdout:4/842: mknod d7/d21/d45/d49/d9b/df4/c114 0 2024-04-22T15:49:46.034 INFO:tasks.workunit.client.0.smithi016.stdout:4/843: write d7/d21/d45/d49/f56 [855108,117051] 0 2024-04-22T15:49:46.035 INFO:tasks.workunit.client.1.smithi027.stdout:8/34: creat d1/fd x:0 0 0 2024-04-22T15:49:46.035 INFO:tasks.workunit.client.1.smithi027.stdout:7/53: dwrite d0/f3 [0,4194304] 0 2024-04-22T15:49:46.035 INFO:tasks.workunit.client.1.smithi027.stdout:7/54: stat d0/ff 0 2024-04-22T15:49:46.036 INFO:tasks.workunit.client.1.smithi027.stdout:8/35: write d1/fd [8419,43322] 0 2024-04-22T15:49:46.037 INFO:tasks.workunit.client.0.smithi016.stdout:8/811: read d2/d82/f7b [1923466,64812] 0 2024-04-22T15:49:46.038 INFO:tasks.workunit.client.0.smithi016.stdout:5/854: dread df/d15/d1a/f20 [0,4194304] 0 2024-04-22T15:49:46.038 INFO:tasks.workunit.client.0.smithi016.stdout:8/812: read d2/d82/d3e/dd5/fd8 [2869985,124376] 0 2024-04-22T15:49:46.040 INFO:tasks.workunit.client.1.smithi027.stdout:7/55: unlink d0/ff 0 2024-04-22T15:49:46.040 INFO:tasks.workunit.client.1.smithi027.stdout:7/56: chown d0/l4 858352 1 2024-04-22T15:49:46.041 INFO:tasks.workunit.client.0.smithi016.stdout:4/844: creat d7/d4e/d2f/d6f/f115 x:0 0 0 2024-04-22T15:49:46.043 INFO:tasks.workunit.client.1.smithi027.stdout:8/36: rename d1/c2 to d1/d3/ce 0 2024-04-22T15:49:46.044 INFO:tasks.workunit.client.1.smithi027.stdout:7/57: symlink d0/l10 0 2024-04-22T15:49:46.045 INFO:tasks.workunit.client.1.smithi027.stdout:7/58: fdatasync d0/fb 0 2024-04-22T15:49:46.046 INFO:tasks.workunit.client.1.smithi027.stdout:8/37: mknod d1/cf 0 2024-04-22T15:49:46.050 INFO:tasks.workunit.client.0.smithi016.stdout:4/845: rename d7/d4e/d102/d106/d88/d9d/dad/dac to d7/d4e/d39/df3/d116 0 2024-04-22T15:49:46.053 INFO:tasks.workunit.client.1.smithi027.stdout:8/38: creat d1/d3/f10 x:0 0 0 2024-04-22T15:49:46.054 INFO:tasks.workunit.client.1.smithi027.stdout:8/39: readlink d1/d3/lc 0 2024-04-22T15:49:46.056 INFO:tasks.workunit.client.0.smithi016.stdout:4/846: rename d7/d4e/d102/d106/d88/d9d/dad/cf9 to d7/d4e/d102/c117 0 2024-04-22T15:49:46.058 INFO:tasks.workunit.client.0.smithi016.stdout:5/855: creat df/d15/d1a/d50/d77/f107 x:0 0 0 2024-04-22T15:49:46.059 INFO:tasks.workunit.client.1.smithi027.stdout:8/40: symlink d1/l11 0 2024-04-22T15:49:46.059 INFO:tasks.workunit.client.1.smithi027.stdout:8/41: readlink d1/d3/l8 0 2024-04-22T15:49:46.061 INFO:tasks.workunit.client.0.smithi016.stdout:4/847: mknod d7/d21/d7c/d103/d10f/c118 0 2024-04-22T15:49:46.062 INFO:tasks.workunit.client.1.smithi027.stdout:8/42: write f0 [296919,127421] 0 2024-04-22T15:49:46.063 INFO:tasks.workunit.client.1.smithi027.stdout:8/43: chown d1/d3/l8 1410644800 1 2024-04-22T15:49:46.063 INFO:tasks.workunit.client.0.smithi016.stdout:5/856: mkdir df/d15/d1a/d50/d77/d7d/d8b/d93/db7/d108 0 2024-04-22T15:49:46.064 INFO:tasks.workunit.client.0.smithi016.stdout:4/848: read d7/d21/d45/d49/d9b/fa2 [435892,130798] 0 2024-04-22T15:49:46.064 INFO:tasks.workunit.client.0.smithi016.stdout:4/849: chown d7/d4e/c8f 59843 1 2024-04-22T15:49:46.066 INFO:tasks.workunit.client.1.smithi027.stdout:8/44: rename d1/l11 to d1/d3/l12 0 2024-04-22T15:49:46.067 INFO:tasks.workunit.client.1.smithi027.stdout:8/45: read - d1/d3/f10 zero size 2024-04-22T15:49:46.067 INFO:tasks.workunit.client.0.smithi016.stdout:5/857: symlink df/d15/d1a/d50/l109 0 2024-04-22T15:49:46.068 INFO:tasks.workunit.client.1.smithi027.stdout:8/46: stat d1/d3/l8 0 2024-04-22T15:49:46.069 INFO:tasks.workunit.client.1.smithi027.stdout:8/47: creat d1/d3/f13 x:0 0 0 2024-04-22T15:49:46.069 INFO:tasks.workunit.client.1.smithi027.stdout:8/48: stat d1/d3/fa 0 2024-04-22T15:49:46.071 INFO:tasks.workunit.client.0.smithi016.stdout:5/858: rename df/d22 to df/d15/d1a/d4e/d10a 0 2024-04-22T15:49:46.072 INFO:tasks.workunit.client.1.smithi027.stdout:8/49: rename d1/d3/lc to d1/l14 0 2024-04-22T15:49:46.076 INFO:tasks.workunit.client.0.smithi016.stdout:5/859: getdents df/d15 0 2024-04-22T15:49:46.078 INFO:tasks.workunit.client.0.smithi016.stdout:5/860: write df/d15/d1a/d4e/d58/d87/fe1 [936356,125618] 0 2024-04-22T15:49:46.080 INFO:tasks.workunit.client.1.smithi027.stdout:8/50: dwrite d1/d3/fa [0,4194304] 0 2024-04-22T15:49:46.081 INFO:tasks.workunit.client.1.smithi027.stdout:8/51: truncate d1/f7 313384 0 2024-04-22T15:49:46.083 INFO:tasks.workunit.client.1.smithi027.stdout:8/52: dread d1/f7 [0,4194304] 0 2024-04-22T15:49:46.084 INFO:tasks.workunit.client.1.smithi027.stdout:8/53: read d1/fd [9422,94355] 0 2024-04-22T15:49:46.086 INFO:tasks.workunit.client.0.smithi016.stdout:5/861: dwrite df/d15/d1a/f71 [0,4194304] 0 2024-04-22T15:49:46.088 INFO:tasks.workunit.client.0.smithi016.stdout:7/833: dwrite d3/d11a/fab [0,4194304] 0 2024-04-22T15:49:46.092 INFO:tasks.workunit.client.0.smithi016.stdout:5/862: unlink df/d15/d1a/d50/d77/d7d/d8b/d8c/ddf/fe9 0 2024-04-22T15:49:46.093 INFO:tasks.workunit.client.0.smithi016.stdout:7/834: mknod d3/de6/c12e 0 2024-04-22T15:49:46.095 INFO:tasks.workunit.client.0.smithi016.stdout:5/863: chown df/d15/d1a/d50/d77/d7d/d8b/d93/dfa 35 1 2024-04-22T15:49:46.097 INFO:tasks.workunit.client.0.smithi016.stdout:5/864: dread df/d15/fa9 [0,4194304] 0 2024-04-22T15:49:46.097 INFO:tasks.workunit.client.0.smithi016.stdout:7/835: symlink d3/de6/l12f 0 2024-04-22T15:49:46.099 INFO:tasks.workunit.client.0.smithi016.stdout:7/836: write d3/dc/d9f/fae [1323062,115687] 0 2024-04-22T15:49:46.100 INFO:tasks.workunit.client.1.smithi027.stdout:6/28: dread f4 [0,4194304] 0 2024-04-22T15:49:46.101 INFO:tasks.workunit.client.1.smithi027.stdout:6/29: read f0 [1918987,118151] 0 2024-04-22T15:49:46.101 INFO:tasks.workunit.client.1.smithi027.stdout:6/30: chown d5/l7 103627 1 2024-04-22T15:49:46.105 INFO:tasks.workunit.client.0.smithi016.stdout:7/837: creat d3/dc/d1b/d5a/d9/df/d94/d4c/d7a/f130 x:0 0 0 2024-04-22T15:49:46.106 INFO:tasks.workunit.client.1.smithi027.stdout:6/31: mkdir d5/d9 0 2024-04-22T15:49:46.107 INFO:tasks.workunit.client.0.smithi016.stdout:5/865: mkdir df/d15/d1a/d50/d77/d94/d10b 0 2024-04-22T15:49:46.108 INFO:tasks.workunit.client.1.smithi027.stdout:4/43: truncate f2 4105884 0 2024-04-22T15:49:46.109 INFO:tasks.workunit.client.1.smithi027.stdout:6/32: dread f4 [0,4194304] 0 2024-04-22T15:49:46.110 INFO:tasks.workunit.client.0.smithi016.stdout:7/838: creat d3/d10b/f131 x:0 0 0 2024-04-22T15:49:46.111 INFO:tasks.workunit.client.0.smithi016.stdout:3/905: write d7/d17/d18/d32/d42/d63/d71/fd9 [4377891,41542] 0 2024-04-22T15:49:46.115 INFO:tasks.workunit.client.1.smithi027.stdout:4/44: dwrite d5/d7/ff [0,4194304] 0 2024-04-22T15:49:46.115 INFO:tasks.workunit.client.1.smithi027.stdout:2/55: getdents dd 0 2024-04-22T15:49:46.116 INFO:tasks.workunit.client.0.smithi016.stdout:7/839: readlink d3/dc/d1b/d5a/d9/df/d94/l104 0 2024-04-22T15:49:46.116 INFO:tasks.workunit.client.0.smithi016.stdout:3/906: stat d7/d17/d18/d32/d42/d63/deb/d21/d55/ddb/lbb 0 2024-04-22T15:49:46.116 INFO:tasks.workunit.client.1.smithi027.stdout:4/45: dread - d5/d7/fb zero size 2024-04-22T15:49:46.117 INFO:tasks.workunit.client.0.smithi016.stdout:5/866: dwrite df/d15/d1a/d50/d77/d7d/d8b/d93/dbb/ff4 [0,4194304] 0 2024-04-22T15:49:46.118 INFO:tasks.workunit.client.0.smithi016.stdout:7/840: stat d3/dc/d1b/d5a/d9/df/d94/f105 0 2024-04-22T15:49:46.118 INFO:tasks.workunit.client.1.smithi027.stdout:2/56: dread f7 [0,4194304] 0 2024-04-22T15:49:46.119 INFO:tasks.workunit.client.1.smithi027.stdout:4/46: creat d5/d7/f10 x:0 0 0 2024-04-22T15:49:46.121 INFO:tasks.workunit.client.1.smithi027.stdout:2/57: dread f7 [0,4194304] 0 2024-04-22T15:49:46.123 INFO:tasks.workunit.client.1.smithi027.stdout:4/47: mkdir d5/d11 0 2024-04-22T15:49:46.124 INFO:tasks.workunit.client.0.smithi016.stdout:0/948: write dc/d13/d70/d8c/f11d [4130060,88495] 0 2024-04-22T15:49:46.126 INFO:tasks.workunit.client.0.smithi016.stdout:9/904: sync 2024-04-22T15:49:46.126 INFO:tasks.workunit.client.0.smithi016.stdout:6/901: sync 2024-04-22T15:49:46.126 INFO:tasks.workunit.client.0.smithi016.stdout:5/867: mkdir df/d15/d1a/d50/d77/d7d/d8b/d8c/d10c 0 2024-04-22T15:49:46.128 INFO:tasks.workunit.client.1.smithi027.stdout:2/58: mkdir dd/df 0 2024-04-22T15:49:46.130 INFO:tasks.workunit.client.0.smithi016.stdout:3/907: dwrite d7/d17/d18/d32/d3c/fcc [0,4194304] 0 2024-04-22T15:49:46.132 INFO:tasks.workunit.client.1.smithi027.stdout:1/51: rmdir d2 39 2024-04-22T15:49:46.134 INFO:tasks.workunit.client.1.smithi027.stdout:0/96: write d0/f1 [223091,58898] 0 2024-04-22T15:49:46.135 INFO:tasks.workunit.client.1.smithi027.stdout:4/48: mknod d5/d11/c12 0 2024-04-22T15:49:46.136 INFO:tasks.workunit.client.0.smithi016.stdout:6/902: chown d3/da/ddd/d101/d11b/d83/cc5 250595 1 2024-04-22T15:49:46.136 INFO:tasks.workunit.client.1.smithi027.stdout:4/49: chown d5/fa 801 1 2024-04-22T15:49:46.137 INFO:tasks.workunit.client.0.smithi016.stdout:3/908: read d7/d17/d18/d27/fb8 [2140194,122136] 0 2024-04-22T15:49:46.137 INFO:tasks.workunit.client.0.smithi016.stdout:6/903: readlink d3/da/db/d14/d16/l2f 0 2024-04-22T15:49:46.138 INFO:tasks.workunit.client.0.smithi016.stdout:9/905: mkdir d9/d57/da7/daa/d124 0 2024-04-22T15:49:46.140 INFO:tasks.workunit.client.0.smithi016.stdout:5/868: fdatasync df/d15/d1a/d50/d77/d7d/d8b/d8c/de4/fd1 0 2024-04-22T15:49:46.141 INFO:tasks.workunit.client.1.smithi027.stdout:5/43: truncate d0/f7 244034 0 2024-04-22T15:49:46.144 INFO:tasks.workunit.client.0.smithi016.stdout:2/845: write d2/d23/d92/f9d [442831,444] 0 2024-04-22T15:49:46.145 INFO:tasks.workunit.client.0.smithi016.stdout:7/841: creat d3/dc/d1b/d5a/f132 x:0 0 0 2024-04-22T15:49:46.146 INFO:tasks.workunit.client.0.smithi016.stdout:7/842: chown d3/dc/d9f 6091601 1 2024-04-22T15:49:46.146 INFO:tasks.workunit.client.1.smithi027.stdout:4/50: creat d5/f13 x:0 0 0 2024-04-22T15:49:46.147 INFO:tasks.workunit.client.0.smithi016.stdout:0/949: dread dc/d13/d14/de5/d65/fec [0,4194304] 0 2024-04-22T15:49:46.148 INFO:tasks.workunit.client.0.smithi016.stdout:0/950: readlink dc/d7d/lde 0 2024-04-22T15:49:46.149 INFO:tasks.workunit.client.1.smithi027.stdout:1/52: unlink d2/f4 0 2024-04-22T15:49:46.149 INFO:tasks.workunit.client.1.smithi027.stdout:1/53: stat d2/lb 0 2024-04-22T15:49:46.150 INFO:tasks.workunit.client.0.smithi016.stdout:9/906: rmdir d9/de/d26/d5a 39 2024-04-22T15:49:46.153 INFO:tasks.workunit.client.0.smithi016.stdout:3/909: mkdir d7/d17/d18/d32/d42/d63/db4/d11a 0 2024-04-22T15:49:46.154 INFO:tasks.workunit.client.0.smithi016.stdout:0/951: dread dc/f68 [0,4194304] 0 2024-04-22T15:49:46.154 INFO:tasks.workunit.client.1.smithi027.stdout:0/97: unlink d0/c13 0 2024-04-22T15:49:46.156 INFO:tasks.workunit.client.0.smithi016.stdout:5/869: symlink df/d15/d1a/d4e/d10a/l10d 0 2024-04-22T15:49:46.158 INFO:tasks.workunit.client.1.smithi027.stdout:9/64: dwrite d0/dc/de/f14 [0,4194304] 0 2024-04-22T15:49:46.159 INFO:tasks.workunit.client.0.smithi016.stdout:2/846: dread d2/d1a/d4c/d9f/dae/fb3 [0,4194304] 0 2024-04-22T15:49:46.163 INFO:tasks.workunit.client.0.smithi016.stdout:9/907: rmdir d9 39 2024-04-22T15:49:46.167 INFO:tasks.workunit.client.0.smithi016.stdout:2/847: dwrite d2/d1a/d7b/dcc/f135 [0,4194304] 0 2024-04-22T15:49:46.170 INFO:tasks.workunit.client.0.smithi016.stdout:5/870: creat df/d15/d1a/d4e/f10e x:0 0 0 2024-04-22T15:49:46.176 INFO:tasks.workunit.client.1.smithi027.stdout:9/65: rename d0/d8/l15 to d0/dc/l16 0 2024-04-22T15:49:46.176 INFO:tasks.workunit.client.0.smithi016.stdout:0/952: dread dc/f1e [0,4194304] 0 2024-04-22T15:49:46.176 INFO:tasks.workunit.client.0.smithi016.stdout:2/848: dwrite d2/d23/f121 [0,4194304] 0 2024-04-22T15:49:46.179 INFO:tasks.workunit.client.0.smithi016.stdout:3/910: mknod d7/d17/d18/d32/d3c/c11b 0 2024-04-22T15:49:46.180 INFO:tasks.workunit.client.1.smithi027.stdout:0/98: truncate d0/f10 1266718 0 2024-04-22T15:49:46.184 INFO:tasks.workunit.client.1.smithi027.stdout:9/66: symlink d0/dc/l17 0 2024-04-22T15:49:46.185 INFO:tasks.workunit.client.0.smithi016.stdout:3/911: dread d7/d17/d18/d32/d42/d63/deb/d21/fcd [0,4194304] 0 2024-04-22T15:49:46.186 INFO:tasks.workunit.client.0.smithi016.stdout:2/849: creat d2/d2c/d97/d34/f141 x:0 0 0 2024-04-22T15:49:46.187 INFO:tasks.workunit.client.1.smithi027.stdout:0/99: creat d0/d1a/f20 x:0 0 0 2024-04-22T15:49:46.188 INFO:tasks.workunit.client.1.smithi027.stdout:0/100: read d0/f14 [63855,91754] 0 2024-04-22T15:49:46.190 INFO:tasks.workunit.client.0.smithi016.stdout:9/908: fdatasync d9/d37/f4a 0 2024-04-22T15:49:46.191 INFO:tasks.workunit.client.0.smithi016.stdout:0/953: symlink dc/d13/d14/l132 0 2024-04-22T15:49:46.191 INFO:tasks.workunit.client.0.smithi016.stdout:3/912: fsync d7/d17/d18/d32/d42/d63/db4/fda 0 2024-04-22T15:49:46.192 INFO:tasks.workunit.client.1.smithi027.stdout:9/67: creat d0/d8/f18 x:0 0 0 2024-04-22T15:49:46.192 INFO:tasks.workunit.client.1.smithi027.stdout:7/59: truncate d0/fb 3150852 0 2024-04-22T15:49:46.192 INFO:tasks.workunit.client.1.smithi027.stdout:7/60: dread - d0/fd zero size 2024-04-22T15:49:46.193 INFO:tasks.workunit.client.1.smithi027.stdout:7/61: readlink d0/l10 0 2024-04-22T15:49:46.193 INFO:tasks.workunit.client.0.smithi016.stdout:5/871: mknod df/d15/d1a/d28/c10f 0 2024-04-22T15:49:46.195 INFO:tasks.workunit.client.0.smithi016.stdout:1/968: dwrite d3/d28/dbd/f14a [0,4194304] 0 2024-04-22T15:49:46.197 INFO:tasks.workunit.client.1.smithi027.stdout:9/68: rmdir d0/dc 39 2024-04-22T15:49:46.199 INFO:tasks.workunit.client.0.smithi016.stdout:9/909: unlink d9/ff9 0 2024-04-22T15:49:46.204 INFO:tasks.workunit.client.1.smithi027.stdout:9/69: dwrite d0/fb [0,4194304] 0 2024-04-22T15:49:46.207 INFO:tasks.workunit.client.1.smithi027.stdout:0/101: mknod d0/c21 0 2024-04-22T15:49:46.209 INFO:tasks.workunit.client.0.smithi016.stdout:2/850: rename d2/d1a/d4c/f7a to d2/d2c/daa/d110/f142 0 2024-04-22T15:49:46.214 INFO:tasks.workunit.client.0.smithi016.stdout:2/851: chown d2/d2c/d97/d34/d60/c117 0 1 2024-04-22T15:49:46.214 INFO:tasks.workunit.client.0.smithi016.stdout:5/872: mkdir df/d15/d110 0 2024-04-22T15:49:46.214 INFO:tasks.workunit.client.1.smithi027.stdout:8/54: chown d1/l14 7361 1 2024-04-22T15:49:46.215 INFO:tasks.workunit.client.1.smithi027.stdout:8/55: write d1/d3/f6 [518142,77685] 0 2024-04-22T15:49:46.218 INFO:tasks.workunit.client.0.smithi016.stdout:4/850: dwrite d7/d21/d45/d49/d67/fbd [0,4194304] 0 2024-04-22T15:49:46.219 INFO:tasks.workunit.client.0.smithi016.stdout:1/969: symlink d3/dea/l150 0 2024-04-22T15:49:46.222 INFO:tasks.workunit.client.1.smithi027.stdout:8/56: dread d1/d3/fa [0,4194304] 0 2024-04-22T15:49:46.225 INFO:tasks.workunit.client.1.smithi027.stdout:0/102: dwrite d0/f18 [0,4194304] 0 2024-04-22T15:49:46.226 INFO:tasks.workunit.client.0.smithi016.stdout:9/910: creat d9/d37/f125 x:0 0 0 2024-04-22T15:49:46.229 INFO:tasks.workunit.client.1.smithi027.stdout:9/70: symlink d0/dc/d12/l19 0 2024-04-22T15:49:46.229 INFO:tasks.workunit.client.1.smithi027.stdout:3/77: sync 2024-04-22T15:49:46.230 INFO:tasks.workunit.client.1.smithi027.stdout:9/71: stat d0/l13 0 2024-04-22T15:49:46.235 INFO:tasks.workunit.client.1.smithi027.stdout:0/103: dwrite d0/f12 [0,4194304] 0 2024-04-22T15:49:46.243 INFO:tasks.workunit.client.1.smithi027.stdout:6/33: dwrite f0 [4194304,4194304] 0 2024-04-22T15:49:46.244 INFO:tasks.workunit.client.1.smithi027.stdout:3/78: rename c4 to c1b 0 2024-04-22T15:49:46.245 INFO:tasks.workunit.client.1.smithi027.stdout:3/79: read f16 [1944611,98285] 0 2024-04-22T15:49:46.245 INFO:tasks.workunit.client.0.smithi016.stdout:1/970: truncate d3/d2e/d80/dbb/fc2 855493 0 2024-04-22T15:49:46.246 INFO:tasks.workunit.client.0.smithi016.stdout:3/913: dread d7/d17/d18/fae [0,4194304] 0 2024-04-22T15:49:46.246 INFO:tasks.workunit.client.1.smithi027.stdout:3/80: dread f11 [0,4194304] 0 2024-04-22T15:49:46.247 INFO:tasks.workunit.client.0.smithi016.stdout:2/852: dwrite d2/d2c/d97/d67/dbf/fe8 [4194304,4194304] 0 2024-04-22T15:49:46.249 INFO:tasks.workunit.client.0.smithi016.stdout:2/853: truncate d2/d4d/d64/f111 986174 0 2024-04-22T15:49:46.249 INFO:tasks.workunit.client.0.smithi016.stdout:9/911: mkdir d9/d20/dac/dae/d126 0 2024-04-22T15:49:46.249 INFO:tasks.workunit.client.1.smithi027.stdout:6/34: dread f0 [0,4194304] 0 2024-04-22T15:49:46.253 INFO:tasks.workunit.client.0.smithi016.stdout:2/854: truncate d2/d9/fa8 630488 0 2024-04-22T15:49:46.253 INFO:tasks.workunit.client.0.smithi016.stdout:2/855: stat d2/d1a 0 2024-04-22T15:49:46.254 INFO:tasks.workunit.client.1.smithi027.stdout:3/81: dwrite f16 [0,4194304] 0 2024-04-22T15:49:46.255 INFO:tasks.workunit.client.1.smithi027.stdout:6/35: dread f4 [0,4194304] 0 2024-04-22T15:49:46.256 INFO:tasks.workunit.client.1.smithi027.stdout:2/59: write f7 [1667810,111932] 0 2024-04-22T15:49:46.257 INFO:tasks.workunit.client.0.smithi016.stdout:4/851: link d7/d4e/f60 d7/d4e/d102/d106/d88/deb/f119 0 2024-04-22T15:49:46.258 INFO:tasks.workunit.client.1.smithi027.stdout:2/60: write f7 [559261,14521] 0 2024-04-22T15:49:46.260 INFO:tasks.workunit.client.0.smithi016.stdout:3/914: chown d7/d17/f2e 2730 1 2024-04-22T15:49:46.263 INFO:tasks.workunit.client.1.smithi027.stdout:6/36: dwrite f0 [4194304,4194304] 0 2024-04-22T15:49:46.263 INFO:tasks.workunit.client.1.smithi027.stdout:5/44: dread d0/f3 [0,4194304] 0 2024-04-22T15:49:46.265 INFO:tasks.workunit.client.1.smithi027.stdout:5/45: chown d0/f3 3175 1 2024-04-22T15:49:46.267 INFO:tasks.workunit.client.0.smithi016.stdout:1/971: creat d3/d2e/d9f/deb/f151 x:0 0 0 2024-04-22T15:49:46.271 INFO:tasks.workunit.client.0.smithi016.stdout:9/912: creat d9/d20/dac/dbd/f127 x:0 0 0 2024-04-22T15:49:46.274 INFO:tasks.workunit.client.0.smithi016.stdout:0/954: getdents dc/d13/d6f 0 2024-04-22T15:49:46.275 INFO:tasks.workunit.client.1.smithi027.stdout:1/54: rmdir d2 39 2024-04-22T15:49:46.278 INFO:tasks.workunit.client.0.smithi016.stdout:4/852: creat d7/d4e/d39/df3/d116/f11a x:0 0 0 2024-04-22T15:49:46.279 INFO:tasks.workunit.client.1.smithi027.stdout:3/82: creat f1c x:0 0 0 2024-04-22T15:49:46.279 INFO:tasks.workunit.client.1.smithi027.stdout:3/83: readlink l3 0 2024-04-22T15:49:46.280 INFO:tasks.workunit.client.0.smithi016.stdout:6/904: dwrite d3/da/db/d44/d68/fa5 [0,4194304] 0 2024-04-22T15:49:46.280 INFO:tasks.workunit.client.0.smithi016.stdout:4/853: write d7/d22/f64 [995417,94963] 0 2024-04-22T15:49:46.282 INFO:tasks.workunit.client.0.smithi016.stdout:1/972: symlink d3/dea/l152 0 2024-04-22T15:49:46.287 INFO:tasks.workunit.client.0.smithi016.stdout:2/856: creat d2/d2c/d97/d12e/f143 x:0 0 0 2024-04-22T15:49:46.288 INFO:tasks.workunit.client.0.smithi016.stdout:0/955: dread dc/d13/d14/de5/f117 [0,4194304] 0 2024-04-22T15:49:46.290 INFO:tasks.workunit.client.0.smithi016.stdout:3/915: mkdir d7/d17/d18/d32/d42/d63/db4/d11a/d11c 0 2024-04-22T15:49:46.293 INFO:tasks.workunit.client.0.smithi016.stdout:3/916: write d7/d17/d18/d32/d42/d63/db4/f10a [718314,74324] 0 2024-04-22T15:49:46.296 INFO:tasks.workunit.client.0.smithi016.stdout:2/857: dwrite d2/d1a/d7b/dcc/f7e [4194304,4194304] 0 2024-04-22T15:49:46.297 INFO:tasks.workunit.client.1.smithi027.stdout:9/72: link d0/l6 d0/dc/df/l1a 0 2024-04-22T15:49:46.298 INFO:tasks.workunit.client.1.smithi027.stdout:9/73: write d0/dc/de/f14 [3773137,117134] 0 2024-04-22T15:49:46.300 INFO:tasks.workunit.client.0.smithi016.stdout:1/973: chown d3/d28/c114 1816694356 1 2024-04-22T15:49:46.302 INFO:tasks.workunit.client.1.smithi027.stdout:8/57: getdents d1 0 2024-04-22T15:49:46.305 INFO:tasks.workunit.client.0.smithi016.stdout:6/905: symlink d3/da/l12a 0 2024-04-22T15:49:46.308 INFO:tasks.workunit.client.1.smithi027.stdout:0/104: creat d0/f22 x:0 0 0 2024-04-22T15:49:46.309 INFO:tasks.workunit.client.1.smithi027.stdout:3/84: creat f1d x:0 0 0 2024-04-22T15:49:46.310 INFO:tasks.workunit.client.1.smithi027.stdout:3/85: readlink l8 0 2024-04-22T15:49:46.310 INFO:tasks.workunit.client.0.smithi016.stdout:4/854: mknod d7/d21/c11b 0 2024-04-22T15:49:46.311 INFO:tasks.workunit.client.1.smithi027.stdout:3/86: dread f13 [0,4194304] 0 2024-04-22T15:49:46.311 INFO:tasks.workunit.client.1.smithi027.stdout:3/87: stat la 0 2024-04-22T15:49:46.312 INFO:tasks.workunit.client.1.smithi027.stdout:2/61: rename c9 to dd/c10 0 2024-04-22T15:49:46.315 INFO:tasks.workunit.client.0.smithi016.stdout:4/855: dread d7/d21/d45/d49/d9b/db6/fc4 [0,4194304] 0 2024-04-22T15:49:46.316 INFO:tasks.workunit.client.1.smithi027.stdout:6/37: mknod d5/d8/ca 0 2024-04-22T15:49:46.317 INFO:tasks.workunit.client.0.smithi016.stdout:1/974: mkdir d3/d4/d15/d38/d11a/d153 0 2024-04-22T15:49:46.318 INFO:tasks.workunit.client.0.smithi016.stdout:0/956: mknod dc/d13/d14/de5/dd8/d113/c133 0 2024-04-22T15:49:46.322 INFO:tasks.workunit.client.0.smithi016.stdout:0/957: dread - dc/d13/d14/de5/d65/d40/d4a/d59/d78/dbd/fda zero size 2024-04-22T15:49:46.322 INFO:tasks.workunit.client.1.smithi027.stdout:8/58: mknod d1/d3/c15 0 2024-04-22T15:49:46.323 INFO:tasks.workunit.client.1.smithi027.stdout:8/59: stat d1/d3/f10 0 2024-04-22T15:49:46.324 INFO:tasks.workunit.client.1.smithi027.stdout:8/60: truncate d1/d3/f10 199867 0 2024-04-22T15:49:46.325 INFO:tasks.workunit.client.0.smithi016.stdout:4/856: dwrite d7/d4e/d102/d106/d88/d9d/dad/fb4 [4194304,4194304] 0 2024-04-22T15:49:46.331 INFO:tasks.workunit.client.1.smithi027.stdout:8/61: dwrite d1/fb [0,4194304] 0 2024-04-22T15:49:46.332 INFO:tasks.workunit.client.1.smithi027.stdout:3/88: creat f1e x:0 0 0 2024-04-22T15:49:46.334 INFO:tasks.workunit.client.1.smithi027.stdout:8/62: write d1/d3/f6 [1029269,21775] 0 2024-04-22T15:49:46.334 INFO:tasks.workunit.client.1.smithi027.stdout:8/63: readlink d1/l14 0 2024-04-22T15:49:46.335 INFO:tasks.workunit.client.0.smithi016.stdout:1/975: creat d3/d4/d19/d20/d2b/dd6/d12d/f154 x:0 0 0 2024-04-22T15:49:46.337 INFO:tasks.workunit.client.0.smithi016.stdout:1/976: stat d3/d4/d19/d20/d2b/dd6/d12d/ff2 0 2024-04-22T15:49:46.338 INFO:tasks.workunit.client.1.smithi027.stdout:1/55: rename f0 to d2/fc 0 2024-04-22T15:49:46.339 INFO:tasks.workunit.client.1.smithi027.stdout:1/56: write d2/f7 [420168,110055] 0 2024-04-22T15:49:46.339 INFO:tasks.workunit.client.1.smithi027.stdout:5/46: write d0/f7 [345197,18164] 0 2024-04-22T15:49:46.340 INFO:tasks.workunit.client.1.smithi027.stdout:1/57: chown d2/f3 1770497059 1 2024-04-22T15:49:46.342 INFO:tasks.workunit.client.0.smithi016.stdout:6/906: truncate d3/da/d12/d13/d97/de0/fe3 1160237 0 2024-04-22T15:49:46.345 INFO:tasks.workunit.client.0.smithi016.stdout:5/873: rmdir df/d15/d1a/d4e 39 2024-04-22T15:49:46.346 INFO:tasks.workunit.client.0.smithi016.stdout:5/874: fdatasync df/d15/d1a/d50/d77/d7d/fc6 0 2024-04-22T15:49:46.346 INFO:tasks.workunit.client.1.smithi027.stdout:9/74: chown d0/l6 242253167 1 2024-04-22T15:49:46.347 INFO:tasks.workunit.client.0.smithi016.stdout:5/875: stat df/d5b/d5d/d5f/lde 0 2024-04-22T15:49:46.347 INFO:tasks.workunit.client.1.smithi027.stdout:5/47: dwrite d0/f5 [0,4194304] 0 2024-04-22T15:49:46.349 INFO:tasks.workunit.client.1.smithi027.stdout:9/75: dread - d0/d8/f18 zero size 2024-04-22T15:49:46.349 INFO:tasks.workunit.client.1.smithi027.stdout:9/76: fdatasync d0/dc/de/f14 0 2024-04-22T15:49:46.349 INFO:tasks.workunit.client.0.smithi016.stdout:0/958: symlink dc/d13/d6f/l134 0 2024-04-22T15:49:46.351 INFO:tasks.workunit.client.0.smithi016.stdout:0/959: write dc/d13/d14/de5/d65/d40/d4a/d59/d9d/fc1 [5032620,101653] 0 2024-04-22T15:49:46.353 INFO:tasks.workunit.client.1.smithi027.stdout:5/48: dread d0/f5 [0,4194304] 0 2024-04-22T15:49:46.354 INFO:tasks.workunit.client.1.smithi027.stdout:0/105: symlink d0/l23 0 2024-04-22T15:49:46.356 INFO:tasks.workunit.client.0.smithi016.stdout:2/858: link d2/d23/d92/c61 d2/d23/d92/d55/c144 0 2024-04-22T15:49:46.357 INFO:tasks.workunit.client.1.smithi027.stdout:3/89: write ff [1834094,18417] 0 2024-04-22T15:49:46.359 INFO:tasks.workunit.client.0.smithi016.stdout:1/977: write d3/d4/d3a/d9e/d124/f13b [6733984,8732] 0 2024-04-22T15:49:46.359 INFO:tasks.workunit.client.0.smithi016.stdout:4/857: dwrite d7/d24/d41/f57 [0,4194304] 0 2024-04-22T15:49:46.360 INFO:tasks.workunit.client.0.smithi016.stdout:4/858: fsync d7/d24/d41/f5d 0 2024-04-22T15:49:46.361 INFO:tasks.workunit.client.0.smithi016.stdout:4/859: fsync d7/d3c/f44 0 2024-04-22T15:49:46.362 INFO:tasks.workunit.client.1.smithi027.stdout:8/64: symlink d1/d3/l16 0 2024-04-22T15:49:46.363 INFO:tasks.workunit.client.0.smithi016.stdout:4/860: dread - d7/d4e/de1/fff zero size 2024-04-22T15:49:46.366 INFO:tasks.workunit.client.0.smithi016.stdout:5/876: fsync df/d15/d1a/f78 0 2024-04-22T15:49:46.367 INFO:tasks.workunit.client.0.smithi016.stdout:5/877: chown df/d5b/ffd 2 1 2024-04-22T15:49:46.369 INFO:tasks.workunit.client.1.smithi027.stdout:7/62: dwrite d0/fb [0,4194304] 0 2024-04-22T15:49:46.369 INFO:tasks.workunit.client.0.smithi016.stdout:7/843: sync 2024-04-22T15:49:46.369 INFO:tasks.workunit.client.0.smithi016.stdout:8/813: sync 2024-04-22T15:49:46.373 INFO:tasks.workunit.client.1.smithi027.stdout:1/58: symlink d2/ld 0 2024-04-22T15:49:46.375 INFO:tasks.workunit.client.0.smithi016.stdout:5/878: dwrite df/d15/d1a/f71 [0,4194304] 0 2024-04-22T15:49:46.375 INFO:tasks.workunit.client.0.smithi016.stdout:2/859: symlink d2/d2c/d97/d34/d60/l145 0 2024-04-22T15:49:46.382 INFO:tasks.workunit.client.1.smithi027.stdout:9/77: mknod d0/d8/c1b 0 2024-04-22T15:49:46.385 INFO:tasks.workunit.client.1.smithi027.stdout:5/49: mkdir d0/df/d10 0 2024-04-22T15:49:46.385 INFO:tasks.workunit.client.0.smithi016.stdout:5/879: read - df/d5b/db3/dc1/ff6 zero size 2024-04-22T15:49:46.386 INFO:tasks.workunit.client.1.smithi027.stdout:5/50: write d0/f8 [657031,101783] 0 2024-04-22T15:49:46.386 INFO:tasks.workunit.client.0.smithi016.stdout:9/913: sync 2024-04-22T15:49:46.386 INFO:tasks.workunit.client.1.smithi027.stdout:5/51: write d0/f8 [1046453,742] 0 2024-04-22T15:49:46.389 INFO:tasks.workunit.client.0.smithi016.stdout:2/860: chown d2/d23/d92/d55/ffe 4 1 2024-04-22T15:49:46.390 INFO:tasks.workunit.client.0.smithi016.stdout:1/978: rename d3/d4/d3a/d6b/d77/f76 to d3/d4/d3a/d96/de4/d133/d136/f155 0 2024-04-22T15:49:46.391 INFO:tasks.workunit.client.1.smithi027.stdout:3/90: creat f1f x:0 0 0 2024-04-22T15:49:46.391 INFO:tasks.workunit.client.1.smithi027.stdout:3/91: chown f1f 113023 1 2024-04-22T15:49:46.392 INFO:tasks.workunit.client.1.smithi027.stdout:3/92: chown f1d 105 1 2024-04-22T15:49:46.392 INFO:tasks.workunit.client.0.smithi016.stdout:7/844: dwrite d3/dc/d1b/d5a/d9/df/d94/d4c/f101 [0,4194304] 0 2024-04-22T15:49:46.392 INFO:tasks.workunit.client.0.smithi016.stdout:3/917: dwrite d7/d17/d18/d32/d42/d63/deb/fbc [0,4194304] 0 2024-04-22T15:49:46.396 INFO:tasks.workunit.client.1.smithi027.stdout:6/38: write f4 [2361996,24422] 0 2024-04-22T15:49:46.398 INFO:tasks.workunit.client.0.smithi016.stdout:6/907: link d3/da/ddd/d101/d11b/d83/cc5 d3/da/ddd/d101/d11b/d35/c12b 0 2024-04-22T15:49:46.402 INFO:tasks.workunit.client.0.smithi016.stdout:5/880: write df/d15/d1a/d50/d77/d7d/d8b/d8c/de4/fd1 [5041944,37466] 0 2024-04-22T15:49:46.402 INFO:tasks.workunit.client.1.smithi027.stdout:4/51: truncate f2 3077301 0 2024-04-22T15:49:46.403 INFO:tasks.workunit.client.1.smithi027.stdout:9/78: rmdir d0/dc/df 39 2024-04-22T15:49:46.404 INFO:tasks.workunit.client.1.smithi027.stdout:9/79: chown d0/l2 11806 1 2024-04-22T15:49:46.405 INFO:tasks.workunit.client.1.smithi027.stdout:9/80: rename d0 to d0/dc/de/d1c 22 2024-04-22T15:49:46.407 INFO:tasks.workunit.client.1.smithi027.stdout:5/52: mkdir d0/d11 0 2024-04-22T15:49:46.409 INFO:tasks.workunit.client.0.smithi016.stdout:5/881: dwrite df/d15/d1a/d50/d77/f62 [0,4194304] 0 2024-04-22T15:49:46.410 INFO:tasks.workunit.client.1.smithi027.stdout:9/81: dwrite d0/d8/f11 [0,4194304] 0 2024-04-22T15:49:46.413 INFO:tasks.workunit.client.1.smithi027.stdout:3/93: creat f20 x:0 0 0 2024-04-22T15:49:46.413 INFO:tasks.workunit.client.0.smithi016.stdout:2/861: symlink d2/d2c/d97/l146 0 2024-04-22T15:49:46.416 INFO:tasks.workunit.client.1.smithi027.stdout:9/82: dread d0/fb [4194304,4194304] 0 2024-04-22T15:49:46.419 INFO:tasks.workunit.client.0.smithi016.stdout:0/960: write dc/d13/d14/de5/d65/d30/d96/fc7 [278559,111031] 0 2024-04-22T15:49:46.421 INFO:tasks.workunit.client.0.smithi016.stdout:2/862: dwrite d2/d51/f13b [0,4194304] 0 2024-04-22T15:49:46.423 INFO:tasks.workunit.client.0.smithi016.stdout:2/863: dread - d2/d23/db8/dd4/d114/d11b/f13d zero size 2024-04-22T15:49:46.426 INFO:tasks.workunit.client.0.smithi016.stdout:1/979: creat d3/d4/da/d7e/f156 x:0 0 0 2024-04-22T15:49:46.427 INFO:tasks.workunit.client.1.smithi027.stdout:6/39: unlink f4 0 2024-04-22T15:49:46.430 INFO:tasks.workunit.client.1.smithi027.stdout:4/52: mknod d5/d7/c14 0 2024-04-22T15:49:46.431 INFO:tasks.workunit.client.1.smithi027.stdout:6/40: dread f0 [0,4194304] 0 2024-04-22T15:49:46.432 INFO:tasks.workunit.client.1.smithi027.stdout:6/41: fsync f0 0 2024-04-22T15:49:46.435 INFO:tasks.workunit.client.0.smithi016.stdout:6/908: rename d3/da/db/d44/d68/d95 to d3/da/dd/d7b/d12c 0 2024-04-22T15:49:46.442 INFO:tasks.workunit.client.1.smithi027.stdout:3/94: creat f21 x:0 0 0 2024-04-22T15:49:46.445 INFO:tasks.workunit.client.1.smithi027.stdout:8/65: truncate d1/d3/fa 2659288 0 2024-04-22T15:49:46.446 INFO:tasks.workunit.client.1.smithi027.stdout:7/63: truncate d0/fb 1126909 0 2024-04-22T15:49:46.446 INFO:tasks.workunit.client.0.smithi016.stdout:9/914: dwrite f7 [0,4194304] 0 2024-04-22T15:49:46.448 INFO:tasks.workunit.client.1.smithi027.stdout:1/59: truncate d2/fc 260150 0 2024-04-22T15:49:46.451 INFO:tasks.workunit.client.0.smithi016.stdout:7/845: symlink d3/dc/d1b/d5a/d59/d119/l133 0 2024-04-22T15:49:46.451 INFO:tasks.workunit.client.1.smithi027.stdout:4/53: creat d5/d7/f15 x:0 0 0 2024-04-22T15:49:46.455 INFO:tasks.workunit.client.1.smithi027.stdout:6/42: mkdir d5/db 0 2024-04-22T15:49:46.455 INFO:tasks.workunit.client.1.smithi027.stdout:1/60: dwrite d2/f3 [0,4194304] 0 2024-04-22T15:49:46.456 INFO:tasks.workunit.client.0.smithi016.stdout:8/814: truncate d2/d48/dad/d8f/da1/fc8 99655 0 2024-04-22T15:49:46.457 INFO:tasks.workunit.client.1.smithi027.stdout:1/61: read d2/f3 [1878722,122215] 0 2024-04-22T15:49:46.458 INFO:tasks.workunit.client.1.smithi027.stdout:4/54: dwrite f0 [0,4194304] 0 2024-04-22T15:49:46.460 INFO:tasks.workunit.client.1.smithi027.stdout:5/53: link d0/f8 d0/df/f12 0 2024-04-22T15:49:46.464 INFO:tasks.workunit.client.0.smithi016.stdout:2/864: rename d2/d2c/d97/lfa to d2/d1a/d7b/dcc/de5/d107/l147 0 2024-04-22T15:49:46.466 INFO:tasks.workunit.client.1.smithi027.stdout:0/106: link d0/ld d0/d1a/l24 0 2024-04-22T15:49:46.467 INFO:tasks.workunit.client.1.smithi027.stdout:1/62: dwrite d2/f3 [0,4194304] 0 2024-04-22T15:49:46.469 INFO:tasks.workunit.client.1.smithi027.stdout:1/63: write d2/fa [111883,57291] 0 2024-04-22T15:49:46.469 INFO:tasks.workunit.client.0.smithi016.stdout:4/861: getdents d7/d4e/d2f/d50 0 2024-04-22T15:49:46.470 INFO:tasks.workunit.client.1.smithi027.stdout:1/64: fdatasync d2/f6 0 2024-04-22T15:49:46.470 INFO:tasks.workunit.client.1.smithi027.stdout:1/65: chown d2/lb 47 1 2024-04-22T15:49:46.474 INFO:tasks.workunit.client.1.smithi027.stdout:9/83: symlink d0/l1d 0 2024-04-22T15:49:46.476 INFO:tasks.workunit.client.1.smithi027.stdout:8/66: symlink d1/l17 0 2024-04-22T15:49:46.477 INFO:tasks.workunit.client.1.smithi027.stdout:7/64: rename d0/dc to d0/d11 0 2024-04-22T15:49:46.479 INFO:tasks.workunit.client.1.smithi027.stdout:9/84: dread d0/d8/f11 [0,4194304] 0 2024-04-22T15:49:46.480 INFO:tasks.workunit.client.0.smithi016.stdout:7/846: mknod d3/dc/d33/c134 0 2024-04-22T15:49:46.480 INFO:tasks.workunit.client.1.smithi027.stdout:6/43: mkdir d5/dc 0 2024-04-22T15:49:46.481 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:46 smithi016 ceph-mon[26620]: pgmap v35: 65 pgs: 65 active+clean; 1.8 GiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 63 MiB/s rd, 165 MiB/s wr, 405 op/s 2024-04-22T15:49:46.483 INFO:tasks.workunit.client.1.smithi027.stdout:9/85: dread d0/d8/f11 [0,4194304] 0 2024-04-22T15:49:46.485 INFO:tasks.workunit.client.0.smithi016.stdout:5/882: dwrite df/d15/d1a/f53 [4194304,4194304] 0 2024-04-22T15:49:46.486 INFO:tasks.workunit.client.1.smithi027.stdout:5/54: rename d0/f5 to d0/df/de/f13 0 2024-04-22T15:49:46.488 INFO:tasks.workunit.client.0.smithi016.stdout:7/847: dread d3/dc/d9f/da1/fbb [0,4194304] 0 2024-04-22T15:49:46.490 INFO:tasks.workunit.client.0.smithi016.stdout:5/883: write df/f57 [2444393,54869] 0 2024-04-22T15:49:46.490 INFO:tasks.workunit.client.1.smithi027.stdout:0/107: chown d0/l19 56 1 2024-04-22T15:49:46.491 INFO:tasks.workunit.client.0.smithi016.stdout:5/884: chown df/d5b 1465775 1 2024-04-22T15:49:46.491 INFO:tasks.workunit.client.0.smithi016.stdout:3/918: dwrite d7/d17/d18/d27/fa3 [0,4194304] 0 2024-04-22T15:49:46.494 INFO:tasks.workunit.client.0.smithi016.stdout:3/919: chown d7/la7 15281859 1 2024-04-22T15:49:46.495 INFO:tasks.workunit.client.0.smithi016.stdout:0/961: creat dc/d13/f135 x:0 0 0 2024-04-22T15:49:46.495 INFO:tasks.workunit.client.0.smithi016.stdout:4/862: dread d7/d24/d51/fbb [0,4194304] 0 2024-04-22T15:49:46.496 INFO:tasks.workunit.client.0.smithi016.stdout:0/962: chown dc/d13/d5d/f6c 226229 1 2024-04-22T15:49:46.496 INFO:tasks.workunit.client.1.smithi027.stdout:1/66: mknod d2/ce 0 2024-04-22T15:49:46.497 INFO:tasks.workunit.client.0.smithi016.stdout:3/920: write d7/d17/d18/d32/d42/d63/d86/fdf [267365,130248] 0 2024-04-22T15:49:46.497 INFO:tasks.workunit.client.1.smithi027.stdout:1/67: fsync d2/f7 0 2024-04-22T15:49:46.499 INFO:tasks.workunit.client.0.smithi016.stdout:3/921: write d7/f15 [6532673,55726] 0 2024-04-22T15:49:46.502 INFO:tasks.workunit.client.1.smithi027.stdout:3/95: link f1d f22 0 2024-04-22T15:49:46.509 INFO:tasks.workunit.client.1.smithi027.stdout:3/96: dwrite f21 [0,4194304] 0 2024-04-22T15:49:46.509 INFO:tasks.workunit.client.1.smithi027.stdout:8/67: creat d1/f18 x:0 0 0 2024-04-22T15:49:46.511 INFO:tasks.workunit.client.0.smithi016.stdout:8/815: rename d2/d10/d47/lc3 to d2/d82/d52/df0/dc1/lfd 0 2024-04-22T15:49:46.512 INFO:tasks.workunit.client.1.smithi027.stdout:3/97: dread f16 [0,4194304] 0 2024-04-22T15:49:46.513 INFO:tasks.workunit.client.1.smithi027.stdout:7/65: mknod d0/c12 0 2024-04-22T15:49:46.519 INFO:tasks.workunit.client.0.smithi016.stdout:1/980: truncate d3/d28/f8b 2187307 0 2024-04-22T15:49:46.520 INFO:tasks.workunit.client.0.smithi016.stdout:7/848: unlink d3/dc/d1b/d5a/d4a/d6e/d115/f122 0 2024-04-22T15:49:46.521 INFO:tasks.workunit.client.1.smithi027.stdout:4/55: creat d5/de/f16 x:0 0 0 2024-04-22T15:49:46.522 INFO:tasks.workunit.client.0.smithi016.stdout:5/885: rmdir df/d15/d1a/d50/d77/d7d 39 2024-04-22T15:49:46.524 INFO:tasks.workunit.client.0.smithi016.stdout:4/863: creat d7/d21/d45/dd6/f11c x:0 0 0 2024-04-22T15:49:46.525 INFO:tasks.workunit.client.0.smithi016.stdout:0/963: symlink dc/d13/d14/de5/def/l136 0 2024-04-22T15:49:46.526 INFO:tasks.workunit.client.0.smithi016.stdout:6/909: truncate d3/da/db/d14/d4c/d58/f114 91006 0 2024-04-22T15:49:46.526 INFO:tasks.workunit.client.0.smithi016.stdout:4/864: write d7/d21/d45/d49/d67/fbd [1140914,7441] 0 2024-04-22T15:49:46.529 INFO:tasks.workunit.client.0.smithi016.stdout:3/922: truncate d7/d17/d18/d32/d3c/d51/f5f 884385 0 2024-04-22T15:49:46.530 INFO:tasks.workunit.client.0.smithi016.stdout:5/886: dwrite df/d5b/db3/f105 [0,4194304] 0 2024-04-22T15:49:46.531 INFO:tasks.workunit.client.1.smithi027.stdout:1/68: symlink d2/lf 0 2024-04-22T15:49:46.531 INFO:tasks.workunit.client.1.smithi027.stdout:2/62: sync 2024-04-22T15:49:46.534 INFO:tasks.workunit.client.0.smithi016.stdout:3/923: write d7/d17/d18/d27/ff3 [622446,31311] 0 2024-04-22T15:49:46.534 INFO:tasks.workunit.client.0.smithi016.stdout:5/887: chown df/d5b/d5d/deb 3592 1 2024-04-22T15:49:46.534 INFO:tasks.workunit.client.0.smithi016.stdout:2/865: rename d2/d2c/d97/d12e to d2/d2c/de7/d148 0 2024-04-22T15:49:46.537 INFO:tasks.workunit.client.0.smithi016.stdout:2/866: write d2/d51/f62 [3293415,89766] 0 2024-04-22T15:49:46.542 INFO:tasks.workunit.client.0.smithi016.stdout:7/849: fdatasync d3/dc/d1b/d5a/d9/df/d94/f3d 0 2024-04-22T15:49:46.542 INFO:tasks.workunit.client.0.smithi016.stdout:9/915: dwrite d9/de/d26/d5a/fb9 [0,4194304] 0 2024-04-22T15:49:46.542 INFO:tasks.workunit.client.1.smithi027.stdout:8/68: creat d1/d3/f19 x:0 0 0 2024-04-22T15:49:46.550 INFO:tasks.workunit.client.0.smithi016.stdout:9/916: dread d9/de/fec [0,4194304] 0 2024-04-22T15:49:46.551 INFO:tasks.workunit.client.1.smithi027.stdout:6/44: mkdir d5/dc/dd 0 2024-04-22T15:49:46.553 INFO:tasks.workunit.client.0.smithi016.stdout:4/865: creat d7/d21/d45/d49/f11d x:0 0 0 2024-04-22T15:49:46.555 INFO:tasks.workunit.client.1.smithi027.stdout:4/56: symlink d5/l17 0 2024-04-22T15:49:46.561 INFO:tasks.workunit.client.0.smithi016.stdout:3/924: rmdir d7/d17/d18/d32/d3c/d51/df1 39 2024-04-22T15:49:46.568 INFO:tasks.workunit.client.1.smithi027.stdout:0/108: symlink d0/l25 0 2024-04-22T15:49:46.568 INFO:tasks.workunit.client.1.smithi027.stdout:1/69: unlink d2/f7 0 2024-04-22T15:49:46.569 INFO:tasks.workunit.client.0.smithi016.stdout:8/816: write d2/d10/d47/fb2 [774361,82218] 0 2024-04-22T15:49:46.570 INFO:tasks.workunit.client.0.smithi016.stdout:8/817: chown d2/d82/d52/cd0 47101 1 2024-04-22T15:49:46.571 INFO:tasks.workunit.client.1.smithi027.stdout:2/63: creat dd/f11 x:0 0 0 2024-04-22T15:49:46.573 INFO:tasks.workunit.client.1.smithi027.stdout:8/69: creat d1/d3/f1a x:0 0 0 2024-04-22T15:49:46.574 INFO:tasks.workunit.client.1.smithi027.stdout:3/98: write f13 [897289,65000] 0 2024-04-22T15:49:46.578 INFO:tasks.workunit.client.0.smithi016.stdout:8/818: dwrite d2/d82/d52/df0/d66/d7e/ff5 [0,4194304] 0 2024-04-22T15:49:46.578 INFO:tasks.workunit.client.1.smithi027.stdout:2/64: dwrite dd/f11 [0,4194304] 0 2024-04-22T15:49:46.582 INFO:tasks.workunit.client.1.smithi027.stdout:2/65: dread dd/f11 [0,4194304] 0 2024-04-22T15:49:46.583 INFO:tasks.workunit.client.0.smithi016.stdout:6/910: getdents d3/da/ddd/d10e 0 2024-04-22T15:49:46.583 INFO:tasks.workunit.client.0.smithi016.stdout:5/888: write df/d5b/d70/fa4 [4804823,38591] 0 2024-04-22T15:49:46.585 INFO:tasks.workunit.client.0.smithi016.stdout:4/866: creat d7/d4e/d39/df3/f11e x:0 0 0 2024-04-22T15:49:46.587 INFO:tasks.workunit.client.1.smithi027.stdout:9/86: creat d0/dc/f1e x:0 0 0 2024-04-22T15:49:46.587 INFO:tasks.workunit.client.1.smithi027.stdout:2/66: dread f7 [0,4194304] 0 2024-04-22T15:49:46.587 INFO:tasks.workunit.client.1.smithi027.stdout:5/55: write d0/df/de/f13 [4973264,12411] 0 2024-04-22T15:49:46.589 INFO:tasks.workunit.client.0.smithi016.stdout:3/925: creat d7/d17/d18/d32/d42/d63/db4/f11d x:0 0 0 2024-04-22T15:49:46.594 INFO:tasks.workunit.client.1.smithi027.stdout:2/67: dwrite f7 [0,4194304] 0 2024-04-22T15:49:46.594 INFO:tasks.workunit.client.0.smithi016.stdout:2/867: mknod d2/d2c/d97/d67/c149 0 2024-04-22T15:49:46.597 INFO:tasks.workunit.client.1.smithi027.stdout:5/56: fdatasync d0/df/de/f13 0 2024-04-22T15:49:46.599 INFO:tasks.workunit.client.1.smithi027.stdout:7/66: truncate d0/f3 2653588 0 2024-04-22T15:49:46.602 INFO:tasks.workunit.client.0.smithi016.stdout:1/981: truncate d3/d4/d3a/d6b/dc3/ff0 2531683 0 2024-04-22T15:49:46.608 INFO:tasks.workunit.client.0.smithi016.stdout:7/850: dwrite d3/dc/d1b/d5a/d59/d64/f6c [0,4194304] 0 2024-04-22T15:49:46.610 INFO:tasks.workunit.client.0.smithi016.stdout:7/851: truncate d3/dc/f9a 1889612 0 2024-04-22T15:49:46.612 INFO:tasks.workunit.client.0.smithi016.stdout:1/982: dwrite d3/d2e/d9f/deb/f151 [0,4194304] 0 2024-04-22T15:49:46.613 INFO:tasks.workunit.client.0.smithi016.stdout:7/852: read - d3/dc/d1b/d31/f11d zero size 2024-04-22T15:49:46.613 INFO:tasks.workunit.client.0.smithi016.stdout:1/983: chown d3/d4/d19/d20/d2b/dd6/d12d/d116 33983 1 2024-04-22T15:49:46.614 INFO:tasks.workunit.client.0.smithi016.stdout:1/984: fsync d3/d4/d15/d38/d63/f109 0 2024-04-22T15:49:46.615 INFO:tasks.workunit.client.0.smithi016.stdout:7/853: truncate d3/dc/d33/d84/f106 1643082 0 2024-04-22T15:49:46.617 INFO:tasks.workunit.client.0.smithi016.stdout:4/867: symlink d7/d4e/d39/l11f 0 2024-04-22T15:49:46.622 INFO:tasks.workunit.client.0.smithi016.stdout:3/926: rmdir d7/d17/d18/d27 39 2024-04-22T15:49:46.623 INFO:tasks.workunit.client.0.smithi016.stdout:3/927: readlink d7/d17/d18/d32/d42/d63/deb/l34 0 2024-04-22T15:49:46.626 INFO:tasks.workunit.client.0.smithi016.stdout:0/964: rename dc/d13/d6f/d8b/ffe to dc/d13/d14/f137 0 2024-04-22T15:49:46.626 INFO:tasks.workunit.client.0.smithi016.stdout:8/819: write d2/d82/f7b [1403400,94107] 0 2024-04-22T15:49:46.627 INFO:tasks.workunit.client.0.smithi016.stdout:2/868: symlink d2/d2c/daa/d110/l14a 0 2024-04-22T15:49:46.628 INFO:tasks.workunit.client.0.smithi016.stdout:8/820: chown d2/d82/d52/df0/d66/l4e 53302641 1 2024-04-22T15:49:46.628 INFO:tasks.workunit.client.0.smithi016.stdout:2/869: chown d2/d23/ff6 585541 1 2024-04-22T15:49:46.635 INFO:tasks.workunit.client.0.smithi016.stdout:5/889: creat df/d15/d1a/d50/d77/d7d/d8b/d8c/ddf/f111 x:0 0 0 2024-04-22T15:49:46.639 INFO:tasks.workunit.client.0.smithi016.stdout:5/890: dread df/f76 [0,4194304] 0 2024-04-22T15:49:46.642 INFO:tasks.workunit.client.0.smithi016.stdout:3/928: fsync d7/d17/d18/d27/ff3 0 2024-04-22T15:49:46.643 INFO:tasks.workunit.client.0.smithi016.stdout:3/929: dread - d7/d17/d18/d32/f10b zero size 2024-04-22T15:49:46.645 INFO:tasks.workunit.client.0.smithi016.stdout:3/930: read d7/d17/d18/d32/d3c/d47/f48 [2113640,90312] 0 2024-04-22T15:49:46.645 INFO:tasks.workunit.client.0.smithi016.stdout:0/965: creat dc/d13/d14/de5/d65/d1f/dd4/df4/f138 x:0 0 0 2024-04-22T15:49:46.646 INFO:tasks.workunit.client.0.smithi016.stdout:0/966: stat dc/d13/d14/de5/d65/c23 0 2024-04-22T15:49:46.647 INFO:tasks.workunit.client.0.smithi016.stdout:3/931: dread d7/d17/d18/f70 [0,4194304] 0 2024-04-22T15:49:46.647 INFO:tasks.workunit.client.0.smithi016.stdout:0/967: readlink dc/d13/d22/la5 0 2024-04-22T15:49:46.647 INFO:tasks.workunit.client.0.smithi016.stdout:8/821: mknod d2/d82/d52/df0/d66/d7e/dc0/cfe 0 2024-04-22T15:49:46.648 INFO:tasks.workunit.client.0.smithi016.stdout:3/932: fdatasync d7/d17/d18/d32/d42/f101 0 2024-04-22T15:49:46.652 INFO:tasks.workunit.client.0.smithi016.stdout:1/985: mknod d3/d4/da/d7e/d104/c157 0 2024-04-22T15:49:46.654 INFO:tasks.workunit.client.0.smithi016.stdout:4/868: mkdir d7/d22/dc7/d10b/d120 0 2024-04-22T15:49:46.654 INFO:tasks.workunit.client.0.smithi016.stdout:0/968: dwrite dc/d13/d14/de5/d65/d40/d4a/d59/d78/fbc [0,4194304] 0 2024-04-22T15:49:46.654 INFO:tasks.workunit.client.0.smithi016.stdout:5/891: chown df/d15/d1a/d4e 1058268094 1 2024-04-22T15:49:46.660 INFO:tasks.workunit.client.0.smithi016.stdout:5/892: dwrite df/d15/d1a/fe5 [0,4194304] 0 2024-04-22T15:49:46.661 INFO:tasks.workunit.client.0.smithi016.stdout:8/822: fdatasync d2/d82/d52/df0/d66/ffb 0 2024-04-22T15:49:46.663 INFO:tasks.workunit.client.0.smithi016.stdout:5/893: write df/d15/d1a/d50/d77/f62 [2611267,70027] 0 2024-04-22T15:49:46.664 INFO:tasks.workunit.client.0.smithi016.stdout:8/823: write d2/d82/d52/df0/d66/d1c/f81 [1134337,105024] 0 2024-04-22T15:49:46.667 INFO:tasks.workunit.client.0.smithi016.stdout:7/854: creat d3/dc/d1b/d31/f135 x:0 0 0 2024-04-22T15:49:46.667 INFO:tasks.workunit.client.0.smithi016.stdout:5/894: stat df/d15/d1a/d4e/f72 0 2024-04-22T15:49:46.669 INFO:tasks.workunit.client.0.smithi016.stdout:1/986: write d3/d4/d15/fce [1007689,60838] 0 2024-04-22T15:49:46.669 INFO:tasks.workunit.client.0.smithi016.stdout:4/869: mkdir d7/d24/d41/d121 0 2024-04-22T15:49:46.670 INFO:tasks.workunit.client.0.smithi016.stdout:0/969: fsync dc/d13/d5d/fbe 0 2024-04-22T15:49:46.674 INFO:tasks.workunit.client.0.smithi016.stdout:2/870: creat d2/d1a/d7b/dcc/f14b x:0 0 0 2024-04-22T15:49:46.678 INFO:tasks.workunit.client.0.smithi016.stdout:6/911: getdents d3/da/dd/d67/d6c/db9 0 2024-04-22T15:49:46.678 INFO:tasks.workunit.client.0.smithi016.stdout:1/987: dwrite d3/d4/d3a/d91/fb5 [0,4194304] 0 2024-04-22T15:49:46.683 INFO:tasks.workunit.client.0.smithi016.stdout:3/933: mkdir d7/d17/d18/d32/d42/d63/db4/d11a/d11c/d11e 0 2024-04-22T15:49:46.686 INFO:tasks.workunit.client.0.smithi016.stdout:7/855: mkdir d3/d11a/d8c/d127/d136 0 2024-04-22T15:49:46.687 INFO:tasks.workunit.client.0.smithi016.stdout:7/856: write d3/de6/fec [981981,116894] 0 2024-04-22T15:49:46.693 INFO:tasks.workunit.client.0.smithi016.stdout:8/824: dread d2/d48/dad/d8f/fb8 [0,4194304] 0 2024-04-22T15:49:46.694 INFO:tasks.workunit.client.0.smithi016.stdout:8/825: chown d2/d82/d3e/lce 528265 1 2024-04-22T15:49:46.695 INFO:tasks.workunit.client.0.smithi016.stdout:4/870: mknod d7/d4e/d102/d106/d88/deb/c122 0 2024-04-22T15:49:46.697 INFO:tasks.workunit.client.0.smithi016.stdout:2/871: creat d2/d23/db8/f14c x:0 0 0 2024-04-22T15:49:46.704 INFO:tasks.workunit.client.0.smithi016.stdout:3/934: write d7/d17/d18/d32/d42/d63/f10e [325557,106554] 0 2024-04-22T15:49:46.704 INFO:tasks.workunit.client.0.smithi016.stdout:3/935: dread - d7/d77/f102 zero size 2024-04-22T15:49:46.706 INFO:tasks.workunit.client.0.smithi016.stdout:3/936: read - d7/d17/d18/d32/d3c/d47/dc3/ff8 zero size 2024-04-22T15:49:46.709 INFO:tasks.workunit.client.0.smithi016.stdout:7/857: dread d3/dc/d9f/da1/fc0 [0,4194304] 0 2024-04-22T15:49:46.711 INFO:tasks.workunit.client.0.smithi016.stdout:8/826: mknod d2/d48/cff 0 2024-04-22T15:49:46.714 INFO:tasks.workunit.client.0.smithi016.stdout:4/871: symlink d7/d21/d45/d49/d9b/db6/l123 0 2024-04-22T15:49:46.717 INFO:tasks.workunit.client.0.smithi016.stdout:2/872: rename d2/d1a/d7b/dc6/f132 to d2/de/d2a/f14d 0 2024-04-22T15:49:46.722 INFO:tasks.workunit.client.0.smithi016.stdout:6/912: symlink d3/da/d12/d13/d38/l12d 0 2024-04-22T15:49:46.722 INFO:tasks.workunit.client.0.smithi016.stdout:6/913: fdatasync d3/da/db/fce 0 2024-04-22T15:49:46.728 INFO:tasks.workunit.client.0.smithi016.stdout:8/827: chown d2/d82/d52/df0/d66/d79 1919 1 2024-04-22T15:49:46.730 INFO:tasks.workunit.client.0.smithi016.stdout:4/872: mkdir d7/d4e/d102/d106/d88/d124 0 2024-04-22T15:49:46.734 INFO:tasks.workunit.client.0.smithi016.stdout:7/858: rename d3/dc/d1b/d5a/d9/df/f11 to d3/dc/d1b/d5a/da4/f137 0 2024-04-22T15:49:46.736 INFO:tasks.workunit.client.0.smithi016.stdout:7/859: fsync d3/dc/d1b/d5a/d9/df/d94/d4c/d7a/f130 0 2024-04-22T15:49:46.736 INFO:tasks.workunit.client.0.smithi016.stdout:2/873: truncate d2/f7 1310536 0 2024-04-22T15:49:46.738 INFO:tasks.workunit.client.0.smithi016.stdout:7/860: write d3/dc/d1b/d5a/d9/df/fcc [1527046,11666] 0 2024-04-22T15:49:46.742 INFO:tasks.workunit.client.0.smithi016.stdout:6/914: read d3/da/d12/d13/d97/de0/fe3 [516752,21986] 0 2024-04-22T15:49:46.742 INFO:tasks.workunit.client.0.smithi016.stdout:5/895: getdents df/d15/d1a/d50/d77/d94 0 2024-04-22T15:49:46.742 INFO:tasks.workunit.client.0.smithi016.stdout:4/873: dread d7/d4e/f29 [0,4194304] 0 2024-04-22T15:49:46.747 INFO:tasks.workunit.client.0.smithi016.stdout:7/861: unlink d3/dc/d1b/d5a/d59/d64/dc9/f121 0 2024-04-22T15:49:46.748 INFO:tasks.workunit.client.0.smithi016.stdout:8/828: dwrite d2/fa2 [0,4194304] 0 2024-04-22T15:49:46.752 INFO:tasks.workunit.client.0.smithi016.stdout:6/915: unlink d3/da/ddd/d101/d11b/d83/f9f 0 2024-04-22T15:49:46.754 INFO:tasks.workunit.client.0.smithi016.stdout:5/896: truncate df/d15/d1a/f99 748319 0 2024-04-22T15:49:46.755 INFO:tasks.workunit.client.0.smithi016.stdout:0/970: truncate dc/d13/d14/de5/d65/d40/d4a/d59/f11b 159154 0 2024-04-22T15:49:46.757 INFO:tasks.workunit.client.0.smithi016.stdout:0/971: chown dc/d13/d14/d55/dba/f109 1703 1 2024-04-22T15:49:46.757 INFO:tasks.workunit.client.0.smithi016.stdout:9/917: sync 2024-04-22T15:49:46.758 INFO:tasks.workunit.client.0.smithi016.stdout:9/918: write d9/d20/f40 [2266285,59366] 0 2024-04-22T15:49:46.759 INFO:tasks.workunit.client.0.smithi016.stdout:9/919: chown l1 0 1 2024-04-22T15:49:46.761 INFO:tasks.workunit.client.0.smithi016.stdout:1/988: write d3/d4/d15/d38/d63/d67/fd0 [469942,92440] 0 2024-04-22T15:49:46.761 INFO:tasks.workunit.client.0.smithi016.stdout:8/829: creat d2/d82/d52/df0/d66/d7e/f100 x:0 0 0 2024-04-22T15:49:46.762 INFO:tasks.workunit.client.0.smithi016.stdout:3/937: write d7/d17/d18/f5b [6628716,67492] 0 2024-04-22T15:49:46.765 INFO:tasks.workunit.client.0.smithi016.stdout:6/916: rmdir d3/da/db/d14/d4c/d58 39 2024-04-22T15:49:46.769 INFO:tasks.workunit.client.0.smithi016.stdout:5/897: mknod df/d15/d1a/d50/d77/d7d/d8b/d93/db7/c112 0 2024-04-22T15:49:46.771 INFO:tasks.workunit.client.0.smithi016.stdout:7/862: symlink d3/d11f/l138 0 2024-04-22T15:49:46.773 INFO:tasks.workunit.client.0.smithi016.stdout:0/972: mkdir dc/d13/d14/de5/d65/d40/d4a/d59/d139 0 2024-04-22T15:49:46.774 INFO:tasks.workunit.client.0.smithi016.stdout:4/874: getdents d7/d3c/d96/dca 0 2024-04-22T15:49:46.776 INFO:tasks.workunit.client.0.smithi016.stdout:8/830: write d2/d30/f73 [2185860,24808] 0 2024-04-22T15:49:46.779 INFO:tasks.workunit.client.0.smithi016.stdout:1/989: unlink d3/d4/d19/d20/d2b/dd6/d12d/ff2 0 2024-04-22T15:49:46.783 INFO:tasks.workunit.client.0.smithi016.stdout:2/874: mknod d2/de/d2a/d59/c14e 0 2024-04-22T15:49:46.784 INFO:tasks.workunit.client.0.smithi016.stdout:2/875: write d2/d1a/d7b/dcc/f135 [1623294,130235] 0 2024-04-22T15:49:46.787 INFO:tasks.workunit.client.0.smithi016.stdout:3/938: dread d7/d17/fd0 [0,4194304] 0 2024-04-22T15:49:46.791 INFO:tasks.workunit.client.0.smithi016.stdout:0/973: mkdir dc/d13/d14/de5/d65/d57/dcb/de1/d112/d13a 0 2024-04-22T15:49:46.794 INFO:tasks.workunit.client.0.smithi016.stdout:8/831: symlink d2/d48/l101 0 2024-04-22T15:49:46.796 INFO:tasks.workunit.client.0.smithi016.stdout:1/990: mkdir d3/d4/d3a/d9e/d158 0 2024-04-22T15:49:46.804 INFO:tasks.workunit.client.1.smithi027.stdout:3/99: symlink l23 0 2024-04-22T15:49:46.806 INFO:tasks.workunit.client.1.smithi027.stdout:6/45: getdents d5/dc/dd 0 2024-04-22T15:49:46.807 INFO:tasks.workunit.client.0.smithi016.stdout:5/898: rename df/c104 to df/d15/d1a/d50/d77/d7d/d8b/d8c/d10c/c113 0 2024-04-22T15:49:46.808 INFO:tasks.workunit.client.0.smithi016.stdout:8/832: mkdir d2/d82/d52/df0/d6f/d102 0 2024-04-22T15:49:46.810 INFO:tasks.workunit.client.0.smithi016.stdout:6/917: write d3/da/dd/d67/d6c/db9/f106 [4844465,2813] 0 2024-04-22T15:49:46.811 INFO:tasks.workunit.client.0.smithi016.stdout:1/991: creat d3/d2e/d9f/f159 x:0 0 0 2024-04-22T15:49:46.812 INFO:tasks.workunit.client.0.smithi016.stdout:1/992: dread - d3/d4/da/d7e/f156 zero size 2024-04-22T15:49:46.814 INFO:tasks.workunit.client.1.smithi027.stdout:9/87: symlink d0/d8/l1f 0 2024-04-22T15:49:46.814 INFO:tasks.workunit.client.0.smithi016.stdout:5/899: read df/d15/d1a/d50/d77/d7d/d8b/d93/fd5 [1084015,56706] 0 2024-04-22T15:49:46.814 INFO:tasks.workunit.client.0.smithi016.stdout:6/918: write d3/da/db/d14/f86 [321016,93823] 0 2024-04-22T15:49:46.815 INFO:tasks.workunit.client.0.smithi016.stdout:7/863: write d3/d11a/d8c/fda [1004225,4316] 0 2024-04-22T15:49:46.815 INFO:tasks.workunit.client.1.smithi027.stdout:9/88: fsync d0/dc/f1e 0 2024-04-22T15:49:46.818 INFO:tasks.workunit.client.0.smithi016.stdout:7/864: dread - d3/dc/d1b/d5a/d9/df/d94/ddf/f102 zero size 2024-04-22T15:49:46.824 INFO:tasks.workunit.client.0.smithi016.stdout:6/919: dwrite d3/da/db/d14/d16/f3d [0,4194304] 0 2024-04-22T15:49:46.824 INFO:tasks.workunit.client.1.smithi027.stdout:7/67: creat d0/f13 x:0 0 0 2024-04-22T15:49:46.827 INFO:tasks.workunit.client.0.smithi016.stdout:6/920: write d3/da/dd/d7b/d12c/ff7 [89461,89520] 0 2024-04-22T15:49:46.828 INFO:tasks.workunit.client.0.smithi016.stdout:3/939: write d7/f22 [420135,87046] 0 2024-04-22T15:49:46.832 INFO:tasks.workunit.client.1.smithi027.stdout:4/57: truncate d5/d7/ff 241408 0 2024-04-22T15:49:46.834 INFO:tasks.workunit.client.1.smithi027.stdout:2/68: dwrite dd/f11 [4194304,4194304] 0 2024-04-22T15:49:46.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:46 smithi027 ceph-mon[34922]: pgmap v35: 65 pgs: 65 active+clean; 1.8 GiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 63 MiB/s rd, 165 MiB/s wr, 405 op/s 2024-04-22T15:49:46.836 INFO:tasks.workunit.client.0.smithi016.stdout:6/921: dread d3/da/dd/d1b/f85 [4194304,4194304] 0 2024-04-22T15:49:46.837 INFO:tasks.workunit.client.1.smithi027.stdout:3/100: mknod c24 0 2024-04-22T15:49:46.841 INFO:tasks.workunit.client.1.smithi027.stdout:6/46: creat d5/fe x:0 0 0 2024-04-22T15:49:46.844 INFO:tasks.workunit.client.1.smithi027.stdout:9/89: mknod d0/d8/c20 0 2024-04-22T15:49:46.846 INFO:tasks.workunit.client.1.smithi027.stdout:0/109: write d0/f10 [330446,16967] 0 2024-04-22T15:49:46.849 INFO:tasks.workunit.client.1.smithi027.stdout:5/57: creat d0/df/d10/f14 x:0 0 0 2024-04-22T15:49:46.852 INFO:tasks.workunit.client.1.smithi027.stdout:4/58: rename f3 to d5/d11/f18 0 2024-04-22T15:49:46.853 INFO:tasks.workunit.client.1.smithi027.stdout:4/59: fdatasync d5/f13 0 2024-04-22T15:49:46.854 INFO:tasks.workunit.client.1.smithi027.stdout:4/60: read - d5/de/f16 zero size 2024-04-22T15:49:46.855 INFO:tasks.workunit.client.1.smithi027.stdout:4/61: truncate d5/d7/f10 229154 0 2024-04-22T15:49:46.858 INFO:tasks.workunit.client.1.smithi027.stdout:2/69: fsync f7 0 2024-04-22T15:49:46.858 INFO:tasks.workunit.client.1.smithi027.stdout:2/70: fsync f7 0 2024-04-22T15:49:46.860 INFO:tasks.workunit.client.1.smithi027.stdout:8/70: link d1/d3/c15 d1/c1b 0 2024-04-22T15:49:46.861 INFO:tasks.workunit.client.1.smithi027.stdout:2/71: dread f7 [0,4194304] 0 2024-04-22T15:49:46.867 INFO:tasks.workunit.client.1.smithi027.stdout:3/101: dwrite f16 [0,4194304] 0 2024-04-22T15:49:46.874 INFO:tasks.workunit.client.1.smithi027.stdout:5/58: unlink d0/l2 0 2024-04-22T15:49:46.882 INFO:tasks.workunit.client.1.smithi027.stdout:4/62: symlink d5/d11/l19 0 2024-04-22T15:49:46.882 INFO:tasks.workunit.client.1.smithi027.stdout:4/63: write f0 [3404689,79467] 0 2024-04-22T15:49:46.885 INFO:tasks.workunit.client.1.smithi027.stdout:8/71: mkdir d1/d3/d1c 0 2024-04-22T15:49:46.885 INFO:tasks.workunit.client.1.smithi027.stdout:8/72: dread - d1/f18 zero size 2024-04-22T15:49:46.890 INFO:tasks.workunit.client.1.smithi027.stdout:1/70: write d2/f8 [426612,12137] 0 2024-04-22T15:49:46.892 INFO:tasks.workunit.client.1.smithi027.stdout:9/90: dwrite d0/fb [4194304,4194304] 0 2024-04-22T15:49:46.893 INFO:tasks.workunit.client.1.smithi027.stdout:3/102: symlink l25 0 2024-04-22T15:49:46.898 INFO:tasks.workunit.client.1.smithi027.stdout:7/68: write d0/d11/fe [3273996,73353] 0 2024-04-22T15:49:46.899 INFO:tasks.workunit.client.0.smithi016.stdout:9/920: sync 2024-04-22T15:49:46.900 INFO:tasks.workunit.client.0.smithi016.stdout:9/921: dread - d9/de/d23/f105 zero size 2024-04-22T15:49:46.901 INFO:tasks.workunit.client.1.smithi027.stdout:6/47: rename d5/fe to d5/d9/ff 0 2024-04-22T15:49:46.904 INFO:tasks.workunit.client.1.smithi027.stdout:4/64: mknod d5/de/c1a 0 2024-04-22T15:49:46.906 INFO:tasks.workunit.client.1.smithi027.stdout:2/72: creat dd/df/f12 x:0 0 0 2024-04-22T15:49:46.907 INFO:tasks.workunit.client.1.smithi027.stdout:4/65: dread d5/d11/f18 [0,4194304] 0 2024-04-22T15:49:46.910 INFO:tasks.workunit.client.1.smithi027.stdout:9/91: mknod d0/d8/c21 0 2024-04-22T15:49:46.912 INFO:tasks.workunit.client.1.smithi027.stdout:5/59: getdents d0/d11 0 2024-04-22T15:49:46.915 INFO:tasks.workunit.client.0.smithi016.stdout:2/876: rename d2/d4d/ce3 to d2/d2c/d97/d34/c14f 0 2024-04-22T15:49:46.920 INFO:tasks.workunit.client.0.smithi016.stdout:8/833: symlink d2/d82/d52/l103 0 2024-04-22T15:49:46.920 INFO:tasks.workunit.client.0.smithi016.stdout:8/834: write d2/d10/f39 [5238448,67011] 0 2024-04-22T15:49:46.925 INFO:tasks.workunit.client.0.smithi016.stdout:1/993: symlink d3/d2e/l15a 0 2024-04-22T15:49:46.928 INFO:tasks.workunit.client.0.smithi016.stdout:8/835: dread d2/d48/dad/d6a/f9a [0,4194304] 0 2024-04-22T15:49:46.929 INFO:tasks.workunit.client.0.smithi016.stdout:8/836: fsync d2/d82/f38 0 2024-04-22T15:49:46.929 INFO:tasks.workunit.client.0.smithi016.stdout:7/865: symlink d3/dc/d1b/d5a/d9/df/d94/ddf/l139 0 2024-04-22T15:49:46.930 INFO:tasks.workunit.client.1.smithi027.stdout:6/48: creat d5/f10 x:0 0 0 2024-04-22T15:49:46.930 INFO:tasks.workunit.client.0.smithi016.stdout:8/837: chown d2/d48/dad/d8f/d96/de4 8941 1 2024-04-22T15:49:46.930 INFO:tasks.workunit.client.1.smithi027.stdout:0/110: getdents d0 0 2024-04-22T15:49:46.932 INFO:tasks.workunit.client.0.smithi016.stdout:9/922: symlink d9/d37/dea/l128 0 2024-04-22T15:49:46.932 INFO:tasks.workunit.client.1.smithi027.stdout:1/71: chown d2/fa 1090200062 1 2024-04-22T15:49:46.939 INFO:tasks.workunit.client.0.smithi016.stdout:5/900: creat df/d5b/d5d/deb/f114 x:0 0 0 2024-04-22T15:49:46.939 INFO:tasks.workunit.client.1.smithi027.stdout:6/49: dwrite d5/f10 [0,4194304] 0 2024-04-22T15:49:46.940 INFO:tasks.workunit.client.1.smithi027.stdout:0/111: dwrite d0/d1a/f1e [0,4194304] 0 2024-04-22T15:49:46.941 INFO:tasks.workunit.client.1.smithi027.stdout:6/50: chown d5/f10 733552 1 2024-04-22T15:49:46.942 INFO:tasks.workunit.client.1.smithi027.stdout:3/103: write f1d [374228,19927] 0 2024-04-22T15:49:46.944 INFO:tasks.workunit.client.0.smithi016.stdout:3/940: mknod d7/d17/d18/d32/d42/d63/db4/d11a/d11c/d11e/c11f 0 2024-04-22T15:49:46.946 INFO:tasks.workunit.client.1.smithi027.stdout:5/60: truncate d0/df/de/f13 2832523 0 2024-04-22T15:49:46.948 INFO:tasks.workunit.client.1.smithi027.stdout:8/73: write d1/d3/fa [3062249,123721] 0 2024-04-22T15:49:46.950 INFO:tasks.workunit.client.0.smithi016.stdout:8/838: write d2/d82/d3e/dd5/fd8 [3279838,33042] 0 2024-04-22T15:49:46.953 INFO:tasks.workunit.client.0.smithi016.stdout:9/923: symlink d9/de/d23/ddb/def/l129 0 2024-04-22T15:49:46.954 INFO:tasks.workunit.client.1.smithi027.stdout:3/104: dread f1d [0,4194304] 0 2024-04-22T15:49:46.956 INFO:tasks.workunit.client.0.smithi016.stdout:2/877: truncate d2/d2c/d97/f31 2624171 0 2024-04-22T15:49:46.956 INFO:tasks.workunit.client.1.smithi027.stdout:3/105: write f13 [1813618,87966] 0 2024-04-22T15:49:46.957 INFO:tasks.workunit.client.1.smithi027.stdout:9/92: creat d0/dc/f22 x:0 0 0 2024-04-22T15:49:46.957 INFO:tasks.workunit.client.1.smithi027.stdout:8/74: dwrite d1/d3/f10 [0,4194304] 0 2024-04-22T15:49:46.958 INFO:tasks.workunit.client.1.smithi027.stdout:3/106: chown f1e 132 1 2024-04-22T15:49:46.959 INFO:tasks.workunit.client.0.smithi016.stdout:1/994: mknod d3/d4/d3a/d6b/c15b 0 2024-04-22T15:49:46.959 INFO:tasks.workunit.client.1.smithi027.stdout:8/75: dread - d1/d3/f13 zero size 2024-04-22T15:49:46.962 INFO:tasks.workunit.client.1.smithi027.stdout:2/73: truncate dd/f11 5785501 0 2024-04-22T15:49:46.964 INFO:tasks.workunit.client.1.smithi027.stdout:3/107: dread f16 [0,4194304] 0 2024-04-22T15:49:46.964 INFO:tasks.workunit.client.1.smithi027.stdout:3/108: fdatasync ff 0 2024-04-22T15:49:46.965 INFO:tasks.workunit.client.0.smithi016.stdout:3/941: mkdir d7/d17/d18/d32/d42/d63/d71/d120 0 2024-04-22T15:49:46.969 INFO:tasks.workunit.client.0.smithi016.stdout:3/942: stat d7/d17/d18/d32/d42/d63/deb/d21/d55/f6f 0 2024-04-22T15:49:46.969 INFO:tasks.workunit.client.0.smithi016.stdout:6/922: creat d3/da/f12e x:0 0 0 2024-04-22T15:49:46.974 INFO:tasks.workunit.client.0.smithi016.stdout:2/878: write d2/f5 [3218044,43802] 0 2024-04-22T15:49:46.983 INFO:tasks.workunit.client.1.smithi027.stdout:6/51: dwrite f0 [0,4194304] 0 2024-04-22T15:49:46.983 INFO:tasks.workunit.client.1.smithi027.stdout:2/74: creat dd/f13 x:0 0 0 2024-04-22T15:49:46.984 INFO:tasks.workunit.client.0.smithi016.stdout:9/924: creat d9/d20/dac/dae/d126/f12a x:0 0 0 2024-04-22T15:49:46.988 INFO:tasks.workunit.client.1.smithi027.stdout:5/61: dwrite d0/df/de/f13 [0,4194304] 0 2024-04-22T15:49:46.989 INFO:tasks.workunit.client.1.smithi027.stdout:5/62: truncate d0/df/f12 1867513 0 2024-04-22T15:49:46.994 INFO:tasks.workunit.client.1.smithi027.stdout:8/76: mknod d1/d3/d1c/c1d 0 2024-04-22T15:49:46.996 INFO:tasks.workunit.client.0.smithi016.stdout:9/925: symlink d9/d57/d91/de4/d25/d36/l12b 0 2024-04-22T15:49:46.997 INFO:tasks.workunit.client.1.smithi027.stdout:2/75: mkdir dd/d14 0 2024-04-22T15:49:46.997 INFO:tasks.workunit.client.0.smithi016.stdout:9/926: chown d9/d57/da7/daa/lf4 2 1 2024-04-22T15:49:46.998 INFO:tasks.workunit.client.1.smithi027.stdout:2/76: write f7 [2775998,117140] 0 2024-04-22T15:49:47.000 INFO:tasks.workunit.client.0.smithi016.stdout:1/995: write d3/f4f [2465634,69840] 0 2024-04-22T15:49:47.001 INFO:tasks.workunit.client.0.smithi016.stdout:0/974: sync 2024-04-22T15:49:47.001 INFO:tasks.workunit.client.0.smithi016.stdout:4/875: sync 2024-04-22T15:49:47.004 INFO:tasks.workunit.client.0.smithi016.stdout:5/901: getdents df/d15/d1a 0 2024-04-22T15:49:47.005 INFO:tasks.workunit.client.1.smithi027.stdout:8/77: mkdir d1/d1e 0 2024-04-22T15:49:47.006 INFO:tasks.workunit.client.0.smithi016.stdout:8/839: getdents d2/d82/d52/df0/d66/d1c/d65 0 2024-04-22T15:49:47.007 INFO:tasks.workunit.client.0.smithi016.stdout:6/923: rename d3/da/dd/f29 to d3/da/dd/d1b/f12f 0 2024-04-22T15:49:47.008 INFO:tasks.workunit.client.1.smithi027.stdout:6/52: creat d5/db/f11 x:0 0 0 2024-04-22T15:49:47.009 INFO:tasks.workunit.client.0.smithi016.stdout:2/879: dwrite d2/d2c/d97/d34/d60/f6f [0,4194304] 0 2024-04-22T15:49:47.009 INFO:tasks.workunit.client.0.smithi016.stdout:9/927: creat d9/d57/d91/dc1/f12c x:0 0 0 2024-04-22T15:49:47.010 INFO:tasks.workunit.client.1.smithi027.stdout:1/72: link d2/c9 d2/c10 0 2024-04-22T15:49:47.010 INFO:tasks.workunit.client.1.smithi027.stdout:8/78: dread d1/fb [0,4194304] 0 2024-04-22T15:49:47.011 INFO:tasks.workunit.client.0.smithi016.stdout:9/928: chown d9/l1e 238 1 2024-04-22T15:49:47.011 INFO:tasks.workunit.client.1.smithi027.stdout:8/79: write d1/f18 [9367,110900] 0 2024-04-22T15:49:47.012 INFO:tasks.workunit.client.1.smithi027.stdout:5/63: link d0/f7 d0/f15 0 2024-04-22T15:49:47.016 INFO:tasks.workunit.client.1.smithi027.stdout:8/80: dread d1/fb [0,4194304] 0 2024-04-22T15:49:47.017 INFO:tasks.workunit.client.1.smithi027.stdout:6/53: rename f0 to d5/d8/f12 0 2024-04-22T15:49:47.018 INFO:tasks.workunit.client.1.smithi027.stdout:6/54: write d5/f10 [2584840,2925] 0 2024-04-22T15:49:47.019 INFO:tasks.workunit.client.0.smithi016.stdout:0/975: rmdir dc/d13/d14/de5/d65/d40/d4a/d59/d98 39 2024-04-22T15:49:47.020 INFO:tasks.workunit.client.0.smithi016.stdout:8/840: dwrite d2/d48/dad/d8f/d96/fd4 [0,4194304] 0 2024-04-22T15:49:47.023 INFO:tasks.workunit.client.0.smithi016.stdout:8/841: chown d2/d82/d52/df0 5869549 1 2024-04-22T15:49:47.027 INFO:tasks.workunit.client.1.smithi027.stdout:8/81: mknod d1/d3/d1c/c1f 0 2024-04-22T15:49:47.028 INFO:tasks.workunit.client.1.smithi027.stdout:8/82: truncate d1/f7 1278345 0 2024-04-22T15:49:47.029 INFO:tasks.workunit.client.0.smithi016.stdout:2/880: read d2/d2c/d97/d67/dbf/fba [271513,28114] 0 2024-04-22T15:49:47.030 INFO:tasks.workunit.client.0.smithi016.stdout:9/929: mknod d9/d57/d91/de4/d25/d36/d116/c12d 0 2024-04-22T15:49:47.034 INFO:tasks.workunit.client.1.smithi027.stdout:6/55: mknod d5/d9/c13 0 2024-04-22T15:49:47.034 INFO:tasks.workunit.client.1.smithi027.stdout:6/56: fsync d5/db/f11 0 2024-04-22T15:49:47.035 INFO:tasks.workunit.client.0.smithi016.stdout:0/976: creat dc/d13/d14/de5/d65/d40/dd6/f13b x:0 0 0 2024-04-22T15:49:47.038 INFO:tasks.workunit.client.1.smithi027.stdout:5/64: creat d0/d11/f16 x:0 0 0 2024-04-22T15:49:47.039 INFO:tasks.workunit.client.1.smithi027.stdout:5/65: read d0/f3 [167566,80904] 0 2024-04-22T15:49:47.041 INFO:tasks.workunit.client.1.smithi027.stdout:8/83: creat d1/d3/d1c/f20 x:0 0 0 2024-04-22T15:49:47.041 INFO:tasks.workunit.client.0.smithi016.stdout:6/924: symlink d3/da/db/l130 0 2024-04-22T15:49:47.041 INFO:tasks.workunit.client.0.smithi016.stdout:0/977: read dc/d13/d14/de5/d65/d57/fca [1796938,104016] 0 2024-04-22T15:49:47.043 INFO:tasks.workunit.client.1.smithi027.stdout:6/57: write d5/d8/f12 [8754464,7042] 0 2024-04-22T15:49:47.044 INFO:tasks.workunit.client.1.smithi027.stdout:6/58: chown d5/l7 15301926 1 2024-04-22T15:49:47.049 INFO:tasks.workunit.client.1.smithi027.stdout:5/66: symlink d0/l17 0 2024-04-22T15:49:47.050 INFO:tasks.workunit.client.0.smithi016.stdout:5/902: dread df/d15/d1a/f78 [0,4194304] 0 2024-04-22T15:49:47.050 INFO:tasks.workunit.client.0.smithi016.stdout:1/996: dwrite d3/d4/d45/f110 [0,4194304] 0 2024-04-22T15:49:47.051 INFO:tasks.workunit.client.1.smithi027.stdout:5/67: stat d0/df/de 0 2024-04-22T15:49:47.051 INFO:tasks.workunit.client.1.smithi027.stdout:6/59: dwrite d5/d8/f12 [0,4194304] 0 2024-04-22T15:49:47.052 INFO:tasks.workunit.client.1.smithi027.stdout:5/68: readlink d0/df/lb 0 2024-04-22T15:49:47.053 INFO:tasks.workunit.client.1.smithi027.stdout:8/84: rename d1/d3/d1c/f20 to d1/f21 0 2024-04-22T15:49:47.054 INFO:tasks.workunit.client.0.smithi016.stdout:8/842: unlink d2/d10/d43/dbe/cdf 0 2024-04-22T15:49:47.055 INFO:tasks.workunit.client.1.smithi027.stdout:6/60: dread d5/f10 [0,4194304] 0 2024-04-22T15:49:47.056 INFO:tasks.workunit.client.0.smithi016.stdout:1/997: chown d3/d4/da/f26 28446 1 2024-04-22T15:49:47.057 INFO:tasks.workunit.client.1.smithi027.stdout:1/73: rename d2/fc to d2/f11 0 2024-04-22T15:49:47.059 INFO:tasks.workunit.client.0.smithi016.stdout:4/876: truncate d7/d24/d51/fbb 89333 0 2024-04-22T15:49:47.059 INFO:tasks.workunit.client.0.smithi016.stdout:6/925: dwrite d3/da/dd/d67/d6c/ff6 [0,4194304] 0 2024-04-22T15:49:47.059 INFO:tasks.workunit.client.0.smithi016.stdout:8/843: read - d2/d82/d3e/d8e/ff6 zero size 2024-04-22T15:49:47.061 INFO:tasks.workunit.client.0.smithi016.stdout:9/930: mkdir d9/de/d23/d12e 0 2024-04-22T15:49:47.062 INFO:tasks.workunit.client.1.smithi027.stdout:6/61: dwrite d5/db/f11 [0,4194304] 0 2024-04-22T15:49:47.062 INFO:tasks.workunit.client.0.smithi016.stdout:4/877: fdatasync d7/d22/fcd 0 2024-04-22T15:49:47.064 INFO:tasks.workunit.client.1.smithi027.stdout:6/62: write d5/d8/f12 [8054487,18831] 0 2024-04-22T15:49:47.065 INFO:tasks.workunit.client.0.smithi016.stdout:0/978: mknod dc/d13/d14/de5/dd1/ddc/c13c 0 2024-04-22T15:49:47.069 INFO:tasks.workunit.client.1.smithi027.stdout:5/69: creat d0/df/de/f18 x:0 0 0 2024-04-22T15:49:47.074 INFO:tasks.workunit.client.0.smithi016.stdout:7/866: sync 2024-04-22T15:49:47.074 INFO:tasks.workunit.client.0.smithi016.stdout:5/903: mkdir df/d90/dad/d115 0 2024-04-22T15:49:47.078 INFO:tasks.workunit.client.0.smithi016.stdout:9/931: unlink d9/de/d26/l47 0 2024-04-22T15:49:47.082 INFO:tasks.workunit.client.0.smithi016.stdout:6/926: truncate d3/da/fc3 41025 0 2024-04-22T15:49:47.084 INFO:tasks.workunit.client.1.smithi027.stdout:5/70: rename d0/lc to d0/df/l19 0 2024-04-22T15:49:47.084 INFO:tasks.workunit.client.0.smithi016.stdout:8/844: symlink d2/d82/d52/df0/d66/d7e/l104 0 2024-04-22T15:49:47.084 INFO:tasks.workunit.client.1.smithi027.stdout:1/74: dread d2/f8 [0,4194304] 0 2024-04-22T15:49:47.085 INFO:tasks.workunit.client.1.smithi027.stdout:5/71: stat d0/df/f12 0 2024-04-22T15:49:47.085 INFO:tasks.workunit.client.0.smithi016.stdout:2/881: dwrite d2/f7 [0,4194304] 0 2024-04-22T15:49:47.085 INFO:tasks.workunit.client.1.smithi027.stdout:5/72: dread - d0/df/d10/f14 zero size 2024-04-22T15:49:47.086 INFO:tasks.workunit.client.0.smithi016.stdout:8/845: read d2/d82/d56/f71 [208321,87253] 0 2024-04-22T15:49:47.086 INFO:tasks.workunit.client.1.smithi027.stdout:5/73: chown d0/df/lb 1322520 1 2024-04-22T15:49:47.087 INFO:tasks.workunit.client.1.smithi027.stdout:1/75: dread d2/f8 [0,4194304] 0 2024-04-22T15:49:47.087 INFO:tasks.workunit.client.1.smithi027.stdout:5/74: chown d0/f6 7871432 1 2024-04-22T15:49:47.088 INFO:tasks.workunit.client.0.smithi016.stdout:0/979: unlink dc/d13/d14/de5/d65/d40/d4a/d59/d78/dbd/fda 0 2024-04-22T15:49:47.088 INFO:tasks.workunit.client.0.smithi016.stdout:0/980: chown dc/d13/d99/fc4 544 1 2024-04-22T15:49:47.092 INFO:tasks.workunit.client.0.smithi016.stdout:0/981: truncate dc/d13/d14/de5/d65/fec 4533528 0 2024-04-22T15:49:47.095 INFO:tasks.workunit.client.0.smithi016.stdout:9/932: creat d9/d5d/db8/f12f x:0 0 0 2024-04-22T15:49:47.096 INFO:tasks.workunit.client.0.smithi016.stdout:9/933: chown d9/d57/da7/daa/lf4 3 1 2024-04-22T15:49:47.096 INFO:tasks.workunit.client.1.smithi027.stdout:5/75: dwrite d0/df/de/f13 [0,4194304] 0 2024-04-22T15:49:47.101 INFO:tasks.workunit.client.0.smithi016.stdout:7/867: rename d3/dc/d1b/d5a/d9/df/fcc to d3/dc/d1b/d5a/d9/d109/f13a 0 2024-04-22T15:49:47.101 INFO:tasks.workunit.client.0.smithi016.stdout:3/943: sync 2024-04-22T15:49:47.107 INFO:tasks.workunit.client.0.smithi016.stdout:8/846: mknod d2/d10/d43/c105 0 2024-04-22T15:49:47.107 INFO:tasks.workunit.client.0.smithi016.stdout:9/934: dwrite d9/de/f43 [0,4194304] 0 2024-04-22T15:49:47.113 INFO:tasks.workunit.client.0.smithi016.stdout:1/998: dread d3/d4/d15/d38/f141 [0,4194304] 0 2024-04-22T15:49:47.115 INFO:tasks.workunit.client.0.smithi016.stdout:1/999: stat d3/f142 0 2024-04-22T15:49:47.116 INFO:tasks.workunit.client.1.smithi027.stdout:8/85: dwrite d1/f7 [0,4194304] 0 2024-04-22T15:49:47.117 INFO:tasks.workunit.client.0.smithi016.stdout:7/868: mkdir d3/d11a/d8c/d13b 0 2024-04-22T15:49:47.118 INFO:tasks.workunit.client.0.smithi016.stdout:7/869: chown d3/dc/d1b/d5a/f56 16374369 1 2024-04-22T15:49:47.118 INFO:tasks.workunit.client.0.smithi016.stdout:6/927: dread d3/da/dd/d1b/f62 [0,4194304] 0 2024-04-22T15:49:47.123 INFO:tasks.workunit.client.1.smithi027.stdout:6/63: write d5/d9/ff [49270,111415] 0 2024-04-22T15:49:47.126 INFO:tasks.workunit.client.0.smithi016.stdout:3/944: unlink d7/d17/d18/d32/d42/c107 0 2024-04-22T15:49:47.128 INFO:tasks.workunit.client.1.smithi027.stdout:8/86: creat d1/d1e/f22 x:0 0 0 2024-04-22T15:49:47.131 INFO:tasks.workunit.client.1.smithi027.stdout:8/87: dread d1/f18 [0,4194304] 0 2024-04-22T15:49:47.132 INFO:tasks.workunit.client.0.smithi016.stdout:8/847: truncate d2/d82/d56/f90 606542 0 2024-04-22T15:49:47.132 INFO:tasks.workunit.client.1.smithi027.stdout:6/64: dwrite d5/d8/f12 [4194304,4194304] 0 2024-04-22T15:49:47.133 INFO:tasks.workunit.client.1.smithi027.stdout:8/88: readlink d1/d3/l8 0 2024-04-22T15:49:47.133 INFO:tasks.workunit.client.1.smithi027.stdout:6/65: chown d5/db/f11 4 1 2024-04-22T15:49:47.136 INFO:tasks.workunit.client.0.smithi016.stdout:7/870: fdatasync d3/dc/d1b/d5a/f56 0 2024-04-22T15:49:47.137 INFO:tasks.workunit.client.1.smithi027.stdout:8/89: mknod d1/d3/c23 0 2024-04-22T15:49:47.140 INFO:tasks.workunit.client.0.smithi016.stdout:5/904: getdents df/d15/d1a/d50/d77 0 2024-04-22T15:49:47.141 INFO:tasks.workunit.client.1.smithi027.stdout:6/66: symlink d5/db/l14 0 2024-04-22T15:49:47.141 INFO:tasks.workunit.client.0.smithi016.stdout:2/882: write d2/d23/d92/d55/fa6 [1463850,52381] 0 2024-04-22T15:49:47.142 INFO:tasks.workunit.client.0.smithi016.stdout:2/883: stat d2/d1a/d7b/dcc/l72 0 2024-04-22T15:49:47.143 INFO:tasks.workunit.client.1.smithi027.stdout:8/90: symlink d1/d1e/l24 0 2024-04-22T15:49:47.145 INFO:tasks.workunit.client.0.smithi016.stdout:9/935: creat d9/d5d/db8/d122/f130 x:0 0 0 2024-04-22T15:49:47.145 INFO:tasks.workunit.client.0.smithi016.stdout:0/982: write dc/d13/d6f/d8b/f9a [1176885,37269] 0 2024-04-22T15:49:47.149 INFO:tasks.workunit.client.0.smithi016.stdout:8/848: rmdir d2/d82/d3e/dd5 39 2024-04-22T15:49:47.149 INFO:tasks.workunit.client.1.smithi027.stdout:8/91: mknod d1/d3/c25 0 2024-04-22T15:49:47.150 INFO:tasks.workunit.client.1.smithi027.stdout:5/76: dwrite d0/f15 [0,4194304] 0 2024-04-22T15:49:47.150 INFO:tasks.workunit.client.1.smithi027.stdout:5/77: write d0/f7 [2093610,104087] 0 2024-04-22T15:49:47.155 INFO:tasks.workunit.client.1.smithi027.stdout:8/92: dread d1/d3/f10 [0,4194304] 0 2024-04-22T15:49:47.159 INFO:tasks.workunit.client.1.smithi027.stdout:8/93: dread f0 [0,4194304] 0 2024-04-22T15:49:47.160 INFO:tasks.workunit.client.1.smithi027.stdout:8/94: fdatasync d1/d3/f6 0 2024-04-22T15:49:47.164 INFO:tasks.workunit.client.1.smithi027.stdout:5/78: dread d0/f6 [0,4194304] 0 2024-04-22T15:49:47.165 INFO:tasks.workunit.client.0.smithi016.stdout:9/936: creat d9/de/d26/d5a/f131 x:0 0 0 2024-04-22T15:49:47.167 INFO:tasks.workunit.client.0.smithi016.stdout:0/983: fsync dc/d13/d14/de5/d65/d40/d4a/d59/d98/f115 0 2024-04-22T15:49:47.168 INFO:tasks.workunit.client.1.smithi027.stdout:5/79: dread d0/df/de/f13 [0,4194304] 0 2024-04-22T15:49:47.169 INFO:tasks.workunit.client.0.smithi016.stdout:7/871: dread d3/dc/d1b/d5a/da4/fc4 [0,4194304] 0 2024-04-22T15:49:47.169 INFO:tasks.workunit.client.0.smithi016.stdout:8/849: chown d2/d82/d52/df0/c8a 1216239 1 2024-04-22T15:49:47.170 INFO:tasks.workunit.client.1.smithi027.stdout:8/95: creat d1/d3/f26 x:0 0 0 2024-04-22T15:49:47.171 INFO:tasks.workunit.client.0.smithi016.stdout:4/878: link d7/d21/d45/c7f d7/d4e/d39/dc3/c125 0 2024-04-22T15:49:47.172 INFO:tasks.workunit.client.0.smithi016.stdout:4/879: chown d7/d4e/d2f/d33/c5c 2212 1 2024-04-22T15:49:47.175 INFO:tasks.workunit.client.0.smithi016.stdout:5/905: symlink df/d15/d1a/d4e/d58/l116 0 2024-04-22T15:49:47.175 INFO:tasks.workunit.client.0.smithi016.stdout:5/906: dread - df/d15/d1a/d4e/f10e zero size 2024-04-22T15:49:47.176 INFO:tasks.workunit.client.1.smithi027.stdout:5/80: unlink d0/df/lb 0 2024-04-22T15:49:47.178 INFO:tasks.workunit.client.1.smithi027.stdout:8/96: write d1/f21 [425671,5892] 0 2024-04-22T15:49:47.179 INFO:tasks.workunit.client.1.smithi027.stdout:6/67: truncate d5/db/f11 1658832 0 2024-04-22T15:49:47.179 INFO:tasks.workunit.client.1.smithi027.stdout:8/97: chown d1/fd 1012 1 2024-04-22T15:49:47.182 INFO:tasks.workunit.client.0.smithi016.stdout:9/937: write d9/d57/d91/de4/d25/da6/f5b [2245060,52181] 0 2024-04-22T15:49:47.182 INFO:tasks.workunit.client.1.smithi027.stdout:5/81: mkdir d0/d1a 0 2024-04-22T15:49:47.183 INFO:tasks.workunit.client.0.smithi016.stdout:0/984: dread dc/d13/d14/de5/d65/d1f/f53 [4194304,4194304] 0 2024-04-22T15:49:47.184 INFO:tasks.workunit.client.1.smithi027.stdout:5/82: dread - d0/df/d10/f14 zero size 2024-04-22T15:49:47.185 INFO:tasks.workunit.client.0.smithi016.stdout:0/985: fsync dc/d13/d14/de5/d65/d57/dcb/fd7 0 2024-04-22T15:49:47.187 INFO:tasks.workunit.client.1.smithi027.stdout:8/98: creat d1/d3/f27 x:0 0 0 2024-04-22T15:49:47.189 INFO:tasks.workunit.client.0.smithi016.stdout:2/884: dwrite d2/d2c/d97/f6a [0,4194304] 0 2024-04-22T15:49:47.189 INFO:tasks.workunit.client.1.smithi027.stdout:6/68: dread d5/f10 [0,4194304] 0 2024-04-22T15:49:47.190 INFO:tasks.workunit.client.0.smithi016.stdout:4/880: symlink d7/d21/d7c/d103/l126 0 2024-04-22T15:49:47.191 INFO:tasks.workunit.client.0.smithi016.stdout:3/945: getdents d7/d17/d18/d32/d42/d63/d86/dd4/d100 0 2024-04-22T15:49:47.191 INFO:tasks.workunit.client.0.smithi016.stdout:2/885: dread d2/d1a/d4c/d9f/dae/fb3 [4194304,4194304] 0 2024-04-22T15:49:47.192 INFO:tasks.workunit.client.1.smithi027.stdout:5/83: dwrite d0/f3 [0,4194304] 0 2024-04-22T15:49:47.194 INFO:tasks.workunit.client.0.smithi016.stdout:5/907: truncate df/d15/d1a/d50/d77/f40 34594 0 2024-04-22T15:49:47.194 INFO:tasks.workunit.client.1.smithi027.stdout:6/69: dread d5/f10 [0,4194304] 0 2024-04-22T15:49:47.195 INFO:tasks.workunit.client.0.smithi016.stdout:9/938: creat d9/d20/dac/dae/d126/f132 x:0 0 0 2024-04-22T15:49:47.197 INFO:tasks.workunit.client.0.smithi016.stdout:9/939: write d9/d57/d91/d103/f10c [75265,97589] 0 2024-04-22T15:49:47.200 INFO:tasks.workunit.client.1.smithi027.stdout:5/84: chown d0/df/l19 152155 1 2024-04-22T15:49:47.202 INFO:tasks.workunit.client.0.smithi016.stdout:5/908: dwrite df/d15/d1a/d50/d77/d7d/fc0 [0,4194304] 0 2024-04-22T15:49:47.203 INFO:tasks.workunit.client.0.smithi016.stdout:5/909: stat df/d15/d1a/d4e/d10a/c9f 0 2024-04-22T15:49:47.205 INFO:tasks.workunit.client.1.smithi027.stdout:5/85: dread d0/f15 [0,4194304] 0 2024-04-22T15:49:47.211 INFO:tasks.workunit.client.1.smithi027.stdout:8/99: write d1/d3/f10 [4272024,109537] 0 2024-04-22T15:49:47.211 INFO:tasks.workunit.client.0.smithi016.stdout:4/881: chown d7/d4e/d2f/d33/c7e 127021 1 2024-04-22T15:49:47.213 INFO:tasks.workunit.client.0.smithi016.stdout:3/946: truncate d7/d17/d18/d27/f4f 4941881 0 2024-04-22T15:49:47.217 INFO:tasks.workunit.client.0.smithi016.stdout:9/940: creat d9/d5d/db8/f133 x:0 0 0 2024-04-22T15:49:47.219 INFO:tasks.workunit.client.0.smithi016.stdout:0/986: write dc/d13/d14/d26/d32/fb0 [1424622,6111] 0 2024-04-22T15:49:47.220 INFO:tasks.workunit.client.0.smithi016.stdout:5/910: mknod df/d15/d1a/c117 0 2024-04-22T15:49:47.220 INFO:tasks.workunit.client.1.smithi027.stdout:6/70: link d5/l6 d5/dc/l15 0 2024-04-22T15:49:47.222 INFO:tasks.workunit.client.0.smithi016.stdout:7/872: rename d3/dc/d9f/f117 to d3/dc/d1b/d31/f13c 0 2024-04-22T15:49:47.223 INFO:tasks.workunit.client.1.smithi027.stdout:5/86: write d0/df/f12 [774716,13544] 0 2024-04-22T15:49:47.225 INFO:tasks.workunit.client.0.smithi016.stdout:9/941: mkdir d9/d20/dac/dbd/d134 0 2024-04-22T15:49:47.226 INFO:tasks.workunit.client.1.smithi027.stdout:6/71: rmdir d5/db 39 2024-04-22T15:49:47.228 INFO:tasks.workunit.client.1.smithi027.stdout:5/87: mknod d0/df/d10/c1b 0 2024-04-22T15:49:47.229 INFO:tasks.workunit.client.1.smithi027.stdout:8/100: write d1/f4 [306221,30638] 0 2024-04-22T15:49:47.230 INFO:tasks.workunit.client.1.smithi027.stdout:8/101: write d1/d3/f1a [681385,125240] 0 2024-04-22T15:49:47.231 INFO:tasks.workunit.client.0.smithi016.stdout:3/947: dwrite d7/d17/d18/d32/d3c/d51/df1/ff7 [0,4194304] 0 2024-04-22T15:49:47.232 INFO:tasks.workunit.client.0.smithi016.stdout:7/873: dwrite d3/dc/d33/d84/de3/ff8 [0,4194304] 0 2024-04-22T15:49:47.232 INFO:tasks.workunit.client.0.smithi016.stdout:8/850: rename d2/la4 to d2/d10/d43/dbe/l106 0 2024-04-22T15:49:47.233 INFO:tasks.workunit.client.1.smithi027.stdout:8/102: dread d1/f21 [0,4194304] 0 2024-04-22T15:49:47.233 INFO:tasks.workunit.client.1.smithi027.stdout:5/88: mkdir d0/d11/d1c 0 2024-04-22T15:49:47.234 INFO:tasks.workunit.client.1.smithi027.stdout:8/103: truncate d1/d3/f6 2072622 0 2024-04-22T15:49:47.234 INFO:tasks.workunit.client.1.smithi027.stdout:5/89: dread - d0/df/de/f18 zero size 2024-04-22T15:49:47.241 INFO:tasks.workunit.client.0.smithi016.stdout:9/942: symlink d9/d57/da7/daa/l135 0 2024-04-22T15:49:47.242 INFO:tasks.workunit.client.0.smithi016.stdout:9/943: write d9/d20/dac/dbd/ff8 [613260,72405] 0 2024-04-22T15:49:47.247 INFO:tasks.workunit.client.0.smithi016.stdout:8/851: rmdir d2/d82 39 2024-04-22T15:49:47.249 INFO:tasks.workunit.client.0.smithi016.stdout:7/874: mknod d3/dc/d1b/d5a/d9/df/c13d 0 2024-04-22T15:49:47.252 INFO:tasks.workunit.client.0.smithi016.stdout:2/886: rename d2/d23/db8/dd4/d114 to d2/d4d/d150 0 2024-04-22T15:49:47.252 INFO:tasks.workunit.client.0.smithi016.stdout:5/911: mknod df/d15/d1a/dd8/c118 0 2024-04-22T15:49:47.254 INFO:tasks.workunit.client.0.smithi016.stdout:8/852: creat d2/d82/d52/df0/d66/d7e/dc0/de2/f107 x:0 0 0 2024-04-22T15:49:47.255 INFO:tasks.workunit.client.0.smithi016.stdout:2/887: unlink d2/de/c14 0 2024-04-22T15:49:47.260 INFO:tasks.workunit.client.0.smithi016.stdout:0/987: rename dc/d13/d14/de5/d65/d40/d4a/d59/d78 to dc/d13/d14/de5/d65/d13d 0 2024-04-22T15:49:47.266 INFO:tasks.workunit.client.0.smithi016.stdout:3/948: rename d7/d17/d18/d32/d3c/lce to d7/d17/d18/d32/d42/d63/d86/l121 0 2024-04-22T15:49:47.266 INFO:tasks.workunit.client.0.smithi016.stdout:5/912: getdents df/d15/d1a/d50/d77/d94 0 2024-04-22T15:49:47.268 INFO:tasks.workunit.client.0.smithi016.stdout:9/944: rename d9/d57/d91/de4/f8e to d9/d57/d91/de4/d25/d36/d87/d109/f136 0 2024-04-22T15:49:47.272 INFO:tasks.workunit.client.0.smithi016.stdout:5/913: fdatasync df/d90/dad/f101 0 2024-04-22T15:49:47.274 INFO:tasks.workunit.client.0.smithi016.stdout:2/888: rename d2/de/d2a/d59/c10b to d2/d2c/de7/c151 0 2024-04-22T15:49:47.283 INFO:tasks.workunit.client.0.smithi016.stdout:2/889: dwrite d2/d2c/f82 [0,4194304] 0 2024-04-22T15:49:47.285 INFO:tasks.workunit.client.0.smithi016.stdout:3/949: dread d7/d17/d18/f2a [0,4194304] 0 2024-04-22T15:49:47.285 INFO:tasks.workunit.client.0.smithi016.stdout:8/853: dwrite d2/f24 [0,4194304] 0 2024-04-22T15:49:47.287 INFO:tasks.workunit.client.0.smithi016.stdout:3/950: fdatasync d7/d17/d18/d32/d42/fa2 0 2024-04-22T15:49:47.288 INFO:tasks.workunit.client.0.smithi016.stdout:3/951: readlink d7/d17/d18/d27/lfb 0 2024-04-22T15:49:47.290 INFO:tasks.workunit.client.0.smithi016.stdout:2/890: mknod d2/d1a/d4c/d9f/da3/c152 0 2024-04-22T15:49:47.291 INFO:tasks.workunit.client.0.smithi016.stdout:8/854: mknod d2/d82/d56/d99/c108 0 2024-04-22T15:49:47.291 INFO:tasks.workunit.client.0.smithi016.stdout:2/891: stat d2/d1a/d4c/d77/dea/f116 0 2024-04-22T15:49:47.293 INFO:tasks.workunit.client.0.smithi016.stdout:3/952: creat d7/d17/d18/d32/d42/d63/d86/dd8/f122 x:0 0 0 2024-04-22T15:49:47.294 INFO:tasks.workunit.client.0.smithi016.stdout:8/855: symlink d2/d32/d92/l109 0 2024-04-22T15:49:47.296 INFO:tasks.workunit.client.0.smithi016.stdout:2/892: mkdir d2/d153 0 2024-04-22T15:49:47.299 INFO:tasks.workunit.client.0.smithi016.stdout:3/953: unlink d7/d17/d18/d32/d3c/d51/f5f 0 2024-04-22T15:49:47.302 INFO:tasks.workunit.client.0.smithi016.stdout:2/893: link d2/d1a/d4c/d77/dea/f116 d2/d23/db8/dd4/f154 0 2024-04-22T15:49:47.303 INFO:tasks.workunit.client.0.smithi016.stdout:8/856: dread d2/d82/d56/f90 [0,4194304] 0 2024-04-22T15:49:47.303 INFO:tasks.workunit.client.0.smithi016.stdout:3/954: truncate d7/d17/d18/d32/d42/d63/d86/dd8/f122 702781 0 2024-04-22T15:49:47.304 INFO:tasks.workunit.client.0.smithi016.stdout:2/894: symlink d2/d2c/d97/l155 0 2024-04-22T15:49:47.306 INFO:tasks.workunit.client.0.smithi016.stdout:8/857: mkdir d2/d82/d52/df0/dc1/d10a 0 2024-04-22T15:49:47.307 INFO:tasks.workunit.client.0.smithi016.stdout:2/895: rmdir d2/d1a/d53 39 2024-04-22T15:49:47.307 INFO:tasks.workunit.client.0.smithi016.stdout:2/896: chown d2/d4d 16 1 2024-04-22T15:49:47.308 INFO:tasks.workunit.client.0.smithi016.stdout:3/955: getdents d7/d17/d18/d32/d42/d63/d86/dd4/d100 0 2024-04-22T15:49:47.310 INFO:tasks.workunit.client.0.smithi016.stdout:3/956: chown d7/d17/d18/d32/d42/d63/f10e 67234072 1 2024-04-22T15:49:47.314 INFO:tasks.workunit.client.0.smithi016.stdout:3/957: creat d7/d17/d18/d32/d3c/d47/f123 x:0 0 0 2024-04-22T15:49:47.317 INFO:tasks.workunit.client.0.smithi016.stdout:3/958: write d7/d17/d18/d32/d8a/ffd [155472,28238] 0 2024-04-22T15:49:47.319 INFO:tasks.workunit.client.0.smithi016.stdout:3/959: write d7/d17/d18/d32/d3c/d51/f114 [760897,115320] 0 2024-04-22T15:49:47.320 INFO:tasks.workunit.client.0.smithi016.stdout:2/897: dread d2/d1a/d4c/d9f/dae/fed [0,4194304] 0 2024-04-22T15:49:47.322 INFO:tasks.workunit.client.0.smithi016.stdout:6/928: sync 2024-04-22T15:49:47.322 INFO:tasks.workunit.client.0.smithi016.stdout:2/898: chown d2/d4d/d8e/f91 8586 1 2024-04-22T15:49:47.325 INFO:tasks.workunit.client.0.smithi016.stdout:2/899: creat d2/d1a/d7b/dc6/f156 x:0 0 0 2024-04-22T15:49:47.326 INFO:tasks.workunit.client.0.smithi016.stdout:8/858: dread d2/d10/d47/f95 [0,4194304] 0 2024-04-22T15:49:47.329 INFO:tasks.workunit.client.0.smithi016.stdout:8/859: creat d2/d10/d47/d67/f10b x:0 0 0 2024-04-22T15:49:47.330 INFO:tasks.workunit.client.0.smithi016.stdout:8/860: readlink d2/d82/d3e/lce 0 2024-04-22T15:49:47.332 INFO:tasks.workunit.client.0.smithi016.stdout:2/900: getdents d2/d1a/d7b/dcc 0 2024-04-22T15:49:47.333 INFO:tasks.workunit.client.0.smithi016.stdout:8/861: creat d2/d32/f10c x:0 0 0 2024-04-22T15:49:47.334 INFO:tasks.workunit.client.0.smithi016.stdout:2/901: write d2/d1a/d7b/dcc/f7e [3448652,4812] 0 2024-04-22T15:49:47.337 INFO:tasks.workunit.client.0.smithi016.stdout:3/960: dread d7/d17/d18/d32/d3c/d47/f48 [0,4194304] 0 2024-04-22T15:49:47.337 INFO:tasks.workunit.client.0.smithi016.stdout:2/902: write d2/f4f [7884333,85314] 0 2024-04-22T15:49:47.337 INFO:tasks.workunit.client.0.smithi016.stdout:8/862: rename d2/d82/d52/df0/d66/fa7 to d2/d77/f10d 0 2024-04-22T15:49:47.339 INFO:tasks.workunit.client.0.smithi016.stdout:2/903: mkdir d2/d4d/d8e/d157 0 2024-04-22T15:49:47.342 INFO:tasks.workunit.client.0.smithi016.stdout:8/863: rename d2/d82/f7b to d2/d82/d52/df0/d66/d7e/d85/de8/f10e 0 2024-04-22T15:49:47.344 INFO:tasks.workunit.client.0.smithi016.stdout:2/904: creat d2/d1a/d53/f158 x:0 0 0 2024-04-22T15:49:47.346 INFO:tasks.workunit.client.0.smithi016.stdout:3/961: rename d7/d17/d18/d32/d42/d63/deb/d21/d55/ddb/db7/c113 to d7/d17/d18/d32/d8a/d116/c124 0 2024-04-22T15:49:47.348 INFO:tasks.workunit.client.0.smithi016.stdout:2/905: mknod d2/d1a/d7b/dcc/de5/c159 0 2024-04-22T15:49:47.350 INFO:tasks.workunit.client.0.smithi016.stdout:2/906: fdatasync d2/d1a/f5e 0 2024-04-22T15:49:47.352 INFO:tasks.workunit.client.0.smithi016.stdout:3/962: mknod d7/d17/d18/d32/d42/d63/deb/d21/d55/ddb/c125 0 2024-04-22T15:49:47.354 INFO:tasks.workunit.client.0.smithi016.stdout:8/864: dwrite d2/d82/d52/df0/d66/d7e/ff5 [0,4194304] 0 2024-04-22T15:49:47.354 INFO:tasks.workunit.client.0.smithi016.stdout:2/907: fdatasync d2/d2c/d97/f6a 0 2024-04-22T15:49:47.358 INFO:tasks.workunit.client.0.smithi016.stdout:2/908: readlink d2/d1a/d4c/d9f/dae/l102 0 2024-04-22T15:49:47.360 INFO:tasks.workunit.client.1.smithi027.stdout:8/104: fdatasync d1/d3/fa 0 2024-04-22T15:49:47.361 INFO:tasks.workunit.client.1.smithi027.stdout:6/72: creat d5/dc/dd/f16 x:0 0 0 2024-04-22T15:49:47.361 INFO:tasks.workunit.client.1.smithi027.stdout:5/90: chown d0/df/l19 12 1 2024-04-22T15:49:47.363 INFO:tasks.workunit.client.1.smithi027.stdout:6/73: write d5/d8/f12 [6621087,9895] 0 2024-04-22T15:49:47.365 INFO:tasks.workunit.client.1.smithi027.stdout:8/105: mkdir d1/d3/d1c/d28 0 2024-04-22T15:49:47.367 INFO:tasks.workunit.client.0.smithi016.stdout:2/909: link d2/d1a/d4c/d9f/dae/fb3 d2/d1a/d7b/dc6/f15a 0 2024-04-22T15:49:47.367 INFO:tasks.workunit.client.1.smithi027.stdout:5/91: symlink d0/df/d10/l1d 0 2024-04-22T15:49:47.368 INFO:tasks.workunit.client.1.smithi027.stdout:5/92: chown d0/d11/f16 41734 1 2024-04-22T15:49:47.374 INFO:tasks.workunit.client.0.smithi016.stdout:8/865: dwrite d2/d82/d56/f71 [0,4194304] 0 2024-04-22T15:49:47.374 INFO:tasks.workunit.client.0.smithi016.stdout:2/910: mkdir d2/d4d/d8e/d157/d15b 0 2024-04-22T15:49:47.378 INFO:tasks.workunit.client.0.smithi016.stdout:2/911: chown d2/d2c/d97/d34/c14f 241296527 1 2024-04-22T15:49:47.380 INFO:tasks.workunit.client.1.smithi027.stdout:6/74: write d5/db/f11 [599648,13177] 0 2024-04-22T15:49:47.381 INFO:tasks.workunit.client.0.smithi016.stdout:8/866: mknod d2/d82/d3e/d8e/de6/c10f 0 2024-04-22T15:49:47.383 INFO:tasks.workunit.client.1.smithi027.stdout:6/75: write d5/db/f11 [587853,103217] 0 2024-04-22T15:49:47.384 INFO:tasks.workunit.client.1.smithi027.stdout:5/93: truncate d0/f15 1359532 0 2024-04-22T15:49:47.385 INFO:tasks.workunit.client.0.smithi016.stdout:3/963: dread d7/d17/d18/f5b [0,4194304] 0 2024-04-22T15:49:47.386 INFO:tasks.workunit.client.1.smithi027.stdout:8/106: dwrite f0 [4194304,4194304] 0 2024-04-22T15:49:47.386 INFO:tasks.workunit.client.0.smithi016.stdout:2/912: dwrite d2/de/f84 [0,4194304] 0 2024-04-22T15:49:47.387 INFO:tasks.workunit.client.0.smithi016.stdout:8/867: mkdir d2/d82/d52/df0/d66/d7e/d110 0 2024-04-22T15:49:47.387 INFO:tasks.workunit.client.0.smithi016.stdout:2/913: readlink d2/l11e 0 2024-04-22T15:49:47.388 INFO:tasks.workunit.client.1.smithi027.stdout:6/76: mkdir d5/dc/d17 0 2024-04-22T15:49:47.394 INFO:tasks.workunit.client.1.smithi027.stdout:8/107: unlink d1/l14 0 2024-04-22T15:49:47.397 INFO:tasks.workunit.client.1.smithi027.stdout:6/77: symlink d5/d9/l18 0 2024-04-22T15:49:47.397 INFO:tasks.workunit.client.0.smithi016.stdout:2/914: mkdir d2/d2c/d97/d67/dbf/d15c 0 2024-04-22T15:49:47.397 INFO:tasks.workunit.client.0.smithi016.stdout:8/868: dwrite d2/f27 [4194304,4194304] 0 2024-04-22T15:49:47.398 INFO:tasks.workunit.client.1.smithi027.stdout:5/94: symlink d0/d1a/l1e 0 2024-04-22T15:49:47.398 INFO:tasks.workunit.client.1.smithi027.stdout:6/78: dread - d5/dc/dd/f16 zero size 2024-04-22T15:49:47.398 INFO:tasks.workunit.client.0.smithi016.stdout:8/869: stat d2/d82/f58 0 2024-04-22T15:49:47.400 INFO:tasks.workunit.client.1.smithi027.stdout:8/108: unlink d1/fd 0 2024-04-22T15:49:47.400 INFO:tasks.workunit.client.1.smithi027.stdout:6/79: symlink d5/d9/l19 0 2024-04-22T15:49:47.402 INFO:tasks.workunit.client.1.smithi027.stdout:5/95: symlink d0/df/de/l1f 0 2024-04-22T15:49:47.402 INFO:tasks.workunit.client.1.smithi027.stdout:5/96: readlink d0/df/d10/l1d 0 2024-04-22T15:49:47.403 INFO:tasks.workunit.client.1.smithi027.stdout:8/109: symlink d1/d1e/l29 0 2024-04-22T15:49:47.407 INFO:tasks.workunit.client.0.smithi016.stdout:2/915: dwrite d2/d4d/d150/d11b/f13d [0,4194304] 0 2024-04-22T15:49:47.408 INFO:tasks.workunit.client.0.smithi016.stdout:2/916: readlink d2/d2c/daa/d110/l14a 0 2024-04-22T15:49:47.410 INFO:tasks.workunit.client.1.smithi027.stdout:5/97: mkdir d0/df/de/d20 0 2024-04-22T15:49:47.411 INFO:tasks.workunit.client.1.smithi027.stdout:5/98: write d0/f8 [1640478,3969] 0 2024-04-22T15:49:47.414 INFO:tasks.workunit.client.1.smithi027.stdout:8/110: rename d1/d3/d1c/c1d to d1/c2a 0 2024-04-22T15:49:47.418 INFO:tasks.workunit.client.1.smithi027.stdout:6/80: rename d5/f10 to d5/f1a 0 2024-04-22T15:49:47.419 INFO:tasks.workunit.client.0.smithi016.stdout:8/870: link d2/d82/d52/df0/d66/d1c/c91 d2/d82/d52/df0/d66/d7e/dc0/c111 0 2024-04-22T15:49:47.419 INFO:tasks.workunit.client.1.smithi027.stdout:7/69: sync 2024-04-22T15:49:47.424 INFO:tasks.workunit.client.1.smithi027.stdout:7/70: creat d0/f14 x:0 0 0 2024-04-22T15:49:47.426 INFO:tasks.workunit.client.0.smithi016.stdout:8/871: mkdir d2/d82/d52/df0/dc1/d10a/d112 0 2024-04-22T15:49:47.427 INFO:tasks.workunit.client.0.smithi016.stdout:3/964: dwrite d7/d17/d18/d32/d42/d63/deb/fc2 [0,4194304] 0 2024-04-22T15:49:47.428 INFO:tasks.workunit.client.1.smithi027.stdout:8/111: symlink d1/d3/d1c/d28/l2b 0 2024-04-22T15:49:47.428 INFO:tasks.workunit.client.1.smithi027.stdout:8/112: chown d1/d1e 5 1 2024-04-22T15:49:47.429 INFO:tasks.workunit.client.0.smithi016.stdout:7/875: sync 2024-04-22T15:49:47.429 INFO:tasks.workunit.client.0.smithi016.stdout:4/882: sync 2024-04-22T15:49:47.429 INFO:tasks.workunit.client.1.smithi027.stdout:6/81: creat d5/dc/d17/f1b x:0 0 0 2024-04-22T15:49:47.430 INFO:tasks.workunit.client.0.smithi016.stdout:7/876: chown d3/dc/d22/l2d 1187892 1 2024-04-22T15:49:47.435 INFO:tasks.workunit.client.1.smithi027.stdout:8/113: rename d1/d3/c25 to d1/d3/d1c/d28/c2c 0 2024-04-22T15:49:47.438 INFO:tasks.workunit.client.1.smithi027.stdout:8/114: readlink d1/d3/l12 0 2024-04-22T15:49:47.439 INFO:tasks.workunit.client.1.smithi027.stdout:8/115: write d1/d3/f6 [294819,50673] 0 2024-04-22T15:49:47.440 INFO:tasks.workunit.client.0.smithi016.stdout:8/872: symlink d2/d82/d3e/dd5/l113 0 2024-04-22T15:49:47.441 INFO:tasks.workunit.client.0.smithi016.stdout:4/883: getdents d7/d4e/d102/d106/d88/d9d/dad/db9/d109 0 2024-04-22T15:49:47.442 INFO:tasks.workunit.client.0.smithi016.stdout:4/884: write d7/d21/d45/d49/d67/fd4 [2520132,83271] 0 2024-04-22T15:49:47.452 INFO:tasks.workunit.client.0.smithi016.stdout:3/965: getdents d7/d17/d18/d32/d3c 0 2024-04-22T15:49:47.453 INFO:tasks.workunit.client.0.smithi016.stdout:2/917: dwrite d2/f2b [0,4194304] 0 2024-04-22T15:49:47.454 INFO:tasks.workunit.client.0.smithi016.stdout:4/885: dwrite d7/d21/fc6 [4194304,4194304] 0 2024-04-22T15:49:47.455 INFO:tasks.workunit.client.0.smithi016.stdout:3/966: stat d7/d17/d18/d32/d42/d63/deb/d21/d55/dac/f108 0 2024-04-22T15:49:47.456 INFO:tasks.workunit.client.1.smithi027.stdout:5/99: dwrite d0/f3 [0,4194304] 0 2024-04-22T15:49:47.456 INFO:tasks.workunit.client.1.smithi027.stdout:8/116: link d1/d1e/l24 d1/d3/d1c/l2d 0 2024-04-22T15:49:47.457 INFO:tasks.workunit.client.1.smithi027.stdout:8/117: dread - d1/d3/f19 zero size 2024-04-22T15:49:47.458 INFO:tasks.workunit.client.1.smithi027.stdout:6/82: write d5/f1a [3695693,107859] 0 2024-04-22T15:49:47.459 INFO:tasks.workunit.client.0.smithi016.stdout:7/877: getdents d3/dc/d1b/d5a 0 2024-04-22T15:49:47.459 INFO:tasks.workunit.client.0.smithi016.stdout:2/918: write d2/d9/fa8 [692158,60083] 0 2024-04-22T15:49:47.460 INFO:tasks.workunit.client.1.smithi027.stdout:6/83: fdatasync d5/d8/f12 0 2024-04-22T15:49:47.460 INFO:tasks.workunit.client.1.smithi027.stdout:6/84: chown c1 2023 1 2024-04-22T15:49:47.463 INFO:tasks.workunit.client.0.smithi016.stdout:8/873: dread d2/d82/d52/df0/d66/f3c [0,4194304] 0 2024-04-22T15:49:47.470 INFO:tasks.workunit.client.1.smithi027.stdout:6/85: dwrite d5/db/f11 [0,4194304] 0 2024-04-22T15:49:47.470 INFO:tasks.workunit.client.1.smithi027.stdout:6/86: chown d5/d9 2032589395 1 2024-04-22T15:49:47.471 INFO:tasks.workunit.client.0.smithi016.stdout:4/886: chown d7/l77 129006 1 2024-04-22T15:49:47.472 INFO:tasks.workunit.client.0.smithi016.stdout:2/919: mkdir d2/d2c/daa/d15d 0 2024-04-22T15:49:47.473 INFO:tasks.workunit.client.0.smithi016.stdout:8/874: symlink d2/d82/d52/df0/l114 0 2024-04-22T15:49:47.474 INFO:tasks.workunit.client.0.smithi016.stdout:2/920: write d2/d2c/d97/d34/fdb [3625782,31615] 0 2024-04-22T15:49:47.475 INFO:tasks.workunit.client.1.smithi027.stdout:6/87: chown d5/l6 22389 1 2024-04-22T15:49:47.475 INFO:tasks.workunit.client.1.smithi027.stdout:6/88: chown d5/dc 53990434 1 2024-04-22T15:49:47.476 INFO:tasks.workunit.client.1.smithi027.stdout:6/89: write d5/db/f11 [753295,114399] 0 2024-04-22T15:49:47.481 INFO:tasks.workunit.client.1.smithi027.stdout:6/90: rename c2 to d5/dc/d17/c1c 0 2024-04-22T15:49:47.482 INFO:tasks.workunit.client.0.smithi016.stdout:2/921: symlink d2/d153/l15e 0 2024-04-22T15:49:47.483 INFO:tasks.workunit.client.1.smithi027.stdout:6/91: symlink d5/dc/d17/l1d 0 2024-04-22T15:49:47.485 INFO:tasks.workunit.client.1.smithi027.stdout:6/92: dread d5/d9/ff [0,4194304] 0 2024-04-22T15:49:47.486 INFO:tasks.workunit.client.0.smithi016.stdout:2/922: creat d2/d1a/d7b/dcc/de5/d107/f15f x:0 0 0 2024-04-22T15:49:47.487 INFO:tasks.workunit.client.0.smithi016.stdout:4/887: dread d7/d24/fed [0,4194304] 0 2024-04-22T15:49:47.487 INFO:tasks.workunit.client.0.smithi016.stdout:2/923: write d2/d9/f80 [1624067,109191] 0 2024-04-22T15:49:47.489 INFO:tasks.workunit.client.0.smithi016.stdout:4/888: creat d7/d4e/de1/f127 x:0 0 0 2024-04-22T15:49:47.491 INFO:tasks.workunit.client.1.smithi027.stdout:6/93: dwrite d5/d8/f12 [0,4194304] 0 2024-04-22T15:49:47.496 INFO:tasks.workunit.client.0.smithi016.stdout:4/889: dwrite d7/d24/f76 [0,4194304] 0 2024-04-22T15:49:47.497 INFO:tasks.workunit.client.1.smithi027.stdout:8/118: write d1/f18 [412788,22000] 0 2024-04-22T15:49:47.499 INFO:tasks.workunit.client.0.smithi016.stdout:3/967: write d7/d17/f80 [1160204,20515] 0 2024-04-22T15:49:47.499 INFO:tasks.workunit.client.1.smithi027.stdout:6/94: dwrite d5/d8/f12 [8388608,4194304] 0 2024-04-22T15:49:47.502 INFO:tasks.workunit.client.0.smithi016.stdout:4/890: creat d7/d4e/d2f/f128 x:0 0 0 2024-04-22T15:49:47.504 INFO:tasks.workunit.client.0.smithi016.stdout:4/891: write d7/d22/f73 [1635384,45048] 0 2024-04-22T15:49:47.505 INFO:tasks.workunit.client.0.smithi016.stdout:7/878: write d3/dc/d22/f6b [1419874,92853] 0 2024-04-22T15:49:47.506 INFO:tasks.workunit.client.1.smithi027.stdout:8/119: dwrite d1/d3/fa [0,4194304] 0 2024-04-22T15:49:47.508 INFO:tasks.workunit.client.1.smithi027.stdout:8/120: dread - d1/d1e/f22 zero size 2024-04-22T15:49:47.518 INFO:tasks.workunit.client.0.smithi016.stdout:7/879: write d3/dc/d1b/d31/d5f/f7b [2947063,91285] 0 2024-04-22T15:49:47.520 INFO:tasks.workunit.client.0.smithi016.stdout:4/892: truncate d7/d24/fed 1504840 0 2024-04-22T15:49:47.521 INFO:tasks.workunit.client.0.smithi016.stdout:2/924: dwrite d2/d23/d92/d55/fa7 [0,4194304] 0 2024-04-22T15:49:47.524 INFO:tasks.workunit.client.0.smithi016.stdout:7/880: truncate d3/f28 3209016 0 2024-04-22T15:49:47.524 INFO:tasks.workunit.client.0.smithi016.stdout:7/881: read - d3/dc/d1b/d31/d5f/f114 zero size 2024-04-22T15:49:47.529 INFO:tasks.workunit.client.0.smithi016.stdout:4/893: write d7/d21/d45/f6d [192817,23692] 0 2024-04-22T15:49:47.530 INFO:tasks.workunit.client.0.smithi016.stdout:7/882: read d3/dc/d1b/d5a/d9/f2a [1825772,65477] 0 2024-04-22T15:49:47.532 INFO:tasks.workunit.client.0.smithi016.stdout:4/894: chown d7/d4e/d2f/d33/l70 16957756 1 2024-04-22T15:49:47.532 INFO:tasks.workunit.client.1.smithi027.stdout:6/95: fsync d5/d8/f12 0 2024-04-22T15:49:47.533 INFO:tasks.workunit.client.0.smithi016.stdout:2/925: getdents d2/d2c/d97/d34 0 2024-04-22T15:49:47.534 INFO:tasks.workunit.client.1.smithi027.stdout:6/96: creat d5/db/f1e x:0 0 0 2024-04-22T15:49:47.535 INFO:tasks.workunit.client.0.smithi016.stdout:4/895: truncate d7/d24/d51/fcf 968178 0 2024-04-22T15:49:47.535 INFO:tasks.workunit.client.0.smithi016.stdout:2/926: write d2/d1a/d7b/dcc/f7e [4685475,82088] 0 2024-04-22T15:49:47.535 INFO:tasks.workunit.client.0.smithi016.stdout:7/883: mknod d3/d11a/d8c/d13b/c13e 0 2024-04-22T15:49:47.535 INFO:tasks.workunit.client.1.smithi027.stdout:6/97: write d5/dc/dd/f16 [546145,58940] 0 2024-04-22T15:49:47.537 INFO:tasks.workunit.client.0.smithi016.stdout:2/927: mkdir d2/d9/d160 0 2024-04-22T15:49:47.538 INFO:tasks.workunit.client.1.smithi027.stdout:6/98: mkdir d5/d9/d1f 0 2024-04-22T15:49:47.538 INFO:tasks.workunit.client.1.smithi027.stdout:6/99: readlink d5/dc/d17/l1d 0 2024-04-22T15:49:47.541 INFO:tasks.workunit.client.1.smithi027.stdout:6/100: mkdir d5/dc/d20 0 2024-04-22T15:49:47.542 INFO:tasks.workunit.client.0.smithi016.stdout:3/968: dread d7/d17/d18/d27/fb8 [0,4194304] 0 2024-04-22T15:49:47.542 INFO:tasks.workunit.client.0.smithi016.stdout:2/928: chown d2/d1a/d7b/dcc/de5/d107/l139 1351982258 1 2024-04-22T15:49:47.545 INFO:tasks.workunit.client.1.smithi027.stdout:6/101: dread d5/d8/f12 [0,4194304] 0 2024-04-22T15:49:47.545 INFO:tasks.workunit.client.0.smithi016.stdout:3/969: truncate d7/d17/d18/d32/d42/f9a 98648 0 2024-04-22T15:49:47.551 INFO:tasks.workunit.client.0.smithi016.stdout:2/929: rename d2/l21 to d2/d4d/l161 0 2024-04-22T15:49:47.553 INFO:tasks.workunit.client.0.smithi016.stdout:3/970: creat d7/d17/d18/d32/d3c/d51/f126 x:0 0 0 2024-04-22T15:49:47.554 INFO:tasks.workunit.client.1.smithi027.stdout:6/102: truncate d5/f1a 707764 0 2024-04-22T15:49:47.557 INFO:tasks.workunit.client.1.smithi027.stdout:6/103: symlink d5/dc/l21 0 2024-04-22T15:49:47.561 INFO:tasks.workunit.client.0.smithi016.stdout:2/930: getdents d2/d9 0 2024-04-22T15:49:47.565 INFO:tasks.workunit.client.0.smithi016.stdout:3/971: rmdir d7/d17/d18/d32/d42/d63/d71/d120 0 2024-04-22T15:49:47.566 INFO:tasks.workunit.client.0.smithi016.stdout:2/931: mkdir d2/de/d2a/d59/d162 0 2024-04-22T15:49:47.570 INFO:tasks.workunit.client.0.smithi016.stdout:3/972: unlink d7/d17/d18/c10c 0 2024-04-22T15:49:47.571 INFO:tasks.workunit.client.0.smithi016.stdout:2/932: symlink d2/d1a/d7b/dcc/de5/d107/l163 0 2024-04-22T15:49:47.575 INFO:tasks.workunit.client.0.smithi016.stdout:3/973: mkdir d7/d17/d18/d32/d3c/d51/d118/d127 0 2024-04-22T15:49:47.578 INFO:tasks.workunit.client.0.smithi016.stdout:3/974: fsync d7/d17/d18/d32/d8a/fa8 0 2024-04-22T15:49:47.579 INFO:tasks.workunit.client.0.smithi016.stdout:3/975: chown d7/d17/d18/d32/d3c/d47/f52 918963 1 2024-04-22T15:49:47.580 INFO:tasks.workunit.client.0.smithi016.stdout:2/933: write d2/d23/d92/fa1 [563164,30773] 0 2024-04-22T15:49:47.580 INFO:tasks.workunit.client.0.smithi016.stdout:3/976: write d7/d17/d18/d32/d42/d63/d86/fdf [798645,116299] 0 2024-04-22T15:49:47.581 INFO:tasks.workunit.client.1.smithi027.stdout:6/104: truncate d5/f1a 919301 0 2024-04-22T15:49:47.583 INFO:tasks.workunit.client.0.smithi016.stdout:3/977: read d7/fc [1177390,30363] 0 2024-04-22T15:49:47.583 INFO:tasks.workunit.client.1.smithi027.stdout:6/105: write d5/d9/ff [86630,72224] 0 2024-04-22T15:49:47.585 INFO:tasks.workunit.client.0.smithi016.stdout:2/934: getdents d2/d1a/d7b/dcc/de5 0 2024-04-22T15:49:47.585 INFO:tasks.workunit.client.0.smithi016.stdout:2/935: dread - d2/d1a/d7b/dc6/f156 zero size 2024-04-22T15:49:47.586 INFO:tasks.workunit.client.0.smithi016.stdout:2/936: read - d2/d1a/d7b/dc6/f156 zero size 2024-04-22T15:49:47.587 INFO:tasks.workunit.client.1.smithi027.stdout:6/106: symlink d5/l22 0 2024-04-22T15:49:47.587 INFO:tasks.workunit.client.0.smithi016.stdout:2/937: chown d2/de/d2a/d59/f78 10211012 1 2024-04-22T15:49:47.588 INFO:tasks.workunit.client.1.smithi027.stdout:6/107: dread - d5/db/f1e zero size 2024-04-22T15:49:47.593 INFO:tasks.workunit.client.0.smithi016.stdout:0/988: sync 2024-04-22T15:49:47.598 INFO:tasks.workunit.client.0.smithi016.stdout:2/938: dwrite d2/f4 [8388608,4194304] 0 2024-04-22T15:49:47.598 INFO:tasks.workunit.client.1.smithi027.stdout:6/108: dwrite d5/f1a [0,4194304] 0 2024-04-22T15:49:47.599 INFO:tasks.workunit.client.1.smithi027.stdout:6/109: write d5/d9/ff [420536,129119] 0 2024-04-22T15:49:47.600 INFO:tasks.workunit.client.0.smithi016.stdout:0/989: creat dc/d13/d14/de5/d65/d40/dd5/d11a/f13e x:0 0 0 2024-04-22T15:49:47.602 INFO:tasks.workunit.client.0.smithi016.stdout:3/978: dread d7/d17/d18/d32/d42/d63/deb/d21/d55/f79 [0,4194304] 0 2024-04-22T15:49:47.605 INFO:tasks.workunit.client.0.smithi016.stdout:2/939: creat d2/d1a/f164 x:0 0 0 2024-04-22T15:49:47.610 INFO:tasks.workunit.client.1.smithi027.stdout:6/110: dwrite d5/dc/d17/f1b [0,4194304] 0 2024-04-22T15:49:47.611 INFO:tasks.workunit.client.0.smithi016.stdout:3/979: rmdir d7/d17/d18 39 2024-04-22T15:49:47.613 INFO:tasks.workunit.client.1.smithi027.stdout:6/111: symlink d5/db/l23 0 2024-04-22T15:49:47.616 INFO:tasks.workunit.client.0.smithi016.stdout:0/990: getdents dc/d13/d14/d105 0 2024-04-22T15:49:47.621 INFO:tasks.workunit.client.1.smithi027.stdout:6/112: dwrite d5/db/f1e [0,4194304] 0 2024-04-22T15:49:47.622 INFO:tasks.workunit.client.1.smithi027.stdout:6/113: truncate d5/f1a 4414354 0 2024-04-22T15:49:47.623 INFO:tasks.workunit.client.1.smithi027.stdout:6/114: chown d5/dc/l15 9986694 1 2024-04-22T15:49:47.625 INFO:tasks.workunit.client.1.smithi027.stdout:6/115: creat d5/d8/f24 x:0 0 0 2024-04-22T15:49:47.626 INFO:tasks.workunit.client.0.smithi016.stdout:2/940: write d2/d1a/d7b/dc6/f15a [1594165,93877] 0 2024-04-22T15:49:47.627 INFO:tasks.workunit.client.0.smithi016.stdout:0/991: write dc/d7d/fa3 [712659,131049] 0 2024-04-22T15:49:47.627 INFO:tasks.workunit.client.1.smithi027.stdout:6/116: rmdir d5/db 39 2024-04-22T15:49:47.628 INFO:tasks.workunit.client.0.smithi016.stdout:2/941: dread - d2/d2c/d97/d34/f141 zero size 2024-04-22T15:49:47.629 INFO:tasks.workunit.client.1.smithi027.stdout:6/117: mknod d5/dc/dd/c25 0 2024-04-22T15:49:47.633 INFO:tasks.workunit.client.1.smithi027.stdout:6/118: read d5/d9/ff [137464,105945] 0 2024-04-22T15:49:47.635 INFO:tasks.workunit.client.1.smithi027.stdout:6/119: mkdir d5/dc/d26 0 2024-04-22T15:49:47.637 INFO:tasks.workunit.client.0.smithi016.stdout:3/980: dwrite d7/d17/d18/d27/f4f [0,4194304] 0 2024-04-22T15:49:47.637 INFO:tasks.workunit.client.1.smithi027.stdout:6/120: creat d5/d8/f27 x:0 0 0 2024-04-22T15:49:47.638 INFO:tasks.workunit.client.0.smithi016.stdout:0/992: dwrite dc/d13/d14/de5/d65/d40/d4a/db2/ffd [0,4194304] 0 2024-04-22T15:49:47.640 INFO:tasks.workunit.client.0.smithi016.stdout:0/993: chown dc/d13/d14/de5/d65/d1f/d48 1171868 1 2024-04-22T15:49:47.640 INFO:tasks.workunit.client.0.smithi016.stdout:2/942: dread d2/d9/f80 [0,4194304] 0 2024-04-22T15:49:47.645 INFO:tasks.workunit.client.0.smithi016.stdout:3/981: mkdir d7/d17/d18/d32/d42/d128 0 2024-04-22T15:49:47.646 INFO:tasks.workunit.client.1.smithi027.stdout:6/121: dwrite d5/d8/f12 [0,4194304] 0 2024-04-22T15:49:47.646 INFO:tasks.workunit.client.1.smithi027.stdout:4/66: sync 2024-04-22T15:49:47.646 INFO:tasks.workunit.client.1.smithi027.stdout:0/112: sync 2024-04-22T15:49:47.647 INFO:tasks.workunit.client.0.smithi016.stdout:0/994: symlink dc/d13/d14/de5/dd1/ddc/df6/l13f 0 2024-04-22T15:49:47.650 INFO:tasks.workunit.client.0.smithi016.stdout:2/943: creat d2/d23/d92/d120/f165 x:0 0 0 2024-04-22T15:49:47.651 INFO:tasks.workunit.client.1.smithi027.stdout:0/113: creat d0/f26 x:0 0 0 2024-04-22T15:49:47.652 INFO:tasks.workunit.client.0.smithi016.stdout:3/982: fdatasync d7/d17/d18/d32/d42/d63/deb/f7c 0 2024-04-22T15:49:47.654 INFO:tasks.workunit.client.0.smithi016.stdout:0/995: mkdir dc/d13/d14/de5/d65/d1f/d48/d140 0 2024-04-22T15:49:47.654 INFO:tasks.workunit.client.0.smithi016.stdout:3/983: stat d7/d17/d18/d32/d8a/ffd 0 2024-04-22T15:49:47.658 INFO:tasks.workunit.client.1.smithi027.stdout:4/67: link d5/d7/c8 d5/d7/c1b 0 2024-04-22T15:49:47.658 INFO:tasks.workunit.client.0.smithi016.stdout:3/984: write d7/d17/d18/f5b [1650316,93424] 0 2024-04-22T15:49:47.659 INFO:tasks.workunit.client.1.smithi027.stdout:4/68: truncate d5/d7/fb 360324 0 2024-04-22T15:49:47.661 INFO:tasks.workunit.client.0.smithi016.stdout:2/944: rename d2/de/f5c to d2/f166 0 2024-04-22T15:49:47.662 INFO:tasks.workunit.client.0.smithi016.stdout:2/945: dread - d2/d23/db8/f14c zero size 2024-04-22T15:49:47.663 INFO:tasks.workunit.client.1.smithi027.stdout:4/69: mknod d5/c1c 0 2024-04-22T15:49:47.665 INFO:tasks.workunit.client.1.smithi027.stdout:4/70: dread d5/d7/fb [0,4194304] 0 2024-04-22T15:49:47.665 INFO:tasks.workunit.client.0.smithi016.stdout:3/985: symlink d7/d17/d18/d32/d42/l129 0 2024-04-22T15:49:47.665 INFO:tasks.workunit.client.0.smithi016.stdout:3/986: chown d7/d17/d18/d32/d42/d63/deb/l34 1657253 1 2024-04-22T15:49:47.666 INFO:tasks.workunit.client.0.smithi016.stdout:2/946: unlink d2/d2c/d97/d67/dbf/fba 0 2024-04-22T15:49:47.667 INFO:tasks.workunit.client.0.smithi016.stdout:5/914: sync 2024-04-22T15:49:47.667 INFO:tasks.workunit.client.0.smithi016.stdout:6/929: sync 2024-04-22T15:49:47.667 INFO:tasks.workunit.client.0.smithi016.stdout:9/945: sync 2024-04-22T15:49:47.668 INFO:tasks.workunit.client.1.smithi027.stdout:4/71: mkdir d5/d11/d1d 0 2024-04-22T15:49:47.674 INFO:tasks.workunit.client.1.smithi027.stdout:6/122: truncate d5/db/f1e 2052940 0 2024-04-22T15:49:47.678 INFO:tasks.workunit.client.0.smithi016.stdout:9/946: mknod d9/d20/dac/dbd/c137 0 2024-04-22T15:49:47.681 INFO:tasks.workunit.client.1.smithi027.stdout:6/123: rename d5/d8/ca to d5/dc/c28 0 2024-04-22T15:49:47.682 INFO:tasks.workunit.client.1.smithi027.stdout:0/114: dwrite d0/d1a/f20 [0,4194304] 0 2024-04-22T15:49:47.683 INFO:tasks.workunit.client.0.smithi016.stdout:0/996: dread dc/d13/d99/fc4 [0,4194304] 0 2024-04-22T15:49:47.683 INFO:tasks.workunit.client.0.smithi016.stdout:5/915: symlink df/d15/d1a/d50/d77/d7d/d8b/d8c/d10c/l119 0 2024-04-22T15:49:47.684 INFO:tasks.workunit.client.0.smithi016.stdout:0/997: chown dc/d13/d14/d26/d32/fb0 549998350 1 2024-04-22T15:49:47.687 INFO:tasks.workunit.client.1.smithi027.stdout:4/72: dwrite d5/d7/fb [0,4194304] 0 2024-04-22T15:49:47.688 INFO:tasks.workunit.client.0.smithi016.stdout:2/947: write d2/d2c/f3c [9425665,65148] 0 2024-04-22T15:49:47.688 INFO:tasks.workunit.client.1.smithi027.stdout:4/73: chown d5/d7/f15 312023 1 2024-04-22T15:49:47.689 INFO:tasks.workunit.client.1.smithi027.stdout:6/124: rmdir d5/dc 39 2024-04-22T15:49:47.690 INFO:tasks.workunit.client.1.smithi027.stdout:6/125: write d5/d8/f24 [738294,70437] 0 2024-04-22T15:49:47.692 INFO:tasks.workunit.client.0.smithi016.stdout:6/930: mkdir d3/da/db/d44/d131 0 2024-04-22T15:49:47.697 INFO:tasks.workunit.client.0.smithi016.stdout:9/947: rename d9/de/d26/c45 to d9/d57/d91/de4/d25/d36/c138 0 2024-04-22T15:49:47.698 INFO:tasks.workunit.client.1.smithi027.stdout:0/115: chown d0/l25 0 1 2024-04-22T15:49:47.701 INFO:tasks.workunit.client.1.smithi027.stdout:4/74: symlink d5/de/l1e 0 2024-04-22T15:49:47.703 INFO:tasks.workunit.client.1.smithi027.stdout:4/75: write d5/fa [1945282,23474] 0 2024-04-22T15:49:47.703 INFO:tasks.workunit.client.0.smithi016.stdout:7/884: sync 2024-04-22T15:49:47.704 INFO:tasks.workunit.client.0.smithi016.stdout:4/896: sync 2024-04-22T15:49:47.704 INFO:tasks.workunit.client.0.smithi016.stdout:8/875: sync 2024-04-22T15:49:47.704 INFO:tasks.workunit.client.0.smithi016.stdout:2/948: fsync d2/d4d/d150/d11b/f9a 0 2024-04-22T15:49:47.704 INFO:tasks.workunit.client.0.smithi016.stdout:3/987: dwrite d7/d17/d18/d32/f9f [0,4194304] 0 2024-04-22T15:49:47.707 INFO:tasks.workunit.client.1.smithi027.stdout:4/76: creat d5/d7/f1f x:0 0 0 2024-04-22T15:49:47.708 INFO:tasks.workunit.client.0.smithi016.stdout:7/885: write d3/de6/fec [698921,7962] 0 2024-04-22T15:49:47.708 INFO:tasks.workunit.client.0.smithi016.stdout:8/876: write d2/d82/d52/f54 [3213533,54733] 0 2024-04-22T15:49:47.709 INFO:tasks.workunit.client.1.smithi027.stdout:6/126: creat d5/f29 x:0 0 0 2024-04-22T15:49:47.709 INFO:tasks.workunit.client.0.smithi016.stdout:3/988: dread - d7/d17/d18/d32/d42/d63/db4/f11d zero size 2024-04-22T15:49:47.709 INFO:tasks.workunit.client.0.smithi016.stdout:7/886: chown d3/f8b 14790507 1 2024-04-22T15:49:47.710 INFO:tasks.workunit.client.0.smithi016.stdout:9/948: symlink d9/d57/d91/l139 0 2024-04-22T15:49:47.714 INFO:tasks.workunit.client.1.smithi027.stdout:4/77: mknod d5/de/c20 0 2024-04-22T15:49:47.715 INFO:tasks.workunit.client.1.smithi027.stdout:4/78: write d5/fa [814744,12548] 0 2024-04-22T15:49:47.716 INFO:tasks.workunit.client.1.smithi027.stdout:6/127: symlink d5/d8/l2a 0 2024-04-22T15:49:47.719 INFO:tasks.workunit.client.0.smithi016.stdout:0/998: link dc/d13/d6f/l134 dc/d13/d14/de5/dd8/l141 0 2024-04-22T15:49:47.720 INFO:tasks.workunit.client.0.smithi016.stdout:4/897: dwrite d7/d21/d45/f6d [0,4194304] 0 2024-04-22T15:49:47.721 INFO:tasks.workunit.client.1.smithi027.stdout:4/79: creat d5/de/f21 x:0 0 0 2024-04-22T15:49:47.722 INFO:tasks.workunit.client.0.smithi016.stdout:4/898: stat d7/d4e/d2f/d50 0 2024-04-22T15:49:47.724 INFO:tasks.workunit.client.0.smithi016.stdout:5/916: dread df/d90/dad/fb2 [0,4194304] 0 2024-04-22T15:49:47.725 INFO:tasks.workunit.client.0.smithi016.stdout:4/899: truncate d7/d21/d45/dd6/f11c 492246 0 2024-04-22T15:49:47.730 INFO:tasks.workunit.client.0.smithi016.stdout:3/989: mkdir d7/d17/d18/d32/d3c/d12a 0 2024-04-22T15:49:47.731 INFO:tasks.workunit.client.1.smithi027.stdout:4/80: truncate d5/d7/ff 280372 0 2024-04-22T15:49:47.733 INFO:tasks.workunit.client.0.smithi016.stdout:9/949: rename d9/de/d23/f9c to d9/d5d/f13a 0 2024-04-22T15:49:47.734 INFO:tasks.workunit.client.0.smithi016.stdout:9/950: chown d9/d37/dea/d11b 1 1 2024-04-22T15:49:47.734 INFO:tasks.workunit.client.0.smithi016.stdout:0/999: fdatasync dc/d13/d14/de5/d65/d30/ff3 0 2024-04-22T15:49:47.735 INFO:tasks.workunit.client.1.smithi027.stdout:4/81: symlink d5/d11/l22 0 2024-04-22T15:49:47.735 INFO:tasks.workunit.client.1.smithi027.stdout:4/82: dread - d5/d7/f15 zero size 2024-04-22T15:49:47.736 INFO:tasks.workunit.client.1.smithi027.stdout:4/83: truncate d5/d7/f10 1075033 0 2024-04-22T15:49:47.737 INFO:tasks.workunit.client.0.smithi016.stdout:5/917: symlink df/d5b/db3/l11a 0 2024-04-22T15:49:47.740 INFO:tasks.workunit.client.0.smithi016.stdout:9/951: mknod d9/d20/dac/dbd/c13b 0 2024-04-22T15:49:47.741 INFO:tasks.workunit.client.0.smithi016.stdout:4/900: symlink d7/d4e/l129 0 2024-04-22T15:49:47.742 INFO:tasks.workunit.client.0.smithi016.stdout:5/918: creat df/d5b/d70/f11b x:0 0 0 2024-04-22T15:49:47.743 INFO:tasks.workunit.client.0.smithi016.stdout:4/901: readlink d7/d4e/d102/d106/d88/d9d/lf1 0 2024-04-22T15:49:47.745 INFO:tasks.workunit.client.1.smithi027.stdout:6/128: rename d5/d8 to d5/dc/d20/d2b 0 2024-04-22T15:49:47.745 INFO:tasks.workunit.client.0.smithi016.stdout:5/919: chown df/d15/d1a/d50/d77/d7a/fcb 0 1 2024-04-22T15:49:47.749 INFO:tasks.workunit.client.1.smithi027.stdout:6/129: dread d5/f1a [0,4194304] 0 2024-04-22T15:49:47.749 INFO:tasks.workunit.client.1.smithi027.stdout:6/130: stat d5/dc/d17/f1b 0 2024-04-22T15:49:47.750 INFO:tasks.workunit.client.0.smithi016.stdout:9/952: getdents d9 0 2024-04-22T15:49:47.750 INFO:tasks.workunit.client.0.smithi016.stdout:2/949: write d2/d1a/f20 [4744858,64837] 0 2024-04-22T15:49:47.751 INFO:tasks.workunit.client.1.smithi027.stdout:6/131: write d5/dc/d20/d2b/f24 [1064097,81433] 0 2024-04-22T15:49:47.753 INFO:tasks.workunit.client.0.smithi016.stdout:4/902: creat d7/d22/dc7/d10b/d120/f12a x:0 0 0 2024-04-22T15:49:47.754 INFO:tasks.workunit.client.1.smithi027.stdout:4/84: read d5/d7/ff [247893,84396] 0 2024-04-22T15:49:47.754 INFO:tasks.workunit.client.0.smithi016.stdout:9/953: fsync d9/d20/dac/dbd/f127 0 2024-04-22T15:49:47.754 INFO:tasks.workunit.client.1.smithi027.stdout:4/85: fdatasync d5/f6 0 2024-04-22T15:49:47.755 INFO:tasks.workunit.client.0.smithi016.stdout:5/920: symlink df/d15/d1a/d4e/d10a/l11c 0 2024-04-22T15:49:47.757 INFO:tasks.workunit.client.0.smithi016.stdout:2/950: mknod d2/d2c/de7/c167 0 2024-04-22T15:49:47.758 INFO:tasks.workunit.client.0.smithi016.stdout:2/951: chown d2/d1a/d7b 63058 1 2024-04-22T15:49:47.759 INFO:tasks.workunit.client.1.smithi027.stdout:4/86: creat d5/f23 x:0 0 0 2024-04-22T15:49:47.759 INFO:tasks.workunit.client.0.smithi016.stdout:9/954: write d9/d20/dac/dae/d126/f12a [304032,56772] 0 2024-04-22T15:49:47.759 INFO:tasks.workunit.client.0.smithi016.stdout:8/877: dwrite d2/d82/d56/f64 [0,4194304] 0 2024-04-22T15:49:47.763 INFO:tasks.workunit.client.0.smithi016.stdout:8/878: chown d2/d48/dad/d8f/fb3 13077 1 2024-04-22T15:49:47.763 INFO:tasks.workunit.client.1.smithi027.stdout:4/87: dread d5/fa [0,4194304] 0 2024-04-22T15:49:47.764 INFO:tasks.workunit.client.1.smithi027.stdout:4/88: chown d5/de/f21 2334 1 2024-04-22T15:49:47.764 INFO:tasks.workunit.client.0.smithi016.stdout:5/921: dread df/d15/d1a/d50/fe3 [0,4194304] 0 2024-04-22T15:49:47.768 INFO:tasks.workunit.client.1.smithi027.stdout:4/89: dread d5/f6 [0,4194304] 0 2024-04-22T15:49:47.769 INFO:tasks.workunit.client.0.smithi016.stdout:4/903: getdents d7/d4e/d102/d106/d88/d124 0 2024-04-22T15:49:47.769 INFO:tasks.workunit.client.1.smithi027.stdout:4/90: write d5/d7/f1f [823852,34016] 0 2024-04-22T15:49:47.770 INFO:tasks.workunit.client.0.smithi016.stdout:8/879: dread - d2/d82/f94 zero size 2024-04-22T15:49:47.770 INFO:tasks.workunit.client.1.smithi027.stdout:4/91: chown d5/de/f16 122426 1 2024-04-22T15:49:47.771 INFO:tasks.workunit.client.0.smithi016.stdout:8/880: chown d2/d10/d47/f95 2083841 1 2024-04-22T15:49:47.771 INFO:tasks.workunit.client.1.smithi027.stdout:4/92: chown d5/de/f16 114 1 2024-04-22T15:49:47.771 INFO:tasks.workunit.client.1.smithi027.stdout:4/93: read d5/f6 [3855546,108171] 0 2024-04-22T15:49:47.772 INFO:tasks.workunit.client.1.smithi027.stdout:4/94: stat d5/de 0 2024-04-22T15:49:47.773 INFO:tasks.workunit.client.0.smithi016.stdout:5/922: creat df/d90/f11d x:0 0 0 2024-04-22T15:49:47.777 INFO:tasks.workunit.client.0.smithi016.stdout:3/990: rename d7/d17/d18/d32/d3c/c11b to d7/d17/d18/d32/c12b 0 2024-04-22T15:49:47.778 INFO:tasks.workunit.client.1.smithi027.stdout:6/132: dwrite d5/db/f1e [0,4194304] 0 2024-04-22T15:49:47.778 INFO:tasks.workunit.client.0.smithi016.stdout:5/923: creat df/d15/d1a/dd8/f11e x:0 0 0 2024-04-22T15:49:47.780 INFO:tasks.workunit.client.0.smithi016.stdout:4/904: creat d7/d21/d45/dfc/f12b x:0 0 0 2024-04-22T15:49:47.781 INFO:tasks.workunit.client.1.smithi027.stdout:6/133: unlink d5/f1a 0 2024-04-22T15:49:47.787 INFO:tasks.workunit.client.0.smithi016.stdout:8/881: fsync d2/f7a 0 2024-04-22T15:49:47.794 INFO:tasks.workunit.client.0.smithi016.stdout:4/905: link d7/d4e/l129 d7/d21/d45/d49/de4/l12c 0 2024-04-22T15:49:47.794 INFO:tasks.workunit.client.0.smithi016.stdout:5/924: dwrite df/d15/d1a/d4e/d10a/f3b [0,4194304] 0 2024-04-22T15:49:47.795 INFO:tasks.workunit.client.0.smithi016.stdout:2/952: dwrite d2/f166 [0,4194304] 0 2024-04-22T15:49:47.796 INFO:tasks.workunit.client.0.smithi016.stdout:9/955: dread d9/d57/d91/f99 [0,4194304] 0 2024-04-22T15:49:47.797 INFO:tasks.workunit.client.1.smithi027.stdout:6/134: truncate d5/db/f1e 2483898 0 2024-04-22T15:49:47.797 INFO:tasks.workunit.client.1.smithi027.stdout:6/135: stat d5/dc/dd/f16 0 2024-04-22T15:49:47.798 INFO:tasks.workunit.client.0.smithi016.stdout:3/991: mknod d7/d17/d18/d32/d3c/d51/c12c 0 2024-04-22T15:49:47.798 INFO:tasks.workunit.client.1.smithi027.stdout:6/136: stat d5/dc/d17 0 2024-04-22T15:49:47.799 INFO:tasks.workunit.client.1.smithi027.stdout:6/137: fsync d5/dc/d20/d2b/f12 0 2024-04-22T15:49:47.802 INFO:tasks.workunit.client.0.smithi016.stdout:8/882: creat d2/d48/dad/f115 x:0 0 0 2024-04-22T15:49:47.815 INFO:tasks.workunit.client.0.smithi016.stdout:5/925: creat df/d15/d1a/d4e/f11f x:0 0 0 2024-04-22T15:49:47.816 INFO:tasks.workunit.client.0.smithi016.stdout:4/906: dwrite d7/d21/d45/d49/d9b/fdf [0,4194304] 0 2024-04-22T15:49:47.816 INFO:tasks.workunit.client.0.smithi016.stdout:2/953: creat d2/d4d/d150/d11b/f168 x:0 0 0 2024-04-22T15:49:47.821 INFO:tasks.workunit.client.1.smithi027.stdout:6/138: truncate d5/dc/d17/f1b 206312 0 2024-04-22T15:49:47.822 INFO:tasks.workunit.client.1.smithi027.stdout:6/139: dread - d5/f29 zero size 2024-04-22T15:49:47.825 INFO:tasks.workunit.client.1.smithi027.stdout:6/140: rename d5/dc/l15 to d5/dc/d20/l2c 0 2024-04-22T15:49:47.831 INFO:tasks.workunit.client.1.smithi027.stdout:6/141: dwrite d5/dc/d20/d2b/f12 [0,4194304] 0 2024-04-22T15:49:47.835 INFO:tasks.workunit.client.1.smithi027.stdout:6/142: creat d5/dc/d26/f2d x:0 0 0 2024-04-22T15:49:47.845 INFO:tasks.workunit.client.0.smithi016.stdout:3/992: fdatasync d7/d17/d18/d32/d3c/d47/f52 0 2024-04-22T15:49:47.846 INFO:tasks.workunit.client.0.smithi016.stdout:5/926: mknod df/d15/d1a/d50/d77/d94/c120 0 2024-04-22T15:49:47.847 INFO:tasks.workunit.client.1.smithi027.stdout:9/93: sync 2024-04-22T15:49:47.847 INFO:tasks.workunit.client.0.smithi016.stdout:5/927: readlink df/d5b/db3/dc1/lea 0 2024-04-22T15:49:47.849 INFO:tasks.workunit.client.0.smithi016.stdout:9/956: rmdir d9/d37/dea/d11b 0 2024-04-22T15:49:47.850 INFO:tasks.workunit.client.0.smithi016.stdout:3/993: symlink d7/d17/d18/d32/d42/d63/db4/d11a/d11c/d11e/l12d 0 2024-04-22T15:49:47.851 INFO:tasks.workunit.client.1.smithi027.stdout:9/94: creat d0/dc/d12/f23 x:0 0 0 2024-04-22T15:49:47.856 INFO:tasks.workunit.client.0.smithi016.stdout:9/957: dwrite d9/de/fdf [4194304,4194304] 0 2024-04-22T15:49:47.856 INFO:tasks.workunit.client.0.smithi016.stdout:3/994: symlink d7/d17/d18/d32/d42/d63/deb/d21/d55/ddb/l12e 0 2024-04-22T15:49:47.857 INFO:tasks.workunit.client.1.smithi027.stdout:9/95: creat d0/dc/de/f24 x:0 0 0 2024-04-22T15:49:47.858 INFO:tasks.workunit.client.1.smithi027.stdout:9/96: chown d0/d8/f18 93813 1 2024-04-22T15:49:47.859 INFO:tasks.workunit.client.0.smithi016.stdout:5/928: truncate df/d15/d1a/f78 3566789 0 2024-04-22T15:49:47.860 INFO:tasks.workunit.client.0.smithi016.stdout:5/929: dread - df/d15/d1a/d4e/f10e zero size 2024-04-22T15:49:47.861 INFO:tasks.workunit.client.1.smithi027.stdout:9/97: dread d0/fb [0,4194304] 0 2024-04-22T15:49:47.862 INFO:tasks.workunit.client.0.smithi016.stdout:3/995: creat d7/d17/d18/d32/d42/d63/db4/f12f x:0 0 0 2024-04-22T15:49:47.863 INFO:tasks.workunit.client.0.smithi016.stdout:9/958: rename d9/d57/d91/de4/ffe to d9/d5d/db8/d122/f13c 0 2024-04-22T15:49:47.869 INFO:tasks.workunit.client.1.smithi027.stdout:9/98: write d0/fb [7907646,75772] 0 2024-04-22T15:49:47.871 INFO:tasks.workunit.client.0.smithi016.stdout:9/959: symlink d9/d5d/db8/d122/l13d 0 2024-04-22T15:49:47.871 INFO:tasks.workunit.client.0.smithi016.stdout:2/954: dread d2/d23/f3b [0,4194304] 0 2024-04-22T15:49:47.872 INFO:tasks.workunit.client.0.smithi016.stdout:9/960: stat d9/de/d23/f78 0 2024-04-22T15:49:47.875 INFO:tasks.workunit.client.0.smithi016.stdout:3/996: mknod d7/d17/d18/c130 0 2024-04-22T15:49:47.877 INFO:tasks.workunit.client.0.smithi016.stdout:9/961: creat d9/d20/dac/dbd/f13e x:0 0 0 2024-04-22T15:49:47.878 INFO:tasks.workunit.client.0.smithi016.stdout:7/887: sync 2024-04-22T15:49:47.878 INFO:tasks.workunit.client.0.smithi016.stdout:6/931: sync 2024-04-22T15:49:47.879 INFO:tasks.workunit.client.0.smithi016.stdout:2/955: link d2/d2c/d97/d67/f75 d2/d23/d92/d120/f169 0 2024-04-22T15:49:47.880 INFO:tasks.workunit.client.0.smithi016.stdout:9/962: mknod d9/d20/dac/d106/c13f 0 2024-04-22T15:49:47.880 INFO:tasks.workunit.client.0.smithi016.stdout:6/932: dread d3/da/f69 [0,4194304] 0 2024-04-22T15:49:47.882 INFO:tasks.workunit.client.0.smithi016.stdout:3/997: rename d7/d17/d18/d32/d42/d63/d86/dd8/l7b to d7/d17/d18/d32/d42/d63/deb/d21/d55/dac/l131 0 2024-04-22T15:49:47.883 INFO:tasks.workunit.client.0.smithi016.stdout:9/963: chown d9/d57/d91/de4/d25/d36/c138 187 1 2024-04-22T15:49:47.883 INFO:tasks.workunit.client.0.smithi016.stdout:7/888: truncate d3/d11a/f90 925848 0 2024-04-22T15:49:47.883 INFO:tasks.workunit.client.0.smithi016.stdout:9/964: readlink d9/l13 0 2024-04-22T15:49:47.884 INFO:tasks.workunit.client.0.smithi016.stdout:6/933: mknod d3/da/dd/d7b/d12c/d10c/c132 0 2024-04-22T15:49:47.884 INFO:tasks.workunit.client.0.smithi016.stdout:9/965: write d9/d57/d91/f110 [487414,113755] 0 2024-04-22T15:49:47.886 INFO:tasks.workunit.client.0.smithi016.stdout:3/998: creat d7/d17/d18/d32/d8a/d116/f132 x:0 0 0 2024-04-22T15:49:47.889 INFO:tasks.workunit.client.0.smithi016.stdout:7/889: mkdir d3/dc/d1b/d31/d13f 0 2024-04-22T15:49:47.891 INFO:tasks.workunit.client.0.smithi016.stdout:2/956: rename d2/d1a/d4c/d77/lb1 to d2/d1a/d4c/d77/l16a 0 2024-04-22T15:49:47.891 INFO:tasks.workunit.client.0.smithi016.stdout:9/966: dread d9/d20/dac/dbd/ff8 [0,4194304] 0 2024-04-22T15:49:47.892 INFO:tasks.workunit.client.0.smithi016.stdout:6/934: truncate d3/da/d12/d13/f27 796686 0 2024-04-22T15:49:47.892 INFO:tasks.workunit.client.0.smithi016.stdout:9/967: dread - d9/de/d26/d5a/f131 zero size 2024-04-22T15:49:47.893 INFO:tasks.workunit.client.0.smithi016.stdout:3/999: symlink d7/d17/d18/d32/d42/d63/d86/dd4/l133 0 2024-04-22T15:49:47.894 INFO:tasks.workunit.client.0.smithi016.stdout:7/890: creat d3/dc/d33/f140 x:0 0 0 2024-04-22T15:49:47.895 INFO:tasks.workunit.client.0.smithi016.stdout:4/907: write d7/d22/f54 [5117499,104081] 0 2024-04-22T15:49:47.896 INFO:tasks.workunit.client.0.smithi016.stdout:6/935: chown d3/f8 917161255 1 2024-04-22T15:49:47.897 INFO:tasks.workunit.client.0.smithi016.stdout:7/891: mknod d3/d11f/c141 0 2024-04-22T15:49:47.897 INFO:tasks.workunit.client.0.smithi016.stdout:4/908: fdatasync d7/d22/fc9 0 2024-04-22T15:49:47.899 INFO:tasks.workunit.client.0.smithi016.stdout:6/936: dread d3/da/f69 [0,4194304] 0 2024-04-22T15:49:47.902 INFO:tasks.workunit.client.0.smithi016.stdout:9/968: dwrite d9/d20/dac/dbd/f13e [0,4194304] 0 2024-04-22T15:49:47.903 INFO:tasks.workunit.client.1.smithi027.stdout:6/143: dwrite d5/dc/d17/f1b [0,4194304] 0 2024-04-22T15:49:47.910 INFO:tasks.workunit.client.0.smithi016.stdout:9/969: creat d9/f140 x:0 0 0 2024-04-22T15:49:47.910 INFO:tasks.workunit.client.0.smithi016.stdout:4/909: dwrite d7/d4e/d2f/f128 [0,4194304] 0 2024-04-22T15:49:47.911 INFO:tasks.workunit.client.0.smithi016.stdout:6/937: read d3/da/d12/d13/dd2/dee/fa9 [563364,68537] 0 2024-04-22T15:49:47.912 INFO:tasks.workunit.client.1.smithi027.stdout:6/144: dwrite d5/dc/dd/f16 [0,4194304] 0 2024-04-22T15:49:47.912 INFO:tasks.workunit.client.0.smithi016.stdout:5/930: truncate df/d15/d1a/d50/d77/d7d/d8b/d8c/ff7 5066266 0 2024-04-22T15:49:47.913 INFO:tasks.workunit.client.1.smithi027.stdout:9/99: write d0/dc/de/f14 [4794647,20540] 0 2024-04-22T15:49:47.914 INFO:tasks.workunit.client.1.smithi027.stdout:9/100: chown d0/d8/l1f 11483901 1 2024-04-22T15:49:47.920 INFO:tasks.workunit.client.1.smithi027.stdout:6/145: rename d5/d9 to d5/db/d2e 0 2024-04-22T15:49:47.921 INFO:tasks.workunit.client.0.smithi016.stdout:6/938: chown d3/da/db/d14/d16/cc6 1186969157 1 2024-04-22T15:49:47.922 INFO:tasks.workunit.client.0.smithi016.stdout:6/939: chown d3/da/d12/ff 36113035 1 2024-04-22T15:49:47.923 INFO:tasks.workunit.client.0.smithi016.stdout:6/940: readlink d3/da/db/d44/l47 0 2024-04-22T15:49:47.929 INFO:tasks.workunit.client.0.smithi016.stdout:6/941: creat d3/da/d12/d13/f133 x:0 0 0 2024-04-22T15:49:47.931 INFO:tasks.workunit.client.1.smithi027.stdout:9/101: rename d0/dc/de/f14 to d0/dc/f25 0 2024-04-22T15:49:47.937 INFO:tasks.workunit.client.0.smithi016.stdout:4/910: link d7/d21/d45/d49/d67/l78 d7/d24/d41/d121/l12d 0 2024-04-22T15:49:47.940 INFO:tasks.workunit.client.1.smithi027.stdout:9/102: dwrite d0/d8/f18 [0,4194304] 0 2024-04-22T15:49:47.942 INFO:tasks.workunit.client.1.smithi027.stdout:9/103: write d0/dc/de/f24 [803012,16170] 0 2024-04-22T15:49:47.942 INFO:tasks.workunit.client.1.smithi027.stdout:9/104: readlink d0/l2 0 2024-04-22T15:49:47.944 INFO:tasks.workunit.client.1.smithi027.stdout:9/105: write d0/dc/d12/f23 [677178,35331] 0 2024-04-22T15:49:47.944 INFO:tasks.workunit.client.0.smithi016.stdout:6/942: getdents d3/da/db/d44/d68 0 2024-04-22T15:49:47.945 INFO:tasks.workunit.client.0.smithi016.stdout:6/943: readlink d3/da/db/d14/d16/l2f 0 2024-04-22T15:49:47.948 INFO:tasks.workunit.client.0.smithi016.stdout:6/944: fsync d3/da/f69 0 2024-04-22T15:49:47.953 INFO:tasks.workunit.client.1.smithi027.stdout:9/106: dwrite d0/dc/f22 [0,4194304] 0 2024-04-22T15:49:47.953 INFO:tasks.workunit.client.0.smithi016.stdout:6/945: creat d3/da/ddd/d101/d11b/dab/f134 x:0 0 0 2024-04-22T15:49:47.956 INFO:tasks.workunit.client.0.smithi016.stdout:7/892: dwrite d3/dc/d1b/d5a/d9/df/d94/ddf/fe2 [0,4194304] 0 2024-04-22T15:49:47.958 INFO:tasks.workunit.client.0.smithi016.stdout:9/970: creat d9/d57/d91/dc1/f141 x:0 0 0 2024-04-22T15:49:47.958 INFO:tasks.workunit.client.0.smithi016.stdout:7/893: readlink d3/dc/d9f/le9 0 2024-04-22T15:49:47.960 INFO:tasks.workunit.client.0.smithi016.stdout:6/946: dread - d3/da/db/d14/d4c/fea zero size 2024-04-22T15:49:47.963 INFO:tasks.workunit.client.0.smithi016.stdout:7/894: mknod d3/d11f/c142 0 2024-04-22T15:49:47.964 INFO:tasks.workunit.client.0.smithi016.stdout:5/931: truncate df/d15/d1a/d4e/d10a/f3b 637773 0 2024-04-22T15:49:47.969 INFO:tasks.workunit.client.0.smithi016.stdout:6/947: truncate d3/da/dd/fb5 928616 0 2024-04-22T15:49:47.969 INFO:tasks.workunit.client.0.smithi016.stdout:6/948: readlink d3/da/dd/d67/l121 0 2024-04-22T15:49:47.969 INFO:tasks.workunit.client.0.smithi016.stdout:7/895: symlink d3/dc/d9f/l143 0 2024-04-22T15:49:47.970 INFO:tasks.workunit.client.1.smithi027.stdout:9/107: getdents d0/dc/de 0 2024-04-22T15:49:47.971 INFO:tasks.workunit.client.0.smithi016.stdout:9/971: dwrite d9/d57/d91/de4/f2e [4194304,4194304] 0 2024-04-22T15:49:47.973 INFO:tasks.workunit.client.0.smithi016.stdout:8/883: sync 2024-04-22T15:49:47.976 INFO:tasks.workunit.client.0.smithi016.stdout:5/932: fsync df/d5b/f98 0 2024-04-22T15:49:47.977 INFO:tasks.workunit.client.0.smithi016.stdout:5/933: chown df/d5b/d5d/deb 22642 1 2024-04-22T15:49:47.978 INFO:tasks.workunit.client.0.smithi016.stdout:9/972: read d9/de/d23/fa5 [306288,62671] 0 2024-04-22T15:49:47.980 INFO:tasks.workunit.client.0.smithi016.stdout:8/884: rename d2/d82/d52/df0/d66/f20 to d2/d48/dad/d8f/d96/f116 0 2024-04-22T15:49:47.983 INFO:tasks.workunit.client.0.smithi016.stdout:5/934: creat df/d15/d1a/d50/d77/d7d/d8b/d93/dfa/d103/f121 x:0 0 0 2024-04-22T15:49:47.988 INFO:tasks.workunit.client.0.smithi016.stdout:7/896: dread d3/dc/d22/f39 [8388608,4194304] 0 2024-04-22T15:49:47.988 INFO:tasks.workunit.client.0.smithi016.stdout:7/897: chown d3 64693 1 2024-04-22T15:49:47.989 INFO:tasks.workunit.client.0.smithi016.stdout:5/935: symlink df/d15/d1a/d4e/d58/l122 0 2024-04-22T15:49:47.991 INFO:tasks.workunit.client.0.smithi016.stdout:5/936: write df/d5b/d5d/ff2 [785764,31732] 0 2024-04-22T15:49:48.000 INFO:tasks.workunit.client.0.smithi016.stdout:7/898: write d3/dc/d1b/d5a/d4a/d6e/ffb [57424,33258] 0 2024-04-22T15:49:48.004 INFO:tasks.workunit.client.0.smithi016.stdout:7/899: rmdir d3/dc/d1b/d5a/d9/df/d94 39 2024-04-22T15:49:48.006 INFO:tasks.workunit.client.0.smithi016.stdout:5/937: dread df/d15/d1a/f71 [0,4194304] 0 2024-04-22T15:49:48.007 INFO:tasks.workunit.client.0.smithi016.stdout:8/885: dread d2/d82/d52/df0/d66/d7e/dc0/fcd [0,4194304] 0 2024-04-22T15:49:48.009 INFO:tasks.workunit.client.0.smithi016.stdout:6/949: dwrite d3/da/d12/d13/f27 [0,4194304] 0 2024-04-22T15:49:48.009 INFO:tasks.workunit.client.0.smithi016.stdout:8/886: chown d2/d30 2498 1 2024-04-22T15:49:48.013 INFO:tasks.workunit.client.0.smithi016.stdout:8/887: creat d2/d82/d52/df0/d66/d1c/d65/f117 x:0 0 0 2024-04-22T15:49:48.014 INFO:tasks.workunit.client.0.smithi016.stdout:5/938: truncate df/d15/d1a/d50/fac 3130597 0 2024-04-22T15:49:48.015 INFO:tasks.workunit.client.0.smithi016.stdout:9/973: dread d9/de/f28 [0,4194304] 0 2024-04-22T15:49:48.016 INFO:tasks.workunit.client.0.smithi016.stdout:6/950: write d3/da/db/d14/d4c/f52 [2937559,129064] 0 2024-04-22T15:49:48.021 INFO:tasks.workunit.client.0.smithi016.stdout:7/900: dwrite d3/dc/d1b/d31/f48 [0,4194304] 0 2024-04-22T15:49:48.021 INFO:tasks.workunit.client.0.smithi016.stdout:9/974: write d9/d5d/db8/d122/f13c [1373238,4491] 0 2024-04-22T15:49:48.022 INFO:tasks.workunit.client.0.smithi016.stdout:5/939: creat df/d15/d1a/d50/d77/d7d/d8b/d8c/f123 x:0 0 0 2024-04-22T15:49:48.022 INFO:tasks.workunit.client.0.smithi016.stdout:8/888: dread d2/d10/d43/da9/fac [0,4194304] 0 2024-04-22T15:49:48.022 INFO:tasks.workunit.client.1.smithi027.stdout:2/77: sync 2024-04-22T15:49:48.022 INFO:tasks.workunit.client.1.smithi027.stdout:3/109: sync 2024-04-22T15:49:48.023 INFO:tasks.workunit.client.0.smithi016.stdout:5/940: readlink df/d15/d1a/d50/d77/l48 0 2024-04-22T15:49:48.023 INFO:tasks.workunit.client.0.smithi016.stdout:7/901: write d3/dc/d33/f7f [1311169,120937] 0 2024-04-22T15:49:48.024 INFO:tasks.workunit.client.0.smithi016.stdout:8/889: read - d2/d82/d52/df0/d66/d1c/d65/def/ff2 zero size 2024-04-22T15:49:48.026 INFO:tasks.workunit.client.0.smithi016.stdout:7/902: chown d3/dc/d1b/d5a/d9/df/c29 32573 1 2024-04-22T15:49:48.026 INFO:tasks.workunit.client.0.smithi016.stdout:5/941: chown df/d15/f4a 64 1 2024-04-22T15:49:48.027 INFO:tasks.workunit.client.0.smithi016.stdout:2/957: sync 2024-04-22T15:49:48.028 INFO:tasks.workunit.client.1.smithi027.stdout:3/110: link f10 f26 0 2024-04-22T15:49:48.029 INFO:tasks.workunit.client.1.smithi027.stdout:3/111: fdatasync f21 0 2024-04-22T15:49:48.029 INFO:tasks.workunit.client.0.smithi016.stdout:8/890: fsync d2/d48/dad/d8f/fb8 0 2024-04-22T15:49:48.030 INFO:tasks.workunit.client.1.smithi027.stdout:3/112: fdatasync ff 0 2024-04-22T15:49:48.030 INFO:tasks.workunit.client.1.smithi027.stdout:2/78: rmdir dd/d14 0 2024-04-22T15:49:48.031 INFO:tasks.workunit.client.0.smithi016.stdout:2/958: read d2/de/d2a/d59/fbe [3474004,111910] 0 2024-04-22T15:49:48.032 INFO:tasks.workunit.client.0.smithi016.stdout:9/975: symlink d9/d20/dac/d106/l142 0 2024-04-22T15:49:48.035 INFO:tasks.workunit.client.0.smithi016.stdout:8/891: dread d2/d82/d56/f71 [0,4194304] 0 2024-04-22T15:49:48.038 INFO:tasks.workunit.client.0.smithi016.stdout:5/942: unlink df/d15/d1a/d50/d77/fce 0 2024-04-22T15:49:48.041 INFO:tasks.workunit.client.0.smithi016.stdout:8/892: write d2/d82/d52/df0/d66/d1c/d65/fbd [1122716,5351] 0 2024-04-22T15:49:48.043 INFO:tasks.workunit.client.1.smithi027.stdout:2/79: dwrite dd/df/f12 [0,4194304] 0 2024-04-22T15:49:48.043 INFO:tasks.workunit.client.0.smithi016.stdout:2/959: truncate d2/d2c/d97/f32 405579 0 2024-04-22T15:49:48.045 INFO:tasks.workunit.client.0.smithi016.stdout:9/976: dwrite d9/d20/dac/dbd/f127 [0,4194304] 0 2024-04-22T15:49:48.048 INFO:tasks.workunit.client.0.smithi016.stdout:9/977: chown d9/de/d23/f78 4604547 1 2024-04-22T15:49:48.053 INFO:tasks.workunit.client.0.smithi016.stdout:7/903: write d3/dc/d33/f3e [5135513,111543] 0 2024-04-22T15:49:48.055 INFO:tasks.workunit.client.1.smithi027.stdout:3/113: dwrite f26 [0,4194304] 0 2024-04-22T15:49:48.056 INFO:tasks.workunit.client.1.smithi027.stdout:3/114: chown f16 6686 1 2024-04-22T15:49:48.056 INFO:tasks.workunit.client.0.smithi016.stdout:5/943: read df/d15/d1a/fd2 [88881,19588] 0 2024-04-22T15:49:48.057 INFO:tasks.workunit.client.0.smithi016.stdout:7/904: chown d3/dc/d1b/d31/d88/c10e 22 1 2024-04-22T15:49:48.058 INFO:tasks.workunit.client.0.smithi016.stdout:6/951: dread d3/da/dd/d7b/d12c/d10c/f120 [0,4194304] 0 2024-04-22T15:49:48.060 INFO:tasks.workunit.client.0.smithi016.stdout:9/978: write d9/de/d26/d5a/fc7 [1011332,44469] 0 2024-04-22T15:49:48.061 INFO:tasks.workunit.client.1.smithi027.stdout:3/115: dread f10 [0,4194304] 0 2024-04-22T15:49:48.064 INFO:tasks.workunit.client.0.smithi016.stdout:7/905: creat d3/dc/d33/d84/de3/f144 x:0 0 0 2024-04-22T15:49:48.064 INFO:tasks.workunit.client.1.smithi027.stdout:3/116: symlink l27 0 2024-04-22T15:49:48.068 INFO:tasks.workunit.client.1.smithi027.stdout:3/117: dread f21 [0,4194304] 0 2024-04-22T15:49:48.068 INFO:tasks.workunit.client.0.smithi016.stdout:8/893: dwrite d2/d82/d52/df0/d66/d7e/d85/de8/f10e [0,4194304] 0 2024-04-22T15:49:48.069 INFO:tasks.workunit.client.0.smithi016.stdout:9/979: creat d9/d57/d91/dc1/f143 x:0 0 0 2024-04-22T15:49:48.071 INFO:tasks.workunit.client.0.smithi016.stdout:6/952: dwrite d3/da/dd/f11f [0,4194304] 0 2024-04-22T15:49:48.071 INFO:tasks.workunit.client.1.smithi027.stdout:3/118: dread f1d [0,4194304] 0 2024-04-22T15:49:48.073 INFO:tasks.workunit.client.1.smithi027.stdout:2/80: truncate dd/df/f12 2447012 0 2024-04-22T15:49:48.075 INFO:tasks.workunit.client.0.smithi016.stdout:8/894: creat d2/d32/f118 x:0 0 0 2024-04-22T15:49:48.081 INFO:tasks.workunit.client.0.smithi016.stdout:6/953: dwrite d3/da/dd/d1b/f8b [0,4194304] 0 2024-04-22T15:49:48.085 INFO:tasks.workunit.client.0.smithi016.stdout:5/944: dread df/d15/d1a/d50/d77/d7a/fcb [0,4194304] 0 2024-04-22T15:49:48.086 INFO:tasks.workunit.client.0.smithi016.stdout:7/906: creat d3/dc/f145 x:0 0 0 2024-04-22T15:49:48.089 INFO:tasks.workunit.client.1.smithi027.stdout:2/81: fsync dd/f11 0 2024-04-22T15:49:48.093 INFO:tasks.workunit.client.1.smithi027.stdout:2/82: dread dd/f11 [4194304,4194304] 0 2024-04-22T15:49:48.093 INFO:tasks.workunit.client.0.smithi016.stdout:5/945: symlink df/d15/d1a/d50/d77/d7d/d8b/d93/dfa/d103/l124 0 2024-04-22T15:49:48.096 INFO:tasks.workunit.client.0.smithi016.stdout:9/980: write d9/d57/d91/de4/d25/da6/f79 [1131710,75513] 0 2024-04-22T15:49:48.097 INFO:tasks.workunit.client.0.smithi016.stdout:7/907: symlink d3/dc/d1b/d5a/d9/df/l146 0 2024-04-22T15:49:48.099 INFO:tasks.workunit.client.0.smithi016.stdout:5/946: unlink df/d15/d1a/d50/d77/c37 0 2024-04-22T15:49:48.100 INFO:tasks.workunit.client.1.smithi027.stdout:1/76: sync 2024-04-22T15:49:48.100 INFO:tasks.workunit.client.1.smithi027.stdout:7/71: sync 2024-04-22T15:49:48.100 INFO:tasks.workunit.client.1.smithi027.stdout:0/116: sync 2024-04-22T15:49:48.100 INFO:tasks.workunit.client.1.smithi027.stdout:5/100: sync 2024-04-22T15:49:48.100 INFO:tasks.workunit.client.1.smithi027.stdout:8/121: sync 2024-04-22T15:49:48.100 INFO:tasks.workunit.client.1.smithi027.stdout:4/95: sync 2024-04-22T15:49:48.101 INFO:tasks.workunit.client.1.smithi027.stdout:5/101: write d0/f8 [963882,82511] 0 2024-04-22T15:49:48.102 INFO:tasks.workunit.client.1.smithi027.stdout:8/122: chown d1/f21 591193348 1 2024-04-22T15:49:48.102 INFO:tasks.workunit.client.1.smithi027.stdout:2/83: rename l4 to dd/df/l15 0 2024-04-22T15:49:48.102 INFO:tasks.workunit.client.1.smithi027.stdout:7/72: chown d0/d11/fe 8101 1 2024-04-22T15:49:48.103 INFO:tasks.workunit.client.1.smithi027.stdout:5/102: dread - d0/d11/f16 zero size 2024-04-22T15:49:48.104 INFO:tasks.workunit.client.0.smithi016.stdout:9/981: symlink d9/d57/d91/dc1/da2/l144 0 2024-04-22T15:49:48.104 INFO:tasks.workunit.client.1.smithi027.stdout:6/146: sync 2024-04-22T15:49:48.105 INFO:tasks.workunit.client.0.smithi016.stdout:4/911: sync 2024-04-22T15:49:48.105 INFO:tasks.workunit.client.0.smithi016.stdout:7/908: rmdir d3/dc/d33/d84/de3 39 2024-04-22T15:49:48.109 INFO:tasks.workunit.client.1.smithi027.stdout:0/117: symlink d0/d1a/l27 0 2024-04-22T15:49:48.110 INFO:tasks.workunit.client.0.smithi016.stdout:6/954: getdents d3/da/db/d14 0 2024-04-22T15:49:48.111 INFO:tasks.workunit.client.1.smithi027.stdout:1/77: symlink d2/l12 0 2024-04-22T15:49:48.111 INFO:tasks.workunit.client.0.smithi016.stdout:6/955: chown d3/da/f72 44195985 1 2024-04-22T15:49:48.112 INFO:tasks.workunit.client.1.smithi027.stdout:3/119: dwrite f15 [0,4194304] 0 2024-04-22T15:49:48.113 INFO:tasks.workunit.client.1.smithi027.stdout:7/73: creat d0/f15 x:0 0 0 2024-04-22T15:49:48.116 INFO:tasks.workunit.client.1.smithi027.stdout:2/84: mknod dd/df/c16 0 2024-04-22T15:49:48.118 INFO:tasks.workunit.client.1.smithi027.stdout:7/74: dread d0/d11/fe [0,4194304] 0 2024-04-22T15:49:48.121 INFO:tasks.workunit.client.1.smithi027.stdout:5/103: creat d0/f21 x:0 0 0 2024-04-22T15:49:48.122 INFO:tasks.workunit.client.0.smithi016.stdout:9/982: mknod d9/de/c145 0 2024-04-22T15:49:48.123 INFO:tasks.workunit.client.1.smithi027.stdout:6/147: write d5/db/d2e/ff [1544101,30562] 0 2024-04-22T15:49:48.123 INFO:tasks.workunit.client.0.smithi016.stdout:6/956: dread - d3/da/dd/d67/d6c/fa6 zero size 2024-04-22T15:49:48.123 INFO:tasks.workunit.client.1.smithi027.stdout:2/85: dwrite f7 [0,4194304] 0 2024-04-22T15:49:48.124 INFO:tasks.workunit.client.1.smithi027.stdout:4/96: mkdir d5/d11/d1d/d24 0 2024-04-22T15:49:48.130 INFO:tasks.workunit.client.1.smithi027.stdout:6/148: dread d5/db/d2e/ff [0,4194304] 0 2024-04-22T15:49:48.133 INFO:tasks.workunit.client.0.smithi016.stdout:6/957: stat d3/da/ddd/d101/d11b/c41 0 2024-04-22T15:49:48.135 INFO:tasks.workunit.client.0.smithi016.stdout:4/912: link d7/d4e/d2f/d33/cb0 d7/d22/dc7/c12e 0 2024-04-22T15:49:48.138 INFO:tasks.workunit.client.0.smithi016.stdout:7/909: creat d3/dc/d33/d84/f147 x:0 0 0 2024-04-22T15:49:48.139 INFO:tasks.workunit.client.1.smithi027.stdout:0/118: dwrite d0/f22 [0,4194304] 0 2024-04-22T15:49:48.139 INFO:tasks.workunit.client.1.smithi027.stdout:6/149: dread d5/db/f1e [0,4194304] 0 2024-04-22T15:49:48.140 INFO:tasks.workunit.client.0.smithi016.stdout:5/947: getdents df/d15/d1a/dd8 0 2024-04-22T15:49:48.141 INFO:tasks.workunit.client.1.smithi027.stdout:1/78: dread d2/f6 [0,4194304] 0 2024-04-22T15:49:48.142 INFO:tasks.workunit.client.0.smithi016.stdout:5/948: write df/d5b/fc4 [2455077,111829] 0 2024-04-22T15:49:48.144 INFO:tasks.workunit.client.1.smithi027.stdout:2/86: fdatasync f7 0 2024-04-22T15:49:48.145 INFO:tasks.workunit.client.1.smithi027.stdout:5/104: mkdir d0/d1a/d22 0 2024-04-22T15:49:48.147 INFO:tasks.workunit.client.0.smithi016.stdout:9/983: write d9/de/d23/fa5 [4529276,95293] 0 2024-04-22T15:49:48.148 INFO:tasks.workunit.client.0.smithi016.stdout:7/910: truncate d3/dc/d33/f44 2284178 0 2024-04-22T15:49:48.149 INFO:tasks.workunit.client.1.smithi027.stdout:3/120: write f1d [1096912,94188] 0 2024-04-22T15:49:48.149 INFO:tasks.workunit.client.1.smithi027.stdout:4/97: creat d5/f25 x:0 0 0 2024-04-22T15:49:48.153 INFO:tasks.workunit.client.0.smithi016.stdout:5/949: rename df/d15/d1a/d50/d77/d7d/d8b/d8c/ddf/c102 to df/d15/d1a/d50/d77/d7d/c125 0 2024-04-22T15:49:48.154 INFO:tasks.workunit.client.1.smithi027.stdout:8/123: truncate d1/fb 863726 0 2024-04-22T15:49:48.155 INFO:tasks.workunit.client.0.smithi016.stdout:5/950: fsync df/d15/d1a/d4e/d58/fb8 0 2024-04-22T15:49:48.155 INFO:tasks.workunit.client.1.smithi027.stdout:8/124: fdatasync d1/f7 0 2024-04-22T15:49:48.156 INFO:tasks.workunit.client.0.smithi016.stdout:9/984: dwrite d9/de/d23/fdc [0,4194304] 0 2024-04-22T15:49:48.156 INFO:tasks.workunit.client.0.smithi016.stdout:4/913: creat d7/d3c/d96/dca/f12f x:0 0 0 2024-04-22T15:49:48.157 INFO:tasks.workunit.client.1.smithi027.stdout:3/121: dwrite f1f [0,4194304] 0 2024-04-22T15:49:48.158 INFO:tasks.workunit.client.0.smithi016.stdout:7/911: creat d3/dc/d9f/f148 x:0 0 0 2024-04-22T15:49:48.159 INFO:tasks.workunit.client.0.smithi016.stdout:9/985: chown d9/d57/d91/de4/f2f 134 1 2024-04-22T15:49:48.159 INFO:tasks.workunit.client.1.smithi027.stdout:0/119: write d0/f12 [1393612,47894] 0 2024-04-22T15:49:48.162 INFO:tasks.workunit.client.1.smithi027.stdout:6/150: symlink d5/db/l2f 0 2024-04-22T15:49:48.166 INFO:tasks.workunit.client.0.smithi016.stdout:9/986: dwrite d9/d57/d91/de4/f2e [4194304,4194304] 0 2024-04-22T15:49:48.176 INFO:tasks.workunit.client.1.smithi027.stdout:8/125: symlink d1/l2e 0 2024-04-22T15:49:48.178 INFO:tasks.workunit.client.1.smithi027.stdout:3/122: mkdir d28 0 2024-04-22T15:49:48.178 INFO:tasks.workunit.client.0.smithi016.stdout:4/914: dread d7/d24/d41/f57 [0,4194304] 0 2024-04-22T15:49:48.179 INFO:tasks.workunit.client.0.smithi016.stdout:4/915: write d7/d4e/d102/d106/de2/ff7 [753727,53129] 0 2024-04-22T15:49:48.182 INFO:tasks.workunit.client.1.smithi027.stdout:6/151: write d5/db/f1e [1499107,79607] 0 2024-04-22T15:49:48.183 INFO:tasks.workunit.client.1.smithi027.stdout:6/152: read - d5/f29 zero size 2024-04-22T15:49:48.183 INFO:tasks.workunit.client.0.smithi016.stdout:5/951: mkdir df/d15/d1a/d28/d126 0 2024-04-22T15:49:48.184 INFO:tasks.workunit.client.1.smithi027.stdout:1/79: mknod d2/c13 0 2024-04-22T15:49:48.187 INFO:tasks.workunit.client.0.smithi016.stdout:9/987: rename d9/d57/d91/dc1/f143 to d9/d37/dea/f146 0 2024-04-22T15:49:48.188 INFO:tasks.workunit.client.1.smithi027.stdout:1/80: dread d2/f6 [0,4194304] 0 2024-04-22T15:49:48.188 INFO:tasks.workunit.client.0.smithi016.stdout:4/916: creat d7/d4e/d81/f130 x:0 0 0 2024-04-22T15:49:48.194 INFO:tasks.workunit.client.1.smithi027.stdout:4/98: link d5/d7/f15 d5/f26 0 2024-04-22T15:49:48.196 INFO:tasks.workunit.client.1.smithi027.stdout:8/126: rename d1/d3/f27 to d1/d1e/f2f 0 2024-04-22T15:49:48.197 INFO:tasks.workunit.client.1.smithi027.stdout:2/87: dwrite dd/f11 [0,4194304] 0 2024-04-22T15:49:48.198 INFO:tasks.workunit.client.1.smithi027.stdout:2/88: dread - dd/f13 zero size 2024-04-22T15:49:48.202 INFO:tasks.workunit.client.0.smithi016.stdout:4/917: creat d7/d4e/d102/d106/d88/d9d/dad/db9/f131 x:0 0 0 2024-04-22T15:49:48.203 INFO:tasks.workunit.client.0.smithi016.stdout:2/960: sync 2024-04-22T15:49:48.204 INFO:tasks.workunit.client.0.smithi016.stdout:2/961: fsync d2/d1a/f164 0 2024-04-22T15:49:48.205 INFO:tasks.workunit.client.0.smithi016.stdout:2/962: fsync d2/d4d/f7f 0 2024-04-22T15:49:48.206 INFO:tasks.workunit.client.0.smithi016.stdout:9/988: symlink d9/d20/dac/l147 0 2024-04-22T15:49:48.207 INFO:tasks.workunit.client.1.smithi027.stdout:5/105: rmdir d0/df/de/d20 0 2024-04-22T15:49:48.208 INFO:tasks.workunit.client.0.smithi016.stdout:9/989: read f7 [3912417,41469] 0 2024-04-22T15:49:48.211 INFO:tasks.workunit.client.1.smithi027.stdout:4/99: rename d5/fa to d5/d11/d1d/f27 0 2024-04-22T15:49:48.211 INFO:tasks.workunit.client.1.smithi027.stdout:5/106: rename d0/d11/d1c to d0/d11/d1c/d23 22 2024-04-22T15:49:48.212 INFO:tasks.workunit.client.1.smithi027.stdout:5/107: chown d0/df/de/f13 14163 1 2024-04-22T15:49:48.214 INFO:tasks.workunit.client.1.smithi027.stdout:5/108: dread - d0/d11/f16 zero size 2024-04-22T15:49:48.219 INFO:tasks.workunit.client.0.smithi016.stdout:2/963: read d2/d2c/d97/f31 [829691,81441] 0 2024-04-22T15:49:48.219 INFO:tasks.workunit.client.1.smithi027.stdout:9/108: sync 2024-04-22T15:49:48.220 INFO:tasks.workunit.client.1.smithi027.stdout:4/100: dwrite d5/de/f21 [0,4194304] 0 2024-04-22T15:49:48.221 INFO:tasks.workunit.client.1.smithi027.stdout:9/109: truncate d0/dc/de/f24 857597 0 2024-04-22T15:49:48.221 INFO:tasks.workunit.client.1.smithi027.stdout:2/89: chown la 20555213 1 2024-04-22T15:49:48.223 INFO:tasks.workunit.client.0.smithi016.stdout:9/990: mkdir d9/d5d/db8/d122/d148 0 2024-04-22T15:49:48.224 INFO:tasks.workunit.client.0.smithi016.stdout:9/991: chown d9/de/d23/f35 6114509 1 2024-04-22T15:49:48.224 INFO:tasks.workunit.client.1.smithi027.stdout:3/123: truncate f1d 732773 0 2024-04-22T15:49:48.227 INFO:tasks.workunit.client.0.smithi016.stdout:4/918: unlink d7/d4e/d39/df3/f11e 0 2024-04-22T15:49:48.227 INFO:tasks.workunit.client.0.smithi016.stdout:4/919: readlink d7/l1b 0 2024-04-22T15:49:48.229 INFO:tasks.workunit.client.1.smithi027.stdout:6/153: mkdir d5/d30 0 2024-04-22T15:49:48.229 INFO:tasks.workunit.client.0.smithi016.stdout:5/952: link df/d15/d1a/f53 df/d5b/d5d/d5f/f127 0 2024-04-22T15:49:48.230 INFO:tasks.workunit.client.0.smithi016.stdout:4/920: chown d7/d4e/d102/d106/d88/deb/f119 21474 1 2024-04-22T15:49:48.231 INFO:tasks.workunit.client.1.smithi027.stdout:0/120: truncate d0/fc 2443808 0 2024-04-22T15:49:48.232 INFO:tasks.workunit.client.1.smithi027.stdout:1/81: symlink d2/l14 0 2024-04-22T15:49:48.233 INFO:tasks.workunit.client.0.smithi016.stdout:7/912: dwrite d3/dc/d33/f44 [0,4194304] 0 2024-04-22T15:49:48.234 INFO:tasks.workunit.client.1.smithi027.stdout:5/109: rename d0/d11 to d0/df/de/d24 0 2024-04-22T15:49:48.237 INFO:tasks.workunit.client.0.smithi016.stdout:5/953: mknod df/d5b/d5d/d5f/c128 0 2024-04-22T15:49:48.244 INFO:tasks.workunit.client.1.smithi027.stdout:4/101: creat d5/d11/f28 x:0 0 0 2024-04-22T15:49:48.245 INFO:tasks.workunit.client.1.smithi027.stdout:7/75: sync 2024-04-22T15:49:48.245 INFO:tasks.workunit.client.1.smithi027.stdout:7/76: dread - d0/f14 zero size 2024-04-22T15:49:48.246 INFO:tasks.workunit.client.1.smithi027.stdout:7/77: dread - d0/f15 zero size 2024-04-22T15:49:48.246 INFO:tasks.workunit.client.1.smithi027.stdout:9/110: symlink d0/dc/df/l26 0 2024-04-22T15:49:48.247 INFO:tasks.workunit.client.1.smithi027.stdout:9/111: stat d0/dc/de 0 2024-04-22T15:49:48.247 INFO:tasks.workunit.client.0.smithi016.stdout:9/992: mkdir d9/d20/dac/d149 0 2024-04-22T15:49:48.248 INFO:tasks.workunit.client.1.smithi027.stdout:9/112: fsync d0/dc/de/f24 0 2024-04-22T15:49:48.248 INFO:tasks.workunit.client.1.smithi027.stdout:9/113: stat d0/dc 0 2024-04-22T15:49:48.250 INFO:tasks.workunit.client.1.smithi027.stdout:2/90: creat dd/df/f17 x:0 0 0 2024-04-22T15:49:48.253 INFO:tasks.workunit.client.0.smithi016.stdout:5/954: dread - df/d15/f7b zero size 2024-04-22T15:49:48.254 INFO:tasks.workunit.client.0.smithi016.stdout:5/955: dread - df/d15/d1a/d4e/f11f zero size 2024-04-22T15:49:48.255 INFO:tasks.workunit.client.0.smithi016.stdout:2/964: link d2/d4d/ff5 d2/d4d/d64/f16b 0 2024-04-22T15:49:48.256 INFO:tasks.workunit.client.0.smithi016.stdout:5/956: write df/d15/d1a/d28/f32 [1282949,12205] 0 2024-04-22T15:49:48.256 INFO:tasks.workunit.client.1.smithi027.stdout:3/124: dread f16 [0,4194304] 0 2024-04-22T15:49:48.258 INFO:tasks.workunit.client.0.smithi016.stdout:4/921: symlink d7/d21/d45/l132 0 2024-04-22T15:49:48.258 INFO:tasks.workunit.client.0.smithi016.stdout:4/922: fdatasync d7/f1e 0 2024-04-22T15:49:48.260 INFO:tasks.workunit.client.0.smithi016.stdout:9/993: truncate d9/d57/d91/de4/d25/d36/ff6 743696 0 2024-04-22T15:49:48.260 INFO:tasks.workunit.client.1.smithi027.stdout:8/127: link d1/cf d1/d3/d1c/d28/c30 0 2024-04-22T15:49:48.261 INFO:tasks.workunit.client.0.smithi016.stdout:9/994: readlink d9/d5d/db8/d122/l13d 0 2024-04-22T15:49:48.263 INFO:tasks.workunit.client.1.smithi027.stdout:4/102: rename d5/cc to d5/d11/d1d/c29 0 2024-04-22T15:49:48.264 INFO:tasks.workunit.client.1.smithi027.stdout:4/103: write d5/d7/f1f [1279457,16761] 0 2024-04-22T15:49:48.265 INFO:tasks.workunit.client.0.smithi016.stdout:2/965: rename d2/d1a/d7b/dcc/c73 to d2/de/d2a/d59/d162/c16c 0 2024-04-22T15:49:48.267 INFO:tasks.workunit.client.0.smithi016.stdout:7/913: write d3/dc/d1b/d5a/d4a/d6e/fa8 [4650510,96320] 0 2024-04-22T15:49:48.269 INFO:tasks.workunit.client.1.smithi027.stdout:2/91: readlink l5 0 2024-04-22T15:49:48.271 INFO:tasks.workunit.client.1.smithi027.stdout:1/82: write d2/f8 [2537344,66681] 0 2024-04-22T15:49:48.273 INFO:tasks.workunit.client.1.smithi027.stdout:2/92: dread dd/f11 [0,4194304] 0 2024-04-22T15:49:48.276 INFO:tasks.workunit.client.1.smithi027.stdout:5/110: getdents d0/df/de/d24/d1c 0 2024-04-22T15:49:48.277 INFO:tasks.workunit.client.0.smithi016.stdout:5/957: write df/d15/d1a/d4e/d10a/f9c [633381,17810] 0 2024-04-22T15:49:48.279 INFO:tasks.workunit.client.0.smithi016.stdout:5/958: fsync df/d15/d1a/d50/d77/d7d/d8b/d8c/de4/fd1 0 2024-04-22T15:49:48.280 INFO:tasks.workunit.client.0.smithi016.stdout:7/914: dread d3/dc/d1b/d5a/d4a/ffa [0,4194304] 0 2024-04-22T15:49:48.280 INFO:tasks.workunit.client.1.smithi027.stdout:8/128: mknod d1/d3/d1c/d28/c31 0 2024-04-22T15:49:48.285 INFO:tasks.workunit.client.1.smithi027.stdout:7/78: unlink d0/fa 0 2024-04-22T15:49:48.285 INFO:tasks.workunit.client.1.smithi027.stdout:5/111: dwrite d0/df/d10/f14 [0,4194304] 0 2024-04-22T15:49:48.286 INFO:tasks.workunit.client.0.smithi016.stdout:2/966: dwrite d2/de/d2a/d59/fbe [4194304,4194304] 0 2024-04-22T15:49:48.286 INFO:tasks.workunit.client.0.smithi016.stdout:4/923: truncate d7/d21/d7c/fe8 356962 0 2024-04-22T15:49:48.292 INFO:tasks.workunit.client.1.smithi027.stdout:3/125: dwrite f5 [0,4194304] 0 2024-04-22T15:49:48.293 INFO:tasks.workunit.client.1.smithi027.stdout:0/121: rename d0/d1a to d0/d28 0 2024-04-22T15:49:48.294 INFO:tasks.workunit.client.0.smithi016.stdout:2/967: dwrite d2/f166 [4194304,4194304] 0 2024-04-22T15:49:48.296 INFO:tasks.workunit.client.0.smithi016.stdout:2/968: truncate d2/d23/ff6 4301377 0 2024-04-22T15:49:48.298 INFO:tasks.workunit.client.0.smithi016.stdout:5/959: unlink df/d15/d1a/d50/d77/l48 0 2024-04-22T15:49:48.298 INFO:tasks.workunit.client.1.smithi027.stdout:8/129: dwrite d1/d3/f13 [0,4194304] 0 2024-04-22T15:49:48.300 INFO:tasks.workunit.client.1.smithi027.stdout:8/130: write d1/d3/f19 [370960,109501] 0 2024-04-22T15:49:48.300 INFO:tasks.workunit.client.1.smithi027.stdout:2/93: rmdir dd/df 39 2024-04-22T15:49:48.307 INFO:tasks.workunit.client.0.smithi016.stdout:9/995: creat d9/de/f14a x:0 0 0 2024-04-22T15:49:48.308 INFO:tasks.workunit.client.0.smithi016.stdout:9/996: chown d9/d37/f49 4545 1 2024-04-22T15:49:48.309 INFO:tasks.workunit.client.0.smithi016.stdout:9/997: chown d9/de/d23/ddb/def/f104 10006274 1 2024-04-22T15:49:48.312 INFO:tasks.workunit.client.0.smithi016.stdout:6/958: sync 2024-04-22T15:49:48.312 INFO:tasks.workunit.client.1.smithi027.stdout:8/131: dwrite d1/d3/f6 [0,4194304] 0 2024-04-22T15:49:48.313 INFO:tasks.workunit.client.0.smithi016.stdout:8/895: sync 2024-04-22T15:49:48.314 INFO:tasks.workunit.client.0.smithi016.stdout:8/896: fsync d2/d82/f58 0 2024-04-22T15:49:48.315 INFO:tasks.workunit.client.1.smithi027.stdout:5/112: creat d0/df/d10/f25 x:0 0 0 2024-04-22T15:49:48.317 INFO:tasks.workunit.client.0.smithi016.stdout:8/897: read d2/d48/dad/d6a/f9a [617248,48112] 0 2024-04-22T15:49:48.319 INFO:tasks.workunit.client.0.smithi016.stdout:8/898: truncate d2/d82/d3e/dd5/df8/ff9 670443 0 2024-04-22T15:49:48.320 INFO:tasks.workunit.client.0.smithi016.stdout:8/899: dread - d2/d82/d52/df0/d66/ffb zero size 2024-04-22T15:49:48.321 INFO:tasks.workunit.client.0.smithi016.stdout:5/960: rmdir df/d5b 39 2024-04-22T15:49:48.323 INFO:tasks.workunit.client.1.smithi027.stdout:5/113: dwrite d0/f7 [0,4194304] 0 2024-04-22T15:49:48.323 INFO:tasks.workunit.client.1.smithi027.stdout:3/126: chown f22 7 1 2024-04-22T15:49:48.324 INFO:tasks.workunit.client.1.smithi027.stdout:3/127: stat f1d 0 2024-04-22T15:49:48.324 INFO:tasks.workunit.client.0.smithi016.stdout:7/915: symlink d3/l149 0 2024-04-22T15:49:48.325 INFO:tasks.workunit.client.1.smithi027.stdout:3/128: fdatasync f1e 0 2024-04-22T15:49:48.325 INFO:tasks.workunit.client.1.smithi027.stdout:5/114: readlink d0/df/d10/l1d 0 2024-04-22T15:49:48.325 INFO:tasks.workunit.client.1.smithi027.stdout:3/129: write ff [389342,30914] 0 2024-04-22T15:49:48.327 INFO:tasks.workunit.client.1.smithi027.stdout:3/130: chown l3 114918 1 2024-04-22T15:49:48.332 INFO:tasks.workunit.client.1.smithi027.stdout:9/114: rename d0/d8/c20 to d0/c27 0 2024-04-22T15:49:48.334 INFO:tasks.workunit.client.1.smithi027.stdout:8/132: mkdir d1/d3/d1c/d32 0 2024-04-22T15:49:48.335 INFO:tasks.workunit.client.0.smithi016.stdout:8/900: creat d2/d48/dad/d8f/da1/f119 x:0 0 0 2024-04-22T15:49:48.336 INFO:tasks.workunit.client.1.smithi027.stdout:9/115: dread d0/dc/d12/f23 [0,4194304] 0 2024-04-22T15:49:48.336 INFO:tasks.workunit.client.0.smithi016.stdout:8/901: fdatasync d2/d82/d52/f54 0 2024-04-22T15:49:48.337 INFO:tasks.workunit.client.0.smithi016.stdout:9/998: mknod d9/d20/c14b 0 2024-04-22T15:49:48.339 INFO:tasks.workunit.client.1.smithi027.stdout:2/94: unlink dd/c10 0 2024-04-22T15:49:48.342 INFO:tasks.workunit.client.0.smithi016.stdout:7/916: mknod d3/dc/d1b/d5a/d9/df/c14a 0 2024-04-22T15:49:48.343 INFO:tasks.workunit.client.0.smithi016.stdout:9/999: fdatasync f2 0 2024-04-22T15:49:48.344 INFO:tasks.workunit.client.1.smithi027.stdout:5/115: mkdir d0/d1a/d22/d26 0 2024-04-22T15:49:48.346 INFO:tasks.workunit.client.1.smithi027.stdout:3/131: creat d28/f29 x:0 0 0 2024-04-22T15:49:48.347 INFO:tasks.workunit.client.0.smithi016.stdout:7/917: dwrite d3/de6/fec [0,4194304] 0 2024-04-22T15:49:48.350 INFO:tasks.workunit.client.1.smithi027.stdout:7/79: getdents d0 0 2024-04-22T15:49:48.355 INFO:tasks.workunit.client.1.smithi027.stdout:7/80: truncate d0/f14 1034878 0 2024-04-22T15:49:48.356 INFO:tasks.workunit.client.1.smithi027.stdout:9/116: dwrite d0/dc/f25 [0,4194304] 0 2024-04-22T15:49:48.357 INFO:tasks.workunit.client.1.smithi027.stdout:7/81: chown d0/l10 3 1 2024-04-22T15:49:48.358 INFO:tasks.workunit.client.1.smithi027.stdout:3/132: dread f1f [0,4194304] 0 2024-04-22T15:49:48.359 INFO:tasks.workunit.client.1.smithi027.stdout:3/133: fsync ff 0 2024-04-22T15:49:48.360 INFO:tasks.workunit.client.0.smithi016.stdout:5/961: dread df/d15/d1a/d50/d77/d7d/d8b/d8c/ff7 [0,4194304] 0 2024-04-22T15:49:48.362 INFO:tasks.workunit.client.1.smithi027.stdout:2/95: creat dd/df/f18 x:0 0 0 2024-04-22T15:49:48.364 INFO:tasks.workunit.client.1.smithi027.stdout:2/96: chown dd/df/f18 1 1 2024-04-22T15:49:48.365 INFO:tasks.workunit.client.1.smithi027.stdout:0/122: chown d0/f12 34672171 1 2024-04-22T15:49:48.368 INFO:tasks.workunit.client.1.smithi027.stdout:5/116: getdents d0/d1a/d22/d26 0 2024-04-22T15:49:48.371 INFO:tasks.workunit.client.1.smithi027.stdout:9/117: rename d0/l3 to d0/dc/de/l28 0 2024-04-22T15:49:48.372 INFO:tasks.workunit.client.1.smithi027.stdout:9/118: dread - d0/dc/f1e zero size 2024-04-22T15:49:48.374 INFO:tasks.workunit.client.0.smithi016.stdout:4/924: dwrite d7/d21/d45/d49/d9b/db6/fc4 [0,4194304] 0 2024-04-22T15:49:48.375 INFO:tasks.workunit.client.1.smithi027.stdout:5/117: dread d0/df/f12 [0,4194304] 0 2024-04-22T15:49:48.377 INFO:tasks.workunit.client.0.smithi016.stdout:2/969: dwrite d2/d1a/d7b/dcc/de5/fe9 [0,4194304] 0 2024-04-22T15:49:48.381 INFO:tasks.workunit.client.1.smithi027.stdout:8/133: getdents d1/d1e 0 2024-04-22T15:49:48.382 INFO:tasks.workunit.client.0.smithi016.stdout:6/959: dwrite d3/da/db/d14/d4c/fea [0,4194304] 0 2024-04-22T15:49:48.389 INFO:tasks.workunit.client.1.smithi027.stdout:3/134: rename f1e to d28/f2a 0 2024-04-22T15:49:48.391 INFO:tasks.workunit.client.0.smithi016.stdout:5/962: creat df/d5b/f129 x:0 0 0 2024-04-22T15:49:48.393 INFO:tasks.workunit.client.0.smithi016.stdout:7/918: dread d3/dc/d1b/d5a/d9/df/d94/d4c/f101 [0,4194304] 0 2024-04-22T15:49:48.404 INFO:tasks.workunit.client.0.smithi016.stdout:5/963: symlink df/d15/d1a/d4e/d58/d87/l12a 0 2024-04-22T15:49:48.405 INFO:tasks.workunit.client.0.smithi016.stdout:4/925: write d7/d4e/d2f/d6f/f97 [4892486,72745] 0 2024-04-22T15:49:48.406 INFO:tasks.workunit.client.1.smithi027.stdout:3/135: truncate f16 2149009 0 2024-04-22T15:49:48.407 INFO:tasks.workunit.client.1.smithi027.stdout:3/136: chown l2 2066 1 2024-04-22T15:49:48.407 INFO:tasks.workunit.client.1.smithi027.stdout:3/137: read f1f [2162008,123692] 0 2024-04-22T15:49:48.408 INFO:tasks.workunit.client.0.smithi016.stdout:7/919: creat d3/dc/d1b/d5a/d59/d64/dfe/f14b x:0 0 0 2024-04-22T15:49:48.409 INFO:tasks.workunit.client.0.smithi016.stdout:4/926: readlink d7/d4e/l10c 0 2024-04-22T15:49:48.411 INFO:tasks.workunit.client.1.smithi027.stdout:0/123: rename d0/c21 to d0/d28/c29 0 2024-04-22T15:49:48.412 INFO:tasks.workunit.client.1.smithi027.stdout:9/119: creat d0/dc/f29 x:0 0 0 2024-04-22T15:49:48.414 INFO:tasks.workunit.client.1.smithi027.stdout:3/138: dwrite d28/f29 [0,4194304] 0 2024-04-22T15:49:48.420 INFO:tasks.workunit.client.1.smithi027.stdout:9/120: unlink d0/d8/l1f 0 2024-04-22T15:49:48.420 INFO:tasks.workunit.client.1.smithi027.stdout:6/154: sync 2024-04-22T15:49:48.420 INFO:tasks.workunit.client.0.smithi016.stdout:6/960: link d3/da/d12/d13/d38/df5/lef d3/da/dd/d67/d111/l135 0 2024-04-22T15:49:48.421 INFO:tasks.workunit.client.1.smithi027.stdout:2/97: truncate dd/df/f12 1657070 0 2024-04-22T15:49:48.422 INFO:tasks.workunit.client.1.smithi027.stdout:3/139: dwrite f5 [0,4194304] 0 2024-04-22T15:49:48.423 INFO:tasks.workunit.client.1.smithi027.stdout:2/98: dread - dd/df/f17 zero size 2024-04-22T15:49:48.424 INFO:tasks.workunit.client.1.smithi027.stdout:3/140: rename d28 to d28/d2b 22 2024-04-22T15:49:48.424 INFO:tasks.workunit.client.0.smithi016.stdout:2/970: dwrite d2/d23/d92/d55/ffe [0,4194304] 0 2024-04-22T15:49:48.426 INFO:tasks.workunit.client.1.smithi027.stdout:6/155: symlink d5/dc/d20/l31 0 2024-04-22T15:49:48.426 INFO:tasks.workunit.client.1.smithi027.stdout:9/121: truncate d0/dc/de/f24 1633888 0 2024-04-22T15:49:48.431 INFO:tasks.workunit.client.1.smithi027.stdout:3/141: symlink d28/l2c 0 2024-04-22T15:49:48.432 INFO:tasks.workunit.client.1.smithi027.stdout:5/118: dwrite d0/df/de/f13 [4194304,4194304] 0 2024-04-22T15:49:48.434 INFO:tasks.workunit.client.1.smithi027.stdout:5/119: read d0/f6 [1125497,66479] 0 2024-04-22T15:49:48.437 INFO:tasks.workunit.client.1.smithi027.stdout:5/120: stat d0/df/de/d24/d1c 0 2024-04-22T15:49:48.437 INFO:tasks.workunit.client.0.smithi016.stdout:6/961: mknod d3/da/ddd/d101/d11b/c136 0 2024-04-22T15:49:48.438 INFO:tasks.workunit.client.0.smithi016.stdout:7/920: write d3/dc/d1b/f6d [925189,11906] 0 2024-04-22T15:49:48.440 INFO:tasks.workunit.client.1.smithi027.stdout:2/99: dwrite dd/df/f17 [0,4194304] 0 2024-04-22T15:49:48.442 INFO:tasks.workunit.client.0.smithi016.stdout:2/971: creat d2/d2c/d97/d67/d138/f16d x:0 0 0 2024-04-22T15:49:48.445 INFO:tasks.workunit.client.0.smithi016.stdout:6/962: rmdir d3/da/db/d14/d16 39 2024-04-22T15:49:48.448 INFO:tasks.workunit.client.1.smithi027.stdout:3/142: symlink d28/l2d 0 2024-04-22T15:49:48.449 INFO:tasks.workunit.client.1.smithi027.stdout:5/121: dwrite d0/df/d10/f25 [0,4194304] 0 2024-04-22T15:49:48.449 INFO:tasks.workunit.client.1.smithi027.stdout:3/143: chown c24 17765 1 2024-04-22T15:49:48.451 INFO:tasks.workunit.client.1.smithi027.stdout:3/144: dread - f1c zero size 2024-04-22T15:49:48.452 INFO:tasks.workunit.client.1.smithi027.stdout:5/122: write d0/df/d10/f14 [3909667,121412] 0 2024-04-22T15:49:48.452 INFO:tasks.workunit.client.0.smithi016.stdout:2/972: mkdir d2/d4d/d64/d16e 0 2024-04-22T15:49:48.453 INFO:tasks.workunit.client.1.smithi027.stdout:0/124: fsync d0/d28/f1c 0 2024-04-22T15:49:48.456 INFO:tasks.workunit.client.0.smithi016.stdout:7/921: symlink d3/dc/d1b/d5a/d4a/l14c 0 2024-04-22T15:49:48.456 INFO:tasks.workunit.client.1.smithi027.stdout:6/156: symlink d5/l32 0 2024-04-22T15:49:48.457 INFO:tasks.workunit.client.1.smithi027.stdout:6/157: chown d5/db/l14 0 1 2024-04-22T15:49:48.460 INFO:tasks.workunit.client.1.smithi027.stdout:6/158: truncate d5/dc/d20/d2b/f24 1299209 0 2024-04-22T15:49:48.461 INFO:tasks.workunit.client.1.smithi027.stdout:2/100: unlink l5 0 2024-04-22T15:49:48.462 INFO:tasks.workunit.client.0.smithi016.stdout:7/922: dread d3/dc/f1e [0,4194304] 0 2024-04-22T15:49:48.463 INFO:tasks.workunit.client.0.smithi016.stdout:2/973: creat d2/d1a/d4c/d77/d12c/f16f x:0 0 0 2024-04-22T15:49:48.467 INFO:tasks.workunit.client.0.smithi016.stdout:2/974: mknod d2/d1a/d4c/d77/dea/c170 0 2024-04-22T15:49:48.467 INFO:tasks.workunit.client.0.smithi016.stdout:7/923: dread d3/dc/d1b/d5a/d9/df/d94/d4c/f101 [0,4194304] 0 2024-04-22T15:49:48.468 INFO:tasks.workunit.client.1.smithi027.stdout:3/145: symlink d28/l2e 0 2024-04-22T15:49:48.470 INFO:tasks.workunit.client.0.smithi016.stdout:2/975: creat d2/d1a/d4c/d9f/f171 x:0 0 0 2024-04-22T15:49:48.471 INFO:tasks.workunit.client.1.smithi027.stdout:9/122: link d0/d8/c21 d0/d8/c2a 0 2024-04-22T15:49:48.472 INFO:tasks.workunit.client.1.smithi027.stdout:9/123: write d0/dc/f25 [4641739,13260] 0 2024-04-22T15:49:48.474 INFO:tasks.workunit.client.1.smithi027.stdout:9/124: write d0/fb [1376693,72816] 0 2024-04-22T15:49:48.475 INFO:tasks.workunit.client.0.smithi016.stdout:2/976: dread d2/d51/f9b [0,4194304] 0 2024-04-22T15:49:48.477 INFO:tasks.workunit.client.1.smithi027.stdout:3/146: fdatasync f10 0 2024-04-22T15:49:48.479 INFO:tasks.workunit.client.1.smithi027.stdout:9/125: dread d0/dc/f22 [0,4194304] 0 2024-04-22T15:49:48.480 INFO:tasks.workunit.client.1.smithi027.stdout:9/126: chown d0/d8/f18 63001 1 2024-04-22T15:49:48.480 INFO:tasks.workunit.client.0.smithi016.stdout:2/977: creat d2/d9/f172 x:0 0 0 2024-04-22T15:49:48.481 INFO:tasks.workunit.client.1.smithi027.stdout:0/125: mkdir d0/d2a 0 2024-04-22T15:49:48.483 INFO:tasks.workunit.client.1.smithi027.stdout:9/127: dread d0/d8/f18 [0,4194304] 0 2024-04-22T15:49:48.487 INFO:tasks.workunit.client.0.smithi016.stdout:2/978: link d2/f2b d2/d23/d92/d120/f173 0 2024-04-22T15:49:48.488 INFO:tasks.workunit.client.0.smithi016.stdout:2/979: fdatasync d2/d23/db8/f14c 0 2024-04-22T15:49:48.489 INFO:tasks.workunit.client.0.smithi016.stdout:7/924: truncate d3/dc/d1b/d5a/d59/d64/f6c 2187412 0 2024-04-22T15:49:48.490 INFO:tasks.workunit.client.1.smithi027.stdout:3/147: rename l3 to d28/l2f 0 2024-04-22T15:49:48.490 INFO:tasks.workunit.client.1.smithi027.stdout:3/148: readlink d28/l2c 0 2024-04-22T15:49:48.495 INFO:tasks.workunit.client.0.smithi016.stdout:7/925: creat d3/dc/d1b/d5a/d59/d119/f14d x:0 0 0 2024-04-22T15:49:48.496 INFO:tasks.workunit.client.1.smithi027.stdout:5/123: link d0/l17 d0/d1a/d22/l27 0 2024-04-22T15:49:48.496 INFO:tasks.workunit.client.1.smithi027.stdout:3/149: dwrite f15 [4194304,4194304] 0 2024-04-22T15:49:48.497 INFO:tasks.workunit.client.1.smithi027.stdout:3/150: chown l8 173558 1 2024-04-22T15:49:48.505 INFO:tasks.workunit.client.1.smithi027.stdout:5/124: dwrite d0/df/de/f18 [0,4194304] 0 2024-04-22T15:49:48.509 INFO:tasks.workunit.client.0.smithi016.stdout:7/926: mkdir d3/dc/d1b/d5a/d9/d14e 0 2024-04-22T15:49:48.510 INFO:tasks.workunit.client.1.smithi027.stdout:2/101: truncate dd/df/f12 209695 0 2024-04-22T15:49:48.511 INFO:tasks.workunit.client.0.smithi016.stdout:2/980: getdents d2/d2c/d97/d34/d60 0 2024-04-22T15:49:48.512 INFO:tasks.workunit.client.0.smithi016.stdout:7/927: mkdir d3/d11a/d14f 0 2024-04-22T15:49:48.513 INFO:tasks.workunit.client.0.smithi016.stdout:7/928: chown d3/dc/d1b/d31/d5f 2 1 2024-04-22T15:49:48.517 INFO:tasks.workunit.client.0.smithi016.stdout:2/981: creat d2/d23/db8/f174 x:0 0 0 2024-04-22T15:49:48.517 INFO:tasks.workunit.client.1.smithi027.stdout:9/128: mknod d0/dc/c2b 0 2024-04-22T15:49:48.520 INFO:tasks.workunit.client.0.smithi016.stdout:2/982: mkdir d2/d1a/d7b/dcc/de5/d107/d175 0 2024-04-22T15:49:48.525 INFO:tasks.workunit.client.1.smithi027.stdout:9/129: mknod d0/dc/df/c2c 0 2024-04-22T15:49:48.525 INFO:tasks.workunit.client.0.smithi016.stdout:2/983: dwrite d2/de/d2a/d59/fbe [0,4194304] 0 2024-04-22T15:49:48.531 INFO:tasks.workunit.client.1.smithi027.stdout:0/126: dwrite d0/f14 [0,4194304] 0 2024-04-22T15:49:48.534 INFO:tasks.workunit.client.0.smithi016.stdout:2/984: symlink d2/d4d/dc4/l176 0 2024-04-22T15:49:48.537 INFO:tasks.workunit.client.1.smithi027.stdout:3/151: dwrite f1d [0,4194304] 0 2024-04-22T15:49:48.541 INFO:tasks.workunit.client.0.smithi016.stdout:2/985: creat d2/d2c/de7/d148/f177 x:0 0 0 2024-04-22T15:49:48.542 INFO:tasks.workunit.client.0.smithi016.stdout:7/929: dread d3/dc/d22/f39 [4194304,4194304] 0 2024-04-22T15:49:48.543 INFO:tasks.workunit.client.1.smithi027.stdout:5/125: truncate d0/df/d10/f25 1206698 0 2024-04-22T15:49:48.543 INFO:tasks.workunit.client.0.smithi016.stdout:2/986: readlink d2/d2c/daa/d110/l14a 0 2024-04-22T15:49:48.545 INFO:tasks.workunit.client.1.smithi027.stdout:9/130: write d0/dc/d12/f23 [1274373,105590] 0 2024-04-22T15:49:48.547 INFO:tasks.workunit.client.1.smithi027.stdout:2/102: dread dd/df/f12 [0,4194304] 0 2024-04-22T15:49:48.549 INFO:tasks.workunit.client.0.smithi016.stdout:2/987: creat d2/d2c/de7/f178 x:0 0 0 2024-04-22T15:49:48.549 INFO:tasks.workunit.client.1.smithi027.stdout:3/152: fdatasync d28/f2a 0 2024-04-22T15:49:48.554 INFO:tasks.workunit.client.1.smithi027.stdout:0/127: truncate d0/fc 2977119 0 2024-04-22T15:49:48.556 INFO:tasks.workunit.client.0.smithi016.stdout:2/988: creat d2/d4d/f179 x:0 0 0 2024-04-22T15:49:48.557 INFO:tasks.workunit.client.0.smithi016.stdout:2/989: fdatasync d2/d2c/d97/f5f 0 2024-04-22T15:49:48.558 INFO:tasks.workunit.client.0.smithi016.stdout:2/990: write d2/d2c/f82 [3478392,106548] 0 2024-04-22T15:49:48.559 INFO:tasks.workunit.client.1.smithi027.stdout:3/153: truncate f16 2510680 0 2024-04-22T15:49:48.560 INFO:tasks.workunit.client.1.smithi027.stdout:0/128: creat d0/d2a/f2b x:0 0 0 2024-04-22T15:49:48.562 INFO:tasks.workunit.client.1.smithi027.stdout:3/154: symlink d28/l30 0 2024-04-22T15:49:48.562 INFO:tasks.workunit.client.1.smithi027.stdout:5/126: write d0/f6 [3968803,28151] 0 2024-04-22T15:49:48.564 INFO:tasks.workunit.client.1.smithi027.stdout:3/155: write f5 [3719079,99626] 0 2024-04-22T15:49:48.564 INFO:tasks.workunit.client.1.smithi027.stdout:3/156: truncate d28/f2a 973627 0 2024-04-22T15:49:48.566 INFO:tasks.workunit.client.0.smithi016.stdout:7/930: dwrite d3/dc/d1b/d5a/d4a/d6e/f73 [0,4194304] 0 2024-04-22T15:49:48.566 INFO:tasks.workunit.client.1.smithi027.stdout:3/157: write ff [5194429,39211] 0 2024-04-22T15:49:48.567 INFO:tasks.workunit.client.0.smithi016.stdout:2/991: mknod d2/d4d/d150/d11b/c17a 0 2024-04-22T15:49:48.567 INFO:tasks.workunit.client.1.smithi027.stdout:3/158: write ff [5192130,15933] 0 2024-04-22T15:49:48.573 INFO:tasks.workunit.client.1.smithi027.stdout:0/129: unlink d0/f15 0 2024-04-22T15:49:48.574 INFO:tasks.workunit.client.1.smithi027.stdout:2/103: dwrite dd/f11 [0,4194304] 0 2024-04-22T15:49:48.577 INFO:tasks.workunit.client.1.smithi027.stdout:3/159: dread f1f [0,4194304] 0 2024-04-22T15:49:48.578 INFO:tasks.workunit.client.0.smithi016.stdout:8/902: sync 2024-04-22T15:49:48.580 INFO:tasks.workunit.client.0.smithi016.stdout:2/992: dread d2/d51/f62 [0,4194304] 0 2024-04-22T15:49:48.581 INFO:tasks.workunit.client.1.smithi027.stdout:3/160: mknod d28/c31 0 2024-04-22T15:49:48.581 INFO:tasks.workunit.client.1.smithi027.stdout:3/161: fsync f1d 0 2024-04-22T15:49:48.583 INFO:tasks.workunit.client.0.smithi016.stdout:2/993: unlink d2/d1a/d7b/dc6/cd3 0 2024-04-22T15:49:48.584 INFO:tasks.workunit.client.1.smithi027.stdout:5/127: getdents d0/d1a 0 2024-04-22T15:49:48.589 INFO:tasks.workunit.client.0.smithi016.stdout:8/903: dwrite d2/d48/dad/f115 [0,4194304] 0 2024-04-22T15:49:48.590 INFO:tasks.workunit.client.0.smithi016.stdout:2/994: symlink d2/d23/l17b 0 2024-04-22T15:49:48.590 INFO:tasks.workunit.client.1.smithi027.stdout:3/162: mknod d28/c32 0 2024-04-22T15:49:48.590 INFO:tasks.workunit.client.1.smithi027.stdout:2/104: creat dd/f19 x:0 0 0 2024-04-22T15:49:48.594 INFO:tasks.workunit.client.1.smithi027.stdout:5/128: symlink d0/df/d10/l28 0 2024-04-22T15:49:48.596 INFO:tasks.workunit.client.1.smithi027.stdout:3/163: mknod d28/c33 0 2024-04-22T15:49:48.598 INFO:tasks.workunit.client.1.smithi027.stdout:2/105: creat dd/df/f1a x:0 0 0 2024-04-22T15:49:48.601 INFO:tasks.workunit.client.1.smithi027.stdout:5/129: dwrite d0/f7 [0,4194304] 0 2024-04-22T15:49:48.603 INFO:tasks.workunit.client.1.smithi027.stdout:2/106: unlink l3 0 2024-04-22T15:49:48.603 INFO:tasks.workunit.client.1.smithi027.stdout:3/164: dread f11 [0,4194304] 0 2024-04-22T15:49:48.605 INFO:tasks.workunit.client.1.smithi027.stdout:5/130: creat d0/df/de/f29 x:0 0 0 2024-04-22T15:49:48.607 INFO:tasks.workunit.client.1.smithi027.stdout:3/165: creat d28/f34 x:0 0 0 2024-04-22T15:49:48.608 INFO:tasks.workunit.client.1.smithi027.stdout:0/130: truncate d0/f22 104365 0 2024-04-22T15:49:48.610 INFO:tasks.workunit.client.1.smithi027.stdout:2/107: dread f7 [0,4194304] 0 2024-04-22T15:49:48.611 INFO:tasks.workunit.client.1.smithi027.stdout:5/131: creat d0/d1a/d22/d26/f2a x:0 0 0 2024-04-22T15:49:48.612 INFO:tasks.workunit.client.1.smithi027.stdout:0/131: readlink d0/l25 0 2024-04-22T15:49:48.613 INFO:tasks.workunit.client.1.smithi027.stdout:2/108: mkdir dd/df/d1b 0 2024-04-22T15:49:48.613 INFO:tasks.workunit.client.0.smithi016.stdout:8/904: dwrite d2/d82/d56/f71 [0,4194304] 0 2024-04-22T15:49:48.613 INFO:tasks.workunit.client.0.smithi016.stdout:8/905: chown d2 1733689 1 2024-04-22T15:49:48.617 INFO:tasks.workunit.client.1.smithi027.stdout:3/166: dwrite d28/f2a [0,4194304] 0 2024-04-22T15:49:48.619 INFO:tasks.workunit.client.1.smithi027.stdout:3/167: write ff [2558881,2469] 0 2024-04-22T15:49:48.619 INFO:tasks.workunit.client.1.smithi027.stdout:5/132: dread d0/f3 [0,4194304] 0 2024-04-22T15:49:48.622 INFO:tasks.workunit.client.1.smithi027.stdout:2/109: creat dd/f1c x:0 0 0 2024-04-22T15:49:48.624 INFO:tasks.workunit.client.1.smithi027.stdout:2/110: fsync dd/f11 0 2024-04-22T15:49:48.625 INFO:tasks.workunit.client.1.smithi027.stdout:2/111: write f7 [2881948,100275] 0 2024-04-22T15:49:48.627 INFO:tasks.workunit.client.1.smithi027.stdout:2/112: truncate dd/f1c 515085 0 2024-04-22T15:49:48.628 INFO:tasks.workunit.client.1.smithi027.stdout:5/133: unlink d0/df/l19 0 2024-04-22T15:49:48.629 INFO:tasks.workunit.client.1.smithi027.stdout:3/168: dwrite f10 [4194304,4194304] 0 2024-04-22T15:49:48.630 INFO:tasks.workunit.client.1.smithi027.stdout:0/132: rename d0/lf to d0/l2c 0 2024-04-22T15:49:48.638 INFO:tasks.workunit.client.1.smithi027.stdout:2/113: dwrite dd/f11 [0,4194304] 0 2024-04-22T15:49:48.641 INFO:tasks.workunit.client.1.smithi027.stdout:0/133: rmdir d0/d2a 39 2024-04-22T15:49:48.643 INFO:tasks.workunit.client.1.smithi027.stdout:3/169: mknod d28/c35 0 2024-04-22T15:49:48.645 INFO:tasks.workunit.client.0.smithi016.stdout:8/906: truncate d2/d82/d52/df0/d66/d1c/f81 362551 0 2024-04-22T15:49:48.646 INFO:tasks.workunit.client.1.smithi027.stdout:0/134: fdatasync d0/d2a/f2b 0 2024-04-22T15:49:48.648 INFO:tasks.workunit.client.1.smithi027.stdout:2/114: mkdir dd/d1d 0 2024-04-22T15:49:48.650 INFO:tasks.workunit.client.1.smithi027.stdout:0/135: truncate d0/d28/f1c 975341 0 2024-04-22T15:49:48.651 INFO:tasks.workunit.client.0.smithi016.stdout:8/907: creat d2/d82/d52/df0/d66/d7e/dc0/de2/f11a x:0 0 0 2024-04-22T15:49:48.653 INFO:tasks.workunit.client.1.smithi027.stdout:3/170: link f1f d28/f36 0 2024-04-22T15:49:48.653 INFO:tasks.workunit.client.1.smithi027.stdout:3/171: chown l19 880428 1 2024-04-22T15:49:48.654 INFO:tasks.workunit.client.1.smithi027.stdout:3/172: write f11 [478294,71765] 0 2024-04-22T15:49:48.655 INFO:tasks.workunit.client.1.smithi027.stdout:0/136: symlink d0/d2a/l2d 0 2024-04-22T15:49:48.655 INFO:tasks.workunit.client.1.smithi027.stdout:0/137: stat d0/d28/f1c 0 2024-04-22T15:49:48.658 INFO:tasks.workunit.client.0.smithi016.stdout:8/908: symlink d2/d82/d52/df0/d66/d7e/d85/l11b 0 2024-04-22T15:49:48.664 INFO:tasks.workunit.client.0.smithi016.stdout:8/909: rename d2/d82/d52/df0/l114 to d2/d77/l11c 0 2024-04-22T15:49:48.665 INFO:tasks.workunit.client.1.smithi027.stdout:0/138: chown d0/cb 0 1 2024-04-22T15:49:48.666 INFO:tasks.workunit.client.1.smithi027.stdout:2/115: creat dd/f1e x:0 0 0 2024-04-22T15:49:48.666 INFO:tasks.workunit.client.1.smithi027.stdout:0/139: truncate d0/d28/f20 4474728 0 2024-04-22T15:49:48.668 INFO:tasks.workunit.client.1.smithi027.stdout:0/140: write d0/f14 [2931126,99989] 0 2024-04-22T15:49:48.668 INFO:tasks.workunit.client.1.smithi027.stdout:0/141: stat d0/d28 0 2024-04-22T15:49:48.670 INFO:tasks.workunit.client.1.smithi027.stdout:2/116: dread dd/f11 [0,4194304] 0 2024-04-22T15:49:48.671 INFO:tasks.workunit.client.0.smithi016.stdout:8/910: mknod d2/c11d 0 2024-04-22T15:49:48.676 INFO:tasks.workunit.client.1.smithi027.stdout:3/173: dwrite f1f [0,4194304] 0 2024-04-22T15:49:48.676 INFO:tasks.workunit.client.1.smithi027.stdout:0/142: dwrite d0/f14 [4194304,4194304] 0 2024-04-22T15:49:48.679 INFO:tasks.workunit.client.0.smithi016.stdout:8/911: mkdir d2/d82/d52/df0/dc1/d10a/d112/d11e 0 2024-04-22T15:49:48.679 INFO:tasks.workunit.client.1.smithi027.stdout:0/143: rmdir d0/d28 39 2024-04-22T15:49:48.681 INFO:tasks.workunit.client.0.smithi016.stdout:8/912: mkdir d2/d82/d52/df0/d66/d79/d11f 0 2024-04-22T15:49:48.682 INFO:tasks.workunit.client.1.smithi027.stdout:2/117: mkdir dd/df/d1b/d1f 0 2024-04-22T15:49:48.684 INFO:tasks.workunit.client.0.smithi016.stdout:8/913: write d2/d82/d52/df0/d66/d7e/ff5 [1276318,124035] 0 2024-04-22T15:49:48.688 INFO:tasks.workunit.client.0.smithi016.stdout:8/914: rename d2/d32/f118 to d2/d48/dad/d8f/d96/dda/f120 0 2024-04-22T15:49:48.689 INFO:tasks.workunit.client.1.smithi027.stdout:2/118: dwrite dd/f19 [0,4194304] 0 2024-04-22T15:49:48.691 INFO:tasks.workunit.client.0.smithi016.stdout:5/964: sync 2024-04-22T15:49:48.696 INFO:tasks.workunit.client.1.smithi027.stdout:2/119: unlink dd/df/f1a 0 2024-04-22T15:49:48.700 INFO:tasks.workunit.client.1.smithi027.stdout:2/120: creat dd/f20 x:0 0 0 2024-04-22T15:49:48.703 INFO:tasks.workunit.client.1.smithi027.stdout:2/121: write dd/df/f12 [1113761,89561] 0 2024-04-22T15:49:48.707 INFO:tasks.workunit.client.1.smithi027.stdout:2/122: unlink dd/df/c16 0 2024-04-22T15:49:48.708 INFO:tasks.workunit.client.1.smithi027.stdout:2/123: readlink la 0 2024-04-22T15:49:48.708 INFO:tasks.workunit.client.1.smithi027.stdout:2/124: read f7 [3291083,107594] 0 2024-04-22T15:49:48.709 INFO:tasks.workunit.client.1.smithi027.stdout:2/125: write f7 [3198791,33846] 0 2024-04-22T15:49:48.709 INFO:tasks.workunit.client.1.smithi027.stdout:2/126: chown dd 13320663 1 2024-04-22T15:49:48.715 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:48 smithi016 ceph-mon[26620]: pgmap v36: 65 pgs: 65 active+clean; 2.0 GiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 73 MiB/s rd, 208 MiB/s wr, 480 op/s 2024-04-22T15:49:48.717 INFO:tasks.workunit.client.0.smithi016.stdout:8/915: dread d2/d48/dad/d8f/d96/fd4 [0,4194304] 0 2024-04-22T15:49:48.717 INFO:tasks.workunit.client.1.smithi027.stdout:2/127: dwrite dd/f1c [0,4194304] 0 2024-04-22T15:49:48.720 INFO:tasks.workunit.client.0.smithi016.stdout:8/916: symlink d2/d82/d52/df0/dc1/d10a/l121 0 2024-04-22T15:49:48.721 INFO:tasks.workunit.client.1.smithi027.stdout:2/128: symlink dd/df/l21 0 2024-04-22T15:49:48.723 INFO:tasks.workunit.client.1.smithi027.stdout:2/129: fdatasync dd/df/f18 0 2024-04-22T15:49:48.725 INFO:tasks.workunit.client.0.smithi016.stdout:8/917: read d2/d10/f41 [679513,125883] 0 2024-04-22T15:49:48.726 INFO:tasks.workunit.client.0.smithi016.stdout:5/965: dread df/d15/d1a/d28/f2b [0,4194304] 0 2024-04-22T15:49:48.726 INFO:tasks.workunit.client.1.smithi027.stdout:2/130: getdents dd/d1d 0 2024-04-22T15:49:48.727 INFO:tasks.workunit.client.1.smithi027.stdout:2/131: write dd/f13 [435506,39006] 0 2024-04-22T15:49:48.731 INFO:tasks.workunit.client.0.smithi016.stdout:8/918: dwrite d2/d82/d52/df0/d6f/fa6 [0,4194304] 0 2024-04-22T15:49:48.732 INFO:tasks.workunit.client.0.smithi016.stdout:5/966: unlink df/d15/d1a/d50/d77/d7d/d8b/d93/dbb/lbf 0 2024-04-22T15:49:48.733 INFO:tasks.workunit.client.1.smithi027.stdout:2/132: unlink cb 0 2024-04-22T15:49:48.734 INFO:tasks.workunit.client.0.smithi016.stdout:5/967: write df/d5b/d70/f11b [740790,119221] 0 2024-04-22T15:49:48.735 INFO:tasks.workunit.client.0.smithi016.stdout:8/919: mknod d2/d48/dad/d8f/d96/c122 0 2024-04-22T15:49:48.738 INFO:tasks.workunit.client.0.smithi016.stdout:5/968: mkdir df/d5b/d5d/d12b 0 2024-04-22T15:49:48.739 INFO:tasks.workunit.client.0.smithi016.stdout:5/969: write df/d15/d1a/fbe [5021942,66692] 0 2024-04-22T15:49:48.740 INFO:tasks.workunit.client.0.smithi016.stdout:8/920: creat d2/d10/f123 x:0 0 0 2024-04-22T15:49:48.741 INFO:tasks.workunit.client.1.smithi027.stdout:2/133: link dd/df/f17 dd/f22 0 2024-04-22T15:49:48.741 INFO:tasks.workunit.client.0.smithi016.stdout:5/970: chown df/d15/d1a/d50/d77/fb4 33 1 2024-04-22T15:49:48.742 INFO:tasks.workunit.client.0.smithi016.stdout:8/921: mkdir d2/d82/d56/d99/d124 0 2024-04-22T15:49:48.744 INFO:tasks.workunit.client.0.smithi016.stdout:5/971: getdents df/d15/d1a/dd8 0 2024-04-22T15:49:48.745 INFO:tasks.workunit.client.1.smithi027.stdout:2/134: creat dd/df/f23 x:0 0 0 2024-04-22T15:49:48.745 INFO:tasks.workunit.client.0.smithi016.stdout:8/922: mknod d2/d82/d52/df0/db0/c125 0 2024-04-22T15:49:48.746 INFO:tasks.workunit.client.0.smithi016.stdout:5/972: creat df/d5b/db3/f12c x:0 0 0 2024-04-22T15:49:48.748 INFO:tasks.workunit.client.0.smithi016.stdout:5/973: stat df/d15/d1a/d50/d77/d7d/d8b/d8c/d10c 0 2024-04-22T15:49:48.750 INFO:tasks.workunit.client.0.smithi016.stdout:5/974: stat df/d15/d1a/l3e 0 2024-04-22T15:49:48.751 INFO:tasks.workunit.client.0.smithi016.stdout:5/975: write df/f1f [2116617,124671] 0 2024-04-22T15:49:48.751 INFO:tasks.workunit.client.0.smithi016.stdout:8/923: dwrite d2/d82/d52/f54 [0,4194304] 0 2024-04-22T15:49:48.753 INFO:tasks.workunit.client.0.smithi016.stdout:8/924: write d2/d82/d56/f64 [3306583,20099] 0 2024-04-22T15:49:48.754 INFO:tasks.workunit.client.0.smithi016.stdout:8/925: chown d2/d82/d52/df0/d66/d7e/dc0 950 1 2024-04-22T15:49:48.760 INFO:tasks.workunit.client.0.smithi016.stdout:8/926: read d2/f24 [2221412,108303] 0 2024-04-22T15:49:48.761 INFO:tasks.workunit.client.1.smithi027.stdout:2/135: truncate dd/f13 388151 0 2024-04-22T15:49:48.762 INFO:tasks.workunit.client.0.smithi016.stdout:8/927: fdatasync d2/f22 0 2024-04-22T15:49:48.764 INFO:tasks.workunit.client.0.smithi016.stdout:5/976: write df/f9b [616843,97630] 0 2024-04-22T15:49:48.766 INFO:tasks.workunit.client.0.smithi016.stdout:8/928: write d2/d82/d52/df0/d66/f3c [6321430,254] 0 2024-04-22T15:49:48.768 INFO:tasks.workunit.client.1.smithi027.stdout:2/136: dwrite dd/df/f12 [0,4194304] 0 2024-04-22T15:49:48.769 INFO:tasks.workunit.client.0.smithi016.stdout:8/929: truncate d2/d82/d52/df0/d66/ffb 657828 0 2024-04-22T15:49:48.773 INFO:tasks.workunit.client.0.smithi016.stdout:4/927: sync 2024-04-22T15:49:48.776 INFO:tasks.workunit.client.0.smithi016.stdout:4/928: unlink d7/d21/lc1 0 2024-04-22T15:49:48.777 INFO:tasks.workunit.client.0.smithi016.stdout:4/929: chown d7/d24/d41/fe9 255992 1 2024-04-22T15:49:48.781 INFO:tasks.workunit.client.1.smithi027.stdout:2/137: dwrite dd/df/f12 [4194304,4194304] 0 2024-04-22T15:49:48.785 INFO:tasks.workunit.client.0.smithi016.stdout:8/930: dwrite d2/d82/d56/f90 [0,4194304] 0 2024-04-22T15:49:48.787 INFO:tasks.workunit.client.0.smithi016.stdout:5/977: dread df/d15/d1a/d28/f3f [0,4194304] 0 2024-04-22T15:49:48.788 INFO:tasks.workunit.client.0.smithi016.stdout:4/930: dwrite d7/d3c/fe6 [0,4194304] 0 2024-04-22T15:49:48.788 INFO:tasks.workunit.client.1.smithi027.stdout:2/138: rename cc to dd/df/d1b/d1f/c24 0 2024-04-22T15:49:48.794 INFO:tasks.workunit.client.0.smithi016.stdout:4/931: rename f5 to d7/d24/d41/f133 0 2024-04-22T15:49:48.795 INFO:tasks.workunit.client.0.smithi016.stdout:8/931: truncate d2/d82/d52/df0/d66/d1c/f8c 6256000 0 2024-04-22T15:49:48.799 INFO:tasks.workunit.client.0.smithi016.stdout:5/978: dwrite df/d15/d1a/fe5 [0,4194304] 0 2024-04-22T15:49:48.802 INFO:tasks.workunit.client.1.smithi027.stdout:2/139: truncate dd/f1c 1811376 0 2024-04-22T15:49:48.803 INFO:tasks.workunit.client.0.smithi016.stdout:8/932: dwrite d2/d82/d52/df0/d66/d7e/fc5 [0,4194304] 0 2024-04-22T15:49:48.804 INFO:tasks.workunit.client.0.smithi016.stdout:5/979: symlink df/d5b/d5d/d5f/l12d 0 2024-04-22T15:49:48.806 INFO:tasks.workunit.client.0.smithi016.stdout:5/980: dread - df/d5b/db3/dc1/ff6 zero size 2024-04-22T15:49:48.814 INFO:tasks.workunit.client.0.smithi016.stdout:8/933: rename d2/d82/d52/df0/d66/d1c/d65/def to d2/d82/d3e/dd5/d126 0 2024-04-22T15:49:48.814 INFO:tasks.workunit.client.0.smithi016.stdout:8/934: chown d2/d48/dad/d8f 162941646 1 2024-04-22T15:49:48.818 INFO:tasks.workunit.client.1.smithi027.stdout:2/140: dread dd/f1c [0,4194304] 0 2024-04-22T15:49:48.820 INFO:tasks.workunit.client.0.smithi016.stdout:8/935: mkdir d2/d48/dad/d6a/d127 0 2024-04-22T15:49:48.821 INFO:tasks.workunit.client.1.smithi027.stdout:2/141: creat dd/df/d1b/f25 x:0 0 0 2024-04-22T15:49:48.822 INFO:tasks.workunit.client.1.smithi027.stdout:2/142: chown dd/f22 14470048 1 2024-04-22T15:49:48.825 INFO:tasks.workunit.client.0.smithi016.stdout:5/981: dwrite df/d15/d1a/d50/d77/d94/fc2 [0,4194304] 0 2024-04-22T15:49:48.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:48 smithi027 ceph-mon[34922]: pgmap v36: 65 pgs: 65 active+clean; 2.0 GiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 73 MiB/s rd, 208 MiB/s wr, 480 op/s 2024-04-22T15:49:48.836 INFO:tasks.workunit.client.1.smithi027.stdout:2/143: rmdir dd/df/d1b 39 2024-04-22T15:49:48.840 INFO:tasks.workunit.client.1.smithi027.stdout:2/144: dread f7 [0,4194304] 0 2024-04-22T15:49:48.844 INFO:tasks.workunit.client.1.smithi027.stdout:4/104: sync 2024-04-22T15:49:48.844 INFO:tasks.workunit.client.1.smithi027.stdout:2/145: creat dd/df/d1b/f26 x:0 0 0 2024-04-22T15:49:48.845 INFO:tasks.workunit.client.1.smithi027.stdout:8/134: sync 2024-04-22T15:49:48.845 INFO:tasks.workunit.client.1.smithi027.stdout:1/83: sync 2024-04-22T15:49:48.845 INFO:tasks.workunit.client.1.smithi027.stdout:7/82: sync 2024-04-22T15:49:48.845 INFO:tasks.workunit.client.1.smithi027.stdout:6/159: sync 2024-04-22T15:49:48.846 INFO:tasks.workunit.client.1.smithi027.stdout:2/146: fdatasync dd/f22 0 2024-04-22T15:49:48.846 INFO:tasks.workunit.client.0.smithi016.stdout:8/936: dread d2/f27 [4194304,4194304] 0 2024-04-22T15:49:48.848 INFO:tasks.workunit.client.0.smithi016.stdout:8/937: write d2/d82/d3e/dd5/df8/ff9 [751256,56625] 0 2024-04-22T15:49:48.848 INFO:tasks.workunit.client.1.smithi027.stdout:1/84: write d2/f8 [179276,80594] 0 2024-04-22T15:49:48.848 INFO:tasks.workunit.client.1.smithi027.stdout:8/135: write d1/f18 [948359,45248] 0 2024-04-22T15:49:48.849 INFO:tasks.workunit.client.1.smithi027.stdout:4/105: fsync d5/d7/f15 0 2024-04-22T15:49:48.850 INFO:tasks.workunit.client.1.smithi027.stdout:1/85: readlink d2/l14 0 2024-04-22T15:49:48.851 INFO:tasks.workunit.client.1.smithi027.stdout:7/83: mkdir d0/d16 0 2024-04-22T15:49:48.853 INFO:tasks.workunit.client.1.smithi027.stdout:6/160: dread d5/dc/d20/d2b/f12 [0,4194304] 0 2024-04-22T15:49:48.854 INFO:tasks.workunit.client.1.smithi027.stdout:6/161: write d5/dc/d20/d2b/f12 [7519948,128870] 0 2024-04-22T15:49:48.860 INFO:tasks.workunit.client.1.smithi027.stdout:6/162: dwrite d5/dc/dd/f16 [0,4194304] 0 2024-04-22T15:49:48.860 INFO:tasks.workunit.client.1.smithi027.stdout:2/147: symlink dd/df/d1b/l27 0 2024-04-22T15:49:48.862 INFO:tasks.workunit.client.1.smithi027.stdout:6/163: write d5/dc/d20/d2b/f24 [1762971,118127] 0 2024-04-22T15:49:48.863 INFO:tasks.workunit.client.1.smithi027.stdout:4/106: mkdir d5/de/d2a 0 2024-04-22T15:49:48.864 INFO:tasks.workunit.client.1.smithi027.stdout:6/164: write d5/dc/d26/f2d [930000,32981] 0 2024-04-22T15:49:48.868 INFO:tasks.workunit.client.1.smithi027.stdout:7/84: fdatasync d0/fd 0 2024-04-22T15:49:48.876 INFO:tasks.workunit.client.1.smithi027.stdout:4/107: mknod d5/d11/d1d/c2b 0 2024-04-22T15:49:48.877 INFO:tasks.workunit.client.1.smithi027.stdout:6/165: mknod d5/dc/d20/c33 0 2024-04-22T15:49:48.878 INFO:tasks.workunit.client.1.smithi027.stdout:6/166: truncate d5/f29 468175 0 2024-04-22T15:49:48.880 INFO:tasks.workunit.client.1.smithi027.stdout:7/85: truncate d0/d11/fe 589882 0 2024-04-22T15:49:48.883 INFO:tasks.workunit.client.0.smithi016.stdout:7/931: sync 2024-04-22T15:49:48.883 INFO:tasks.workunit.client.0.smithi016.stdout:2/995: sync 2024-04-22T15:49:48.883 INFO:tasks.workunit.client.0.smithi016.stdout:6/963: sync 2024-04-22T15:49:48.884 INFO:tasks.workunit.client.1.smithi027.stdout:4/108: creat d5/d7/f2c x:0 0 0 2024-04-22T15:49:48.884 INFO:tasks.workunit.client.1.smithi027.stdout:6/167: dwrite d5/dc/d20/d2b/f27 [0,4194304] 0 2024-04-22T15:49:48.888 INFO:tasks.workunit.client.1.smithi027.stdout:4/109: dread d5/f6 [0,4194304] 0 2024-04-22T15:49:48.889 INFO:tasks.workunit.client.0.smithi016.stdout:6/964: dwrite d3/da/d12/d13/d97/de0/fe4 [0,4194304] 0 2024-04-22T15:49:48.894 INFO:tasks.workunit.client.1.smithi027.stdout:1/86: getdents d2 0 2024-04-22T15:49:48.895 INFO:tasks.workunit.client.1.smithi027.stdout:1/87: fdatasync d2/fa 0 2024-04-22T15:49:48.903 INFO:tasks.workunit.client.1.smithi027.stdout:2/148: rename dd/f13 to dd/df/f28 0 2024-04-22T15:49:48.908 INFO:tasks.workunit.client.1.smithi027.stdout:1/88: creat d2/f15 x:0 0 0 2024-04-22T15:49:48.915 INFO:tasks.workunit.client.1.smithi027.stdout:4/110: mkdir d5/d11/d1d/d24/d2d 0 2024-04-22T15:49:48.921 INFO:tasks.workunit.client.1.smithi027.stdout:6/168: write d5/db/d2e/ff [436728,105771] 0 2024-04-22T15:49:48.922 INFO:tasks.workunit.client.1.smithi027.stdout:6/169: write d5/dc/d17/f1b [3822403,90921] 0 2024-04-22T15:49:48.923 INFO:tasks.workunit.client.1.smithi027.stdout:4/111: mknod d5/d11/d1d/d24/d2d/c2e 0 2024-04-22T15:49:48.924 INFO:tasks.workunit.client.1.smithi027.stdout:6/170: write d5/dc/d17/f1b [75382,73462] 0 2024-04-22T15:49:48.925 INFO:tasks.workunit.client.1.smithi027.stdout:4/112: chown d5/d11/d1d/f27 26 1 2024-04-22T15:49:48.925 INFO:tasks.workunit.client.1.smithi027.stdout:6/171: fsync d5/dc/d26/f2d 0 2024-04-22T15:49:48.931 INFO:tasks.workunit.client.1.smithi027.stdout:7/86: write d0/d11/fe [732631,13230] 0 2024-04-22T15:49:48.932 INFO:tasks.workunit.client.0.smithi016.stdout:4/932: sync 2024-04-22T15:49:48.932 INFO:tasks.workunit.client.1.smithi027.stdout:4/113: mkdir d5/de/d2f 0 2024-04-22T15:49:48.935 INFO:tasks.workunit.client.1.smithi027.stdout:6/172: rmdir d5/dc/dd 39 2024-04-22T15:49:48.939 INFO:tasks.workunit.client.1.smithi027.stdout:2/149: dwrite dd/df/f17 [4194304,4194304] 0 2024-04-22T15:49:48.940 INFO:tasks.workunit.client.1.smithi027.stdout:7/87: write d0/fb [1401632,129646] 0 2024-04-22T15:49:48.941 INFO:tasks.workunit.client.1.smithi027.stdout:7/88: fdatasync d0/f3 0 2024-04-22T15:49:48.942 INFO:tasks.workunit.client.1.smithi027.stdout:4/114: mkdir d5/de/d30 0 2024-04-22T15:49:48.946 INFO:tasks.workunit.client.1.smithi027.stdout:6/173: rename d5/db/d2e/d1f to d5/dc/d20/d2b/d34 0 2024-04-22T15:49:48.949 INFO:tasks.workunit.client.1.smithi027.stdout:7/89: dwrite d0/f13 [0,4194304] 0 2024-04-22T15:49:48.950 INFO:tasks.workunit.client.1.smithi027.stdout:1/89: unlink d2/f6 0 2024-04-22T15:49:48.950 INFO:tasks.workunit.client.1.smithi027.stdout:7/90: chown d0/f3 71 1 2024-04-22T15:49:48.950 INFO:tasks.workunit.client.1.smithi027.stdout:6/174: dread d5/db/d2e/ff [0,4194304] 0 2024-04-22T15:49:48.951 INFO:tasks.workunit.client.1.smithi027.stdout:7/91: write d0/fd [1046290,37528] 0 2024-04-22T15:49:48.952 INFO:tasks.workunit.client.1.smithi027.stdout:7/92: stat d0 0 2024-04-22T15:49:48.952 INFO:tasks.workunit.client.1.smithi027.stdout:6/175: chown d5/dc/d17/l1d 4661 1 2024-04-22T15:49:48.953 INFO:tasks.workunit.client.1.smithi027.stdout:7/93: truncate d0/d11/fe 1358842 0 2024-04-22T15:49:48.954 INFO:tasks.workunit.client.1.smithi027.stdout:7/94: rename d0/d11 to d0/d11/d17 22 2024-04-22T15:49:48.956 INFO:tasks.workunit.client.1.smithi027.stdout:2/150: creat dd/df/d1b/f29 x:0 0 0 2024-04-22T15:49:48.959 INFO:tasks.workunit.client.0.smithi016.stdout:5/982: mkdir df/d15/d1a/d50/d77/d7d/d12e 0 2024-04-22T15:49:48.965 INFO:tasks.workunit.client.0.smithi016.stdout:7/932: creat d3/dc/d1b/d5a/d59/d64/dc9/f150 x:0 0 0 2024-04-22T15:49:48.966 INFO:tasks.workunit.client.0.smithi016.stdout:5/983: dwrite df/d15/d1a/d50/d77/d7d/d8b/d93/dfa/d103/f121 [0,4194304] 0 2024-04-22T15:49:48.968 INFO:tasks.workunit.client.1.smithi027.stdout:6/176: creat d5/dc/d20/d2b/f35 x:0 0 0 2024-04-22T15:49:48.971 INFO:tasks.workunit.client.1.smithi027.stdout:7/95: mknod d0/d11/c18 0 2024-04-22T15:49:48.973 INFO:tasks.workunit.client.1.smithi027.stdout:7/96: write d0/f14 [1134844,19954] 0 2024-04-22T15:49:48.976 INFO:tasks.workunit.client.0.smithi016.stdout:8/938: mkdir d2/d128 0 2024-04-22T15:49:48.977 INFO:tasks.workunit.client.0.smithi016.stdout:8/939: dread - d2/d10/d47/d67/f10b zero size 2024-04-22T15:49:48.978 INFO:tasks.workunit.client.0.smithi016.stdout:8/940: chown d2/d82/d3e/dd5/fd8 112461 1 2024-04-22T15:49:48.979 INFO:tasks.workunit.client.0.smithi016.stdout:8/941: write d2/d32/f10c [760028,35861] 0 2024-04-22T15:49:48.981 INFO:tasks.workunit.client.1.smithi027.stdout:9/131: sync 2024-04-22T15:49:48.984 INFO:tasks.workunit.client.1.smithi027.stdout:6/177: symlink d5/dc/d20/d2b/d34/l36 0 2024-04-22T15:49:48.986 INFO:tasks.workunit.client.0.smithi016.stdout:5/984: creat df/d15/d1a/d50/d77/d94/f12f x:0 0 0 2024-04-22T15:49:48.987 INFO:tasks.workunit.client.1.smithi027.stdout:7/97: creat d0/f19 x:0 0 0 2024-04-22T15:49:48.989 INFO:tasks.workunit.client.1.smithi027.stdout:2/151: mknod dd/c2a 0 2024-04-22T15:49:48.991 INFO:tasks.workunit.client.0.smithi016.stdout:6/965: dwrite d3/da/dd/f18 [0,4194304] 0 2024-04-22T15:49:48.991 INFO:tasks.workunit.client.1.smithi027.stdout:7/98: dread d0/f14 [0,4194304] 0 2024-04-22T15:49:48.992 INFO:tasks.workunit.client.1.smithi027.stdout:4/115: write f2 [1414297,125019] 0 2024-04-22T15:49:48.992 INFO:tasks.workunit.client.0.smithi016.stdout:6/966: dread - d3/da/dd/d67/d6c/db9/dfa/f128 zero size 2024-04-22T15:49:48.993 INFO:tasks.workunit.client.0.smithi016.stdout:2/996: creat d2/d2c/d97/d67/f17c x:0 0 0 2024-04-22T15:49:48.993 INFO:tasks.workunit.client.1.smithi027.stdout:4/116: fdatasync d5/de/f16 0 2024-04-22T15:49:48.994 INFO:tasks.workunit.client.1.smithi027.stdout:4/117: dread - d5/f13 zero size 2024-04-22T15:49:48.998 INFO:tasks.workunit.client.0.smithi016.stdout:5/985: truncate df/d15/d1a/d28/f2b 3669163 0 2024-04-22T15:49:49.000 INFO:tasks.workunit.client.0.smithi016.stdout:7/933: dwrite d3/de6/f2c [0,4194304] 0 2024-04-22T15:49:49.001 INFO:tasks.workunit.client.0.smithi016.stdout:7/934: chown d3/de6/lc5 247 1 2024-04-22T15:49:49.002 INFO:tasks.workunit.client.0.smithi016.stdout:4/933: truncate d7/d24/f3d 2542306 0 2024-04-22T15:49:49.002 INFO:tasks.workunit.client.1.smithi027.stdout:9/132: dwrite d0/d8/f18 [4194304,4194304] 0 2024-04-22T15:49:49.003 INFO:tasks.workunit.client.0.smithi016.stdout:2/997: chown d2/de/d2a/f49 508913260 1 2024-04-22T15:49:49.004 INFO:tasks.workunit.client.0.smithi016.stdout:7/935: read d3/dc/d33/f4d [431358,26496] 0 2024-04-22T15:49:49.005 INFO:tasks.workunit.client.1.smithi027.stdout:1/90: dwrite d2/f11 [0,4194304] 0 2024-04-22T15:49:49.006 INFO:tasks.workunit.client.1.smithi027.stdout:7/99: symlink d0/l1a 0 2024-04-22T15:49:49.006 INFO:tasks.workunit.client.0.smithi016.stdout:6/967: creat d3/da/ddd/d101/d11b/d83/f137 x:0 0 0 2024-04-22T15:49:49.007 INFO:tasks.workunit.client.1.smithi027.stdout:9/133: write d0/d8/f11 [1064966,78717] 0 2024-04-22T15:49:49.008 INFO:tasks.workunit.client.0.smithi016.stdout:8/942: link d2/d48/dad/d6a/f9a d2/d82/d56/d99/d124/f129 0 2024-04-22T15:49:49.012 INFO:tasks.workunit.client.0.smithi016.stdout:5/986: truncate df/d15/d1a/d50/d77/f4c 2165617 0 2024-04-22T15:49:49.018 INFO:tasks.workunit.client.0.smithi016.stdout:4/934: creat d7/d4e/d102/d106/d88/f134 x:0 0 0 2024-04-22T15:49:49.021 INFO:tasks.workunit.client.0.smithi016.stdout:2/998: symlink d2/d1a/d7b/dc6/l17d 0 2024-04-22T15:49:49.022 INFO:tasks.workunit.client.0.smithi016.stdout:2/999: readlink d2/d23/db8/dd4/l105 0 2024-04-22T15:49:49.023 INFO:tasks.workunit.client.0.smithi016.stdout:7/936: mknod d3/dc/d1b/d5a/d4a/d6e/c151 0 2024-04-22T15:49:49.025 INFO:tasks.workunit.client.0.smithi016.stdout:6/968: dread d3/da/d12/d13/dd2/dee/fda [0,4194304] 0 2024-04-22T15:49:49.026 INFO:tasks.workunit.client.1.smithi027.stdout:2/152: dwrite dd/f1c [0,4194304] 0 2024-04-22T15:49:49.030 INFO:tasks.workunit.client.1.smithi027.stdout:9/134: creat d0/dc/de/f2d x:0 0 0 2024-04-22T15:49:49.033 INFO:tasks.workunit.client.0.smithi016.stdout:6/969: fsync d3/da/d12/d13/dd2/dee/fa9 0 2024-04-22T15:49:49.034 INFO:tasks.workunit.client.1.smithi027.stdout:6/178: link d5/dc/d20/c33 d5/db/c37 0 2024-04-22T15:49:49.034 INFO:tasks.workunit.client.0.smithi016.stdout:6/970: stat d3/da/dd/d1b 0 2024-04-22T15:49:49.035 INFO:tasks.workunit.client.0.smithi016.stdout:6/971: readlink d3/da/d12/d13/l5e 0 2024-04-22T15:49:49.035 INFO:tasks.workunit.client.1.smithi027.stdout:6/179: write d5/db/f11 [4511917,109886] 0 2024-04-22T15:49:49.036 INFO:tasks.workunit.client.0.smithi016.stdout:5/987: mknod df/d5b/c130 0 2024-04-22T15:49:49.042 INFO:tasks.workunit.client.1.smithi027.stdout:1/91: symlink d2/l16 0 2024-04-22T15:49:49.043 INFO:tasks.workunit.client.1.smithi027.stdout:4/118: dwrite d5/f9 [0,4194304] 0 2024-04-22T15:49:49.044 INFO:tasks.workunit.client.1.smithi027.stdout:4/119: dread - d5/f26 zero size 2024-04-22T15:49:49.051 INFO:tasks.workunit.client.0.smithi016.stdout:6/972: creat d3/da/d12/d13/dd2/dee/f138 x:0 0 0 2024-04-22T15:49:49.054 INFO:tasks.workunit.client.0.smithi016.stdout:5/988: rename df/d5b/d5d/deb/f114 to df/d15/d1a/d50/d77/d7d/d8b/d8c/f131 0 2024-04-22T15:49:49.057 INFO:tasks.workunit.client.0.smithi016.stdout:4/935: rmdir d7/d4e/d102/d106/d88/deb/d101 0 2024-04-22T15:49:49.059 INFO:tasks.workunit.client.1.smithi027.stdout:2/153: link dd/df/f17 dd/df/d1b/f2b 0 2024-04-22T15:49:49.059 INFO:tasks.workunit.client.0.smithi016.stdout:7/937: link d3/dc/d9f/ce5 d3/d10b/c152 0 2024-04-22T15:49:49.062 INFO:tasks.workunit.client.1.smithi027.stdout:2/154: dread f7 [0,4194304] 0 2024-04-22T15:49:49.062 INFO:tasks.workunit.client.1.smithi027.stdout:9/135: rename d0/l6 to d0/dc/de/l2e 0 2024-04-22T15:49:49.064 INFO:tasks.workunit.client.0.smithi016.stdout:8/943: getdents d2/d48/dad/d8f/d96 0 2024-04-22T15:49:49.069 INFO:tasks.workunit.client.1.smithi027.stdout:4/120: truncate f0 3461593 0 2024-04-22T15:49:49.069 INFO:tasks.workunit.client.0.smithi016.stdout:8/944: dread d2/d10/f41 [0,4194304] 0 2024-04-22T15:49:49.070 INFO:tasks.workunit.client.1.smithi027.stdout:7/100: getdents d0/d11 0 2024-04-22T15:49:49.071 INFO:tasks.workunit.client.1.smithi027.stdout:7/101: fsync d0/f13 0 2024-04-22T15:49:49.073 INFO:tasks.workunit.client.1.smithi027.stdout:6/180: rename d5/l22 to d5/dc/d20/d2b/d34/l38 0 2024-04-22T15:49:49.074 INFO:tasks.workunit.client.1.smithi027.stdout:6/181: chown d5/dc/l21 100 1 2024-04-22T15:49:49.075 INFO:tasks.workunit.client.0.smithi016.stdout:7/938: fsync d3/dc/d1b/d5a/d9/d109/f13a 0 2024-04-22T15:49:49.080 INFO:tasks.workunit.client.1.smithi027.stdout:5/134: sync 2024-04-22T15:49:49.082 INFO:tasks.workunit.client.0.smithi016.stdout:6/973: dwrite d3/da/dd/d7b/d12c/fd3 [0,4194304] 0 2024-04-22T15:49:49.083 INFO:tasks.workunit.client.1.smithi027.stdout:9/136: rename d0/dc/c2b to d0/dc/df/c2f 0 2024-04-22T15:49:49.083 INFO:tasks.workunit.client.0.smithi016.stdout:6/974: write d3/da/d12/d13/dd2/dee/f105 [970608,39265] 0 2024-04-22T15:49:49.086 INFO:tasks.workunit.client.1.smithi027.stdout:6/182: creat d5/dc/d17/f39 x:0 0 0 2024-04-22T15:49:49.091 INFO:tasks.workunit.client.1.smithi027.stdout:7/102: symlink d0/d16/l1b 0 2024-04-22T15:49:49.093 INFO:tasks.workunit.client.0.smithi016.stdout:5/989: dwrite df/d15/d1a/d4e/d10a/f3b [0,4194304] 0 2024-04-22T15:49:49.093 INFO:tasks.workunit.client.0.smithi016.stdout:8/945: creat d2/d48/dad/d8f/d96/de4/f12a x:0 0 0 2024-04-22T15:49:49.094 INFO:tasks.workunit.client.0.smithi016.stdout:6/975: fdatasync d3/da/dd/fb2 0 2024-04-22T15:49:49.095 INFO:tasks.workunit.client.1.smithi027.stdout:4/121: dwrite f0 [0,4194304] 0 2024-04-22T15:49:49.098 INFO:tasks.workunit.client.0.smithi016.stdout:4/936: creat d7/d21/d45/f135 x:0 0 0 2024-04-22T15:49:49.101 INFO:tasks.workunit.client.1.smithi027.stdout:6/183: rename d5/dc/d20/l31 to d5/dc/d20/d2b/d34/l3a 0 2024-04-22T15:49:49.102 INFO:tasks.workunit.client.1.smithi027.stdout:7/103: dwrite d0/fb [0,4194304] 0 2024-04-22T15:49:49.103 INFO:tasks.workunit.client.0.smithi016.stdout:7/939: mknod d3/dc/d33/d84/de3/c153 0 2024-04-22T15:49:49.104 INFO:tasks.workunit.client.1.smithi027.stdout:1/92: link d2/lb d2/l17 0 2024-04-22T15:49:49.106 INFO:tasks.workunit.client.1.smithi027.stdout:1/93: write d2/f15 [399499,121567] 0 2024-04-22T15:49:49.106 INFO:tasks.workunit.client.1.smithi027.stdout:1/94: readlink d2/l16 0 2024-04-22T15:49:49.110 INFO:tasks.workunit.client.1.smithi027.stdout:2/155: getdents dd/df/d1b/d1f 0 2024-04-22T15:49:49.112 INFO:tasks.workunit.client.1.smithi027.stdout:4/122: rmdir d5/d11/d1d/d24/d2d 39 2024-04-22T15:49:49.114 INFO:tasks.workunit.client.0.smithi016.stdout:4/937: read d7/d4e/d2f/d6f/fae [476432,40738] 0 2024-04-22T15:49:49.118 INFO:tasks.workunit.client.1.smithi027.stdout:2/156: creat dd/df/d1b/d1f/f2c x:0 0 0 2024-04-22T15:49:49.118 INFO:tasks.workunit.client.0.smithi016.stdout:8/946: mknod d2/d48/dad/d8f/c12b 0 2024-04-22T15:49:49.119 INFO:tasks.workunit.client.1.smithi027.stdout:1/95: dread d2/fa [0,4194304] 0 2024-04-22T15:49:49.122 INFO:tasks.workunit.client.1.smithi027.stdout:1/96: write d2/fa [23162,57559] 0 2024-04-22T15:49:49.122 INFO:tasks.workunit.client.1.smithi027.stdout:4/123: dwrite f0 [0,4194304] 0 2024-04-22T15:49:49.122 INFO:tasks.workunit.client.0.smithi016.stdout:7/940: mknod d3/dc/d1b/d31/d13f/c154 0 2024-04-22T15:49:49.123 INFO:tasks.workunit.client.1.smithi027.stdout:5/135: dwrite d0/df/d10/f25 [0,4194304] 0 2024-04-22T15:49:49.124 INFO:tasks.workunit.client.0.smithi016.stdout:4/938: dread - d7/d4e/d39/fa4 zero size 2024-04-22T15:49:49.125 INFO:tasks.workunit.client.1.smithi027.stdout:5/136: stat d0/df/de/f29 0 2024-04-22T15:49:49.126 INFO:tasks.workunit.client.0.smithi016.stdout:4/939: truncate d7/d22/fc9 113294 0 2024-04-22T15:49:49.126 INFO:tasks.workunit.client.0.smithi016.stdout:4/940: stat d7/d21/fc6 0 2024-04-22T15:49:49.129 INFO:tasks.workunit.client.1.smithi027.stdout:4/124: symlink d5/l31 0 2024-04-22T15:49:49.129 INFO:tasks.workunit.client.1.smithi027.stdout:4/125: read - d5/d11/f28 zero size 2024-04-22T15:49:49.130 INFO:tasks.workunit.client.1.smithi027.stdout:4/126: dread - d5/f26 zero size 2024-04-22T15:49:49.130 INFO:tasks.workunit.client.0.smithi016.stdout:8/947: creat d2/d82/d3e/dd5/d126/f12c x:0 0 0 2024-04-22T15:49:49.130 INFO:tasks.workunit.client.1.smithi027.stdout:7/104: getdents d0/d16 0 2024-04-22T15:49:49.135 INFO:tasks.workunit.client.1.smithi027.stdout:4/127: dread d5/f6 [0,4194304] 0 2024-04-22T15:49:49.136 INFO:tasks.workunit.client.0.smithi016.stdout:6/976: dread d3/da/db/d14/d16/f20 [0,4194304] 0 2024-04-22T15:49:49.137 INFO:tasks.workunit.client.1.smithi027.stdout:7/105: dread d0/d11/fe [0,4194304] 0 2024-04-22T15:49:49.138 INFO:tasks.workunit.client.0.smithi016.stdout:6/977: fdatasync d3/da/d12/d13/f133 0 2024-04-22T15:49:49.138 INFO:tasks.workunit.client.0.smithi016.stdout:7/941: read d3/dc/d1b/d5a/da4/f137 [1324049,20505] 0 2024-04-22T15:49:49.139 INFO:tasks.workunit.client.1.smithi027.stdout:2/157: creat dd/f2d x:0 0 0 2024-04-22T15:49:49.139 INFO:tasks.workunit.client.0.smithi016.stdout:5/990: truncate df/d15/f45 2609337 0 2024-04-22T15:49:49.140 INFO:tasks.workunit.client.1.smithi027.stdout:2/158: read - dd/f20 zero size 2024-04-22T15:49:49.140 INFO:tasks.workunit.client.0.smithi016.stdout:5/991: fdatasync df/d15/d1a/d50/fff 0 2024-04-22T15:49:49.141 INFO:tasks.workunit.client.0.smithi016.stdout:8/948: unlink d2/d82/c2d 0 2024-04-22T15:49:49.144 INFO:tasks.workunit.client.1.smithi027.stdout:7/106: dwrite d0/f3 [0,4194304] 0 2024-04-22T15:49:49.146 INFO:tasks.workunit.client.1.smithi027.stdout:7/107: dread - d0/f19 zero size 2024-04-22T15:49:49.148 INFO:tasks.workunit.client.1.smithi027.stdout:2/159: creat dd/df/d1b/f2e x:0 0 0 2024-04-22T15:49:49.149 INFO:tasks.workunit.client.1.smithi027.stdout:2/160: chown dd/df/f12 373880239 1 2024-04-22T15:49:49.151 INFO:tasks.workunit.client.0.smithi016.stdout:4/941: creat d7/f136 x:0 0 0 2024-04-22T15:49:49.154 INFO:tasks.workunit.client.0.smithi016.stdout:5/992: symlink df/d90/dad/d115/l132 0 2024-04-22T15:49:49.155 INFO:tasks.workunit.client.0.smithi016.stdout:5/993: fsync df/d15/d1a/d4e/d58/fd4 0 2024-04-22T15:49:49.156 INFO:tasks.workunit.client.0.smithi016.stdout:5/994: chown df/d15/d1a/d50/d77/cc7 97435 1 2024-04-22T15:49:49.157 INFO:tasks.workunit.client.0.smithi016.stdout:8/949: mkdir d2/d82/d3e/d12d 0 2024-04-22T15:49:49.159 INFO:tasks.workunit.client.1.smithi027.stdout:1/97: rename d2/f11 to d2/f18 0 2024-04-22T15:49:49.160 INFO:tasks.workunit.client.0.smithi016.stdout:7/942: write d3/dc/d1b/d5a/d59/d64/f89 [5858704,48360] 0 2024-04-22T15:49:49.160 INFO:tasks.workunit.client.1.smithi027.stdout:4/128: symlink d5/de/d2f/l32 0 2024-04-22T15:49:49.161 INFO:tasks.workunit.client.1.smithi027.stdout:4/129: readlink d5/l17 0 2024-04-22T15:49:49.161 INFO:tasks.workunit.client.1.smithi027.stdout:4/130: fdatasync d5/f13 0 2024-04-22T15:49:49.163 INFO:tasks.workunit.client.1.smithi027.stdout:7/108: mkdir d0/d11/d1c 0 2024-04-22T15:49:49.163 INFO:tasks.workunit.client.1.smithi027.stdout:7/109: fsync d0/d11/fe 0 2024-04-22T15:49:49.164 INFO:tasks.workunit.client.0.smithi016.stdout:5/995: mkdir df/d15/d1a/d50/d77/d7d/d8b/d93/dbb/d133 0 2024-04-22T15:49:49.167 INFO:tasks.workunit.client.1.smithi027.stdout:2/161: rmdir dd/df 39 2024-04-22T15:49:49.168 INFO:tasks.workunit.client.1.smithi027.stdout:2/162: fsync dd/f1c 0 2024-04-22T15:49:49.170 INFO:tasks.workunit.client.0.smithi016.stdout:7/943: write d3/de6/ffc [469538,22922] 0 2024-04-22T15:49:49.172 INFO:tasks.workunit.client.0.smithi016.stdout:5/996: rmdir df/d15/d1a/dd8 39 2024-04-22T15:49:49.178 INFO:tasks.workunit.client.1.smithi027.stdout:7/110: rename d0/l4 to d0/d11/l1d 0 2024-04-22T15:49:49.178 INFO:tasks.workunit.client.1.smithi027.stdout:7/111: chown d0/l1 1543917 1 2024-04-22T15:49:49.179 INFO:tasks.workunit.client.0.smithi016.stdout:8/950: truncate d2/d82/d52/df0/d66/d1c/f8c 1578795 0 2024-04-22T15:49:49.180 INFO:tasks.workunit.client.1.smithi027.stdout:7/112: write d0/f3 [4656225,6968] 0 2024-04-22T15:49:49.180 INFO:tasks.workunit.client.1.smithi027.stdout:5/137: getdents d0/df/de/d24 0 2024-04-22T15:49:49.180 INFO:tasks.workunit.client.0.smithi016.stdout:8/951: fdatasync d2/d10/d47/d67/f10b 0 2024-04-22T15:49:49.181 INFO:tasks.workunit.client.1.smithi027.stdout:5/138: chown d0/d1a/d22/d26 319 1 2024-04-22T15:49:49.182 INFO:tasks.workunit.client.0.smithi016.stdout:4/942: creat d7/d4e/d39/f137 x:0 0 0 2024-04-22T15:49:49.182 INFO:tasks.workunit.client.0.smithi016.stdout:4/943: chown d7/d4e/d2f/d6f 15790 1 2024-04-22T15:49:49.183 INFO:tasks.workunit.client.0.smithi016.stdout:7/944: symlink d3/dc/d1b/d31/d5f/l155 0 2024-04-22T15:49:49.184 INFO:tasks.workunit.client.1.smithi027.stdout:1/98: symlink d2/l19 0 2024-04-22T15:49:49.187 INFO:tasks.workunit.client.1.smithi027.stdout:5/139: dwrite d0/df/de/f29 [0,4194304] 0 2024-04-22T15:49:49.197 INFO:tasks.workunit.client.0.smithi016.stdout:8/952: symlink d2/d48/dad/d8f/l12e 0 2024-04-22T15:49:49.198 INFO:tasks.workunit.client.0.smithi016.stdout:8/953: dread - d2/d82/d52/df0/d66/d7e/dc0/de2/f11a zero size 2024-04-22T15:49:49.201 INFO:tasks.workunit.client.0.smithi016.stdout:7/945: symlink d3/dc/d1b/l156 0 2024-04-22T15:49:49.201 INFO:tasks.workunit.client.0.smithi016.stdout:5/997: creat df/d15/d1a/d4e/f134 x:0 0 0 2024-04-22T15:49:49.202 INFO:tasks.workunit.client.0.smithi016.stdout:8/954: dread d2/d82/d3e/dd5/df8/ff9 [0,4194304] 0 2024-04-22T15:49:49.203 INFO:tasks.workunit.client.0.smithi016.stdout:8/955: dread - d2/d82/d3e/dd5/d126/ff2 zero size 2024-04-22T15:49:49.203 INFO:tasks.workunit.client.0.smithi016.stdout:4/944: rmdir d7/d4e/d102/d106/d88/d9d/dad/db9/d109 0 2024-04-22T15:49:49.207 INFO:tasks.workunit.client.0.smithi016.stdout:7/946: symlink d3/dc/d1b/d5a/d59/d64/dfe/l157 0 2024-04-22T15:49:49.207 INFO:tasks.workunit.client.0.smithi016.stdout:8/956: mkdir d2/d82/d52/df0/db0/d12f 0 2024-04-22T15:49:49.208 INFO:tasks.workunit.client.0.smithi016.stdout:8/957: write d2/d10/d43/da9/fac [73752,109352] 0 2024-04-22T15:49:49.210 INFO:tasks.workunit.client.0.smithi016.stdout:8/958: dread d2/d10/d43/da9/fac [0,4194304] 0 2024-04-22T15:49:49.211 INFO:tasks.workunit.client.0.smithi016.stdout:7/947: creat d3/dc/d22/f158 x:0 0 0 2024-04-22T15:49:49.212 INFO:tasks.workunit.client.0.smithi016.stdout:4/945: symlink d7/d21/d45/l138 0 2024-04-22T15:49:49.213 INFO:tasks.workunit.client.0.smithi016.stdout:5/998: write df/fb6 [984763,116189] 0 2024-04-22T15:49:49.215 INFO:tasks.workunit.client.0.smithi016.stdout:8/959: rmdir d2/d10 39 2024-04-22T15:49:49.216 INFO:tasks.workunit.client.0.smithi016.stdout:7/948: fdatasync d3/de6/fad 0 2024-04-22T15:49:49.217 INFO:tasks.workunit.client.0.smithi016.stdout:4/946: creat d7/d24/d51/f139 x:0 0 0 2024-04-22T15:49:49.219 INFO:tasks.workunit.client.0.smithi016.stdout:5/999: chown df/d15/d1a/d50/d77/d7d/caa 25526 1 2024-04-22T15:49:49.221 INFO:tasks.workunit.client.0.smithi016.stdout:4/947: creat d7/d22/f13a x:0 0 0 2024-04-22T15:49:49.222 INFO:tasks.workunit.client.0.smithi016.stdout:4/948: dread - d7/d4e/d102/d106/d88/d9d/dad/db9/f131 zero size 2024-04-22T15:49:49.224 INFO:tasks.workunit.client.0.smithi016.stdout:7/949: creat d3/dc/d1b/d5a/d9/f159 x:0 0 0 2024-04-22T15:49:49.226 INFO:tasks.workunit.client.0.smithi016.stdout:4/949: creat d7/d21/d45/d49/d67/f13b x:0 0 0 2024-04-22T15:49:49.228 INFO:tasks.workunit.client.0.smithi016.stdout:7/950: creat d3/dc/d1b/d5a/d9/f15a x:0 0 0 2024-04-22T15:49:49.228 INFO:tasks.workunit.client.0.smithi016.stdout:8/960: dwrite d2/d10/d43/da9/fac [0,4194304] 0 2024-04-22T15:49:49.229 INFO:tasks.workunit.client.0.smithi016.stdout:4/950: mknod d7/d4e/d102/d106/d88/d9d/dad/c13c 0 2024-04-22T15:49:49.230 INFO:tasks.workunit.client.1.smithi027.stdout:3/174: sync 2024-04-22T15:49:49.230 INFO:tasks.workunit.client.1.smithi027.stdout:0/144: sync 2024-04-22T15:49:49.230 INFO:tasks.workunit.client.1.smithi027.stdout:8/136: sync 2024-04-22T15:49:49.230 INFO:tasks.workunit.client.0.smithi016.stdout:7/951: truncate d3/de6/fad 383450 0 2024-04-22T15:49:49.231 INFO:tasks.workunit.client.0.smithi016.stdout:4/951: truncate d7/d21/d45/d49/d9b/fdf 4757723 0 2024-04-22T15:49:49.231 INFO:tasks.workunit.client.0.smithi016.stdout:7/952: write d3/dc/f145 [213244,37175] 0 2024-04-22T15:49:49.232 INFO:tasks.workunit.client.0.smithi016.stdout:8/961: unlink d2/d82/d52/df0/d66/d7e/fc5 0 2024-04-22T15:49:49.234 INFO:tasks.workunit.client.0.smithi016.stdout:7/953: chown d3/dc/d1b/d5a/d9/df/d94/ddf/l139 362794 1 2024-04-22T15:49:49.235 INFO:tasks.workunit.client.0.smithi016.stdout:4/952: creat d7/d4e/d2f/d50/f13d x:0 0 0 2024-04-22T15:49:49.236 INFO:tasks.workunit.client.0.smithi016.stdout:8/962: rmdir d2/d10/d47/d67 39 2024-04-22T15:49:49.236 INFO:tasks.workunit.client.0.smithi016.stdout:7/954: dread - d3/dc/d1b/d5a/d59/d119/f14d zero size 2024-04-22T15:49:49.237 INFO:tasks.workunit.client.0.smithi016.stdout:4/953: write d7/d4e/d2f/d33/ffd [1729055,89855] 0 2024-04-22T15:49:49.237 INFO:tasks.workunit.client.0.smithi016.stdout:7/955: dread - d3/d11a/f11b zero size 2024-04-22T15:49:49.238 INFO:tasks.workunit.client.0.smithi016.stdout:8/963: creat d2/d30/f130 x:0 0 0 2024-04-22T15:49:49.240 INFO:tasks.workunit.client.0.smithi016.stdout:4/954: mknod d7/d4e/d2f/d50/c13e 0 2024-04-22T15:49:49.241 INFO:tasks.workunit.client.0.smithi016.stdout:4/955: dread - d7/d4e/d39/f137 zero size 2024-04-22T15:49:49.243 INFO:tasks.workunit.client.1.smithi027.stdout:7/113: truncate d0/f14 1526351 0 2024-04-22T15:49:49.244 INFO:tasks.workunit.client.0.smithi016.stdout:8/964: dwrite d2/d48/dad/d8f/fb3 [0,4194304] 0 2024-04-22T15:49:49.245 INFO:tasks.workunit.client.1.smithi027.stdout:4/131: link d5/l31 d5/de/d30/l33 0 2024-04-22T15:49:49.245 INFO:tasks.workunit.client.1.smithi027.stdout:5/140: write d0/f21 [45242,126962] 0 2024-04-22T15:49:49.246 INFO:tasks.workunit.client.1.smithi027.stdout:7/114: fdatasync d0/d11/fe 0 2024-04-22T15:49:49.247 INFO:tasks.workunit.client.1.smithi027.stdout:0/145: chown d0/f12 3 1 2024-04-22T15:49:49.247 INFO:tasks.workunit.client.1.smithi027.stdout:4/132: read d5/f6 [2966765,9767] 0 2024-04-22T15:49:49.247 INFO:tasks.workunit.client.1.smithi027.stdout:3/175: unlink d28/f29 0 2024-04-22T15:49:49.249 INFO:tasks.workunit.client.1.smithi027.stdout:5/141: truncate d0/df/de/d24/f16 115183 0 2024-04-22T15:49:49.255 INFO:tasks.workunit.client.1.smithi027.stdout:7/115: unlink d0/c12 0 2024-04-22T15:49:49.255 INFO:tasks.workunit.client.1.smithi027.stdout:4/133: dwrite d5/f9 [0,4194304] 0 2024-04-22T15:49:49.257 INFO:tasks.workunit.client.1.smithi027.stdout:0/146: fdatasync d0/f1 0 2024-04-22T15:49:49.257 INFO:tasks.workunit.client.0.smithi016.stdout:4/956: getdents d7/d4e/d39/df3 0 2024-04-22T15:49:49.262 INFO:tasks.workunit.client.1.smithi027.stdout:4/134: chown d5/de/d2f/l32 240620 1 2024-04-22T15:49:49.262 INFO:tasks.workunit.client.1.smithi027.stdout:3/176: rename f20 to d28/f37 0 2024-04-22T15:49:49.262 INFO:tasks.workunit.client.1.smithi027.stdout:5/142: fsync d0/f3 0 2024-04-22T15:49:49.265 INFO:tasks.workunit.client.1.smithi027.stdout:3/177: write f1f [432380,22064] 0 2024-04-22T15:49:49.267 INFO:tasks.workunit.client.1.smithi027.stdout:1/99: dread d2/f8 [0,4194304] 0 2024-04-22T15:49:49.267 INFO:tasks.workunit.client.1.smithi027.stdout:5/143: write d0/f15 [2006762,93313] 0 2024-04-22T15:49:49.267 INFO:tasks.workunit.client.1.smithi027.stdout:3/178: truncate f22 4931462 0 2024-04-22T15:49:49.268 INFO:tasks.workunit.client.1.smithi027.stdout:7/116: dwrite d0/f13 [4194304,4194304] 0 2024-04-22T15:49:49.269 INFO:tasks.workunit.client.1.smithi027.stdout:1/100: read d2/fa [165233,2043] 0 2024-04-22T15:49:49.269 INFO:tasks.workunit.client.1.smithi027.stdout:4/135: mknod d5/d11/d1d/c34 0 2024-04-22T15:49:49.270 INFO:tasks.workunit.client.1.smithi027.stdout:5/144: stat d0/d1a/l1e 0 2024-04-22T15:49:49.271 INFO:tasks.workunit.client.1.smithi027.stdout:0/147: creat d0/f2e x:0 0 0 2024-04-22T15:49:49.271 INFO:tasks.workunit.client.1.smithi027.stdout:7/117: chown d0/f19 43 1 2024-04-22T15:49:49.273 INFO:tasks.workunit.client.1.smithi027.stdout:1/101: write d2/fa [464338,124721] 0 2024-04-22T15:49:49.279 INFO:tasks.workunit.client.1.smithi027.stdout:3/179: dwrite f15 [0,4194304] 0 2024-04-22T15:49:49.285 INFO:tasks.workunit.client.1.smithi027.stdout:4/136: mknod d5/de/d2a/c35 0 2024-04-22T15:49:49.285 INFO:tasks.workunit.client.1.smithi027.stdout:4/137: chown d5/d11/l22 1 1 2024-04-22T15:49:49.286 INFO:tasks.workunit.client.1.smithi027.stdout:3/180: creat d28/f38 x:0 0 0 2024-04-22T15:49:49.287 INFO:tasks.workunit.client.1.smithi027.stdout:4/138: stat d5/de/d2a/c35 0 2024-04-22T15:49:49.289 INFO:tasks.workunit.client.0.smithi016.stdout:8/965: dwrite d2/f8 [0,4194304] 0 2024-04-22T15:49:49.291 INFO:tasks.workunit.client.0.smithi016.stdout:8/966: write d2/d82/d52/df0/d66/d1c/d65/f9c [2154437,37064] 0 2024-04-22T15:49:49.291 INFO:tasks.workunit.client.1.smithi027.stdout:5/145: link d0/f3 d0/d1a/d22/f2b 0 2024-04-22T15:49:49.294 INFO:tasks.workunit.client.1.smithi027.stdout:3/181: rename d28/c32 to d28/c39 0 2024-04-22T15:49:49.296 INFO:tasks.workunit.client.1.smithi027.stdout:4/139: creat d5/d11/f36 x:0 0 0 2024-04-22T15:49:49.300 INFO:tasks.workunit.client.0.smithi016.stdout:8/967: creat d2/d77/dfa/f131 x:0 0 0 2024-04-22T15:49:49.300 INFO:tasks.workunit.client.1.smithi027.stdout:5/146: dwrite d0/df/de/f18 [0,4194304] 0 2024-04-22T15:49:49.303 INFO:tasks.workunit.client.1.smithi027.stdout:4/140: creat d5/de/f37 x:0 0 0 2024-04-22T15:49:49.304 INFO:tasks.workunit.client.0.smithi016.stdout:8/968: mknod d2/d82/d52/df0/d66/d1c/c132 0 2024-04-22T15:49:49.305 INFO:tasks.workunit.client.1.smithi027.stdout:1/102: dread d2/f3 [0,4194304] 0 2024-04-22T15:49:49.306 INFO:tasks.workunit.client.1.smithi027.stdout:1/103: stat d2/fa 0 2024-04-22T15:49:49.306 INFO:tasks.workunit.client.1.smithi027.stdout:3/182: rename d28/l2f to d28/l3a 0 2024-04-22T15:49:49.309 INFO:tasks.workunit.client.1.smithi027.stdout:5/147: mknod d0/d1a/d22/d26/c2c 0 2024-04-22T15:49:49.311 INFO:tasks.workunit.client.1.smithi027.stdout:4/141: dwrite d5/d7/f10 [0,4194304] 0 2024-04-22T15:49:49.312 INFO:tasks.workunit.client.1.smithi027.stdout:0/148: write d0/f22 [786265,117124] 0 2024-04-22T15:49:49.314 INFO:tasks.workunit.client.0.smithi016.stdout:8/969: unlink d2/d10/c15 0 2024-04-22T15:49:49.319 INFO:tasks.workunit.client.0.smithi016.stdout:8/970: symlink d2/d32/l133 0 2024-04-22T15:49:49.319 INFO:tasks.workunit.client.1.smithi027.stdout:4/142: dwrite d5/d7/f2c [0,4194304] 0 2024-04-22T15:49:49.321 INFO:tasks.workunit.client.1.smithi027.stdout:5/148: rename d0/f21 to d0/df/de/f2d 0 2024-04-22T15:49:49.324 INFO:tasks.workunit.client.1.smithi027.stdout:1/104: mkdir d2/d1a 0 2024-04-22T15:49:49.325 INFO:tasks.workunit.client.1.smithi027.stdout:1/105: stat d2/f3 0 2024-04-22T15:49:49.326 INFO:tasks.workunit.client.0.smithi016.stdout:8/971: creat d2/d77/dfa/f134 x:0 0 0 2024-04-22T15:49:49.329 INFO:tasks.workunit.client.1.smithi027.stdout:1/106: dread d2/f3 [0,4194304] 0 2024-04-22T15:49:49.332 INFO:tasks.workunit.client.0.smithi016.stdout:8/972: symlink d2/d82/d52/df0/d6f/d102/l135 0 2024-04-22T15:49:49.333 INFO:tasks.workunit.client.1.smithi027.stdout:4/143: symlink d5/d11/d1d/l38 0 2024-04-22T15:49:49.335 INFO:tasks.workunit.client.1.smithi027.stdout:5/149: rename d0/df/de/f29 to d0/df/de/d24/f2e 0 2024-04-22T15:49:49.336 INFO:tasks.workunit.client.1.smithi027.stdout:0/149: mknod d0/d28/c2f 0 2024-04-22T15:49:49.340 INFO:tasks.workunit.client.1.smithi027.stdout:4/144: truncate d5/d7/f15 511657 0 2024-04-22T15:49:49.342 INFO:tasks.workunit.client.1.smithi027.stdout:4/145: read d5/d7/f10 [2515464,102026] 0 2024-04-22T15:49:49.343 INFO:tasks.workunit.client.1.smithi027.stdout:4/146: readlink d5/d11/l19 0 2024-04-22T15:49:49.345 INFO:tasks.workunit.client.1.smithi027.stdout:3/183: getdents d28 0 2024-04-22T15:49:49.345 INFO:tasks.workunit.client.1.smithi027.stdout:3/184: chown d28/c35 19057670 1 2024-04-22T15:49:49.346 INFO:tasks.workunit.client.1.smithi027.stdout:1/107: mknod d2/d1a/c1b 0 2024-04-22T15:49:49.350 INFO:tasks.workunit.client.1.smithi027.stdout:4/147: symlink d5/de/d2a/l39 0 2024-04-22T15:49:49.351 INFO:tasks.workunit.client.1.smithi027.stdout:3/185: rmdir d28 39 2024-04-22T15:49:49.351 INFO:tasks.workunit.client.1.smithi027.stdout:3/186: fdatasync f22 0 2024-04-22T15:49:49.352 INFO:tasks.workunit.client.1.smithi027.stdout:3/187: write f5 [439486,70379] 0 2024-04-22T15:49:49.352 INFO:tasks.workunit.client.1.smithi027.stdout:1/108: creat d2/d1a/f1c x:0 0 0 2024-04-22T15:49:49.353 INFO:tasks.workunit.client.1.smithi027.stdout:3/188: truncate f21 4210633 0 2024-04-22T15:49:49.355 INFO:tasks.workunit.client.1.smithi027.stdout:0/150: dread d0/d28/f1e [0,4194304] 0 2024-04-22T15:49:49.356 INFO:tasks.workunit.client.1.smithi027.stdout:4/148: rename f0 to d5/de/d2a/f3a 0 2024-04-22T15:49:49.356 INFO:tasks.workunit.client.1.smithi027.stdout:4/149: dread - d5/d11/f28 zero size 2024-04-22T15:49:49.357 INFO:tasks.workunit.client.0.smithi016.stdout:8/973: truncate d2/d82/d52/df0/d66/d7e/d85/de8/f10e 2612913 0 2024-04-22T15:49:49.358 INFO:tasks.workunit.client.1.smithi027.stdout:4/150: write d5/f23 [163621,83661] 0 2024-04-22T15:49:49.362 INFO:tasks.workunit.client.1.smithi027.stdout:0/151: dwrite d0/d28/f20 [0,4194304] 0 2024-04-22T15:49:49.364 INFO:tasks.workunit.client.1.smithi027.stdout:0/152: read - d0/d2a/f2b zero size 2024-04-22T15:49:49.367 INFO:tasks.workunit.client.1.smithi027.stdout:5/150: dwrite d0/df/f12 [0,4194304] 0 2024-04-22T15:49:49.368 INFO:tasks.workunit.client.1.smithi027.stdout:4/151: write d5/d11/d1d/f27 [2556316,77738] 0 2024-04-22T15:49:49.369 INFO:tasks.workunit.client.1.smithi027.stdout:5/151: write d0/df/d10/f25 [5107966,23603] 0 2024-04-22T15:49:49.370 INFO:tasks.workunit.client.1.smithi027.stdout:4/152: fdatasync d5/de/f37 0 2024-04-22T15:49:49.372 INFO:tasks.workunit.client.1.smithi027.stdout:3/189: creat d28/f3b x:0 0 0 2024-04-22T15:49:49.372 INFO:tasks.workunit.client.1.smithi027.stdout:3/190: chown l8 109700034 1 2024-04-22T15:49:49.374 INFO:tasks.workunit.client.1.smithi027.stdout:3/191: write f11 [969684,35308] 0 2024-04-22T15:49:49.378 INFO:tasks.workunit.client.0.smithi016.stdout:8/974: dread d2/d82/d52/df0/d66/d7e/d85/de8/f10e [0,4194304] 0 2024-04-22T15:49:49.378 INFO:tasks.workunit.client.1.smithi027.stdout:1/109: rename d2/d1a to d2/d1d 0 2024-04-22T15:49:49.379 INFO:tasks.workunit.client.0.smithi016.stdout:8/975: write d2/d10/f123 [70658,13039] 0 2024-04-22T15:49:49.380 INFO:tasks.workunit.client.1.smithi027.stdout:1/110: truncate d2/d1d/f1c 456213 0 2024-04-22T15:49:49.381 INFO:tasks.workunit.client.1.smithi027.stdout:4/153: rename d5/d11/c12 to d5/de/c3b 0 2024-04-22T15:49:49.382 INFO:tasks.workunit.client.0.smithi016.stdout:8/976: dread d2/f8 [0,4194304] 0 2024-04-22T15:49:49.383 INFO:tasks.workunit.client.1.smithi027.stdout:4/154: truncate d5/d11/f28 203684 0 2024-04-22T15:49:49.389 INFO:tasks.workunit.client.1.smithi027.stdout:3/192: rename f22 to d28/f3c 0 2024-04-22T15:49:49.389 INFO:tasks.workunit.client.0.smithi016.stdout:6/978: sync 2024-04-22T15:49:49.391 INFO:tasks.workunit.client.1.smithi027.stdout:4/155: write d5/d7/fb [1715401,84279] 0 2024-04-22T15:49:49.392 INFO:tasks.workunit.client.1.smithi027.stdout:1/111: symlink d2/l1e 0 2024-04-22T15:49:49.392 INFO:tasks.workunit.client.1.smithi027.stdout:4/156: chown d5/d7 16693070 1 2024-04-22T15:49:49.396 INFO:tasks.workunit.client.1.smithi027.stdout:5/152: link d0/df/d10/l28 d0/d1a/l2f 0 2024-04-22T15:49:49.397 INFO:tasks.workunit.client.1.smithi027.stdout:3/193: rmdir d28 39 2024-04-22T15:49:49.397 INFO:tasks.workunit.client.1.smithi027.stdout:3/194: chown f1c 2008 1 2024-04-22T15:49:49.400 INFO:tasks.workunit.client.1.smithi027.stdout:5/153: dread d0/f7 [0,4194304] 0 2024-04-22T15:49:49.400 INFO:tasks.workunit.client.1.smithi027.stdout:4/157: creat d5/de/d2f/f3c x:0 0 0 2024-04-22T15:49:49.400 INFO:tasks.workunit.client.1.smithi027.stdout:5/154: stat d0/f15 0 2024-04-22T15:49:49.401 INFO:tasks.workunit.client.0.smithi016.stdout:6/979: write d3/da/d12/d13/dd2/dee/fb4 [4980574,15952] 0 2024-04-22T15:49:49.404 INFO:tasks.workunit.client.1.smithi027.stdout:4/158: symlink d5/d11/l3d 0 2024-04-22T15:49:49.405 INFO:tasks.workunit.client.1.smithi027.stdout:4/159: chown d5/d7/fb 3520838 1 2024-04-22T15:49:49.406 INFO:tasks.workunit.client.1.smithi027.stdout:5/155: mknod d0/c30 0 2024-04-22T15:49:49.410 INFO:tasks.workunit.client.1.smithi027.stdout:1/112: creat d2/f1f x:0 0 0 2024-04-22T15:49:49.410 INFO:tasks.workunit.client.1.smithi027.stdout:3/195: rename d28/c35 to d28/c3d 0 2024-04-22T15:49:49.413 INFO:tasks.workunit.client.1.smithi027.stdout:3/196: creat d28/f3e x:0 0 0 2024-04-22T15:49:49.413 INFO:tasks.workunit.client.1.smithi027.stdout:4/160: symlink d5/d11/d1d/d24/l3e 0 2024-04-22T15:49:49.415 INFO:tasks.workunit.client.1.smithi027.stdout:4/161: readlink d5/de/d30/l33 0 2024-04-22T15:49:49.415 INFO:tasks.workunit.client.1.smithi027.stdout:1/113: dwrite d2/f1f [0,4194304] 0 2024-04-22T15:49:49.416 INFO:tasks.workunit.client.1.smithi027.stdout:3/197: unlink l1 0 2024-04-22T15:49:49.416 INFO:tasks.workunit.client.1.smithi027.stdout:4/162: write d5/f9 [448939,322] 0 2024-04-22T15:49:49.417 INFO:tasks.workunit.client.0.smithi016.stdout:6/980: dread d3/da/dd/d7b/fb0 [0,4194304] 0 2024-04-22T15:49:49.418 INFO:tasks.workunit.client.1.smithi027.stdout:4/163: dread - d5/de/f16 zero size 2024-04-22T15:49:49.420 INFO:tasks.workunit.client.0.smithi016.stdout:6/981: truncate d3/da/dd/d67/d6c/ff6 4736187 0 2024-04-22T15:49:49.420 INFO:tasks.workunit.client.1.smithi027.stdout:5/156: getdents d0/df/de 0 2024-04-22T15:49:49.422 INFO:tasks.workunit.client.1.smithi027.stdout:1/114: mknod d2/d1d/c20 0 2024-04-22T15:49:49.423 INFO:tasks.workunit.client.1.smithi027.stdout:3/198: dwrite f5 [0,4194304] 0 2024-04-22T15:49:49.425 INFO:tasks.workunit.client.1.smithi027.stdout:4/164: creat d5/de/d2a/f3f x:0 0 0 2024-04-22T15:49:49.426 INFO:tasks.workunit.client.1.smithi027.stdout:4/165: dread - d5/d11/f36 zero size 2024-04-22T15:49:49.428 INFO:tasks.workunit.client.0.smithi016.stdout:6/982: symlink d3/da/db/l139 0 2024-04-22T15:49:49.431 INFO:tasks.workunit.client.1.smithi027.stdout:3/199: rmdir d28 39 2024-04-22T15:49:49.431 INFO:tasks.workunit.client.1.smithi027.stdout:3/200: readlink l23 0 2024-04-22T15:49:49.432 INFO:tasks.workunit.client.1.smithi027.stdout:4/166: dread d5/d7/f2c [0,4194304] 0 2024-04-22T15:49:49.433 INFO:tasks.workunit.client.1.smithi027.stdout:4/167: fdatasync d5/f23 0 2024-04-22T15:49:49.434 INFO:tasks.workunit.client.1.smithi027.stdout:4/168: write d5/d11/d1d/f27 [5029005,27862] 0 2024-04-22T15:49:49.437 INFO:tasks.workunit.client.1.smithi027.stdout:4/169: write d5/f13 [240456,24962] 0 2024-04-22T15:49:49.437 INFO:tasks.workunit.client.0.smithi016.stdout:6/983: mknod d3/da/dd/c13a 0 2024-04-22T15:49:49.440 INFO:tasks.workunit.client.0.smithi016.stdout:6/984: fsync d3/da/db/d14/f86 0 2024-04-22T15:49:49.441 INFO:tasks.workunit.client.0.smithi016.stdout:6/985: chown d3/da/d12/d13/d97 730200228 1 2024-04-22T15:49:49.442 INFO:tasks.workunit.client.1.smithi027.stdout:5/157: dwrite d0/f15 [0,4194304] 0 2024-04-22T15:49:49.443 INFO:tasks.workunit.client.1.smithi027.stdout:5/158: stat d0/df/de/d24/f2e 0 2024-04-22T15:49:49.447 INFO:tasks.workunit.client.1.smithi027.stdout:3/201: rename d28/f3e to d28/f3f 0 2024-04-22T15:49:49.449 INFO:tasks.workunit.client.1.smithi027.stdout:1/115: truncate d2/f1f 4137092 0 2024-04-22T15:49:49.451 INFO:tasks.workunit.client.0.smithi016.stdout:6/986: unlink d3/da/dd/d7b/d12c/d9e/ce1 0 2024-04-22T15:49:49.455 INFO:tasks.workunit.client.0.smithi016.stdout:6/987: truncate d3/da/db/d14/d16/f5f 970839 0 2024-04-22T15:49:49.456 INFO:tasks.workunit.client.1.smithi027.stdout:1/116: dread d2/fa [0,4194304] 0 2024-04-22T15:49:49.457 INFO:tasks.workunit.client.0.smithi016.stdout:6/988: dread d3/da/fc3 [0,4194304] 0 2024-04-22T15:49:49.460 INFO:tasks.workunit.client.1.smithi027.stdout:4/170: dwrite d5/d7/ff [0,4194304] 0 2024-04-22T15:49:49.460 INFO:tasks.workunit.client.1.smithi027.stdout:1/117: dread d2/f15 [0,4194304] 0 2024-04-22T15:49:49.462 INFO:tasks.workunit.client.0.smithi016.stdout:6/989: rmdir d3/da/d12/d13/dd2 39 2024-04-22T15:49:49.465 INFO:tasks.workunit.client.1.smithi027.stdout:5/159: truncate d0/df/d10/f14 1721275 0 2024-04-22T15:49:49.467 INFO:tasks.workunit.client.1.smithi027.stdout:5/160: symlink d0/df/de/l31 0 2024-04-22T15:49:49.468 INFO:tasks.workunit.client.0.smithi016.stdout:7/956: sync 2024-04-22T15:49:49.469 INFO:tasks.workunit.client.0.smithi016.stdout:6/990: mknod d3/da/ddd/d101/d11b/c13b 0 2024-04-22T15:49:49.470 INFO:tasks.workunit.client.0.smithi016.stdout:7/957: chown d3/dc/d1b/d5a/d9/df/d94/ddf 212 1 2024-04-22T15:49:49.470 INFO:tasks.workunit.client.1.smithi027.stdout:5/161: mkdir d0/d1a/d22/d26/d32 0 2024-04-22T15:49:49.471 INFO:tasks.workunit.client.1.smithi027.stdout:4/171: creat d5/d11/d1d/d24/f40 x:0 0 0 2024-04-22T15:49:49.472 INFO:tasks.workunit.client.0.smithi016.stdout:6/991: chown d3/da/db/d14/d4c/f52 130 1 2024-04-22T15:49:49.475 INFO:tasks.workunit.client.0.smithi016.stdout:7/958: fsync d3/dc/d1b/d5a/d4a/f55 0 2024-04-22T15:49:49.477 INFO:tasks.workunit.client.1.smithi027.stdout:5/162: rename d0/d1a/d22/f2b to d0/df/d10/f33 0 2024-04-22T15:49:49.479 INFO:tasks.workunit.client.1.smithi027.stdout:5/163: read d0/df/de/f2d [97037,40696] 0 2024-04-22T15:49:49.480 INFO:tasks.workunit.client.1.smithi027.stdout:4/172: dwrite d5/f25 [0,4194304] 0 2024-04-22T15:49:49.480 INFO:tasks.workunit.client.1.smithi027.stdout:1/118: getdents d2 0 2024-04-22T15:49:49.481 INFO:tasks.workunit.client.0.smithi016.stdout:7/959: dwrite d3/d11a/f11b [0,4194304] 0 2024-04-22T15:49:49.482 INFO:tasks.workunit.client.1.smithi027.stdout:9/137: sync 2024-04-22T15:49:49.488 INFO:tasks.workunit.client.1.smithi027.stdout:5/164: creat d0/df/de/d24/d1c/f34 x:0 0 0 2024-04-22T15:49:49.492 INFO:tasks.workunit.client.1.smithi027.stdout:5/165: dread d0/f6 [0,4194304] 0 2024-04-22T15:49:49.510 INFO:tasks.workunit.client.0.smithi016.stdout:4/957: sync 2024-04-22T15:49:49.510 INFO:tasks.workunit.client.0.smithi016.stdout:8/977: sync 2024-04-22T15:49:49.512 INFO:tasks.workunit.client.0.smithi016.stdout:4/958: truncate d7/d4e/d39/f137 345729 0 2024-04-22T15:49:49.512 INFO:tasks.workunit.client.0.smithi016.stdout:4/959: write d7/d4e/d2f/d50/f13d [918415,71517] 0 2024-04-22T15:49:49.523 INFO:tasks.workunit.client.0.smithi016.stdout:7/960: creat d3/d11a/f15b x:0 0 0 2024-04-22T15:49:49.524 INFO:tasks.workunit.client.0.smithi016.stdout:7/961: write d3/dc/d33/d84/f147 [400296,58361] 0 2024-04-22T15:49:49.525 INFO:tasks.workunit.client.1.smithi027.stdout:9/138: mkdir d0/dc/d12/d30 0 2024-04-22T15:49:49.528 INFO:tasks.workunit.client.0.smithi016.stdout:4/960: dread - d7/d24/d41/fe9 zero size 2024-04-22T15:49:49.532 INFO:tasks.workunit.client.0.smithi016.stdout:8/978: getdents d2/d48/db9 0 2024-04-22T15:49:49.535 INFO:tasks.workunit.client.1.smithi027.stdout:9/139: rename d0/dc/f1e to d0/dc/df/f31 0 2024-04-22T15:49:49.537 INFO:tasks.workunit.client.1.smithi027.stdout:9/140: write d0/dc/de/f2d [378284,60570] 0 2024-04-22T15:49:49.537 INFO:tasks.workunit.client.0.smithi016.stdout:6/992: dwrite d3/da/fc3 [0,4194304] 0 2024-04-22T15:49:49.538 INFO:tasks.workunit.client.1.smithi027.stdout:5/166: dwrite d0/f7 [0,4194304] 0 2024-04-22T15:49:49.539 INFO:tasks.workunit.client.0.smithi016.stdout:6/993: readlink d3/lde 0 2024-04-22T15:49:49.540 INFO:tasks.workunit.client.0.smithi016.stdout:8/979: mknod d2/d82/d3e/d8e/de6/c136 0 2024-04-22T15:49:49.540 INFO:tasks.workunit.client.0.smithi016.stdout:8/980: stat d2/d10/d43/dbe 0 2024-04-22T15:49:49.541 INFO:tasks.workunit.client.1.smithi027.stdout:1/119: truncate d2/f8 1054059 0 2024-04-22T15:49:49.542 INFO:tasks.workunit.client.0.smithi016.stdout:8/981: write d2/d82/d3e/d8e/ff6 [376069,108462] 0 2024-04-22T15:49:49.544 INFO:tasks.workunit.client.1.smithi027.stdout:4/173: truncate d5/de/f21 151029 0 2024-04-22T15:49:49.545 INFO:tasks.workunit.client.1.smithi027.stdout:4/174: stat d5/de/d2a/f3f 0 2024-04-22T15:49:49.553 INFO:tasks.workunit.client.0.smithi016.stdout:8/982: dread d2/d32/f10c [0,4194304] 0 2024-04-22T15:49:49.554 INFO:tasks.workunit.client.1.smithi027.stdout:1/120: rename d2/l12 to d2/l21 0 2024-04-22T15:49:49.555 INFO:tasks.workunit.client.0.smithi016.stdout:8/983: write d2/d82/d52/df0/d66/d7e/f100 [25515,58424] 0 2024-04-22T15:49:49.555 INFO:tasks.workunit.client.0.smithi016.stdout:4/961: write d7/d21/d7c/fe8 [530730,91421] 0 2024-04-22T15:49:49.558 INFO:tasks.workunit.client.0.smithi016.stdout:7/962: link d3/dc/d1b/d31/l8d d3/d11a/def/l15c 0 2024-04-22T15:49:49.560 INFO:tasks.workunit.client.0.smithi016.stdout:6/994: mknod d3/da/dd/d67/d6c/db9/dfa/d118/c13c 0 2024-04-22T15:49:49.561 INFO:tasks.workunit.client.1.smithi027.stdout:4/175: dwrite d5/d11/f18 [0,4194304] 0 2024-04-22T15:49:49.562 INFO:tasks.workunit.client.1.smithi027.stdout:4/176: read - d5/de/f16 zero size 2024-04-22T15:49:49.563 INFO:tasks.workunit.client.0.smithi016.stdout:6/995: read d3/da/dd/d7b/d12c/d10c/f120 [3377053,101493] 0 2024-04-22T15:49:49.566 INFO:tasks.workunit.client.0.smithi016.stdout:4/962: symlink d7/d21/d45/d49/l13f 0 2024-04-22T15:49:49.570 INFO:tasks.workunit.client.0.smithi016.stdout:8/984: dwrite d2/f5a [0,4194304] 0 2024-04-22T15:49:49.574 INFO:tasks.workunit.client.0.smithi016.stdout:4/963: mknod d7/d21/d7c/d103/c140 0 2024-04-22T15:49:49.575 INFO:tasks.workunit.client.1.smithi027.stdout:9/141: truncate d0/dc/d12/f23 1194497 0 2024-04-22T15:49:49.575 INFO:tasks.workunit.client.1.smithi027.stdout:9/142: stat d0/dc/de 0 2024-04-22T15:49:49.577 INFO:tasks.workunit.client.1.smithi027.stdout:9/143: write d0/dc/de/f24 [1123338,70381] 0 2024-04-22T15:49:49.582 INFO:tasks.workunit.client.1.smithi027.stdout:5/167: truncate d0/df/f12 3093572 0 2024-04-22T15:49:49.587 INFO:tasks.workunit.client.0.smithi016.stdout:7/963: dread d3/f74 [0,4194304] 0 2024-04-22T15:49:49.589 INFO:tasks.workunit.client.1.smithi027.stdout:1/121: truncate d2/f3 3529336 0 2024-04-22T15:49:49.590 INFO:tasks.workunit.client.1.smithi027.stdout:4/177: dwrite d5/f26 [0,4194304] 0 2024-04-22T15:49:49.591 INFO:tasks.workunit.client.1.smithi027.stdout:9/144: creat d0/dc/d12/d30/f32 x:0 0 0 2024-04-22T15:49:49.592 INFO:tasks.workunit.client.0.smithi016.stdout:4/964: dread d7/d3c/d96/fb8 [0,4194304] 0 2024-04-22T15:49:49.592 INFO:tasks.workunit.client.0.smithi016.stdout:6/996: truncate d3/da/d12/d13/d38/df5/f109 1748215 0 2024-04-22T15:49:49.595 INFO:tasks.workunit.client.0.smithi016.stdout:7/964: creat d3/dc/d1b/d5a/d59/d119/f15d x:0 0 0 2024-04-22T15:49:49.595 INFO:tasks.workunit.client.0.smithi016.stdout:4/965: write d7/d21/d45/d49/d67/fbd [4117286,81841] 0 2024-04-22T15:49:49.597 INFO:tasks.workunit.client.0.smithi016.stdout:7/965: chown d3/d11a/d14f 747 1 2024-04-22T15:49:49.599 INFO:tasks.workunit.client.1.smithi027.stdout:9/145: dwrite d0/d8/f11 [0,4194304] 0 2024-04-22T15:49:49.601 INFO:tasks.workunit.client.1.smithi027.stdout:9/146: truncate d0/dc/d12/d30/f32 173220 0 2024-04-22T15:49:49.603 INFO:tasks.workunit.client.1.smithi027.stdout:5/168: write d0/f6 [1250445,76349] 0 2024-04-22T15:49:49.605 INFO:tasks.workunit.client.0.smithi016.stdout:6/997: dwrite d3/da/dd/d1b/f12f [0,4194304] 0 2024-04-22T15:49:49.605 INFO:tasks.workunit.client.0.smithi016.stdout:7/966: write d3/dc/d33/f4d [4095793,19284] 0 2024-04-22T15:49:49.606 INFO:tasks.workunit.client.1.smithi027.stdout:1/122: symlink d2/l22 0 2024-04-22T15:49:49.608 INFO:tasks.workunit.client.1.smithi027.stdout:9/147: symlink d0/dc/de/l33 0 2024-04-22T15:49:49.608 INFO:tasks.workunit.client.1.smithi027.stdout:5/169: dread d0/f6 [0,4194304] 0 2024-04-22T15:49:49.610 INFO:tasks.workunit.client.1.smithi027.stdout:9/148: write d0/d8/f18 [5990036,44861] 0 2024-04-22T15:49:49.613 INFO:tasks.workunit.client.0.smithi016.stdout:6/998: symlink d3/da/ddd/d101/d11b/de9/l13d 0 2024-04-22T15:49:49.615 INFO:tasks.workunit.client.1.smithi027.stdout:6/184: sync 2024-04-22T15:49:49.615 INFO:tasks.workunit.client.1.smithi027.stdout:8/137: sync 2024-04-22T15:49:49.615 INFO:tasks.workunit.client.1.smithi027.stdout:2/163: sync 2024-04-22T15:49:49.622 INFO:tasks.workunit.client.0.smithi016.stdout:7/967: symlink d3/dc/d1b/d5a/d9/df/l15e 0 2024-04-22T15:49:49.624 INFO:tasks.workunit.client.1.smithi027.stdout:5/170: creat d0/df/de/d24/f35 x:0 0 0 2024-04-22T15:49:49.625 INFO:tasks.workunit.client.1.smithi027.stdout:9/149: symlink d0/d8/l34 0 2024-04-22T15:49:49.626 INFO:tasks.workunit.client.1.smithi027.stdout:4/178: link d5/de/d2f/l32 d5/de/d30/l41 0 2024-04-22T15:49:49.626 INFO:tasks.workunit.client.0.smithi016.stdout:6/999: dwrite d3/da/db/d14/d4c/d58/f117 [0,4194304] 0 2024-04-22T15:49:49.631 INFO:tasks.workunit.client.1.smithi027.stdout:8/138: creat d1/d1e/f33 x:0 0 0 2024-04-22T15:49:49.632 INFO:tasks.workunit.client.1.smithi027.stdout:9/150: dwrite d0/dc/f25 [0,4194304] 0 2024-04-22T15:49:49.635 INFO:tasks.workunit.client.1.smithi027.stdout:1/123: mknod d2/d1d/c23 0 2024-04-22T15:49:49.635 INFO:tasks.workunit.client.0.smithi016.stdout:4/966: dread d7/f1e [4194304,4194304] 0 2024-04-22T15:49:49.640 INFO:tasks.workunit.client.0.smithi016.stdout:7/968: getdents d3/dc/d9f 0 2024-04-22T15:49:49.640 INFO:tasks.workunit.client.1.smithi027.stdout:9/151: dwrite d0/fb [0,4194304] 0 2024-04-22T15:49:49.642 INFO:tasks.workunit.client.1.smithi027.stdout:9/152: chown d0/dc/d12/d30/f32 2 1 2024-04-22T15:49:49.642 INFO:tasks.workunit.client.1.smithi027.stdout:5/171: symlink d0/d1a/l36 0 2024-04-22T15:49:49.643 INFO:tasks.workunit.client.1.smithi027.stdout:5/172: readlink d0/d1a/l36 0 2024-04-22T15:49:49.644 INFO:tasks.workunit.client.1.smithi027.stdout:9/153: dread d0/dc/d12/d30/f32 [0,4194304] 0 2024-04-22T15:49:49.648 INFO:tasks.workunit.client.0.smithi016.stdout:4/967: mkdir d7/d21/d141 0 2024-04-22T15:49:49.649 INFO:tasks.workunit.client.1.smithi027.stdout:6/185: rename d5/dc/d20/d2b/d34/l36 to d5/dc/dd/l3b 0 2024-04-22T15:49:49.651 INFO:tasks.workunit.client.0.smithi016.stdout:7/969: mkdir d3/dc/d33/d84/d15f 0 2024-04-22T15:49:49.654 INFO:tasks.workunit.client.0.smithi016.stdout:4/968: dwrite d7/f1e [0,4194304] 0 2024-04-22T15:49:49.655 INFO:tasks.workunit.client.0.smithi016.stdout:4/969: dread - d7/d24/d51/f139 zero size 2024-04-22T15:49:49.657 INFO:tasks.workunit.client.0.smithi016.stdout:7/970: mknod d3/dc/d1b/d5a/c160 0 2024-04-22T15:49:49.660 INFO:tasks.workunit.client.0.smithi016.stdout:4/970: read d7/d4e/d2f/d6f/fa8 [2586629,10677] 0 2024-04-22T15:49:49.662 INFO:tasks.workunit.client.1.smithi027.stdout:5/173: unlink d0/df/de/f2d 0 2024-04-22T15:49:49.662 INFO:tasks.workunit.client.0.smithi016.stdout:7/971: mknod d3/dc/d1b/d31/d88/c161 0 2024-04-22T15:49:49.663 INFO:tasks.workunit.client.1.smithi027.stdout:9/154: mknod d0/dc/d12/d30/c35 0 2024-04-22T15:49:49.664 INFO:tasks.workunit.client.0.smithi016.stdout:7/972: write d3/de6/f85 [1793770,123851] 0 2024-04-22T15:49:49.665 INFO:tasks.workunit.client.0.smithi016.stdout:4/971: symlink d7/d4e/d102/l142 0 2024-04-22T15:49:49.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:49 smithi016 ceph-mon[26620]: pgmap v37: 65 pgs: 65 active+clean; 2.3 GiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 79 MiB/s rd, 223 MiB/s wr, 520 op/s 2024-04-22T15:49:49.668 INFO:tasks.workunit.client.1.smithi027.stdout:9/155: dread d0/fb [0,4194304] 0 2024-04-22T15:49:49.668 INFO:tasks.workunit.client.1.smithi027.stdout:8/139: rename d1/d3/f26 to d1/d3/d1c/f34 0 2024-04-22T15:49:49.670 INFO:tasks.workunit.client.1.smithi027.stdout:9/156: write d0/dc/f29 [794867,109794] 0 2024-04-22T15:49:49.670 INFO:tasks.workunit.client.1.smithi027.stdout:8/140: write d1/f7 [1399259,84820] 0 2024-04-22T15:49:49.671 INFO:tasks.workunit.client.1.smithi027.stdout:9/157: readlink d0/l1d 0 2024-04-22T15:49:49.673 INFO:tasks.workunit.client.1.smithi027.stdout:2/164: creat dd/df/f2f x:0 0 0 2024-04-22T15:49:49.677 INFO:tasks.workunit.client.0.smithi016.stdout:4/972: rename d7/d4e/d2f/d33 to d7/d21/d143 0 2024-04-22T15:49:49.677 INFO:tasks.workunit.client.1.smithi027.stdout:1/124: dwrite d2/f1f [0,4194304] 0 2024-04-22T15:49:49.677 INFO:tasks.workunit.client.1.smithi027.stdout:6/186: symlink d5/dc/l3c 0 2024-04-22T15:49:49.683 INFO:tasks.workunit.client.0.smithi016.stdout:4/973: mkdir d7/d4e/d81/d144 0 2024-04-22T15:49:49.684 INFO:tasks.workunit.client.0.smithi016.stdout:4/974: chown d7/d21/d7c/d103/d10f/c118 54451286 1 2024-04-22T15:49:49.685 INFO:tasks.workunit.client.1.smithi027.stdout:1/125: creat d2/d1d/f24 x:0 0 0 2024-04-22T15:49:49.685 INFO:tasks.workunit.client.1.smithi027.stdout:1/126: stat d2/f15 0 2024-04-22T15:49:49.686 INFO:tasks.workunit.client.0.smithi016.stdout:4/975: dread - d7/d4e/d102/d106/d88/f134 zero size 2024-04-22T15:49:49.687 INFO:tasks.workunit.client.1.smithi027.stdout:6/187: mkdir d5/dc/d26/d3d 0 2024-04-22T15:49:49.689 INFO:tasks.workunit.client.0.smithi016.stdout:8/985: sync 2024-04-22T15:49:49.691 INFO:tasks.workunit.client.1.smithi027.stdout:8/141: dread d1/d3/f10 [0,4194304] 0 2024-04-22T15:49:49.692 INFO:tasks.workunit.client.1.smithi027.stdout:8/142: truncate d1/d3/f1a 1584213 0 2024-04-22T15:49:49.693 INFO:tasks.workunit.client.0.smithi016.stdout:4/976: mkdir d7/d4e/d102/d106/d88/d9d/dad/db9/d145 0 2024-04-22T15:49:49.693 INFO:tasks.workunit.client.0.smithi016.stdout:7/973: dread d3/dc/d1b/d31/d5f/f77 [0,4194304] 0 2024-04-22T15:49:49.693 INFO:tasks.workunit.client.1.smithi027.stdout:8/143: read - d1/d1e/f33 zero size 2024-04-22T15:49:49.694 INFO:tasks.workunit.client.1.smithi027.stdout:8/144: write d1/f18 [94901,73074] 0 2024-04-22T15:49:49.695 INFO:tasks.workunit.client.0.smithi016.stdout:4/977: write d7/d21/d45/d49/d67/fbd [1111832,104318] 0 2024-04-22T15:49:49.698 INFO:tasks.workunit.client.1.smithi027.stdout:9/158: rename d0/dc/de/f2d to d0/f36 0 2024-04-22T15:49:49.699 INFO:tasks.workunit.client.1.smithi027.stdout:6/188: dread d5/dc/d20/d2b/f12 [0,4194304] 0 2024-04-22T15:49:49.700 INFO:tasks.workunit.client.1.smithi027.stdout:9/159: write d0/d8/f18 [7847621,10561] 0 2024-04-22T15:49:49.703 INFO:tasks.workunit.client.0.smithi016.stdout:8/986: mknod d2/d82/d3e/c137 0 2024-04-22T15:49:49.706 INFO:tasks.workunit.client.0.smithi016.stdout:7/974: truncate d3/dc/d22/f39 6705239 0 2024-04-22T15:49:49.709 INFO:tasks.workunit.client.1.smithi027.stdout:5/174: write d0/df/d10/f14 [2631196,67114] 0 2024-04-22T15:49:49.709 INFO:tasks.workunit.client.0.smithi016.stdout:8/987: read d2/d48/dad/d8f/da1/fc8 [59005,67504] 0 2024-04-22T15:49:49.711 INFO:tasks.workunit.client.1.smithi027.stdout:2/165: rename dd/f20 to dd/df/d1b/d1f/f30 0 2024-04-22T15:49:49.712 INFO:tasks.workunit.client.1.smithi027.stdout:2/166: truncate dd/df/d1b/f25 469635 0 2024-04-22T15:49:49.713 INFO:tasks.workunit.client.1.smithi027.stdout:6/189: unlink d5/dc/d17/f1b 0 2024-04-22T15:49:49.714 INFO:tasks.workunit.client.1.smithi027.stdout:6/190: fdatasync d5/dc/d17/f39 0 2024-04-22T15:49:49.715 INFO:tasks.workunit.client.0.smithi016.stdout:4/978: creat d7/f146 x:0 0 0 2024-04-22T15:49:49.716 INFO:tasks.workunit.client.1.smithi027.stdout:1/127: rename d2/l22 to d2/l25 0 2024-04-22T15:49:49.719 INFO:tasks.workunit.client.0.smithi016.stdout:8/988: fsync d2/d48/dad/d6a/f9a 0 2024-04-22T15:49:49.722 INFO:tasks.workunit.client.0.smithi016.stdout:8/989: truncate d2/d82/d52/df0/d66/d7e/dc0/de2/f11a 217064 0 2024-04-22T15:49:49.722 INFO:tasks.workunit.client.1.smithi027.stdout:5/175: chown d0/df/d10/l28 1507 1 2024-04-22T15:49:49.726 INFO:tasks.workunit.client.1.smithi027.stdout:2/167: symlink dd/df/d1b/l31 0 2024-04-22T15:49:49.727 INFO:tasks.workunit.client.0.smithi016.stdout:4/979: dwrite d7/d24/d41/f57 [0,4194304] 0 2024-04-22T15:49:49.727 INFO:tasks.workunit.client.0.smithi016.stdout:4/980: chown d7/f19 11918 1 2024-04-22T15:49:49.728 INFO:tasks.workunit.client.0.smithi016.stdout:7/975: getdents d3/dc/d1b/d5a/d9/d14e 0 2024-04-22T15:49:49.730 INFO:tasks.workunit.client.1.smithi027.stdout:6/191: mkdir d5/dc/d20/d2b/d34/d3e 0 2024-04-22T15:49:49.731 INFO:tasks.workunit.client.1.smithi027.stdout:1/128: mknod d2/d1d/c26 0 2024-04-22T15:49:49.732 INFO:tasks.workunit.client.1.smithi027.stdout:8/145: link d1/f21 d1/d3/d1c/d28/f35 0 2024-04-22T15:49:49.733 INFO:tasks.workunit.client.1.smithi027.stdout:8/146: dread - d1/d3/d1c/f34 zero size 2024-04-22T15:49:49.734 INFO:tasks.workunit.client.0.smithi016.stdout:4/981: creat d7/d21/d45/dd6/f147 x:0 0 0 2024-04-22T15:49:49.736 INFO:tasks.workunit.client.0.smithi016.stdout:4/982: write d7/d21/d45/d49/fd8 [413645,119015] 0 2024-04-22T15:49:49.737 INFO:tasks.workunit.client.0.smithi016.stdout:4/983: chown d7/d4e/d102/d106/d88/d9d/dad 29 1 2024-04-22T15:49:49.738 INFO:tasks.workunit.client.0.smithi016.stdout:4/984: readlink d7/d4e/d102/l142 0 2024-04-22T15:49:49.738 INFO:tasks.workunit.client.0.smithi016.stdout:7/976: unlink d3/dc/d1b/feb 0 2024-04-22T15:49:49.740 INFO:tasks.workunit.client.1.smithi027.stdout:5/176: dwrite d0/df/de/d24/f2e [0,4194304] 0 2024-04-22T15:49:49.741 INFO:tasks.workunit.client.1.smithi027.stdout:2/168: creat dd/df/d1b/f32 x:0 0 0 2024-04-22T15:49:49.743 INFO:tasks.workunit.client.0.smithi016.stdout:8/990: rmdir d2/d82/d3e/d12d 0 2024-04-22T15:49:49.744 INFO:tasks.workunit.client.1.smithi027.stdout:9/160: creat d0/dc/f37 x:0 0 0 2024-04-22T15:49:49.745 INFO:tasks.workunit.client.0.smithi016.stdout:4/985: dwrite d7/d4e/f60 [4194304,4194304] 0 2024-04-22T15:49:49.747 INFO:tasks.workunit.client.1.smithi027.stdout:7/118: sync 2024-04-22T15:49:49.748 INFO:tasks.workunit.client.1.smithi027.stdout:0/153: sync 2024-04-22T15:49:49.748 INFO:tasks.workunit.client.1.smithi027.stdout:3/202: sync 2024-04-22T15:49:49.748 INFO:tasks.workunit.client.0.smithi016.stdout:8/991: write d2/d48/dad/d8f/da1/f119 [802862,40373] 0 2024-04-22T15:49:49.748 INFO:tasks.workunit.client.0.smithi016.stdout:4/986: chown d7/d4e/d39/df3/d116 8131 1 2024-04-22T15:49:49.749 INFO:tasks.workunit.client.1.smithi027.stdout:8/147: rmdir d1/d1e 39 2024-04-22T15:49:49.749 INFO:tasks.workunit.client.0.smithi016.stdout:4/987: dread - d7/d4e/d39/dc3/f10d zero size 2024-04-22T15:49:49.749 INFO:tasks.workunit.client.1.smithi027.stdout:7/119: read - d0/f15 zero size 2024-04-22T15:49:49.749 INFO:tasks.workunit.client.1.smithi027.stdout:5/177: dwrite d0/df/d10/f33 [0,4194304] 0 2024-04-22T15:49:49.750 INFO:tasks.workunit.client.0.smithi016.stdout:7/977: read d3/dc/d1b/d5a/d9/df/d94/f72 [7353678,47843] 0 2024-04-22T15:49:49.750 INFO:tasks.workunit.client.0.smithi016.stdout:7/978: chown d3/d11a 2785451 1 2024-04-22T15:49:49.755 INFO:tasks.workunit.client.0.smithi016.stdout:8/992: rename d2/d82/d52/df0/d66/d7e/d110 to d2/d30/d138 0 2024-04-22T15:49:49.756 INFO:tasks.workunit.client.0.smithi016.stdout:8/993: write d2/d48/dad/d8f/d96/de4/f12a [521865,27481] 0 2024-04-22T15:49:49.757 INFO:tasks.workunit.client.0.smithi016.stdout:4/988: creat d7/d4e/d102/d106/d88/f148 x:0 0 0 2024-04-22T15:49:49.758 INFO:tasks.workunit.client.0.smithi016.stdout:8/994: chown d2/d10/d43/da9/fac 11717 1 2024-04-22T15:49:49.759 INFO:tasks.workunit.client.1.smithi027.stdout:3/203: dwrite f1f [0,4194304] 0 2024-04-22T15:49:49.759 INFO:tasks.workunit.client.1.smithi027.stdout:3/204: chown c24 575724429 1 2024-04-22T15:49:49.760 INFO:tasks.workunit.client.0.smithi016.stdout:8/995: chown d2/d82/d52/df0/d66/ffb 0 1 2024-04-22T15:49:49.761 INFO:tasks.workunit.client.1.smithi027.stdout:3/205: write d28/f38 [895258,50715] 0 2024-04-22T15:49:49.762 INFO:tasks.workunit.client.0.smithi016.stdout:7/979: write d3/dc/d1b/d5a/d59/d119/f14d [906735,69515] 0 2024-04-22T15:49:49.763 INFO:tasks.workunit.client.0.smithi016.stdout:7/980: chown d3/dc/d1b/d31/d13f 4 1 2024-04-22T15:49:49.764 INFO:tasks.workunit.client.1.smithi027.stdout:8/148: mkdir d1/d3/d36 0 2024-04-22T15:49:49.769 INFO:tasks.workunit.client.1.smithi027.stdout:5/178: rmdir d0/df/de/d24/d1c 39 2024-04-22T15:49:49.771 INFO:tasks.workunit.client.1.smithi027.stdout:8/149: dwrite d1/f4 [0,4194304] 0 2024-04-22T15:49:49.771 INFO:tasks.workunit.client.0.smithi016.stdout:7/981: mknod d3/dc/d1b/d31/d5f/c162 0 2024-04-22T15:49:49.772 INFO:tasks.workunit.client.1.smithi027.stdout:5/179: read d0/df/de/d24/f16 [69936,48142] 0 2024-04-22T15:49:49.775 INFO:tasks.workunit.client.1.smithi027.stdout:3/206: mkdir d28/d40 0 2024-04-22T15:49:49.777 INFO:tasks.workunit.client.0.smithi016.stdout:7/982: truncate d3/dc/d1b/f96 204116 0 2024-04-22T15:49:49.783 INFO:tasks.workunit.client.0.smithi016.stdout:7/983: creat d3/dc/d1b/d5a/d9/df/d94/f163 x:0 0 0 2024-04-22T15:49:49.784 INFO:tasks.workunit.client.1.smithi027.stdout:2/169: creat dd/f33 x:0 0 0 2024-04-22T15:49:49.786 INFO:tasks.workunit.client.0.smithi016.stdout:7/984: read d3/de6/f7c [1018547,28963] 0 2024-04-22T15:49:49.787 INFO:tasks.workunit.client.0.smithi016.stdout:7/985: stat d3/l12 0 2024-04-22T15:49:49.788 INFO:tasks.workunit.client.1.smithi027.stdout:9/161: rename d0/dc/de/l33 to d0/dc/l38 0 2024-04-22T15:49:49.791 INFO:tasks.workunit.client.1.smithi027.stdout:1/129: dwrite d2/f15 [0,4194304] 0 2024-04-22T15:49:49.793 INFO:tasks.workunit.client.1.smithi027.stdout:8/150: dwrite d1/d1e/f33 [0,4194304] 0 2024-04-22T15:49:49.793 INFO:tasks.workunit.client.0.smithi016.stdout:4/989: truncate d7/d21/d45/dd6/f11c 302504 0 2024-04-22T15:49:49.797 INFO:tasks.workunit.client.0.smithi016.stdout:7/986: symlink d3/dc/l164 0 2024-04-22T15:49:49.801 INFO:tasks.workunit.client.1.smithi027.stdout:0/154: truncate d0/d28/f20 3430476 0 2024-04-22T15:49:49.805 INFO:tasks.workunit.client.1.smithi027.stdout:7/120: rename d0/f19 to d0/d16/f1e 0 2024-04-22T15:49:49.806 INFO:tasks.workunit.client.0.smithi016.stdout:4/990: fdatasync d7/d24/fed 0 2024-04-22T15:49:49.807 INFO:tasks.workunit.client.1.smithi027.stdout:5/180: creat d0/d1a/d22/d26/d32/f37 x:0 0 0 2024-04-22T15:49:49.810 INFO:tasks.workunit.client.0.smithi016.stdout:7/987: mkdir d3/dc/d9f/da1/dfd/d165 0 2024-04-22T15:49:49.811 INFO:tasks.workunit.client.0.smithi016.stdout:7/988: write d3/de6/f2c [536385,100448] 0 2024-04-22T15:49:49.813 INFO:tasks.workunit.client.0.smithi016.stdout:7/989: write d3/dc/d1b/d5a/d9/df/faa [2106985,97521] 0 2024-04-22T15:49:49.820 INFO:tasks.workunit.client.1.smithi027.stdout:2/170: symlink dd/l34 0 2024-04-22T15:49:49.821 INFO:tasks.workunit.client.1.smithi027.stdout:2/171: read dd/df/d1b/f2b [1601546,41063] 0 2024-04-22T15:49:49.822 INFO:tasks.workunit.client.1.smithi027.stdout:0/155: rmdir d0/d2a 39 2024-04-22T15:49:49.825 INFO:tasks.workunit.client.1.smithi027.stdout:7/121: mknod d0/d16/c1f 0 2024-04-22T15:49:49.826 INFO:tasks.workunit.client.0.smithi016.stdout:4/991: mkdir d7/d4e/d149 0 2024-04-22T15:49:49.826 INFO:tasks.workunit.client.1.smithi027.stdout:7/122: chown d0/d11/fe 1849641993 1 2024-04-22T15:49:49.828 INFO:tasks.workunit.client.1.smithi027.stdout:5/181: truncate d0/df/de/d24/d1c/f34 283963 0 2024-04-22T15:49:49.829 INFO:tasks.workunit.client.1.smithi027.stdout:1/130: fsync d2/f18 0 2024-04-22T15:49:49.831 INFO:tasks.workunit.client.1.smithi027.stdout:3/207: rmdir d28/d40 0 2024-04-22T15:49:49.832 INFO:tasks.workunit.client.0.smithi016.stdout:4/992: dwrite d7/d3c/fe6 [4194304,4194304] 0 2024-04-22T15:49:49.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:49 smithi027 ceph-mon[34922]: pgmap v37: 65 pgs: 65 active+clean; 2.3 GiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 79 MiB/s rd, 223 MiB/s wr, 520 op/s 2024-04-22T15:49:49.842 INFO:tasks.workunit.client.1.smithi027.stdout:5/182: dread d0/df/de/d24/f16 [0,4194304] 0 2024-04-22T15:49:49.845 INFO:tasks.workunit.client.1.smithi027.stdout:8/151: write d1/fb [1349933,121506] 0 2024-04-22T15:49:49.849 INFO:tasks.workunit.client.1.smithi027.stdout:8/152: dread d1/d3/f19 [0,4194304] 0 2024-04-22T15:49:49.860 INFO:tasks.workunit.client.1.smithi027.stdout:5/183: dread d0/f8 [0,4194304] 0 2024-04-22T15:49:49.862 INFO:tasks.workunit.client.1.smithi027.stdout:1/131: fdatasync d2/fa 0 2024-04-22T15:49:49.865 INFO:tasks.workunit.client.1.smithi027.stdout:5/184: dwrite d0/f7 [0,4194304] 0 2024-04-22T15:49:49.866 INFO:tasks.workunit.client.1.smithi027.stdout:2/172: creat dd/f35 x:0 0 0 2024-04-22T15:49:49.870 INFO:tasks.workunit.client.1.smithi027.stdout:0/156: truncate d0/f22 559949 0 2024-04-22T15:49:49.874 INFO:tasks.workunit.client.1.smithi027.stdout:2/173: dwrite dd/f1e [0,4194304] 0 2024-04-22T15:49:49.881 INFO:tasks.workunit.client.1.smithi027.stdout:5/185: write d0/f6 [4405939,44184] 0 2024-04-22T15:49:49.883 INFO:tasks.workunit.client.1.smithi027.stdout:8/153: rename d1/d3/f19 to d1/d1e/f37 0 2024-04-22T15:49:49.890 INFO:tasks.workunit.client.1.smithi027.stdout:7/123: getdents d0 0 2024-04-22T15:49:49.892 INFO:tasks.workunit.client.1.smithi027.stdout:2/174: rmdir dd/df/d1b 39 2024-04-22T15:49:49.898 INFO:tasks.workunit.client.1.smithi027.stdout:3/208: getdents d28 0 2024-04-22T15:49:49.903 INFO:tasks.workunit.client.1.smithi027.stdout:0/157: symlink d0/d2a/l30 0 2024-04-22T15:49:49.908 INFO:tasks.workunit.client.1.smithi027.stdout:1/132: creat d2/f27 x:0 0 0 2024-04-22T15:49:49.912 INFO:tasks.workunit.client.0.smithi016.stdout:8/996: sync 2024-04-22T15:49:49.913 INFO:tasks.workunit.client.0.smithi016.stdout:8/997: readlink d2/d82/d52/df0/d66/l1d 0 2024-04-22T15:49:49.914 INFO:tasks.workunit.client.1.smithi027.stdout:3/209: dwrite d28/f37 [0,4194304] 0 2024-04-22T15:49:49.916 INFO:tasks.workunit.client.1.smithi027.stdout:0/158: dread d0/f14 [0,4194304] 0 2024-04-22T15:49:49.926 INFO:tasks.workunit.client.1.smithi027.stdout:4/179: sync 2024-04-22T15:49:49.931 INFO:tasks.workunit.client.1.smithi027.stdout:3/210: creat d28/f41 x:0 0 0 2024-04-22T15:49:49.931 INFO:tasks.workunit.client.1.smithi027.stdout:4/180: dwrite d5/d7/f15 [0,4194304] 0 2024-04-22T15:49:49.935 INFO:tasks.workunit.client.1.smithi027.stdout:0/159: rename d0/d28/l1d to d0/d28/l31 0 2024-04-22T15:49:49.935 INFO:tasks.workunit.client.1.smithi027.stdout:0/160: fsync d0/f12 0 2024-04-22T15:49:49.939 INFO:tasks.workunit.client.1.smithi027.stdout:1/133: symlink d2/l28 0 2024-04-22T15:49:49.940 INFO:tasks.workunit.client.1.smithi027.stdout:2/175: write dd/df/f28 [999078,64826] 0 2024-04-22T15:49:49.941 INFO:tasks.workunit.client.1.smithi027.stdout:5/186: getdents d0/df/de 0 2024-04-22T15:49:49.941 INFO:tasks.workunit.client.1.smithi027.stdout:2/176: read - dd/f35 zero size 2024-04-22T15:49:49.943 INFO:tasks.workunit.client.1.smithi027.stdout:8/154: link d1/d3/d1c/l2d d1/d1e/l38 0 2024-04-22T15:49:49.945 INFO:tasks.workunit.client.1.smithi027.stdout:2/177: dread dd/f11 [4194304,4194304] 0 2024-04-22T15:49:49.948 INFO:tasks.workunit.client.1.smithi027.stdout:5/187: dwrite d0/df/d10/f25 [0,4194304] 0 2024-04-22T15:49:49.948 INFO:tasks.workunit.client.1.smithi027.stdout:7/124: getdents d0/d11 0 2024-04-22T15:49:49.951 INFO:tasks.workunit.client.1.smithi027.stdout:6/192: sync 2024-04-22T15:49:49.951 INFO:tasks.workunit.client.1.smithi027.stdout:2/178: dread dd/f1e [0,4194304] 0 2024-04-22T15:49:49.953 INFO:tasks.workunit.client.1.smithi027.stdout:3/211: rename f13 to d28/f42 0 2024-04-22T15:49:49.953 INFO:tasks.workunit.client.1.smithi027.stdout:6/193: write d5/dc/d26/f2d [1990524,63488] 0 2024-04-22T15:49:49.955 INFO:tasks.workunit.client.1.smithi027.stdout:4/181: mkdir d5/d42 0 2024-04-22T15:49:49.955 INFO:tasks.workunit.client.1.smithi027.stdout:6/194: fsync d5/dc/d20/d2b/f24 0 2024-04-22T15:49:49.958 INFO:tasks.workunit.client.1.smithi027.stdout:7/125: dwrite d0/fd [0,4194304] 0 2024-04-22T15:49:49.961 INFO:tasks.workunit.client.1.smithi027.stdout:7/126: write d0/f13 [6464006,124257] 0 2024-04-22T15:49:49.965 INFO:tasks.workunit.client.1.smithi027.stdout:4/182: dwrite d5/d11/f28 [0,4194304] 0 2024-04-22T15:49:49.970 INFO:tasks.workunit.client.1.smithi027.stdout:4/183: dread d5/f6 [0,4194304] 0 2024-04-22T15:49:49.971 INFO:tasks.workunit.client.0.smithi016.stdout:7/990: creat d3/dc/d1b/f166 x:0 0 0 2024-04-22T15:49:49.977 INFO:tasks.workunit.client.0.smithi016.stdout:4/993: creat d7/d4e/de1/f14a x:0 0 0 2024-04-22T15:49:49.977 INFO:tasks.workunit.client.0.smithi016.stdout:4/994: dread - d7/d22/f13a zero size 2024-04-22T15:49:49.977 INFO:tasks.workunit.client.1.smithi027.stdout:5/188: rmdir d0/df 39 2024-04-22T15:49:49.978 INFO:tasks.workunit.client.0.smithi016.stdout:4/995: truncate d7/d4e/d2f/d6f/f115 1010811 0 2024-04-22T15:49:49.979 INFO:tasks.workunit.client.0.smithi016.stdout:8/998: creat d2/d82/d56/d99/f139 x:0 0 0 2024-04-22T15:49:49.982 INFO:tasks.workunit.client.1.smithi027.stdout:5/189: dread d0/f15 [0,4194304] 0 2024-04-22T15:49:49.982 INFO:tasks.workunit.client.1.smithi027.stdout:3/212: creat d28/f43 x:0 0 0 2024-04-22T15:49:49.984 INFO:tasks.workunit.client.0.smithi016.stdout:4/996: dwrite d7/d21/d143/ffd [0,4194304] 0 2024-04-22T15:49:49.986 INFO:tasks.workunit.client.0.smithi016.stdout:4/997: write d7/d21/d45/d49/f11d [1033432,121944] 0 2024-04-22T15:49:49.989 INFO:tasks.workunit.client.1.smithi027.stdout:6/195: creat d5/dc/d20/f3f x:0 0 0 2024-04-22T15:49:49.991 INFO:tasks.workunit.client.1.smithi027.stdout:6/196: chown d5/dc/d20/d2b/f24 65526274 1 2024-04-22T15:49:49.992 INFO:tasks.workunit.client.0.smithi016.stdout:4/998: dwrite d7/d21/d45/d49/d67/fbd [0,4194304] 0 2024-04-22T15:49:49.996 INFO:tasks.workunit.client.1.smithi027.stdout:7/127: symlink d0/l20 0 2024-04-22T15:49:49.996 INFO:tasks.workunit.client.1.smithi027.stdout:7/128: stat d0/l20 0 2024-04-22T15:49:49.997 INFO:tasks.workunit.client.0.smithi016.stdout:8/999: getdents d2/d82/d52/df0/d6f 0 2024-04-22T15:49:49.999 INFO:tasks.workunit.client.1.smithi027.stdout:0/161: mknod d0/c32 0 2024-04-22T15:49:50.000 INFO:tasks.workunit.client.1.smithi027.stdout:1/134: mknod d2/c29 0 2024-04-22T15:49:50.001 INFO:tasks.workunit.client.0.smithi016.stdout:4/999: dwrite d7/d4e/d39/dc3/f10d [0,4194304] 0 2024-04-22T15:49:50.002 INFO:tasks.workunit.client.1.smithi027.stdout:1/135: read - d2/f27 zero size 2024-04-22T15:49:50.004 INFO:tasks.workunit.client.1.smithi027.stdout:4/184: rename d5/de/d2a/f3f to d5/de/f43 0 2024-04-22T15:49:50.005 INFO:tasks.workunit.client.1.smithi027.stdout:7/129: dwrite d0/fd [0,4194304] 0 2024-04-22T15:49:50.007 INFO:tasks.workunit.client.1.smithi027.stdout:9/162: sync 2024-04-22T15:49:50.008 INFO:tasks.workunit.client.1.smithi027.stdout:7/130: chown d0/fb 3231 1 2024-04-22T15:49:50.009 INFO:tasks.workunit.client.1.smithi027.stdout:2/179: unlink dd/df/d1b/f2b 0 2024-04-22T15:49:50.009 INFO:tasks.workunit.client.1.smithi027.stdout:8/155: dwrite d1/d1e/f37 [0,4194304] 0 2024-04-22T15:49:50.011 INFO:tasks.workunit.client.1.smithi027.stdout:2/180: fdatasync dd/df/d1b/f26 0 2024-04-22T15:49:50.015 INFO:tasks.workunit.client.1.smithi027.stdout:2/181: write dd/df/d1b/f25 [1367583,121988] 0 2024-04-22T15:49:50.015 INFO:tasks.workunit.client.1.smithi027.stdout:3/213: unlink d28/f42 0 2024-04-22T15:49:50.021 INFO:tasks.workunit.client.1.smithi027.stdout:0/162: mknod d0/d28/c33 0 2024-04-22T15:49:50.021 INFO:tasks.workunit.client.1.smithi027.stdout:1/136: readlink d2/ld 0 2024-04-22T15:49:50.022 INFO:tasks.workunit.client.1.smithi027.stdout:4/185: mknod d5/de/c44 0 2024-04-22T15:49:50.023 INFO:tasks.workunit.client.1.smithi027.stdout:7/131: dread d0/fd [0,4194304] 0 2024-04-22T15:49:50.025 INFO:tasks.workunit.client.1.smithi027.stdout:8/156: dwrite d1/d1e/f22 [0,4194304] 0 2024-04-22T15:49:50.025 INFO:tasks.workunit.client.1.smithi027.stdout:0/163: truncate d0/d28/f1f 932011 0 2024-04-22T15:49:50.026 INFO:tasks.workunit.client.0.smithi016.stdout:7/991: dwrite d3/dc/d1b/d5a/d4a/f4f [0,4194304] 0 2024-04-22T15:49:50.031 INFO:tasks.workunit.client.1.smithi027.stdout:1/137: dread d2/f18 [0,4194304] 0 2024-04-22T15:49:50.032 INFO:tasks.workunit.client.1.smithi027.stdout:9/163: rename d0/fb to d0/dc/d12/d30/f39 0 2024-04-22T15:49:50.032 INFO:tasks.workunit.client.1.smithi027.stdout:8/157: dread d1/d1e/f33 [0,4194304] 0 2024-04-22T15:49:50.032 INFO:tasks.workunit.client.1.smithi027.stdout:5/190: unlink d0/f15 0 2024-04-22T15:49:50.032 INFO:tasks.workunit.client.0.smithi016.stdout:7/992: dread - d3/dc/d1b/d31/d88/de1/ff5 zero size 2024-04-22T15:49:50.037 INFO:tasks.workunit.client.0.smithi016.stdout:7/993: rename d3/dc/d1b/d5a/d9/f25 to d3/dc/d9f/da1/f167 0 2024-04-22T15:49:50.042 INFO:tasks.workunit.client.0.smithi016.stdout:7/994: truncate d3/dc/d33/d84/de3/ff2 451788 0 2024-04-22T15:49:50.052 INFO:tasks.workunit.client.0.smithi016.stdout:7/995: getdents d3/dc/d1b/d5a/d59/d119 0 2024-04-22T15:49:50.053 INFO:tasks.workunit.client.0.smithi016.stdout:7/996: write d3/dc/d1b/d5a/d59/d64/dc9/f150 [398950,98724] 0 2024-04-22T15:49:50.061 INFO:tasks.workunit.client.0.smithi016.stdout:7/997: dwrite d3/dc/d1b/d5a/d9/df/fd8 [4194304,4194304] 0 2024-04-22T15:49:50.071 INFO:tasks.workunit.client.0.smithi016.stdout:7/998: dread d3/dc/d1b/d5a/d9/df/d94/f105 [0,4194304] 0 2024-04-22T15:49:50.072 INFO:tasks.workunit.client.0.smithi016.stdout:7/999: read - d3/dc/d1b/f166 zero size 2024-04-22T15:49:50.079 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm -rf -- ./tmp.8wrsDzmH6d 2024-04-22T15:49:50.106 INFO:tasks.workunit.client.1.smithi027.stdout:2/182: creat dd/df/d1b/d1f/f36 x:0 0 0 2024-04-22T15:49:50.106 INFO:tasks.workunit.client.1.smithi027.stdout:3/214: fdatasync d28/f3c 0 2024-04-22T15:49:50.107 INFO:tasks.workunit.client.1.smithi027.stdout:7/132: mknod d0/d16/c21 0 2024-04-22T15:49:50.108 INFO:tasks.workunit.client.1.smithi027.stdout:2/183: chown dd/df/d1b/f2e 124366 1 2024-04-22T15:49:50.108 INFO:tasks.workunit.client.1.smithi027.stdout:6/197: write d5/db/d2e/ff [2449181,110428] 0 2024-04-22T15:49:50.109 INFO:tasks.workunit.client.1.smithi027.stdout:1/138: symlink d2/d1d/l2a 0 2024-04-22T15:49:50.110 INFO:tasks.workunit.client.1.smithi027.stdout:9/164: creat d0/dc/de/f3a x:0 0 0 2024-04-22T15:49:50.112 INFO:tasks.workunit.client.1.smithi027.stdout:3/215: creat d28/f44 x:0 0 0 2024-04-22T15:49:50.112 INFO:tasks.workunit.client.1.smithi027.stdout:0/164: mkdir d0/d34 0 2024-04-22T15:49:50.115 INFO:tasks.workunit.client.1.smithi027.stdout:8/158: unlink d1/d3/ce 0 2024-04-22T15:49:50.115 INFO:tasks.workunit.client.1.smithi027.stdout:9/165: dread d0/dc/f25 [4194304,4194304] 0 2024-04-22T15:49:50.119 INFO:tasks.workunit.client.1.smithi027.stdout:3/216: dwrite d28/f44 [0,4194304] 0 2024-04-22T15:49:50.120 INFO:tasks.workunit.client.1.smithi027.stdout:7/133: link d0/f13 d0/f22 0 2024-04-22T15:49:50.122 INFO:tasks.workunit.client.1.smithi027.stdout:2/184: mknod dd/c37 0 2024-04-22T15:49:50.127 INFO:tasks.workunit.client.1.smithi027.stdout:0/165: creat d0/d2a/f35 x:0 0 0 2024-04-22T15:49:50.128 INFO:tasks.workunit.client.1.smithi027.stdout:6/198: mkdir d5/d30/d40 0 2024-04-22T15:49:50.129 INFO:tasks.workunit.client.1.smithi027.stdout:8/159: creat d1/d3/d1c/d28/f39 x:0 0 0 2024-04-22T15:49:50.131 INFO:tasks.workunit.client.1.smithi027.stdout:9/166: mknod d0/dc/d12/c3b 0 2024-04-22T15:49:50.133 INFO:tasks.workunit.client.1.smithi027.stdout:5/191: link d0/df/d10/f14 d0/df/de/f38 0 2024-04-22T15:49:50.135 INFO:tasks.workunit.client.1.smithi027.stdout:3/217: rename d28/f41 to d28/f45 0 2024-04-22T15:49:50.135 INFO:tasks.workunit.client.1.smithi027.stdout:7/134: rmdir d0/d11 39 2024-04-22T15:49:50.140 INFO:tasks.workunit.client.1.smithi027.stdout:6/199: mknod d5/dc/d20/d2b/d34/c41 0 2024-04-22T15:49:50.145 INFO:tasks.workunit.client.1.smithi027.stdout:9/167: rename d0/l2 to d0/dc/df/l3c 0 2024-04-22T15:49:50.147 INFO:tasks.workunit.client.1.smithi027.stdout:2/185: dread dd/df/f12 [4194304,4194304] 0 2024-04-22T15:49:50.147 INFO:tasks.workunit.client.1.smithi027.stdout:5/192: symlink d0/df/d10/l39 0 2024-04-22T15:49:50.148 INFO:tasks.workunit.client.1.smithi027.stdout:5/193: write d0/f6 [2885005,44817] 0 2024-04-22T15:49:50.149 INFO:tasks.workunit.client.1.smithi027.stdout:3/218: mknod d28/c46 0 2024-04-22T15:49:50.151 INFO:tasks.workunit.client.1.smithi027.stdout:7/135: readlink d0/d11/l1d 0 2024-04-22T15:49:50.152 INFO:tasks.workunit.client.1.smithi027.stdout:0/166: mknod d0/d34/c36 0 2024-04-22T15:49:50.154 INFO:tasks.workunit.client.1.smithi027.stdout:8/160: symlink d1/d3/d36/l3a 0 2024-04-22T15:49:50.158 INFO:tasks.workunit.client.1.smithi027.stdout:2/186: write dd/f1e [752039,56168] 0 2024-04-22T15:49:50.159 INFO:tasks.workunit.client.1.smithi027.stdout:3/219: symlink d28/l47 0 2024-04-22T15:49:50.162 INFO:tasks.workunit.client.1.smithi027.stdout:7/136: fdatasync d0/fd 0 2024-04-22T15:49:50.163 INFO:tasks.workunit.client.1.smithi027.stdout:0/167: stat d0/d28/f20 0 2024-04-22T15:49:50.163 INFO:tasks.workunit.client.1.smithi027.stdout:6/200: mknod d5/dc/c42 0 2024-04-22T15:49:50.164 INFO:tasks.workunit.client.1.smithi027.stdout:1/139: dwrite d2/f8 [0,4194304] 0 2024-04-22T15:49:50.166 INFO:tasks.workunit.client.1.smithi027.stdout:0/168: stat d0/d28/l24 0 2024-04-22T15:49:50.166 INFO:tasks.workunit.client.1.smithi027.stdout:1/140: write d2/f1f [2209172,125489] 0 2024-04-22T15:49:50.168 INFO:tasks.workunit.client.1.smithi027.stdout:5/194: link d0/d1a/d22/d26/f2a d0/d1a/f3a 0 2024-04-22T15:49:50.169 INFO:tasks.workunit.client.1.smithi027.stdout:2/187: rmdir dd/df/d1b 39 2024-04-22T15:49:50.171 INFO:tasks.workunit.client.1.smithi027.stdout:8/161: dread d1/d3/fa [0,4194304] 0 2024-04-22T15:49:50.173 INFO:tasks.workunit.client.1.smithi027.stdout:3/220: creat d28/f48 x:0 0 0 2024-04-22T15:49:50.174 INFO:tasks.workunit.client.1.smithi027.stdout:3/221: chown l6 12620 1 2024-04-22T15:49:50.174 INFO:tasks.workunit.client.1.smithi027.stdout:8/162: write d1/d1e/f22 [899020,76788] 0 2024-04-22T15:49:50.177 INFO:tasks.workunit.client.1.smithi027.stdout:5/195: dwrite d0/f6 [0,4194304] 0 2024-04-22T15:49:50.178 INFO:tasks.workunit.client.1.smithi027.stdout:7/137: creat d0/d16/f23 x:0 0 0 2024-04-22T15:49:50.186 INFO:tasks.workunit.client.1.smithi027.stdout:3/222: dwrite d28/f3c [4194304,4194304] 0 2024-04-22T15:49:50.187 INFO:tasks.workunit.client.1.smithi027.stdout:6/201: mknod d5/dc/d20/c43 0 2024-04-22T15:49:50.188 INFO:tasks.workunit.client.1.smithi027.stdout:3/223: dread - d28/f3f zero size 2024-04-22T15:49:50.189 INFO:tasks.workunit.client.1.smithi027.stdout:9/168: dwrite d0/dc/f25 [0,4194304] 0 2024-04-22T15:49:50.191 INFO:tasks.workunit.client.1.smithi027.stdout:4/186: sync 2024-04-22T15:49:50.194 INFO:tasks.workunit.client.1.smithi027.stdout:0/169: unlink d0/d28/f20 0 2024-04-22T15:49:50.201 INFO:tasks.workunit.client.1.smithi027.stdout:7/138: symlink d0/d16/l24 0 2024-04-22T15:49:50.203 INFO:tasks.workunit.client.1.smithi027.stdout:7/139: chown d0/f13 40692 1 2024-04-22T15:49:50.203 INFO:tasks.workunit.client.1.smithi027.stdout:5/196: rename d0/df/d10/f25 to d0/d1a/d22/d26/f3b 0 2024-04-22T15:49:50.205 INFO:tasks.workunit.client.1.smithi027.stdout:8/163: dwrite d1/d3/f10 [0,4194304] 0 2024-04-22T15:49:50.208 INFO:tasks.workunit.client.1.smithi027.stdout:1/141: symlink d2/l2b 0 2024-04-22T15:49:50.208 INFO:tasks.workunit.client.1.smithi027.stdout:3/224: creat d28/f49 x:0 0 0 2024-04-22T15:49:50.209 INFO:tasks.workunit.client.1.smithi027.stdout:0/170: write d0/f14 [5573641,32073] 0 2024-04-22T15:49:50.210 INFO:tasks.workunit.client.1.smithi027.stdout:2/188: mkdir dd/df/d1b/d38 0 2024-04-22T15:49:50.210 INFO:tasks.workunit.client.1.smithi027.stdout:2/189: readlink la 0 2024-04-22T15:49:50.213 INFO:tasks.workunit.client.1.smithi027.stdout:7/140: creat d0/d16/f25 x:0 0 0 2024-04-22T15:49:50.216 INFO:tasks.workunit.client.1.smithi027.stdout:2/190: dread dd/f1c [0,4194304] 0 2024-04-22T15:49:50.217 INFO:tasks.workunit.client.1.smithi027.stdout:6/202: mknod d5/c44 0 2024-04-22T15:49:50.219 INFO:tasks.workunit.client.1.smithi027.stdout:1/142: symlink d2/d1d/l2c 0 2024-04-22T15:49:50.226 INFO:tasks.workunit.client.1.smithi027.stdout:0/171: mkdir d0/d28/d37 0 2024-04-22T15:49:50.226 INFO:tasks.workunit.client.1.smithi027.stdout:4/187: rename d5/d11/d1d/d24/d2d to d5/d45 0 2024-04-22T15:49:50.237 INFO:tasks.workunit.client.1.smithi027.stdout:8/164: mknod d1/c3b 0 2024-04-22T15:49:50.237 INFO:tasks.workunit.client.1.smithi027.stdout:8/165: chown d1/d3/d1c/d28 1775 1 2024-04-22T15:49:50.245 INFO:tasks.workunit.client.1.smithi027.stdout:0/172: mkdir d0/d2a/d38 0 2024-04-22T15:49:50.250 INFO:tasks.workunit.client.1.smithi027.stdout:7/141: mkdir d0/d26 0 2024-04-22T15:49:50.251 INFO:tasks.workunit.client.1.smithi027.stdout:7/142: write d0/fd [3393434,17112] 0 2024-04-22T15:49:50.252 INFO:tasks.workunit.client.1.smithi027.stdout:4/188: mknod d5/d11/d1d/c46 0 2024-04-22T15:49:50.257 INFO:tasks.workunit.client.1.smithi027.stdout:5/197: creat d0/f3c x:0 0 0 2024-04-22T15:49:50.258 INFO:tasks.workunit.client.1.smithi027.stdout:2/191: mknod dd/d1d/c39 0 2024-04-22T15:49:50.261 INFO:tasks.workunit.client.1.smithi027.stdout:8/166: unlink d1/d3/d1c/d28/f39 0 2024-04-22T15:49:50.263 INFO:tasks.workunit.client.1.smithi027.stdout:6/203: link d5/dc/dd/f16 d5/dc/d20/d2b/d34/d3e/f45 0 2024-04-22T15:49:50.265 INFO:tasks.workunit.client.1.smithi027.stdout:9/169: getdents d0/dc/d12 0 2024-04-22T15:49:50.268 INFO:tasks.workunit.client.1.smithi027.stdout:4/189: mkdir d5/d45/d47 0 2024-04-22T15:49:50.268 INFO:tasks.workunit.client.1.smithi027.stdout:5/198: mkdir d0/df/d3d 0 2024-04-22T15:49:50.271 INFO:tasks.workunit.client.1.smithi027.stdout:5/199: readlink d0/df/d10/l28 0 2024-04-22T15:49:50.272 INFO:tasks.workunit.client.1.smithi027.stdout:8/167: creat d1/d3/f3c x:0 0 0 2024-04-22T15:49:50.272 INFO:tasks.workunit.client.1.smithi027.stdout:6/204: dwrite d5/dc/d20/d2b/f24 [0,4194304] 0 2024-04-22T15:49:50.274 INFO:tasks.workunit.client.1.smithi027.stdout:7/143: truncate d0/f22 5667762 0 2024-04-22T15:49:50.275 INFO:tasks.workunit.client.1.smithi027.stdout:8/168: chown d1/d1e/l29 21330 1 2024-04-22T15:49:50.276 INFO:tasks.workunit.client.1.smithi027.stdout:4/190: creat d5/f48 x:0 0 0 2024-04-22T15:49:50.277 INFO:tasks.workunit.client.1.smithi027.stdout:0/173: dwrite d0/d28/f1e [0,4194304] 0 2024-04-22T15:49:50.278 INFO:tasks.workunit.client.1.smithi027.stdout:4/191: chown d5/d11/f28 22 1 2024-04-22T15:49:50.279 INFO:tasks.workunit.client.1.smithi027.stdout:4/192: dread - d5/de/d2f/f3c zero size 2024-04-22T15:49:50.281 INFO:tasks.workunit.client.1.smithi027.stdout:2/192: symlink dd/l3a 0 2024-04-22T15:49:50.284 INFO:tasks.workunit.client.1.smithi027.stdout:1/143: getdents d2 0 2024-04-22T15:49:50.289 INFO:tasks.workunit.client.1.smithi027.stdout:5/200: mkdir d0/df/de/d24/d1c/d3e 0 2024-04-22T15:49:50.290 INFO:tasks.workunit.client.1.smithi027.stdout:2/193: dwrite dd/df/f28 [0,4194304] 0 2024-04-22T15:49:50.291 INFO:tasks.workunit.client.1.smithi027.stdout:2/194: chown dd/f2d 56 1 2024-04-22T15:49:50.294 INFO:tasks.workunit.client.1.smithi027.stdout:2/195: read dd/f11 [3574668,74687] 0 2024-04-22T15:49:50.294 INFO:tasks.workunit.client.1.smithi027.stdout:6/205: mknod d5/dc/d20/c46 0 2024-04-22T15:49:50.295 INFO:tasks.workunit.client.1.smithi027.stdout:2/196: readlink dd/le 0 2024-04-22T15:49:50.297 INFO:tasks.workunit.client.1.smithi027.stdout:0/174: fdatasync d0/f18 0 2024-04-22T15:49:50.299 INFO:tasks.workunit.client.1.smithi027.stdout:1/144: rename d2/l21 to d2/d1d/l2d 0 2024-04-22T15:49:50.300 INFO:tasks.workunit.client.1.smithi027.stdout:9/170: creat d0/dc/f3d x:0 0 0 2024-04-22T15:49:50.301 INFO:tasks.workunit.client.1.smithi027.stdout:5/201: chown d0/f3 149111 1 2024-04-22T15:49:50.301 INFO:tasks.workunit.client.1.smithi027.stdout:6/206: creat d5/d30/f47 x:0 0 0 2024-04-22T15:49:50.303 INFO:tasks.workunit.client.1.smithi027.stdout:6/207: read d5/f29 [287432,8521] 0 2024-04-22T15:49:50.304 INFO:tasks.workunit.client.1.smithi027.stdout:8/169: creat d1/d3/d1c/d32/f3d x:0 0 0 2024-04-22T15:49:50.306 INFO:tasks.workunit.client.1.smithi027.stdout:0/175: unlink d0/d28/f1e 0 2024-04-22T15:49:50.306 INFO:tasks.workunit.client.1.smithi027.stdout:4/193: dwrite d5/de/d2a/f3a [0,4194304] 0 2024-04-22T15:49:50.306 INFO:tasks.workunit.client.1.smithi027.stdout:6/208: chown d5/dc/d20/f3f 125659 1 2024-04-22T15:49:50.309 INFO:tasks.workunit.client.1.smithi027.stdout:0/176: write d0/f12 [4112569,30853] 0 2024-04-22T15:49:50.311 INFO:tasks.workunit.client.1.smithi027.stdout:2/197: creat dd/df/d1b/d38/f3b x:0 0 0 2024-04-22T15:49:50.311 INFO:tasks.workunit.client.1.smithi027.stdout:0/177: chown d0/f12 356895030 1 2024-04-22T15:49:50.311 INFO:tasks.workunit.client.1.smithi027.stdout:2/198: readlink la 0 2024-04-22T15:49:50.312 INFO:tasks.workunit.client.1.smithi027.stdout:9/171: dread d0/dc/d12/d30/f32 [0,4194304] 0 2024-04-22T15:49:50.317 INFO:tasks.workunit.client.1.smithi027.stdout:6/209: mkdir d5/dc/d17/d48 0 2024-04-22T15:49:50.317 INFO:tasks.workunit.client.1.smithi027.stdout:8/170: symlink d1/d3/d1c/l3e 0 2024-04-22T15:49:50.319 INFO:tasks.workunit.client.1.smithi027.stdout:7/144: write d0/f14 [1105655,91757] 0 2024-04-22T15:49:50.329 INFO:tasks.workunit.client.1.smithi027.stdout:6/210: rename d5/dc/d20/d2b/d34/d3e to d5/dc/d26/d49 0 2024-04-22T15:49:50.332 INFO:tasks.workunit.client.1.smithi027.stdout:4/194: link d5/d11/f28 d5/de/d2a/f49 0 2024-04-22T15:49:50.333 INFO:tasks.workunit.client.1.smithi027.stdout:4/195: fsync d5/d11/d1d/f27 0 2024-04-22T15:49:50.334 INFO:tasks.workunit.client.1.smithi027.stdout:0/178: symlink d0/d2a/d38/l39 0 2024-04-22T15:49:50.337 INFO:tasks.workunit.client.1.smithi027.stdout:1/145: dread - d2/d1d/f24 zero size 2024-04-22T15:49:50.338 INFO:tasks.workunit.client.1.smithi027.stdout:9/172: symlink d0/dc/l3e 0 2024-04-22T15:49:50.338 INFO:tasks.workunit.client.1.smithi027.stdout:1/146: dread - d2/d1d/f24 zero size 2024-04-22T15:49:50.338 INFO:tasks.workunit.client.1.smithi027.stdout:5/202: link d0/d1a/d22/l27 d0/df/de/l3f 0 2024-04-22T15:49:50.340 INFO:tasks.workunit.client.1.smithi027.stdout:6/211: symlink d5/dc/d26/l4a 0 2024-04-22T15:49:50.341 INFO:tasks.workunit.client.1.smithi027.stdout:6/212: chown d5/dc/d20/d2b/f24 32843548 1 2024-04-22T15:49:50.342 INFO:tasks.workunit.client.1.smithi027.stdout:6/213: write d5/db/d2e/ff [726322,22538] 0 2024-04-22T15:49:50.343 INFO:tasks.workunit.client.1.smithi027.stdout:2/199: link dd/f19 dd/df/f3c 0 2024-04-22T15:49:50.344 INFO:tasks.workunit.client.1.smithi027.stdout:2/200: chown f7 0 1 2024-04-22T15:49:50.345 INFO:tasks.workunit.client.1.smithi027.stdout:2/201: chown dd/df/d1b/f25 932748 1 2024-04-22T15:49:50.346 INFO:tasks.workunit.client.1.smithi027.stdout:7/145: creat d0/d26/f27 x:0 0 0 2024-04-22T15:49:50.346 INFO:tasks.workunit.client.1.smithi027.stdout:7/146: chown d0/l10 2362 1 2024-04-22T15:49:50.347 INFO:tasks.workunit.client.1.smithi027.stdout:0/179: creat d0/d34/f3a x:0 0 0 2024-04-22T15:49:50.348 INFO:tasks.workunit.client.1.smithi027.stdout:9/173: creat d0/d8/f3f x:0 0 0 2024-04-22T15:49:50.351 INFO:tasks.workunit.client.1.smithi027.stdout:5/203: fsync d0/df/de/d24/f16 0 2024-04-22T15:49:50.358 INFO:tasks.workunit.client.1.smithi027.stdout:4/196: creat d5/d45/d47/f4a x:0 0 0 2024-04-22T15:49:50.358 INFO:tasks.workunit.client.1.smithi027.stdout:3/225: sync 2024-04-22T15:49:50.358 INFO:tasks.workunit.client.1.smithi027.stdout:4/197: chown d5/de/c1a 7800833 1 2024-04-22T15:49:50.359 INFO:tasks.workunit.client.1.smithi027.stdout:3/226: dread - d28/f43 zero size 2024-04-22T15:49:50.360 INFO:tasks.workunit.client.1.smithi027.stdout:4/198: write d5/d11/d1d/f27 [5130211,127711] 0 2024-04-22T15:49:50.364 INFO:tasks.workunit.client.1.smithi027.stdout:1/147: unlink d2/l1e 0 2024-04-22T15:49:50.364 INFO:tasks.workunit.client.1.smithi027.stdout:3/227: dread d28/f38 [0,4194304] 0 2024-04-22T15:49:50.365 INFO:tasks.workunit.client.1.smithi027.stdout:9/174: creat d0/dc/de/f40 x:0 0 0 2024-04-22T15:49:50.368 INFO:tasks.workunit.client.1.smithi027.stdout:5/204: mkdir d0/df/de/d40 0 2024-04-22T15:49:50.371 INFO:tasks.workunit.client.1.smithi027.stdout:3/228: dwrite f5 [0,4194304] 0 2024-04-22T15:49:50.373 INFO:tasks.workunit.client.1.smithi027.stdout:8/171: rename d1/d1e/f37 to d1/f3f 0 2024-04-22T15:49:50.375 INFO:tasks.workunit.client.1.smithi027.stdout:2/202: symlink dd/df/l3d 0 2024-04-22T15:49:50.376 INFO:tasks.workunit.client.1.smithi027.stdout:2/203: fdatasync dd/df/d1b/f32 0 2024-04-22T15:49:50.379 INFO:tasks.workunit.client.1.smithi027.stdout:0/180: creat d0/d28/d37/f3b x:0 0 0 2024-04-22T15:49:50.379 INFO:tasks.workunit.client.1.smithi027.stdout:4/199: symlink d5/d11/l4b 0 2024-04-22T15:49:50.382 INFO:tasks.workunit.client.1.smithi027.stdout:0/181: chown d0/d28/f1f 55 1 2024-04-22T15:49:50.384 INFO:tasks.workunit.client.1.smithi027.stdout:6/214: dwrite d5/dc/d20/d2b/f12 [4194304,4194304] 0 2024-04-22T15:49:50.387 INFO:tasks.workunit.client.1.smithi027.stdout:3/229: symlink d28/l4a 0 2024-04-22T15:49:50.388 INFO:tasks.workunit.client.1.smithi027.stdout:8/172: chown d1/c2a 166 1 2024-04-22T15:49:50.394 INFO:tasks.workunit.client.1.smithi027.stdout:7/147: rename d0/f3 to d0/f28 0 2024-04-22T15:49:50.396 INFO:tasks.workunit.client.1.smithi027.stdout:2/204: creat dd/df/d1b/d1f/f3e x:0 0 0 2024-04-22T15:49:50.399 INFO:tasks.workunit.client.1.smithi027.stdout:4/200: rename f2 to d5/d7/f4c 0 2024-04-22T15:49:50.401 INFO:tasks.workunit.client.1.smithi027.stdout:4/201: read - d5/d11/d1d/d24/f40 zero size 2024-04-22T15:49:50.403 INFO:tasks.workunit.client.1.smithi027.stdout:1/148: symlink d2/l2e 0 2024-04-22T15:49:50.404 INFO:tasks.workunit.client.1.smithi027.stdout:2/205: dwrite dd/df/d1b/f25 [0,4194304] 0 2024-04-22T15:49:50.411 INFO:tasks.workunit.client.1.smithi027.stdout:6/215: creat d5/dc/d17/f4b x:0 0 0 2024-04-22T15:49:50.413 INFO:tasks.workunit.client.1.smithi027.stdout:5/205: dwrite d0/df/de/f38 [0,4194304] 0 2024-04-22T15:49:50.414 INFO:tasks.workunit.client.1.smithi027.stdout:3/230: mkdir d28/d4b 0 2024-04-22T15:49:50.414 INFO:tasks.workunit.client.1.smithi027.stdout:5/206: dread - d0/f3c zero size 2024-04-22T15:49:50.415 INFO:tasks.workunit.client.1.smithi027.stdout:3/231: chown l2 12537 1 2024-04-22T15:49:50.417 INFO:tasks.workunit.client.1.smithi027.stdout:3/232: write d28/f34 [343316,17471] 0 2024-04-22T15:49:50.420 INFO:tasks.workunit.client.1.smithi027.stdout:9/175: truncate d0/dc/f22 298787 0 2024-04-22T15:49:50.425 INFO:tasks.workunit.client.1.smithi027.stdout:4/202: mknod d5/d11/d1d/c4d 0 2024-04-22T15:49:50.426 INFO:tasks.workunit.client.1.smithi027.stdout:4/203: write d5/d7/f15 [721954,19839] 0 2024-04-22T15:49:50.429 INFO:tasks.workunit.client.1.smithi027.stdout:0/182: fdatasync d0/f22 0 2024-04-22T15:49:50.431 INFO:tasks.workunit.client.1.smithi027.stdout:6/216: unlink d5/dc/l3c 0 2024-04-22T15:49:50.433 INFO:tasks.workunit.client.1.smithi027.stdout:8/173: dwrite d1/f4 [4194304,4194304] 0 2024-04-22T15:49:50.434 INFO:tasks.workunit.client.1.smithi027.stdout:8/174: dread - d1/d3/f3c zero size 2024-04-22T15:49:50.436 INFO:tasks.workunit.client.1.smithi027.stdout:9/176: creat d0/dc/de/f41 x:0 0 0 2024-04-22T15:49:50.442 INFO:tasks.workunit.client.1.smithi027.stdout:8/175: dwrite d1/d1e/f22 [0,4194304] 0 2024-04-22T15:49:50.445 INFO:tasks.workunit.client.1.smithi027.stdout:1/149: creat d2/d1d/f2f x:0 0 0 2024-04-22T15:49:50.446 INFO:tasks.workunit.client.1.smithi027.stdout:4/204: readlink d5/de/d2f/l32 0 2024-04-22T15:49:50.450 INFO:tasks.workunit.client.1.smithi027.stdout:2/206: rename dd/df/l15 to dd/df/d1b/l3f 0 2024-04-22T15:49:50.452 INFO:tasks.workunit.client.1.smithi027.stdout:0/183: chown d0/l3 4265010 1 2024-04-22T15:49:50.454 INFO:tasks.workunit.client.1.smithi027.stdout:5/207: symlink d0/df/de/d24/d1c/d3e/l41 0 2024-04-22T15:49:50.456 INFO:tasks.workunit.client.1.smithi027.stdout:3/233: creat d28/d4b/f4c x:0 0 0 2024-04-22T15:49:50.458 INFO:tasks.workunit.client.1.smithi027.stdout:3/234: write d28/f3c [6325461,103514] 0 2024-04-22T15:49:50.459 INFO:tasks.workunit.client.1.smithi027.stdout:0/184: dwrite d0/f18 [0,4194304] 0 2024-04-22T15:49:50.463 INFO:tasks.workunit.client.1.smithi027.stdout:8/176: creat d1/d3/f40 x:0 0 0 2024-04-22T15:49:50.464 INFO:tasks.workunit.client.1.smithi027.stdout:8/177: read - d1/d3/d1c/f34 zero size 2024-04-22T15:49:50.465 INFO:tasks.workunit.client.1.smithi027.stdout:8/178: write d1/f18 [1962864,47473] 0 2024-04-22T15:49:50.467 INFO:tasks.workunit.client.1.smithi027.stdout:4/205: symlink d5/d11/l4e 0 2024-04-22T15:49:50.472 INFO:tasks.workunit.client.1.smithi027.stdout:6/217: symlink d5/d30/d40/l4c 0 2024-04-22T15:49:50.473 INFO:tasks.workunit.client.1.smithi027.stdout:5/208: fsync d0/f8 0 2024-04-22T15:49:50.475 INFO:tasks.workunit.client.1.smithi027.stdout:5/209: fdatasync d0/df/de/d24/f2e 0 2024-04-22T15:49:50.477 INFO:tasks.workunit.client.1.smithi027.stdout:9/177: unlink d0/d8/c2a 0 2024-04-22T15:49:50.480 INFO:tasks.workunit.client.1.smithi027.stdout:0/185: creat d0/d2a/f3c x:0 0 0 2024-04-22T15:49:50.482 INFO:tasks.workunit.client.1.smithi027.stdout:8/179: mkdir d1/d3/d1c/d32/d41 0 2024-04-22T15:49:50.483 INFO:tasks.workunit.client.1.smithi027.stdout:1/150: mkdir d2/d30 0 2024-04-22T15:49:50.484 INFO:tasks.workunit.client.1.smithi027.stdout:4/206: mknod d5/de/c4f 0 2024-04-22T15:49:50.486 INFO:tasks.workunit.client.1.smithi027.stdout:4/207: write d5/d11/d1d/d24/f40 [810338,8777] 0 2024-04-22T15:49:50.487 INFO:tasks.workunit.client.1.smithi027.stdout:4/208: fdatasync d5/de/d2f/f3c 0 2024-04-22T15:49:50.488 INFO:tasks.workunit.client.1.smithi027.stdout:6/218: creat d5/db/f4d x:0 0 0 2024-04-22T15:49:50.488 INFO:tasks.workunit.client.1.smithi027.stdout:4/209: fsync d5/f23 0 2024-04-22T15:49:50.489 INFO:tasks.workunit.client.1.smithi027.stdout:4/210: chown d5/de/d2f/l32 467 1 2024-04-22T15:49:50.490 INFO:tasks.workunit.client.1.smithi027.stdout:6/219: write d5/dc/d20/d2b/f12 [11571285,47655] 0 2024-04-22T15:49:50.490 INFO:tasks.workunit.client.1.smithi027.stdout:5/210: symlink d0/df/d10/l42 0 2024-04-22T15:49:50.494 INFO:tasks.workunit.client.1.smithi027.stdout:7/148: link d0/f13 d0/f29 0 2024-04-22T15:49:50.495 INFO:tasks.workunit.client.1.smithi027.stdout:0/186: rmdir d0/d2a/d38 39 2024-04-22T15:49:50.496 INFO:tasks.workunit.client.1.smithi027.stdout:4/211: dread d5/f26 [0,4194304] 0 2024-04-22T15:49:50.497 INFO:tasks.workunit.client.1.smithi027.stdout:8/180: mkdir d1/d3/d1c/d28/d42 0 2024-04-22T15:49:50.500 INFO:tasks.workunit.client.1.smithi027.stdout:9/178: dwrite d0/dc/d12/d30/f39 [0,4194304] 0 2024-04-22T15:49:50.507 INFO:tasks.workunit.client.1.smithi027.stdout:7/149: write d0/d16/f1e [733870,38329] 0 2024-04-22T15:49:50.507 INFO:tasks.workunit.client.1.smithi027.stdout:4/212: mknod d5/d45/d47/c50 0 2024-04-22T15:49:50.509 INFO:tasks.workunit.client.1.smithi027.stdout:7/150: readlink d0/d16/l1b 0 2024-04-22T15:49:50.510 INFO:tasks.workunit.client.1.smithi027.stdout:3/235: dwrite f16 [0,4194304] 0 2024-04-22T15:49:50.512 INFO:tasks.workunit.client.1.smithi027.stdout:4/213: write d5/d11/f36 [656716,20266] 0 2024-04-22T15:49:50.512 INFO:tasks.workunit.client.1.smithi027.stdout:3/236: read f5 [3397308,75967] 0 2024-04-22T15:49:50.512 INFO:tasks.workunit.client.1.smithi027.stdout:1/151: symlink d2/l31 0 2024-04-22T15:49:50.513 INFO:tasks.workunit.client.1.smithi027.stdout:8/181: mkdir d1/d3/d1c/d43 0 2024-04-22T15:49:50.514 INFO:tasks.workunit.client.1.smithi027.stdout:4/214: truncate d5/f48 530406 0 2024-04-22T15:49:50.515 INFO:tasks.workunit.client.1.smithi027.stdout:8/182: truncate d1/fb 1508568 0 2024-04-22T15:49:50.515 INFO:tasks.workunit.client.1.smithi027.stdout:4/215: write d5/f48 [272679,49971] 0 2024-04-22T15:49:50.515 INFO:tasks.workunit.client.1.smithi027.stdout:2/207: getdents dd/df/d1b/d1f 0 2024-04-22T15:49:50.517 INFO:tasks.workunit.client.1.smithi027.stdout:4/216: chown d5/d11/f36 11216182 1 2024-04-22T15:49:50.518 INFO:tasks.workunit.client.1.smithi027.stdout:9/179: write d0/f36 [984118,45510] 0 2024-04-22T15:49:50.519 INFO:tasks.workunit.client.1.smithi027.stdout:9/180: write d0/dc/de/f41 [535433,95416] 0 2024-04-22T15:49:50.520 INFO:tasks.workunit.client.1.smithi027.stdout:6/220: creat d5/dc/d17/d48/f4e x:0 0 0 2024-04-22T15:49:50.522 INFO:tasks.workunit.client.1.smithi027.stdout:6/221: write d5/dc/d20/d2b/f27 [4485069,92864] 0 2024-04-22T15:49:50.527 INFO:tasks.workunit.client.1.smithi027.stdout:5/211: mknod d0/df/de/d40/c43 0 2024-04-22T15:49:50.530 INFO:tasks.workunit.client.1.smithi027.stdout:7/151: unlink d0/d16/l24 0 2024-04-22T15:49:50.532 INFO:tasks.workunit.client.1.smithi027.stdout:3/237: rename la to d28/d4b/l4d 0 2024-04-22T15:49:50.534 INFO:tasks.workunit.client.1.smithi027.stdout:2/208: unlink dd/df/f12 0 2024-04-22T15:49:50.537 INFO:tasks.workunit.client.1.smithi027.stdout:9/181: dwrite d0/dc/f25 [0,4194304] 0 2024-04-22T15:49:50.538 INFO:tasks.workunit.client.1.smithi027.stdout:6/222: dwrite d5/dc/d17/d48/f4e [0,4194304] 0 2024-04-22T15:49:50.546 INFO:tasks.workunit.client.1.smithi027.stdout:1/152: fdatasync d2/fa 0 2024-04-22T15:49:50.547 INFO:tasks.workunit.client.1.smithi027.stdout:1/153: write d2/d1d/f24 [951906,94695] 0 2024-04-22T15:49:50.547 INFO:tasks.workunit.client.1.smithi027.stdout:7/152: mknod d0/d11/c2a 0 2024-04-22T15:49:50.549 INFO:tasks.workunit.client.1.smithi027.stdout:1/154: write d2/d1d/f1c [967287,123416] 0 2024-04-22T15:49:50.550 INFO:tasks.workunit.client.1.smithi027.stdout:3/238: unlink d28/c46 0 2024-04-22T15:49:50.552 INFO:tasks.workunit.client.1.smithi027.stdout:8/183: mknod d1/d3/d1c/d28/d42/c44 0 2024-04-22T15:49:50.553 INFO:tasks.workunit.client.1.smithi027.stdout:8/184: rename d1/d3 to d1/d3/d1c/d28/d45 22 2024-04-22T15:49:50.554 INFO:tasks.workunit.client.1.smithi027.stdout:8/185: readlink d1/d3/d36/l3a 0 2024-04-22T15:49:50.555 INFO:tasks.workunit.client.1.smithi027.stdout:2/209: creat dd/df/d1b/f40 x:0 0 0 2024-04-22T15:49:50.556 INFO:tasks.workunit.client.1.smithi027.stdout:2/210: write dd/f35 [882199,15272] 0 2024-04-22T15:49:50.557 INFO:tasks.workunit.client.1.smithi027.stdout:9/182: write d0/dc/df/f31 [747525,69572] 0 2024-04-22T15:49:50.558 INFO:tasks.workunit.client.1.smithi027.stdout:0/187: write d0/fc [3115965,128992] 0 2024-04-22T15:49:50.559 INFO:tasks.workunit.client.1.smithi027.stdout:6/223: fdatasync d5/dc/d26/d49/f45 0 2024-04-22T15:49:50.560 INFO:tasks.workunit.client.1.smithi027.stdout:0/188: truncate d0/d34/f3a 804989 0 2024-04-22T15:49:50.562 INFO:tasks.workunit.client.1.smithi027.stdout:5/212: creat d0/df/d3d/f44 x:0 0 0 2024-04-22T15:49:50.566 INFO:tasks.workunit.client.1.smithi027.stdout:7/153: mknod d0/d11/c2b 0 2024-04-22T15:49:50.567 INFO:tasks.workunit.client.1.smithi027.stdout:7/154: write d0/d16/f1e [350571,96227] 0 2024-04-22T15:49:50.568 INFO:tasks.workunit.client.1.smithi027.stdout:7/155: chown d0/fd 10329 1 2024-04-22T15:49:50.570 INFO:tasks.workunit.client.1.smithi027.stdout:1/155: readlink d2/lf 0 2024-04-22T15:49:50.572 INFO:tasks.workunit.client.1.smithi027.stdout:3/239: creat d28/f4e x:0 0 0 2024-04-22T15:49:50.572 INFO:tasks.workunit.client.1.smithi027.stdout:3/240: readlink d28/l47 0 2024-04-22T15:49:50.576 INFO:tasks.workunit.client.1.smithi027.stdout:4/217: creat d5/d11/d1d/d24/f51 x:0 0 0 2024-04-22T15:49:50.578 INFO:tasks.workunit.client.1.smithi027.stdout:2/211: creat dd/df/d1b/f41 x:0 0 0 2024-04-22T15:49:50.582 INFO:tasks.workunit.client.1.smithi027.stdout:3/241: dread d28/f37 [0,4194304] 0 2024-04-22T15:49:50.582 INFO:tasks.workunit.client.1.smithi027.stdout:3/242: readlink l2 0 2024-04-22T15:49:50.582 INFO:tasks.workunit.client.1.smithi027.stdout:5/213: rename d0/f7 to d0/df/f45 0 2024-04-22T15:49:50.583 INFO:tasks.workunit.client.1.smithi027.stdout:3/243: stat d28/l47 0 2024-04-22T15:49:50.585 INFO:tasks.workunit.client.1.smithi027.stdout:4/218: dwrite d5/f23 [0,4194304] 0 2024-04-22T15:49:50.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:50 smithi027 ceph-mon[34922]: Standby manager daemon smithi016.rnsnnr restarted 2024-04-22T15:49:50.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:50 smithi027 ceph-mon[34922]: Standby manager daemon smithi016.rnsnnr started 2024-04-22T15:49:50.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:50 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi016.rnsnnr/crt"}]: dispatch 2024-04-22T15:49:50.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:50 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:49:50.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:50 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi016.rnsnnr/key"}]: dispatch 2024-04-22T15:49:50.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:50 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:49:50.587 INFO:tasks.workunit.client.1.smithi027.stdout:5/214: truncate d0/d1a/d22/d26/d32/f37 176515 0 2024-04-22T15:49:50.588 INFO:tasks.workunit.client.1.smithi027.stdout:1/156: creat d2/d1d/f32 x:0 0 0 2024-04-22T15:49:50.590 INFO:tasks.workunit.client.1.smithi027.stdout:5/215: readlink d0/df/d10/l28 0 2024-04-22T15:49:50.590 INFO:tasks.workunit.client.1.smithi027.stdout:3/244: dread f1f [0,4194304] 0 2024-04-22T15:49:50.591 INFO:tasks.workunit.client.1.smithi027.stdout:9/183: unlink d0/dc/f22 0 2024-04-22T15:49:50.592 INFO:tasks.workunit.client.1.smithi027.stdout:3/245: chown d28/c39 73 1 2024-04-22T15:49:50.592 INFO:tasks.workunit.client.1.smithi027.stdout:5/216: write d0/df/d10/f14 [3516271,26370] 0 2024-04-22T15:49:50.593 INFO:tasks.workunit.client.1.smithi027.stdout:8/186: truncate d1/d3/d1c/d28/f35 479273 0 2024-04-22T15:49:50.602 INFO:tasks.workunit.client.1.smithi027.stdout:4/219: rename d5/d7/f15 to d5/d45/f52 0 2024-04-22T15:49:50.606 INFO:tasks.workunit.client.1.smithi027.stdout:5/217: fsync d0/f8 0 2024-04-22T15:49:50.607 INFO:tasks.workunit.client.1.smithi027.stdout:9/184: mknod d0/dc/d12/d30/c42 0 2024-04-22T15:49:50.608 INFO:tasks.workunit.client.1.smithi027.stdout:9/185: chown d0/dc/de/f40 10965 1 2024-04-22T15:49:50.610 INFO:tasks.workunit.client.1.smithi027.stdout:3/246: dread ff [0,4194304] 0 2024-04-22T15:49:50.659 INFO:tasks.workunit.client.1.smithi027.stdout:2/212: symlink dd/l42 0 2024-04-22T15:49:50.663 INFO:tasks.workunit.client.1.smithi027.stdout:2/213: dread dd/df/f28 [0,4194304] 0 2024-04-22T15:49:50.703 INFO:tasks.workunit.client.1.smithi027.stdout:8/187: rename d1/fb to d1/d3/f46 0 2024-04-22T15:49:50.704 INFO:tasks.workunit.client.1.smithi027.stdout:8/188: write d1/d3/d1c/d32/f3d [741987,115091] 0 2024-04-22T15:49:50.706 INFO:tasks.workunit.client.1.smithi027.stdout:4/220: creat d5/d11/d1d/f53 x:0 0 0 2024-04-22T15:49:50.707 INFO:tasks.workunit.client.1.smithi027.stdout:7/156: truncate d0/f28 999389 0 2024-04-22T15:49:50.708 INFO:tasks.workunit.client.1.smithi027.stdout:7/157: write d0/f15 [157709,118416] 0 2024-04-22T15:49:50.712 INFO:tasks.workunit.client.1.smithi027.stdout:8/189: dread d1/d3/f46 [0,4194304] 0 2024-04-22T15:49:50.717 INFO:tasks.workunit.client.1.smithi027.stdout:7/158: dwrite d0/d16/f23 [0,4194304] 0 2024-04-22T15:49:50.719 INFO:tasks.workunit.client.1.smithi027.stdout:5/218: creat d0/d1a/d22/d26/d32/f46 x:0 0 0 2024-04-22T15:49:50.720 INFO:tasks.workunit.client.1.smithi027.stdout:9/186: readlink d0/dc/l38 0 2024-04-22T15:49:50.723 INFO:tasks.workunit.client.1.smithi027.stdout:5/219: dread d0/f8 [0,4194304] 0 2024-04-22T15:49:50.723 INFO:tasks.workunit.client.1.smithi027.stdout:3/247: mknod d28/d4b/c4f 0 2024-04-22T15:49:50.725 INFO:tasks.workunit.client.1.smithi027.stdout:3/248: write f16 [1547765,92912] 0 2024-04-22T15:49:50.731 INFO:tasks.workunit.client.1.smithi027.stdout:0/189: getdents d0 0 2024-04-22T15:49:50.732 INFO:tasks.workunit.client.1.smithi027.stdout:3/249: dwrite d28/f3c [0,4194304] 0 2024-04-22T15:49:50.737 INFO:tasks.workunit.client.1.smithi027.stdout:4/221: mkdir d5/d11/d1d/d54 0 2024-04-22T15:49:50.751 INFO:tasks.workunit.client.1.smithi027.stdout:9/187: symlink d0/dc/de/l43 0 2024-04-22T15:49:50.759 INFO:tasks.workunit.client.1.smithi027.stdout:2/214: fsync dd/f22 0 2024-04-22T15:49:50.759 INFO:tasks.workunit.client.1.smithi027.stdout:2/215: chown dd/df/f18 48549 1 2024-04-22T15:49:50.760 INFO:tasks.workunit.client.1.smithi027.stdout:0/190: creat d0/d28/d37/f3d x:0 0 0 2024-04-22T15:49:50.761 INFO:tasks.workunit.client.1.smithi027.stdout:2/216: truncate dd/df/f18 683315 0 2024-04-22T15:49:50.762 INFO:tasks.workunit.client.1.smithi027.stdout:3/250: mknod d28/c50 0 2024-04-22T15:49:50.766 INFO:tasks.workunit.client.1.smithi027.stdout:7/159: mkdir d0/d11/d1c/d2c 0 2024-04-22T15:49:50.768 INFO:tasks.workunit.client.1.smithi027.stdout:7/160: read d0/d16/f1e [54060,80974] 0 2024-04-22T15:49:50.772 INFO:tasks.workunit.client.1.smithi027.stdout:4/222: symlink d5/d42/l55 0 2024-04-22T15:49:50.774 INFO:tasks.workunit.client.1.smithi027.stdout:3/251: mknod d28/d4b/c51 0 2024-04-22T15:49:50.779 INFO:tasks.workunit.client.1.smithi027.stdout:7/161: rename d0/d26/f27 to d0/d16/f2d 0 2024-04-22T15:49:50.785 INFO:tasks.workunit.client.1.smithi027.stdout:3/252: mkdir d28/d4b/d52 0 2024-04-22T15:49:50.786 INFO:tasks.workunit.client.1.smithi027.stdout:3/253: write f26 [3660636,29066] 0 2024-04-22T15:49:50.789 INFO:tasks.workunit.client.1.smithi027.stdout:4/223: truncate d5/d45/f52 3265007 0 2024-04-22T15:49:50.791 INFO:tasks.workunit.client.1.smithi027.stdout:3/254: creat d28/d4b/f53 x:0 0 0 2024-04-22T15:49:50.795 INFO:tasks.workunit.client.1.smithi027.stdout:4/224: link d5/d7/c8 d5/de/d2a/c56 0 2024-04-22T15:49:50.795 INFO:tasks.workunit.client.1.smithi027.stdout:3/255: dread f5 [0,4194304] 0 2024-04-22T15:49:50.797 INFO:tasks.workunit.client.1.smithi027.stdout:4/225: write d5/d11/d1d/f27 [728963,32787] 0 2024-04-22T15:49:50.800 INFO:tasks.workunit.client.1.smithi027.stdout:3/256: creat d28/d4b/d52/f54 x:0 0 0 2024-04-22T15:49:50.801 INFO:tasks.workunit.client.1.smithi027.stdout:3/257: chown d28/l2e 1017846567 1 2024-04-22T15:49:50.804 INFO:tasks.workunit.client.1.smithi027.stdout:4/226: dwrite d5/d7/fb [4194304,4194304] 0 2024-04-22T15:49:50.804 INFO:tasks.workunit.client.1.smithi027.stdout:3/258: symlink d28/d4b/d52/l55 0 2024-04-22T15:49:50.805 INFO:tasks.workunit.client.1.smithi027.stdout:4/227: stat d5/de/l1e 0 2024-04-22T15:49:50.805 INFO:tasks.workunit.client.1.smithi027.stdout:3/259: write d28/f3c [4698824,109605] 0 2024-04-22T15:49:50.813 INFO:tasks.workunit.client.1.smithi027.stdout:4/228: symlink d5/d11/d1d/d54/l57 0 2024-04-22T15:49:50.814 INFO:tasks.workunit.client.1.smithi027.stdout:3/260: dwrite f11 [0,4194304] 0 2024-04-22T15:49:50.815 INFO:tasks.workunit.client.1.smithi027.stdout:3/261: chown d28/f49 3211370 1 2024-04-22T15:49:50.817 INFO:tasks.workunit.client.1.smithi027.stdout:4/229: mkdir d5/de/d2a/d58 0 2024-04-22T15:49:50.819 INFO:tasks.workunit.client.1.smithi027.stdout:4/230: chown d5/d7/f2c 7 1 2024-04-22T15:49:50.819 INFO:tasks.workunit.client.1.smithi027.stdout:4/231: readlink d5/d11/l22 0 2024-04-22T15:49:50.823 INFO:tasks.workunit.client.1.smithi027.stdout:4/232: dread d5/f25 [0,4194304] 0 2024-04-22T15:49:50.825 INFO:tasks.workunit.client.1.smithi027.stdout:4/233: creat d5/d45/f59 x:0 0 0 2024-04-22T15:49:50.829 INFO:tasks.workunit.client.1.smithi027.stdout:3/262: fdatasync f11 0 2024-04-22T15:49:50.829 INFO:tasks.workunit.client.1.smithi027.stdout:4/234: mkdir d5/d45/d5a 0 2024-04-22T15:49:50.833 INFO:tasks.workunit.client.1.smithi027.stdout:3/263: creat d28/f56 x:0 0 0 2024-04-22T15:49:50.836 INFO:tasks.workunit.client.1.smithi027.stdout:4/235: dwrite d5/de/d2a/f3a [0,4194304] 0 2024-04-22T15:49:50.839 INFO:tasks.workunit.client.1.smithi027.stdout:3/264: creat d28/d4b/f57 x:0 0 0 2024-04-22T15:49:50.842 INFO:tasks.workunit.client.1.smithi027.stdout:4/236: unlink d5/d11/d1d/c46 0 2024-04-22T15:49:50.843 INFO:tasks.workunit.client.1.smithi027.stdout:3/265: mknod d28/d4b/c58 0 2024-04-22T15:49:50.846 INFO:tasks.workunit.client.1.smithi027.stdout:3/266: creat d28/d4b/f59 x:0 0 0 2024-04-22T15:49:50.846 INFO:tasks.workunit.client.1.smithi027.stdout:9/188: creat d0/f44 x:0 0 0 2024-04-22T15:49:50.849 INFO:tasks.workunit.client.1.smithi027.stdout:3/267: fsync d28/f36 0 2024-04-22T15:49:50.855 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:50 smithi016 ceph-mon[26620]: Standby manager daemon smithi016.rnsnnr restarted 2024-04-22T15:49:50.855 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:50 smithi016 ceph-mon[26620]: Standby manager daemon smithi016.rnsnnr started 2024-04-22T15:49:50.856 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:50 smithi016 ceph-mon[26620]: from='mgr.? 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi016.rnsnnr/crt"}]: dispatch 2024-04-22T15:49:50.856 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:50 smithi016 ceph-mon[26620]: from='mgr.? 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:49:50.856 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:50 smithi016 ceph-mon[26620]: from='mgr.? 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi016.rnsnnr/key"}]: dispatch 2024-04-22T15:49:50.856 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:50 smithi016 ceph-mon[26620]: from='mgr.? 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:49:50.856 INFO:tasks.workunit.client.1.smithi027.stdout:3/268: symlink d28/d4b/l5a 0 2024-04-22T15:49:50.859 INFO:tasks.workunit.client.1.smithi027.stdout:9/189: mknod d0/dc/c45 0 2024-04-22T15:49:50.861 INFO:tasks.workunit.client.1.smithi027.stdout:9/190: write d0/dc/de/f24 [2640893,80876] 0 2024-04-22T15:49:50.864 INFO:tasks.workunit.client.1.smithi027.stdout:1/157: dwrite d2/f3 [0,4194304] 0 2024-04-22T15:49:50.869 INFO:tasks.workunit.client.1.smithi027.stdout:9/191: unlink d0/dc/l38 0 2024-04-22T15:49:50.877 INFO:tasks.workunit.client.1.smithi027.stdout:1/158: unlink d2/l17 0 2024-04-22T15:49:50.881 INFO:tasks.workunit.client.1.smithi027.stdout:9/192: symlink d0/dc/l46 0 2024-04-22T15:49:50.883 INFO:tasks.workunit.client.1.smithi027.stdout:1/159: readlink d2/l2b 0 2024-04-22T15:49:50.891 INFO:tasks.workunit.client.1.smithi027.stdout:1/160: rename d2/c10 to d2/d30/c33 0 2024-04-22T15:49:50.893 INFO:tasks.workunit.client.1.smithi027.stdout:9/193: creat d0/f47 x:0 0 0 2024-04-22T15:49:50.894 INFO:tasks.workunit.client.1.smithi027.stdout:9/194: write d0/dc/de/f3a [981101,97954] 0 2024-04-22T15:49:50.899 INFO:tasks.workunit.client.1.smithi027.stdout:9/195: dread d0/dc/df/f31 [0,4194304] 0 2024-04-22T15:49:50.903 INFO:tasks.workunit.client.1.smithi027.stdout:9/196: creat d0/d8/f48 x:0 0 0 2024-04-22T15:49:50.916 INFO:tasks.workunit.client.1.smithi027.stdout:5/220: write d0/df/d10/f33 [174977,91246] 0 2024-04-22T15:49:50.918 INFO:tasks.workunit.client.1.smithi027.stdout:8/190: dwrite d1/f21 [0,4194304] 0 2024-04-22T15:49:50.925 INFO:tasks.workunit.client.1.smithi027.stdout:5/221: dwrite d0/f3 [0,4194304] 0 2024-04-22T15:49:50.926 INFO:tasks.workunit.client.1.smithi027.stdout:5/222: dread - d0/d1a/d22/d26/f2a zero size 2024-04-22T15:49:50.927 INFO:tasks.workunit.client.1.smithi027.stdout:8/191: mkdir d1/d1e/d47 0 2024-04-22T15:49:50.935 INFO:tasks.workunit.client.1.smithi027.stdout:0/191: truncate d0/f18 919167 0 2024-04-22T15:49:50.936 INFO:tasks.workunit.client.1.smithi027.stdout:2/217: truncate dd/f1e 1578525 0 2024-04-22T15:49:50.937 INFO:tasks.workunit.client.1.smithi027.stdout:0/192: dread d0/d28/f1f [0,4194304] 0 2024-04-22T15:49:50.938 INFO:tasks.workunit.client.1.smithi027.stdout:5/223: symlink d0/d1a/l47 0 2024-04-22T15:49:50.944 INFO:tasks.workunit.client.1.smithi027.stdout:0/193: dwrite d0/fc [0,4194304] 0 2024-04-22T15:49:50.945 INFO:tasks.workunit.client.1.smithi027.stdout:2/218: fdatasync dd/f19 0 2024-04-22T15:49:50.948 INFO:tasks.workunit.client.1.smithi027.stdout:5/224: creat d0/d1a/d22/d26/f48 x:0 0 0 2024-04-22T15:49:50.954 INFO:tasks.workunit.client.1.smithi027.stdout:5/225: fdatasync d0/f8 0 2024-04-22T15:49:50.958 INFO:tasks.workunit.client.1.smithi027.stdout:2/219: mkdir dd/d43 0 2024-04-22T15:49:50.963 INFO:tasks.workunit.client.1.smithi027.stdout:5/226: mknod d0/df/c49 0 2024-04-22T15:49:50.966 INFO:tasks.workunit.client.1.smithi027.stdout:5/227: dread d0/f8 [0,4194304] 0 2024-04-22T15:49:50.966 INFO:tasks.workunit.client.1.smithi027.stdout:2/220: creat dd/d43/f44 x:0 0 0 2024-04-22T15:49:50.967 INFO:tasks.workunit.client.1.smithi027.stdout:5/228: dread d0/df/de/d24/f16 [0,4194304] 0 2024-04-22T15:49:50.969 INFO:tasks.workunit.client.1.smithi027.stdout:5/229: symlink d0/d1a/l4a 0 2024-04-22T15:49:50.970 INFO:tasks.workunit.client.1.smithi027.stdout:5/230: read d0/df/f45 [3377665,77029] 0 2024-04-22T15:49:50.976 INFO:tasks.workunit.client.1.smithi027.stdout:5/231: dwrite d0/f3 [0,4194304] 0 2024-04-22T15:49:50.977 INFO:tasks.workunit.client.1.smithi027.stdout:5/232: fdatasync d0/df/de/d24/f2e 0 2024-04-22T15:49:50.981 INFO:tasks.workunit.client.1.smithi027.stdout:5/233: mknod d0/df/d3d/c4b 0 2024-04-22T15:49:50.993 INFO:tasks.workunit.client.1.smithi027.stdout:3/269: getdents d28/d4b 0 2024-04-22T15:49:50.995 INFO:tasks.workunit.client.1.smithi027.stdout:3/270: rename f1f to d28/d4b/f5b 0 2024-04-22T15:49:50.999 INFO:tasks.workunit.client.1.smithi027.stdout:5/234: rmdir d0/df 39 2024-04-22T15:49:51.001 INFO:tasks.workunit.client.1.smithi027.stdout:5/235: mkdir d0/d1a/d22/d4c 0 2024-04-22T15:49:51.002 INFO:tasks.workunit.client.1.smithi027.stdout:5/236: chown d0/df/d10/l39 96 1 2024-04-22T15:49:51.004 INFO:tasks.workunit.client.1.smithi027.stdout:5/237: symlink d0/df/de/l4d 0 2024-04-22T15:49:51.005 INFO:tasks.workunit.client.1.smithi027.stdout:1/161: getdents d2/d30 0 2024-04-22T15:49:51.007 INFO:tasks.workunit.client.1.smithi027.stdout:1/162: symlink d2/d30/l34 0 2024-04-22T15:49:51.009 INFO:tasks.workunit.client.1.smithi027.stdout:1/163: stat d2/c9 0 2024-04-22T15:49:51.010 INFO:tasks.workunit.client.1.smithi027.stdout:5/238: dwrite d0/df/de/d24/f2e [0,4194304] 0 2024-04-22T15:49:51.011 INFO:tasks.workunit.client.1.smithi027.stdout:1/164: write d2/f1f [4023532,92305] 0 2024-04-22T15:49:51.017 INFO:tasks.workunit.client.1.smithi027.stdout:1/165: creat d2/f35 x:0 0 0 2024-04-22T15:49:51.018 INFO:tasks.workunit.client.1.smithi027.stdout:6/224: sync 2024-04-22T15:49:51.018 INFO:tasks.workunit.client.1.smithi027.stdout:5/239: getdents d0/df/de/d24/d1c/d3e 0 2024-04-22T15:49:51.022 INFO:tasks.workunit.client.1.smithi027.stdout:1/166: mkdir d2/d36 0 2024-04-22T15:49:51.029 INFO:tasks.workunit.client.1.smithi027.stdout:6/225: dwrite d5/dc/d20/d2b/f35 [0,4194304] 0 2024-04-22T15:49:51.030 INFO:tasks.workunit.client.1.smithi027.stdout:1/167: rename d2/d1d/f2f to d2/f37 0 2024-04-22T15:49:51.030 INFO:tasks.workunit.client.1.smithi027.stdout:5/240: dwrite d0/d1a/d22/d26/d32/f46 [0,4194304] 0 2024-04-22T15:49:51.033 INFO:tasks.workunit.client.1.smithi027.stdout:5/241: fsync d0/df/de/d24/f35 0 2024-04-22T15:49:51.033 INFO:tasks.workunit.client.1.smithi027.stdout:6/226: write d5/dc/d17/f4b [450154,4675] 0 2024-04-22T15:49:51.039 INFO:tasks.workunit.client.1.smithi027.stdout:5/242: dread d0/df/d10/f14 [0,4194304] 0 2024-04-22T15:49:51.041 INFO:tasks.workunit.client.1.smithi027.stdout:6/227: read d5/f29 [414190,24868] 0 2024-04-22T15:49:51.046 INFO:tasks.workunit.client.1.smithi027.stdout:0/194: unlink d0/f18 0 2024-04-22T15:49:51.047 INFO:tasks.workunit.client.1.smithi027.stdout:5/243: creat d0/df/d3d/f4e x:0 0 0 2024-04-22T15:49:51.048 INFO:tasks.workunit.client.1.smithi027.stdout:6/228: dwrite d5/dc/d20/d2b/f35 [0,4194304] 0 2024-04-22T15:49:51.050 INFO:tasks.workunit.client.1.smithi027.stdout:5/244: write d0/df/de/f18 [4461735,125902] 0 2024-04-22T15:49:51.055 INFO:tasks.workunit.client.1.smithi027.stdout:5/245: dread d0/df/f12 [0,4194304] 0 2024-04-22T15:49:51.057 INFO:tasks.workunit.client.1.smithi027.stdout:0/195: dwrite d0/f12 [0,4194304] 0 2024-04-22T15:49:51.057 INFO:tasks.workunit.client.1.smithi027.stdout:5/246: fsync d0/df/de/f18 0 2024-04-22T15:49:51.058 INFO:tasks.workunit.client.1.smithi027.stdout:6/229: mknod d5/d30/d40/c4f 0 2024-04-22T15:49:51.059 INFO:tasks.workunit.client.1.smithi027.stdout:0/196: write d0/d28/d37/f3b [21365,38890] 0 2024-04-22T15:49:51.060 INFO:tasks.workunit.client.1.smithi027.stdout:0/197: stat d0/d28/c2f 0 2024-04-22T15:49:51.063 INFO:tasks.workunit.client.1.smithi027.stdout:5/247: unlink d0/df/de/d40/c43 0 2024-04-22T15:49:51.065 INFO:tasks.workunit.client.1.smithi027.stdout:8/192: truncate d1/f3f 2346789 0 2024-04-22T15:49:51.067 INFO:tasks.workunit.client.1.smithi027.stdout:0/198: dwrite d0/d28/d37/f3b [0,4194304] 0 2024-04-22T15:49:51.069 INFO:tasks.workunit.client.1.smithi027.stdout:8/193: dread d1/d3/f46 [0,4194304] 0 2024-04-22T15:49:51.069 INFO:tasks.workunit.client.1.smithi027.stdout:5/248: rmdir d0/df/d10 39 2024-04-22T15:49:51.075 INFO:tasks.workunit.client.1.smithi027.stdout:8/194: mknod d1/d1e/c48 0 2024-04-22T15:49:51.078 INFO:tasks.workunit.client.1.smithi027.stdout:8/195: mkdir d1/d3/d1c/d49 0 2024-04-22T15:49:51.083 INFO:tasks.workunit.client.1.smithi027.stdout:5/249: link d0/d1a/d22/d26/d32/f46 d0/d1a/d22/d26/d32/f4f 0 2024-04-22T15:49:51.085 INFO:tasks.workunit.client.1.smithi027.stdout:2/221: truncate dd/f11 3227561 0 2024-04-22T15:49:51.087 INFO:tasks.workunit.client.1.smithi027.stdout:0/199: getdents d0/d28 0 2024-04-22T15:49:51.089 INFO:tasks.workunit.client.1.smithi027.stdout:5/250: write d0/df/de/f38 [3437098,102788] 0 2024-04-22T15:49:51.089 INFO:tasks.workunit.client.1.smithi027.stdout:2/222: truncate dd/df/d1b/d1f/f3e 334456 0 2024-04-22T15:49:51.092 INFO:tasks.workunit.client.1.smithi027.stdout:8/196: dwrite d1/d1e/f33 [0,4194304] 0 2024-04-22T15:49:51.093 INFO:tasks.workunit.client.1.smithi027.stdout:8/197: dread - d1/d3/f3c zero size 2024-04-22T15:49:51.095 INFO:tasks.workunit.client.1.smithi027.stdout:8/198: write d1/f18 [106621,102808] 0 2024-04-22T15:49:51.096 INFO:tasks.workunit.client.1.smithi027.stdout:0/200: dread d0/fc [0,4194304] 0 2024-04-22T15:49:51.096 INFO:tasks.workunit.client.1.smithi027.stdout:3/271: write f5 [3937009,51159] 0 2024-04-22T15:49:51.096 INFO:tasks.workunit.client.1.smithi027.stdout:8/199: write d1/f21 [2664426,35113] 0 2024-04-22T15:49:51.097 INFO:tasks.workunit.client.1.smithi027.stdout:2/223: rename dd/df/f2f to dd/d1d/f45 0 2024-04-22T15:49:51.099 INFO:tasks.workunit.client.1.smithi027.stdout:8/200: write d1/d1e/f33 [4085497,62320] 0 2024-04-22T15:49:51.099 INFO:tasks.workunit.client.1.smithi027.stdout:5/251: mknod d0/df/de/d24/d1c/c50 0 2024-04-22T15:49:51.100 INFO:tasks.workunit.client.1.smithi027.stdout:8/201: stat d1/c3b 0 2024-04-22T15:49:51.108 INFO:tasks.workunit.client.1.smithi027.stdout:0/201: fdatasync d0/f22 0 2024-04-22T15:49:51.109 INFO:tasks.workunit.client.1.smithi027.stdout:0/202: read - d0/f2e zero size 2024-04-22T15:49:51.110 INFO:tasks.workunit.client.1.smithi027.stdout:1/168: rmdir d2 39 2024-04-22T15:49:51.113 INFO:tasks.workunit.client.1.smithi027.stdout:2/224: chown dd/df/d1b/l3f 54 1 2024-04-22T15:49:51.116 INFO:tasks.workunit.client.1.smithi027.stdout:5/252: creat d0/df/d3d/f51 x:0 0 0 2024-04-22T15:49:51.119 INFO:tasks.workunit.client.1.smithi027.stdout:8/202: dread d1/d3/f13 [0,4194304] 0 2024-04-22T15:49:51.120 INFO:tasks.workunit.client.1.smithi027.stdout:0/203: creat d0/f3e x:0 0 0 2024-04-22T15:49:51.123 INFO:tasks.workunit.client.1.smithi027.stdout:5/253: mkdir d0/df/de/d52 0 2024-04-22T15:49:51.133 INFO:tasks.workunit.client.1.smithi027.stdout:8/203: mknod d1/d3/d1c/d32/d41/c4a 0 2024-04-22T15:49:51.134 INFO:tasks.workunit.client.1.smithi027.stdout:1/169: mkdir d2/d36/d38 0 2024-04-22T15:49:51.135 INFO:tasks.workunit.client.1.smithi027.stdout:1/170: fdatasync d2/f35 0 2024-04-22T15:49:51.135 INFO:tasks.workunit.client.1.smithi027.stdout:2/225: link dd/df/d1b/l27 dd/df/d1b/d1f/l46 0 2024-04-22T15:49:51.136 INFO:tasks.workunit.client.1.smithi027.stdout:1/171: write d2/f15 [1818836,87680] 0 2024-04-22T15:49:51.137 INFO:tasks.workunit.client.1.smithi027.stdout:1/172: chown d2/f15 1926 1 2024-04-22T15:49:51.141 INFO:tasks.workunit.client.1.smithi027.stdout:8/204: mknod d1/d3/c4b 0 2024-04-22T15:49:51.142 INFO:tasks.workunit.client.1.smithi027.stdout:8/205: readlink d1/d3/l9 0 2024-04-22T15:49:51.145 INFO:tasks.workunit.client.1.smithi027.stdout:0/204: link d0/la d0/d2a/d38/l3f 0 2024-04-22T15:49:51.146 INFO:tasks.workunit.client.1.smithi027.stdout:8/206: fdatasync d1/d3/f13 0 2024-04-22T15:49:51.149 INFO:tasks.workunit.client.1.smithi027.stdout:1/173: mknod d2/c39 0 2024-04-22T15:49:51.150 INFO:tasks.workunit.client.1.smithi027.stdout:1/174: write d2/f1f [3631555,63631] 0 2024-04-22T15:49:51.152 INFO:tasks.workunit.client.1.smithi027.stdout:0/205: dwrite d0/f10 [0,4194304] 0 2024-04-22T15:49:51.152 INFO:tasks.workunit.client.1.smithi027.stdout:8/207: creat d1/d1e/f4c x:0 0 0 2024-04-22T15:49:51.153 INFO:tasks.workunit.client.1.smithi027.stdout:1/175: write d2/d1d/f24 [585634,7419] 0 2024-04-22T15:49:51.154 INFO:tasks.workunit.client.1.smithi027.stdout:0/206: chown d0/d2a/l2d 819 1 2024-04-22T15:49:51.154 INFO:tasks.workunit.client.1.smithi027.stdout:8/208: write d1/d3/f6 [794972,92752] 0 2024-04-22T15:49:51.155 INFO:tasks.workunit.client.1.smithi027.stdout:0/207: write d0/f10 [1045830,116014] 0 2024-04-22T15:49:51.166 INFO:tasks.workunit.client.1.smithi027.stdout:0/208: rename d0/f10 to d0/d28/d37/f40 0 2024-04-22T15:49:51.168 INFO:tasks.workunit.client.1.smithi027.stdout:8/209: unlink d1/d3/d1c/d28/f35 0 2024-04-22T15:49:51.169 INFO:tasks.workunit.client.1.smithi027.stdout:8/210: write d1/d3/f6 [1783427,111373] 0 2024-04-22T15:49:51.172 INFO:tasks.workunit.client.1.smithi027.stdout:1/176: creat d2/d36/d38/f3a x:0 0 0 2024-04-22T15:49:51.176 INFO:tasks.workunit.client.1.smithi027.stdout:0/209: mknod d0/d2a/d38/c41 0 2024-04-22T15:49:51.176 INFO:tasks.workunit.client.1.smithi027.stdout:5/254: truncate d0/df/de/f13 1328360 0 2024-04-22T15:49:51.177 INFO:tasks.workunit.client.1.smithi027.stdout:5/255: chown d0/d1a/d22/d26/d32/f37 10841201 1 2024-04-22T15:49:51.180 INFO:tasks.workunit.client.1.smithi027.stdout:2/226: truncate f7 3883109 0 2024-04-22T15:49:51.181 INFO:tasks.workunit.client.1.smithi027.stdout:1/177: rmdir d2/d1d 39 2024-04-22T15:49:51.183 INFO:tasks.workunit.client.1.smithi027.stdout:5/256: mkdir d0/df/de/d24/d1c/d3e/d53 0 2024-04-22T15:49:51.185 INFO:tasks.workunit.client.1.smithi027.stdout:8/211: rename d1/d1e/l38 to d1/d3/d1c/d28/d42/l4d 0 2024-04-22T15:49:51.189 INFO:tasks.workunit.client.1.smithi027.stdout:5/257: rename d0/df/de/d24/f16 to d0/d1a/d22/d26/d32/f54 0 2024-04-22T15:49:51.190 INFO:tasks.workunit.client.1.smithi027.stdout:8/212: mknod d1/d3/d36/c4e 0 2024-04-22T15:49:51.190 INFO:tasks.workunit.client.1.smithi027.stdout:5/258: dread - d0/d1a/f3a zero size 2024-04-22T15:49:51.191 INFO:tasks.workunit.client.1.smithi027.stdout:8/213: write d1/d1e/f33 [3220515,44701] 0 2024-04-22T15:49:51.191 INFO:tasks.workunit.client.1.smithi027.stdout:2/227: mkdir dd/d47 0 2024-04-22T15:49:51.192 INFO:tasks.workunit.client.1.smithi027.stdout:2/228: dread - dd/df/d1b/f2e zero size 2024-04-22T15:49:51.196 INFO:tasks.workunit.client.1.smithi027.stdout:5/259: rename d0/df/de/d24/d1c/d3e/d53 to d0/df/de/d24/d1c/d3e/d55 0 2024-04-22T15:49:51.198 INFO:tasks.workunit.client.1.smithi027.stdout:1/178: link d2/d1d/c20 d2/d36/c3b 0 2024-04-22T15:49:51.204 INFO:tasks.workunit.client.1.smithi027.stdout:2/229: creat dd/d47/f48 x:0 0 0 2024-04-22T15:49:51.206 INFO:tasks.workunit.client.1.smithi027.stdout:1/179: creat d2/d30/f3c x:0 0 0 2024-04-22T15:49:51.212 INFO:tasks.workunit.client.1.smithi027.stdout:5/260: creat d0/df/f56 x:0 0 0 2024-04-22T15:49:51.215 INFO:tasks.workunit.client.1.smithi027.stdout:8/214: unlink d1/f3f 0 2024-04-22T15:49:51.219 INFO:tasks.workunit.client.1.smithi027.stdout:5/261: chown d0/d1a/d22/l27 17291836 1 2024-04-22T15:49:51.220 INFO:tasks.workunit.client.1.smithi027.stdout:8/215: chown d1/d3/d1c/d28/c2c 19 1 2024-04-22T15:49:51.223 INFO:tasks.workunit.client.1.smithi027.stdout:5/262: creat d0/d1a/d22/d26/f57 x:0 0 0 2024-04-22T15:49:51.224 INFO:tasks.workunit.client.1.smithi027.stdout:5/263: fsync d0/df/de/d24/d1c/f34 0 2024-04-22T15:49:51.224 INFO:tasks.workunit.client.1.smithi027.stdout:5/264: fsync d0/f6 0 2024-04-22T15:49:51.225 INFO:tasks.workunit.client.1.smithi027.stdout:8/216: chown d1/f21 9211 1 2024-04-22T15:49:51.226 INFO:tasks.workunit.client.1.smithi027.stdout:5/265: chown d0/df/de/d24/d1c/c50 149178 1 2024-04-22T15:49:51.227 INFO:tasks.workunit.client.1.smithi027.stdout:5/266: readlink d0/df/de/d24/d1c/d3e/l41 0 2024-04-22T15:49:51.230 INFO:tasks.workunit.client.1.smithi027.stdout:8/217: dwrite d1/d3/f1a [0,4194304] 0 2024-04-22T15:49:51.232 INFO:tasks.workunit.client.1.smithi027.stdout:8/218: write d1/d3/f1a [3385423,47065] 0 2024-04-22T15:49:51.232 INFO:tasks.workunit.client.1.smithi027.stdout:5/267: mknod d0/d1a/d22/c58 0 2024-04-22T15:49:51.233 INFO:tasks.workunit.client.1.smithi027.stdout:5/268: chown d0/d1a/d22/d26 35 1 2024-04-22T15:49:51.234 INFO:tasks.workunit.client.1.smithi027.stdout:8/219: chown d1/d3/d1c/d28/c2c 448979403 1 2024-04-22T15:49:51.235 INFO:tasks.workunit.client.1.smithi027.stdout:8/220: read - d1/d3/f3c zero size 2024-04-22T15:49:51.235 INFO:tasks.workunit.client.1.smithi027.stdout:8/221: chown d1/f4 107 1 2024-04-22T15:49:51.240 INFO:tasks.workunit.client.1.smithi027.stdout:8/222: fsync d1/d3/f46 0 2024-04-22T15:49:51.245 INFO:tasks.workunit.client.1.smithi027.stdout:2/230: dwrite dd/df/f3c [0,4194304] 0 2024-04-22T15:49:51.245 INFO:tasks.workunit.client.1.smithi027.stdout:2/231: chown dd/f22 446329 1 2024-04-22T15:49:51.252 INFO:tasks.workunit.client.1.smithi027.stdout:5/269: getdents d0/df/de/d40 0 2024-04-22T15:49:51.254 INFO:tasks.workunit.client.1.smithi027.stdout:2/232: write dd/df/f17 [225164,74305] 0 2024-04-22T15:49:51.254 INFO:tasks.workunit.client.1.smithi027.stdout:5/270: rename d0 to d0/df/de/d24/d59 22 2024-04-22T15:49:51.255 INFO:tasks.workunit.client.1.smithi027.stdout:8/223: getdents d1/d3/d1c/d32/d41 0 2024-04-22T15:49:51.259 INFO:tasks.workunit.client.1.smithi027.stdout:5/271: dread d0/f6 [0,4194304] 0 2024-04-22T15:49:51.263 INFO:tasks.workunit.client.1.smithi027.stdout:8/224: mkdir d1/d4f 0 2024-04-22T15:49:51.263 INFO:tasks.workunit.client.1.smithi027.stdout:8/225: readlink d1/d3/d36/l3a 0 2024-04-22T15:49:51.269 INFO:tasks.workunit.client.1.smithi027.stdout:2/233: truncate dd/df/d1b/f25 2482388 0 2024-04-22T15:49:51.270 INFO:tasks.workunit.client.1.smithi027.stdout:5/272: truncate d0/df/de/f18 1027259 0 2024-04-22T15:49:51.272 INFO:tasks.workunit.client.1.smithi027.stdout:8/226: symlink d1/d3/d1c/d32/d41/l50 0 2024-04-22T15:49:51.272 INFO:tasks.workunit.client.1.smithi027.stdout:8/227: chown d1/f18 0 1 2024-04-22T15:49:51.282 INFO:tasks.workunit.client.1.smithi027.stdout:2/234: dwrite dd/d1d/f45 [0,4194304] 0 2024-04-22T15:49:51.282 INFO:tasks.workunit.client.1.smithi027.stdout:8/228: dwrite d1/d1e/f22 [4194304,4194304] 0 2024-04-22T15:49:51.286 INFO:tasks.workunit.client.1.smithi027.stdout:5/273: dread d0/df/d10/f14 [0,4194304] 0 2024-04-22T15:49:51.288 INFO:tasks.workunit.client.1.smithi027.stdout:8/229: creat d1/d3/d1c/d28/f51 x:0 0 0 2024-04-22T15:49:51.292 INFO:tasks.workunit.client.1.smithi027.stdout:5/274: rmdir d0/df/d3d 39 2024-04-22T15:49:51.296 INFO:tasks.workunit.client.1.smithi027.stdout:2/235: creat dd/df/f49 x:0 0 0 2024-04-22T15:49:51.297 INFO:tasks.workunit.client.1.smithi027.stdout:8/230: mkdir d1/d3/d1c/d32/d41/d52 0 2024-04-22T15:49:51.304 INFO:tasks.workunit.client.1.smithi027.stdout:5/275: dread d0/df/de/f18 [0,4194304] 0 2024-04-22T15:49:51.306 INFO:tasks.workunit.client.1.smithi027.stdout:8/231: mkdir d1/d1e/d47/d53 0 2024-04-22T15:49:51.314 INFO:tasks.workunit.client.1.smithi027.stdout:8/232: mknod d1/d3/d1c/d32/c54 0 2024-04-22T15:49:51.315 INFO:tasks.workunit.client.1.smithi027.stdout:8/233: fdatasync d1/f7 0 2024-04-22T15:49:51.315 INFO:tasks.workunit.client.1.smithi027.stdout:8/234: chown d1/d3/d1c/d28/d42 35 1 2024-04-22T15:49:51.316 INFO:tasks.workunit.client.1.smithi027.stdout:8/235: stat d1/d3/d1c/d28/f51 0 2024-04-22T15:49:51.319 INFO:tasks.workunit.client.1.smithi027.stdout:5/276: link d0/df/de/l31 d0/d1a/d22/d26/l5a 0 2024-04-22T15:49:51.321 INFO:tasks.workunit.client.1.smithi027.stdout:2/236: dwrite dd/f11 [0,4194304] 0 2024-04-22T15:49:51.324 INFO:tasks.workunit.client.1.smithi027.stdout:5/277: write d0/df/d3d/f51 [1040658,30452] 0 2024-04-22T15:49:51.325 INFO:tasks.workunit.client.1.smithi027.stdout:5/278: read d0/f3 [2674201,19920] 0 2024-04-22T15:49:51.327 INFO:tasks.workunit.client.1.smithi027.stdout:8/236: symlink d1/d1e/d47/d53/l55 0 2024-04-22T15:49:51.328 INFO:tasks.workunit.client.1.smithi027.stdout:2/237: creat dd/df/d1b/d1f/f4a x:0 0 0 2024-04-22T15:49:51.329 INFO:tasks.workunit.client.1.smithi027.stdout:2/238: write dd/df/d1b/d1f/f3e [798646,47130] 0 2024-04-22T15:49:51.330 INFO:tasks.workunit.client.1.smithi027.stdout:7/162: sync 2024-04-22T15:49:51.330 INFO:tasks.workunit.client.1.smithi027.stdout:2/239: fsync dd/df/f49 0 2024-04-22T15:49:51.331 INFO:tasks.workunit.client.1.smithi027.stdout:7/163: write d0/f14 [1730417,20935] 0 2024-04-22T15:49:51.332 INFO:tasks.workunit.client.1.smithi027.stdout:2/240: write dd/df/d1b/f32 [750757,114349] 0 2024-04-22T15:49:51.338 INFO:tasks.workunit.client.1.smithi027.stdout:8/237: mkdir d1/d3/d1c/d56 0 2024-04-22T15:49:51.339 INFO:tasks.workunit.client.1.smithi027.stdout:2/241: dwrite dd/f1c [0,4194304] 0 2024-04-22T15:49:51.348 INFO:tasks.workunit.client.1.smithi027.stdout:7/164: unlink d0/d11/l1d 0 2024-04-22T15:49:51.351 INFO:tasks.workunit.client.1.smithi027.stdout:8/238: mkdir d1/d3/d36/d57 0 2024-04-22T15:49:51.356 INFO:tasks.workunit.client.1.smithi027.stdout:5/279: getdents d0/d1a/d22/d26 0 2024-04-22T15:49:51.358 INFO:tasks.workunit.client.1.smithi027.stdout:2/242: dread dd/f22 [4194304,4194304] 0 2024-04-22T15:49:51.358 INFO:tasks.workunit.client.1.smithi027.stdout:8/239: symlink d1/d3/d36/d57/l58 0 2024-04-22T15:49:51.358 INFO:tasks.workunit.client.1.smithi027.stdout:7/165: dread d0/d11/fe [0,4194304] 0 2024-04-22T15:49:51.361 INFO:tasks.workunit.client.1.smithi027.stdout:2/243: write dd/df/d1b/d1f/f3e [981579,109312] 0 2024-04-22T15:49:51.361 INFO:tasks.workunit.client.1.smithi027.stdout:7/166: read d0/f14 [1250602,84995] 0 2024-04-22T15:49:51.367 INFO:tasks.workunit.client.1.smithi027.stdout:7/167: dread d0/f14 [0,4194304] 0 2024-04-22T15:49:51.369 INFO:tasks.workunit.client.1.smithi027.stdout:2/244: mkdir dd/d1d/d4b 0 2024-04-22T15:49:51.372 INFO:tasks.workunit.client.1.smithi027.stdout:5/280: write d0/d1a/d22/d26/d32/f54 [566099,55574] 0 2024-04-22T15:49:51.373 INFO:tasks.workunit.client.1.smithi027.stdout:8/240: truncate d1/d3/f46 284050 0 2024-04-22T15:49:51.376 INFO:tasks.workunit.client.1.smithi027.stdout:7/168: symlink d0/d16/l2e 0 2024-04-22T15:49:51.381 INFO:tasks.workunit.client.1.smithi027.stdout:5/281: mknod d0/d1a/d22/d26/d32/c5b 0 2024-04-22T15:49:51.382 INFO:tasks.workunit.client.1.smithi027.stdout:5/282: write d0/df/de/d24/d1c/f34 [1089242,39198] 0 2024-04-22T15:49:51.384 INFO:tasks.workunit.client.1.smithi027.stdout:8/241: unlink d1/c1b 0 2024-04-22T15:49:51.387 INFO:tasks.workunit.client.1.smithi027.stdout:7/169: mkdir d0/d16/d2f 0 2024-04-22T15:49:51.390 INFO:tasks.workunit.client.1.smithi027.stdout:8/242: dwrite d1/d1e/f33 [0,4194304] 0 2024-04-22T15:49:51.393 INFO:tasks.workunit.client.1.smithi027.stdout:2/245: getdents dd/d1d 0 2024-04-22T15:49:51.394 INFO:tasks.workunit.client.1.smithi027.stdout:7/170: creat d0/f30 x:0 0 0 2024-04-22T15:49:51.395 INFO:tasks.workunit.client.1.smithi027.stdout:7/171: chown d0/d16/f2d 3 1 2024-04-22T15:49:51.396 INFO:tasks.workunit.client.1.smithi027.stdout:8/243: link d1/d1e/f4c d1/d3/f59 0 2024-04-22T15:49:51.396 INFO:tasks.workunit.client.1.smithi027.stdout:7/172: read d0/fb [4061739,85762] 0 2024-04-22T15:49:51.397 INFO:tasks.workunit.client.1.smithi027.stdout:7/173: chown d0/d11 5163270 1 2024-04-22T15:49:51.401 INFO:tasks.workunit.client.1.smithi027.stdout:7/174: mkdir d0/d11/d1c/d31 0 2024-04-22T15:49:51.403 INFO:tasks.workunit.client.1.smithi027.stdout:7/175: write d0/d16/f25 [662519,114175] 0 2024-04-22T15:49:51.403 INFO:tasks.workunit.client.1.smithi027.stdout:7/176: readlink d0/d16/l2e 0 2024-04-22T15:49:51.404 INFO:tasks.workunit.client.1.smithi027.stdout:7/177: fsync d0/f30 0 2024-04-22T15:49:51.410 INFO:tasks.workunit.client.1.smithi027.stdout:7/178: rename d0/l1a to d0/l32 0 2024-04-22T15:49:51.412 INFO:tasks.workunit.client.1.smithi027.stdout:2/246: dwrite dd/df/d1b/f25 [0,4194304] 0 2024-04-22T15:49:51.414 INFO:tasks.workunit.client.1.smithi027.stdout:2/247: write dd/df/d1b/f40 [12921,53853] 0 2024-04-22T15:49:51.416 INFO:tasks.workunit.client.1.smithi027.stdout:7/179: creat d0/d11/d1c/d2c/f33 x:0 0 0 2024-04-22T15:49:51.417 INFO:tasks.workunit.client.1.smithi027.stdout:8/244: dread d1/f7 [0,4194304] 0 2024-04-22T15:49:51.420 INFO:tasks.workunit.client.1.smithi027.stdout:7/180: symlink d0/d11/d1c/l34 0 2024-04-22T15:49:51.423 INFO:tasks.workunit.client.1.smithi027.stdout:8/245: mknod d1/d3/d1c/d49/c5a 0 2024-04-22T15:49:51.425 INFO:tasks.workunit.client.1.smithi027.stdout:7/181: dwrite d0/fd [0,4194304] 0 2024-04-22T15:49:51.425 INFO:tasks.workunit.client.1.smithi027.stdout:8/246: stat d1/d3/d1c/l2d 0 2024-04-22T15:49:51.437 INFO:tasks.workunit.client.1.smithi027.stdout:8/247: dwrite d1/d3/d1c/d28/f51 [0,4194304] 0 2024-04-22T15:49:51.437 INFO:tasks.workunit.client.1.smithi027.stdout:7/182: dwrite d0/d16/f23 [4194304,4194304] 0 2024-04-22T15:49:51.438 INFO:tasks.workunit.client.1.smithi027.stdout:7/183: dread - d0/d16/f2d zero size 2024-04-22T15:49:51.444 INFO:tasks.workunit.client.1.smithi027.stdout:8/248: symlink d1/d1e/d47/d53/l5b 0 2024-04-22T15:49:51.453 INFO:tasks.workunit.client.1.smithi027.stdout:8/249: mknod d1/d3/d1c/d49/c5c 0 2024-04-22T15:49:51.456 INFO:tasks.workunit.client.1.smithi027.stdout:8/250: creat d1/d3/d1c/d49/f5d x:0 0 0 2024-04-22T15:49:51.458 INFO:tasks.workunit.client.1.smithi027.stdout:8/251: mknod d1/d1e/d47/d53/c5e 0 2024-04-22T15:49:51.460 INFO:tasks.workunit.client.1.smithi027.stdout:8/252: mkdir d1/d3/d1c/d43/d5f 0 2024-04-22T15:49:51.462 INFO:tasks.workunit.client.1.smithi027.stdout:8/253: creat d1/d3/d1c/d32/d41/d52/f60 x:0 0 0 2024-04-22T15:49:51.464 INFO:tasks.workunit.client.1.smithi027.stdout:7/184: write d0/d16/f1e [1005402,5200] 0 2024-04-22T15:49:51.465 INFO:tasks.workunit.client.1.smithi027.stdout:7/185: dread - d0/d16/f2d zero size 2024-04-22T15:49:51.468 INFO:tasks.workunit.client.1.smithi027.stdout:7/186: truncate d0/fb 2132788 0 2024-04-22T15:49:51.471 INFO:tasks.workunit.client.1.smithi027.stdout:7/187: symlink d0/d11/d1c/l35 0 2024-04-22T15:49:51.474 INFO:tasks.workunit.client.1.smithi027.stdout:7/188: rename d0/f22 to d0/d26/f36 0 2024-04-22T15:49:51.477 INFO:tasks.workunit.client.1.smithi027.stdout:8/254: truncate d1/d3/fa 3535520 0 2024-04-22T15:49:51.478 INFO:tasks.workunit.client.1.smithi027.stdout:7/189: chown d0/fb 659398 1 2024-04-22T15:49:51.481 INFO:tasks.workunit.client.1.smithi027.stdout:8/255: mkdir d1/d3/d1c/d32/d41/d52/d61 0 2024-04-22T15:49:51.481 INFO:tasks.workunit.client.1.smithi027.stdout:7/190: rmdir d0/d11 39 2024-04-22T15:49:51.482 INFO:tasks.workunit.client.1.smithi027.stdout:7/191: chown d0/d16/f25 7 1 2024-04-22T15:49:51.483 INFO:tasks.workunit.client.1.smithi027.stdout:8/256: rmdir d1/d3/d1c/d28 39 2024-04-22T15:49:51.483 INFO:tasks.workunit.client.1.smithi027.stdout:7/192: write d0/d16/f25 [1381679,3672] 0 2024-04-22T15:49:51.494 INFO:tasks.workunit.client.1.smithi027.stdout:8/257: write d1/d3/f59 [61877,32823] 0 2024-04-22T15:49:51.499 INFO:tasks.workunit.client.1.smithi027.stdout:8/258: mkdir d1/d3/d1c/d56/d62 0 2024-04-22T15:49:51.499 INFO:tasks.workunit.client.1.smithi027.stdout:8/259: truncate d1/d3/d1c/f34 942635 0 2024-04-22T15:49:51.502 INFO:tasks.workunit.client.1.smithi027.stdout:7/193: dread d0/d26/f36 [0,4194304] 0 2024-04-22T15:49:51.507 INFO:tasks.workunit.client.1.smithi027.stdout:8/260: dwrite d1/d3/d1c/d32/d41/d52/f60 [0,4194304] 0 2024-04-22T15:49:51.514 INFO:tasks.workunit.client.1.smithi027.stdout:7/194: link d0/d26/f36 d0/f37 0 2024-04-22T15:49:51.516 INFO:tasks.workunit.client.1.smithi027.stdout:8/261: dwrite d1/d3/d1c/d32/d41/d52/f60 [0,4194304] 0 2024-04-22T15:49:51.520 INFO:tasks.workunit.client.1.smithi027.stdout:7/195: unlink d0/d16/f1e 0 2024-04-22T15:49:51.534 INFO:tasks.workunit.client.1.smithi027.stdout:7/196: write d0/f13 [2263344,115200] 0 2024-04-22T15:49:51.536 INFO:tasks.workunit.client.1.smithi027.stdout:7/197: readlink d0/d11/d1c/l34 0 2024-04-22T15:49:51.538 INFO:tasks.workunit.client.1.smithi027.stdout:7/198: creat d0/d11/d1c/d2c/f38 x:0 0 0 2024-04-22T15:49:51.540 INFO:tasks.workunit.client.1.smithi027.stdout:7/199: dread d0/f15 [0,4194304] 0 2024-04-22T15:49:51.542 INFO:tasks.workunit.client.1.smithi027.stdout:7/200: readlink d0/l32 0 2024-04-22T15:49:51.542 INFO:tasks.workunit.client.1.smithi027.stdout:7/201: chown d0/l1 691280 1 2024-04-22T15:49:51.543 INFO:tasks.workunit.client.1.smithi027.stdout:7/202: dread - d0/d11/d1c/d2c/f33 zero size 2024-04-22T15:49:51.546 INFO:tasks.workunit.client.1.smithi027.stdout:7/203: getdents d0/d11/d1c/d31 0 2024-04-22T15:49:51.547 INFO:tasks.workunit.client.1.smithi027.stdout:7/204: stat d0/d11 0 2024-04-22T15:49:51.551 INFO:tasks.workunit.client.1.smithi027.stdout:7/205: link d0/f13 d0/f39 0 2024-04-22T15:49:51.552 INFO:tasks.workunit.client.1.smithi027.stdout:7/206: chown d0/d11/d1c 65 1 2024-04-22T15:49:51.557 INFO:tasks.workunit.client.1.smithi027.stdout:4/237: sync 2024-04-22T15:49:51.557 INFO:tasks.workunit.client.1.smithi027.stdout:7/207: dwrite d0/d16/f23 [0,4194304] 0 2024-04-22T15:49:51.559 INFO:tasks.workunit.client.1.smithi027.stdout:7/208: read d0/d16/f25 [283304,11271] 0 2024-04-22T15:49:51.560 INFO:tasks.workunit.client.1.smithi027.stdout:7/209: stat d0 0 2024-04-22T15:49:51.564 INFO:tasks.workunit.client.1.smithi027.stdout:7/210: mkdir d0/d11/d3a 0 2024-04-22T15:49:51.567 INFO:tasks.workunit.client.1.smithi027.stdout:7/211: dread d0/f14 [0,4194304] 0 2024-04-22T15:49:51.568 INFO:tasks.workunit.client.1.smithi027.stdout:4/238: dwrite d5/d11/f28 [4194304,4194304] 0 2024-04-22T15:49:51.569 INFO:tasks.workunit.client.1.smithi027.stdout:4/239: truncate d5/de/f43 80800 0 2024-04-22T15:49:51.692 INFO:tasks.workunit.client.1.smithi027.stdout:7/212: creat d0/d16/f3b x:0 0 0 2024-04-22T15:49:51.693 INFO:tasks.workunit.client.1.smithi027.stdout:7/213: rename d0 to d0/d16/d3c 22 2024-04-22T15:49:51.697 INFO:tasks.workunit.client.1.smithi027.stdout:7/214: creat d0/d16/f3d x:0 0 0 2024-04-22T15:49:51.700 INFO:tasks.workunit.client.1.smithi027.stdout:4/240: truncate d5/d11/f18 1070754 0 2024-04-22T15:49:51.701 INFO:tasks.workunit.client.1.smithi027.stdout:4/241: readlink d5/d42/l55 0 2024-04-22T15:49:51.702 INFO:tasks.workunit.client.1.smithi027.stdout:4/242: creat d5/d7/f5b x:0 0 0 2024-04-22T15:49:51.705 INFO:tasks.workunit.client.1.smithi027.stdout:4/243: write d5/de/d2a/f3a [1606876,79151] 0 2024-04-22T15:49:51.707 INFO:tasks.workunit.client.1.smithi027.stdout:7/215: dwrite d0/f37 [0,4194304] 0 2024-04-22T15:49:51.711 INFO:tasks.workunit.client.1.smithi027.stdout:7/216: rmdir d0/d16 39 2024-04-22T15:49:51.713 INFO:tasks.workunit.client.1.smithi027.stdout:4/244: link d5/de/f43 d5/d11/d1d/f5c 0 2024-04-22T15:49:51.719 INFO:tasks.workunit.client.1.smithi027.stdout:4/245: mkdir d5/de/d30/d5d 0 2024-04-22T15:49:51.722 INFO:tasks.workunit.client.1.smithi027.stdout:9/197: sync 2024-04-22T15:49:51.726 INFO:tasks.workunit.client.1.smithi027.stdout:4/246: creat d5/d11/d1d/f5e x:0 0 0 2024-04-22T15:49:51.731 INFO:tasks.workunit.client.1.smithi027.stdout:4/247: mkdir d5/de/d5f 0 2024-04-22T15:49:51.734 INFO:tasks.workunit.client.1.smithi027.stdout:9/198: link d0/dc/d12/f23 d0/dc/de/f49 0 2024-04-22T15:49:51.735 INFO:tasks.workunit.client.1.smithi027.stdout:9/199: chown d0/dc/d12 5 1 2024-04-22T15:49:51.735 INFO:tasks.workunit.client.1.smithi027.stdout:4/248: symlink d5/de/d30/d5d/l60 0 2024-04-22T15:49:51.738 INFO:tasks.workunit.client.1.smithi027.stdout:4/249: creat d5/d42/f61 x:0 0 0 2024-04-22T15:49:51.739 INFO:tasks.workunit.client.1.smithi027.stdout:4/250: write d5/d11/d1d/f5e [843396,48104] 0 2024-04-22T15:49:51.740 INFO:tasks.workunit.client.1.smithi027.stdout:4/251: fdatasync d5/f6 0 2024-04-22T15:49:51.797 INFO:tasks.workunit.client.1.smithi027.stdout:9/200: getdents d0/dc/de 0 2024-04-22T15:49:51.799 INFO:tasks.workunit.client.1.smithi027.stdout:9/201: mknod d0/dc/d12/d30/c4a 0 2024-04-22T15:49:51.803 INFO:tasks.workunit.client.1.smithi027.stdout:9/202: dread d0/dc/f25 [0,4194304] 0 2024-04-22T15:49:51.804 INFO:tasks.workunit.client.1.smithi027.stdout:4/252: write d5/de/f43 [653894,3458] 0 2024-04-22T15:49:51.809 INFO:tasks.workunit.client.1.smithi027.stdout:4/253: rmdir d5/de/d5f 0 2024-04-22T15:49:51.810 INFO:tasks.workunit.client.1.smithi027.stdout:9/203: dwrite d0/dc/de/f3a [0,4194304] 0 2024-04-22T15:49:51.817 INFO:tasks.workunit.client.1.smithi027.stdout:9/204: mknod d0/dc/c4b 0 2024-04-22T15:49:51.819 INFO:tasks.workunit.client.1.smithi027.stdout:4/254: getdents d5/d45 0 2024-04-22T15:49:51.821 INFO:tasks.workunit.client.1.smithi027.stdout:4/255: creat d5/d45/d47/f62 x:0 0 0 2024-04-22T15:49:51.823 INFO:tasks.workunit.client.1.smithi027.stdout:9/205: creat d0/dc/f4c x:0 0 0 2024-04-22T15:49:51.823 INFO:tasks.workunit.client.1.smithi027.stdout:9/206: chown d0/dc/df 2626 1 2024-04-22T15:49:51.824 INFO:tasks.workunit.client.1.smithi027.stdout:9/207: stat d0/dc/d12 0 2024-04-22T15:49:51.825 INFO:tasks.workunit.client.1.smithi027.stdout:9/208: write d0/dc/f37 [865356,125940] 0 2024-04-22T15:49:51.825 INFO:tasks.workunit.client.1.smithi027.stdout:4/256: mkdir d5/d11/d1d/d24/d63 0 2024-04-22T15:49:51.827 INFO:tasks.workunit.client.1.smithi027.stdout:4/257: creat d5/d45/f64 x:0 0 0 2024-04-22T15:49:51.828 INFO:tasks.workunit.client.1.smithi027.stdout:4/258: dread - d5/de/f16 zero size 2024-04-22T15:49:51.831 INFO:tasks.workunit.client.1.smithi027.stdout:9/209: link d0/d8/f11 d0/dc/d12/d30/f4d 0 2024-04-22T15:49:51.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:51 smithi027 ceph-mon[34922]: mgrmap e26: smithi027.ukdgzs(active, since 69s), standbys: smithi016.rnsnnr 2024-04-22T15:49:51.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:51 smithi027 ceph-mon[34922]: pgmap v38: 65 pgs: 65 active+clean; 2.3 GiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 61 MiB/s rd, 181 MiB/s wr, 386 op/s 2024-04-22T15:49:51.845 INFO:tasks.workunit.client.1.smithi027.stdout:4/259: getdents d5/d45 0 2024-04-22T15:49:51.846 INFO:tasks.workunit.client.1.smithi027.stdout:6/230: sync 2024-04-22T15:49:51.846 INFO:tasks.workunit.client.1.smithi027.stdout:3/272: sync 2024-04-22T15:49:51.846 INFO:tasks.workunit.client.1.smithi027.stdout:4/260: chown d5/de/d2a/c35 74537 1 2024-04-22T15:49:51.846 INFO:tasks.workunit.client.1.smithi027.stdout:3/273: dread - d28/d4b/f59 zero size 2024-04-22T15:49:51.847 INFO:tasks.workunit.client.1.smithi027.stdout:3/274: write f5 [1355883,77156] 0 2024-04-22T15:49:51.850 INFO:tasks.workunit.client.1.smithi027.stdout:6/231: mknod d5/db/d2e/c50 0 2024-04-22T15:49:51.850 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:51 smithi016 ceph-mon[26620]: mgrmap e26: smithi027.ukdgzs(active, since 69s), standbys: smithi016.rnsnnr 2024-04-22T15:49:51.851 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:51 smithi016 ceph-mon[26620]: pgmap v38: 65 pgs: 65 active+clean; 2.3 GiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 61 MiB/s rd, 181 MiB/s wr, 386 op/s 2024-04-22T15:49:51.855 INFO:tasks.workunit.client.1.smithi027.stdout:9/210: dwrite d0/dc/df/f31 [0,4194304] 0 2024-04-22T15:49:51.856 INFO:tasks.workunit.client.1.smithi027.stdout:9/211: chown d0/d8 1 1 2024-04-22T15:49:51.857 INFO:tasks.workunit.client.1.smithi027.stdout:4/261: rename d5/de/d2a/l39 to d5/d11/d1d/d24/l65 0 2024-04-22T15:49:51.873 INFO:tasks.workunit.client.1.smithi027.stdout:6/232: rename d5/dc/d26/d49/f45 to d5/dc/d26/d3d/f51 0 2024-04-22T15:49:51.873 INFO:tasks.workunit.client.1.smithi027.stdout:9/212: stat d0/dc/df/c2f 0 2024-04-22T15:49:51.874 INFO:tasks.workunit.client.1.smithi027.stdout:6/233: dread - d5/db/f4d zero size 2024-04-22T15:49:51.875 INFO:tasks.workunit.client.1.smithi027.stdout:9/213: read d0/dc/de/f3a [2226142,42968] 0 2024-04-22T15:49:51.877 INFO:tasks.workunit.client.1.smithi027.stdout:4/262: dwrite d5/f13 [0,4194304] 0 2024-04-22T15:49:51.879 INFO:tasks.workunit.client.1.smithi027.stdout:6/234: mkdir d5/dc/dd/d52 0 2024-04-22T15:49:51.881 INFO:tasks.workunit.client.1.smithi027.stdout:9/214: unlink d0/d8/f3f 0 2024-04-22T15:49:51.885 INFO:tasks.workunit.client.1.smithi027.stdout:4/263: mknod d5/de/d30/d5d/c66 0 2024-04-22T15:49:51.886 INFO:tasks.workunit.client.1.smithi027.stdout:4/264: read - d5/d7/f5b zero size 2024-04-22T15:49:51.889 INFO:tasks.workunit.client.1.smithi027.stdout:6/235: mkdir d5/dc/d17/d53 0 2024-04-22T15:49:51.895 INFO:tasks.workunit.client.1.smithi027.stdout:6/236: dwrite d5/dc/d17/d48/f4e [0,4194304] 0 2024-04-22T15:49:51.903 INFO:tasks.workunit.client.1.smithi027.stdout:9/215: rename d0/dc/l16 to d0/d8/l4e 0 2024-04-22T15:49:51.904 INFO:tasks.workunit.client.1.smithi027.stdout:9/216: fdatasync d0/dc/f4c 0 2024-04-22T15:49:51.905 INFO:tasks.workunit.client.1.smithi027.stdout:6/237: mkdir d5/d30/d54 0 2024-04-22T15:49:51.908 INFO:tasks.workunit.client.1.smithi027.stdout:4/265: link d5/f13 d5/de/d2a/d58/f67 0 2024-04-22T15:49:51.909 INFO:tasks.workunit.client.1.smithi027.stdout:4/266: readlink d5/de/d30/d5d/l60 0 2024-04-22T15:49:51.911 INFO:tasks.workunit.client.1.smithi027.stdout:6/238: dwrite d5/dc/d20/d2b/f35 [0,4194304] 0 2024-04-22T15:49:51.919 INFO:tasks.workunit.client.1.smithi027.stdout:6/239: dwrite d5/dc/d20/f3f [0,4194304] 0 2024-04-22T15:49:51.920 INFO:tasks.workunit.client.1.smithi027.stdout:6/240: chown d5/dc/d20/d2b/d34 2076866 1 2024-04-22T15:49:51.921 INFO:tasks.workunit.client.1.smithi027.stdout:4/267: creat d5/de/d2a/d58/f68 x:0 0 0 2024-04-22T15:49:51.921 INFO:tasks.workunit.client.1.smithi027.stdout:6/241: chown d5/c44 55240276 1 2024-04-22T15:49:51.929 INFO:tasks.workunit.client.1.smithi027.stdout:9/217: link d0/dc/de/f49 d0/dc/d12/d30/f4f 0 2024-04-22T15:49:51.940 INFO:tasks.workunit.client.1.smithi027.stdout:9/218: truncate d0/dc/d12/d30/f4d 1398706 0 2024-04-22T15:49:51.944 INFO:tasks.workunit.client.1.smithi027.stdout:9/219: fsync d0/dc/d12/d30/f32 0 2024-04-22T15:49:51.945 INFO:tasks.workunit.client.1.smithi027.stdout:4/268: truncate d5/de/f43 127754 0 2024-04-22T15:49:51.947 INFO:tasks.workunit.client.1.smithi027.stdout:6/242: truncate d5/dc/d20/f3f 3713377 0 2024-04-22T15:49:51.949 INFO:tasks.workunit.client.1.smithi027.stdout:0/210: sync 2024-04-22T15:49:51.949 INFO:tasks.workunit.client.1.smithi027.stdout:2/248: sync 2024-04-22T15:49:51.951 INFO:tasks.workunit.client.1.smithi027.stdout:0/211: read d0/d28/f1f [757967,130445] 0 2024-04-22T15:49:51.952 INFO:tasks.workunit.client.1.smithi027.stdout:9/220: dread d0/dc/de/f3a [0,4194304] 0 2024-04-22T15:49:51.953 INFO:tasks.workunit.client.1.smithi027.stdout:9/221: write d0/dc/f3d [1040376,13326] 0 2024-04-22T15:49:51.954 INFO:tasks.workunit.client.1.smithi027.stdout:6/243: mknod d5/dc/d20/d2b/c55 0 2024-04-22T15:49:51.959 INFO:tasks.workunit.client.1.smithi027.stdout:2/249: mknod dd/df/d1b/c4c 0 2024-04-22T15:49:51.961 INFO:tasks.workunit.client.1.smithi027.stdout:0/212: creat d0/d34/f42 x:0 0 0 2024-04-22T15:49:51.967 INFO:tasks.workunit.client.1.smithi027.stdout:4/269: rmdir d5/d11/d1d/d24/d63 0 2024-04-22T15:49:51.968 INFO:tasks.workunit.client.1.smithi027.stdout:4/270: write d5/d42/f61 [1035921,10429] 0 2024-04-22T15:49:51.971 INFO:tasks.workunit.client.1.smithi027.stdout:3/275: sync 2024-04-22T15:49:51.971 INFO:tasks.workunit.client.1.smithi027.stdout:5/283: sync 2024-04-22T15:49:51.972 INFO:tasks.workunit.client.1.smithi027.stdout:7/217: sync 2024-04-22T15:49:51.972 INFO:tasks.workunit.client.1.smithi027.stdout:8/262: sync 2024-04-22T15:49:51.972 INFO:tasks.workunit.client.1.smithi027.stdout:1/180: sync 2024-04-22T15:49:51.973 INFO:tasks.workunit.client.1.smithi027.stdout:8/263: fdatasync d1/d1e/f22 0 2024-04-22T15:49:51.974 INFO:tasks.workunit.client.1.smithi027.stdout:2/250: rmdir dd 39 2024-04-22T15:49:51.976 INFO:tasks.workunit.client.1.smithi027.stdout:1/181: dread d2/fa [0,4194304] 0 2024-04-22T15:49:51.977 INFO:tasks.workunit.client.1.smithi027.stdout:1/182: chown d2/l2e 41350351 1 2024-04-22T15:49:51.977 INFO:tasks.workunit.client.1.smithi027.stdout:6/244: rename d5/dc/dd/l3b to d5/db/l56 0 2024-04-22T15:49:51.979 INFO:tasks.workunit.client.1.smithi027.stdout:9/222: write d0/dc/de/f3a [4697543,49224] 0 2024-04-22T15:49:51.983 INFO:tasks.workunit.client.1.smithi027.stdout:3/276: mkdir d28/d4b/d5c 0 2024-04-22T15:49:51.984 INFO:tasks.workunit.client.1.smithi027.stdout:3/277: write d28/f56 [768893,117529] 0 2024-04-22T15:49:51.990 INFO:tasks.workunit.client.1.smithi027.stdout:8/264: creat d1/d3/d1c/d49/f63 x:0 0 0 2024-04-22T15:49:51.990 INFO:tasks.workunit.client.1.smithi027.stdout:8/265: chown d1/d3/d1c/c1f 57 1 2024-04-22T15:49:51.995 INFO:tasks.workunit.client.1.smithi027.stdout:2/251: dread dd/df/f17 [4194304,4194304] 0 2024-04-22T15:49:51.997 INFO:tasks.workunit.client.1.smithi027.stdout:9/223: mknod d0/d8/c50 0 2024-04-22T15:49:51.998 INFO:tasks.workunit.client.1.smithi027.stdout:3/278: write d28/f3f [460862,120134] 0 2024-04-22T15:49:51.999 INFO:tasks.workunit.client.1.smithi027.stdout:4/271: dwrite d5/de/d2a/d58/f67 [0,4194304] 0 2024-04-22T15:49:52.001 INFO:tasks.workunit.client.1.smithi027.stdout:5/284: symlink d0/df/de/d24/l5c 0 2024-04-22T15:49:52.002 INFO:tasks.workunit.client.1.smithi027.stdout:5/285: dread - d0/df/d3d/f44 zero size 2024-04-22T15:49:52.002 INFO:tasks.workunit.client.1.smithi027.stdout:5/286: readlink d0/d1a/l2f 0 2024-04-22T15:49:52.007 INFO:tasks.workunit.client.1.smithi027.stdout:3/279: mkdir d28/d5d 0 2024-04-22T15:49:52.007 INFO:tasks.workunit.client.1.smithi027.stdout:8/266: fsync d1/d3/d1c/d49/f63 0 2024-04-22T15:49:52.008 INFO:tasks.workunit.client.1.smithi027.stdout:7/218: rename d0/f15 to d0/f3e 0 2024-04-22T15:49:52.010 INFO:tasks.workunit.client.1.smithi027.stdout:4/272: mknod d5/de/d2a/c69 0 2024-04-22T15:49:52.012 INFO:tasks.workunit.client.1.smithi027.stdout:5/287: mkdir d0/d1a/d22/d26/d5d 0 2024-04-22T15:49:52.014 INFO:tasks.workunit.client.1.smithi027.stdout:5/288: chown d0/d1a/d22/d26/f57 14738 1 2024-04-22T15:49:52.015 INFO:tasks.workunit.client.1.smithi027.stdout:8/267: dwrite d1/d3/f40 [0,4194304] 0 2024-04-22T15:49:52.015 INFO:tasks.workunit.client.1.smithi027.stdout:0/213: rename d0/ld to d0/d2a/l43 0 2024-04-22T15:49:52.016 INFO:tasks.workunit.client.1.smithi027.stdout:0/214: write d0/d2a/f35 [25904,124287] 0 2024-04-22T15:49:52.017 INFO:tasks.workunit.client.1.smithi027.stdout:5/289: chown d0/df/d10/c1b 3 1 2024-04-22T15:49:52.018 INFO:tasks.workunit.client.1.smithi027.stdout:9/224: mkdir d0/d51 0 2024-04-22T15:49:52.023 INFO:tasks.workunit.client.1.smithi027.stdout:6/245: dwrite d5/dc/d26/d3d/f51 [0,4194304] 0 2024-04-22T15:49:52.024 INFO:tasks.workunit.client.1.smithi027.stdout:6/246: chown d5/dc/d26 286775 1 2024-04-22T15:49:52.028 INFO:tasks.workunit.client.1.smithi027.stdout:8/268: rename d1/l2e to d1/d3/d1c/d56/l64 0 2024-04-22T15:49:52.029 INFO:tasks.workunit.client.1.smithi027.stdout:8/269: write d1/d1e/f4c [707572,19787] 0 2024-04-22T15:49:52.033 INFO:tasks.workunit.client.1.smithi027.stdout:0/215: creat d0/d34/f44 x:0 0 0 2024-04-22T15:49:52.034 INFO:tasks.workunit.client.1.smithi027.stdout:7/219: getdents d0/d11/d1c/d31 0 2024-04-22T15:49:52.036 INFO:tasks.workunit.client.1.smithi027.stdout:4/273: unlink d5/d45/f52 0 2024-04-22T15:49:52.037 INFO:tasks.workunit.client.1.smithi027.stdout:2/252: getdents dd/d43 0 2024-04-22T15:49:52.038 INFO:tasks.workunit.client.1.smithi027.stdout:6/247: symlink d5/d30/d40/l57 0 2024-04-22T15:49:52.039 INFO:tasks.workunit.client.1.smithi027.stdout:6/248: fdatasync d5/dc/d20/d2b/f27 0 2024-04-22T15:49:52.043 INFO:tasks.workunit.client.1.smithi027.stdout:8/270: mknod d1/d3/c65 0 2024-04-22T15:49:52.046 INFO:tasks.workunit.client.1.smithi027.stdout:0/216: write d0/d28/d37/f40 [3854520,4428] 0 2024-04-22T15:49:52.048 INFO:tasks.workunit.client.1.smithi027.stdout:0/217: write d0/d2a/f2b [625146,61112] 0 2024-04-22T15:49:52.049 INFO:tasks.workunit.client.1.smithi027.stdout:0/218: dread - d0/f3e zero size 2024-04-22T15:49:52.049 INFO:tasks.workunit.client.1.smithi027.stdout:4/274: creat d5/de/d30/d5d/f6a x:0 0 0 2024-04-22T15:49:52.051 INFO:tasks.workunit.client.1.smithi027.stdout:0/219: truncate d0/d28/d37/f3d 704840 0 2024-04-22T15:49:52.051 INFO:tasks.workunit.client.1.smithi027.stdout:0/220: write d0/f12 [3561476,31956] 0 2024-04-22T15:49:52.052 INFO:tasks.workunit.client.1.smithi027.stdout:0/221: readlink d0/d2a/l30 0 2024-04-22T15:49:52.052 INFO:tasks.workunit.client.1.smithi027.stdout:0/222: chown d0/d2a 1317441 1 2024-04-22T15:49:52.054 INFO:tasks.workunit.client.1.smithi027.stdout:3/280: getdents d28 0 2024-04-22T15:49:52.054 INFO:tasks.workunit.client.1.smithi027.stdout:5/290: truncate d0/df/d10/f14 4021918 0 2024-04-22T15:49:52.055 INFO:tasks.workunit.client.1.smithi027.stdout:3/281: fsync f15 0 2024-04-22T15:49:52.056 INFO:tasks.workunit.client.1.smithi027.stdout:6/249: write d5/dc/d20/f3f [679887,64169] 0 2024-04-22T15:49:52.057 INFO:tasks.workunit.client.1.smithi027.stdout:7/220: mkdir d0/d3f 0 2024-04-22T15:49:52.059 INFO:tasks.workunit.client.1.smithi027.stdout:0/223: dread d0/f12 [0,4194304] 0 2024-04-22T15:49:52.060 INFO:tasks.workunit.client.1.smithi027.stdout:0/224: dread d0/d28/f1c [0,4194304] 0 2024-04-22T15:49:52.067 INFO:tasks.workunit.client.1.smithi027.stdout:0/225: dwrite d0/f12 [0,4194304] 0 2024-04-22T15:49:52.069 INFO:tasks.workunit.client.1.smithi027.stdout:8/271: symlink d1/d3/d1c/d56/d62/l66 0 2024-04-22T15:49:52.070 INFO:tasks.workunit.client.1.smithi027.stdout:8/272: read - d1/d3/d1c/d49/f63 zero size 2024-04-22T15:49:52.071 INFO:tasks.workunit.client.1.smithi027.stdout:9/225: getdents d0 0 2024-04-22T15:49:52.075 INFO:tasks.workunit.client.1.smithi027.stdout:9/226: dread d0/dc/de/f3a [4194304,4194304] 0 2024-04-22T15:49:52.076 INFO:tasks.workunit.client.1.smithi027.stdout:3/282: rename l23 to d28/d5d/l5e 0 2024-04-22T15:49:52.076 INFO:tasks.workunit.client.1.smithi027.stdout:6/250: symlink d5/db/d2e/l58 0 2024-04-22T15:49:52.080 INFO:tasks.workunit.client.1.smithi027.stdout:6/251: read d5/db/f1e [1609373,124933] 0 2024-04-22T15:49:52.083 INFO:tasks.workunit.client.1.smithi027.stdout:0/226: mkdir d0/d28/d45 0 2024-04-22T15:49:52.086 INFO:tasks.workunit.client.1.smithi027.stdout:8/273: rmdir d1/d1e/d47 39 2024-04-22T15:49:52.086 INFO:tasks.workunit.client.1.smithi027.stdout:5/291: dwrite d0/d1a/d22/d26/f2a [0,4194304] 0 2024-04-22T15:49:52.092 INFO:tasks.workunit.client.1.smithi027.stdout:6/252: dwrite d5/dc/d20/d2b/f12 [0,4194304] 0 2024-04-22T15:49:52.092 INFO:tasks.workunit.client.1.smithi027.stdout:3/283: symlink d28/d4b/d52/l5f 0 2024-04-22T15:49:52.093 INFO:tasks.workunit.client.1.smithi027.stdout:9/227: dread d0/dc/d12/d30/f4d [0,4194304] 0 2024-04-22T15:49:52.093 INFO:tasks.workunit.client.1.smithi027.stdout:3/284: stat d28/l3a 0 2024-04-22T15:49:52.095 INFO:tasks.workunit.client.1.smithi027.stdout:7/221: link d0/f14 d0/d11/d1c/d2c/f40 0 2024-04-22T15:49:52.095 INFO:tasks.workunit.client.1.smithi027.stdout:3/285: truncate d28/d4b/f4c 869878 0 2024-04-22T15:49:52.095 INFO:tasks.workunit.client.1.smithi027.stdout:7/222: fsync d0/f39 0 2024-04-22T15:49:52.097 INFO:tasks.workunit.client.1.smithi027.stdout:4/275: symlink d5/d11/d1d/l6b 0 2024-04-22T15:49:52.100 INFO:tasks.workunit.client.1.smithi027.stdout:0/227: rename d0/l2c to d0/d28/d37/l46 0 2024-04-22T15:49:52.108 INFO:tasks.workunit.client.1.smithi027.stdout:6/253: mkdir d5/db/d2e/d59 0 2024-04-22T15:49:52.112 INFO:tasks.workunit.client.1.smithi027.stdout:7/223: write d0/d16/f23 [926018,125909] 0 2024-04-22T15:49:52.113 INFO:tasks.workunit.client.1.smithi027.stdout:7/224: chown d0/d11/c18 484949514 1 2024-04-22T15:49:52.115 INFO:tasks.workunit.client.1.smithi027.stdout:8/274: dread d1/f4 [0,4194304] 0 2024-04-22T15:49:52.116 INFO:tasks.workunit.client.1.smithi027.stdout:8/275: dread - d1/d3/f3c zero size 2024-04-22T15:49:52.116 INFO:tasks.workunit.client.1.smithi027.stdout:0/228: mknod d0/d28/d37/c47 0 2024-04-22T15:49:52.117 INFO:tasks.workunit.client.1.smithi027.stdout:8/276: fsync d1/d3/d1c/d32/d41/d52/f60 0 2024-04-22T15:49:52.121 INFO:tasks.workunit.client.1.smithi027.stdout:3/286: symlink d28/d4b/d5c/l60 0 2024-04-22T15:49:52.121 INFO:tasks.workunit.client.1.smithi027.stdout:3/287: stat f1d 0 2024-04-22T15:49:52.122 INFO:tasks.workunit.client.1.smithi027.stdout:3/288: fsync f1c 0 2024-04-22T15:49:52.122 INFO:tasks.workunit.client.1.smithi027.stdout:3/289: write f21 [2925270,17964] 0 2024-04-22T15:49:52.124 INFO:tasks.workunit.client.1.smithi027.stdout:7/225: rmdir d0/d11/d1c/d2c 39 2024-04-22T15:49:52.124 INFO:tasks.workunit.client.1.smithi027.stdout:3/290: write d28/f44 [4456027,77627] 0 2024-04-22T15:49:52.126 INFO:tasks.workunit.client.1.smithi027.stdout:4/276: mkdir d5/d11/d6c 0 2024-04-22T15:49:52.126 INFO:tasks.workunit.client.1.smithi027.stdout:5/292: write d0/df/d10/f33 [4080205,9401] 0 2024-04-22T15:49:52.131 INFO:tasks.workunit.client.1.smithi027.stdout:8/277: symlink d1/d3/d1c/d56/l67 0 2024-04-22T15:49:52.136 INFO:tasks.workunit.client.1.smithi027.stdout:8/278: dwrite f0 [0,4194304] 0 2024-04-22T15:49:52.138 INFO:tasks.workunit.client.1.smithi027.stdout:6/254: truncate d5/dc/d20/d2b/f35 4173668 0 2024-04-22T15:49:52.140 INFO:tasks.workunit.client.1.smithi027.stdout:3/291: truncate d28/f38 670040 0 2024-04-22T15:49:52.141 INFO:tasks.workunit.client.1.smithi027.stdout:4/277: creat d5/d11/f6d x:0 0 0 2024-04-22T15:49:52.142 INFO:tasks.workunit.client.1.smithi027.stdout:3/292: fdatasync d28/d4b/f57 0 2024-04-22T15:49:52.145 INFO:tasks.workunit.client.1.smithi027.stdout:5/293: unlink d0/df/de/d24/l5c 0 2024-04-22T15:49:52.147 INFO:tasks.workunit.client.1.smithi027.stdout:8/279: dwrite d1/d3/f59 [0,4194304] 0 2024-04-22T15:49:52.148 INFO:tasks.workunit.client.1.smithi027.stdout:5/294: chown d0/df/f45 311178851 1 2024-04-22T15:49:52.148 INFO:tasks.workunit.client.1.smithi027.stdout:8/280: chown d1/d3/l12 7 1 2024-04-22T15:49:52.149 INFO:tasks.workunit.client.1.smithi027.stdout:3/293: dread f5 [0,4194304] 0 2024-04-22T15:49:52.156 INFO:tasks.workunit.client.1.smithi027.stdout:5/295: dwrite d0/f3 [0,4194304] 0 2024-04-22T15:49:52.162 INFO:tasks.workunit.client.1.smithi027.stdout:5/296: dread d0/d1a/d22/d26/d32/f4f [0,4194304] 0 2024-04-22T15:49:52.163 INFO:tasks.workunit.client.1.smithi027.stdout:0/229: creat d0/d2a/f48 x:0 0 0 2024-04-22T15:49:52.166 INFO:tasks.workunit.client.1.smithi027.stdout:8/281: creat d1/d3/d1c/d56/d62/f68 x:0 0 0 2024-04-22T15:49:52.169 INFO:tasks.workunit.client.1.smithi027.stdout:7/226: link d0/l10 d0/d3f/l41 0 2024-04-22T15:49:52.173 INFO:tasks.workunit.client.1.smithi027.stdout:4/278: mknod d5/d11/d6c/c6e 0 2024-04-22T15:49:52.178 INFO:tasks.workunit.client.1.smithi027.stdout:0/230: mkdir d0/d2a/d38/d49 0 2024-04-22T15:49:52.180 INFO:tasks.workunit.client.1.smithi027.stdout:3/294: dread d28/d4b/f5b [0,4194304] 0 2024-04-22T15:49:52.183 INFO:tasks.workunit.client.1.smithi027.stdout:5/297: dread d0/df/de/f13 [0,4194304] 0 2024-04-22T15:49:52.184 INFO:tasks.workunit.client.1.smithi027.stdout:7/227: truncate d0/d11/fe 1236577 0 2024-04-22T15:49:52.186 INFO:tasks.workunit.client.1.smithi027.stdout:3/295: unlink d28/c33 0 2024-04-22T15:49:52.187 INFO:tasks.workunit.client.1.smithi027.stdout:6/255: link d5/db/l14 d5/l5a 0 2024-04-22T15:49:52.188 INFO:tasks.workunit.client.1.smithi027.stdout:6/256: truncate d5/d30/f47 875624 0 2024-04-22T15:49:52.190 INFO:tasks.workunit.client.1.smithi027.stdout:7/228: truncate d0/d11/d1c/d2c/f40 2382529 0 2024-04-22T15:49:52.193 INFO:tasks.workunit.client.1.smithi027.stdout:4/279: rename d5/de/d2f/l32 to d5/d11/d1d/l6f 0 2024-04-22T15:49:52.195 INFO:tasks.workunit.client.1.smithi027.stdout:3/296: creat d28/d4b/d52/f61 x:0 0 0 2024-04-22T15:49:52.198 INFO:tasks.workunit.client.1.smithi027.stdout:0/231: rmdir d0/d28/d45 0 2024-04-22T15:49:52.199 INFO:tasks.workunit.client.1.smithi027.stdout:0/232: stat d0/cb 0 2024-04-22T15:49:52.200 INFO:tasks.workunit.client.1.smithi027.stdout:0/233: chown d0/d2a/l2d 10284772 1 2024-04-22T15:49:52.201 INFO:tasks.workunit.client.1.smithi027.stdout:8/282: getdents d1/d3/d1c/d32/d41 0 2024-04-22T15:49:52.202 INFO:tasks.workunit.client.1.smithi027.stdout:0/234: truncate d0/d28/d37/f40 4892923 0 2024-04-22T15:49:52.202 INFO:tasks.workunit.client.1.smithi027.stdout:5/298: creat d0/df/f5e x:0 0 0 2024-04-22T15:49:52.203 INFO:tasks.workunit.client.1.smithi027.stdout:3/297: mkdir d28/d5d/d62 0 2024-04-22T15:49:52.204 INFO:tasks.workunit.client.1.smithi027.stdout:6/257: dread d5/dc/d20/d2b/f12 [4194304,4194304] 0 2024-04-22T15:49:52.204 INFO:tasks.workunit.client.1.smithi027.stdout:5/299: truncate d0/df/f56 775937 0 2024-04-22T15:49:52.206 INFO:tasks.workunit.client.1.smithi027.stdout:7/229: getdents d0/d11/d1c/d31 0 2024-04-22T15:49:52.208 INFO:tasks.workunit.client.1.smithi027.stdout:8/283: creat d1/d3/d1c/d32/f69 x:0 0 0 2024-04-22T15:49:52.212 INFO:tasks.workunit.client.1.smithi027.stdout:0/235: unlink d0/d28/d37/f3d 0 2024-04-22T15:49:52.214 INFO:tasks.workunit.client.1.smithi027.stdout:0/236: chown d0/l19 1736255008 1 2024-04-22T15:49:52.214 INFO:tasks.workunit.client.1.smithi027.stdout:3/298: creat d28/f63 x:0 0 0 2024-04-22T15:49:52.217 INFO:tasks.workunit.client.1.smithi027.stdout:6/258: rmdir d5/dc/d20 39 2024-04-22T15:49:52.219 INFO:tasks.workunit.client.1.smithi027.stdout:1/183: sync 2024-04-22T15:49:52.219 INFO:tasks.workunit.client.1.smithi027.stdout:3/299: dwrite d28/d4b/d52/f54 [0,4194304] 0 2024-04-22T15:49:52.224 INFO:tasks.workunit.client.1.smithi027.stdout:7/230: rmdir d0 39 2024-04-22T15:49:52.226 INFO:tasks.workunit.client.1.smithi027.stdout:6/259: dwrite d5/dc/d26/f2d [0,4194304] 0 2024-04-22T15:49:52.231 INFO:tasks.workunit.client.1.smithi027.stdout:8/284: chown d1/d1e/d47/d53/l5b 269611 1 2024-04-22T15:49:52.233 INFO:tasks.workunit.client.1.smithi027.stdout:8/285: chown d1/d3/l12 1 1 2024-04-22T15:49:52.235 INFO:tasks.workunit.client.1.smithi027.stdout:0/237: creat d0/d2a/d38/f4a x:0 0 0 2024-04-22T15:49:52.239 INFO:tasks.workunit.client.1.smithi027.stdout:0/238: stat d0/d28/f1f 0 2024-04-22T15:49:52.239 INFO:tasks.workunit.client.1.smithi027.stdout:6/260: dwrite d5/dc/dd/f16 [0,4194304] 0 2024-04-22T15:49:52.239 INFO:tasks.workunit.client.1.smithi027.stdout:9/228: sync 2024-04-22T15:49:52.240 INFO:tasks.workunit.client.1.smithi027.stdout:2/253: sync 2024-04-22T15:49:52.242 INFO:tasks.workunit.client.1.smithi027.stdout:6/261: dread d5/d30/f47 [0,4194304] 0 2024-04-22T15:49:52.243 INFO:tasks.workunit.client.1.smithi027.stdout:0/239: dread d0/d34/f3a [0,4194304] 0 2024-04-22T15:49:52.244 INFO:tasks.workunit.client.1.smithi027.stdout:1/184: mkdir d2/d36/d38/d3d 0 2024-04-22T15:49:52.244 INFO:tasks.workunit.client.1.smithi027.stdout:5/300: dwrite d0/df/de/f18 [0,4194304] 0 2024-04-22T15:49:52.245 INFO:tasks.workunit.client.1.smithi027.stdout:0/240: write d0/d2a/f3c [897888,98503] 0 2024-04-22T15:49:52.247 INFO:tasks.workunit.client.1.smithi027.stdout:4/280: getdents d5/de 0 2024-04-22T15:49:52.247 INFO:tasks.workunit.client.1.smithi027.stdout:0/241: dread - d0/f2e zero size 2024-04-22T15:49:52.249 INFO:tasks.workunit.client.1.smithi027.stdout:8/286: write d1/d1e/f2f [776044,77462] 0 2024-04-22T15:49:52.255 INFO:tasks.workunit.client.1.smithi027.stdout:3/300: creat d28/d5d/d62/f64 x:0 0 0 2024-04-22T15:49:52.256 INFO:tasks.workunit.client.1.smithi027.stdout:3/301: chown d28/d4b/c4f 181178 1 2024-04-22T15:49:52.258 INFO:tasks.workunit.client.1.smithi027.stdout:9/229: unlink d0/dc/d12/d30/f4d 0 2024-04-22T15:49:52.262 INFO:tasks.workunit.client.1.smithi027.stdout:7/231: creat d0/d26/f42 x:0 0 0 2024-04-22T15:49:52.263 INFO:tasks.workunit.client.1.smithi027.stdout:7/232: stat d0/f14 0 2024-04-22T15:49:52.263 INFO:tasks.workunit.client.1.smithi027.stdout:4/281: chown d5/d11/d1d/l6f 4835016 1 2024-04-22T15:49:52.264 INFO:tasks.workunit.client.1.smithi027.stdout:4/282: chown d5/d11/d1d 129 1 2024-04-22T15:49:52.265 INFO:tasks.workunit.client.1.smithi027.stdout:0/242: symlink d0/d28/l4b 0 2024-04-22T15:49:52.266 INFO:tasks.workunit.client.1.smithi027.stdout:9/230: dwrite d0/dc/de/f40 [0,4194304] 0 2024-04-22T15:49:52.266 INFO:tasks.workunit.client.1.smithi027.stdout:0/243: write d0/f12 [4974115,60563] 0 2024-04-22T15:49:52.274 INFO:tasks.workunit.client.1.smithi027.stdout:3/302: creat d28/d4b/d52/f65 x:0 0 0 2024-04-22T15:49:52.276 INFO:tasks.workunit.client.1.smithi027.stdout:5/301: symlink d0/df/l5f 0 2024-04-22T15:49:52.277 INFO:tasks.workunit.client.1.smithi027.stdout:5/302: chown d0/f3c 15332 1 2024-04-22T15:49:52.277 INFO:tasks.workunit.client.1.smithi027.stdout:6/262: rename d5/dc/d20/c46 to d5/dc/d26/d49/c5b 0 2024-04-22T15:49:52.279 INFO:tasks.workunit.client.1.smithi027.stdout:6/263: chown d5/dc/d20/d2b 505327622 1 2024-04-22T15:49:52.281 INFO:tasks.workunit.client.1.smithi027.stdout:9/231: mknod d0/c52 0 2024-04-22T15:49:52.282 INFO:tasks.workunit.client.1.smithi027.stdout:0/244: symlink d0/d28/d37/l4c 0 2024-04-22T15:49:52.287 INFO:tasks.workunit.client.1.smithi027.stdout:2/254: creat dd/f4d x:0 0 0 2024-04-22T15:49:52.287 INFO:tasks.workunit.client.1.smithi027.stdout:3/303: creat d28/d5d/d62/f66 x:0 0 0 2024-04-22T15:49:52.290 INFO:tasks.workunit.client.1.smithi027.stdout:5/303: creat d0/df/d10/f60 x:0 0 0 2024-04-22T15:49:52.291 INFO:tasks.workunit.client.1.smithi027.stdout:5/304: fdatasync d0/df/de/d24/f35 0 2024-04-22T15:49:52.292 INFO:tasks.workunit.client.1.smithi027.stdout:7/233: rename d0/d16/l1b to d0/d11/d1c/d31/l43 0 2024-04-22T15:49:52.293 INFO:tasks.workunit.client.1.smithi027.stdout:8/287: truncate d1/d3/f6 3856891 0 2024-04-22T15:49:52.298 INFO:tasks.workunit.client.1.smithi027.stdout:3/304: mknod d28/d4b/c67 0 2024-04-22T15:49:52.299 INFO:tasks.workunit.client.1.smithi027.stdout:3/305: write d28/f3f [621589,99547] 0 2024-04-22T15:49:52.301 INFO:tasks.workunit.client.1.smithi027.stdout:1/185: getdents d2/d1d 0 2024-04-22T15:49:52.303 INFO:tasks.workunit.client.1.smithi027.stdout:6/264: mkdir d5/dc/d17/d53/d5c 0 2024-04-22T15:49:52.304 INFO:tasks.workunit.client.1.smithi027.stdout:6/265: truncate d5/dc/d17/f39 541145 0 2024-04-22T15:49:52.306 INFO:tasks.workunit.client.1.smithi027.stdout:0/245: symlink d0/d2a/l4d 0 2024-04-22T15:49:52.307 INFO:tasks.workunit.client.1.smithi027.stdout:0/246: chown d0/f3e 102731 1 2024-04-22T15:49:52.307 INFO:tasks.workunit.client.1.smithi027.stdout:2/255: mkdir dd/d1d/d4b/d4e 0 2024-04-22T15:49:52.309 INFO:tasks.workunit.client.1.smithi027.stdout:5/305: getdents d0/d1a/d22/d26/d5d 0 2024-04-22T15:49:52.310 INFO:tasks.workunit.client.1.smithi027.stdout:4/283: dread d5/d7/f2c [0,4194304] 0 2024-04-22T15:49:52.311 INFO:tasks.workunit.client.1.smithi027.stdout:3/306: rename d28/d4b/d5c/l60 to d28/d5d/l68 0 2024-04-22T15:49:52.312 INFO:tasks.workunit.client.1.smithi027.stdout:1/186: fdatasync d2/f37 0 2024-04-22T15:49:52.313 INFO:tasks.workunit.client.1.smithi027.stdout:7/234: symlink d0/d11/d3a/l44 0 2024-04-22T15:49:52.313 INFO:tasks.workunit.client.1.smithi027.stdout:9/232: dwrite d0/dc/df/f31 [0,4194304] 0 2024-04-22T15:49:52.315 INFO:tasks.workunit.client.1.smithi027.stdout:6/266: rmdir d5/db 39 2024-04-22T15:49:52.318 INFO:tasks.workunit.client.1.smithi027.stdout:2/256: dwrite dd/df/d1b/f26 [0,4194304] 0 2024-04-22T15:49:52.319 INFO:tasks.workunit.client.1.smithi027.stdout:0/247: mkdir d0/d34/d4e 0 2024-04-22T15:49:52.320 INFO:tasks.workunit.client.1.smithi027.stdout:0/248: write d0/f12 [1991257,22666] 0 2024-04-22T15:49:52.326 INFO:tasks.workunit.client.1.smithi027.stdout:4/284: creat d5/de/d2f/f70 x:0 0 0 2024-04-22T15:49:52.331 INFO:tasks.workunit.client.1.smithi027.stdout:3/307: mkdir d28/d69 0 2024-04-22T15:49:52.331 INFO:tasks.workunit.client.1.smithi027.stdout:3/308: readlink d28/l2c 0 2024-04-22T15:49:52.331 INFO:tasks.workunit.client.1.smithi027.stdout:3/309: chown f15 59331034 1 2024-04-22T15:49:52.332 INFO:tasks.workunit.client.1.smithi027.stdout:3/310: chown d28/f43 87815741 1 2024-04-22T15:49:52.334 INFO:tasks.workunit.client.1.smithi027.stdout:1/187: rmdir d2/d36/d38 39 2024-04-22T15:49:52.335 INFO:tasks.workunit.client.1.smithi027.stdout:1/188: fdatasync d2/f3 0 2024-04-22T15:49:52.336 INFO:tasks.workunit.client.1.smithi027.stdout:6/267: mkdir d5/dc/d17/d48/d5d 0 2024-04-22T15:49:52.338 INFO:tasks.workunit.client.1.smithi027.stdout:2/257: rmdir dd/df 39 2024-04-22T15:49:52.339 INFO:tasks.workunit.client.1.smithi027.stdout:0/249: fsync d0/d28/f1f 0 2024-04-22T15:49:52.341 INFO:tasks.workunit.client.1.smithi027.stdout:5/306: getdents d0/df/de/d52 0 2024-04-22T15:49:52.342 INFO:tasks.workunit.client.1.smithi027.stdout:5/307: write d0/df/d10/f33 [1321387,34116] 0 2024-04-22T15:49:52.342 INFO:tasks.workunit.client.1.smithi027.stdout:4/285: chown d5/d45/d47/f62 1 1 2024-04-22T15:49:52.343 INFO:tasks.workunit.client.1.smithi027.stdout:5/308: fdatasync d0/df/f5e 0 2024-04-22T15:49:52.343 INFO:tasks.workunit.client.1.smithi027.stdout:3/311: creat d28/d4b/d52/f6a x:0 0 0 2024-04-22T15:49:52.345 INFO:tasks.workunit.client.1.smithi027.stdout:3/312: write d28/d5d/d62/f64 [702031,95508] 0 2024-04-22T15:49:52.347 INFO:tasks.workunit.client.1.smithi027.stdout:3/313: chown d28/f36 2309258 1 2024-04-22T15:49:52.355 INFO:tasks.workunit.client.1.smithi027.stdout:4/286: symlink d5/d45/l71 0 2024-04-22T15:49:52.358 INFO:tasks.workunit.client.1.smithi027.stdout:3/314: dread f11 [0,4194304] 0 2024-04-22T15:49:52.358 INFO:tasks.workunit.client.1.smithi027.stdout:9/233: creat d0/dc/f53 x:0 0 0 2024-04-22T15:49:52.358 INFO:tasks.workunit.client.1.smithi027.stdout:3/315: fsync f10 0 2024-04-22T15:49:52.360 INFO:tasks.workunit.client.1.smithi027.stdout:5/309: creat d0/df/de/d24/d1c/d3e/d55/f61 x:0 0 0 2024-04-22T15:49:52.361 INFO:tasks.workunit.client.1.smithi027.stdout:1/189: mknod d2/d36/d38/c3e 0 2024-04-22T15:49:52.362 INFO:tasks.workunit.client.1.smithi027.stdout:1/190: stat d2/d1d/c1b 0 2024-04-22T15:49:52.363 INFO:tasks.workunit.client.1.smithi027.stdout:6/268: fsync d5/db/f1e 0 2024-04-22T15:49:52.366 INFO:tasks.workunit.client.1.smithi027.stdout:4/287: rename d5/d11/d1d/d24/f40 to d5/de/d30/f72 0 2024-04-22T15:49:52.375 INFO:tasks.workunit.client.1.smithi027.stdout:0/250: rmdir d0/d2a/d38/d49 0 2024-04-22T15:49:52.379 INFO:tasks.workunit.client.1.smithi027.stdout:3/316: rename ff to d28/d4b/d5c/f6b 0 2024-04-22T15:49:52.381 INFO:tasks.workunit.client.1.smithi027.stdout:9/234: write d0/dc/de/f3a [759195,101327] 0 2024-04-22T15:49:52.382 INFO:tasks.workunit.client.1.smithi027.stdout:3/317: chown d28/d4b/f5b 18 1 2024-04-22T15:49:52.382 INFO:tasks.workunit.client.1.smithi027.stdout:0/251: dwrite d0/d34/f42 [0,4194304] 0 2024-04-22T15:49:52.382 INFO:tasks.workunit.client.1.smithi027.stdout:3/318: fsync f1d 0 2024-04-22T15:49:52.384 INFO:tasks.workunit.client.1.smithi027.stdout:5/310: mkdir d0/df/de/d24/d62 0 2024-04-22T15:49:52.385 INFO:tasks.workunit.client.1.smithi027.stdout:3/319: truncate d28/d4b/f57 154002 0 2024-04-22T15:49:52.387 INFO:tasks.workunit.client.1.smithi027.stdout:3/320: write f15 [6197774,126358] 0 2024-04-22T15:49:52.389 INFO:tasks.workunit.client.1.smithi027.stdout:2/258: getdents dd/d47 0 2024-04-22T15:49:52.392 INFO:tasks.workunit.client.1.smithi027.stdout:9/235: unlink d0/dc/f53 0 2024-04-22T15:49:52.393 INFO:tasks.workunit.client.1.smithi027.stdout:6/269: dwrite d5/f29 [0,4194304] 0 2024-04-22T15:49:52.394 INFO:tasks.workunit.client.1.smithi027.stdout:9/236: fsync d0/dc/f3d 0 2024-04-22T15:49:52.396 INFO:tasks.workunit.client.1.smithi027.stdout:9/237: write d0/dc/f3d [1820823,47341] 0 2024-04-22T15:49:52.397 INFO:tasks.workunit.client.1.smithi027.stdout:9/238: write d0/dc/de/f3a [1501990,92186] 0 2024-04-22T15:49:52.401 INFO:tasks.workunit.client.1.smithi027.stdout:5/311: mknod d0/df/d10/c63 0 2024-04-22T15:49:52.403 INFO:tasks.workunit.client.1.smithi027.stdout:3/321: creat d28/f6c x:0 0 0 2024-04-22T15:49:52.405 INFO:tasks.workunit.client.1.smithi027.stdout:2/259: rmdir dd 39 2024-04-22T15:49:52.417 INFO:tasks.workunit.client.1.smithi027.stdout:3/322: dread d28/f3c [4194304,4194304] 0 2024-04-22T15:49:52.417 INFO:tasks.workunit.client.1.smithi027.stdout:3/323: fsync d28/f4e 0 2024-04-22T15:49:52.428 INFO:tasks.workunit.client.1.smithi027.stdout:9/239: stat d0/dc/d12/f23 0 2024-04-22T15:49:52.428 INFO:tasks.workunit.client.1.smithi027.stdout:9/240: chown d0/dc/de/f3a 4 1 2024-04-22T15:49:52.429 INFO:tasks.workunit.client.1.smithi027.stdout:0/252: symlink d0/d2a/l4f 0 2024-04-22T15:49:52.430 INFO:tasks.workunit.client.1.smithi027.stdout:0/253: write d0/d2a/f35 [383391,61361] 0 2024-04-22T15:49:52.432 INFO:tasks.workunit.client.1.smithi027.stdout:1/191: rename d2/d36/d38/c3e to d2/d36/c3f 0 2024-04-22T15:49:52.433 INFO:tasks.workunit.client.1.smithi027.stdout:3/324: fdatasync d28/d4b/d5c/f6b 0 2024-04-22T15:49:52.433 INFO:tasks.workunit.client.1.smithi027.stdout:6/270: mknod d5/d30/d54/c5e 0 2024-04-22T15:49:52.434 INFO:tasks.workunit.client.1.smithi027.stdout:3/325: write d28/f34 [1277249,21019] 0 2024-04-22T15:49:52.435 INFO:tasks.workunit.client.1.smithi027.stdout:9/241: mknod d0/dc/df/c54 0 2024-04-22T15:49:52.436 INFO:tasks.workunit.client.1.smithi027.stdout:6/271: truncate d5/dc/d26/f2d 4440865 0 2024-04-22T15:49:52.439 INFO:tasks.workunit.client.1.smithi027.stdout:1/192: creat d2/d36/d38/f40 x:0 0 0 2024-04-22T15:49:52.444 INFO:tasks.workunit.client.1.smithi027.stdout:3/326: truncate d28/f36 4566976 0 2024-04-22T15:49:52.444 INFO:tasks.workunit.client.1.smithi027.stdout:3/327: chown d28/c3d 919 1 2024-04-22T15:49:52.445 INFO:tasks.workunit.client.1.smithi027.stdout:9/242: mknod d0/dc/d12/c55 0 2024-04-22T15:49:52.445 INFO:tasks.workunit.client.1.smithi027.stdout:1/193: dwrite d2/f1f [0,4194304] 0 2024-04-22T15:49:52.449 INFO:tasks.workunit.client.1.smithi027.stdout:0/254: mknod d0/d2a/c50 0 2024-04-22T15:49:52.450 INFO:tasks.workunit.client.1.smithi027.stdout:5/312: truncate d0/df/d3d/f51 307703 0 2024-04-22T15:49:52.451 INFO:tasks.workunit.client.1.smithi027.stdout:0/255: chown d0/d2a/f35 2413665 1 2024-04-22T15:49:52.452 INFO:tasks.workunit.client.1.smithi027.stdout:0/256: stat d0/cb 0 2024-04-22T15:49:52.453 INFO:tasks.workunit.client.1.smithi027.stdout:1/194: dwrite d2/d1d/f1c [0,4194304] 0 2024-04-22T15:49:52.456 INFO:tasks.workunit.client.1.smithi027.stdout:3/328: rmdir d28/d5d/d62 39 2024-04-22T15:49:52.462 INFO:tasks.workunit.client.1.smithi027.stdout:5/313: dwrite d0/f3c [0,4194304] 0 2024-04-22T15:49:52.463 INFO:tasks.workunit.client.1.smithi027.stdout:6/272: symlink d5/dc/l5f 0 2024-04-22T15:49:52.464 INFO:tasks.workunit.client.1.smithi027.stdout:6/273: write d5/db/f11 [3539717,6547] 0 2024-04-22T15:49:52.467 INFO:tasks.workunit.client.1.smithi027.stdout:1/195: creat d2/d30/f41 x:0 0 0 2024-04-22T15:49:52.468 INFO:tasks.workunit.client.1.smithi027.stdout:6/274: write d5/dc/d20/d2b/f27 [1175247,35369] 0 2024-04-22T15:49:52.474 INFO:tasks.workunit.client.1.smithi027.stdout:3/329: mknod d28/d4b/d5c/c6d 0 2024-04-22T15:49:52.475 INFO:tasks.workunit.client.1.smithi027.stdout:3/330: write d28/d4b/d52/f6a [67485,84439] 0 2024-04-22T15:49:52.476 INFO:tasks.workunit.client.1.smithi027.stdout:1/196: symlink d2/d1d/l42 0 2024-04-22T15:49:52.476 INFO:tasks.workunit.client.1.smithi027.stdout:3/331: fsync d28/d4b/d52/f65 0 2024-04-22T15:49:52.477 INFO:tasks.workunit.client.1.smithi027.stdout:6/275: symlink d5/dc/dd/d52/l60 0 2024-04-22T15:49:52.478 INFO:tasks.workunit.client.1.smithi027.stdout:1/197: creat d2/d30/f43 x:0 0 0 2024-04-22T15:49:52.482 INFO:tasks.workunit.client.1.smithi027.stdout:3/332: mkdir d28/d69/d6e 0 2024-04-22T15:49:52.482 INFO:tasks.workunit.client.1.smithi027.stdout:5/314: dread d0/df/f12 [0,4194304] 0 2024-04-22T15:49:52.484 INFO:tasks.workunit.client.1.smithi027.stdout:0/257: link d0/d2a/c50 d0/d2a/c51 0 2024-04-22T15:49:52.486 INFO:tasks.workunit.client.1.smithi027.stdout:0/258: chown d0/d2a/l30 183 1 2024-04-22T15:49:52.486 INFO:tasks.workunit.client.1.smithi027.stdout:6/276: dwrite d5/dc/dd/f16 [0,4194304] 0 2024-04-22T15:49:52.486 INFO:tasks.workunit.client.1.smithi027.stdout:5/315: truncate d0/d1a/d22/d26/d32/f54 1282952 0 2024-04-22T15:49:52.486 INFO:tasks.workunit.client.1.smithi027.stdout:3/333: chown d28/d5d/d62/f66 1 1 2024-04-22T15:49:52.487 INFO:tasks.workunit.client.1.smithi027.stdout:1/198: mkdir d2/d36/d38/d3d/d44 0 2024-04-22T15:49:52.487 INFO:tasks.workunit.client.1.smithi027.stdout:0/259: stat d0/d2a/l2d 0 2024-04-22T15:49:52.495 INFO:tasks.workunit.client.1.smithi027.stdout:5/316: dread d0/df/de/d24/d1c/f34 [0,4194304] 0 2024-04-22T15:49:52.497 INFO:tasks.workunit.client.1.smithi027.stdout:6/277: write d5/dc/d20/d2b/f12 [8569328,46630] 0 2024-04-22T15:49:52.497 INFO:tasks.workunit.client.1.smithi027.stdout:6/278: rename d5 to d5/d30/d61 22 2024-04-22T15:49:52.500 INFO:tasks.workunit.client.1.smithi027.stdout:0/260: mknod d0/d2a/c52 0 2024-04-22T15:49:52.509 INFO:tasks.workunit.client.1.smithi027.stdout:5/317: rename d0/df/de/d24/d1c/d3e/d55/f61 to d0/df/d10/f64 0 2024-04-22T15:49:52.511 INFO:tasks.workunit.client.1.smithi027.stdout:6/279: creat d5/db/d2e/f62 x:0 0 0 2024-04-22T15:49:52.512 INFO:tasks.workunit.client.1.smithi027.stdout:1/199: mknod d2/d36/d38/d3d/d44/c45 0 2024-04-22T15:49:52.512 INFO:tasks.workunit.client.1.smithi027.stdout:3/334: dwrite f5 [0,4194304] 0 2024-04-22T15:49:52.513 INFO:tasks.workunit.client.1.smithi027.stdout:3/335: write f10 [8724256,104595] 0 2024-04-22T15:49:52.517 INFO:tasks.workunit.client.1.smithi027.stdout:0/261: rename d0/d2a/c52 to d0/d2a/d38/c53 0 2024-04-22T15:49:52.520 INFO:tasks.workunit.client.1.smithi027.stdout:5/318: unlink d0/d1a/d22/d26/d32/f37 0 2024-04-22T15:49:52.526 INFO:tasks.workunit.client.1.smithi027.stdout:1/200: dwrite d2/d30/f43 [0,4194304] 0 2024-04-22T15:49:52.526 INFO:tasks.workunit.client.1.smithi027.stdout:3/336: dwrite f1c [0,4194304] 0 2024-04-22T15:49:52.530 INFO:tasks.workunit.client.1.smithi027.stdout:3/337: write d28/f56 [1057329,52870] 0 2024-04-22T15:49:52.530 INFO:tasks.workunit.client.1.smithi027.stdout:6/280: dread d5/db/d2e/ff [0,4194304] 0 2024-04-22T15:49:52.534 INFO:tasks.workunit.client.1.smithi027.stdout:0/262: dwrite d0/f3e [0,4194304] 0 2024-04-22T15:49:52.535 INFO:tasks.workunit.client.1.smithi027.stdout:0/263: truncate d0/f2e 154483 0 2024-04-22T15:49:52.536 INFO:tasks.workunit.client.1.smithi027.stdout:0/264: stat d0/d28/f1c 0 2024-04-22T15:49:52.541 INFO:tasks.workunit.client.1.smithi027.stdout:1/201: symlink d2/d30/l46 0 2024-04-22T15:49:52.546 INFO:tasks.workunit.client.1.smithi027.stdout:6/281: symlink d5/dc/d20/l63 0 2024-04-22T15:49:52.547 INFO:tasks.workunit.client.1.smithi027.stdout:3/338: creat d28/d4b/d5c/f6f x:0 0 0 2024-04-22T15:49:52.550 INFO:tasks.workunit.client.1.smithi027.stdout:1/202: rename d2/f15 to d2/d30/f47 0 2024-04-22T15:49:52.552 INFO:tasks.workunit.client.1.smithi027.stdout:5/319: getdents d0/d1a/d22/d26 0 2024-04-22T15:49:52.553 INFO:tasks.workunit.client.1.smithi027.stdout:5/320: chown d0/df/d10/l42 992639584 1 2024-04-22T15:49:52.555 INFO:tasks.workunit.client.1.smithi027.stdout:3/339: symlink d28/d4b/d5c/l70 0 2024-04-22T15:49:52.557 INFO:tasks.workunit.client.1.smithi027.stdout:1/203: symlink d2/d1d/l48 0 2024-04-22T15:49:52.558 INFO:tasks.workunit.client.1.smithi027.stdout:0/265: link d0/cb d0/d34/d4e/c54 0 2024-04-22T15:49:52.561 INFO:tasks.workunit.client.1.smithi027.stdout:5/321: dwrite d0/df/de/d24/f35 [0,4194304] 0 2024-04-22T15:49:52.561 INFO:tasks.workunit.client.1.smithi027.stdout:6/282: getdents d5/dc/d17/d53 0 2024-04-22T15:49:52.563 INFO:tasks.workunit.client.1.smithi027.stdout:6/283: readlink d5/dc/l5f 0 2024-04-22T15:49:52.570 INFO:tasks.workunit.client.1.smithi027.stdout:6/284: mknod d5/dc/d17/d53/c64 0 2024-04-22T15:49:52.574 INFO:tasks.workunit.client.1.smithi027.stdout:5/322: symlink d0/df/de/d24/d62/l65 0 2024-04-22T15:49:52.576 INFO:tasks.workunit.client.1.smithi027.stdout:5/323: write d0/df/de/f18 [2229518,119247] 0 2024-04-22T15:49:52.577 INFO:tasks.workunit.client.1.smithi027.stdout:3/340: dread f1d [0,4194304] 0 2024-04-22T15:49:52.582 INFO:tasks.workunit.client.1.smithi027.stdout:5/324: rename d0/df/c49 to d0/d1a/d22/d26/d32/c66 0 2024-04-22T15:49:52.583 INFO:tasks.workunit.client.1.smithi027.stdout:0/266: write d0/d28/f1c [609278,16820] 0 2024-04-22T15:49:52.584 INFO:tasks.workunit.client.1.smithi027.stdout:1/204: dread d2/f8 [0,4194304] 0 2024-04-22T15:49:52.585 INFO:tasks.workunit.client.1.smithi027.stdout:0/267: write d0/d2a/f48 [8069,74797] 0 2024-04-22T15:49:52.585 INFO:tasks.workunit.client.1.smithi027.stdout:3/341: creat d28/d5d/f71 x:0 0 0 2024-04-22T15:49:52.586 INFO:tasks.workunit.client.1.smithi027.stdout:1/205: rename d2/d36 to d2/d36/d38/d49 22 2024-04-22T15:49:52.587 INFO:tasks.workunit.client.1.smithi027.stdout:0/268: write d0/d2a/f2b [458337,22704] 0 2024-04-22T15:49:52.587 INFO:tasks.workunit.client.1.smithi027.stdout:5/325: fsync d0/d1a/d22/d26/f3b 0 2024-04-22T15:49:52.588 INFO:tasks.workunit.client.1.smithi027.stdout:0/269: readlink d0/d2a/l2d 0 2024-04-22T15:49:52.594 INFO:tasks.workunit.client.1.smithi027.stdout:0/270: rename d0/f26 to d0/d2a/d38/f55 0 2024-04-22T15:49:52.594 INFO:tasks.workunit.client.1.smithi027.stdout:5/326: dwrite d0/f3 [4194304,4194304] 0 2024-04-22T15:49:52.599 INFO:tasks.workunit.client.1.smithi027.stdout:3/342: rmdir d28/d69/d6e 0 2024-04-22T15:49:52.600 INFO:tasks.workunit.client.1.smithi027.stdout:0/271: chown d0/d2a/c51 1354165 1 2024-04-22T15:49:52.603 INFO:tasks.workunit.client.1.smithi027.stdout:5/327: creat d0/d1a/d22/d26/d32/f67 x:0 0 0 2024-04-22T15:49:52.604 INFO:tasks.workunit.client.1.smithi027.stdout:5/328: dread - d0/df/d10/f60 zero size 2024-04-22T15:49:52.605 INFO:tasks.workunit.client.1.smithi027.stdout:3/343: dwrite d28/d4b/d52/f65 [0,4194304] 0 2024-04-22T15:49:52.607 INFO:tasks.workunit.client.1.smithi027.stdout:0/272: truncate d0/d34/f3a 371967 0 2024-04-22T15:49:52.609 INFO:tasks.workunit.client.1.smithi027.stdout:0/273: write d0/f3e [705883,31051] 0 2024-04-22T15:49:52.610 INFO:tasks.workunit.client.1.smithi027.stdout:1/206: getdents d2/d36 0 2024-04-22T15:49:52.613 INFO:tasks.workunit.client.1.smithi027.stdout:3/344: symlink d28/d5d/l72 0 2024-04-22T15:49:52.614 INFO:tasks.workunit.client.1.smithi027.stdout:5/329: dwrite d0/f3 [4194304,4194304] 0 2024-04-22T15:49:52.616 INFO:tasks.workunit.client.1.smithi027.stdout:1/207: rename d2/c39 to d2/d36/d38/d3d/d44/c4a 0 2024-04-22T15:49:52.618 INFO:tasks.workunit.client.1.smithi027.stdout:1/208: write d2/d1d/f24 [293217,126065] 0 2024-04-22T15:49:52.620 INFO:tasks.workunit.client.1.smithi027.stdout:5/330: creat d0/df/de/d24/d62/f68 x:0 0 0 2024-04-22T15:49:52.622 INFO:tasks.workunit.client.1.smithi027.stdout:5/331: write d0/df/f5e [563791,73582] 0 2024-04-22T15:49:52.627 INFO:tasks.workunit.client.1.smithi027.stdout:5/332: dread d0/f3 [4194304,4194304] 0 2024-04-22T15:49:52.628 INFO:tasks.workunit.client.1.smithi027.stdout:5/333: readlink d0/df/d10/l42 0 2024-04-22T15:49:52.629 INFO:tasks.workunit.client.1.smithi027.stdout:1/209: rename d2/d1d/f24 to d2/d36/f4b 0 2024-04-22T15:49:52.631 INFO:tasks.workunit.client.1.smithi027.stdout:1/210: write d2/d30/f3c [586401,78969] 0 2024-04-22T15:49:52.631 INFO:tasks.workunit.client.1.smithi027.stdout:3/345: mknod d28/d5d/c73 0 2024-04-22T15:49:52.632 INFO:tasks.workunit.client.1.smithi027.stdout:0/274: dread d0/f14 [0,4194304] 0 2024-04-22T15:49:52.632 INFO:tasks.workunit.client.1.smithi027.stdout:1/211: truncate d2/d30/f43 4217988 0 2024-04-22T15:49:52.644 INFO:tasks.workunit.client.1.smithi027.stdout:0/275: chown d0/c7 29003450 1 2024-04-22T15:49:52.645 INFO:tasks.workunit.client.1.smithi027.stdout:3/346: dread d28/f36 [0,4194304] 0 2024-04-22T15:49:52.645 INFO:tasks.workunit.client.1.smithi027.stdout:3/347: readlink l17 0 2024-04-22T15:49:52.652 INFO:tasks.workunit.client.1.smithi027.stdout:1/212: dwrite d2/f18 [4194304,4194304] 0 2024-04-22T15:49:52.657 INFO:tasks.workunit.client.1.smithi027.stdout:0/276: mknod d0/d2a/c56 0 2024-04-22T15:49:52.663 INFO:tasks.workunit.client.1.smithi027.stdout:5/334: getdents d0/d1a 0 2024-04-22T15:49:52.666 INFO:tasks.workunit.client.1.smithi027.stdout:0/277: mknod d0/d28/c57 0 2024-04-22T15:49:52.668 INFO:tasks.workunit.client.1.smithi027.stdout:1/213: unlink d2/l16 0 2024-04-22T15:49:52.671 INFO:tasks.workunit.client.1.smithi027.stdout:5/335: stat d0/df/de/l31 0 2024-04-22T15:49:52.672 INFO:tasks.workunit.client.1.smithi027.stdout:5/336: write d0/df/de/f18 [66048,19233] 0 2024-04-22T15:49:52.672 INFO:tasks.workunit.client.1.smithi027.stdout:1/214: stat d2/c13 0 2024-04-22T15:49:52.673 INFO:tasks.workunit.client.1.smithi027.stdout:3/348: link c1a d28/d69/c74 0 2024-04-22T15:49:52.674 INFO:tasks.workunit.client.1.smithi027.stdout:3/349: stat d28/d5d/l72 0 2024-04-22T15:49:52.674 INFO:tasks.workunit.client.1.smithi027.stdout:3/350: write f11 [4397815,72226] 0 2024-04-22T15:49:52.675 INFO:tasks.workunit.client.1.smithi027.stdout:0/278: mkdir d0/d2a/d58 0 2024-04-22T15:49:52.681 INFO:tasks.workunit.client.1.smithi027.stdout:1/215: dwrite d2/f3 [0,4194304] 0 2024-04-22T15:49:52.681 INFO:tasks.workunit.client.1.smithi027.stdout:3/351: symlink d28/l75 0 2024-04-22T15:49:52.683 INFO:tasks.workunit.client.1.smithi027.stdout:3/352: chown l2 91081101 1 2024-04-22T15:49:52.683 INFO:tasks.workunit.client.1.smithi027.stdout:1/216: readlink d2/d30/l46 0 2024-04-22T15:49:52.683 INFO:tasks.workunit.client.1.smithi027.stdout:0/279: mkdir d0/d34/d59 0 2024-04-22T15:49:52.684 INFO:tasks.workunit.client.1.smithi027.stdout:3/353: write f5 [3219814,33922] 0 2024-04-22T15:49:52.684 INFO:tasks.workunit.client.1.smithi027.stdout:0/280: fdatasync d0/d2a/d38/f4a 0 2024-04-22T15:49:52.685 INFO:tasks.workunit.client.1.smithi027.stdout:1/217: dread d2/fa [0,4194304] 0 2024-04-22T15:49:52.685 INFO:tasks.workunit.client.1.smithi027.stdout:0/281: fsync d0/d2a/d38/f4a 0 2024-04-22T15:49:52.688 INFO:tasks.workunit.client.1.smithi027.stdout:3/354: mknod d28/d5d/d62/c76 0 2024-04-22T15:49:52.689 INFO:tasks.workunit.client.1.smithi027.stdout:0/282: creat d0/d34/f5a x:0 0 0 2024-04-22T15:49:52.689 INFO:tasks.workunit.client.1.smithi027.stdout:3/355: write d28/d4b/f4c [128778,104829] 0 2024-04-22T15:49:52.690 INFO:tasks.workunit.client.1.smithi027.stdout:0/283: dread - d0/d34/f5a zero size 2024-04-22T15:49:52.691 INFO:tasks.workunit.client.1.smithi027.stdout:5/337: getdents d0/d1a/d22/d26/d32 0 2024-04-22T15:49:52.692 INFO:tasks.workunit.client.1.smithi027.stdout:3/356: write d28/d4b/d52/f61 [330310,38244] 0 2024-04-22T15:49:52.693 INFO:tasks.workunit.client.1.smithi027.stdout:1/218: write d2/d36/d38/f40 [218213,58285] 0 2024-04-22T15:49:52.693 INFO:tasks.workunit.client.1.smithi027.stdout:0/284: write d0/d2a/f48 [1037822,66604] 0 2024-04-22T15:49:52.696 INFO:tasks.workunit.client.1.smithi027.stdout:0/285: truncate d0/d34/f44 725936 0 2024-04-22T15:49:52.698 INFO:tasks.workunit.client.1.smithi027.stdout:3/357: read f10 [535490,66992] 0 2024-04-22T15:49:52.699 INFO:tasks.workunit.client.1.smithi027.stdout:0/286: chown d0/d34/f5a 59 1 2024-04-22T15:49:52.700 INFO:tasks.workunit.client.1.smithi027.stdout:8/288: sync 2024-04-22T15:49:52.701 INFO:tasks.workunit.client.1.smithi027.stdout:7/235: sync 2024-04-22T15:49:52.701 INFO:tasks.workunit.client.1.smithi027.stdout:2/260: sync 2024-04-22T15:49:52.702 INFO:tasks.workunit.client.1.smithi027.stdout:7/236: dread - d0/d11/d1c/d2c/f33 zero size 2024-04-22T15:49:52.702 INFO:tasks.workunit.client.1.smithi027.stdout:5/338: dwrite d0/d1a/d22/d26/f48 [0,4194304] 0 2024-04-22T15:49:52.705 INFO:tasks.workunit.client.1.smithi027.stdout:3/358: creat d28/d69/f77 x:0 0 0 2024-04-22T15:49:52.707 INFO:tasks.workunit.client.1.smithi027.stdout:3/359: read f16 [266343,65666] 0 2024-04-22T15:49:52.711 INFO:tasks.workunit.client.1.smithi027.stdout:0/287: creat d0/d28/f5b x:0 0 0 2024-04-22T15:49:52.711 INFO:tasks.workunit.client.1.smithi027.stdout:5/339: dread d0/d1a/d22/d26/d32/f54 [0,4194304] 0 2024-04-22T15:49:52.712 INFO:tasks.workunit.client.1.smithi027.stdout:2/261: chown dd/f19 36919594 1 2024-04-22T15:49:52.716 INFO:tasks.workunit.client.1.smithi027.stdout:8/289: rmdir d1/d3 39 2024-04-22T15:49:52.716 INFO:tasks.workunit.client.1.smithi027.stdout:8/290: fsync d1/f7 0 2024-04-22T15:49:52.722 INFO:tasks.workunit.client.1.smithi027.stdout:3/360: dread d28/f37 [0,4194304] 0 2024-04-22T15:49:52.723 INFO:tasks.workunit.client.1.smithi027.stdout:3/361: readlink d28/l30 0 2024-04-22T15:49:52.724 INFO:tasks.workunit.client.1.smithi027.stdout:0/288: write d0/f14 [8277438,105635] 0 2024-04-22T15:49:52.726 INFO:tasks.workunit.client.1.smithi027.stdout:5/340: creat d0/df/d3d/f69 x:0 0 0 2024-04-22T15:49:52.726 INFO:tasks.workunit.client.1.smithi027.stdout:2/262: chown dd/df/d1b 1 1 2024-04-22T15:49:52.727 INFO:tasks.workunit.client.1.smithi027.stdout:2/263: chown dd/d1d/c39 1227474590 1 2024-04-22T15:49:52.731 INFO:tasks.workunit.client.1.smithi027.stdout:7/237: fdatasync d0/d11/fe 0 2024-04-22T15:49:52.731 INFO:tasks.workunit.client.1.smithi027.stdout:9/243: sync 2024-04-22T15:49:52.732 INFO:tasks.workunit.client.1.smithi027.stdout:4/288: sync 2024-04-22T15:49:52.732 INFO:tasks.workunit.client.1.smithi027.stdout:6/285: sync 2024-04-22T15:49:52.732 INFO:tasks.workunit.client.1.smithi027.stdout:6/286: fsync d5/db/f11 0 2024-04-22T15:49:52.732 INFO:tasks.workunit.client.1.smithi027.stdout:4/289: fsync d5/d45/d47/f4a 0 2024-04-22T15:49:52.734 INFO:tasks.workunit.client.1.smithi027.stdout:3/362: creat d28/d69/f78 x:0 0 0 2024-04-22T15:49:52.735 INFO:tasks.workunit.client.1.smithi027.stdout:3/363: chown l25 102 1 2024-04-22T15:49:52.736 INFO:tasks.workunit.client.1.smithi027.stdout:5/341: creat d0/df/de/d40/f6a x:0 0 0 2024-04-22T15:49:52.738 INFO:tasks.workunit.client.1.smithi027.stdout:5/342: fsync d0/df/f56 0 2024-04-22T15:49:52.741 INFO:tasks.workunit.client.1.smithi027.stdout:1/219: getdents d2/d36/d38 0 2024-04-22T15:49:52.742 INFO:tasks.workunit.client.1.smithi027.stdout:8/291: read d1/d3/f6 [2202981,2636] 0 2024-04-22T15:49:52.743 INFO:tasks.workunit.client.1.smithi027.stdout:2/264: dread dd/df/d1b/f25 [0,4194304] 0 2024-04-22T15:49:52.744 INFO:tasks.workunit.client.1.smithi027.stdout:3/364: dwrite d28/d4b/d52/f6a [0,4194304] 0 2024-04-22T15:49:52.748 INFO:tasks.workunit.client.1.smithi027.stdout:0/289: creat d0/d34/d59/f5c x:0 0 0 2024-04-22T15:49:52.749 INFO:tasks.workunit.client.1.smithi027.stdout:0/290: write d0/d2a/f35 [149999,127372] 0 2024-04-22T15:49:52.760 INFO:tasks.workunit.client.1.smithi027.stdout:1/220: rmdir d2/d1d 39 2024-04-22T15:49:52.760 INFO:tasks.workunit.client.1.smithi027.stdout:5/343: fsync d0/df/d10/f64 0 2024-04-22T15:49:52.762 INFO:tasks.workunit.client.1.smithi027.stdout:5/344: fsync d0/f3c 0 2024-04-22T15:49:52.771 INFO:tasks.workunit.client.1.smithi027.stdout:1/221: dread d2/d30/f3c [0,4194304] 0 2024-04-22T15:49:52.773 INFO:tasks.workunit.client.1.smithi027.stdout:9/244: dwrite d0/dc/d12/d30/f32 [0,4194304] 0 2024-04-22T15:49:52.775 INFO:tasks.workunit.client.1.smithi027.stdout:4/290: rename d5/d7/c1b to d5/d11/d1d/c73 0 2024-04-22T15:49:52.778 INFO:tasks.workunit.client.1.smithi027.stdout:6/287: link d5/dc/d26/d3d/f51 d5/dc/d20/d2b/d34/f65 0 2024-04-22T15:49:52.782 INFO:tasks.workunit.client.1.smithi027.stdout:9/245: dwrite d0/dc/de/f3a [0,4194304] 0 2024-04-22T15:49:52.785 INFO:tasks.workunit.client.1.smithi027.stdout:3/365: mkdir d28/d5d/d79 0 2024-04-22T15:49:52.787 INFO:tasks.workunit.client.1.smithi027.stdout:6/288: dwrite d5/db/d2e/f62 [0,4194304] 0 2024-04-22T15:49:52.789 INFO:tasks.workunit.client.1.smithi027.stdout:2/265: mknod dd/d1d/d4b/d4e/c4f 0 2024-04-22T15:49:52.791 INFO:tasks.workunit.client.1.smithi027.stdout:2/266: fsync dd/df/f23 0 2024-04-22T15:49:52.794 INFO:tasks.workunit.client.1.smithi027.stdout:0/291: unlink d0/d2a/d38/c53 0 2024-04-22T15:49:52.794 INFO:tasks.workunit.client.1.smithi027.stdout:3/366: dwrite d28/f2a [0,4194304] 0 2024-04-22T15:49:52.796 INFO:tasks.workunit.client.1.smithi027.stdout:1/222: rmdir d2/d30 39 2024-04-22T15:49:52.798 INFO:tasks.workunit.client.1.smithi027.stdout:7/238: truncate d0/d11/d1c/d2c/f40 2309714 0 2024-04-22T15:49:52.800 INFO:tasks.workunit.client.1.smithi027.stdout:4/291: symlink d5/de/d30/d5d/l74 0 2024-04-22T15:49:52.801 INFO:tasks.workunit.client.1.smithi027.stdout:2/267: dwrite dd/df/f18 [0,4194304] 0 2024-04-22T15:49:52.804 INFO:tasks.workunit.client.1.smithi027.stdout:2/268: truncate dd/df/d1b/f29 529800 0 2024-04-22T15:49:52.806 INFO:tasks.workunit.client.1.smithi027.stdout:2/269: truncate dd/f11 4934122 0 2024-04-22T15:49:52.810 INFO:tasks.workunit.client.1.smithi027.stdout:9/246: symlink d0/dc/df/l56 0 2024-04-22T15:49:52.813 INFO:tasks.workunit.client.1.smithi027.stdout:5/345: mknod d0/d1a/d22/d26/d5d/c6b 0 2024-04-22T15:49:52.813 INFO:tasks.workunit.client.1.smithi027.stdout:5/346: stat d0/df/de/d24/f35 0 2024-04-22T15:49:52.816 INFO:tasks.workunit.client.1.smithi027.stdout:6/289: symlink d5/dc/d20/l66 0 2024-04-22T15:49:52.817 INFO:tasks.workunit.client.1.smithi027.stdout:6/290: truncate d5/dc/d26/d3d/f51 4466775 0 2024-04-22T15:49:52.820 INFO:tasks.workunit.client.1.smithi027.stdout:0/292: unlink d0/d28/c2f 0 2024-04-22T15:49:52.821 INFO:tasks.workunit.client.1.smithi027.stdout:3/367: creat d28/d4b/d52/f7a x:0 0 0 2024-04-22T15:49:52.821 INFO:tasks.workunit.client.1.smithi027.stdout:0/293: write d0/d2a/f35 [705427,16400] 0 2024-04-22T15:49:52.822 INFO:tasks.workunit.client.1.smithi027.stdout:3/368: write d28/f3b [692360,1583] 0 2024-04-22T15:49:52.823 INFO:tasks.workunit.client.1.smithi027.stdout:3/369: fdatasync d28/d5d/f71 0 2024-04-22T15:49:52.827 INFO:tasks.workunit.client.1.smithi027.stdout:7/239: rmdir d0/d11/d1c/d2c 39 2024-04-22T15:49:52.828 INFO:tasks.workunit.client.1.smithi027.stdout:0/294: dwrite d0/d34/f42 [0,4194304] 0 2024-04-22T15:49:52.830 INFO:tasks.workunit.client.1.smithi027.stdout:4/292: mkdir d5/d11/d75 0 2024-04-22T15:49:52.831 INFO:tasks.workunit.client.1.smithi027.stdout:0/295: read d0/d34/f42 [2262316,73441] 0 2024-04-22T15:49:52.834 INFO:tasks.workunit.client.1.smithi027.stdout:8/292: getdents d1/d3/d1c/d43 0 2024-04-22T15:49:52.838 INFO:tasks.workunit.client.1.smithi027.stdout:2/270: symlink dd/df/d1b/d38/l50 0 2024-04-22T15:49:52.840 INFO:tasks.workunit.client.1.smithi027.stdout:0/296: dwrite d0/d34/f42 [4194304,4194304] 0 2024-04-22T15:49:52.842 INFO:tasks.workunit.client.1.smithi027.stdout:2/271: fdatasync dd/df/d1b/d1f/f4a 0 2024-04-22T15:49:52.844 INFO:tasks.workunit.client.1.smithi027.stdout:5/347: mknod d0/d1a/d22/d26/d5d/c6c 0 2024-04-22T15:49:52.844 INFO:tasks.workunit.client.1.smithi027.stdout:0/297: read d0/d28/f1f [381260,90118] 0 2024-04-22T15:49:52.847 INFO:tasks.workunit.client.1.smithi027.stdout:2/272: truncate dd/d47/f48 474317 0 2024-04-22T15:49:52.847 INFO:tasks.workunit.client.1.smithi027.stdout:8/293: dwrite d1/d3/d1c/d49/f5d [0,4194304] 0 2024-04-22T15:49:52.848 INFO:tasks.workunit.client.1.smithi027.stdout:5/348: truncate d0/df/d3d/f44 492002 0 2024-04-22T15:49:52.856 INFO:tasks.workunit.client.1.smithi027.stdout:3/370: unlink d28/d5d/f71 0 2024-04-22T15:49:52.858 INFO:tasks.workunit.client.1.smithi027.stdout:7/240: rmdir d0/d11/d1c 39 2024-04-22T15:49:52.862 INFO:tasks.workunit.client.1.smithi027.stdout:4/293: creat d5/d45/d47/f76 x:0 0 0 2024-04-22T15:49:52.865 INFO:tasks.workunit.client.1.smithi027.stdout:7/241: dwrite d0/d16/f3b [0,4194304] 0 2024-04-22T15:49:52.867 INFO:tasks.workunit.client.1.smithi027.stdout:9/247: mknod d0/d8/c57 0 2024-04-22T15:49:52.870 INFO:tasks.workunit.client.1.smithi027.stdout:0/298: creat d0/d34/d59/f5d x:0 0 0 2024-04-22T15:49:52.871 INFO:tasks.workunit.client.1.smithi027.stdout:0/299: write d0/d2a/f35 [1114305,83286] 0 2024-04-22T15:49:52.873 INFO:tasks.workunit.client.1.smithi027.stdout:6/291: creat d5/dc/d17/d48/d5d/f67 x:0 0 0 2024-04-22T15:49:52.877 INFO:tasks.workunit.client.1.smithi027.stdout:2/273: rename dd/df/d1b/f29 to dd/d1d/d4b/d4e/f51 0 2024-04-22T15:49:52.877 INFO:tasks.workunit.client.1.smithi027.stdout:2/274: stat dd/d47/f48 0 2024-04-22T15:49:52.878 INFO:tasks.workunit.client.1.smithi027.stdout:2/275: readlink dd/l42 0 2024-04-22T15:49:52.880 INFO:tasks.workunit.client.1.smithi027.stdout:8/294: creat d1/d3/d1c/d56/f6a x:0 0 0 2024-04-22T15:49:52.881 INFO:tasks.workunit.client.1.smithi027.stdout:8/295: fsync d1/d3/d1c/d56/d62/f68 0 2024-04-22T15:49:52.883 INFO:tasks.workunit.client.1.smithi027.stdout:3/371: rename d28/f2a to d28/f7b 0 2024-04-22T15:49:52.887 INFO:tasks.workunit.client.1.smithi027.stdout:3/372: dread f1d [4194304,4194304] 0 2024-04-22T15:49:52.891 INFO:tasks.workunit.client.1.smithi027.stdout:7/242: creat d0/d11/d3a/f45 x:0 0 0 2024-04-22T15:49:52.891 INFO:tasks.workunit.client.1.smithi027.stdout:2/276: dread dd/df/f17 [0,4194304] 0 2024-04-22T15:49:52.892 INFO:tasks.workunit.client.1.smithi027.stdout:7/243: chown d0/d11/d3a 7123399 1 2024-04-22T15:49:52.893 INFO:tasks.workunit.client.1.smithi027.stdout:9/248: mkdir d0/dc/d12/d58 0 2024-04-22T15:49:52.899 INFO:tasks.workunit.client.1.smithi027.stdout:6/292: symlink d5/dc/d20/d2b/d34/l68 0 2024-04-22T15:49:52.905 INFO:tasks.workunit.client.1.smithi027.stdout:1/223: getdents d2/d36/d38/d3d 0 2024-04-22T15:49:52.907 INFO:tasks.workunit.client.1.smithi027.stdout:3/373: write d28/f45 [908013,42553] 0 2024-04-22T15:49:52.909 INFO:tasks.workunit.client.1.smithi027.stdout:4/294: mknod d5/c77 0 2024-04-22T15:49:52.910 INFO:tasks.workunit.client.1.smithi027.stdout:4/295: write d5/de/d2a/d58/f68 [834428,13955] 0 2024-04-22T15:49:52.912 INFO:tasks.workunit.client.1.smithi027.stdout:2/277: mkdir dd/d1d/d4b/d52 0 2024-04-22T15:49:52.919 INFO:tasks.workunit.client.1.smithi027.stdout:6/293: chown d5/dc/d17/c1c 389 1 2024-04-22T15:49:52.920 INFO:tasks.workunit.client.1.smithi027.stdout:6/294: chown d5/dc/d17/d53/d5c 572 1 2024-04-22T15:49:52.920 INFO:tasks.workunit.client.1.smithi027.stdout:1/224: rmdir d2/d36/d38/d3d 39 2024-04-22T15:49:52.923 INFO:tasks.workunit.client.1.smithi027.stdout:3/374: rmdir d28/d69 39 2024-04-22T15:49:52.927 INFO:tasks.workunit.client.1.smithi027.stdout:2/278: mknod dd/d43/c53 0 2024-04-22T15:49:52.929 INFO:tasks.workunit.client.1.smithi027.stdout:7/244: rmdir d0/d11/d1c/d31 39 2024-04-22T15:49:52.931 INFO:tasks.workunit.client.1.smithi027.stdout:9/249: write d0/dc/de/f49 [997523,108403] 0 2024-04-22T15:49:52.934 INFO:tasks.workunit.client.1.smithi027.stdout:6/295: creat d5/dc/dd/f69 x:0 0 0 2024-04-22T15:49:52.940 INFO:tasks.workunit.client.1.smithi027.stdout:4/296: rename d5/de/l1e to d5/d42/l78 0 2024-04-22T15:49:52.943 INFO:tasks.workunit.client.1.smithi027.stdout:7/245: mkdir d0/d11/d3a/d46 0 2024-04-22T15:49:52.943 INFO:tasks.workunit.client.1.smithi027.stdout:7/246: chown d0/d11/c2b 60980 1 2024-04-22T15:49:52.944 INFO:tasks.workunit.client.1.smithi027.stdout:0/300: getdents d0/d28/d37 0 2024-04-22T15:49:52.947 INFO:tasks.workunit.client.1.smithi027.stdout:9/250: rename d0/dc/f29 to d0/dc/d12/d58/f59 0 2024-04-22T15:49:52.951 INFO:tasks.workunit.client.1.smithi027.stdout:1/225: dwrite d2/d36/f4b [0,4194304] 0 2024-04-22T15:49:52.965 INFO:tasks.workunit.client.1.smithi027.stdout:6/296: rename d5/dc/d17/d53/d5c to d5/d30/d54/d6a 0 2024-04-22T15:49:52.966 INFO:tasks.workunit.client.1.smithi027.stdout:6/297: chown d5/dc/d17/d48/d5d 32184038 1 2024-04-22T15:49:52.973 INFO:tasks.workunit.client.1.smithi027.stdout:6/298: dwrite d5/dc/d17/f39 [0,4194304] 0 2024-04-22T15:49:52.973 INFO:tasks.workunit.client.1.smithi027.stdout:0/301: mkdir d0/d2a/d5e 0 2024-04-22T15:49:52.976 INFO:tasks.workunit.client.1.smithi027.stdout:1/226: truncate d2/d30/f41 435435 0 2024-04-22T15:49:52.984 INFO:tasks.workunit.client.1.smithi027.stdout:7/247: rename d0/fd to d0/d11/d1c/d31/f47 0 2024-04-22T15:49:52.987 INFO:tasks.workunit.client.1.smithi027.stdout:2/279: getdents dd/d43 0 2024-04-22T15:49:52.987 INFO:tasks.workunit.client.1.smithi027.stdout:2/280: fdatasync dd/f35 0 2024-04-22T15:49:52.988 INFO:tasks.workunit.client.1.smithi027.stdout:2/281: truncate dd/d47/f48 1089095 0 2024-04-22T15:49:52.991 INFO:tasks.workunit.client.1.smithi027.stdout:4/297: truncate d5/d7/f1f 738528 0 2024-04-22T15:49:52.994 INFO:tasks.workunit.client.1.smithi027.stdout:0/302: mkdir d0/d28/d5f 0 2024-04-22T15:49:52.995 INFO:tasks.workunit.client.1.smithi027.stdout:0/303: stat d0/d28/d37/l46 0 2024-04-22T15:49:52.996 INFO:tasks.workunit.client.1.smithi027.stdout:2/282: dwrite dd/df/d1b/d1f/f4a [0,4194304] 0 2024-04-22T15:49:52.998 INFO:tasks.workunit.client.1.smithi027.stdout:3/375: getdents d28/d69 0 2024-04-22T15:49:53.004 INFO:tasks.workunit.client.1.smithi027.stdout:3/376: dread d28/d4b/f4c [0,4194304] 0 2024-04-22T15:49:53.004 INFO:tasks.workunit.client.1.smithi027.stdout:3/377: dread - d28/d4b/f53 zero size 2024-04-22T15:49:53.005 INFO:tasks.workunit.client.1.smithi027.stdout:9/251: write d0/d8/f11 [1433716,12770] 0 2024-04-22T15:49:53.012 INFO:tasks.workunit.client.1.smithi027.stdout:3/378: dwrite d28/d4b/d52/f54 [0,4194304] 0 2024-04-22T15:49:53.013 INFO:tasks.workunit.client.1.smithi027.stdout:3/379: readlink d28/l75 0 2024-04-22T15:49:53.014 INFO:tasks.workunit.client.1.smithi027.stdout:1/227: mknod d2/c4c 0 2024-04-22T15:49:53.015 INFO:tasks.workunit.client.1.smithi027.stdout:9/252: dread d0/dc/de/f24 [0,4194304] 0 2024-04-22T15:49:53.021 INFO:tasks.workunit.client.1.smithi027.stdout:7/248: truncate d0/f37 3736812 0 2024-04-22T15:49:53.022 INFO:tasks.workunit.client.1.smithi027.stdout:7/249: write d0/d26/f42 [1008330,114072] 0 2024-04-22T15:49:53.027 INFO:tasks.workunit.client.1.smithi027.stdout:3/380: creat d28/d4b/f7c x:0 0 0 2024-04-22T15:49:53.032 INFO:tasks.workunit.client.1.smithi027.stdout:9/253: rmdir d0/dc/d12 39 2024-04-22T15:49:53.039 INFO:tasks.workunit.client.1.smithi027.stdout:4/298: creat d5/d11/d1d/d24/f79 x:0 0 0 2024-04-22T15:49:53.041 INFO:tasks.workunit.client.1.smithi027.stdout:6/299: link d5/db/l14 d5/dc/d17/d48/d5d/l6b 0 2024-04-22T15:49:53.042 INFO:tasks.workunit.client.1.smithi027.stdout:6/300: write d5/dc/d17/f39 [2478836,126222] 0 2024-04-22T15:49:53.043 INFO:tasks.workunit.client.1.smithi027.stdout:6/301: stat d5/db/f1e 0 2024-04-22T15:49:53.043 INFO:tasks.workunit.client.1.smithi027.stdout:6/302: fsync d5/db/d2e/f62 0 2024-04-22T15:49:53.045 INFO:tasks.workunit.client.1.smithi027.stdout:0/304: truncate d0/f14 8189342 0 2024-04-22T15:49:53.047 INFO:tasks.workunit.client.1.smithi027.stdout:2/283: truncate dd/df/d1b/f25 2337933 0 2024-04-22T15:49:53.048 INFO:tasks.workunit.client.1.smithi027.stdout:2/284: chown dd/df/d1b/d1f/f3e 0 1 2024-04-22T15:49:53.049 INFO:tasks.workunit.client.1.smithi027.stdout:4/299: creat d5/d11/f7a x:0 0 0 2024-04-22T15:49:53.049 INFO:tasks.workunit.client.1.smithi027.stdout:4/300: dread - d5/d45/d47/f62 zero size 2024-04-22T15:49:53.056 INFO:tasks.workunit.client.1.smithi027.stdout:0/305: mkdir d0/d28/d60 0 2024-04-22T15:49:53.058 INFO:tasks.workunit.client.1.smithi027.stdout:7/250: fdatasync d0/d11/d1c/d2c/f40 0 2024-04-22T15:49:53.065 INFO:tasks.workunit.client.1.smithi027.stdout:4/301: mkdir d5/d11/d1d/d7b 0 2024-04-22T15:49:53.066 INFO:tasks.workunit.client.1.smithi027.stdout:3/381: link d28/d4b/f5b d28/d4b/f7d 0 2024-04-22T15:49:53.069 INFO:tasks.workunit.client.1.smithi027.stdout:0/306: unlink d0/d2a/c56 0 2024-04-22T15:49:53.074 INFO:tasks.workunit.client.1.smithi027.stdout:2/285: mknod dd/c54 0 2024-04-22T15:49:53.076 INFO:tasks.workunit.client.1.smithi027.stdout:4/302: mknod d5/de/d2f/c7c 0 2024-04-22T15:49:53.077 INFO:tasks.workunit.client.1.smithi027.stdout:4/303: chown d5/d11/d1d/d24/f51 1746695 1 2024-04-22T15:49:53.079 INFO:tasks.workunit.client.1.smithi027.stdout:7/251: creat d0/d3f/f48 x:0 0 0 2024-04-22T15:49:53.080 INFO:tasks.workunit.client.1.smithi027.stdout:6/303: truncate d5/dc/d26/d3d/f51 3110343 0 2024-04-22T15:49:53.081 INFO:tasks.workunit.client.1.smithi027.stdout:2/286: creat dd/d43/f55 x:0 0 0 2024-04-22T15:49:53.082 INFO:tasks.workunit.client.1.smithi027.stdout:3/382: creat d28/d5d/d79/f7e x:0 0 0 2024-04-22T15:49:53.085 INFO:tasks.workunit.client.1.smithi027.stdout:7/252: fdatasync d0/d11/fe 0 2024-04-22T15:49:53.086 INFO:tasks.workunit.client.1.smithi027.stdout:7/253: chown d0/d11/d3a/f45 17 1 2024-04-22T15:49:53.090 INFO:tasks.workunit.client.1.smithi027.stdout:6/304: creat d5/dc/d17/d48/d5d/f6c x:0 0 0 2024-04-22T15:49:53.091 INFO:tasks.workunit.client.1.smithi027.stdout:2/287: creat dd/d1d/d4b/d4e/f56 x:0 0 0 2024-04-22T15:49:53.091 INFO:tasks.workunit.client.1.smithi027.stdout:6/305: dread d5/dc/d17/f4b [0,4194304] 0 2024-04-22T15:49:53.097 INFO:tasks.workunit.client.1.smithi027.stdout:2/288: dwrite dd/d1d/f45 [0,4194304] 0 2024-04-22T15:49:53.107 INFO:tasks.workunit.client.1.smithi027.stdout:6/306: mknod d5/dc/d17/c6d 0 2024-04-22T15:49:53.108 INFO:tasks.workunit.client.1.smithi027.stdout:4/304: link d5/d7/f10 d5/d45/f7d 0 2024-04-22T15:49:53.109 INFO:tasks.workunit.client.1.smithi027.stdout:6/307: write d5/dc/d20/f3f [540328,19587] 0 2024-04-22T15:49:53.113 INFO:tasks.workunit.client.1.smithi027.stdout:2/289: symlink dd/d1d/d4b/l57 0 2024-04-22T15:49:53.114 INFO:tasks.workunit.client.1.smithi027.stdout:3/383: dwrite f16 [4194304,4194304] 0 2024-04-22T15:49:53.115 INFO:tasks.workunit.client.1.smithi027.stdout:2/290: readlink dd/df/d1b/l3f 0 2024-04-22T15:49:53.116 INFO:tasks.workunit.client.1.smithi027.stdout:3/384: chown d28/d4b/f4c 59344611 1 2024-04-22T15:49:53.116 INFO:tasks.workunit.client.1.smithi027.stdout:7/254: fdatasync d0/f29 0 2024-04-22T15:49:53.117 INFO:tasks.workunit.client.1.smithi027.stdout:0/307: dread d0/f14 [0,4194304] 0 2024-04-22T15:49:53.117 INFO:tasks.workunit.client.1.smithi027.stdout:7/255: chown d0/l20 102 1 2024-04-22T15:49:53.118 INFO:tasks.workunit.client.1.smithi027.stdout:0/308: fsync d0/f3e 0 2024-04-22T15:49:53.119 INFO:tasks.workunit.client.1.smithi027.stdout:7/256: fdatasync d0/d16/f23 0 2024-04-22T15:49:53.120 INFO:tasks.workunit.client.1.smithi027.stdout:0/309: dread - d0/d34/f5a zero size 2024-04-22T15:49:53.122 INFO:tasks.workunit.client.1.smithi027.stdout:4/305: symlink d5/de/d2f/l7e 0 2024-04-22T15:49:53.125 INFO:tasks.workunit.client.1.smithi027.stdout:6/308: rename d5/d30/d40/l57 to d5/d30/d54/l6e 0 2024-04-22T15:49:53.125 INFO:tasks.workunit.client.1.smithi027.stdout:6/309: dread - d5/dc/d17/d48/d5d/f67 zero size 2024-04-22T15:49:53.132 INFO:tasks.workunit.client.1.smithi027.stdout:3/385: dread d28/d4b/f5b [0,4194304] 0 2024-04-22T15:49:53.135 INFO:tasks.workunit.client.1.smithi027.stdout:4/306: mknod d5/d11/c7f 0 2024-04-22T15:49:53.137 INFO:tasks.workunit.client.1.smithi027.stdout:2/291: unlink dd/df/d1b/l27 0 2024-04-22T15:49:53.137 INFO:tasks.workunit.client.1.smithi027.stdout:2/292: readlink dd/df/l3d 0 2024-04-22T15:49:53.141 INFO:tasks.workunit.client.1.smithi027.stdout:7/257: mkdir d0/d49 0 2024-04-22T15:49:53.143 INFO:tasks.workunit.client.1.smithi027.stdout:4/307: creat d5/d11/f80 x:0 0 0 2024-04-22T15:49:53.144 INFO:tasks.workunit.client.1.smithi027.stdout:2/293: unlink dd/f33 0 2024-04-22T15:49:53.145 INFO:tasks.workunit.client.1.smithi027.stdout:2/294: truncate dd/d43/f44 285270 0 2024-04-22T15:49:53.145 INFO:tasks.workunit.client.1.smithi027.stdout:6/310: symlink d5/l6f 0 2024-04-22T15:49:53.147 INFO:tasks.workunit.client.1.smithi027.stdout:2/295: chown dd/d1d/d4b/d4e/f51 7 1 2024-04-22T15:49:53.147 INFO:tasks.workunit.client.1.smithi027.stdout:0/310: creat d0/d2a/f61 x:0 0 0 2024-04-22T15:49:53.149 INFO:tasks.workunit.client.1.smithi027.stdout:2/296: write dd/df/d1b/d1f/f4a [1072626,18414] 0 2024-04-22T15:49:53.154 INFO:tasks.workunit.client.1.smithi027.stdout:6/311: creat d5/dc/d17/d48/d5d/f70 x:0 0 0 2024-04-22T15:49:53.155 INFO:tasks.workunit.client.1.smithi027.stdout:0/311: dwrite d0/d34/d59/f5d [0,4194304] 0 2024-04-22T15:49:53.160 INFO:tasks.workunit.client.1.smithi027.stdout:7/258: link d0/d3f/f48 d0/d16/f4a 0 2024-04-22T15:49:53.165 INFO:tasks.workunit.client.1.smithi027.stdout:3/386: write d28/f38 [238259,27184] 0 2024-04-22T15:49:53.166 INFO:tasks.workunit.client.1.smithi027.stdout:3/387: fdatasync d28/d4b/d5c/f6f 0 2024-04-22T15:49:53.168 INFO:tasks.workunit.client.1.smithi027.stdout:6/312: mknod d5/d30/d54/d6a/c71 0 2024-04-22T15:49:53.170 INFO:tasks.workunit.client.1.smithi027.stdout:0/312: write d0/d34/f42 [253313,39214] 0 2024-04-22T15:49:53.171 INFO:tasks.workunit.client.1.smithi027.stdout:0/313: chown d0/d34/d59/f5c 0 1 2024-04-22T15:49:53.174 INFO:tasks.workunit.client.1.smithi027.stdout:3/388: dwrite d28/d4b/d52/f6a [0,4194304] 0 2024-04-22T15:49:53.177 INFO:tasks.workunit.client.1.smithi027.stdout:2/297: link dd/f19 dd/d1d/d4b/f58 0 2024-04-22T15:49:53.178 INFO:tasks.workunit.client.1.smithi027.stdout:2/298: stat dd/df/d1b/f2e 0 2024-04-22T15:49:53.178 INFO:tasks.workunit.client.1.smithi027.stdout:0/314: dread d0/d2a/f3c [0,4194304] 0 2024-04-22T15:49:53.178 INFO:tasks.workunit.client.1.smithi027.stdout:4/308: rmdir d5/d11/d75 0 2024-04-22T15:49:53.179 INFO:tasks.workunit.client.1.smithi027.stdout:2/299: fdatasync dd/d47/f48 0 2024-04-22T15:49:53.184 INFO:tasks.workunit.client.1.smithi027.stdout:2/300: dread dd/df/f17 [4194304,4194304] 0 2024-04-22T15:49:53.186 INFO:tasks.workunit.client.1.smithi027.stdout:3/389: mknod d28/d69/c7f 0 2024-04-22T15:49:53.186 INFO:tasks.workunit.client.1.smithi027.stdout:3/390: stat d28/f43 0 2024-04-22T15:49:53.192 INFO:tasks.workunit.client.1.smithi027.stdout:4/309: mkdir d5/d45/d81 0 2024-04-22T15:49:53.193 INFO:tasks.workunit.client.1.smithi027.stdout:4/310: readlink d5/d11/d1d/l6f 0 2024-04-22T15:49:53.197 INFO:tasks.workunit.client.1.smithi027.stdout:7/259: creat d0/f4b x:0 0 0 2024-04-22T15:49:53.201 INFO:tasks.workunit.client.1.smithi027.stdout:0/315: write d0/f22 [1148084,93163] 0 2024-04-22T15:49:53.208 INFO:tasks.workunit.client.1.smithi027.stdout:7/260: mkdir d0/d4c 0 2024-04-22T15:49:53.209 INFO:tasks.workunit.client.1.smithi027.stdout:2/301: rename dd/df/f3c to dd/df/f59 0 2024-04-22T15:49:53.209 INFO:tasks.workunit.client.1.smithi027.stdout:2/302: readlink dd/l42 0 2024-04-22T15:49:53.212 INFO:tasks.workunit.client.1.smithi027.stdout:3/391: rename f21 to d28/d5d/f80 0 2024-04-22T15:49:53.212 INFO:tasks.workunit.client.1.smithi027.stdout:0/316: getdents d0/d2a/d5e 0 2024-04-22T15:49:53.217 INFO:tasks.workunit.client.1.smithi027.stdout:3/392: mknod d28/d4b/c81 0 2024-04-22T15:49:53.218 INFO:tasks.workunit.client.1.smithi027.stdout:4/311: getdents d5/d11/d6c 0 2024-04-22T15:49:53.219 INFO:tasks.workunit.client.1.smithi027.stdout:0/317: dwrite d0/d2a/f2b [0,4194304] 0 2024-04-22T15:49:53.220 INFO:tasks.workunit.client.1.smithi027.stdout:0/318: chown d0/d2a/d38/f55 277871 1 2024-04-22T15:49:53.225 INFO:tasks.workunit.client.1.smithi027.stdout:3/393: creat d28/d69/f82 x:0 0 0 2024-04-22T15:49:53.226 INFO:tasks.workunit.client.1.smithi027.stdout:3/394: dread - d28/d5d/d62/f66 zero size 2024-04-22T15:49:53.228 INFO:tasks.workunit.client.1.smithi027.stdout:4/312: rename d5/de/d30/f72 to d5/d45/d5a/f82 0 2024-04-22T15:49:53.229 INFO:tasks.workunit.client.1.smithi027.stdout:4/313: truncate d5/d45/f64 603741 0 2024-04-22T15:49:53.234 INFO:tasks.workunit.client.1.smithi027.stdout:4/314: creat d5/d11/d1d/d7b/f83 x:0 0 0 2024-04-22T15:49:53.236 INFO:tasks.workunit.client.1.smithi027.stdout:7/261: dwrite d0/f37 [0,4194304] 0 2024-04-22T15:49:53.237 INFO:tasks.workunit.client.1.smithi027.stdout:7/262: chown d0/d11 106841 1 2024-04-22T15:49:53.239 INFO:tasks.workunit.client.1.smithi027.stdout:4/315: dread d5/de/f43 [0,4194304] 0 2024-04-22T15:49:53.241 INFO:tasks.workunit.client.1.smithi027.stdout:3/395: getdents d28/d69 0 2024-04-22T15:49:53.245 INFO:tasks.workunit.client.1.smithi027.stdout:4/316: unlink d5/d11/l4e 0 2024-04-22T15:49:53.249 INFO:tasks.workunit.client.1.smithi027.stdout:4/317: creat d5/de/d2f/f84 x:0 0 0 2024-04-22T15:49:53.253 INFO:tasks.workunit.client.1.smithi027.stdout:7/263: dwrite d0/f28 [0,4194304] 0 2024-04-22T15:49:53.254 INFO:tasks.workunit.client.1.smithi027.stdout:4/318: creat d5/d11/d1d/d7b/f85 x:0 0 0 2024-04-22T15:49:53.258 INFO:tasks.workunit.client.1.smithi027.stdout:4/319: readlink d5/d11/d1d/d24/l3e 0 2024-04-22T15:49:53.260 INFO:tasks.workunit.client.1.smithi027.stdout:4/320: dread - d5/de/d2f/f70 zero size 2024-04-22T15:49:53.261 INFO:tasks.workunit.client.1.smithi027.stdout:4/321: chown d5/de/d30 1146341636 1 2024-04-22T15:49:53.262 INFO:tasks.workunit.client.1.smithi027.stdout:3/396: dwrite d28/d4b/f4c [0,4194304] 0 2024-04-22T15:49:53.264 INFO:tasks.workunit.client.1.smithi027.stdout:7/264: rename d0/f28 to d0/d11/f4d 0 2024-04-22T15:49:53.265 INFO:tasks.workunit.client.1.smithi027.stdout:7/265: dread - d0/f4b zero size 2024-04-22T15:49:53.269 INFO:tasks.workunit.client.1.smithi027.stdout:4/322: creat d5/de/f86 x:0 0 0 2024-04-22T15:49:53.271 INFO:tasks.workunit.client.1.smithi027.stdout:3/397: rename d28/d4b/c81 to d28/d4b/d52/c83 0 2024-04-22T15:49:53.273 INFO:tasks.workunit.client.1.smithi027.stdout:7/266: mkdir d0/d4c/d4e 0 2024-04-22T15:49:53.274 INFO:tasks.workunit.client.1.smithi027.stdout:4/323: rename d5/de/c1a to d5/d45/d47/c87 0 2024-04-22T15:49:53.274 INFO:tasks.workunit.client.1.smithi027.stdout:5/349: sync 2024-04-22T15:49:53.277 INFO:tasks.workunit.client.1.smithi027.stdout:4/324: mknod d5/de/c88 0 2024-04-22T15:49:53.278 INFO:tasks.workunit.client.1.smithi027.stdout:3/398: dwrite d28/d4b/f7c [0,4194304] 0 2024-04-22T15:49:53.279 INFO:tasks.workunit.client.1.smithi027.stdout:4/325: write d5/de/d2a/f3a [1202010,16562] 0 2024-04-22T15:49:53.283 INFO:tasks.workunit.client.1.smithi027.stdout:5/350: mknod d0/df/de/d24/d1c/d3e/d55/c6d 0 2024-04-22T15:49:53.290 INFO:tasks.workunit.client.1.smithi027.stdout:5/351: dwrite d0/f6 [4194304,4194304] 0 2024-04-22T15:49:53.292 INFO:tasks.workunit.client.1.smithi027.stdout:3/399: rename d28/d5d/d62/f64 to d28/d5d/f84 0 2024-04-22T15:49:53.294 INFO:tasks.workunit.client.1.smithi027.stdout:3/400: write d28/f44 [2562962,20244] 0 2024-04-22T15:49:53.299 INFO:tasks.workunit.client.1.smithi027.stdout:4/326: getdents d5/d42 0 2024-04-22T15:49:53.299 INFO:tasks.workunit.client.1.smithi027.stdout:5/352: dwrite d0/f3c [4194304,4194304] 0 2024-04-22T15:49:53.301 INFO:tasks.workunit.client.1.smithi027.stdout:5/353: read - d0/df/d3d/f4e zero size 2024-04-22T15:49:53.306 INFO:tasks.workunit.client.1.smithi027.stdout:5/354: symlink d0/df/d10/l6e 0 2024-04-22T15:49:53.319 INFO:tasks.workunit.client.1.smithi027.stdout:5/355: write d0/f8 [2350638,31318] 0 2024-04-22T15:49:53.322 INFO:tasks.workunit.client.1.smithi027.stdout:5/356: mknod d0/d1a/d22/d4c/c6f 0 2024-04-22T15:49:53.322 INFO:tasks.workunit.client.1.smithi027.stdout:5/357: readlink d0/d1a/d22/l27 0 2024-04-22T15:49:53.336 INFO:tasks.workunit.client.1.smithi027.stdout:5/358: dread d0/f6 [0,4194304] 0 2024-04-22T15:49:53.337 INFO:tasks.workunit.client.1.smithi027.stdout:5/359: write d0/df/d3d/f69 [95105,77519] 0 2024-04-22T15:49:53.340 INFO:tasks.workunit.client.1.smithi027.stdout:5/360: creat d0/d1a/d22/d26/d32/f70 x:0 0 0 2024-04-22T15:49:53.341 INFO:tasks.workunit.client.1.smithi027.stdout:5/361: mknod d0/df/de/d24/d62/c71 0 2024-04-22T15:49:53.343 INFO:tasks.workunit.client.1.smithi027.stdout:5/362: mkdir d0/d1a/d22/d72 0 2024-04-22T15:49:53.343 INFO:tasks.workunit.client.1.smithi027.stdout:5/363: chown d0/df 53 1 2024-04-22T15:49:53.345 INFO:tasks.workunit.client.1.smithi027.stdout:5/364: mknod d0/d1a/d22/d26/d32/c73 0 2024-04-22T15:49:53.345 INFO:tasks.workunit.client.1.smithi027.stdout:5/365: stat d0/df/d3d/f69 0 2024-04-22T15:49:53.347 INFO:tasks.workunit.client.1.smithi027.stdout:5/366: dread d0/df/de/d24/d1c/f34 [0,4194304] 0 2024-04-22T15:49:53.348 INFO:tasks.workunit.client.1.smithi027.stdout:5/367: chown d0/d1a/d22/d26/c2c 82738689 1 2024-04-22T15:49:53.349 INFO:tasks.workunit.client.1.smithi027.stdout:5/368: chown d0/d1a/d22/d26/d5d 31563 1 2024-04-22T15:49:53.351 INFO:tasks.workunit.client.1.smithi027.stdout:5/369: mknod d0/d1a/d22/d72/c74 0 2024-04-22T15:49:53.355 INFO:tasks.workunit.client.1.smithi027.stdout:5/370: creat d0/df/de/d52/f75 x:0 0 0 2024-04-22T15:49:53.356 INFO:tasks.workunit.client.1.smithi027.stdout:5/371: read d0/d1a/d22/d26/f48 [1219603,127890] 0 2024-04-22T15:49:53.356 INFO:tasks.workunit.client.1.smithi027.stdout:5/372: chown d0/d1a/d22/d72 1363 1 2024-04-22T15:49:53.358 INFO:tasks.workunit.client.1.smithi027.stdout:5/373: rename d0 to d0/df/de/d24/d1c/d3e/d55/d76 22 2024-04-22T15:49:53.363 INFO:tasks.workunit.client.1.smithi027.stdout:5/374: creat d0/df/de/d52/f77 x:0 0 0 2024-04-22T15:49:53.366 INFO:tasks.workunit.client.1.smithi027.stdout:5/375: dread d0/f6 [0,4194304] 0 2024-04-22T15:49:53.368 INFO:tasks.workunit.client.1.smithi027.stdout:5/376: mknod d0/d1a/c78 0 2024-04-22T15:49:53.371 INFO:tasks.workunit.client.1.smithi027.stdout:5/377: rename d0/d1a/d22/d4c to d0/d1a/d22/d26/d32/d79 0 2024-04-22T15:49:53.381 INFO:tasks.workunit.client.1.smithi027.stdout:5/378: getdents d0/d1a/d22/d26/d32 0 2024-04-22T15:49:53.386 INFO:tasks.workunit.client.1.smithi027.stdout:5/379: creat d0/df/f7a x:0 0 0 2024-04-22T15:49:53.389 INFO:tasks.workunit.client.1.smithi027.stdout:5/380: dread d0/f3c [0,4194304] 0 2024-04-22T15:49:53.395 INFO:tasks.workunit.client.1.smithi027.stdout:5/381: truncate d0/f6 5063709 0 2024-04-22T15:49:53.404 INFO:tasks.workunit.client.1.smithi027.stdout:1/228: sync 2024-04-22T15:49:53.404 INFO:tasks.workunit.client.1.smithi027.stdout:8/296: sync 2024-04-22T15:49:53.405 INFO:tasks.workunit.client.1.smithi027.stdout:1/229: write d2/f35 [694288,23655] 0 2024-04-22T15:49:53.405 INFO:tasks.workunit.client.1.smithi027.stdout:8/297: fsync d1/d3/d1c/d56/d62/f68 0 2024-04-22T15:49:53.407 INFO:tasks.workunit.client.1.smithi027.stdout:1/230: write d2/d1d/f1c [2195221,47302] 0 2024-04-22T15:49:53.408 INFO:tasks.workunit.client.1.smithi027.stdout:8/298: creat d1/d3/f6b x:0 0 0 2024-04-22T15:49:53.408 INFO:tasks.workunit.client.1.smithi027.stdout:8/299: dread - d1/d3/f3c zero size 2024-04-22T15:49:53.411 INFO:tasks.workunit.client.1.smithi027.stdout:1/231: mknod d2/d36/c4d 0 2024-04-22T15:49:53.411 INFO:tasks.workunit.client.1.smithi027.stdout:1/232: fsync d2/d36/f4b 0 2024-04-22T15:49:53.420 INFO:tasks.workunit.client.1.smithi027.stdout:1/233: link d2/d36/f4b d2/d36/d38/d3d/f4e 0 2024-04-22T15:49:53.424 INFO:tasks.workunit.client.1.smithi027.stdout:1/234: dread d2/f3 [0,4194304] 0 2024-04-22T15:49:53.424 INFO:tasks.workunit.client.1.smithi027.stdout:1/235: write d2/d36/d38/f40 [123586,68041] 0 2024-04-22T15:49:53.427 INFO:tasks.workunit.client.1.smithi027.stdout:1/236: unlink d2/d30/f43 0 2024-04-22T15:49:53.429 INFO:tasks.workunit.client.1.smithi027.stdout:1/237: mknod d2/d1d/c4f 0 2024-04-22T15:49:53.433 INFO:tasks.workunit.client.1.smithi027.stdout:1/238: creat d2/d36/d38/f50 x:0 0 0 2024-04-22T15:49:53.437 INFO:tasks.workunit.client.1.smithi027.stdout:1/239: rename d2/lf to d2/d30/l51 0 2024-04-22T15:49:53.438 INFO:tasks.workunit.client.1.smithi027.stdout:1/240: rename d2/d1d to d2/d1d/d52 22 2024-04-22T15:49:53.438 INFO:tasks.workunit.client.1.smithi027.stdout:1/241: fdatasync d2/d1d/f32 0 2024-04-22T15:49:53.440 INFO:tasks.workunit.client.1.smithi027.stdout:1/242: write d2/f3 [3381302,66734] 0 2024-04-22T15:49:53.449 INFO:tasks.workunit.client.1.smithi027.stdout:1/243: readlink d2/d30/l51 0 2024-04-22T15:49:53.450 INFO:tasks.workunit.client.1.smithi027.stdout:1/244: truncate d2/d36/f4b 4578199 0 2024-04-22T15:49:53.454 INFO:tasks.workunit.client.1.smithi027.stdout:1/245: truncate d2/d30/f47 3528763 0 2024-04-22T15:49:53.455 INFO:tasks.workunit.client.1.smithi027.stdout:1/246: rename d2/d30 to d2/d30/d53 22 2024-04-22T15:49:53.458 INFO:tasks.workunit.client.1.smithi027.stdout:1/247: mknod d2/d36/d38/c54 0 2024-04-22T15:49:53.462 INFO:tasks.workunit.client.1.smithi027.stdout:1/248: creat d2/d36/f55 x:0 0 0 2024-04-22T15:49:53.463 INFO:tasks.workunit.client.1.smithi027.stdout:1/249: mkdir d2/d1d/d56 0 2024-04-22T15:49:53.464 INFO:tasks.workunit.client.1.smithi027.stdout:1/250: stat d2/d1d/l2c 0 2024-04-22T15:49:53.468 INFO:tasks.workunit.client.1.smithi027.stdout:1/251: creat d2/f57 x:0 0 0 2024-04-22T15:49:53.469 INFO:tasks.workunit.client.1.smithi027.stdout:1/252: fsync d2/d1d/f1c 0 2024-04-22T15:49:53.475 INFO:tasks.workunit.client.1.smithi027.stdout:1/253: dwrite d2/f37 [0,4194304] 0 2024-04-22T15:49:53.481 INFO:tasks.workunit.client.1.smithi027.stdout:1/254: dwrite d2/f1f [0,4194304] 0 2024-04-22T15:49:53.483 INFO:tasks.workunit.client.1.smithi027.stdout:1/255: unlink d2/d30/l34 0 2024-04-22T15:49:53.487 INFO:tasks.workunit.client.1.smithi027.stdout:1/256: symlink d2/l58 0 2024-04-22T15:49:53.490 INFO:tasks.workunit.client.1.smithi027.stdout:1/257: unlink d2/d36/d38/f40 0 2024-04-22T15:49:53.491 INFO:tasks.workunit.client.1.smithi027.stdout:1/258: stat d2/d36 0 2024-04-22T15:49:53.501 INFO:tasks.workunit.client.1.smithi027.stdout:1/259: dwrite d2/f57 [0,4194304] 0 2024-04-22T15:49:53.503 INFO:tasks.workunit.client.1.smithi027.stdout:1/260: write d2/f8 [2430280,23917] 0 2024-04-22T15:49:53.505 INFO:tasks.workunit.client.1.smithi027.stdout:1/261: mknod d2/d1d/c59 0 2024-04-22T15:49:53.509 INFO:tasks.workunit.client.1.smithi027.stdout:1/262: dread d2/f37 [0,4194304] 0 2024-04-22T15:49:53.512 INFO:tasks.workunit.client.1.smithi027.stdout:1/263: creat d2/d36/d38/d3d/f5a x:0 0 0 2024-04-22T15:49:53.513 INFO:tasks.workunit.client.1.smithi027.stdout:1/264: write d2/f1f [3246113,30571] 0 2024-04-22T15:49:53.516 INFO:tasks.workunit.client.1.smithi027.stdout:1/265: creat d2/d1d/f5b x:0 0 0 2024-04-22T15:49:53.518 INFO:tasks.workunit.client.1.smithi027.stdout:1/266: symlink d2/d30/l5c 0 2024-04-22T15:49:53.522 INFO:tasks.workunit.client.1.smithi027.stdout:1/267: dread d2/d36/d38/d3d/f4e [0,4194304] 0 2024-04-22T15:49:53.534 INFO:tasks.workunit.client.1.smithi027.stdout:1/268: dwrite d2/d36/d38/d3d/f4e [0,4194304] 0 2024-04-22T15:49:53.540 INFO:tasks.workunit.client.1.smithi027.stdout:1/269: link d2/d1d/c23 d2/d1d/c5d 0 2024-04-22T15:49:53.544 INFO:tasks.workunit.client.1.smithi027.stdout:1/270: mknod d2/d36/d38/c5e 0 2024-04-22T15:49:53.549 INFO:tasks.workunit.client.1.smithi027.stdout:1/271: dwrite d2/f57 [0,4194304] 0 2024-04-22T15:49:53.550 INFO:tasks.workunit.client.1.smithi027.stdout:1/272: truncate d2/f35 844351 0 2024-04-22T15:49:53.560 INFO:tasks.workunit.client.1.smithi027.stdout:9/254: sync 2024-04-22T15:49:53.575 INFO:tasks.workunit.client.1.smithi027.stdout:1/273: write d2/d36/f4b [2503604,36612] 0 2024-04-22T15:49:53.579 INFO:tasks.workunit.client.1.smithi027.stdout:1/274: stat d2/d36/c3f 0 2024-04-22T15:49:53.587 INFO:tasks.workunit.client.1.smithi027.stdout:9/255: truncate d0/dc/d12/d30/f32 2237988 0 2024-04-22T15:49:53.590 INFO:tasks.workunit.client.1.smithi027.stdout:1/275: getdents d2/d36/d38 0 2024-04-22T15:49:53.596 INFO:tasks.workunit.client.1.smithi027.stdout:9/256: truncate d0/dc/d12/d30/f4f 1067197 0 2024-04-22T15:49:53.597 INFO:tasks.workunit.client.1.smithi027.stdout:1/276: mkdir d2/d5f 0 2024-04-22T15:49:53.600 INFO:tasks.workunit.client.1.smithi027.stdout:1/277: write d2/f27 [980188,93318] 0 2024-04-22T15:49:53.601 INFO:tasks.workunit.client.1.smithi027.stdout:9/257: mknod d0/dc/c5a 0 2024-04-22T15:49:53.601 INFO:tasks.workunit.client.1.smithi027.stdout:9/258: readlink d0/dc/df/l56 0 2024-04-22T15:49:53.609 INFO:tasks.workunit.client.1.smithi027.stdout:1/278: creat d2/f60 x:0 0 0 2024-04-22T15:49:53.609 INFO:tasks.workunit.client.1.smithi027.stdout:1/279: chown d2/d30/f41 8 1 2024-04-22T15:49:53.614 INFO:tasks.workunit.client.1.smithi027.stdout:9/259: chown d0/dc/d12/d30/f4f 651 1 2024-04-22T15:49:53.615 INFO:tasks.workunit.client.1.smithi027.stdout:1/280: write d2/fa [931923,16398] 0 2024-04-22T15:49:53.618 INFO:tasks.workunit.client.1.smithi027.stdout:1/281: chown d2/d36/c3b 51 1 2024-04-22T15:49:53.619 INFO:tasks.workunit.client.1.smithi027.stdout:9/260: dread d0/dc/de/f3a [0,4194304] 0 2024-04-22T15:49:53.622 INFO:tasks.workunit.client.1.smithi027.stdout:9/261: mkdir d0/dc/d12/d30/d5b 0 2024-04-22T15:49:53.624 INFO:tasks.workunit.client.1.smithi027.stdout:1/282: dwrite d2/d1d/f32 [0,4194304] 0 2024-04-22T15:49:53.624 INFO:tasks.workunit.client.1.smithi027.stdout:9/262: readlink d0/l1d 0 2024-04-22T15:49:53.626 INFO:tasks.workunit.client.1.smithi027.stdout:9/263: chown d0/d8/f18 1140957451 1 2024-04-22T15:49:53.626 INFO:tasks.workunit.client.1.smithi027.stdout:1/283: write d2/d36/f55 [541208,106411] 0 2024-04-22T15:49:53.627 INFO:tasks.workunit.client.1.smithi027.stdout:1/284: write d2/f60 [556676,63993] 0 2024-04-22T15:49:53.628 INFO:tasks.workunit.client.1.smithi027.stdout:9/264: mkdir d0/d8/d5c 0 2024-04-22T15:49:53.631 INFO:tasks.workunit.client.1.smithi027.stdout:9/265: mknod d0/d8/d5c/c5d 0 2024-04-22T15:49:53.635 INFO:tasks.workunit.client.1.smithi027.stdout:1/285: dwrite d2/d36/d38/d3d/f4e [0,4194304] 0 2024-04-22T15:49:53.638 INFO:tasks.workunit.client.1.smithi027.stdout:1/286: mknod d2/d1d/c61 0 2024-04-22T15:49:53.642 INFO:tasks.workunit.client.1.smithi027.stdout:1/287: unlink d2/d36/d38/d3d/f5a 0 2024-04-22T15:49:53.649 INFO:tasks.workunit.client.1.smithi027.stdout:1/288: creat d2/d36/f62 x:0 0 0 2024-04-22T15:49:53.649 INFO:tasks.workunit.client.1.smithi027.stdout:1/289: fsync d2/d36/f55 0 2024-04-22T15:49:53.658 INFO:tasks.workunit.client.1.smithi027.stdout:1/290: write d2/d30/f3c [1037200,63554] 0 2024-04-22T15:49:53.661 INFO:tasks.workunit.client.1.smithi027.stdout:1/291: creat d2/d36/d38/d3d/d44/f63 x:0 0 0 2024-04-22T15:49:53.664 INFO:tasks.workunit.client.1.smithi027.stdout:1/292: dread d2/f60 [0,4194304] 0 2024-04-22T15:49:53.664 INFO:tasks.workunit.client.1.smithi027.stdout:1/293: readlink d2/d30/l46 0 2024-04-22T15:49:53.670 INFO:tasks.workunit.client.1.smithi027.stdout:1/294: dwrite d2/d1d/f5b [0,4194304] 0 2024-04-22T15:49:53.679 INFO:tasks.workunit.client.1.smithi027.stdout:1/295: getdents d2/d30 0 2024-04-22T15:49:53.686 INFO:tasks.workunit.client.1.smithi027.stdout:1/296: dwrite d2/d36/d38/d3d/f4e [0,4194304] 0 2024-04-22T15:49:53.814 INFO:tasks.workunit.client.1.smithi027.stdout:6/313: sync 2024-04-22T15:49:53.815 INFO:tasks.workunit.client.1.smithi027.stdout:6/314: chown d5/dc/d20/f3f 3502 1 2024-04-22T15:49:53.817 INFO:tasks.workunit.client.1.smithi027.stdout:6/315: creat d5/dc/d17/d53/f72 x:0 0 0 2024-04-22T15:49:53.818 INFO:tasks.workunit.client.1.smithi027.stdout:6/316: mknod d5/dc/dd/c73 0 2024-04-22T15:49:53.819 INFO:tasks.workunit.client.1.smithi027.stdout:6/317: chown d5/dc/d17/d48/f4e 0 1 2024-04-22T15:49:53.819 INFO:tasks.workunit.client.1.smithi027.stdout:6/318: chown d5/d30/d40 96611998 1 2024-04-22T15:49:53.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:53 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:53.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:53 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:53.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:53 smithi027 ceph-mon[34922]: pgmap v39: 65 pgs: 65 active+clean; 2.7 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 81 MiB/s rd, 267 MiB/s wr, 547 op/s 2024-04-22T15:49:53.903 INFO:tasks.workunit.client.1.smithi027.stdout:6/319: creat d5/dc/f74 x:0 0 0 2024-04-22T15:49:53.910 INFO:tasks.workunit.client.1.smithi027.stdout:6/320: creat d5/f75 x:0 0 0 2024-04-22T15:49:53.910 INFO:tasks.workunit.client.1.smithi027.stdout:6/321: truncate d5/dc/d17/f39 4906398 0 2024-04-22T15:49:53.911 INFO:tasks.workunit.client.1.smithi027.stdout:6/322: unlink d5/dc/c42 0 2024-04-22T15:49:53.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:53 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:53.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:53 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:53.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:53 smithi016 ceph-mon[26620]: pgmap v39: 65 pgs: 65 active+clean; 2.7 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 81 MiB/s rd, 267 MiB/s wr, 547 op/s 2024-04-22T15:49:53.924 INFO:tasks.workunit.client.1.smithi027.stdout:6/323: dread d5/f29 [0,4194304] 0 2024-04-22T15:49:53.925 INFO:tasks.workunit.client.1.smithi027.stdout:6/324: readlink d5/dc/d20/d2b/l2a 0 2024-04-22T15:49:53.928 INFO:tasks.workunit.client.1.smithi027.stdout:6/325: stat d5/dc/d26/d49/c5b 0 2024-04-22T15:49:53.929 INFO:tasks.workunit.client.1.smithi027.stdout:6/326: dread - d5/dc/d17/d48/d5d/f67 zero size 2024-04-22T15:49:53.932 INFO:tasks.workunit.client.1.smithi027.stdout:6/327: rmdir d5/db/d2e 39 2024-04-22T15:49:53.937 INFO:tasks.workunit.client.1.smithi027.stdout:0/319: sync 2024-04-22T15:49:53.938 INFO:tasks.workunit.client.1.smithi027.stdout:2/303: sync 2024-04-22T15:49:53.944 INFO:tasks.workunit.client.1.smithi027.stdout:2/304: dread - dd/df/d1b/d1f/f30 zero size 2024-04-22T15:49:53.944 INFO:tasks.workunit.client.1.smithi027.stdout:6/328: dread d5/dc/dd/f16 [0,4194304] 0 2024-04-22T15:49:53.945 INFO:tasks.workunit.client.1.smithi027.stdout:6/329: chown d5/f29 1985511 1 2024-04-22T15:49:53.947 INFO:tasks.workunit.client.1.smithi027.stdout:0/320: link d0/d2a/f48 d0/d34/d59/f62 0 2024-04-22T15:49:53.947 INFO:tasks.workunit.client.1.smithi027.stdout:0/321: readlink d0/d2a/l4d 0 2024-04-22T15:49:53.953 INFO:tasks.workunit.client.1.smithi027.stdout:0/322: dwrite d0/f1 [0,4194304] 0 2024-04-22T15:49:53.954 INFO:tasks.workunit.client.1.smithi027.stdout:6/330: fdatasync d5/dc/d20/d2b/f35 0 2024-04-22T15:49:53.959 INFO:tasks.workunit.client.1.smithi027.stdout:0/323: mkdir d0/d28/d63 0 2024-04-22T15:49:53.963 INFO:tasks.workunit.client.1.smithi027.stdout:6/331: rmdir d5/db 39 2024-04-22T15:49:53.964 INFO:tasks.workunit.client.1.smithi027.stdout:6/332: chown d5/dc/d20/d2b/d34/c41 1003483 1 2024-04-22T15:49:53.964 INFO:tasks.workunit.client.1.smithi027.stdout:6/333: fdatasync d5/dc/d17/f39 0 2024-04-22T15:49:53.967 INFO:tasks.workunit.client.1.smithi027.stdout:0/324: rename d0/d2a/d38/c41 to d0/d34/d4e/c64 0 2024-04-22T15:49:53.968 INFO:tasks.workunit.client.1.smithi027.stdout:0/325: write d0/d28/f1c [33304,68801] 0 2024-04-22T15:49:53.969 INFO:tasks.workunit.client.1.smithi027.stdout:0/326: readlink d0/d2a/l30 0 2024-04-22T15:49:53.973 INFO:tasks.workunit.client.1.smithi027.stdout:6/334: truncate d5/dc/d17/f4b 1071970 0 2024-04-22T15:49:53.977 INFO:tasks.workunit.client.1.smithi027.stdout:2/305: dwrite dd/df/f28 [0,4194304] 0 2024-04-22T15:49:53.977 INFO:tasks.workunit.client.1.smithi027.stdout:0/327: creat d0/d2a/d38/f65 x:0 0 0 2024-04-22T15:49:53.980 INFO:tasks.workunit.client.1.smithi027.stdout:2/306: write dd/df/d1b/f2e [512654,122830] 0 2024-04-22T15:49:53.981 INFO:tasks.workunit.client.1.smithi027.stdout:6/335: dwrite d5/dc/d20/d2b/f27 [0,4194304] 0 2024-04-22T15:49:53.986 INFO:tasks.workunit.client.1.smithi027.stdout:0/328: mkdir d0/d2a/d58/d66 0 2024-04-22T15:49:53.986 INFO:tasks.workunit.client.1.smithi027.stdout:6/336: dread d5/dc/d20/d2b/f27 [0,4194304] 0 2024-04-22T15:49:53.987 INFO:tasks.workunit.client.1.smithi027.stdout:0/329: write d0/f3e [3160549,13372] 0 2024-04-22T15:49:53.990 INFO:tasks.workunit.client.1.smithi027.stdout:0/330: dread - d0/d2a/d38/f4a zero size 2024-04-22T15:49:53.995 INFO:tasks.workunit.client.1.smithi027.stdout:0/331: dread d0/d34/d59/f5d [0,4194304] 0 2024-04-22T15:49:53.997 INFO:tasks.workunit.client.1.smithi027.stdout:6/337: rename d5/dc/d17/c1c to d5/dc/dd/d52/c76 0 2024-04-22T15:49:54.002 INFO:tasks.workunit.client.1.smithi027.stdout:6/338: rmdir d5/dc/d17/d53 39 2024-04-22T15:49:54.004 INFO:tasks.workunit.client.1.smithi027.stdout:6/339: rename d5 to d5/dc/d20/d77 22 2024-04-22T15:49:54.004 INFO:tasks.workunit.client.1.smithi027.stdout:6/340: write d5/dc/d17/d48/d5d/f70 [36105,99007] 0 2024-04-22T15:49:54.014 INFO:tasks.workunit.client.1.smithi027.stdout:0/332: dread d0/fc [0,4194304] 0 2024-04-22T15:49:54.015 INFO:tasks.workunit.client.1.smithi027.stdout:0/333: chown d0/l23 554564 1 2024-04-22T15:49:54.016 INFO:tasks.workunit.client.1.smithi027.stdout:6/341: dread d5/db/d2e/ff [0,4194304] 0 2024-04-22T15:49:54.017 INFO:tasks.workunit.client.1.smithi027.stdout:2/307: write dd/f19 [3856945,75478] 0 2024-04-22T15:49:54.020 INFO:tasks.workunit.client.1.smithi027.stdout:6/342: creat d5/dc/d20/d2b/f78 x:0 0 0 2024-04-22T15:49:54.023 INFO:tasks.workunit.client.1.smithi027.stdout:2/308: mkdir dd/d1d/d5a 0 2024-04-22T15:49:54.024 INFO:tasks.workunit.client.1.smithi027.stdout:0/334: mkdir d0/d28/d63/d67 0 2024-04-22T15:49:54.025 INFO:tasks.workunit.client.1.smithi027.stdout:2/309: fsync dd/d1d/d4b/d4e/f56 0 2024-04-22T15:49:54.026 INFO:tasks.workunit.client.1.smithi027.stdout:2/310: chown dd/d47/f48 1068827 1 2024-04-22T15:49:54.027 INFO:tasks.workunit.client.1.smithi027.stdout:6/343: dwrite d5/dc/d20/d2b/f12 [4194304,4194304] 0 2024-04-22T15:49:54.029 INFO:tasks.workunit.client.1.smithi027.stdout:0/335: creat d0/d34/d4e/f68 x:0 0 0 2024-04-22T15:49:54.031 INFO:tasks.workunit.client.1.smithi027.stdout:2/311: mknod dd/df/d1b/c5b 0 2024-04-22T15:49:54.033 INFO:tasks.workunit.client.1.smithi027.stdout:2/312: chown dd/df/f59 7 1 2024-04-22T15:49:54.034 INFO:tasks.workunit.client.1.smithi027.stdout:0/336: getdents d0/d28/d5f 0 2024-04-22T15:49:54.035 INFO:tasks.workunit.client.1.smithi027.stdout:2/313: write dd/d1d/d4b/f58 [1165398,47109] 0 2024-04-22T15:49:54.036 INFO:tasks.workunit.client.1.smithi027.stdout:6/344: dwrite d5/db/f4d [0,4194304] 0 2024-04-22T15:49:54.039 INFO:tasks.workunit.client.1.smithi027.stdout:0/337: rename d0/d28/c57 to d0/c69 0 2024-04-22T15:49:54.042 INFO:tasks.workunit.client.1.smithi027.stdout:3/401: sync 2024-04-22T15:49:54.042 INFO:tasks.workunit.client.1.smithi027.stdout:4/327: sync 2024-04-22T15:49:54.042 INFO:tasks.workunit.client.1.smithi027.stdout:7/267: sync 2024-04-22T15:49:54.043 INFO:tasks.workunit.client.1.smithi027.stdout:2/314: creat dd/df/d1b/d38/f5c x:0 0 0 2024-04-22T15:49:54.044 INFO:tasks.workunit.client.1.smithi027.stdout:7/268: fsync d0/d11/d3a/f45 0 2024-04-22T15:49:54.045 INFO:tasks.workunit.client.1.smithi027.stdout:6/345: unlink d5/dc/dd/f69 0 2024-04-22T15:49:54.046 INFO:tasks.workunit.client.1.smithi027.stdout:6/346: chown d5/dc/d20/d2b/f35 83627 1 2024-04-22T15:49:54.047 INFO:tasks.workunit.client.1.smithi027.stdout:6/347: chown d5/dc/d20/d2b/d34/l68 4503 1 2024-04-22T15:49:54.048 INFO:tasks.workunit.client.1.smithi027.stdout:3/402: creat d28/d69/f85 x:0 0 0 2024-04-22T15:49:54.048 INFO:tasks.workunit.client.1.smithi027.stdout:3/403: chown d28/c31 5731 1 2024-04-22T15:49:54.056 INFO:tasks.workunit.client.1.smithi027.stdout:2/315: creat dd/d1d/d4b/f5d x:0 0 0 2024-04-22T15:49:54.057 INFO:tasks.workunit.client.1.smithi027.stdout:2/316: write dd/df/d1b/f40 [155522,27327] 0 2024-04-22T15:49:54.059 INFO:tasks.workunit.client.1.smithi027.stdout:7/269: dwrite d0/d11/fe [0,4194304] 0 2024-04-22T15:49:54.059 INFO:tasks.workunit.client.1.smithi027.stdout:4/328: truncate d5/d11/d1d/f5c 674936 0 2024-04-22T15:49:54.061 INFO:tasks.workunit.client.1.smithi027.stdout:0/338: mkdir d0/d2a/d6a 0 2024-04-22T15:49:54.066 INFO:tasks.workunit.client.1.smithi027.stdout:6/348: rmdir d5/dc/d20/d2b 39 2024-04-22T15:49:54.066 INFO:tasks.workunit.client.1.smithi027.stdout:2/317: dwrite dd/df/d1b/f40 [0,4194304] 0 2024-04-22T15:49:54.073 INFO:tasks.workunit.client.1.smithi027.stdout:6/349: dwrite d5/f75 [0,4194304] 0 2024-04-22T15:49:54.075 INFO:tasks.workunit.client.1.smithi027.stdout:4/329: chown d5/f25 30848 1 2024-04-22T15:49:54.079 INFO:tasks.workunit.client.1.smithi027.stdout:7/270: unlink d0/d16/f2d 0 2024-04-22T15:49:54.082 INFO:tasks.workunit.client.1.smithi027.stdout:0/339: creat d0/d2a/d38/f6b x:0 0 0 2024-04-22T15:49:54.083 INFO:tasks.workunit.client.1.smithi027.stdout:2/318: readlink dd/df/d1b/d1f/l46 0 2024-04-22T15:49:54.085 INFO:tasks.workunit.client.1.smithi027.stdout:4/330: dwrite d5/d11/d1d/f53 [0,4194304] 0 2024-04-22T15:49:54.087 INFO:tasks.workunit.client.1.smithi027.stdout:6/350: mkdir d5/dc/dd/d79 0 2024-04-22T15:49:54.089 INFO:tasks.workunit.client.1.smithi027.stdout:7/271: creat d0/d11/d1c/d2c/f4f x:0 0 0 2024-04-22T15:49:54.094 INFO:tasks.workunit.client.1.smithi027.stdout:2/319: creat dd/df/d1b/f5e x:0 0 0 2024-04-22T15:49:54.099 INFO:tasks.workunit.client.1.smithi027.stdout:4/331: rename d5/de/d2f to d5/d11/d89 0 2024-04-22T15:49:54.102 INFO:tasks.workunit.client.1.smithi027.stdout:2/320: dwrite dd/d1d/f45 [0,4194304] 0 2024-04-22T15:49:54.104 INFO:tasks.workunit.client.1.smithi027.stdout:2/321: write dd/d1d/d4b/f5d [955794,61136] 0 2024-04-22T15:49:54.110 INFO:tasks.workunit.client.1.smithi027.stdout:7/272: dread d0/d16/f25 [0,4194304] 0 2024-04-22T15:49:54.112 INFO:tasks.workunit.client.1.smithi027.stdout:4/332: creat d5/d45/d47/f8a x:0 0 0 2024-04-22T15:49:54.114 INFO:tasks.workunit.client.1.smithi027.stdout:6/351: creat d5/f7a x:0 0 0 2024-04-22T15:49:54.118 INFO:tasks.workunit.client.1.smithi027.stdout:7/273: mknod d0/d11/d3a/c50 0 2024-04-22T15:49:54.118 INFO:tasks.workunit.client.1.smithi027.stdout:4/333: dread - d5/d11/d1d/d24/f79 zero size 2024-04-22T15:49:54.118 INFO:tasks.workunit.client.1.smithi027.stdout:0/340: dwrite d0/d2a/f3c [0,4194304] 0 2024-04-22T15:49:54.119 INFO:tasks.workunit.client.1.smithi027.stdout:0/341: readlink d0/l23 0 2024-04-22T15:49:54.122 INFO:tasks.workunit.client.1.smithi027.stdout:6/352: rename d5/dc/d17/d48 to d5/dc/dd/d79/d7b 0 2024-04-22T15:49:54.124 INFO:tasks.workunit.client.1.smithi027.stdout:4/334: creat d5/d11/d6c/f8b x:0 0 0 2024-04-22T15:49:54.131 INFO:tasks.workunit.client.1.smithi027.stdout:2/322: dread dd/df/d1b/f26 [0,4194304] 0 2024-04-22T15:49:54.132 INFO:tasks.workunit.client.1.smithi027.stdout:2/323: write dd/f4d [1036960,82401] 0 2024-04-22T15:49:54.132 INFO:tasks.workunit.client.1.smithi027.stdout:4/335: mkdir d5/d11/d1d/d54/d8c 0 2024-04-22T15:49:54.134 INFO:tasks.workunit.client.1.smithi027.stdout:4/336: read d5/d42/f61 [131269,46205] 0 2024-04-22T15:49:54.134 INFO:tasks.workunit.client.1.smithi027.stdout:0/342: mkdir d0/d2a/d6a/d6c 0 2024-04-22T15:49:54.134 INFO:tasks.workunit.client.1.smithi027.stdout:4/337: fdatasync d5/de/f86 0 2024-04-22T15:49:54.135 INFO:tasks.workunit.client.1.smithi027.stdout:0/343: truncate d0/d2a/f35 1216848 0 2024-04-22T15:49:54.136 INFO:tasks.workunit.client.1.smithi027.stdout:6/353: link d5/dc/dd/d79/d7b/d5d/f70 d5/dc/d26/d3d/f7c 0 2024-04-22T15:49:54.138 INFO:tasks.workunit.client.1.smithi027.stdout:7/274: getdents d0/d11/d1c/d31 0 2024-04-22T15:49:54.141 INFO:tasks.workunit.client.1.smithi027.stdout:6/354: mknod d5/dc/d26/d3d/c7d 0 2024-04-22T15:49:54.144 INFO:tasks.workunit.client.1.smithi027.stdout:7/275: mknod d0/d11/d1c/d2c/c51 0 2024-04-22T15:49:54.144 INFO:tasks.workunit.client.1.smithi027.stdout:4/338: dread d5/d11/d1d/f5c [0,4194304] 0 2024-04-22T15:49:54.145 INFO:tasks.workunit.client.1.smithi027.stdout:0/344: dwrite d0/d2a/d38/f65 [0,4194304] 0 2024-04-22T15:49:54.146 INFO:tasks.workunit.client.1.smithi027.stdout:7/276: fsync d0/d16/f3d 0 2024-04-22T15:49:54.147 INFO:tasks.workunit.client.1.smithi027.stdout:4/339: readlink d5/d11/l3d 0 2024-04-22T15:49:54.149 INFO:tasks.workunit.client.1.smithi027.stdout:0/345: chown d0/d2a/d38/f65 2699318 1 2024-04-22T15:49:54.149 INFO:tasks.workunit.client.1.smithi027.stdout:2/324: dwrite dd/df/d1b/d1f/f2c [0,4194304] 0 2024-04-22T15:49:54.153 INFO:tasks.workunit.client.1.smithi027.stdout:2/325: dread - dd/df/d1b/d38/f5c zero size 2024-04-22T15:49:54.153 INFO:tasks.workunit.client.1.smithi027.stdout:0/346: dread d0/d34/f44 [0,4194304] 0 2024-04-22T15:49:54.155 INFO:tasks.workunit.client.1.smithi027.stdout:4/340: creat d5/d11/f8d x:0 0 0 2024-04-22T15:49:54.156 INFO:tasks.workunit.client.1.smithi027.stdout:4/341: write d5/d45/d47/f62 [592893,44624] 0 2024-04-22T15:49:54.157 INFO:tasks.workunit.client.1.smithi027.stdout:7/277: mkdir d0/d16/d52 0 2024-04-22T15:49:54.159 INFO:tasks.workunit.client.1.smithi027.stdout:4/342: read d5/f25 [1860748,64775] 0 2024-04-22T15:49:54.161 INFO:tasks.workunit.client.1.smithi027.stdout:4/343: write d5/de/d2a/d58/f67 [3815512,27761] 0 2024-04-22T15:49:54.161 INFO:tasks.workunit.client.1.smithi027.stdout:2/326: creat dd/df/d1b/f5f x:0 0 0 2024-04-22T15:49:54.162 INFO:tasks.workunit.client.1.smithi027.stdout:6/355: write d5/dc/d17/f4b [1217921,44542] 0 2024-04-22T15:49:54.167 INFO:tasks.workunit.client.1.smithi027.stdout:8/300: sync 2024-04-22T15:49:54.167 INFO:tasks.workunit.client.1.smithi027.stdout:5/382: sync 2024-04-22T15:49:54.169 INFO:tasks.workunit.client.1.smithi027.stdout:0/347: symlink d0/d28/d63/d67/l6d 0 2024-04-22T15:49:54.169 INFO:tasks.workunit.client.1.smithi027.stdout:2/327: chown dd/df/d1b/d1f/f4a 324913059 1 2024-04-22T15:49:54.170 INFO:tasks.workunit.client.1.smithi027.stdout:5/383: fdatasync d0/df/d10/f64 0 2024-04-22T15:49:54.171 INFO:tasks.workunit.client.1.smithi027.stdout:6/356: dwrite d5/dc/d17/f4b [0,4194304] 0 2024-04-22T15:49:54.176 INFO:tasks.workunit.client.1.smithi027.stdout:2/328: dread dd/df/d1b/d1f/f4a [0,4194304] 0 2024-04-22T15:49:54.177 INFO:tasks.workunit.client.1.smithi027.stdout:2/329: dread - dd/df/d1b/d38/f5c zero size 2024-04-22T15:49:54.179 INFO:tasks.workunit.client.1.smithi027.stdout:4/344: write d5/d11/d1d/f5c [1196186,58449] 0 2024-04-22T15:49:54.214 INFO:tasks.workunit.client.1.smithi027.stdout:0/348: rmdir d0/d28/d60 0 2024-04-22T15:49:54.214 INFO:tasks.workunit.client.1.smithi027.stdout:2/330: dwrite dd/f19 [0,4194304] 0 2024-04-22T15:49:54.215 INFO:tasks.workunit.client.1.smithi027.stdout:4/345: chown d5/d11/f18 165396029 1 2024-04-22T15:49:54.217 INFO:tasks.workunit.client.1.smithi027.stdout:5/384: getdents d0/df/de/d24/d1c/d3e 0 2024-04-22T15:49:54.220 INFO:tasks.workunit.client.1.smithi027.stdout:0/349: symlink d0/d28/d5f/l6e 0 2024-04-22T15:49:54.221 INFO:tasks.workunit.client.1.smithi027.stdout:5/385: read - d0/df/de/d24/d62/f68 zero size 2024-04-22T15:49:54.222 INFO:tasks.workunit.client.1.smithi027.stdout:2/331: write dd/df/d1b/f26 [3446078,46206] 0 2024-04-22T15:49:54.223 INFO:tasks.workunit.client.1.smithi027.stdout:7/278: dwrite d0/fb [0,4194304] 0 2024-04-22T15:49:54.225 INFO:tasks.workunit.client.1.smithi027.stdout:8/301: dwrite d1/f21 [4194304,4194304] 0 2024-04-22T15:49:54.236 INFO:tasks.workunit.client.1.smithi027.stdout:0/350: dwrite d0/fc [0,4194304] 0 2024-04-22T15:49:54.238 INFO:tasks.workunit.client.1.smithi027.stdout:2/332: dwrite dd/df/d1b/d38/f5c [0,4194304] 0 2024-04-22T15:49:54.238 INFO:tasks.workunit.client.1.smithi027.stdout:8/302: mknod d1/d3/d1c/d56/d62/c6c 0 2024-04-22T15:49:54.241 INFO:tasks.workunit.client.1.smithi027.stdout:8/303: truncate d1/f18 2281619 0 2024-04-22T15:49:54.244 INFO:tasks.workunit.client.1.smithi027.stdout:5/386: dwrite d0/df/f45 [4194304,4194304] 0 2024-04-22T15:49:54.244 INFO:tasks.workunit.client.1.smithi027.stdout:8/304: chown d1/d3/d1c/d32/f3d 4053794 1 2024-04-22T15:49:54.245 INFO:tasks.workunit.client.1.smithi027.stdout:7/279: link d0/d11/c2a d0/d3f/c53 0 2024-04-22T15:49:54.249 INFO:tasks.workunit.client.1.smithi027.stdout:0/351: mknod d0/d34/c6f 0 2024-04-22T15:49:54.250 INFO:tasks.workunit.client.1.smithi027.stdout:0/352: dread - d0/d2a/d38/f6b zero size 2024-04-22T15:49:54.255 INFO:tasks.workunit.client.1.smithi027.stdout:2/333: mkdir dd/df/d1b/d38/d60 0 2024-04-22T15:49:54.260 INFO:tasks.workunit.client.1.smithi027.stdout:5/387: creat d0/d1a/d22/d72/f7b x:0 0 0 2024-04-22T15:49:54.262 INFO:tasks.workunit.client.1.smithi027.stdout:0/353: mkdir d0/d2a/d38/d70 0 2024-04-22T15:49:54.262 INFO:tasks.workunit.client.1.smithi027.stdout:5/388: fsync d0/d1a/d22/d26/f3b 0 2024-04-22T15:49:54.263 INFO:tasks.workunit.client.1.smithi027.stdout:2/334: dwrite dd/d43/f55 [0,4194304] 0 2024-04-22T15:49:54.269 INFO:tasks.workunit.client.1.smithi027.stdout:7/280: mkdir d0/d11/d1c/d54 0 2024-04-22T15:49:54.272 INFO:tasks.workunit.client.1.smithi027.stdout:2/335: creat dd/d1d/f61 x:0 0 0 2024-04-22T15:49:54.275 INFO:tasks.workunit.client.1.smithi027.stdout:5/389: truncate d0/d1a/d22/d26/d32/f54 585504 0 2024-04-22T15:49:54.276 INFO:tasks.workunit.client.1.smithi027.stdout:8/305: fsync d1/d3/d1c/f34 0 2024-04-22T15:49:54.277 INFO:tasks.workunit.client.1.smithi027.stdout:7/281: creat d0/d11/d1c/d2c/f55 x:0 0 0 2024-04-22T15:49:54.278 INFO:tasks.workunit.client.1.smithi027.stdout:2/336: chown dd/df/d1b/d1f/f36 9177 1 2024-04-22T15:49:54.278 INFO:tasks.workunit.client.1.smithi027.stdout:2/337: stat dd 0 2024-04-22T15:49:54.283 INFO:tasks.workunit.client.1.smithi027.stdout:7/282: mkdir d0/d11/d1c/d31/d56 0 2024-04-22T15:49:54.285 INFO:tasks.workunit.client.1.smithi027.stdout:0/354: truncate d0/f12 95132 0 2024-04-22T15:49:54.286 INFO:tasks.workunit.client.1.smithi027.stdout:2/338: creat dd/df/d1b/f62 x:0 0 0 2024-04-22T15:49:54.290 INFO:tasks.workunit.client.1.smithi027.stdout:8/306: unlink d1/d3/d1c/d28/d42/l4d 0 2024-04-22T15:49:54.292 INFO:tasks.workunit.client.1.smithi027.stdout:1/297: sync 2024-04-22T15:49:54.292 INFO:tasks.workunit.client.1.smithi027.stdout:9/266: sync 2024-04-22T15:49:54.293 INFO:tasks.workunit.client.1.smithi027.stdout:2/339: creat dd/d1d/f63 x:0 0 0 2024-04-22T15:49:54.294 INFO:tasks.workunit.client.1.smithi027.stdout:0/355: dread d0/d34/f3a [0,4194304] 0 2024-04-22T15:49:54.294 INFO:tasks.workunit.client.1.smithi027.stdout:5/390: dread d0/df/de/d24/f2e [0,4194304] 0 2024-04-22T15:49:54.295 INFO:tasks.workunit.client.1.smithi027.stdout:7/283: creat d0/d4c/d4e/f57 x:0 0 0 2024-04-22T15:49:54.296 INFO:tasks.workunit.client.1.smithi027.stdout:2/340: read dd/d43/f44 [125302,9750] 0 2024-04-22T15:49:54.298 INFO:tasks.workunit.client.1.smithi027.stdout:0/356: dread - d0/d34/d59/f5c zero size 2024-04-22T15:49:54.299 INFO:tasks.workunit.client.1.smithi027.stdout:1/298: mkdir d2/d5f/d64 0 2024-04-22T15:49:54.299 INFO:tasks.workunit.client.1.smithi027.stdout:2/341: readlink dd/df/d1b/d38/l50 0 2024-04-22T15:49:54.302 INFO:tasks.workunit.client.1.smithi027.stdout:8/307: dwrite d1/d3/f40 [4194304,4194304] 0 2024-04-22T15:49:54.302 INFO:tasks.workunit.client.1.smithi027.stdout:1/299: write d2/d30/f3c [892337,124611] 0 2024-04-22T15:49:54.303 INFO:tasks.workunit.client.1.smithi027.stdout:0/357: write d0/d2a/d38/f4a [908940,125734] 0 2024-04-22T15:49:54.303 INFO:tasks.workunit.client.1.smithi027.stdout:2/342: chown dd/l3a 87 1 2024-04-22T15:49:54.304 INFO:tasks.workunit.client.1.smithi027.stdout:1/300: readlink d2/d1d/l2d 0 2024-04-22T15:49:54.306 INFO:tasks.workunit.client.1.smithi027.stdout:1/301: chown d2/d30/l46 3 1 2024-04-22T15:49:54.306 INFO:tasks.workunit.client.1.smithi027.stdout:7/284: fdatasync d0/d11/d1c/d2c/f40 0 2024-04-22T15:49:54.308 INFO:tasks.workunit.client.1.smithi027.stdout:9/267: dread d0/dc/d12/f23 [0,4194304] 0 2024-04-22T15:49:54.308 INFO:tasks.workunit.client.1.smithi027.stdout:2/343: fdatasync dd/df/f49 0 2024-04-22T15:49:54.311 INFO:tasks.workunit.client.1.smithi027.stdout:5/391: mknod d0/df/de/d24/d1c/d3e/c7c 0 2024-04-22T15:49:54.314 INFO:tasks.workunit.client.1.smithi027.stdout:8/308: dwrite d1/d3/f40 [0,4194304] 0 2024-04-22T15:49:54.324 INFO:tasks.workunit.client.1.smithi027.stdout:7/285: rmdir d0/d11/d1c/d31 39 2024-04-22T15:49:54.330 INFO:tasks.workunit.client.1.smithi027.stdout:8/309: rmdir d1/d3/d1c/d56/d62 39 2024-04-22T15:49:54.330 INFO:tasks.workunit.client.1.smithi027.stdout:1/302: mkdir d2/d65 0 2024-04-22T15:49:54.330 INFO:tasks.workunit.client.1.smithi027.stdout:8/310: write d1/d1e/f22 [3087426,23826] 0 2024-04-22T15:49:54.331 INFO:tasks.workunit.client.1.smithi027.stdout:1/303: readlink d2/l58 0 2024-04-22T15:49:54.332 INFO:tasks.workunit.client.1.smithi027.stdout:1/304: read - d2/d36/f62 zero size 2024-04-22T15:49:54.339 INFO:tasks.workunit.client.1.smithi027.stdout:8/311: rename d1/d3/f3c to d1/d3/f6d 0 2024-04-22T15:49:54.339 INFO:tasks.workunit.client.1.smithi027.stdout:2/344: dread f7 [0,4194304] 0 2024-04-22T15:49:54.340 INFO:tasks.workunit.client.1.smithi027.stdout:0/358: truncate d0/d28/d37/f40 1055912 0 2024-04-22T15:49:54.343 INFO:tasks.workunit.client.1.smithi027.stdout:0/359: dread d0/d2a/f2b [0,4194304] 0 2024-04-22T15:49:54.344 INFO:tasks.workunit.client.1.smithi027.stdout:5/392: getdents d0/d1a/d22/d26/d32 0 2024-04-22T15:49:54.349 INFO:tasks.workunit.client.1.smithi027.stdout:7/286: creat d0/d11/d1c/d31/d56/f58 x:0 0 0 2024-04-22T15:49:54.350 INFO:tasks.workunit.client.1.smithi027.stdout:7/287: write d0/f37 [2614934,39345] 0 2024-04-22T15:49:54.350 INFO:tasks.workunit.client.1.smithi027.stdout:2/345: write dd/f22 [4160242,17511] 0 2024-04-22T15:49:54.351 INFO:tasks.workunit.client.1.smithi027.stdout:2/346: stat dd/df/f28 0 2024-04-22T15:49:54.352 INFO:tasks.workunit.client.1.smithi027.stdout:8/312: symlink d1/d3/d1c/d32/d41/l6e 0 2024-04-22T15:49:54.352 INFO:tasks.workunit.client.1.smithi027.stdout:2/347: chown dd/d43/f44 30838001 1 2024-04-22T15:49:54.354 INFO:tasks.workunit.client.1.smithi027.stdout:2/348: read dd/df/d1b/f40 [1513004,79026] 0 2024-04-22T15:49:54.355 INFO:tasks.workunit.client.1.smithi027.stdout:9/268: getdents d0/dc/d12/d30 0 2024-04-22T15:49:54.356 INFO:tasks.workunit.client.1.smithi027.stdout:9/269: write d0/dc/de/f41 [1179787,35460] 0 2024-04-22T15:49:54.359 INFO:tasks.workunit.client.1.smithi027.stdout:1/305: truncate d2/d36/d38/d3d/f4e 2797441 0 2024-04-22T15:49:54.362 INFO:tasks.workunit.client.1.smithi027.stdout:5/393: unlink d0/df/d3d/f4e 0 2024-04-22T15:49:54.366 INFO:tasks.workunit.client.1.smithi027.stdout:0/360: dwrite d0/d34/d59/f62 [0,4194304] 0 2024-04-22T15:49:54.371 INFO:tasks.workunit.client.1.smithi027.stdout:8/313: symlink d1/d3/d1c/d43/l6f 0 2024-04-22T15:49:54.374 INFO:tasks.workunit.client.1.smithi027.stdout:3/404: sync 2024-04-22T15:49:54.375 INFO:tasks.workunit.client.1.smithi027.stdout:9/270: chown d0/dc/df/l3c 11768608 1 2024-04-22T15:49:54.375 INFO:tasks.workunit.client.1.smithi027.stdout:9/271: stat d0/d8/c1b 0 2024-04-22T15:49:54.380 INFO:tasks.workunit.client.1.smithi027.stdout:5/394: rename d0/df/f7a to d0/df/de/d24/d1c/d3e/d55/f7d 0 2024-04-22T15:49:54.381 INFO:tasks.workunit.client.1.smithi027.stdout:7/288: creat d0/d16/d2f/f59 x:0 0 0 2024-04-22T15:49:54.382 INFO:tasks.workunit.client.1.smithi027.stdout:0/361: creat d0/d34/f71 x:0 0 0 2024-04-22T15:49:54.384 INFO:tasks.workunit.client.1.smithi027.stdout:8/314: unlink d1/d3/f6 0 2024-04-22T15:49:54.386 INFO:tasks.workunit.client.1.smithi027.stdout:2/349: mknod dd/c64 0 2024-04-22T15:49:54.388 INFO:tasks.workunit.client.1.smithi027.stdout:3/405: creat d28/d69/f86 x:0 0 0 2024-04-22T15:49:54.389 INFO:tasks.workunit.client.1.smithi027.stdout:3/406: fsync d28/d4b/f59 0 2024-04-22T15:49:54.391 INFO:tasks.workunit.client.1.smithi027.stdout:1/306: creat d2/d65/f66 x:0 0 0 2024-04-22T15:49:54.394 INFO:tasks.workunit.client.1.smithi027.stdout:7/289: unlink d0/d11/d3a/l44 0 2024-04-22T15:49:54.396 INFO:tasks.workunit.client.1.smithi027.stdout:3/407: dwrite d28/f48 [0,4194304] 0 2024-04-22T15:49:54.398 INFO:tasks.workunit.client.1.smithi027.stdout:3/408: chown d28/f3c 1679 1 2024-04-22T15:49:54.399 INFO:tasks.workunit.client.1.smithi027.stdout:8/315: rename d1/d1e/f22 to d1/d1e/d47/f70 0 2024-04-22T15:49:54.400 INFO:tasks.workunit.client.1.smithi027.stdout:3/409: write d28/f49 [926090,66413] 0 2024-04-22T15:49:54.401 INFO:tasks.workunit.client.1.smithi027.stdout:1/307: dwrite d2/d30/f3c [0,4194304] 0 2024-04-22T15:49:54.404 INFO:tasks.workunit.client.1.smithi027.stdout:2/350: unlink dd/d1d/d4b/f5d 0 2024-04-22T15:49:54.405 INFO:tasks.workunit.client.1.smithi027.stdout:8/316: chown d1/d3/d1c/d32/d41/d52/d61 8243356 1 2024-04-22T15:49:54.406 INFO:tasks.workunit.client.1.smithi027.stdout:9/272: mknod d0/dc/c5e 0 2024-04-22T15:49:54.407 INFO:tasks.workunit.client.1.smithi027.stdout:8/317: write d1/d3/f59 [1775005,125754] 0 2024-04-22T15:49:54.416 INFO:tasks.workunit.client.1.smithi027.stdout:0/362: dread d0/d28/d37/f40 [0,4194304] 0 2024-04-22T15:49:54.417 INFO:tasks.workunit.client.1.smithi027.stdout:3/410: rename d28/d4b/d5c/f6b to d28/d5d/d79/f87 0 2024-04-22T15:49:54.420 INFO:tasks.workunit.client.1.smithi027.stdout:7/290: mkdir d0/d11/d1c/d54/d5a 0 2024-04-22T15:49:54.421 INFO:tasks.workunit.client.1.smithi027.stdout:7/291: readlink d0/l32 0 2024-04-22T15:49:54.422 INFO:tasks.workunit.client.1.smithi027.stdout:7/292: chown d0/d16/d52 51956320 1 2024-04-22T15:49:54.423 INFO:tasks.workunit.client.1.smithi027.stdout:1/308: dwrite d2/f37 [0,4194304] 0 2024-04-22T15:49:54.426 INFO:tasks.workunit.client.1.smithi027.stdout:2/351: getdents dd/d43 0 2024-04-22T15:49:54.428 INFO:tasks.workunit.client.1.smithi027.stdout:1/309: creat d2/d65/f67 x:0 0 0 2024-04-22T15:49:54.431 INFO:tasks.workunit.client.1.smithi027.stdout:7/293: dwrite d0/d11/fe [0,4194304] 0 2024-04-22T15:49:54.433 INFO:tasks.workunit.client.1.smithi027.stdout:3/411: link d28/d4b/d52/f61 d28/f88 0 2024-04-22T15:49:54.434 INFO:tasks.workunit.client.1.smithi027.stdout:0/363: getdents d0/d28/d63 0 2024-04-22T15:49:54.435 INFO:tasks.workunit.client.1.smithi027.stdout:0/364: readlink d0/d28/d5f/l6e 0 2024-04-22T15:49:54.439 INFO:tasks.workunit.client.1.smithi027.stdout:7/294: creat d0/d3f/f5b x:0 0 0 2024-04-22T15:49:54.441 INFO:tasks.workunit.client.1.smithi027.stdout:3/412: mknod d28/d4b/d5c/c89 0 2024-04-22T15:49:54.442 INFO:tasks.workunit.client.1.smithi027.stdout:3/413: chown d28/f3b 61435841 1 2024-04-22T15:49:54.442 INFO:tasks.workunit.client.1.smithi027.stdout:8/318: getdents d1/d1e 0 2024-04-22T15:49:54.444 INFO:tasks.workunit.client.1.smithi027.stdout:2/352: creat dd/df/d1b/d38/d60/f65 x:0 0 0 2024-04-22T15:49:54.446 INFO:tasks.workunit.client.1.smithi027.stdout:0/365: rename d0/c32 to d0/d2a/d58/c72 0 2024-04-22T15:49:54.448 INFO:tasks.workunit.client.1.smithi027.stdout:0/366: read - d0/d2a/d38/f55 zero size 2024-04-22T15:49:54.449 INFO:tasks.workunit.client.1.smithi027.stdout:8/319: dwrite d1/f7 [0,4194304] 0 2024-04-22T15:49:54.454 INFO:tasks.workunit.client.1.smithi027.stdout:3/414: mknod d28/d4b/c8a 0 2024-04-22T15:49:54.455 INFO:tasks.workunit.client.1.smithi027.stdout:3/415: chown f1d 1074011 1 2024-04-22T15:49:54.457 INFO:tasks.workunit.client.1.smithi027.stdout:1/310: creat d2/f68 x:0 0 0 2024-04-22T15:49:54.458 INFO:tasks.workunit.client.1.smithi027.stdout:0/367: truncate d0/d34/d59/f5d 1264818 0 2024-04-22T15:49:54.458 INFO:tasks.workunit.client.1.smithi027.stdout:8/320: dwrite d1/d3/d1c/d49/f5d [4194304,4194304] 0 2024-04-22T15:49:54.461 INFO:tasks.workunit.client.1.smithi027.stdout:1/311: read d2/d36/f55 [485288,86685] 0 2024-04-22T15:49:54.464 INFO:tasks.workunit.client.1.smithi027.stdout:2/353: mknod dd/d1d/d5a/c66 0 2024-04-22T15:49:54.469 INFO:tasks.workunit.client.1.smithi027.stdout:9/273: dwrite d0/dc/de/f49 [0,4194304] 0 2024-04-22T15:49:54.471 INFO:tasks.workunit.client.1.smithi027.stdout:9/274: read d0/dc/f37 [371924,51965] 0 2024-04-22T15:49:54.471 INFO:tasks.workunit.client.1.smithi027.stdout:8/321: creat d1/d3/d1c/d43/f71 x:0 0 0 2024-04-22T15:49:54.473 INFO:tasks.workunit.client.1.smithi027.stdout:8/322: fdatasync d1/d3/d1c/d56/f6a 0 2024-04-22T15:49:54.474 INFO:tasks.workunit.client.1.smithi027.stdout:8/323: write d1/d3/d1c/d49/f63 [738835,41693] 0 2024-04-22T15:49:54.476 INFO:tasks.workunit.client.1.smithi027.stdout:1/312: mkdir d2/d36/d38/d69 0 2024-04-22T15:49:54.481 INFO:tasks.workunit.client.1.smithi027.stdout:8/324: dwrite d1/d3/f6b [0,4194304] 0 2024-04-22T15:49:54.485 INFO:tasks.workunit.client.1.smithi027.stdout:8/325: truncate d1/d3/d1c/d49/f63 1066212 0 2024-04-22T15:49:54.487 INFO:tasks.workunit.client.1.smithi027.stdout:7/295: dwrite d0/f14 [0,4194304] 0 2024-04-22T15:49:54.489 INFO:tasks.workunit.client.1.smithi027.stdout:7/296: write d0/d3f/f5b [371904,129790] 0 2024-04-22T15:49:54.490 INFO:tasks.workunit.client.1.smithi027.stdout:0/368: link d0/d2a/f61 d0/d28/d5f/f73 0 2024-04-22T15:49:54.490 INFO:tasks.workunit.client.1.smithi027.stdout:7/297: chown d0/d16 395314 1 2024-04-22T15:49:54.496 INFO:tasks.workunit.client.1.smithi027.stdout:1/313: symlink d2/d65/l6a 0 2024-04-22T15:49:54.501 INFO:tasks.workunit.client.1.smithi027.stdout:2/354: mknod dd/c67 0 2024-04-22T15:49:54.502 INFO:tasks.workunit.client.1.smithi027.stdout:6/357: sync 2024-04-22T15:49:54.503 INFO:tasks.workunit.client.1.smithi027.stdout:4/346: sync 2024-04-22T15:49:54.509 INFO:tasks.workunit.client.1.smithi027.stdout:4/347: dwrite d5/d45/f59 [0,4194304] 0 2024-04-22T15:49:54.511 INFO:tasks.workunit.client.1.smithi027.stdout:0/369: mknod d0/d34/c74 0 2024-04-22T15:49:54.514 INFO:tasks.workunit.client.1.smithi027.stdout:7/298: write d0/d11/d1c/d31/f47 [2395801,117975] 0 2024-04-22T15:49:54.518 INFO:tasks.workunit.client.1.smithi027.stdout:5/395: sync 2024-04-22T15:49:54.521 INFO:tasks.workunit.client.1.smithi027.stdout:2/355: creat dd/d1d/d4b/f68 x:0 0 0 2024-04-22T15:49:54.522 INFO:tasks.workunit.client.1.smithi027.stdout:6/358: creat d5/dc/dd/d52/f7e x:0 0 0 2024-04-22T15:49:54.523 INFO:tasks.workunit.client.1.smithi027.stdout:3/416: link d28/d4b/f5b d28/d5d/f8b 0 2024-04-22T15:49:54.529 INFO:tasks.workunit.client.1.smithi027.stdout:5/396: symlink d0/d1a/d22/l7e 0 2024-04-22T15:49:54.530 INFO:tasks.workunit.client.1.smithi027.stdout:1/314: fsync d2/d36/f4b 0 2024-04-22T15:49:54.530 INFO:tasks.workunit.client.1.smithi027.stdout:2/356: mknod dd/d1d/c69 0 2024-04-22T15:49:54.530 INFO:tasks.workunit.client.1.smithi027.stdout:2/357: chown dd/df/d1b/d1f/f30 14055837 1 2024-04-22T15:49:54.531 INFO:tasks.workunit.client.1.smithi027.stdout:6/359: creat d5/d30/d54/f7f x:0 0 0 2024-04-22T15:49:54.531 INFO:tasks.workunit.client.1.smithi027.stdout:6/360: dread - d5/dc/f74 zero size 2024-04-22T15:49:54.535 INFO:tasks.workunit.client.1.smithi027.stdout:0/370: mknod d0/d2a/d5e/c75 0 2024-04-22T15:49:54.537 INFO:tasks.workunit.client.1.smithi027.stdout:2/358: dwrite dd/df/f18 [0,4194304] 0 2024-04-22T15:49:54.538 INFO:tasks.workunit.client.1.smithi027.stdout:4/348: rename d5/cd to d5/de/d30/d5d/c8e 0 2024-04-22T15:49:54.540 INFO:tasks.workunit.client.1.smithi027.stdout:4/349: write d5/de/d2a/f49 [2163505,6439] 0 2024-04-22T15:49:54.541 INFO:tasks.workunit.client.1.smithi027.stdout:9/275: link d0/dc/de/f24 d0/dc/f5f 0 2024-04-22T15:49:54.546 INFO:tasks.workunit.client.1.smithi027.stdout:5/397: creat d0/df/de/d24/d62/f7f x:0 0 0 2024-04-22T15:49:54.552 INFO:tasks.workunit.client.1.smithi027.stdout:3/417: mknod d28/c8c 0 2024-04-22T15:49:54.553 INFO:tasks.workunit.client.1.smithi027.stdout:8/326: getdents d1 0 2024-04-22T15:49:54.557 INFO:tasks.workunit.client.1.smithi027.stdout:3/418: dwrite d28/d4b/f53 [0,4194304] 0 2024-04-22T15:49:54.562 INFO:tasks.workunit.client.1.smithi027.stdout:7/299: creat d0/d11/f5c x:0 0 0 2024-04-22T15:49:54.563 INFO:tasks.workunit.client.1.smithi027.stdout:7/300: chown d0/d16/f3d 1106 1 2024-04-22T15:49:54.566 INFO:tasks.workunit.client.1.smithi027.stdout:9/276: mknod d0/d8/d5c/c60 0 2024-04-22T15:49:54.567 INFO:tasks.workunit.client.1.smithi027.stdout:5/398: symlink d0/df/de/d24/d1c/d3e/d55/l80 0 2024-04-22T15:49:54.568 INFO:tasks.workunit.client.1.smithi027.stdout:8/327: rmdir d1/d3/d1c/d28/d42 39 2024-04-22T15:49:54.572 INFO:tasks.workunit.client.1.smithi027.stdout:3/419: truncate f10 7414706 0 2024-04-22T15:49:54.573 INFO:tasks.workunit.client.1.smithi027.stdout:0/371: symlink d0/d2a/d38/d70/l76 0 2024-04-22T15:49:54.574 INFO:tasks.workunit.client.1.smithi027.stdout:3/420: read f1c [1269893,31353] 0 2024-04-22T15:49:54.574 INFO:tasks.workunit.client.1.smithi027.stdout:2/359: mknod dd/c6a 0 2024-04-22T15:49:54.576 INFO:tasks.workunit.client.1.smithi027.stdout:3/421: chown l6 0 1 2024-04-22T15:49:54.576 INFO:tasks.workunit.client.1.smithi027.stdout:8/328: dwrite d1/d3/d1c/d32/f3d [0,4194304] 0 2024-04-22T15:49:54.576 INFO:tasks.workunit.client.1.smithi027.stdout:0/372: write d0/d34/d59/f5c [252030,83053] 0 2024-04-22T15:49:54.577 INFO:tasks.workunit.client.1.smithi027.stdout:4/350: unlink d5/f6 0 2024-04-22T15:49:54.578 INFO:tasks.workunit.client.1.smithi027.stdout:0/373: read - d0/d2a/d38/f55 zero size 2024-04-22T15:49:54.580 INFO:tasks.workunit.client.1.smithi027.stdout:7/301: mknod d0/d11/d1c/d54/c5d 0 2024-04-22T15:49:54.583 INFO:tasks.workunit.client.1.smithi027.stdout:9/277: fdatasync d0/f36 0 2024-04-22T15:49:54.590 INFO:tasks.workunit.client.1.smithi027.stdout:1/315: symlink d2/d36/d38/l6b 0 2024-04-22T15:49:54.591 INFO:tasks.workunit.client.1.smithi027.stdout:5/399: mkdir d0/df/d3d/d81 0 2024-04-22T15:49:54.591 INFO:tasks.workunit.client.1.smithi027.stdout:5/400: chown d0/df/d10/l6e 19394 1 2024-04-22T15:49:54.593 INFO:tasks.workunit.client.1.smithi027.stdout:5/401: write d0/df/de/d52/f77 [1034177,69110] 0 2024-04-22T15:49:54.594 INFO:tasks.workunit.client.1.smithi027.stdout:6/361: link d5/d30/d54/c5e d5/db/d2e/d59/c80 0 2024-04-22T15:49:54.595 INFO:tasks.workunit.client.1.smithi027.stdout:6/362: dread - d5/dc/dd/d52/f7e zero size 2024-04-22T15:49:54.598 INFO:tasks.workunit.client.1.smithi027.stdout:2/360: fdatasync dd/d43/f44 0 2024-04-22T15:49:54.600 INFO:tasks.workunit.client.1.smithi027.stdout:2/361: write dd/d1d/f45 [3478670,77570] 0 2024-04-22T15:49:54.600 INFO:tasks.workunit.client.1.smithi027.stdout:2/362: readlink dd/l34 0 2024-04-22T15:49:54.605 INFO:tasks.workunit.client.1.smithi027.stdout:3/422: rename l19 to d28/d5d/d62/l8d 0 2024-04-22T15:49:54.613 INFO:tasks.workunit.client.1.smithi027.stdout:9/278: creat d0/dc/df/f61 x:0 0 0 2024-04-22T15:49:54.615 INFO:tasks.workunit.client.1.smithi027.stdout:4/351: dread d5/f9 [0,4194304] 0 2024-04-22T15:49:54.615 INFO:tasks.workunit.client.1.smithi027.stdout:0/374: write d0/d34/f3a [748204,53015] 0 2024-04-22T15:49:54.618 INFO:tasks.workunit.client.1.smithi027.stdout:0/375: write d0/d2a/d38/f6b [905435,53876] 0 2024-04-22T15:49:54.628 INFO:tasks.workunit.client.1.smithi027.stdout:9/279: dread d0/dc/f3d [0,4194304] 0 2024-04-22T15:49:54.630 INFO:tasks.workunit.client.1.smithi027.stdout:2/363: creat dd/d1d/f6b x:0 0 0 2024-04-22T15:49:54.630 INFO:tasks.workunit.client.1.smithi027.stdout:8/329: mknod d1/d3/d1c/d56/c72 0 2024-04-22T15:49:54.630 INFO:tasks.workunit.client.1.smithi027.stdout:9/280: stat d0/d8/f48 0 2024-04-22T15:49:54.634 INFO:tasks.workunit.client.1.smithi027.stdout:0/376: dwrite d0/f1 [4194304,4194304] 0 2024-04-22T15:49:54.634 INFO:tasks.workunit.client.1.smithi027.stdout:4/352: creat d5/d7/f8f x:0 0 0 2024-04-22T15:49:54.635 INFO:tasks.workunit.client.1.smithi027.stdout:0/377: chown d0/d2a/f48 29547793 1 2024-04-22T15:49:54.636 INFO:tasks.workunit.client.1.smithi027.stdout:4/353: readlink d5/d11/d1d/l6b 0 2024-04-22T15:49:54.640 INFO:tasks.workunit.client.1.smithi027.stdout:6/363: creat d5/dc/d17/d53/f81 x:0 0 0 2024-04-22T15:49:54.641 INFO:tasks.workunit.client.1.smithi027.stdout:6/364: fsync d5/f75 0 2024-04-22T15:49:54.643 INFO:tasks.workunit.client.1.smithi027.stdout:2/364: unlink dd/f2d 0 2024-04-22T15:49:54.645 INFO:tasks.workunit.client.1.smithi027.stdout:5/402: truncate d0/d1a/d22/d26/d32/f54 80314 0 2024-04-22T15:49:54.646 INFO:tasks.workunit.client.1.smithi027.stdout:5/403: read d0/f8 [1861777,21556] 0 2024-04-22T15:49:54.648 INFO:tasks.workunit.client.1.smithi027.stdout:8/330: symlink d1/d3/d36/l73 0 2024-04-22T15:49:54.652 INFO:tasks.workunit.client.1.smithi027.stdout:1/316: mknod d2/d36/d38/c6c 0 2024-04-22T15:49:54.658 INFO:tasks.workunit.client.1.smithi027.stdout:7/302: getdents d0/d11/d1c/d31 0 2024-04-22T15:49:54.659 INFO:tasks.workunit.client.1.smithi027.stdout:2/365: creat dd/df/d1b/d38/f6c x:0 0 0 2024-04-22T15:49:54.663 INFO:tasks.workunit.client.1.smithi027.stdout:5/404: creat d0/d1a/f82 x:0 0 0 2024-04-22T15:49:54.665 INFO:tasks.workunit.client.1.smithi027.stdout:8/331: chown d1/d3/d1c/l2d 63661628 1 2024-04-22T15:49:54.670 INFO:tasks.workunit.client.1.smithi027.stdout:8/332: dwrite d1/d3/f1a [0,4194304] 0 2024-04-22T15:49:54.673 INFO:tasks.workunit.client.1.smithi027.stdout:9/281: truncate d0/d8/f18 5249860 0 2024-04-22T15:49:54.674 INFO:tasks.workunit.client.1.smithi027.stdout:6/365: symlink d5/dc/l82 0 2024-04-22T15:49:54.676 INFO:tasks.workunit.client.1.smithi027.stdout:4/354: truncate d5/d45/f59 2079766 0 2024-04-22T15:49:54.678 INFO:tasks.workunit.client.1.smithi027.stdout:4/355: write d5/d11/d1d/f27 [3140450,129390] 0 2024-04-22T15:49:54.680 INFO:tasks.workunit.client.1.smithi027.stdout:4/356: truncate d5/d11/d6c/f8b 661869 0 2024-04-22T15:49:54.681 INFO:tasks.workunit.client.1.smithi027.stdout:4/357: stat d5/d7/f8f 0 2024-04-22T15:49:54.682 INFO:tasks.workunit.client.1.smithi027.stdout:4/358: truncate d5/d11/f7a 109332 0 2024-04-22T15:49:54.689 INFO:tasks.workunit.client.1.smithi027.stdout:0/378: link d0/d2a/l4d d0/d2a/d58/l77 0 2024-04-22T15:49:54.691 INFO:tasks.workunit.client.1.smithi027.stdout:0/379: dread - d0/d34/f71 zero size 2024-04-22T15:49:54.691 INFO:tasks.workunit.client.1.smithi027.stdout:7/303: dwrite d0/d3f/f48 [0,4194304] 0 2024-04-22T15:49:54.693 INFO:tasks.workunit.client.1.smithi027.stdout:0/380: write d0/d2a/d38/f4a [232914,101766] 0 2024-04-22T15:49:54.698 INFO:tasks.workunit.client.1.smithi027.stdout:2/366: fdatasync dd/df/d1b/f25 0 2024-04-22T15:49:54.700 INFO:tasks.workunit.client.1.smithi027.stdout:4/359: symlink d5/de/l90 0 2024-04-22T15:49:54.702 INFO:tasks.workunit.client.1.smithi027.stdout:5/405: mkdir d0/df/de/d24/d83 0 2024-04-22T15:49:54.707 INFO:tasks.workunit.client.1.smithi027.stdout:1/317: creat d2/d36/d38/d3d/d44/f6d x:0 0 0 2024-04-22T15:49:54.709 INFO:tasks.workunit.client.1.smithi027.stdout:5/406: dwrite d0/df/de/d40/f6a [0,4194304] 0 2024-04-22T15:49:54.713 INFO:tasks.workunit.client.1.smithi027.stdout:8/333: symlink d1/d3/d1c/d28/d42/l74 0 2024-04-22T15:49:54.714 INFO:tasks.workunit.client.1.smithi027.stdout:8/334: stat d1/f21 0 2024-04-22T15:49:54.715 INFO:tasks.workunit.client.1.smithi027.stdout:8/335: write d1/d3/d1c/f34 [1452345,114832] 0 2024-04-22T15:49:54.715 INFO:tasks.workunit.client.1.smithi027.stdout:8/336: readlink d1/d3/l12 0 2024-04-22T15:49:54.722 INFO:tasks.workunit.client.1.smithi027.stdout:6/366: unlink d5/dc/d20/d2b/c55 0 2024-04-22T15:49:54.724 INFO:tasks.workunit.client.1.smithi027.stdout:2/367: creat dd/df/d1b/d38/d60/f6d x:0 0 0 2024-04-22T15:49:54.725 INFO:tasks.workunit.client.1.smithi027.stdout:2/368: chown dd/df/d1b/d38/f5c 364813 1 2024-04-22T15:49:54.726 INFO:tasks.workunit.client.1.smithi027.stdout:4/360: symlink d5/de/d30/l91 0 2024-04-22T15:49:54.733 INFO:tasks.workunit.client.1.smithi027.stdout:6/367: creat d5/dc/d26/d3d/f83 x:0 0 0 2024-04-22T15:49:54.734 INFO:tasks.workunit.client.1.smithi027.stdout:6/368: fdatasync d5/dc/d26/f2d 0 2024-04-22T15:49:54.738 INFO:tasks.workunit.client.1.smithi027.stdout:6/369: write d5/db/f11 [489822,30506] 0 2024-04-22T15:49:54.738 INFO:tasks.workunit.client.1.smithi027.stdout:2/369: unlink dd/f22 0 2024-04-22T15:49:54.741 INFO:tasks.workunit.client.1.smithi027.stdout:1/318: symlink d2/d5f/d64/l6e 0 2024-04-22T15:49:54.741 INFO:tasks.workunit.client.1.smithi027.stdout:2/370: dread dd/d43/f55 [0,4194304] 0 2024-04-22T15:49:54.742 INFO:tasks.workunit.client.1.smithi027.stdout:2/371: write dd/df/f18 [4840499,44455] 0 2024-04-22T15:49:54.751 INFO:tasks.workunit.client.1.smithi027.stdout:6/370: fdatasync d5/db/d2e/ff 0 2024-04-22T15:49:54.753 INFO:tasks.workunit.client.1.smithi027.stdout:5/407: dwrite d0/d1a/d22/d26/f48 [0,4194304] 0 2024-04-22T15:49:54.761 INFO:tasks.workunit.client.1.smithi027.stdout:6/371: dwrite d5/dc/d26/d3d/f83 [0,4194304] 0 2024-04-22T15:49:54.762 INFO:tasks.workunit.client.1.smithi027.stdout:6/372: readlink d5/dc/d20/d2b/d34/l68 0 2024-04-22T15:49:54.763 INFO:tasks.workunit.client.1.smithi027.stdout:7/304: getdents d0/d3f 0 2024-04-22T15:49:54.765 INFO:tasks.workunit.client.1.smithi027.stdout:7/305: write d0/d11/d1c/d2c/f38 [120961,110671] 0 2024-04-22T15:49:54.769 INFO:tasks.workunit.client.1.smithi027.stdout:5/408: unlink d0/f3 0 2024-04-22T15:49:54.779 INFO:tasks.workunit.client.1.smithi027.stdout:8/337: truncate d1/f7 2934639 0 2024-04-22T15:49:54.781 INFO:tasks.workunit.client.1.smithi027.stdout:5/409: symlink d0/df/de/d40/l84 0 2024-04-22T15:49:54.783 INFO:tasks.workunit.client.1.smithi027.stdout:5/410: truncate d0/df/d3d/f44 692821 0 2024-04-22T15:49:54.783 INFO:tasks.workunit.client.1.smithi027.stdout:6/373: mkdir d5/db/d84 0 2024-04-22T15:49:54.786 INFO:tasks.workunit.client.1.smithi027.stdout:2/372: link f7 dd/d1d/f6e 0 2024-04-22T15:49:54.788 INFO:tasks.workunit.client.1.smithi027.stdout:7/306: getdents d0/d11/d1c/d54/d5a 0 2024-04-22T15:49:54.789 INFO:tasks.workunit.client.1.smithi027.stdout:7/307: stat d0/d11/c2b 0 2024-04-22T15:49:54.793 INFO:tasks.workunit.client.1.smithi027.stdout:8/338: dread d1/d3/f59 [0,4194304] 0 2024-04-22T15:49:54.794 INFO:tasks.workunit.client.1.smithi027.stdout:5/411: mknod d0/df/de/d52/c85 0 2024-04-22T15:49:54.795 INFO:tasks.workunit.client.1.smithi027.stdout:8/339: chown d1/d3/f59 397 1 2024-04-22T15:49:54.796 INFO:tasks.workunit.client.1.smithi027.stdout:7/308: dwrite d0/d4c/d4e/f57 [0,4194304] 0 2024-04-22T15:49:54.797 INFO:tasks.workunit.client.1.smithi027.stdout:8/340: fsync d1/d3/d1c/d28/f51 0 2024-04-22T15:49:54.799 INFO:tasks.workunit.client.1.smithi027.stdout:6/374: dread d5/f29 [0,4194304] 0 2024-04-22T15:49:54.799 INFO:tasks.workunit.client.1.smithi027.stdout:8/341: chown d1/d1e/d47/f70 13 1 2024-04-22T15:49:54.801 INFO:tasks.workunit.client.1.smithi027.stdout:8/342: write d1/d3/d1c/d49/f5d [8751094,85645] 0 2024-04-22T15:49:54.807 INFO:tasks.workunit.client.1.smithi027.stdout:4/361: write d5/d7/f1f [403417,95210] 0 2024-04-22T15:49:54.808 INFO:tasks.workunit.client.1.smithi027.stdout:8/343: dwrite d1/f18 [0,4194304] 0 2024-04-22T15:49:54.825 INFO:tasks.workunit.client.1.smithi027.stdout:2/373: creat dd/f6f x:0 0 0 2024-04-22T15:49:54.826 INFO:tasks.workunit.client.1.smithi027.stdout:7/309: getdents d0/d3f 0 2024-04-22T15:49:54.827 INFO:tasks.workunit.client.1.smithi027.stdout:4/362: dwrite d5/f25 [0,4194304] 0 2024-04-22T15:49:54.828 INFO:tasks.workunit.client.1.smithi027.stdout:7/310: truncate d0/f3e 671689 0 2024-04-22T15:49:54.832 INFO:tasks.workunit.client.1.smithi027.stdout:2/374: dread dd/f4d [0,4194304] 0 2024-04-22T15:49:54.833 INFO:tasks.workunit.client.1.smithi027.stdout:2/375: chown dd/d1d/d4b/d52 25153627 1 2024-04-22T15:49:54.837 INFO:tasks.workunit.client.1.smithi027.stdout:4/363: creat d5/d11/d1d/d7b/f92 x:0 0 0 2024-04-22T15:49:54.838 INFO:tasks.workunit.client.1.smithi027.stdout:4/364: write d5/d11/d1d/f27 [1730264,99878] 0 2024-04-22T15:49:54.839 INFO:tasks.workunit.client.1.smithi027.stdout:4/365: chown d5/d11/f36 0 1 2024-04-22T15:49:54.841 INFO:tasks.workunit.client.1.smithi027.stdout:4/366: dread d5/de/f21 [0,4194304] 0 2024-04-22T15:49:54.842 INFO:tasks.workunit.client.1.smithi027.stdout:4/367: fdatasync d5/de/f16 0 2024-04-22T15:49:54.843 INFO:tasks.workunit.client.1.smithi027.stdout:7/311: symlink d0/d16/d52/l5e 0 2024-04-22T15:49:54.846 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:54 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:54.847 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:54 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:54.849 INFO:tasks.workunit.client.1.smithi027.stdout:7/312: creat d0/d11/d1c/d31/f5f x:0 0 0 2024-04-22T15:49:54.854 INFO:tasks.workunit.client.1.smithi027.stdout:7/313: symlink d0/d16/d2f/l60 0 2024-04-22T15:49:54.854 INFO:tasks.workunit.client.1.smithi027.stdout:7/314: stat d0/d4c/d4e/f57 0 2024-04-22T15:49:54.860 INFO:tasks.workunit.client.1.smithi027.stdout:7/315: dwrite d0/f29 [0,4194304] 0 2024-04-22T15:49:54.865 INFO:tasks.workunit.client.1.smithi027.stdout:5/412: rename d0/df/de/d52 to d0/d1a/d22/d72/d86 0 2024-04-22T15:49:54.867 INFO:tasks.workunit.client.1.smithi027.stdout:7/316: dread d0/d26/f42 [0,4194304] 0 2024-04-22T15:49:54.869 INFO:tasks.workunit.client.1.smithi027.stdout:6/375: rename d5/dc/d17/f4b to d5/db/d2e/d59/f85 0 2024-04-22T15:49:54.876 INFO:tasks.workunit.client.1.smithi027.stdout:7/317: mknod d0/d16/d2f/c61 0 2024-04-22T15:49:54.876 INFO:tasks.workunit.client.1.smithi027.stdout:6/376: dread d5/dc/d20/d2b/d34/f65 [0,4194304] 0 2024-04-22T15:49:54.880 INFO:tasks.workunit.client.1.smithi027.stdout:6/377: read d5/db/f11 [2744248,68729] 0 2024-04-22T15:49:54.882 INFO:tasks.workunit.client.1.smithi027.stdout:6/378: stat d5/dc/l5f 0 2024-04-22T15:49:54.883 INFO:tasks.workunit.client.1.smithi027.stdout:7/318: dwrite d0/f37 [0,4194304] 0 2024-04-22T15:49:54.883 INFO:tasks.workunit.client.1.smithi027.stdout:6/379: fsync d5/dc/d20/f3f 0 2024-04-22T15:49:54.885 INFO:tasks.workunit.client.1.smithi027.stdout:7/319: write d0/d11/f4d [2143132,120388] 0 2024-04-22T15:49:54.902 INFO:tasks.workunit.client.1.smithi027.stdout:2/376: rename dd/d1d to dd/df/d70 0 2024-04-22T15:49:54.904 INFO:tasks.workunit.client.1.smithi027.stdout:7/320: getdents d0/d11/d3a/d46 0 2024-04-22T15:49:54.910 INFO:tasks.workunit.client.1.smithi027.stdout:7/321: mkdir d0/d11/d1c/d31/d62 0 2024-04-22T15:49:54.916 INFO:tasks.workunit.client.1.smithi027.stdout:2/377: dread dd/df/f28 [0,4194304] 0 2024-04-22T15:49:54.916 INFO:tasks.workunit.client.1.smithi027.stdout:3/423: sync 2024-04-22T15:49:54.924 INFO:tasks.workunit.client.1.smithi027.stdout:7/322: link d0/f39 d0/d49/f63 0 2024-04-22T15:49:54.926 INFO:tasks.workunit.client.1.smithi027.stdout:3/424: dwrite f1c [0,4194304] 0 2024-04-22T15:49:54.926 INFO:tasks.workunit.client.1.smithi027.stdout:7/323: write d0/fb [2803675,67607] 0 2024-04-22T15:49:54.927 INFO:tasks.workunit.client.1.smithi027.stdout:3/425: write d28/f38 [1441315,80927] 0 2024-04-22T15:49:54.927 INFO:tasks.workunit.client.1.smithi027.stdout:7/324: chown d0/d11/d3a/f45 1 1 2024-04-22T15:49:54.928 INFO:tasks.workunit.client.1.smithi027.stdout:4/368: write d5/d7/f10 [1676451,18570] 0 2024-04-22T15:49:54.929 INFO:tasks.workunit.client.1.smithi027.stdout:4/369: chown d5/c77 15474 1 2024-04-22T15:49:54.942 INFO:tasks.workunit.client.1.smithi027.stdout:7/325: symlink d0/d3f/l64 0 2024-04-22T15:49:54.942 INFO:tasks.workunit.client.1.smithi027.stdout:7/326: chown d0/d11/d1c/l34 56094157 1 2024-04-22T15:49:54.947 INFO:tasks.workunit.client.1.smithi027.stdout:7/327: mkdir d0/d16/d65 0 2024-04-22T15:49:54.948 INFO:tasks.workunit.client.1.smithi027.stdout:7/328: dread - d0/d16/f3d zero size 2024-04-22T15:49:54.954 INFO:tasks.workunit.client.1.smithi027.stdout:3/426: link d28/f49 d28/d4b/f8e 0 2024-04-22T15:49:54.962 INFO:tasks.workunit.client.1.smithi027.stdout:3/427: dwrite f11 [4194304,4194304] 0 2024-04-22T15:49:54.966 INFO:tasks.workunit.client.1.smithi027.stdout:7/329: creat d0/d11/d1c/d31/d56/f66 x:0 0 0 2024-04-22T15:49:54.967 INFO:tasks.workunit.client.1.smithi027.stdout:7/330: stat d0/d16/f23 0 2024-04-22T15:49:54.968 INFO:tasks.workunit.client.1.smithi027.stdout:7/331: write d0/d11/d1c/d31/d56/f58 [801188,93664] 0 2024-04-22T15:49:54.973 INFO:tasks.workunit.client.1.smithi027.stdout:4/370: creat d5/f93 x:0 0 0 2024-04-22T15:49:54.974 INFO:tasks.workunit.client.1.smithi027.stdout:4/371: readlink d5/de/d30/l91 0 2024-04-22T15:49:54.990 INFO:tasks.workunit.client.1.smithi027.stdout:4/372: rmdir d5/d11 39 2024-04-22T15:49:54.993 INFO:tasks.workunit.client.1.smithi027.stdout:7/332: link d0/d16/d2f/f59 d0/d4c/d4e/f67 0 2024-04-22T15:49:54.998 INFO:tasks.workunit.client.1.smithi027.stdout:4/373: write d5/d11/f36 [824541,47049] 0 2024-04-22T15:49:54.999 INFO:tasks.workunit.client.1.smithi027.stdout:7/333: symlink d0/d16/d2f/l68 0 2024-04-22T15:49:54.999 INFO:tasks.workunit.client.1.smithi027.stdout:4/374: chown d5/f25 156206 1 2024-04-22T15:49:55.005 INFO:tasks.workunit.client.1.smithi027.stdout:7/334: creat d0/d26/f69 x:0 0 0 2024-04-22T15:49:55.005 INFO:tasks.workunit.client.1.smithi027.stdout:7/335: dread - d0/d16/f3d zero size 2024-04-22T15:49:55.007 INFO:tasks.workunit.client.1.smithi027.stdout:4/375: creat d5/d45/f94 x:0 0 0 2024-04-22T15:49:55.010 INFO:tasks.workunit.client.1.smithi027.stdout:4/376: creat d5/d11/d6c/f95 x:0 0 0 2024-04-22T15:49:55.015 INFO:tasks.workunit.client.1.smithi027.stdout:4/377: dwrite d5/d11/d6c/f8b [0,4194304] 0 2024-04-22T15:49:55.016 INFO:tasks.workunit.client.1.smithi027.stdout:4/378: chown d5/d11/d1d/d24 19752 1 2024-04-22T15:49:55.020 INFO:tasks.workunit.client.1.smithi027.stdout:4/379: creat d5/d45/d5a/f96 x:0 0 0 2024-04-22T15:49:55.023 INFO:tasks.workunit.client.1.smithi027.stdout:4/380: creat d5/d7/f97 x:0 0 0 2024-04-22T15:49:55.023 INFO:tasks.workunit.client.1.smithi027.stdout:6/380: write d5/dc/dd/d79/d7b/d5d/f70 [1083655,67152] 0 2024-04-22T15:49:55.028 INFO:tasks.workunit.client.1.smithi027.stdout:6/381: mkdir d5/dc/d26/d49/d86 0 2024-04-22T15:49:55.029 INFO:tasks.workunit.client.1.smithi027.stdout:4/381: creat d5/d45/d81/f98 x:0 0 0 2024-04-22T15:49:55.030 INFO:tasks.workunit.client.1.smithi027.stdout:4/382: write d5/de/f43 [1900218,7470] 0 2024-04-22T15:49:55.031 INFO:tasks.workunit.client.1.smithi027.stdout:4/383: dread - d5/d11/d6c/f95 zero size 2024-04-22T15:49:55.032 INFO:tasks.workunit.client.1.smithi027.stdout:6/382: unlink d5/dc/dd/d79/d7b/d5d/f67 0 2024-04-22T15:49:55.036 INFO:tasks.workunit.client.1.smithi027.stdout:4/384: creat d5/de/d30/d5d/f99 x:0 0 0 2024-04-22T15:49:55.040 INFO:tasks.workunit.client.1.smithi027.stdout:6/383: rmdir d5/db/d84 0 2024-04-22T15:49:55.041 INFO:tasks.workunit.client.1.smithi027.stdout:6/384: write d5/dc/dd/d79/d7b/f4e [1379737,118666] 0 2024-04-22T15:49:55.107 INFO:tasks.workunit.client.1.smithi027.stdout:9/282: sync 2024-04-22T15:49:55.117 INFO:tasks.workunit.client.1.smithi027.stdout:9/283: dread d0/dc/df/f31 [0,4194304] 0 2024-04-22T15:49:55.134 INFO:tasks.workunit.client.1.smithi027.stdout:4/385: link d5/d11/d1d/d24/f51 d5/de/d2a/f9a 0 2024-04-22T15:49:55.134 INFO:tasks.workunit.client.1.smithi027.stdout:4/386: dread - d5/d7/f8f zero size 2024-04-22T15:49:55.136 INFO:tasks.workunit.client.1.smithi027.stdout:6/385: write d5/f29 [3141243,46429] 0 2024-04-22T15:49:55.138 INFO:tasks.workunit.client.1.smithi027.stdout:9/284: truncate d0/dc/f37 1307991 0 2024-04-22T15:49:55.138 INFO:tasks.workunit.client.1.smithi027.stdout:6/386: read d5/db/d2e/f62 [2674931,60244] 0 2024-04-22T15:49:55.140 INFO:tasks.workunit.client.1.smithi027.stdout:9/285: read d0/dc/de/f41 [26489,28430] 0 2024-04-22T15:49:55.141 INFO:tasks.workunit.client.1.smithi027.stdout:4/387: creat d5/d11/d6c/f9b x:0 0 0 2024-04-22T15:49:55.147 INFO:tasks.workunit.client.1.smithi027.stdout:9/286: dwrite d0/dc/de/f40 [4194304,4194304] 0 2024-04-22T15:49:55.152 INFO:tasks.workunit.client.1.smithi027.stdout:4/388: rename d5/d7/f97 to d5/d11/d1d/d24/f9c 0 2024-04-22T15:49:55.153 INFO:tasks.workunit.client.1.smithi027.stdout:9/287: unlink d0/dc/f5f 0 2024-04-22T15:49:55.155 INFO:tasks.workunit.client.1.smithi027.stdout:4/389: rename d5/de/d30/l33 to d5/d11/d6c/l9d 0 2024-04-22T15:49:55.156 INFO:tasks.workunit.client.1.smithi027.stdout:9/288: fdatasync d0/dc/f3d 0 2024-04-22T15:49:55.158 INFO:tasks.workunit.client.1.smithi027.stdout:4/390: creat d5/d42/f9e x:0 0 0 2024-04-22T15:49:55.161 INFO:tasks.workunit.client.1.smithi027.stdout:4/391: creat d5/de/d30/d5d/f9f x:0 0 0 2024-04-22T15:49:55.161 INFO:tasks.workunit.client.1.smithi027.stdout:5/413: dwrite d0/df/d10/f14 [0,4194304] 0 2024-04-22T15:49:55.163 INFO:tasks.workunit.client.1.smithi027.stdout:5/414: readlink d0/df/de/l3f 0 2024-04-22T15:49:55.168 INFO:tasks.workunit.client.1.smithi027.stdout:9/289: dwrite d0/f47 [0,4194304] 0 2024-04-22T15:49:55.171 INFO:tasks.workunit.client.1.smithi027.stdout:2/378: write dd/df/d70/f6e [1676948,13997] 0 2024-04-22T15:49:55.172 INFO:tasks.workunit.client.1.smithi027.stdout:5/415: readlink d0/df/d10/l39 0 2024-04-22T15:49:55.174 INFO:tasks.workunit.client.1.smithi027.stdout:9/290: dread d0/dc/de/f49 [0,4194304] 0 2024-04-22T15:49:55.175 INFO:tasks.workunit.client.1.smithi027.stdout:9/291: read d0/dc/d12/d30/f4f [510983,111906] 0 2024-04-22T15:49:55.176 INFO:tasks.workunit.client.1.smithi027.stdout:5/416: creat d0/d1a/d22/d26/d32/d79/f87 x:0 0 0 2024-04-22T15:49:55.179 INFO:tasks.workunit.client.1.smithi027.stdout:5/417: mkdir d0/d1a/d22/d26/d32/d88 0 2024-04-22T15:49:55.180 INFO:tasks.workunit.client.1.smithi027.stdout:5/418: chown d0/d1a/d22/c58 757005 1 2024-04-22T15:49:55.208 INFO:tasks.workunit.client.1.smithi027.stdout:6/387: dwrite d5/d30/f47 [0,4194304] 0 2024-04-22T15:49:55.211 INFO:tasks.workunit.client.1.smithi027.stdout:6/388: read d5/dc/d26/f2d [2511836,76083] 0 2024-04-22T15:49:55.215 INFO:tasks.workunit.client.1.smithi027.stdout:4/392: write d5/de/f21 [290476,13442] 0 2024-04-22T15:49:55.217 INFO:tasks.workunit.client.1.smithi027.stdout:5/419: write d0/d1a/d22/d26/d32/f4f [2288869,40469] 0 2024-04-22T15:49:55.218 INFO:tasks.workunit.client.1.smithi027.stdout:6/389: dwrite d5/f29 [0,4194304] 0 2024-04-22T15:49:55.220 INFO:tasks.workunit.client.1.smithi027.stdout:4/393: unlink d5/d11/f6d 0 2024-04-22T15:49:55.221 INFO:tasks.workunit.client.1.smithi027.stdout:5/420: truncate d0/df/de/d24/f2e 4638584 0 2024-04-22T15:49:55.221 INFO:tasks.workunit.client.1.smithi027.stdout:4/394: readlink d5/d11/l3d 0 2024-04-22T15:49:55.222 INFO:tasks.workunit.client.1.smithi027.stdout:4/395: chown d5/d45 237706863 1 2024-04-22T15:49:55.225 INFO:tasks.workunit.client.1.smithi027.stdout:5/421: mkdir d0/df/de/d40/d89 0 2024-04-22T15:49:55.227 INFO:tasks.workunit.client.1.smithi027.stdout:6/390: symlink d5/dc/d20/l87 0 2024-04-22T15:49:55.228 INFO:tasks.workunit.client.1.smithi027.stdout:9/292: dwrite d0/dc/f37 [0,4194304] 0 2024-04-22T15:49:55.234 INFO:tasks.workunit.client.1.smithi027.stdout:4/396: dwrite d5/d45/d47/f76 [0,4194304] 0 2024-04-22T15:49:55.236 INFO:tasks.workunit.client.1.smithi027.stdout:4/397: truncate d5/de/f43 2737744 0 2024-04-22T15:49:55.296 INFO:tasks.workunit.client.1.smithi027.stdout:0/381: sync 2024-04-22T15:49:55.296 INFO:tasks.workunit.client.1.smithi027.stdout:1/319: sync 2024-04-22T15:49:55.297 INFO:tasks.workunit.client.1.smithi027.stdout:1/320: stat d2/d1d/l2d 0 2024-04-22T15:49:55.322 INFO:tasks.workunit.client.1.smithi027.stdout:5/422: unlink d0/d1a/l36 0 2024-04-22T15:49:55.324 INFO:tasks.workunit.client.1.smithi027.stdout:6/391: creat d5/dc/d26/f88 x:0 0 0 2024-04-22T15:49:55.325 INFO:tasks.workunit.client.1.smithi027.stdout:6/392: chown d5/dc/d17/f39 481 1 2024-04-22T15:49:55.326 INFO:tasks.workunit.client.1.smithi027.stdout:6/393: truncate d5/d30/d54/f7f 25271 0 2024-04-22T15:49:55.328 INFO:tasks.workunit.client.1.smithi027.stdout:9/293: creat d0/d8/d5c/f62 x:0 0 0 2024-04-22T15:49:55.330 INFO:tasks.workunit.client.1.smithi027.stdout:4/398: unlink d5/d11/d1d/f53 0 2024-04-22T15:49:55.331 INFO:tasks.workunit.client.1.smithi027.stdout:4/399: read - d5/de/f16 zero size 2024-04-22T15:49:55.333 INFO:tasks.workunit.client.1.smithi027.stdout:0/382: symlink d0/d2a/d6a/l78 0 2024-04-22T15:49:55.334 INFO:tasks.workunit.client.1.smithi027.stdout:6/394: dwrite d5/dc/d26/d3d/f7c [0,4194304] 0 2024-04-22T15:49:55.335 INFO:tasks.workunit.client.1.smithi027.stdout:0/383: chown d0/d2a/d38/f6b 776926 1 2024-04-22T15:49:55.336 INFO:tasks.workunit.client.1.smithi027.stdout:1/321: rename d2/d1d/c26 to d2/d36/d38/d3d/d44/c6f 0 2024-04-22T15:49:55.336 INFO:tasks.workunit.client.1.smithi027.stdout:6/395: write d5/dc/dd/d52/f7e [970184,124928] 0 2024-04-22T15:49:55.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:54 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:55.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:54 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:55.337 INFO:tasks.workunit.client.1.smithi027.stdout:5/423: symlink d0/df/d3d/l8a 0 2024-04-22T15:49:55.341 INFO:tasks.workunit.client.1.smithi027.stdout:5/424: read d0/df/de/d24/f35 [1214982,109729] 0 2024-04-22T15:49:55.341 INFO:tasks.workunit.client.1.smithi027.stdout:9/294: symlink d0/d8/d5c/l63 0 2024-04-22T15:49:55.344 INFO:tasks.workunit.client.1.smithi027.stdout:0/384: symlink d0/d2a/d38/l79 0 2024-04-22T15:49:55.345 INFO:tasks.workunit.client.1.smithi027.stdout:0/385: truncate d0/f2e 240684 0 2024-04-22T15:49:55.347 INFO:tasks.workunit.client.1.smithi027.stdout:5/425: fdatasync d0/df/de/f13 0 2024-04-22T15:49:55.349 INFO:tasks.workunit.client.1.smithi027.stdout:9/295: unlink d0/dc/f37 0 2024-04-22T15:49:55.358 INFO:tasks.workunit.client.1.smithi027.stdout:4/400: dread d5/f13 [0,4194304] 0 2024-04-22T15:49:55.360 INFO:tasks.workunit.client.1.smithi027.stdout:0/386: dread d0/f14 [4194304,4194304] 0 2024-04-22T15:49:55.361 INFO:tasks.workunit.client.1.smithi027.stdout:0/387: truncate d0/d2a/f48 4947931 0 2024-04-22T15:49:55.361 INFO:tasks.workunit.client.1.smithi027.stdout:0/388: fdatasync d0/d34/f71 0 2024-04-22T15:49:55.362 INFO:tasks.workunit.client.1.smithi027.stdout:0/389: chown d0/d28/f1c 14204612 1 2024-04-22T15:49:55.362 INFO:tasks.workunit.client.1.smithi027.stdout:5/426: creat d0/d1a/d22/d26/d5d/f8b x:0 0 0 2024-04-22T15:49:55.363 INFO:tasks.workunit.client.1.smithi027.stdout:5/427: readlink d0/df/de/d24/d1c/d3e/l41 0 2024-04-22T15:49:55.364 INFO:tasks.workunit.client.1.smithi027.stdout:5/428: write d0/d1a/d22/d72/d86/f77 [1573124,32942] 0 2024-04-22T15:49:55.369 INFO:tasks.workunit.client.1.smithi027.stdout:6/396: creat d5/f89 x:0 0 0 2024-04-22T15:49:55.370 INFO:tasks.workunit.client.1.smithi027.stdout:1/322: link d2/f68 d2/f70 0 2024-04-22T15:49:55.376 INFO:tasks.workunit.client.1.smithi027.stdout:0/390: fsync d0/d28/d37/f40 0 2024-04-22T15:49:55.376 INFO:tasks.workunit.client.1.smithi027.stdout:9/296: dread d0/dc/de/f24 [0,4194304] 0 2024-04-22T15:49:55.378 INFO:tasks.workunit.client.1.smithi027.stdout:5/429: mkdir d0/df/d3d/d8c 0 2024-04-22T15:49:55.379 INFO:tasks.workunit.client.1.smithi027.stdout:6/397: dwrite d5/dc/d26/f2d [0,4194304] 0 2024-04-22T15:49:55.379 INFO:tasks.workunit.client.1.smithi027.stdout:3/428: sync 2024-04-22T15:49:55.379 INFO:tasks.workunit.client.1.smithi027.stdout:8/344: sync 2024-04-22T15:49:55.381 INFO:tasks.workunit.client.1.smithi027.stdout:1/323: dwrite d2/f37 [0,4194304] 0 2024-04-22T15:49:55.381 INFO:tasks.workunit.client.1.smithi027.stdout:3/429: stat d28/d5d/d79 0 2024-04-22T15:49:55.384 INFO:tasks.workunit.client.1.smithi027.stdout:1/324: chown d2/d36/c4d 0 1 2024-04-22T15:49:55.387 INFO:tasks.workunit.client.1.smithi027.stdout:0/391: creat d0/d34/f7a x:0 0 0 2024-04-22T15:49:55.387 INFO:tasks.workunit.client.1.smithi027.stdout:1/325: fsync d2/f35 0 2024-04-22T15:49:55.392 INFO:tasks.workunit.client.1.smithi027.stdout:8/345: mknod d1/d3/d1c/d32/c75 0 2024-04-22T15:49:55.393 INFO:tasks.workunit.client.1.smithi027.stdout:5/430: truncate d0/df/f12 1977724 0 2024-04-22T15:49:55.395 INFO:tasks.workunit.client.1.smithi027.stdout:6/398: dwrite d5/dc/d26/d3d/f7c [0,4194304] 0 2024-04-22T15:49:55.397 INFO:tasks.workunit.client.1.smithi027.stdout:4/401: write d5/d11/f18 [361962,14215] 0 2024-04-22T15:49:55.401 INFO:tasks.workunit.client.1.smithi027.stdout:3/430: dwrite d28/f63 [0,4194304] 0 2024-04-22T15:49:55.406 INFO:tasks.workunit.client.1.smithi027.stdout:5/431: chown d0/df/de/d24/d62/f68 689 1 2024-04-22T15:49:55.406 INFO:tasks.workunit.client.1.smithi027.stdout:1/326: creat d2/d5f/f71 x:0 0 0 2024-04-22T15:49:55.407 INFO:tasks.workunit.client.1.smithi027.stdout:0/392: dwrite d0/d2a/f3c [0,4194304] 0 2024-04-22T15:49:55.408 INFO:tasks.workunit.client.1.smithi027.stdout:6/399: dread d5/dc/dd/f16 [0,4194304] 0 2024-04-22T15:49:55.415 INFO:tasks.workunit.client.1.smithi027.stdout:4/402: mknod d5/d11/d1d/ca0 0 2024-04-22T15:49:55.417 INFO:tasks.workunit.client.1.smithi027.stdout:0/393: fsync d0/d2a/d38/f55 0 2024-04-22T15:49:55.419 INFO:tasks.workunit.client.1.smithi027.stdout:0/394: fdatasync d0/d2a/d38/f65 0 2024-04-22T15:49:55.419 INFO:tasks.workunit.client.1.smithi027.stdout:1/327: fsync d2/d36/d38/d3d/f4e 0 2024-04-22T15:49:55.420 INFO:tasks.workunit.client.1.smithi027.stdout:1/328: stat d2/d1d 0 2024-04-22T15:49:55.421 INFO:tasks.workunit.client.1.smithi027.stdout:0/395: write d0/d34/f42 [4311420,124872] 0 2024-04-22T15:49:55.427 INFO:tasks.workunit.client.1.smithi027.stdout:7/336: sync 2024-04-22T15:49:55.428 INFO:tasks.workunit.client.1.smithi027.stdout:2/379: sync 2024-04-22T15:49:55.428 INFO:tasks.workunit.client.1.smithi027.stdout:4/403: unlink d5/d7/f4c 0 2024-04-22T15:49:55.428 INFO:tasks.workunit.client.1.smithi027.stdout:7/337: dread - d0/f30 zero size 2024-04-22T15:49:55.438 INFO:tasks.workunit.client.1.smithi027.stdout:5/432: dread d0/df/de/d24/f35 [0,4194304] 0 2024-04-22T15:49:55.440 INFO:tasks.workunit.client.1.smithi027.stdout:9/297: dwrite d0/d8/f18 [0,4194304] 0 2024-04-22T15:49:55.443 INFO:tasks.workunit.client.1.smithi027.stdout:6/400: dread d5/dc/d20/d2b/f35 [0,4194304] 0 2024-04-22T15:49:55.446 INFO:tasks.workunit.client.1.smithi027.stdout:4/404: creat d5/d11/d1d/d54/fa1 x:0 0 0 2024-04-22T15:49:55.447 INFO:tasks.workunit.client.1.smithi027.stdout:4/405: dread - d5/d11/d1d/d54/fa1 zero size 2024-04-22T15:49:55.448 INFO:tasks.workunit.client.1.smithi027.stdout:7/338: symlink d0/d11/d1c/d54/l6a 0 2024-04-22T15:49:55.449 INFO:tasks.workunit.client.1.smithi027.stdout:3/431: write f26 [6720153,5174] 0 2024-04-22T15:49:55.451 INFO:tasks.workunit.client.1.smithi027.stdout:5/433: creat d0/d1a/d22/d26/d32/d79/f8d x:0 0 0 2024-04-22T15:49:55.451 INFO:tasks.workunit.client.1.smithi027.stdout:8/346: truncate d1/d1e/f33 16465 0 2024-04-22T15:49:55.455 INFO:tasks.workunit.client.1.smithi027.stdout:2/380: symlink dd/l71 0 2024-04-22T15:49:55.458 INFO:tasks.workunit.client.1.smithi027.stdout:6/401: unlink d5/c44 0 2024-04-22T15:49:55.459 INFO:tasks.workunit.client.1.smithi027.stdout:1/329: creat d2/f72 x:0 0 0 2024-04-22T15:49:55.462 INFO:tasks.workunit.client.1.smithi027.stdout:7/339: rename d0/d16/d2f/l68 to d0/d26/l6b 0 2024-04-22T15:49:55.464 INFO:tasks.workunit.client.1.smithi027.stdout:3/432: creat d28/d5d/d62/f8f x:0 0 0 2024-04-22T15:49:55.464 INFO:tasks.workunit.client.1.smithi027.stdout:8/347: rmdir d1/d3/d1c/d32/d41 39 2024-04-22T15:49:55.466 INFO:tasks.workunit.client.1.smithi027.stdout:2/381: mknod dd/df/d1b/d38/c72 0 2024-04-22T15:49:55.469 INFO:tasks.workunit.client.1.smithi027.stdout:1/330: fsync d2/d30/f47 0 2024-04-22T15:49:55.472 INFO:tasks.workunit.client.1.smithi027.stdout:1/331: write d2/d5f/f71 [738925,6090] 0 2024-04-22T15:49:55.473 INFO:tasks.workunit.client.1.smithi027.stdout:0/396: dwrite d0/d28/d37/f40 [0,4194304] 0 2024-04-22T15:49:55.474 INFO:tasks.workunit.client.1.smithi027.stdout:0/397: stat d0/d28/f1c 0 2024-04-22T15:49:55.476 INFO:tasks.workunit.client.1.smithi027.stdout:7/340: write d0/f3e [484979,68422] 0 2024-04-22T15:49:55.481 INFO:tasks.workunit.client.1.smithi027.stdout:3/433: chown d28/d5d/l5e 194 1 2024-04-22T15:49:55.482 INFO:tasks.workunit.client.1.smithi027.stdout:3/434: stat d28/f4e 0 2024-04-22T15:49:55.483 INFO:tasks.workunit.client.1.smithi027.stdout:9/298: dwrite d0/dc/d12/d30/f32 [0,4194304] 0 2024-04-22T15:49:55.486 INFO:tasks.workunit.client.1.smithi027.stdout:6/402: mkdir d5/dc/d17/d8a 0 2024-04-22T15:49:55.491 INFO:tasks.workunit.client.1.smithi027.stdout:0/398: symlink d0/d2a/d38/l7b 0 2024-04-22T15:49:55.497 INFO:tasks.workunit.client.1.smithi027.stdout:5/434: rename d0/d1a/d22/d26/d32/d79/f8d to d0/df/d3d/f8e 0 2024-04-22T15:49:55.497 INFO:tasks.workunit.client.1.smithi027.stdout:0/399: dwrite d0/f3e [0,4194304] 0 2024-04-22T15:49:55.500 INFO:tasks.workunit.client.1.smithi027.stdout:5/435: chown d0/df/d3d/f44 5330076 1 2024-04-22T15:49:55.500 INFO:tasks.workunit.client.1.smithi027.stdout:9/299: rmdir d0/d8/d5c 39 2024-04-22T15:49:55.506 INFO:tasks.workunit.client.1.smithi027.stdout:6/403: rmdir d5/d30/d40 39 2024-04-22T15:49:55.508 INFO:tasks.workunit.client.1.smithi027.stdout:7/341: write d0/d49/f63 [255032,130714] 0 2024-04-22T15:49:55.511 INFO:tasks.workunit.client.1.smithi027.stdout:0/400: dwrite d0/d2a/d38/f65 [4194304,4194304] 0 2024-04-22T15:49:55.512 INFO:tasks.workunit.client.1.smithi027.stdout:3/435: mknod d28/c90 0 2024-04-22T15:49:55.512 INFO:tasks.workunit.client.1.smithi027.stdout:9/300: dwrite d0/f47 [0,4194304] 0 2024-04-22T15:49:55.516 INFO:tasks.workunit.client.1.smithi027.stdout:1/332: rename d2/f37 to d2/d36/d38/d3d/d44/f73 0 2024-04-22T15:49:55.518 INFO:tasks.workunit.client.1.smithi027.stdout:5/436: mkdir d0/df/de/d24/d1c/d8f 0 2024-04-22T15:49:55.519 INFO:tasks.workunit.client.1.smithi027.stdout:8/348: getdents d1/d3/d1c/d32/d41/d52/d61 0 2024-04-22T15:49:55.523 INFO:tasks.workunit.client.1.smithi027.stdout:6/404: creat d5/dc/dd/d79/f8b x:0 0 0 2024-04-22T15:49:55.525 INFO:tasks.workunit.client.1.smithi027.stdout:7/342: dwrite d0/f29 [0,4194304] 0 2024-04-22T15:49:55.527 INFO:tasks.workunit.client.1.smithi027.stdout:3/436: dwrite d28/f3c [0,4194304] 0 2024-04-22T15:49:55.528 INFO:tasks.workunit.client.1.smithi027.stdout:8/349: read d1/f4 [2356517,36725] 0 2024-04-22T15:49:55.534 INFO:tasks.workunit.client.1.smithi027.stdout:6/405: dwrite d5/f7a [0,4194304] 0 2024-04-22T15:49:55.535 INFO:tasks.workunit.client.1.smithi027.stdout:2/382: link dd/df/d1b/l3f dd/df/l73 0 2024-04-22T15:49:55.539 INFO:tasks.workunit.client.1.smithi027.stdout:6/406: write d5/dc/d20/d2b/f12 [13144284,7927] 0 2024-04-22T15:49:55.542 INFO:tasks.workunit.client.1.smithi027.stdout:9/301: mkdir d0/d64 0 2024-04-22T15:49:55.543 INFO:tasks.workunit.client.1.smithi027.stdout:5/437: mkdir d0/d1a/d22/d26/d32/d88/d90 0 2024-04-22T15:49:55.543 INFO:tasks.workunit.client.1.smithi027.stdout:8/350: truncate d1/d3/d1c/d56/d62/f68 539312 0 2024-04-22T15:49:55.544 INFO:tasks.workunit.client.1.smithi027.stdout:2/383: rmdir dd/df/d70/d4b 39 2024-04-22T15:49:55.546 INFO:tasks.workunit.client.1.smithi027.stdout:7/343: creat d0/d11/d3a/d46/f6c x:0 0 0 2024-04-22T15:49:55.547 INFO:tasks.workunit.client.1.smithi027.stdout:2/384: write dd/df/d1b/d38/f5c [5035603,51255] 0 2024-04-22T15:49:55.550 INFO:tasks.workunit.client.1.smithi027.stdout:6/407: mkdir d5/dc/d17/d53/d8c 0 2024-04-22T15:49:55.553 INFO:tasks.workunit.client.1.smithi027.stdout:6/408: chown d5/db/d2e/ff 470354931 1 2024-04-22T15:49:55.553 INFO:tasks.workunit.client.1.smithi027.stdout:8/351: mkdir d1/d1e/d47/d53/d76 0 2024-04-22T15:49:55.557 INFO:tasks.workunit.client.1.smithi027.stdout:1/333: write d2/f68 [89059,119295] 0 2024-04-22T15:49:55.557 INFO:tasks.workunit.client.1.smithi027.stdout:7/344: mknod d0/d4c/c6d 0 2024-04-22T15:49:55.560 INFO:tasks.workunit.client.1.smithi027.stdout:0/401: getdents d0/d28 0 2024-04-22T15:49:55.561 INFO:tasks.workunit.client.1.smithi027.stdout:9/302: creat d0/dc/d12/d30/d5b/f65 x:0 0 0 2024-04-22T15:49:55.563 INFO:tasks.workunit.client.1.smithi027.stdout:2/385: dread dd/df/d1b/f40 [0,4194304] 0 2024-04-22T15:49:55.563 INFO:tasks.workunit.client.1.smithi027.stdout:9/303: chown d0/dc/d12/d30/f32 53 1 2024-04-22T15:49:55.564 INFO:tasks.workunit.client.1.smithi027.stdout:2/386: write dd/f19 [4997803,35019] 0 2024-04-22T15:49:55.565 INFO:tasks.workunit.client.1.smithi027.stdout:8/352: creat d1/d1e/d47/d53/f77 x:0 0 0 2024-04-22T15:49:55.565 INFO:tasks.workunit.client.1.smithi027.stdout:2/387: chown dd/df/d70/d5a 1483890895 1 2024-04-22T15:49:55.567 INFO:tasks.workunit.client.1.smithi027.stdout:5/438: creat d0/d1a/d22/d26/d32/d88/d90/f91 x:0 0 0 2024-04-22T15:49:55.569 INFO:tasks.workunit.client.1.smithi027.stdout:7/345: symlink d0/d16/d52/l6e 0 2024-04-22T15:49:55.571 INFO:tasks.workunit.client.1.smithi027.stdout:0/402: mkdir d0/d2a/d38/d70/d7c 0 2024-04-22T15:49:55.574 INFO:tasks.workunit.client.1.smithi027.stdout:6/409: rename d5/d30/d54/l6e to d5/dc/dd/l8d 0 2024-04-22T15:49:55.579 INFO:tasks.workunit.client.1.smithi027.stdout:1/334: mkdir d2/d36/d74 0 2024-04-22T15:49:55.583 INFO:tasks.workunit.client.1.smithi027.stdout:7/346: dread d0/d16/f3b [0,4194304] 0 2024-04-22T15:49:55.584 INFO:tasks.workunit.client.1.smithi027.stdout:6/410: creat d5/dc/dd/d52/f8e x:0 0 0 2024-04-22T15:49:55.584 INFO:tasks.workunit.client.1.smithi027.stdout:1/335: dread d2/f60 [0,4194304] 0 2024-04-22T15:49:55.585 INFO:tasks.workunit.client.1.smithi027.stdout:9/304: symlink d0/dc/l66 0 2024-04-22T15:49:55.586 INFO:tasks.workunit.client.1.smithi027.stdout:0/403: dread d0/fc [0,4194304] 0 2024-04-22T15:49:55.587 INFO:tasks.workunit.client.1.smithi027.stdout:9/305: chown d0/dc/de/f49 18 1 2024-04-22T15:49:55.589 INFO:tasks.workunit.client.1.smithi027.stdout:0/404: write d0/d2a/d38/f6b [1812955,101973] 0 2024-04-22T15:49:55.595 INFO:tasks.workunit.client.1.smithi027.stdout:6/411: dread d5/f29 [0,4194304] 0 2024-04-22T15:49:55.595 INFO:tasks.workunit.client.1.smithi027.stdout:1/336: unlink d2/d36/d38/l6b 0 2024-04-22T15:49:55.595 INFO:tasks.workunit.client.1.smithi027.stdout:6/412: chown d5/dc/l21 55905332 1 2024-04-22T15:49:55.596 INFO:tasks.workunit.client.1.smithi027.stdout:6/413: read d5/dc/d26/d3d/f83 [3598088,41056] 0 2024-04-22T15:49:55.596 INFO:tasks.workunit.client.1.smithi027.stdout:8/353: getdents d1/d1e 0 2024-04-22T15:49:55.597 INFO:tasks.workunit.client.1.smithi027.stdout:6/414: chown d5/dc/d17/d53 51956 1 2024-04-22T15:49:55.598 INFO:tasks.workunit.client.1.smithi027.stdout:9/306: write d0/f44 [70925,117409] 0 2024-04-22T15:49:55.598 INFO:tasks.workunit.client.1.smithi027.stdout:6/415: write d5/dc/d17/d53/f72 [34820,60835] 0 2024-04-22T15:49:55.599 INFO:tasks.workunit.client.1.smithi027.stdout:2/388: creat dd/df/d1b/d1f/f74 x:0 0 0 2024-04-22T15:49:55.600 INFO:tasks.workunit.client.1.smithi027.stdout:2/389: readlink dd/l34 0 2024-04-22T15:49:55.600 INFO:tasks.workunit.client.1.smithi027.stdout:0/405: fsync d0/d2a/f61 0 2024-04-22T15:49:55.692 INFO:tasks.workunit.client.1.smithi027.stdout:4/406: sync 2024-04-22T15:49:55.693 INFO:tasks.workunit.client.1.smithi027.stdout:4/407: fdatasync d5/d11/d89/f70 0 2024-04-22T15:49:55.709 INFO:tasks.workunit.client.1.smithi027.stdout:8/354: creat d1/d1e/d47/d53/f78 x:0 0 0 2024-04-22T15:49:55.719 INFO:tasks.workunit.client.1.smithi027.stdout:0/406: creat d0/d28/d63/f7d x:0 0 0 2024-04-22T15:49:55.721 INFO:tasks.workunit.client.1.smithi027.stdout:2/390: truncate dd/df/d1b/d38/f3b 258071 0 2024-04-22T15:49:55.734 INFO:tasks.workunit.client.1.smithi027.stdout:2/391: mkdir dd/df/d1b/d38/d60/d75 0 2024-04-22T15:49:55.734 INFO:tasks.workunit.client.1.smithi027.stdout:2/392: stat dd/l34 0 2024-04-22T15:49:55.735 INFO:tasks.workunit.client.1.smithi027.stdout:2/393: chown dd/df/f23 54751948 1 2024-04-22T15:49:55.737 INFO:tasks.workunit.client.1.smithi027.stdout:8/355: mknod d1/d3/d1c/d56/c79 0 2024-04-22T15:49:55.743 INFO:tasks.workunit.client.1.smithi027.stdout:2/394: truncate dd/df/d70/d4b/f68 76670 0 2024-04-22T15:49:55.751 INFO:tasks.workunit.client.1.smithi027.stdout:2/395: unlink dd/df/d70/f63 0 2024-04-22T15:49:55.752 INFO:tasks.workunit.client.1.smithi027.stdout:0/407: getdents d0/d28/d63/d67 0 2024-04-22T15:49:55.752 INFO:tasks.workunit.client.1.smithi027.stdout:0/408: stat d0/f2e 0 2024-04-22T15:49:55.758 INFO:tasks.workunit.client.1.smithi027.stdout:0/409: dread d0/f22 [0,4194304] 0 2024-04-22T15:49:55.758 INFO:tasks.workunit.client.1.smithi027.stdout:0/410: stat d0/d34/f7a 0 2024-04-22T15:49:55.760 INFO:tasks.workunit.client.1.smithi027.stdout:8/356: getdents d1/d3/d1c/d28 0 2024-04-22T15:49:55.763 INFO:tasks.workunit.client.1.smithi027.stdout:8/357: creat d1/d3/d1c/d32/d41/f7a x:0 0 0 2024-04-22T15:49:55.796 INFO:tasks.workunit.client.1.smithi027.stdout:1/337: creat d2/f75 x:0 0 0 2024-04-22T15:49:55.797 INFO:tasks.workunit.client.1.smithi027.stdout:1/338: dread d2/f68 [0,4194304] 0 2024-04-22T15:49:55.799 INFO:tasks.workunit.client.1.smithi027.stdout:1/339: fsync d2/d30/f47 0 2024-04-22T15:49:55.800 INFO:tasks.workunit.client.1.smithi027.stdout:1/340: stat d2/f68 0 2024-04-22T15:49:55.802 INFO:tasks.workunit.client.1.smithi027.stdout:1/341: creat d2/d5f/f76 x:0 0 0 2024-04-22T15:49:55.803 INFO:tasks.workunit.client.1.smithi027.stdout:3/437: sync 2024-04-22T15:49:55.806 INFO:tasks.workunit.client.1.smithi027.stdout:1/342: symlink d2/d36/l77 0 2024-04-22T15:49:55.807 INFO:tasks.workunit.client.1.smithi027.stdout:3/438: dread d28/d5d/f84 [0,4194304] 0 2024-04-22T15:49:55.809 INFO:tasks.workunit.client.1.smithi027.stdout:3/439: truncate d28/d4b/d52/f7a 956428 0 2024-04-22T15:49:55.810 INFO:tasks.workunit.client.1.smithi027.stdout:1/343: getdents d2/d30 0 2024-04-22T15:49:55.814 INFO:tasks.workunit.client.1.smithi027.stdout:1/344: mknod d2/d1d/d56/c78 0 2024-04-22T15:49:55.816 INFO:tasks.workunit.client.1.smithi027.stdout:3/440: dwrite d28/f45 [0,4194304] 0 2024-04-22T15:49:55.820 INFO:tasks.workunit.client.1.smithi027.stdout:7/347: truncate d0/d49/f63 897018 0 2024-04-22T15:49:55.821 INFO:tasks.workunit.client.1.smithi027.stdout:7/348: dread - d0/d11/d1c/d31/f5f zero size 2024-04-22T15:49:55.821 INFO:tasks.workunit.client.1.smithi027.stdout:1/345: creat d2/d36/d74/f79 x:0 0 0 2024-04-22T15:49:55.827 INFO:tasks.workunit.client.1.smithi027.stdout:7/349: symlink d0/d11/d1c/d54/l6f 0 2024-04-22T15:49:55.827 INFO:tasks.workunit.client.1.smithi027.stdout:7/350: stat d0/d16/d65 0 2024-04-22T15:49:55.828 INFO:tasks.workunit.client.1.smithi027.stdout:7/351: dread - d0/d16/d2f/f59 zero size 2024-04-22T15:49:55.829 INFO:tasks.workunit.client.1.smithi027.stdout:3/441: getdents d28/d5d/d62 0 2024-04-22T15:49:55.836 INFO:tasks.workunit.client.1.smithi027.stdout:5/439: sync 2024-04-22T15:49:55.842 INFO:tasks.workunit.client.1.smithi027.stdout:0/411: fsync d0/d28/d63/f7d 0 2024-04-22T15:49:55.844 INFO:tasks.workunit.client.1.smithi027.stdout:3/442: dread d28/d4b/f4c [0,4194304] 0 2024-04-22T15:49:55.845 INFO:tasks.workunit.client.1.smithi027.stdout:0/412: write d0/d28/d37/f3b [164231,55000] 0 2024-04-22T15:49:55.851 INFO:tasks.workunit.client.1.smithi027.stdout:3/443: dread d28/f88 [0,4194304] 0 2024-04-22T15:49:55.852 INFO:tasks.workunit.client.1.smithi027.stdout:5/440: dread d0/f3c [0,4194304] 0 2024-04-22T15:49:55.853 INFO:tasks.workunit.client.1.smithi027.stdout:9/307: truncate d0/dc/de/f3a 277029 0 2024-04-22T15:49:55.864 INFO:tasks.workunit.client.1.smithi027.stdout:9/308: getdents d0/d64 0 2024-04-22T15:49:55.865 INFO:tasks.workunit.client.1.smithi027.stdout:5/441: mknod d0/df/de/d24/d1c/d8f/c92 0 2024-04-22T15:49:55.866 INFO:tasks.workunit.client.1.smithi027.stdout:5/442: stat d0/df/d3d/f44 0 2024-04-22T15:49:55.868 INFO:tasks.workunit.client.1.smithi027.stdout:3/444: dread d28/d4b/f7c [0,4194304] 0 2024-04-22T15:49:55.868 INFO:tasks.workunit.client.1.smithi027.stdout:9/309: creat d0/dc/d12/d30/f67 x:0 0 0 2024-04-22T15:49:55.869 INFO:tasks.workunit.client.1.smithi027.stdout:9/310: chown d0/dc 1356 1 2024-04-22T15:49:55.869 INFO:tasks.workunit.client.1.smithi027.stdout:3/445: write d28/f48 [2296609,105684] 0 2024-04-22T15:49:55.870 INFO:tasks.workunit.client.1.smithi027.stdout:3/446: fdatasync d28/d69/f77 0 2024-04-22T15:49:55.872 INFO:tasks.workunit.client.1.smithi027.stdout:5/443: mknod d0/d1a/d22/d26/d5d/c93 0 2024-04-22T15:49:55.880 INFO:tasks.workunit.client.1.smithi027.stdout:5/444: creat d0/df/de/d24/d1c/d8f/f94 x:0 0 0 2024-04-22T15:49:55.881 INFO:tasks.workunit.client.1.smithi027.stdout:5/445: chown d0/df/de/f38 1 1 2024-04-22T15:49:55.881 INFO:tasks.workunit.client.1.smithi027.stdout:3/447: fsync d28/d4b/f5b 0 2024-04-22T15:49:55.882 INFO:tasks.workunit.client.1.smithi027.stdout:5/446: write d0/d1a/d22/d26/d32/f67 [879518,64989] 0 2024-04-22T15:49:55.883 INFO:tasks.workunit.client.1.smithi027.stdout:9/311: creat d0/f68 x:0 0 0 2024-04-22T15:49:55.884 INFO:tasks.workunit.client.1.smithi027.stdout:9/312: fsync d0/d8/f11 0 2024-04-22T15:49:55.886 INFO:tasks.workunit.client.1.smithi027.stdout:9/313: write d0/dc/d12/d58/f59 [1715721,70045] 0 2024-04-22T15:49:55.890 INFO:tasks.workunit.client.1.smithi027.stdout:2/396: unlink dd/df/f28 0 2024-04-22T15:49:55.895 INFO:tasks.workunit.client.1.smithi027.stdout:9/314: dwrite d0/dc/d12/d30/f4f [4194304,4194304] 0 2024-04-22T15:49:55.899 INFO:tasks.workunit.client.1.smithi027.stdout:6/416: rename d5/db/c37 to d5/dc/c8f 0 2024-04-22T15:49:55.900 INFO:tasks.workunit.client.1.smithi027.stdout:9/315: symlink d0/dc/l69 0 2024-04-22T15:49:55.902 INFO:tasks.workunit.client.1.smithi027.stdout:4/408: rename d5/de/l90 to d5/d11/d6c/la2 0 2024-04-22T15:49:55.908 INFO:tasks.workunit.client.1.smithi027.stdout:6/417: dwrite d5/db/d2e/ff [0,4194304] 0 2024-04-22T15:49:55.909 INFO:tasks.workunit.client.1.smithi027.stdout:4/409: mkdir d5/d11/d6c/da3 0 2024-04-22T15:49:55.911 INFO:tasks.workunit.client.1.smithi027.stdout:4/410: dread - d5/d7/f5b zero size 2024-04-22T15:49:55.914 INFO:tasks.workunit.client.1.smithi027.stdout:8/358: unlink d1/d3/f6b 0 2024-04-22T15:49:55.917 INFO:tasks.workunit.client.1.smithi027.stdout:4/411: symlink d5/d45/d47/la4 0 2024-04-22T15:49:55.919 INFO:tasks.workunit.client.1.smithi027.stdout:8/359: creat d1/d3/d1c/d32/d41/f7b x:0 0 0 2024-04-22T15:49:55.920 INFO:tasks.workunit.client.1.smithi027.stdout:2/397: creat dd/df/d1b/f76 x:0 0 0 2024-04-22T15:49:55.921 INFO:tasks.workunit.client.1.smithi027.stdout:2/398: readlink dd/df/d1b/d1f/l46 0 2024-04-22T15:49:55.922 INFO:tasks.workunit.client.1.smithi027.stdout:8/360: mknod d1/d3/d36/d57/c7c 0 2024-04-22T15:49:55.922 INFO:tasks.workunit.client.1.smithi027.stdout:2/399: chown dd/df/l3d 401316773 1 2024-04-22T15:49:55.924 INFO:tasks.workunit.client.1.smithi027.stdout:6/418: dread d5/dc/d20/d2b/f12 [8388608,4194304] 0 2024-04-22T15:49:55.927 INFO:tasks.workunit.client.1.smithi027.stdout:3/448: rename c14 to d28/d4b/d52/c91 0 2024-04-22T15:49:55.928 INFO:tasks.workunit.client.1.smithi027.stdout:9/316: dread d0/dc/d12/d58/f59 [0,4194304] 0 2024-04-22T15:49:55.929 INFO:tasks.workunit.client.1.smithi027.stdout:3/449: stat d28/d4b/d5c/c6d 0 2024-04-22T15:49:55.929 INFO:tasks.workunit.client.1.smithi027.stdout:4/412: dread d5/d45/d5a/f82 [0,4194304] 0 2024-04-22T15:49:55.932 INFO:tasks.workunit.client.1.smithi027.stdout:9/317: fdatasync d0/dc/d12/d30/d5b/f65 0 2024-04-22T15:49:55.932 INFO:tasks.workunit.client.1.smithi027.stdout:3/450: dread d28/d4b/d52/f7a [0,4194304] 0 2024-04-22T15:49:55.933 INFO:tasks.workunit.client.1.smithi027.stdout:6/419: rename d5/dc to d5/dc/d20/d90 22 2024-04-22T15:49:55.934 INFO:tasks.workunit.client.1.smithi027.stdout:1/346: rmdir d2/d36/d74 39 2024-04-22T15:49:55.935 INFO:tasks.workunit.client.1.smithi027.stdout:6/420: readlink d5/dc/d26/l4a 0 2024-04-22T15:49:55.938 INFO:tasks.workunit.client.1.smithi027.stdout:2/400: dread f7 [0,4194304] 0 2024-04-22T15:49:55.939 INFO:tasks.workunit.client.1.smithi027.stdout:2/401: write dd/df/d1b/f32 [482235,110915] 0 2024-04-22T15:49:55.941 INFO:tasks.workunit.client.1.smithi027.stdout:9/318: fdatasync d0/dc/d12/d58/f59 0 2024-04-22T15:49:55.941 INFO:tasks.workunit.client.1.smithi027.stdout:2/402: dread - dd/df/d1b/d38/d60/f6d zero size 2024-04-22T15:49:55.942 INFO:tasks.workunit.client.1.smithi027.stdout:2/403: stat dd/df/d1b/f41 0 2024-04-22T15:49:55.943 INFO:tasks.workunit.client.1.smithi027.stdout:9/319: dread - d0/dc/d12/d30/d5b/f65 zero size 2024-04-22T15:49:55.943 INFO:tasks.workunit.client.1.smithi027.stdout:2/404: chown dd/df/d70/d4b/d4e/c4f 0 1 2024-04-22T15:49:55.944 INFO:tasks.workunit.client.1.smithi027.stdout:1/347: creat d2/d5f/f7a x:0 0 0 2024-04-22T15:49:55.945 INFO:tasks.workunit.client.1.smithi027.stdout:4/413: rename d5/d11/l19 to d5/de/d2a/d58/la5 0 2024-04-22T15:49:55.946 INFO:tasks.workunit.client.1.smithi027.stdout:7/352: truncate d0/d11/fe 3456285 0 2024-04-22T15:49:55.948 INFO:tasks.workunit.client.1.smithi027.stdout:6/421: creat d5/dc/d17/d53/f91 x:0 0 0 2024-04-22T15:49:55.948 INFO:tasks.workunit.client.1.smithi027.stdout:2/405: dread dd/df/d1b/f32 [0,4194304] 0 2024-04-22T15:49:55.949 INFO:tasks.workunit.client.1.smithi027.stdout:6/422: fdatasync d5/dc/d17/d53/f72 0 2024-04-22T15:49:55.949 INFO:tasks.workunit.client.1.smithi027.stdout:2/406: write dd/f6f [111711,82304] 0 2024-04-22T15:49:55.954 INFO:tasks.workunit.client.1.smithi027.stdout:9/320: creat d0/dc/d12/d58/f6a x:0 0 0 2024-04-22T15:49:55.956 INFO:tasks.workunit.client.1.smithi027.stdout:6/423: dwrite d5/db/d2e/f62 [0,4194304] 0 2024-04-22T15:49:55.962 INFO:tasks.workunit.client.1.smithi027.stdout:7/353: creat d0/d11/d1c/d31/f70 x:0 0 0 2024-04-22T15:49:55.963 INFO:tasks.workunit.client.1.smithi027.stdout:7/354: chown d0/d26 182 1 2024-04-22T15:49:55.965 INFO:tasks.workunit.client.1.smithi027.stdout:2/407: creat dd/df/d70/d5a/f77 x:0 0 0 2024-04-22T15:49:55.968 INFO:tasks.workunit.client.1.smithi027.stdout:7/355: dread d0/d16/f25 [0,4194304] 0 2024-04-22T15:49:55.970 INFO:tasks.workunit.client.1.smithi027.stdout:1/348: mknod d2/d36/d74/c7b 0 2024-04-22T15:49:55.973 INFO:tasks.workunit.client.1.smithi027.stdout:6/424: mknod d5/dc/d26/d3d/c92 0 2024-04-22T15:49:55.980 INFO:tasks.workunit.client.1.smithi027.stdout:3/451: getdents d28/d69 0 2024-04-22T15:49:55.980 INFO:tasks.workunit.client.1.smithi027.stdout:3/452: fsync d28/f3b 0 2024-04-22T15:49:55.988 INFO:tasks.workunit.client.1.smithi027.stdout:9/321: rename d0/d8/d5c to d0/dc/d12/d58/d6b 0 2024-04-22T15:49:55.990 INFO:tasks.workunit.client.1.smithi027.stdout:6/425: mknod d5/d30/d54/d6a/c93 0 2024-04-22T15:49:55.994 INFO:tasks.workunit.client.1.smithi027.stdout:5/447: dwrite d0/f6 [4194304,4194304] 0 2024-04-22T15:49:55.996 INFO:tasks.workunit.client.1.smithi027.stdout:7/356: creat d0/d49/f71 x:0 0 0 2024-04-22T15:49:56.000 INFO:tasks.workunit.client.1.smithi027.stdout:3/453: rename l6 to d28/d5d/d79/l92 0 2024-04-22T15:49:56.001 INFO:tasks.workunit.client.1.smithi027.stdout:9/322: creat d0/dc/de/f6c x:0 0 0 2024-04-22T15:49:56.003 INFO:tasks.workunit.client.1.smithi027.stdout:2/408: creat dd/df/d1b/d1f/f78 x:0 0 0 2024-04-22T15:49:56.005 INFO:tasks.workunit.client.1.smithi027.stdout:5/448: symlink d0/df/de/d24/d1c/l95 0 2024-04-22T15:49:56.006 INFO:tasks.workunit.client.1.smithi027.stdout:1/349: creat d2/d36/f7c x:0 0 0 2024-04-22T15:49:56.008 INFO:tasks.workunit.client.1.smithi027.stdout:7/357: mkdir d0/d16/d52/d72 0 2024-04-22T15:49:56.008 INFO:tasks.workunit.client.1.smithi027.stdout:7/358: stat d0/d11 0 2024-04-22T15:49:56.009 INFO:tasks.workunit.client.1.smithi027.stdout:3/454: mknod d28/d69/c93 0 2024-04-22T15:49:56.014 INFO:tasks.workunit.client.1.smithi027.stdout:8/361: write d1/d1e/f33 [583137,56232] 0 2024-04-22T15:49:56.014 INFO:tasks.workunit.client.1.smithi027.stdout:5/449: symlink d0/d1a/d22/l96 0 2024-04-22T15:49:56.017 INFO:tasks.workunit.client.1.smithi027.stdout:7/359: creat d0/d4c/d4e/f73 x:0 0 0 2024-04-22T15:49:56.018 INFO:tasks.workunit.client.1.smithi027.stdout:3/455: mkdir d28/d4b/d52/d94 0 2024-04-22T15:49:56.021 INFO:tasks.workunit.client.1.smithi027.stdout:5/450: dwrite d0/d1a/d22/d72/f7b [0,4194304] 0 2024-04-22T15:49:56.025 INFO:tasks.workunit.client.1.smithi027.stdout:1/350: mkdir d2/d36/d38/d69/d7d 0 2024-04-22T15:49:56.029 INFO:tasks.workunit.client.1.smithi027.stdout:8/362: dwrite d1/f4 [0,4194304] 0 2024-04-22T15:49:56.030 INFO:tasks.workunit.client.1.smithi027.stdout:7/360: mknod d0/d11/d1c/d31/c74 0 2024-04-22T15:49:56.038 INFO:tasks.workunit.client.1.smithi027.stdout:5/451: dread d0/f6 [0,4194304] 0 2024-04-22T15:49:56.039 INFO:tasks.workunit.client.1.smithi027.stdout:7/361: creat d0/d3f/f75 x:0 0 0 2024-04-22T15:49:56.040 INFO:tasks.workunit.client.1.smithi027.stdout:2/409: link dd/df/d70/d4b/l57 dd/df/l79 0 2024-04-22T15:49:56.040 INFO:tasks.workunit.client.1.smithi027.stdout:5/452: write d0/d1a/f82 [102463,23305] 0 2024-04-22T15:49:56.043 INFO:tasks.workunit.client.1.smithi027.stdout:8/363: mkdir d1/d3/d1c/d43/d5f/d7d 0 2024-04-22T15:49:56.046 INFO:tasks.workunit.client.1.smithi027.stdout:4/414: write d5/f23 [4871536,113717] 0 2024-04-22T15:49:56.047 INFO:tasks.workunit.client.1.smithi027.stdout:2/410: creat dd/df/d1b/f7a x:0 0 0 2024-04-22T15:49:56.048 INFO:tasks.workunit.client.1.smithi027.stdout:4/415: stat d5/d11/l3d 0 2024-04-22T15:49:56.049 INFO:tasks.workunit.client.1.smithi027.stdout:4/416: fdatasync d5/d11/f80 0 2024-04-22T15:49:56.050 INFO:tasks.workunit.client.1.smithi027.stdout:7/362: dwrite d0/d16/f3b [0,4194304] 0 2024-04-22T15:49:56.051 INFO:tasks.workunit.client.1.smithi027.stdout:5/453: mkdir d0/d97 0 2024-04-22T15:49:56.053 INFO:tasks.workunit.client.1.smithi027.stdout:7/363: write d0/d4c/d4e/f57 [1384906,22191] 0 2024-04-22T15:49:56.056 INFO:tasks.workunit.client.1.smithi027.stdout:8/364: mknod d1/d1e/d47/d53/d76/c7e 0 2024-04-22T15:49:56.056 INFO:tasks.workunit.client.1.smithi027.stdout:8/365: stat d1/d3/l8 0 2024-04-22T15:49:56.057 INFO:tasks.workunit.client.1.smithi027.stdout:8/366: stat d1/d3/d1c/f34 0 2024-04-22T15:49:56.057 INFO:tasks.workunit.client.1.smithi027.stdout:8/367: read - d1/d3/f6d zero size 2024-04-22T15:49:56.058 INFO:tasks.workunit.client.1.smithi027.stdout:4/417: mkdir d5/d11/da6 0 2024-04-22T15:49:56.058 INFO:tasks.workunit.client.1.smithi027.stdout:8/368: fsync d1/d3/f1a 0 2024-04-22T15:49:56.059 INFO:tasks.workunit.client.1.smithi027.stdout:4/418: chown d5/d11/l4b 1 1 2024-04-22T15:49:56.059 INFO:tasks.workunit.client.1.smithi027.stdout:8/369: chown d1/d3/f13 0 1 2024-04-22T15:49:56.066 INFO:tasks.workunit.client.1.smithi027.stdout:0/413: sync 2024-04-22T15:49:56.067 INFO:tasks.workunit.client.1.smithi027.stdout:0/414: write d0/d2a/d38/f6b [1974839,48643] 0 2024-04-22T15:49:56.067 INFO:tasks.workunit.client.1.smithi027.stdout:1/351: getdents d2/d36 0 2024-04-22T15:49:56.070 INFO:tasks.workunit.client.1.smithi027.stdout:6/426: truncate d5/dc/d26/f2d 1695277 0 2024-04-22T15:49:56.072 INFO:tasks.workunit.client.1.smithi027.stdout:5/454: creat d0/df/d3d/d81/f98 x:0 0 0 2024-04-22T15:49:56.072 INFO:tasks.workunit.client.1.smithi027.stdout:3/456: rmdir d28/d4b/d52 39 2024-04-22T15:49:56.073 INFO:tasks.workunit.client.1.smithi027.stdout:3/457: readlink d28/l30 0 2024-04-22T15:49:56.074 INFO:tasks.workunit.client.1.smithi027.stdout:9/323: truncate d0/dc/de/f40 1411079 0 2024-04-22T15:49:56.074 INFO:tasks.workunit.client.1.smithi027.stdout:9/324: readlink d0/dc/d12/l19 0 2024-04-22T15:49:56.075 INFO:tasks.workunit.client.1.smithi027.stdout:2/411: creat dd/f7b x:0 0 0 2024-04-22T15:49:56.077 INFO:tasks.workunit.client.1.smithi027.stdout:2/412: write dd/df/d70/d4b/f68 [561475,54139] 0 2024-04-22T15:49:56.083 INFO:tasks.workunit.client.1.smithi027.stdout:2/413: dwrite dd/f19 [0,4194304] 0 2024-04-22T15:49:56.083 INFO:tasks.workunit.client.1.smithi027.stdout:2/414: readlink dd/l3a 0 2024-04-22T15:49:56.085 INFO:tasks.workunit.client.1.smithi027.stdout:4/419: symlink d5/d11/d6c/da3/la7 0 2024-04-22T15:49:56.086 INFO:tasks.workunit.client.1.smithi027.stdout:4/420: chown d5/d11/d1d/d54 2 1 2024-04-22T15:49:56.087 INFO:tasks.workunit.client.1.smithi027.stdout:5/455: mkdir d0/d1a/d22/d26/d5d/d99 0 2024-04-22T15:49:56.088 INFO:tasks.workunit.client.1.smithi027.stdout:4/421: write d5/de/d30/d5d/f9f [274733,97967] 0 2024-04-22T15:49:56.090 INFO:tasks.workunit.client.1.smithi027.stdout:3/458: mkdir d28/d4b/d5c/d95 0 2024-04-22T15:49:56.091 INFO:tasks.workunit.client.1.smithi027.stdout:4/422: write d5/de/d30/d5d/f6a [341242,65899] 0 2024-04-22T15:49:56.092 INFO:tasks.workunit.client.1.smithi027.stdout:3/459: fdatasync d28/d5d/d62/f66 0 2024-04-22T15:49:56.103 INFO:tasks.workunit.client.1.smithi027.stdout:1/352: mkdir d2/d36/d38/d7e 0 2024-04-22T15:49:56.106 INFO:tasks.workunit.client.1.smithi027.stdout:8/370: creat d1/d3/d1c/d56/f7f x:0 0 0 2024-04-22T15:49:56.108 INFO:tasks.workunit.client.1.smithi027.stdout:2/415: mknod dd/df/d1b/d38/c7c 0 2024-04-22T15:49:56.110 INFO:tasks.workunit.client.1.smithi027.stdout:0/415: dwrite d0/f14 [0,4194304] 0 2024-04-22T15:49:56.110 INFO:tasks.workunit.client.1.smithi027.stdout:6/427: unlink d5/l5a 0 2024-04-22T15:49:56.111 INFO:tasks.workunit.client.1.smithi027.stdout:3/460: dread f15 [0,4194304] 0 2024-04-22T15:49:56.113 INFO:tasks.workunit.client.1.smithi027.stdout:5/456: rename d0/d1a/d22/d26/f3b to d0/d1a/d22/d26/d5d/f9a 0 2024-04-22T15:49:56.119 INFO:tasks.workunit.client.1.smithi027.stdout:8/371: creat d1/d3/d1c/d43/f80 x:0 0 0 2024-04-22T15:49:56.120 INFO:tasks.workunit.client.1.smithi027.stdout:2/416: mknod dd/df/d1b/d38/c7d 0 2024-04-22T15:49:56.121 INFO:tasks.workunit.client.1.smithi027.stdout:8/372: truncate d1/d3/d1c/d43/f71 391634 0 2024-04-22T15:49:56.121 INFO:tasks.workunit.client.1.smithi027.stdout:6/428: rmdir d5/dc/d20 39 2024-04-22T15:49:56.122 INFO:tasks.workunit.client.1.smithi027.stdout:3/461: rmdir d28/d69 39 2024-04-22T15:49:56.122 INFO:tasks.workunit.client.1.smithi027.stdout:8/373: stat d1/d3/d1c/d32/d41/l6e 0 2024-04-22T15:49:56.123 INFO:tasks.workunit.client.1.smithi027.stdout:8/374: dread - d1/d3/d1c/d32/d41/f7a zero size 2024-04-22T15:49:56.126 INFO:tasks.workunit.client.1.smithi027.stdout:4/423: mkdir d5/da8 0 2024-04-22T15:49:56.129 INFO:tasks.workunit.client.1.smithi027.stdout:5/457: unlink d0/d1a/d22/d26/d5d/c93 0 2024-04-22T15:49:56.131 INFO:tasks.workunit.client.1.smithi027.stdout:9/325: symlink d0/d51/l6d 0 2024-04-22T15:49:56.137 INFO:tasks.workunit.client.1.smithi027.stdout:2/417: write f7 [4743366,40706] 0 2024-04-22T15:49:56.137 INFO:tasks.workunit.client.1.smithi027.stdout:9/326: dwrite d0/dc/d12/d30/f32 [0,4194304] 0 2024-04-22T15:49:56.139 INFO:tasks.workunit.client.1.smithi027.stdout:9/327: chown d0/dc/d12/d58 43266 1 2024-04-22T15:49:56.140 INFO:tasks.workunit.client.1.smithi027.stdout:8/375: symlink d1/d3/d1c/d32/d41/l81 0 2024-04-22T15:49:56.142 INFO:tasks.workunit.client.1.smithi027.stdout:9/328: write d0/dc/d12/d30/f67 [454189,111589] 0 2024-04-22T15:49:56.145 INFO:tasks.workunit.client.1.smithi027.stdout:4/424: mkdir d5/de/da9 0 2024-04-22T15:49:56.145 INFO:tasks.workunit.client.1.smithi027.stdout:9/329: fdatasync d0/d8/f48 0 2024-04-22T15:49:56.147 INFO:tasks.workunit.client.1.smithi027.stdout:5/458: creat d0/df/de/d24/d1c/d3e/f9b x:0 0 0 2024-04-22T15:49:56.147 INFO:tasks.workunit.client.1.smithi027.stdout:4/425: readlink d5/d11/d6c/da3/la7 0 2024-04-22T15:49:56.148 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:55 smithi016 ceph-mon[26620]: pgmap v40: 65 pgs: 65 active+clean; 2.7 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 50 MiB/s rd, 180 MiB/s wr, 344 op/s 2024-04-22T15:49:56.149 INFO:tasks.workunit.client.1.smithi027.stdout:1/353: creat d2/f7f x:0 0 0 2024-04-22T15:49:56.152 INFO:tasks.workunit.client.1.smithi027.stdout:2/418: creat dd/df/d70/d5a/f7e x:0 0 0 2024-04-22T15:49:56.152 INFO:tasks.workunit.client.1.smithi027.stdout:5/459: fsync d0/d1a/d22/d72/d86/f77 0 2024-04-22T15:49:56.153 INFO:tasks.workunit.client.1.smithi027.stdout:0/416: rmdir d0/d2a/d38/d70/d7c 0 2024-04-22T15:49:56.154 INFO:tasks.workunit.client.1.smithi027.stdout:7/364: dread d0/d11/fe [0,4194304] 0 2024-04-22T15:49:56.156 INFO:tasks.workunit.client.1.smithi027.stdout:9/330: rmdir d0/d8 39 2024-04-22T15:49:56.157 INFO:tasks.workunit.client.1.smithi027.stdout:4/426: truncate d5/de/d2a/d58/f68 1574363 0 2024-04-22T15:49:56.157 INFO:tasks.workunit.client.1.smithi027.stdout:8/376: dwrite d1/d3/d1c/d56/f6a [0,4194304] 0 2024-04-22T15:49:56.158 INFO:tasks.workunit.client.1.smithi027.stdout:1/354: truncate d2/d36/d74/f79 457607 0 2024-04-22T15:49:56.161 INFO:tasks.workunit.client.1.smithi027.stdout:1/355: stat d2/d36/d38/d7e 0 2024-04-22T15:49:56.163 INFO:tasks.workunit.client.1.smithi027.stdout:9/331: write d0/dc/d12/d30/f4f [9144258,34039] 0 2024-04-22T15:49:56.163 INFO:tasks.workunit.client.1.smithi027.stdout:4/427: write d5/d11/d89/f84 [975708,43315] 0 2024-04-22T15:49:56.166 INFO:tasks.workunit.client.1.smithi027.stdout:2/419: creat dd/d47/f7f x:0 0 0 2024-04-22T15:49:56.170 INFO:tasks.workunit.client.1.smithi027.stdout:8/377: write d1/d3/d1c/d32/d41/f7a [980049,57378] 0 2024-04-22T15:49:56.174 INFO:tasks.workunit.client.1.smithi027.stdout:1/356: read d2/d1d/f32 [1077912,25668] 0 2024-04-22T15:49:56.174 INFO:tasks.workunit.client.1.smithi027.stdout:3/462: dread d28/d4b/d52/f6a [0,4194304] 0 2024-04-22T15:49:56.176 INFO:tasks.workunit.client.1.smithi027.stdout:0/417: dwrite d0/d28/d37/f40 [0,4194304] 0 2024-04-22T15:49:56.176 INFO:tasks.workunit.client.1.smithi027.stdout:2/420: chown dd/df/d1b/f76 73 1 2024-04-22T15:49:56.179 INFO:tasks.workunit.client.1.smithi027.stdout:4/428: truncate d5/d45/d81/f98 970714 0 2024-04-22T15:49:56.179 INFO:tasks.workunit.client.1.smithi027.stdout:0/418: stat d0/l16 0 2024-04-22T15:49:56.179 INFO:tasks.workunit.client.1.smithi027.stdout:5/460: dwrite d0/d1a/d22/d26/d5d/f8b [0,4194304] 0 2024-04-22T15:49:56.181 INFO:tasks.workunit.client.1.smithi027.stdout:0/419: fsync d0/d34/f5a 0 2024-04-22T15:49:56.182 INFO:tasks.workunit.client.1.smithi027.stdout:5/461: truncate d0/df/d3d/f8e 354100 0 2024-04-22T15:49:56.186 INFO:tasks.workunit.client.1.smithi027.stdout:7/365: dread d0/d16/f4a [0,4194304] 0 2024-04-22T15:49:56.192 INFO:tasks.workunit.client.1.smithi027.stdout:7/366: dread d0/d26/f42 [0,4194304] 0 2024-04-22T15:49:56.195 INFO:tasks.workunit.client.1.smithi027.stdout:3/463: write d28/f37 [4345456,79872] 0 2024-04-22T15:49:56.198 INFO:tasks.workunit.client.1.smithi027.stdout:4/429: mknod d5/de/d2a/d58/caa 0 2024-04-22T15:49:56.199 INFO:tasks.workunit.client.1.smithi027.stdout:7/367: dread d0/d3f/f5b [0,4194304] 0 2024-04-22T15:49:56.200 INFO:tasks.workunit.client.1.smithi027.stdout:0/420: mknod d0/d34/d4e/c7e 0 2024-04-22T15:49:56.202 INFO:tasks.workunit.client.1.smithi027.stdout:7/368: write d0/d11/f5c [440783,26143] 0 2024-04-22T15:49:56.203 INFO:tasks.workunit.client.1.smithi027.stdout:5/462: write d0/df/de/d24/d1c/d3e/d55/f7d [878808,77851] 0 2024-04-22T15:49:56.203 INFO:tasks.workunit.client.1.smithi027.stdout:0/421: write d0/d34/f71 [16031,9939] 0 2024-04-22T15:49:56.203 INFO:tasks.workunit.client.1.smithi027.stdout:8/378: truncate d1/d3/f46 1256475 0 2024-04-22T15:49:56.204 INFO:tasks.workunit.client.1.smithi027.stdout:7/369: fdatasync d0/d4c/d4e/f73 0 2024-04-22T15:49:56.204 INFO:tasks.workunit.client.1.smithi027.stdout:5/463: chown d0/d1a/d22/d72/d86/f75 15 1 2024-04-22T15:49:56.205 INFO:tasks.workunit.client.1.smithi027.stdout:1/357: getdents d2/d36/d38/d69/d7d 0 2024-04-22T15:49:56.206 INFO:tasks.workunit.client.1.smithi027.stdout:1/358: write d2/f7f [109773,49375] 0 2024-04-22T15:49:56.207 INFO:tasks.workunit.client.1.smithi027.stdout:3/464: truncate d28/d69/f78 315849 0 2024-04-22T15:49:56.208 INFO:tasks.workunit.client.1.smithi027.stdout:2/421: link dd/df/d1b/d1f/f2c dd/df/d1b/d38/d60/f80 0 2024-04-22T15:49:56.217 INFO:tasks.workunit.client.1.smithi027.stdout:7/370: symlink d0/d11/d3a/d46/l76 0 2024-04-22T15:49:56.218 INFO:tasks.workunit.client.1.smithi027.stdout:5/464: unlink d0/d1a/d22/d26/d32/c5b 0 2024-04-22T15:49:56.218 INFO:tasks.workunit.client.1.smithi027.stdout:8/379: write d1/d3/f59 [508799,22835] 0 2024-04-22T15:49:56.219 INFO:tasks.workunit.client.1.smithi027.stdout:1/359: mknod d2/d5f/d64/c80 0 2024-04-22T15:49:56.223 INFO:tasks.workunit.client.1.smithi027.stdout:8/380: fsync d1/d3/d1c/f34 0 2024-04-22T15:49:56.229 INFO:tasks.workunit.client.1.smithi027.stdout:9/332: dwrite d0/dc/f3d [0,4194304] 0 2024-04-22T15:49:56.232 INFO:tasks.workunit.client.1.smithi027.stdout:3/465: write d28/d4b/d52/f7a [496827,17501] 0 2024-04-22T15:49:56.235 INFO:tasks.workunit.client.1.smithi027.stdout:1/360: rename d2/d36/d38 to d2/d5f/d64/d81 0 2024-04-22T15:49:56.238 INFO:tasks.workunit.client.1.smithi027.stdout:2/422: truncate dd/df/d1b/d38/d60/f80 2322070 0 2024-04-22T15:49:56.244 INFO:tasks.workunit.client.1.smithi027.stdout:4/430: dwrite d5/d42/f61 [0,4194304] 0 2024-04-22T15:49:56.244 INFO:tasks.workunit.client.1.smithi027.stdout:7/371: dwrite d0/d11/d1c/d2c/f55 [0,4194304] 0 2024-04-22T15:49:56.246 INFO:tasks.workunit.client.1.smithi027.stdout:5/465: dwrite d0/df/de/f18 [0,4194304] 0 2024-04-22T15:49:56.247 INFO:tasks.workunit.client.1.smithi027.stdout:8/381: dwrite d1/d3/f40 [8388608,4194304] 0 2024-04-22T15:49:56.250 INFO:tasks.workunit.client.1.smithi027.stdout:5/466: read d0/f3c [6642929,2948] 0 2024-04-22T15:49:56.257 INFO:tasks.workunit.client.1.smithi027.stdout:9/333: dread d0/dc/de/f49 [4194304,4194304] 0 2024-04-22T15:49:56.259 INFO:tasks.workunit.client.1.smithi027.stdout:7/372: creat d0/d16/d52/f77 x:0 0 0 2024-04-22T15:49:56.264 INFO:tasks.workunit.client.1.smithi027.stdout:7/373: fsync d0/d16/f3b 0 2024-04-22T15:49:56.264 INFO:tasks.workunit.client.1.smithi027.stdout:3/466: creat d28/d5d/f96 x:0 0 0 2024-04-22T15:49:56.265 INFO:tasks.workunit.client.1.smithi027.stdout:9/334: chown d0/dc/d12/l19 5506 1 2024-04-22T15:49:56.268 INFO:tasks.workunit.client.1.smithi027.stdout:2/423: dwrite dd/df/f59 [0,4194304] 0 2024-04-22T15:49:56.268 INFO:tasks.workunit.client.1.smithi027.stdout:8/382: dread d1/d3/d1c/f34 [0,4194304] 0 2024-04-22T15:49:56.272 INFO:tasks.workunit.client.1.smithi027.stdout:4/431: dwrite d5/de/f16 [0,4194304] 0 2024-04-22T15:49:56.273 INFO:tasks.workunit.client.1.smithi027.stdout:1/361: dread d2/d36/f55 [0,4194304] 0 2024-04-22T15:49:56.274 INFO:tasks.workunit.client.1.smithi027.stdout:8/383: fsync d1/d1e/d47/d53/f78 0 2024-04-22T15:49:56.275 INFO:tasks.workunit.client.1.smithi027.stdout:3/467: symlink d28/d5d/l97 0 2024-04-22T15:49:56.276 INFO:tasks.workunit.client.1.smithi027.stdout:2/424: creat dd/df/d70/d5a/f81 x:0 0 0 2024-04-22T15:49:56.277 INFO:tasks.workunit.client.1.smithi027.stdout:7/374: truncate d0/d4c/d4e/f67 516938 0 2024-04-22T15:49:56.281 INFO:tasks.workunit.client.1.smithi027.stdout:9/335: rename d0/dc/df to d0/dc/d12/d58/d6b/d6e 0 2024-04-22T15:49:56.282 INFO:tasks.workunit.client.1.smithi027.stdout:3/468: rmdir d28/d69 39 2024-04-22T15:49:56.285 INFO:tasks.workunit.client.1.smithi027.stdout:4/432: unlink d5/d11/d6c/da3/la7 0 2024-04-22T15:49:56.285 INFO:tasks.workunit.client.1.smithi027.stdout:1/362: dwrite d2/d36/f55 [0,4194304] 0 2024-04-22T15:49:56.285 INFO:tasks.workunit.client.1.smithi027.stdout:3/469: truncate d28/d4b/d52/f54 4916675 0 2024-04-22T15:49:56.286 INFO:tasks.workunit.client.1.smithi027.stdout:8/384: mkdir d1/d3/d1c/d43/d82 0 2024-04-22T15:49:56.289 INFO:tasks.workunit.client.1.smithi027.stdout:7/375: mkdir d0/d26/d78 0 2024-04-22T15:49:56.292 INFO:tasks.workunit.client.1.smithi027.stdout:2/425: write dd/f4d [1820384,125302] 0 2024-04-22T15:49:56.292 INFO:tasks.workunit.client.1.smithi027.stdout:7/376: readlink d0/d11/d1c/d54/l6f 0 2024-04-22T15:49:56.294 INFO:tasks.workunit.client.1.smithi027.stdout:2/426: chown dd/df/d1b/f76 1593 1 2024-04-22T15:49:56.296 INFO:tasks.workunit.client.1.smithi027.stdout:7/377: truncate d0/d26/f69 109068 0 2024-04-22T15:49:56.298 INFO:tasks.workunit.client.1.smithi027.stdout:4/433: dwrite d5/d45/d47/f4a [0,4194304] 0 2024-04-22T15:49:56.300 INFO:tasks.workunit.client.1.smithi027.stdout:4/434: stat d5/f13 0 2024-04-22T15:49:56.300 INFO:tasks.workunit.client.1.smithi027.stdout:3/470: chown d28/d69/c93 134970 1 2024-04-22T15:49:56.300 INFO:tasks.workunit.client.1.smithi027.stdout:1/363: mkdir d2/d5f/d64/d81/d3d/d82 0 2024-04-22T15:49:56.303 INFO:tasks.workunit.client.1.smithi027.stdout:9/336: dwrite d0/d8/f11 [0,4194304] 0 2024-04-22T15:49:56.304 INFO:tasks.workunit.client.1.smithi027.stdout:8/385: creat d1/d3/d1c/d32/d41/f83 x:0 0 0 2024-04-22T15:49:56.314 INFO:tasks.workunit.client.1.smithi027.stdout:3/471: symlink d28/d5d/d79/l98 0 2024-04-22T15:49:56.315 INFO:tasks.workunit.client.1.smithi027.stdout:9/337: dwrite d0/f44 [0,4194304] 0 2024-04-22T15:49:56.315 INFO:tasks.workunit.client.1.smithi027.stdout:4/435: dwrite d5/f93 [0,4194304] 0 2024-04-22T15:49:56.319 INFO:tasks.workunit.client.1.smithi027.stdout:1/364: creat d2/d36/d74/f83 x:0 0 0 2024-04-22T15:49:56.321 INFO:tasks.workunit.client.1.smithi027.stdout:1/365: mknod d2/d5f/c84 0 2024-04-22T15:49:56.323 INFO:tasks.workunit.client.1.smithi027.stdout:1/366: chown d2/f57 236 1 2024-04-22T15:49:56.323 INFO:tasks.workunit.client.1.smithi027.stdout:9/338: mknod d0/c6f 0 2024-04-22T15:49:56.329 INFO:tasks.workunit.client.1.smithi027.stdout:1/367: dread d2/d36/f55 [0,4194304] 0 2024-04-22T15:49:56.330 INFO:tasks.workunit.client.1.smithi027.stdout:9/339: symlink d0/d64/l70 0 2024-04-22T15:49:56.335 INFO:tasks.workunit.client.1.smithi027.stdout:4/436: dread d5/f13 [0,4194304] 0 2024-04-22T15:49:56.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:55 smithi027 ceph-mon[34922]: pgmap v40: 65 pgs: 65 active+clean; 2.7 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 50 MiB/s rd, 180 MiB/s wr, 344 op/s 2024-04-22T15:49:56.336 INFO:tasks.workunit.client.1.smithi027.stdout:9/340: rename d0/d8/c57 to d0/d64/c71 0 2024-04-22T15:49:56.337 INFO:tasks.workunit.client.1.smithi027.stdout:1/368: dwrite d2/d36/d74/f83 [0,4194304] 0 2024-04-22T15:49:56.338 INFO:tasks.workunit.client.1.smithi027.stdout:4/437: write d5/d11/d1d/f5e [66916,30044] 0 2024-04-22T15:49:56.341 INFO:tasks.workunit.client.1.smithi027.stdout:2/427: write dd/df/d1b/d38/f3b [1284633,94132] 0 2024-04-22T15:49:56.341 INFO:tasks.workunit.client.1.smithi027.stdout:4/438: dread - d5/de/d30/d5d/f99 zero size 2024-04-22T15:49:56.344 INFO:tasks.workunit.client.1.smithi027.stdout:9/341: creat d0/dc/f72 x:0 0 0 2024-04-22T15:49:56.346 INFO:tasks.workunit.client.1.smithi027.stdout:3/472: write d28/f88 [1281587,41275] 0 2024-04-22T15:49:56.348 INFO:tasks.workunit.client.1.smithi027.stdout:3/473: stat l8 0 2024-04-22T15:49:56.354 INFO:tasks.workunit.client.1.smithi027.stdout:1/369: dwrite d2/d65/f67 [0,4194304] 0 2024-04-22T15:49:56.356 INFO:tasks.workunit.client.1.smithi027.stdout:9/342: mknod d0/dc/de/c73 0 2024-04-22T15:49:56.359 INFO:tasks.workunit.client.1.smithi027.stdout:3/474: rename d28/l2d to d28/d4b/d52/l99 0 2024-04-22T15:49:56.359 INFO:tasks.workunit.client.1.smithi027.stdout:1/370: stat d2/d5f/d64/l6e 0 2024-04-22T15:49:56.360 INFO:tasks.workunit.client.1.smithi027.stdout:2/428: dwrite dd/d47/f48 [0,4194304] 0 2024-04-22T15:49:56.361 INFO:tasks.workunit.client.1.smithi027.stdout:4/439: dwrite d5/d11/f7a [0,4194304] 0 2024-04-22T15:49:56.363 INFO:tasks.workunit.client.1.smithi027.stdout:4/440: chown d5/d11/d6c/da3 1023011 1 2024-04-22T15:49:56.371 INFO:tasks.workunit.client.1.smithi027.stdout:3/475: mkdir d28/d4b/d52/d9a 0 2024-04-22T15:49:56.374 INFO:tasks.workunit.client.1.smithi027.stdout:3/476: fdatasync d28/f37 0 2024-04-22T15:49:56.374 INFO:tasks.workunit.client.1.smithi027.stdout:9/343: dwrite d0/d8/f18 [0,4194304] 0 2024-04-22T15:49:56.374 INFO:tasks.workunit.client.1.smithi027.stdout:1/371: mkdir d2/d5f/d64/d81/d3d/d82/d85 0 2024-04-22T15:49:56.375 INFO:tasks.workunit.client.1.smithi027.stdout:4/441: dread d5/f93 [0,4194304] 0 2024-04-22T15:49:56.383 INFO:tasks.workunit.client.1.smithi027.stdout:3/477: getdents d28/d4b/d5c 0 2024-04-22T15:49:56.383 INFO:tasks.workunit.client.1.smithi027.stdout:1/372: dwrite d2/f75 [0,4194304] 0 2024-04-22T15:49:56.384 INFO:tasks.workunit.client.1.smithi027.stdout:1/373: readlink d2/l31 0 2024-04-22T15:49:56.384 INFO:tasks.workunit.client.1.smithi027.stdout:3/478: write d28/d5d/f96 [811729,78854] 0 2024-04-22T15:49:56.389 INFO:tasks.workunit.client.1.smithi027.stdout:3/479: mknod d28/d4b/d5c/c9b 0 2024-04-22T15:49:56.392 INFO:tasks.workunit.client.1.smithi027.stdout:3/480: mkdir d28/d4b/d5c/d95/d9c 0 2024-04-22T15:49:56.396 INFO:tasks.workunit.client.1.smithi027.stdout:4/442: dread d5/d11/d6c/f8b [0,4194304] 0 2024-04-22T15:49:56.405 INFO:tasks.workunit.client.1.smithi027.stdout:3/481: dread d28/d4b/f5b [0,4194304] 0 2024-04-22T15:49:56.409 INFO:tasks.workunit.client.1.smithi027.stdout:6/429: sync 2024-04-22T15:49:56.412 INFO:tasks.workunit.client.1.smithi027.stdout:1/374: write d2/d30/f41 [2188,102220] 0 2024-04-22T15:49:56.413 INFO:tasks.workunit.client.1.smithi027.stdout:3/482: mknod d28/d5d/d79/c9d 0 2024-04-22T15:49:56.419 INFO:tasks.workunit.client.1.smithi027.stdout:4/443: link d5/de/d30/d5d/f99 d5/de/fab 0 2024-04-22T15:49:56.422 INFO:tasks.workunit.client.1.smithi027.stdout:2/429: truncate dd/df/d70/d4b/d4e/f51 285215 0 2024-04-22T15:49:56.424 INFO:tasks.workunit.client.1.smithi027.stdout:1/375: creat d2/d5f/d64/d81/d3d/d44/f86 x:0 0 0 2024-04-22T15:49:56.426 INFO:tasks.workunit.client.1.smithi027.stdout:2/430: dread - dd/df/d1b/d1f/f30 zero size 2024-04-22T15:49:56.428 INFO:tasks.workunit.client.1.smithi027.stdout:6/430: mknod d5/dc/d20/d2b/c94 0 2024-04-22T15:49:56.429 INFO:tasks.workunit.client.1.smithi027.stdout:9/344: dwrite d0/dc/de/f24 [0,4194304] 0 2024-04-22T15:49:56.431 INFO:tasks.workunit.client.1.smithi027.stdout:6/431: readlink d5/db/d2e/l19 0 2024-04-22T15:49:56.432 INFO:tasks.workunit.client.1.smithi027.stdout:4/444: dwrite d5/d11/d6c/f95 [0,4194304] 0 2024-04-22T15:49:56.434 INFO:tasks.workunit.client.1.smithi027.stdout:4/445: chown d5/d11/d1d/ca0 65266319 1 2024-04-22T15:49:56.435 INFO:tasks.workunit.client.1.smithi027.stdout:2/431: unlink dd/df/d1b/d38/d60/f6d 0 2024-04-22T15:49:56.436 INFO:tasks.workunit.client.1.smithi027.stdout:2/432: chown dd/df/d70/d5a/f7e 65393 1 2024-04-22T15:49:56.436 INFO:tasks.workunit.client.1.smithi027.stdout:9/345: fsync d0/dc/de/f41 0 2024-04-22T15:49:56.438 INFO:tasks.workunit.client.1.smithi027.stdout:9/346: chown d0/dc/de 839364335 1 2024-04-22T15:49:56.444 INFO:tasks.workunit.client.1.smithi027.stdout:3/483: write d28/f49 [1538195,24975] 0 2024-04-22T15:49:56.447 INFO:tasks.workunit.client.1.smithi027.stdout:4/446: creat d5/d45/d47/fac x:0 0 0 2024-04-22T15:49:56.448 INFO:tasks.workunit.client.1.smithi027.stdout:4/447: read d5/d11/d1d/f27 [4515620,7965] 0 2024-04-22T15:49:56.452 INFO:tasks.workunit.client.1.smithi027.stdout:9/347: fsync d0/dc/d12/f23 0 2024-04-22T15:49:56.452 INFO:tasks.workunit.client.1.smithi027.stdout:6/432: mknod d5/dc/d17/d8a/c95 0 2024-04-22T15:49:56.453 INFO:tasks.workunit.client.1.smithi027.stdout:9/348: truncate d0/dc/f4c 1040239 0 2024-04-22T15:49:56.453 INFO:tasks.workunit.client.1.smithi027.stdout:3/484: symlink d28/d69/l9e 0 2024-04-22T15:49:56.457 INFO:tasks.workunit.client.1.smithi027.stdout:4/448: creat d5/d45/fad x:0 0 0 2024-04-22T15:49:56.463 INFO:tasks.workunit.client.1.smithi027.stdout:3/485: unlink d28/d5d/f84 0 2024-04-22T15:49:56.465 INFO:tasks.workunit.client.1.smithi027.stdout:3/486: truncate d28/d4b/d52/f7a 1800663 0 2024-04-22T15:49:56.466 INFO:tasks.workunit.client.1.smithi027.stdout:3/487: write d28/f37 [49852,64354] 0 2024-04-22T15:49:56.467 INFO:tasks.workunit.client.1.smithi027.stdout:3/488: chown d28/d4b/d52/f6a 78349848 1 2024-04-22T15:49:56.467 INFO:tasks.workunit.client.1.smithi027.stdout:1/376: truncate d2/d5f/d64/d81/d3d/d44/f73 1938712 0 2024-04-22T15:49:56.469 INFO:tasks.workunit.client.1.smithi027.stdout:2/433: creat dd/df/d1b/d1f/f82 x:0 0 0 2024-04-22T15:49:56.471 INFO:tasks.workunit.client.1.smithi027.stdout:6/433: mknod d5/dc/d20/c96 0 2024-04-22T15:49:56.474 INFO:tasks.workunit.client.1.smithi027.stdout:4/449: creat d5/d11/da6/fae x:0 0 0 2024-04-22T15:49:56.475 INFO:tasks.workunit.client.1.smithi027.stdout:4/450: write d5/d45/d47/f76 [3523225,119666] 0 2024-04-22T15:49:56.476 INFO:tasks.workunit.client.1.smithi027.stdout:6/434: creat d5/dc/d20/d2b/f97 x:0 0 0 2024-04-22T15:49:56.478 INFO:tasks.workunit.client.1.smithi027.stdout:3/489: symlink d28/d4b/d52/d94/l9f 0 2024-04-22T15:49:56.481 INFO:tasks.workunit.client.1.smithi027.stdout:2/434: link dd/df/d1b/d38/f6c dd/df/d1b/d38/d60/f83 0 2024-04-22T15:49:56.484 INFO:tasks.workunit.client.1.smithi027.stdout:3/490: dwrite d28/d5d/f96 [0,4194304] 0 2024-04-22T15:49:56.486 INFO:tasks.workunit.client.1.smithi027.stdout:9/349: getdents d0/dc/d12/d58 0 2024-04-22T15:49:56.488 INFO:tasks.workunit.client.1.smithi027.stdout:9/350: write d0/dc/d12/d30/f67 [27159,99462] 0 2024-04-22T15:49:56.489 INFO:tasks.workunit.client.1.smithi027.stdout:9/351: fdatasync d0/d8/f48 0 2024-04-22T15:49:56.490 INFO:tasks.workunit.client.1.smithi027.stdout:4/451: rename d5/f25 to d5/de/d2a/d58/faf 0 2024-04-22T15:49:56.494 INFO:tasks.workunit.client.1.smithi027.stdout:6/435: unlink d5/db/l2f 0 2024-04-22T15:49:56.496 INFO:tasks.workunit.client.1.smithi027.stdout:6/436: fsync d5/dc/f74 0 2024-04-22T15:49:56.496 INFO:tasks.workunit.client.1.smithi027.stdout:6/437: readlink d5/l7 0 2024-04-22T15:49:56.497 INFO:tasks.workunit.client.1.smithi027.stdout:9/352: dwrite d0/dc/d12/d30/f32 [0,4194304] 0 2024-04-22T15:49:56.499 INFO:tasks.workunit.client.1.smithi027.stdout:1/377: link d2/d1d/l2c d2/d5f/d64/d81/d3d/d82/d85/l87 0 2024-04-22T15:49:56.500 INFO:tasks.workunit.client.1.smithi027.stdout:2/435: mkdir dd/df/d70/d4b/d84 0 2024-04-22T15:49:56.503 INFO:tasks.workunit.client.1.smithi027.stdout:9/353: dread d0/f36 [0,4194304] 0 2024-04-22T15:49:56.503 INFO:tasks.workunit.client.1.smithi027.stdout:9/354: chown d0/f36 5737539 1 2024-04-22T15:49:56.511 INFO:tasks.workunit.client.1.smithi027.stdout:4/452: creat d5/d45/d5a/fb0 x:0 0 0 2024-04-22T15:49:56.513 INFO:tasks.workunit.client.1.smithi027.stdout:6/438: symlink d5/dc/dd/d79/d7b/d5d/l98 0 2024-04-22T15:49:56.513 INFO:tasks.workunit.client.1.smithi027.stdout:6/439: chown d5 1 1 2024-04-22T15:49:56.517 INFO:tasks.workunit.client.1.smithi027.stdout:3/491: rename c1b to d28/d4b/d5c/d95/ca0 0 2024-04-22T15:49:56.521 INFO:tasks.workunit.client.1.smithi027.stdout:2/436: creat dd/df/d1b/d1f/f85 x:0 0 0 2024-04-22T15:49:56.523 INFO:tasks.workunit.client.1.smithi027.stdout:4/453: rename d5/d11/l3d to d5/d11/d1d/d54/lb1 0 2024-04-22T15:49:56.526 INFO:tasks.workunit.client.1.smithi027.stdout:4/454: truncate d5/d11/d1d/d24/f51 75669 0 2024-04-22T15:49:56.526 INFO:tasks.workunit.client.1.smithi027.stdout:9/355: link d0/dc/l17 d0/d51/l74 0 2024-04-22T15:49:56.526 INFO:tasks.workunit.client.1.smithi027.stdout:4/455: chown d5/de/f21 177 1 2024-04-22T15:49:56.530 INFO:tasks.workunit.client.1.smithi027.stdout:3/492: dread f26 [0,4194304] 0 2024-04-22T15:49:56.531 INFO:tasks.workunit.client.1.smithi027.stdout:9/356: rmdir d0/dc/d12/d58/d6b 39 2024-04-22T15:49:56.535 INFO:tasks.workunit.client.1.smithi027.stdout:3/493: chown d28/d5d/d79/l92 279175855 1 2024-04-22T15:49:56.537 INFO:tasks.workunit.client.1.smithi027.stdout:9/357: mknod d0/dc/d12/d30/d5b/c75 0 2024-04-22T15:49:56.537 INFO:tasks.workunit.client.1.smithi027.stdout:9/358: dread - d0/d8/f48 zero size 2024-04-22T15:49:56.538 INFO:tasks.workunit.client.1.smithi027.stdout:9/359: chown d0/d8/ca 25 1 2024-04-22T15:49:56.539 INFO:tasks.workunit.client.1.smithi027.stdout:9/360: write d0/d8/f48 [563085,70651] 0 2024-04-22T15:49:56.540 INFO:tasks.workunit.client.1.smithi027.stdout:2/437: rename dd/df/d1b/d1f/f4a to dd/df/f86 0 2024-04-22T15:49:56.542 INFO:tasks.workunit.client.1.smithi027.stdout:3/494: creat d28/d5d/d79/fa1 x:0 0 0 2024-04-22T15:49:56.542 INFO:tasks.workunit.client.1.smithi027.stdout:6/440: getdents d5/dc/dd/d79/d7b/d5d 0 2024-04-22T15:49:56.544 INFO:tasks.workunit.client.1.smithi027.stdout:9/361: dread d0/d8/f11 [0,4194304] 0 2024-04-22T15:49:56.545 INFO:tasks.workunit.client.1.smithi027.stdout:9/362: dread - d0/f68 zero size 2024-04-22T15:49:56.548 INFO:tasks.workunit.client.1.smithi027.stdout:4/456: rename d5/de/d2a/c69 to d5/d7/cb2 0 2024-04-22T15:49:56.551 INFO:tasks.workunit.client.1.smithi027.stdout:9/363: dread d0/dc/de/f41 [0,4194304] 0 2024-04-22T15:49:56.555 INFO:tasks.workunit.client.1.smithi027.stdout:3/495: dwrite d28/d4b/f4c [0,4194304] 0 2024-04-22T15:49:56.557 INFO:tasks.workunit.client.1.smithi027.stdout:2/438: rename dd/d43/f44 to dd/df/d70/d5a/f87 0 2024-04-22T15:49:56.559 INFO:tasks.workunit.client.1.smithi027.stdout:6/441: dread d5/dc/dd/d79/d7b/d5d/f70 [0,4194304] 0 2024-04-22T15:49:56.565 INFO:tasks.workunit.client.1.smithi027.stdout:3/496: rmdir d28/d69 39 2024-04-22T15:49:56.570 INFO:tasks.workunit.client.1.smithi027.stdout:9/364: rename d0/dc/c45 to d0/dc/d12/d30/c76 0 2024-04-22T15:49:56.575 INFO:tasks.workunit.client.1.smithi027.stdout:3/497: write d28/d4b/d52/f6a [3679628,49151] 0 2024-04-22T15:49:56.592 INFO:tasks.workunit.client.1.smithi027.stdout:0/422: sync 2024-04-22T15:49:56.599 INFO:tasks.workunit.client.1.smithi027.stdout:0/423: rmdir d0/d2a/d58 39 2024-04-22T15:49:56.601 INFO:tasks.workunit.client.1.smithi027.stdout:4/457: getdents d5/d11/d1d 0 2024-04-22T15:49:56.601 INFO:tasks.workunit.client.1.smithi027.stdout:9/365: mknod d0/dc/d12/d58/d6b/d6e/c77 0 2024-04-22T15:49:56.603 INFO:tasks.workunit.client.1.smithi027.stdout:3/498: symlink d28/d4b/d52/d9a/la2 0 2024-04-22T15:49:56.604 INFO:tasks.workunit.client.1.smithi027.stdout:0/424: symlink d0/l7f 0 2024-04-22T15:49:56.605 INFO:tasks.workunit.client.1.smithi027.stdout:4/458: mkdir d5/d45/db3 0 2024-04-22T15:49:56.607 INFO:tasks.workunit.client.1.smithi027.stdout:0/425: write d0/d28/d37/f3b [1028772,30836] 0 2024-04-22T15:49:56.609 INFO:tasks.workunit.client.1.smithi027.stdout:4/459: write d5/d45/d5a/fb0 [642348,92534] 0 2024-04-22T15:49:56.610 INFO:tasks.workunit.client.1.smithi027.stdout:0/426: chown d0/d28/l27 42 1 2024-04-22T15:49:56.611 INFO:tasks.workunit.client.1.smithi027.stdout:4/460: fsync d5/f23 0 2024-04-22T15:49:56.612 INFO:tasks.workunit.client.1.smithi027.stdout:3/499: dwrite d28/d5d/d62/f66 [0,4194304] 0 2024-04-22T15:49:56.613 INFO:tasks.workunit.client.1.smithi027.stdout:4/461: chown d5/d45/d47/c50 0 1 2024-04-22T15:49:56.615 INFO:tasks.workunit.client.1.smithi027.stdout:0/427: creat d0/d34/f80 x:0 0 0 2024-04-22T15:49:56.615 INFO:tasks.workunit.client.1.smithi027.stdout:6/442: write d5/db/f1e [2349767,121248] 0 2024-04-22T15:49:56.615 INFO:tasks.workunit.client.1.smithi027.stdout:3/500: fsync d28/d4b/f7c 0 2024-04-22T15:49:56.617 INFO:tasks.workunit.client.1.smithi027.stdout:4/462: rmdir d5/d11/d1d/d54 39 2024-04-22T15:49:56.620 INFO:tasks.workunit.client.1.smithi027.stdout:0/428: read d0/fc [2789860,106298] 0 2024-04-22T15:49:56.622 INFO:tasks.workunit.client.1.smithi027.stdout:3/501: write d28/d69/f78 [1322368,66660] 0 2024-04-22T15:49:56.625 INFO:tasks.workunit.client.1.smithi027.stdout:4/463: mknod d5/d11/d1d/d54/cb4 0 2024-04-22T15:49:56.626 INFO:tasks.workunit.client.1.smithi027.stdout:2/439: dwrite dd/f1e [0,4194304] 0 2024-04-22T15:49:56.627 INFO:tasks.workunit.client.1.smithi027.stdout:9/366: dread d0/dc/d12/d58/d6b/d6e/f31 [0,4194304] 0 2024-04-22T15:49:56.633 INFO:tasks.workunit.client.1.smithi027.stdout:4/464: symlink d5/d11/da6/lb5 0 2024-04-22T15:49:56.635 INFO:tasks.workunit.client.1.smithi027.stdout:0/429: getdents d0/d28/d37 0 2024-04-22T15:49:56.636 INFO:tasks.workunit.client.1.smithi027.stdout:0/430: write d0/f2e [403269,114102] 0 2024-04-22T15:49:56.636 INFO:tasks.workunit.client.1.smithi027.stdout:9/367: mknod d0/dc/d12/d30/d5b/c78 0 2024-04-22T15:49:56.638 INFO:tasks.workunit.client.1.smithi027.stdout:3/502: mkdir d28/d4b/d5c/d95/d9c/da3 0 2024-04-22T15:49:56.638 INFO:tasks.workunit.client.1.smithi027.stdout:3/503: fdatasync f1c 0 2024-04-22T15:49:56.638 INFO:tasks.workunit.client.1.smithi027.stdout:9/368: readlink d0/dc/d12/d58/d6b/d6e/l56 0 2024-04-22T15:49:56.639 INFO:tasks.workunit.client.1.smithi027.stdout:9/369: dread - d0/dc/f72 zero size 2024-04-22T15:49:56.644 INFO:tasks.workunit.client.1.smithi027.stdout:0/431: creat d0/d34/f81 x:0 0 0 2024-04-22T15:49:56.655 INFO:tasks.workunit.client.1.smithi027.stdout:2/440: getdents dd/df 0 2024-04-22T15:49:56.657 INFO:tasks.workunit.client.1.smithi027.stdout:2/441: write dd/df/f23 [1024097,4752] 0 2024-04-22T15:49:56.657 INFO:tasks.workunit.client.1.smithi027.stdout:3/504: rmdir d28/d4b/d5c/d95/d9c/da3 0 2024-04-22T15:49:56.658 INFO:tasks.workunit.client.1.smithi027.stdout:2/442: readlink dd/df/l3d 0 2024-04-22T15:49:56.661 INFO:tasks.workunit.client.1.smithi027.stdout:9/370: getdents d0/d8 0 2024-04-22T15:49:56.663 INFO:tasks.workunit.client.1.smithi027.stdout:0/432: write d0/f12 [651344,18197] 0 2024-04-22T15:49:56.665 INFO:tasks.workunit.client.1.smithi027.stdout:3/505: link d28/d69/f82 d28/d4b/d52/d94/fa4 0 2024-04-22T15:49:56.666 INFO:tasks.workunit.client.1.smithi027.stdout:9/371: rename d0/d8/ca to d0/d64/c79 0 2024-04-22T15:49:56.667 INFO:tasks.workunit.client.1.smithi027.stdout:0/433: creat d0/d28/d63/d67/f82 x:0 0 0 2024-04-22T15:49:56.670 INFO:tasks.workunit.client.1.smithi027.stdout:3/506: unlink d28/d5d/l97 0 2024-04-22T15:49:56.670 INFO:tasks.workunit.client.1.smithi027.stdout:3/507: fdatasync d28/f44 0 2024-04-22T15:49:56.671 INFO:tasks.workunit.client.1.smithi027.stdout:3/508: fsync d28/f45 0 2024-04-22T15:49:56.671 INFO:tasks.workunit.client.1.smithi027.stdout:9/372: creat d0/dc/d12/d30/f7a x:0 0 0 2024-04-22T15:49:56.672 INFO:tasks.workunit.client.1.smithi027.stdout:2/443: getdents dd/df 0 2024-04-22T15:49:56.677 INFO:tasks.workunit.client.1.smithi027.stdout:0/434: mkdir d0/d2a/d6a/d6c/d83 0 2024-04-22T15:49:56.680 INFO:tasks.workunit.client.1.smithi027.stdout:2/444: unlink dd/df/d1b/d38/c7c 0 2024-04-22T15:49:56.682 INFO:tasks.workunit.client.1.smithi027.stdout:3/509: rename d28/d5d/l68 to d28/d5d/d79/la5 0 2024-04-22T15:49:56.683 INFO:tasks.workunit.client.1.smithi027.stdout:0/435: symlink d0/d34/l84 0 2024-04-22T15:49:56.687 INFO:tasks.workunit.client.1.smithi027.stdout:3/510: dread d28/f49 [0,4194304] 0 2024-04-22T15:49:56.687 INFO:tasks.workunit.client.1.smithi027.stdout:3/511: readlink l27 0 2024-04-22T15:49:56.688 INFO:tasks.workunit.client.1.smithi027.stdout:0/436: mkdir d0/d34/d4e/d85 0 2024-04-22T15:49:56.692 INFO:tasks.workunit.client.1.smithi027.stdout:3/512: creat d28/d4b/d5c/fa6 x:0 0 0 2024-04-22T15:49:56.693 INFO:tasks.workunit.client.1.smithi027.stdout:3/513: truncate d28/f49 2256369 0 2024-04-22T15:49:56.693 INFO:tasks.workunit.client.1.smithi027.stdout:0/437: mknod d0/d28/d5f/c86 0 2024-04-22T15:49:56.695 INFO:tasks.workunit.client.1.smithi027.stdout:9/373: unlink d0/d8/f48 0 2024-04-22T15:49:56.696 INFO:tasks.workunit.client.1.smithi027.stdout:9/374: truncate d0/dc/d12/d58/d6b/f62 879433 0 2024-04-22T15:49:56.697 INFO:tasks.workunit.client.1.smithi027.stdout:9/375: write d0/dc/f3d [4254885,80144] 0 2024-04-22T15:49:56.701 INFO:tasks.workunit.client.1.smithi027.stdout:0/438: symlink d0/d2a/d38/l87 0 2024-04-22T15:49:56.702 INFO:tasks.workunit.client.1.smithi027.stdout:2/445: getdents dd/df/d70/d5a 0 2024-04-22T15:49:56.706 INFO:tasks.workunit.client.1.smithi027.stdout:3/514: mknod d28/d5d/ca7 0 2024-04-22T15:49:56.711 INFO:tasks.workunit.client.1.smithi027.stdout:0/439: symlink d0/d28/d5f/l88 0 2024-04-22T15:49:56.712 INFO:tasks.workunit.client.1.smithi027.stdout:0/440: write d0/d2a/d38/f55 [157936,60539] 0 2024-04-22T15:49:56.714 INFO:tasks.workunit.client.1.smithi027.stdout:2/446: creat dd/df/f88 x:0 0 0 2024-04-22T15:49:56.716 INFO:tasks.workunit.client.1.smithi027.stdout:9/376: fsync d0/dc/de/f3a 0 2024-04-22T15:49:56.723 INFO:tasks.workunit.client.1.smithi027.stdout:9/377: creat d0/d64/f7b x:0 0 0 2024-04-22T15:49:56.726 INFO:tasks.workunit.client.1.smithi027.stdout:0/441: dwrite d0/d34/f44 [0,4194304] 0 2024-04-22T15:49:56.729 INFO:tasks.workunit.client.1.smithi027.stdout:2/447: mknod dd/df/d1b/d38/d60/d75/c89 0 2024-04-22T15:49:56.730 INFO:tasks.workunit.client.1.smithi027.stdout:3/515: link d28/d4b/d5c/f6f d28/d5d/fa8 0 2024-04-22T15:49:56.732 INFO:tasks.workunit.client.1.smithi027.stdout:9/378: creat d0/d64/f7c x:0 0 0 2024-04-22T15:49:56.734 INFO:tasks.workunit.client.1.smithi027.stdout:0/442: mkdir d0/d2a/d38/d89 0 2024-04-22T15:49:56.734 INFO:tasks.workunit.client.1.smithi027.stdout:0/443: fsync d0/f1 0 2024-04-22T15:49:56.735 INFO:tasks.workunit.client.1.smithi027.stdout:9/379: truncate d0/dc/d12/d30/f67 991451 0 2024-04-22T15:49:56.736 INFO:tasks.workunit.client.1.smithi027.stdout:9/380: readlink d0/dc/l3e 0 2024-04-22T15:49:56.736 INFO:tasks.workunit.client.1.smithi027.stdout:2/448: rename dd/df/d1b/d38/f3b to dd/df/d70/d4b/d52/f8a 0 2024-04-22T15:49:56.737 INFO:tasks.workunit.client.1.smithi027.stdout:0/444: write d0/d2a/d38/f6b [2897803,10637] 0 2024-04-22T15:49:56.740 INFO:tasks.workunit.client.1.smithi027.stdout:3/516: symlink d28/d4b/d52/la9 0 2024-04-22T15:49:56.741 INFO:tasks.workunit.client.1.smithi027.stdout:3/517: dread d28/f36 [4194304,4194304] 0 2024-04-22T15:49:56.743 INFO:tasks.workunit.client.1.smithi027.stdout:3/518: write d28/d5d/d79/fa1 [1029453,72993] 0 2024-04-22T15:49:56.747 INFO:tasks.workunit.client.1.smithi027.stdout:9/381: readlink d0/dc/l17 0 2024-04-22T15:49:56.748 INFO:tasks.workunit.client.1.smithi027.stdout:9/382: read - d0/dc/de/f6c zero size 2024-04-22T15:49:56.749 INFO:tasks.workunit.client.1.smithi027.stdout:9/383: fsync d0/dc/f3d 0 2024-04-22T15:49:56.749 INFO:tasks.workunit.client.1.smithi027.stdout:2/449: mknod dd/df/d1b/d38/d60/d75/c8b 0 2024-04-22T15:49:56.755 INFO:tasks.workunit.client.1.smithi027.stdout:2/450: rename dd/c67 to dd/df/d1b/d38/c8c 0 2024-04-22T15:49:56.757 INFO:tasks.workunit.client.1.smithi027.stdout:0/445: link d0/d2a/l30 d0/d28/d37/l8a 0 2024-04-22T15:49:56.760 INFO:tasks.workunit.client.1.smithi027.stdout:9/384: creat d0/dc/d12/d58/f7d x:0 0 0 2024-04-22T15:49:56.760 INFO:tasks.workunit.client.1.smithi027.stdout:2/451: mkdir dd/df/d70/d4b/d84/d8d 0 2024-04-22T15:49:56.763 INFO:tasks.workunit.client.1.smithi027.stdout:0/446: rmdir d0/d28/d63 39 2024-04-22T15:49:56.764 INFO:tasks.workunit.client.1.smithi027.stdout:9/385: rmdir d0/d51 39 2024-04-22T15:49:56.766 INFO:tasks.workunit.client.1.smithi027.stdout:9/386: read d0/dc/f4c [841429,98190] 0 2024-04-22T15:49:56.767 INFO:tasks.workunit.client.1.smithi027.stdout:0/447: mkdir d0/d2a/d5e/d8b 0 2024-04-22T15:49:56.768 INFO:tasks.workunit.client.1.smithi027.stdout:2/452: dwrite dd/df/f59 [4194304,4194304] 0 2024-04-22T15:49:56.771 INFO:tasks.workunit.client.1.smithi027.stdout:0/448: mknod d0/d34/c8c 0 2024-04-22T15:49:56.775 INFO:tasks.workunit.client.1.smithi027.stdout:9/387: dwrite d0/dc/f72 [0,4194304] 0 2024-04-22T15:49:56.782 INFO:tasks.workunit.client.1.smithi027.stdout:0/449: write d0/d28/d63/d67/f82 [1034125,19359] 0 2024-04-22T15:49:56.782 INFO:tasks.workunit.client.1.smithi027.stdout:0/450: write d0/d2a/d38/f65 [4270675,22309] 0 2024-04-22T15:49:56.785 INFO:tasks.workunit.client.1.smithi027.stdout:0/451: creat d0/d28/d63/f8d x:0 0 0 2024-04-22T15:49:56.786 INFO:tasks.workunit.client.1.smithi027.stdout:0/452: truncate d0/d28/f5b 352965 0 2024-04-22T15:49:56.791 INFO:tasks.workunit.client.1.smithi027.stdout:5/467: sync 2024-04-22T15:49:56.795 INFO:tasks.workunit.client.1.smithi027.stdout:5/468: dread - d0/df/d10/f60 zero size 2024-04-22T15:49:56.797 INFO:tasks.workunit.client.1.smithi027.stdout:9/388: link d0/dc/c5e d0/dc/d12/d58/c7e 0 2024-04-22T15:49:56.798 INFO:tasks.workunit.client.1.smithi027.stdout:2/453: getdents dd/df/d70/d4b/d84 0 2024-04-22T15:49:56.800 INFO:tasks.workunit.client.1.smithi027.stdout:5/469: creat d0/df/de/d24/d1c/d3e/d55/f9c x:0 0 0 2024-04-22T15:49:56.803 INFO:tasks.workunit.client.1.smithi027.stdout:0/453: dread d0/f1 [0,4194304] 0 2024-04-22T15:49:56.806 INFO:tasks.workunit.client.1.smithi027.stdout:9/389: dwrite d0/d64/f7b [0,4194304] 0 2024-04-22T15:49:56.814 INFO:tasks.workunit.client.1.smithi027.stdout:0/454: mkdir d0/d2a/d6a/d8e 0 2024-04-22T15:49:56.815 INFO:tasks.workunit.client.1.smithi027.stdout:0/455: write d0/d34/d59/f5c [549002,4794] 0 2024-04-22T15:49:56.816 INFO:tasks.workunit.client.1.smithi027.stdout:0/456: fdatasync d0/f2e 0 2024-04-22T15:49:56.819 INFO:tasks.workunit.client.1.smithi027.stdout:5/470: creat d0/df/de/d24/f9d x:0 0 0 2024-04-22T15:49:56.822 INFO:tasks.workunit.client.1.smithi027.stdout:9/390: creat d0/d51/f7f x:0 0 0 2024-04-22T15:49:56.824 INFO:tasks.workunit.client.1.smithi027.stdout:2/454: getdents dd/df/d1b/d38/d60/d75 0 2024-04-22T15:49:56.826 INFO:tasks.workunit.client.1.smithi027.stdout:0/457: rename d0/d34/f44 to d0/d34/d4e/d85/f8f 0 2024-04-22T15:49:56.837 INFO:tasks.workunit.client.1.smithi027.stdout:9/391: truncate d0/dc/de/f49 1710543 0 2024-04-22T15:49:56.837 INFO:tasks.workunit.client.1.smithi027.stdout:9/392: stat d0/dc/l17 0 2024-04-22T15:49:56.841 INFO:tasks.workunit.client.1.smithi027.stdout:5/471: getdents d0/df/d3d/d81 0 2024-04-22T15:49:56.844 INFO:tasks.workunit.client.1.smithi027.stdout:9/393: dwrite d0/dc/f72 [0,4194304] 0 2024-04-22T15:49:56.849 INFO:tasks.workunit.client.1.smithi027.stdout:2/455: getdents dd/df/d70/d4b 0 2024-04-22T15:49:56.849 INFO:tasks.workunit.client.1.smithi027.stdout:2/456: stat dd/df/d70/d4b 0 2024-04-22T15:49:56.850 INFO:tasks.workunit.client.1.smithi027.stdout:0/458: link d0/l23 d0/d2a/d58/l90 0 2024-04-22T15:49:56.862 INFO:tasks.workunit.client.1.smithi027.stdout:2/457: creat dd/f8e x:0 0 0 2024-04-22T15:49:56.867 INFO:tasks.workunit.client.1.smithi027.stdout:0/459: getdents d0 0 2024-04-22T15:49:56.871 INFO:tasks.workunit.client.1.smithi027.stdout:2/458: dwrite dd/df/f59 [4194304,4194304] 0 2024-04-22T15:49:56.877 INFO:tasks.workunit.client.1.smithi027.stdout:2/459: read - dd/df/d1b/f41 zero size 2024-04-22T15:49:56.881 INFO:tasks.workunit.client.1.smithi027.stdout:0/460: mkdir d0/d2a/d91 0 2024-04-22T15:49:56.886 INFO:tasks.workunit.client.1.smithi027.stdout:2/460: creat dd/df/d70/d4b/d52/f8f x:0 0 0 2024-04-22T15:49:56.886 INFO:tasks.workunit.client.1.smithi027.stdout:0/461: dread d0/d28/d63/d67/f82 [0,4194304] 0 2024-04-22T15:49:56.887 INFO:tasks.workunit.client.1.smithi027.stdout:0/462: stat d0/d34/f71 0 2024-04-22T15:49:56.888 INFO:tasks.workunit.client.1.smithi027.stdout:0/463: chown d0/d34/f5a 210935 1 2024-04-22T15:49:56.889 INFO:tasks.workunit.client.1.smithi027.stdout:0/464: dread - d0/d34/f81 zero size 2024-04-22T15:49:56.894 INFO:tasks.workunit.client.1.smithi027.stdout:2/461: mknod dd/df/d70/d4b/c90 0 2024-04-22T15:49:56.897 INFO:tasks.workunit.client.1.smithi027.stdout:9/394: write d0/dc/f4c [1562186,48886] 0 2024-04-22T15:49:56.898 INFO:tasks.workunit.client.1.smithi027.stdout:9/395: fdatasync d0/dc/d12/d30/f32 0 2024-04-22T15:49:56.900 INFO:tasks.workunit.client.1.smithi027.stdout:5/472: truncate d0/df/de/f18 3003225 0 2024-04-22T15:49:56.903 INFO:tasks.workunit.client.1.smithi027.stdout:0/465: truncate d0/f22 2106289 0 2024-04-22T15:49:56.903 INFO:tasks.workunit.client.1.smithi027.stdout:0/466: readlink d0/l25 0 2024-04-22T15:49:56.907 INFO:tasks.workunit.client.1.smithi027.stdout:0/467: creat d0/d2a/d38/f92 x:0 0 0 2024-04-22T15:49:56.910 INFO:tasks.workunit.client.1.smithi027.stdout:0/468: dread d0/f2e [0,4194304] 0 2024-04-22T15:49:56.912 INFO:tasks.workunit.client.1.smithi027.stdout:5/473: truncate d0/d1a/d22/d72/d86/f77 784257 0 2024-04-22T15:49:56.915 INFO:tasks.workunit.client.1.smithi027.stdout:0/469: creat d0/d2a/d6a/d6c/f93 x:0 0 0 2024-04-22T15:49:56.920 INFO:tasks.workunit.client.1.smithi027.stdout:0/470: rename d0/l19 to d0/d28/d37/l94 0 2024-04-22T15:49:56.921 INFO:tasks.workunit.client.1.smithi027.stdout:0/471: read - d0/d2a/d38/f92 zero size 2024-04-22T15:49:56.925 INFO:tasks.workunit.client.1.smithi027.stdout:9/396: dwrite d0/dc/d12/d58/d6b/d6e/f31 [0,4194304] 0 2024-04-22T15:49:56.928 INFO:tasks.workunit.client.1.smithi027.stdout:0/472: dwrite d0/d28/d37/f3b [0,4194304] 0 2024-04-22T15:49:56.934 INFO:tasks.workunit.client.1.smithi027.stdout:5/474: getdents d0/df/d10 0 2024-04-22T15:49:56.935 INFO:tasks.workunit.client.1.smithi027.stdout:5/475: dread - d0/df/d10/f60 zero size 2024-04-22T15:49:56.935 INFO:tasks.workunit.client.1.smithi027.stdout:0/473: symlink d0/l95 0 2024-04-22T15:49:56.937 INFO:tasks.workunit.client.1.smithi027.stdout:5/476: write d0/df/de/d24/d1c/d8f/f94 [220108,20720] 0 2024-04-22T15:49:56.942 INFO:tasks.workunit.client.1.smithi027.stdout:9/397: link d0/f36 d0/dc/f80 0 2024-04-22T15:49:56.943 INFO:tasks.workunit.client.1.smithi027.stdout:9/398: fdatasync d0/d64/f7c 0 2024-04-22T15:49:56.944 INFO:tasks.workunit.client.1.smithi027.stdout:9/399: write d0/f47 [2607078,84597] 0 2024-04-22T15:49:56.945 INFO:tasks.workunit.client.1.smithi027.stdout:9/400: write d0/dc/f72 [616524,68590] 0 2024-04-22T15:49:56.947 INFO:tasks.workunit.client.1.smithi027.stdout:9/401: write d0/f68 [42249,114127] 0 2024-04-22T15:49:56.950 INFO:tasks.workunit.client.1.smithi027.stdout:9/402: dread d0/f68 [0,4194304] 0 2024-04-22T15:49:56.954 INFO:tasks.workunit.client.1.smithi027.stdout:0/474: truncate d0/f14 3059977 0 2024-04-22T15:49:56.957 INFO:tasks.workunit.client.1.smithi027.stdout:9/403: link d0/l13 d0/d64/l81 0 2024-04-22T15:49:56.974 INFO:tasks.workunit.client.1.smithi027.stdout:7/378: sync 2024-04-22T15:49:56.974 INFO:tasks.workunit.client.1.smithi027.stdout:1/378: sync 2024-04-22T15:49:56.974 INFO:tasks.workunit.client.1.smithi027.stdout:8/386: sync 2024-04-22T15:49:56.976 INFO:tasks.workunit.client.1.smithi027.stdout:9/404: mknod d0/dc/d12/d58/d6b/d6e/c82 0 2024-04-22T15:49:56.978 INFO:tasks.workunit.client.1.smithi027.stdout:1/379: chown d2/l28 615 1 2024-04-22T15:49:56.978 INFO:tasks.workunit.client.1.smithi027.stdout:0/475: dwrite d0/d34/f7a [0,4194304] 0 2024-04-22T15:49:56.985 INFO:tasks.workunit.client.1.smithi027.stdout:8/387: rmdir d1/d3 39 2024-04-22T15:49:56.986 INFO:tasks.workunit.client.1.smithi027.stdout:9/405: stat d0/c27 0 2024-04-22T15:49:56.987 INFO:tasks.workunit.client.1.smithi027.stdout:0/476: dwrite d0/d2a/d38/f92 [0,4194304] 0 2024-04-22T15:49:56.988 INFO:tasks.workunit.client.1.smithi027.stdout:1/380: rename d2/d5f/d64/d81/c6c to d2/d65/c88 0 2024-04-22T15:49:56.989 INFO:tasks.workunit.client.1.smithi027.stdout:9/406: dread d0/f36 [0,4194304] 0 2024-04-22T15:49:56.990 INFO:tasks.workunit.client.1.smithi027.stdout:9/407: chown d0/dc/d12/d30/c42 129 1 2024-04-22T15:49:57.000 INFO:tasks.workunit.client.1.smithi027.stdout:7/379: dread d0/f14 [0,4194304] 0 2024-04-22T15:49:57.001 INFO:tasks.workunit.client.1.smithi027.stdout:9/408: creat d0/dc/d12/d30/d5b/f83 x:0 0 0 2024-04-22T15:49:57.004 INFO:tasks.workunit.client.1.smithi027.stdout:0/477: mknod d0/d2a/d6a/d6c/d83/c96 0 2024-04-22T15:49:57.005 INFO:tasks.workunit.client.1.smithi027.stdout:1/381: rename d2/d36/l77 to d2/d30/l89 0 2024-04-22T15:49:57.007 INFO:tasks.workunit.client.1.smithi027.stdout:8/388: creat d1/d3/d1c/d43/d82/f84 x:0 0 0 2024-04-22T15:49:57.011 INFO:tasks.workunit.client.1.smithi027.stdout:1/382: creat d2/d5f/d64/d81/d69/f8a x:0 0 0 2024-04-22T15:49:57.014 INFO:tasks.workunit.client.1.smithi027.stdout:8/389: dwrite d1/d3/f1a [4194304,4194304] 0 2024-04-22T15:49:57.030 INFO:tasks.workunit.client.1.smithi027.stdout:9/409: truncate d0/d64/f7b 57404 0 2024-04-22T15:49:57.032 INFO:tasks.workunit.client.1.smithi027.stdout:1/383: rename d2/d1d/c5d to d2/d5f/c8b 0 2024-04-22T15:49:57.034 INFO:tasks.workunit.client.1.smithi027.stdout:0/478: write d0/f14 [3207214,35962] 0 2024-04-22T15:49:57.034 INFO:tasks.workunit.client.1.smithi027.stdout:7/380: dwrite d0/d16/d2f/f59 [0,4194304] 0 2024-04-22T15:49:57.041 INFO:tasks.workunit.client.1.smithi027.stdout:9/410: creat d0/dc/d12/d58/d6b/d6e/f84 x:0 0 0 2024-04-22T15:49:57.045 INFO:tasks.workunit.client.1.smithi027.stdout:8/390: truncate d1/d3/d1c/d32/f3d 2147587 0 2024-04-22T15:49:57.047 INFO:tasks.workunit.client.1.smithi027.stdout:0/479: symlink d0/d2a/d6a/l97 0 2024-04-22T15:49:57.051 INFO:tasks.workunit.client.1.smithi027.stdout:1/384: mknod d2/c8c 0 2024-04-22T15:49:57.052 INFO:tasks.workunit.client.1.smithi027.stdout:1/385: chown d2/d5f/d64/d81/d3d/d44/c45 97888319 1 2024-04-22T15:49:57.053 INFO:tasks.workunit.client.1.smithi027.stdout:1/386: fdatasync d2/f7f 0 2024-04-22T15:49:57.054 INFO:tasks.workunit.client.1.smithi027.stdout:0/480: truncate d0/d28/d63/d67/f82 466461 0 2024-04-22T15:49:57.054 INFO:tasks.workunit.client.1.smithi027.stdout:0/481: fdatasync d0/d34/f71 0 2024-04-22T15:49:57.055 INFO:tasks.workunit.client.1.smithi027.stdout:0/482: fsync d0/f2e 0 2024-04-22T15:49:57.057 INFO:tasks.workunit.client.1.smithi027.stdout:8/391: link d1/d3/d1c/d56/f6a d1/d3/d36/f85 0 2024-04-22T15:49:57.064 INFO:tasks.workunit.client.1.smithi027.stdout:7/381: getdents d0/d11/d1c 0 2024-04-22T15:49:57.067 INFO:tasks.workunit.client.1.smithi027.stdout:9/411: getdents d0/dc/d12/d58/d6b/d6e 0 2024-04-22T15:49:57.068 INFO:tasks.workunit.client.1.smithi027.stdout:9/412: chown d0/dc/d12/d58/d6b/d6e/c77 185245950 1 2024-04-22T15:49:57.071 INFO:tasks.workunit.client.1.smithi027.stdout:0/483: truncate d0/f1 2233700 0 2024-04-22T15:49:57.080 INFO:tasks.workunit.client.1.smithi027.stdout:0/484: mknod d0/d34/d59/c98 0 2024-04-22T15:49:57.081 INFO:tasks.workunit.client.1.smithi027.stdout:1/387: getdents d2/d5f/d64 0 2024-04-22T15:49:57.081 INFO:tasks.workunit.client.1.smithi027.stdout:7/382: link d0/d11/d1c/d31/f47 d0/d16/f79 0 2024-04-22T15:49:57.083 INFO:tasks.workunit.client.1.smithi027.stdout:7/383: read d0/d11/f4d [2096238,60019] 0 2024-04-22T15:49:57.086 INFO:tasks.workunit.client.1.smithi027.stdout:8/392: write d1/f7 [1672659,114134] 0 2024-04-22T15:49:57.087 INFO:tasks.workunit.client.1.smithi027.stdout:9/413: link d0/dc/d12/d58/d6b/d6e/c2c d0/dc/d12/d58/d6b/d6e/c85 0 2024-04-22T15:49:57.091 INFO:tasks.workunit.client.1.smithi027.stdout:8/393: creat d1/d3/d1c/d32/d41/f86 x:0 0 0 2024-04-22T15:49:57.093 INFO:tasks.workunit.client.1.smithi027.stdout:7/384: truncate d0/f37 1614223 0 2024-04-22T15:49:57.093 INFO:tasks.workunit.client.1.smithi027.stdout:7/385: fdatasync d0/f30 0 2024-04-22T15:49:57.095 INFO:tasks.workunit.client.1.smithi027.stdout:9/414: rename d0/d51/l74 to d0/dc/de/l86 0 2024-04-22T15:49:57.098 INFO:tasks.workunit.client.1.smithi027.stdout:7/386: creat d0/d4c/f7a x:0 0 0 2024-04-22T15:49:57.098 INFO:tasks.workunit.client.1.smithi027.stdout:1/388: getdents d2/d1d/d56 0 2024-04-22T15:49:57.100 INFO:tasks.workunit.client.1.smithi027.stdout:1/389: write d2/d5f/d64/d81/d3d/d44/f6d [458747,27654] 0 2024-04-22T15:49:57.100 INFO:tasks.workunit.client.1.smithi027.stdout:9/415: creat d0/d64/f87 x:0 0 0 2024-04-22T15:49:57.101 INFO:tasks.workunit.client.1.smithi027.stdout:8/394: mknod d1/d4f/c87 0 2024-04-22T15:49:57.101 INFO:tasks.workunit.client.1.smithi027.stdout:1/390: fdatasync d2/fa 0 2024-04-22T15:49:57.102 INFO:tasks.workunit.client.1.smithi027.stdout:9/416: chown d0/dc/c5a 306918 1 2024-04-22T15:49:57.104 INFO:tasks.workunit.client.1.smithi027.stdout:9/417: chown d0/dc/d12/d58/d6b/d6e/f84 174753247 1 2024-04-22T15:49:57.104 INFO:tasks.workunit.client.1.smithi027.stdout:8/395: mkdir d1/d3/d36/d88 0 2024-04-22T15:49:57.107 INFO:tasks.workunit.client.1.smithi027.stdout:9/418: mknod d0/dc/d12/d30/c88 0 2024-04-22T15:49:57.109 INFO:tasks.workunit.client.1.smithi027.stdout:8/396: creat d1/d3/d1c/d32/d41/f89 x:0 0 0 2024-04-22T15:49:57.113 INFO:tasks.workunit.client.1.smithi027.stdout:1/391: link d2/d30/l5c d2/d65/l8d 0 2024-04-22T15:49:57.119 INFO:tasks.workunit.client.1.smithi027.stdout:8/397: truncate d1/d3/d1c/d28/f51 1359741 0 2024-04-22T15:49:57.122 INFO:tasks.workunit.client.1.smithi027.stdout:1/392: rename d2/l58 to d2/d36/d74/l8e 0 2024-04-22T15:49:57.125 INFO:tasks.workunit.client.1.smithi027.stdout:7/387: dread d0/d16/f23 [4194304,4194304] 0 2024-04-22T15:49:57.126 INFO:tasks.workunit.client.1.smithi027.stdout:1/393: mknod d2/d5f/d64/c8f 0 2024-04-22T15:49:57.129 INFO:tasks.workunit.client.1.smithi027.stdout:9/419: rmdir d0 39 2024-04-22T15:49:57.132 INFO:tasks.workunit.client.1.smithi027.stdout:6/443: sync 2024-04-22T15:49:57.133 INFO:tasks.workunit.client.1.smithi027.stdout:1/394: dwrite d2/f27 [0,4194304] 0 2024-04-22T15:49:57.135 INFO:tasks.workunit.client.1.smithi027.stdout:8/398: link d1/f21 d1/d3/d1c/d32/f8a 0 2024-04-22T15:49:57.136 INFO:tasks.workunit.client.1.smithi027.stdout:8/399: fsync d1/d3/d1c/d49/f5d 0 2024-04-22T15:49:57.137 INFO:tasks.workunit.client.1.smithi027.stdout:8/400: chown d1/d4f 12173827 1 2024-04-22T15:49:57.138 INFO:tasks.workunit.client.1.smithi027.stdout:9/420: chown d0/dc/d12/d58/d6b/l63 80187220 1 2024-04-22T15:49:57.141 INFO:tasks.workunit.client.1.smithi027.stdout:1/395: mknod d2/d36/d74/c90 0 2024-04-22T15:49:57.146 INFO:tasks.workunit.client.1.smithi027.stdout:8/401: creat d1/d3/d1c/d43/d82/f8b x:0 0 0 2024-04-22T15:49:57.147 INFO:tasks.workunit.client.1.smithi027.stdout:7/388: write d0/d26/f36 [1802193,128435] 0 2024-04-22T15:49:57.148 INFO:tasks.workunit.client.1.smithi027.stdout:9/421: truncate d0/f36 803811 0 2024-04-22T15:49:57.149 INFO:tasks.workunit.client.1.smithi027.stdout:7/389: fdatasync d0/d4c/d4e/f67 0 2024-04-22T15:49:57.150 INFO:tasks.workunit.client.1.smithi027.stdout:7/390: stat d0/d3f/l64 0 2024-04-22T15:49:57.156 INFO:tasks.workunit.client.1.smithi027.stdout:9/422: dwrite d0/dc/de/f24 [0,4194304] 0 2024-04-22T15:49:57.160 INFO:tasks.workunit.client.1.smithi027.stdout:6/444: rename d5/d30/d54/d6a to d5/dc/d99 0 2024-04-22T15:49:57.163 INFO:tasks.workunit.client.1.smithi027.stdout:8/402: creat d1/d3/d1c/d43/d5f/d7d/f8c x:0 0 0 2024-04-22T15:49:57.166 INFO:tasks.workunit.client.1.smithi027.stdout:6/445: stat d5/dc/c28 0 2024-04-22T15:49:57.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:56 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:49:57.167 INFO:tasks.workunit.client.1.smithi027.stdout:1/396: dwrite d2/f68 [0,4194304] 0 2024-04-22T15:49:57.169 INFO:tasks.workunit.client.1.smithi027.stdout:7/391: rename d0/d11/d1c/d31/d56/f66 to d0/d3f/f7b 0 2024-04-22T15:49:57.173 INFO:tasks.workunit.client.1.smithi027.stdout:8/403: creat d1/d4f/f8d x:0 0 0 2024-04-22T15:49:57.175 INFO:tasks.workunit.client.1.smithi027.stdout:6/446: dread d5/dc/dd/d79/d7b/d5d/f70 [0,4194304] 0 2024-04-22T15:49:57.176 INFO:tasks.workunit.client.1.smithi027.stdout:9/423: creat d0/dc/d12/d58/f89 x:0 0 0 2024-04-22T15:49:57.179 INFO:tasks.workunit.client.1.smithi027.stdout:9/424: write d0/dc/d12/d30/f32 [4770009,9994] 0 2024-04-22T15:49:57.180 INFO:tasks.workunit.client.1.smithi027.stdout:7/392: dwrite d0/d16/d52/f77 [0,4194304] 0 2024-04-22T15:49:57.182 INFO:tasks.workunit.client.1.smithi027.stdout:7/393: chown d0/d11/d1c/d31/f47 1920 1 2024-04-22T15:49:57.183 INFO:tasks.workunit.client.1.smithi027.stdout:7/394: rename d0 to d0/d16/d7c 22 2024-04-22T15:49:57.188 INFO:tasks.workunit.client.1.smithi027.stdout:6/447: mkdir d5/dc/dd/d79/d7b/d9a 0 2024-04-22T15:49:57.190 INFO:tasks.workunit.client.1.smithi027.stdout:1/397: truncate d2/f27 1987217 0 2024-04-22T15:49:57.192 INFO:tasks.workunit.client.1.smithi027.stdout:7/395: symlink d0/d11/d1c/d31/d56/l7d 0 2024-04-22T15:49:57.200 INFO:tasks.workunit.client.1.smithi027.stdout:7/396: dwrite d0/d11/d1c/d31/d56/f58 [0,4194304] 0 2024-04-22T15:49:57.201 INFO:tasks.workunit.client.1.smithi027.stdout:7/397: stat d0/d11/d1c/d31/d62 0 2024-04-22T15:49:57.206 INFO:tasks.workunit.client.1.smithi027.stdout:9/425: link d0/f47 d0/dc/d12/d58/f8a 0 2024-04-22T15:49:57.207 INFO:tasks.workunit.client.1.smithi027.stdout:8/404: link d1/d3/c15 d1/d3/d36/c8e 0 2024-04-22T15:49:57.209 INFO:tasks.workunit.client.1.smithi027.stdout:8/405: write d1/d3/d1c/d43/d82/f8b [622236,79405] 0 2024-04-22T15:49:57.214 INFO:tasks.workunit.client.1.smithi027.stdout:8/406: rmdir d1/d3/d1c/d28/d42 39 2024-04-22T15:49:57.216 INFO:tasks.workunit.client.1.smithi027.stdout:6/448: dwrite d5/dc/d26/d3d/f83 [0,4194304] 0 2024-04-22T15:49:57.218 INFO:tasks.workunit.client.1.smithi027.stdout:7/398: link d0/d16/f3b d0/d11/d1c/f7e 0 2024-04-22T15:49:57.221 INFO:tasks.workunit.client.1.smithi027.stdout:8/407: unlink d1/d3/f6d 0 2024-04-22T15:49:57.223 INFO:tasks.workunit.client.1.smithi027.stdout:1/398: write d2/d5f/d64/d81/d3d/f4e [1455954,90700] 0 2024-04-22T15:49:57.224 INFO:tasks.workunit.client.1.smithi027.stdout:8/408: chown d1/f21 25477896 1 2024-04-22T15:49:57.225 INFO:tasks.workunit.client.1.smithi027.stdout:9/426: dread d0/dc/de/f40 [0,4194304] 0 2024-04-22T15:49:57.226 INFO:tasks.workunit.client.1.smithi027.stdout:8/409: stat d1/d3/l8 0 2024-04-22T15:49:57.227 INFO:tasks.workunit.client.1.smithi027.stdout:1/399: chown d2/d65/f67 57150 1 2024-04-22T15:49:57.229 INFO:tasks.workunit.client.1.smithi027.stdout:7/399: dwrite d0/d4c/f7a [0,4194304] 0 2024-04-22T15:49:57.230 INFO:tasks.workunit.client.1.smithi027.stdout:6/449: getdents d5/dc/dd/d79/d7b 0 2024-04-22T15:49:57.232 INFO:tasks.workunit.client.1.smithi027.stdout:9/427: write d0/dc/f25 [1002842,114653] 0 2024-04-22T15:49:57.232 INFO:tasks.workunit.client.1.smithi027.stdout:6/450: read d5/dc/d20/d2b/f12 [9918405,9557] 0 2024-04-22T15:49:57.236 INFO:tasks.workunit.client.1.smithi027.stdout:7/400: dread d0/d16/d52/f77 [0,4194304] 0 2024-04-22T15:49:57.237 INFO:tasks.workunit.client.1.smithi027.stdout:1/400: stat d2/d65/c88 0 2024-04-22T15:49:57.239 INFO:tasks.workunit.client.1.smithi027.stdout:1/401: readlink d2/d30/l51 0 2024-04-22T15:49:57.240 INFO:tasks.workunit.client.1.smithi027.stdout:7/401: read d0/d16/f25 [250783,80181] 0 2024-04-22T15:49:57.241 INFO:tasks.workunit.client.1.smithi027.stdout:7/402: chown d0/l32 84951 1 2024-04-22T15:49:57.243 INFO:tasks.workunit.client.1.smithi027.stdout:6/451: dwrite d5/dc/d17/d53/f72 [0,4194304] 0 2024-04-22T15:49:57.246 INFO:tasks.workunit.client.1.smithi027.stdout:9/428: fsync d0/f47 0 2024-04-22T15:49:57.252 INFO:tasks.workunit.client.1.smithi027.stdout:8/410: mkdir d1/d3/d1c/d32/d41/d52/d61/d8f 0 2024-04-22T15:49:57.254 INFO:tasks.workunit.client.1.smithi027.stdout:6/452: dwrite d5/dc/dd/d52/f8e [0,4194304] 0 2024-04-22T15:49:57.255 INFO:tasks.workunit.client.1.smithi027.stdout:9/429: dread d0/dc/d12/d58/f59 [0,4194304] 0 2024-04-22T15:49:57.257 INFO:tasks.workunit.client.1.smithi027.stdout:9/430: chown d0/dc/d12/d58/d6b/d6e/f84 39343 1 2024-04-22T15:49:57.260 INFO:tasks.workunit.client.1.smithi027.stdout:1/402: rename d2/d5f/d64/d81/d69/f8a to d2/d1d/f91 0 2024-04-22T15:49:57.264 INFO:tasks.workunit.client.1.smithi027.stdout:9/431: dwrite d0/dc/d12/d58/d6b/d6e/f61 [0,4194304] 0 2024-04-22T15:49:57.267 INFO:tasks.workunit.client.1.smithi027.stdout:7/403: mknod d0/d11/c7f 0 2024-04-22T15:49:57.267 INFO:tasks.workunit.client.1.smithi027.stdout:7/404: readlink d0/d3f/l41 0 2024-04-22T15:49:57.268 INFO:tasks.workunit.client.1.smithi027.stdout:7/405: fdatasync d0/d4c/f7a 0 2024-04-22T15:49:57.270 INFO:tasks.workunit.client.1.smithi027.stdout:6/453: mknod d5/db/d2e/c9b 0 2024-04-22T15:49:57.271 INFO:tasks.workunit.client.1.smithi027.stdout:7/406: fsync d0/d11/d3a/d46/f6c 0 2024-04-22T15:49:57.271 INFO:tasks.workunit.client.1.smithi027.stdout:8/411: creat d1/d3/d1c/d43/d5f/d7d/f90 x:0 0 0 2024-04-22T15:49:57.273 INFO:tasks.workunit.client.1.smithi027.stdout:7/407: chown d0/f39 15243977 1 2024-04-22T15:49:57.273 INFO:tasks.workunit.client.1.smithi027.stdout:8/412: write d1/d1e/f33 [1141711,40527] 0 2024-04-22T15:49:57.275 INFO:tasks.workunit.client.1.smithi027.stdout:7/408: write d0/d11/d3a/d46/f6c [856832,30462] 0 2024-04-22T15:49:57.275 INFO:tasks.workunit.client.1.smithi027.stdout:8/413: dread - d1/d3/d1c/d56/f7f zero size 2024-04-22T15:49:57.282 INFO:tasks.workunit.client.1.smithi027.stdout:8/414: read d1/d3/d1c/d32/f8a [2238647,116867] 0 2024-04-22T15:49:57.282 INFO:tasks.workunit.client.1.smithi027.stdout:8/415: chown d1/d3/d1c/d43/d5f/d7d/f8c 10 1 2024-04-22T15:49:57.283 INFO:tasks.workunit.client.1.smithi027.stdout:1/403: truncate d2/d5f/d64/d81/f3a 1017435 0 2024-04-22T15:49:57.283 INFO:tasks.workunit.client.1.smithi027.stdout:1/404: read - d2/d65/f66 zero size 2024-04-22T15:49:57.284 INFO:tasks.workunit.client.1.smithi027.stdout:9/432: mkdir d0/d8b 0 2024-04-22T15:49:57.284 INFO:tasks.workunit.client.1.smithi027.stdout:1/405: fsync d2/f35 0 2024-04-22T15:49:57.286 INFO:tasks.workunit.client.1.smithi027.stdout:1/406: write d2/d65/f67 [4854172,40995] 0 2024-04-22T15:49:57.291 INFO:tasks.workunit.client.1.smithi027.stdout:6/454: rename d5/db/l56 to d5/dc/d17/d8a/l9c 0 2024-04-22T15:49:57.294 INFO:tasks.workunit.client.1.smithi027.stdout:8/416: rmdir d1/d3/d1c/d49 39 2024-04-22T15:49:57.297 INFO:tasks.workunit.client.1.smithi027.stdout:9/433: symlink d0/dc/d12/d58/d6b/l8c 0 2024-04-22T15:49:57.299 INFO:tasks.workunit.client.1.smithi027.stdout:9/434: write d0/dc/d12/d58/d6b/d6e/f61 [5021131,56487] 0 2024-04-22T15:49:57.302 INFO:tasks.workunit.client.1.smithi027.stdout:1/407: rename d2/d5f/f71 to d2/d5f/d64/d81/d3d/d44/f92 0 2024-04-22T15:49:57.305 INFO:tasks.workunit.client.1.smithi027.stdout:7/409: unlink d0/d11/d1c/d31/c74 0 2024-04-22T15:49:57.308 INFO:tasks.workunit.client.1.smithi027.stdout:9/435: dwrite d0/dc/d12/d58/f59 [0,4194304] 0 2024-04-22T15:49:57.311 INFO:tasks.workunit.client.1.smithi027.stdout:8/417: dread d1/d3/d1c/d32/d41/d52/f60 [0,4194304] 0 2024-04-22T15:49:57.313 INFO:tasks.workunit.client.1.smithi027.stdout:1/408: stat d2/d65/l8d 0 2024-04-22T15:49:57.314 INFO:tasks.workunit.client.1.smithi027.stdout:6/455: dread d5/dc/d26/f2d [0,4194304] 0 2024-04-22T15:49:57.315 INFO:tasks.workunit.client.1.smithi027.stdout:7/410: mknod d0/d16/d2f/c80 0 2024-04-22T15:49:57.318 INFO:tasks.workunit.client.1.smithi027.stdout:7/411: stat d0/f39 0 2024-04-22T15:49:57.320 INFO:tasks.workunit.client.1.smithi027.stdout:9/436: dwrite d0/dc/d12/d58/d6b/d6e/f84 [0,4194304] 0 2024-04-22T15:49:57.325 INFO:tasks.workunit.client.1.smithi027.stdout:1/409: rename d2/d5f/d64/d81/d3d/f4e to d2/d5f/d64/d81/d3d/d82/f93 0 2024-04-22T15:49:57.326 INFO:tasks.workunit.client.1.smithi027.stdout:6/456: dwrite d5/d30/d54/f7f [0,4194304] 0 2024-04-22T15:49:57.327 INFO:tasks.workunit.client.1.smithi027.stdout:7/412: truncate d0/d3f/f5b 118908 0 2024-04-22T15:49:57.332 INFO:tasks.workunit.client.1.smithi027.stdout:9/437: creat d0/d51/f8d x:0 0 0 2024-04-22T15:49:57.334 INFO:tasks.workunit.client.1.smithi027.stdout:9/438: fsync d0/dc/d12/d30/f67 0 2024-04-22T15:49:57.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:56 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:49:57.336 INFO:tasks.workunit.client.1.smithi027.stdout:8/418: dwrite d1/d3/d1c/d32/f8a [4194304,4194304] 0 2024-04-22T15:49:57.339 INFO:tasks.workunit.client.1.smithi027.stdout:8/419: truncate d1/d3/d1c/d32/d41/f86 765619 0 2024-04-22T15:49:57.341 INFO:tasks.workunit.client.1.smithi027.stdout:1/410: rename d2/d36/d74/c7b to d2/d36/c94 0 2024-04-22T15:49:57.343 INFO:tasks.workunit.client.1.smithi027.stdout:9/439: creat d0/dc/f8e x:0 0 0 2024-04-22T15:49:57.348 INFO:tasks.workunit.client.1.smithi027.stdout:8/420: dwrite d1/d3/f59 [0,4194304] 0 2024-04-22T15:49:57.348 INFO:tasks.workunit.client.1.smithi027.stdout:1/411: creat d2/d5f/d64/d81/d3d/d82/d85/f95 x:0 0 0 2024-04-22T15:49:57.353 INFO:tasks.workunit.client.1.smithi027.stdout:1/412: creat d2/d1d/f96 x:0 0 0 2024-04-22T15:49:57.362 INFO:tasks.workunit.client.1.smithi027.stdout:1/413: link d2/d1d/l2d d2/d30/l97 0 2024-04-22T15:49:57.363 INFO:tasks.workunit.client.1.smithi027.stdout:1/414: chown d2/d5f/d64/l6e 26 1 2024-04-22T15:49:57.366 INFO:tasks.workunit.client.1.smithi027.stdout:1/415: rename d2/d1d/d56/c78 to d2/d1d/d56/c98 0 2024-04-22T15:49:57.369 INFO:tasks.workunit.client.1.smithi027.stdout:1/416: write d2/f35 [1660646,78484] 0 2024-04-22T15:49:57.369 INFO:tasks.workunit.client.1.smithi027.stdout:7/413: write d0/d11/fe [2026933,82295] 0 2024-04-22T15:49:57.371 INFO:tasks.workunit.client.1.smithi027.stdout:6/457: dread d5/db/f11 [0,4194304] 0 2024-04-22T15:49:57.372 INFO:tasks.workunit.client.1.smithi027.stdout:8/421: dread d1/d3/d1c/d49/f5d [4194304,4194304] 0 2024-04-22T15:49:57.373 INFO:tasks.workunit.client.1.smithi027.stdout:4/465: sync 2024-04-22T15:49:57.374 INFO:tasks.workunit.client.1.smithi027.stdout:1/417: fsync d2/d36/f7c 0 2024-04-22T15:49:57.375 INFO:tasks.workunit.client.1.smithi027.stdout:7/414: stat d0/d11/d1c/d2c/f38 0 2024-04-22T15:49:57.377 INFO:tasks.workunit.client.1.smithi027.stdout:9/440: write d0/dc/de/f40 [1523961,84049] 0 2024-04-22T15:49:57.379 INFO:tasks.workunit.client.1.smithi027.stdout:1/418: dread d2/f68 [0,4194304] 0 2024-04-22T15:49:57.380 INFO:tasks.workunit.client.1.smithi027.stdout:4/466: mknod d5/d7/cb6 0 2024-04-22T15:49:57.381 INFO:tasks.workunit.client.1.smithi027.stdout:4/467: stat d5/d45/d5a/f96 0 2024-04-22T15:49:57.381 INFO:tasks.workunit.client.1.smithi027.stdout:9/441: dread d0/dc/de/f3a [0,4194304] 0 2024-04-22T15:49:57.383 INFO:tasks.workunit.client.1.smithi027.stdout:7/415: creat d0/d16/f81 x:0 0 0 2024-04-22T15:49:57.387 INFO:tasks.workunit.client.1.smithi027.stdout:1/419: dread - d2/d5f/d64/d81/f50 zero size 2024-04-22T15:49:57.390 INFO:tasks.workunit.client.1.smithi027.stdout:9/442: chown d0/f36 121 1 2024-04-22T15:49:57.390 INFO:tasks.workunit.client.1.smithi027.stdout:8/422: dread d1/d3/d1c/d28/f51 [0,4194304] 0 2024-04-22T15:49:57.395 INFO:tasks.workunit.client.1.smithi027.stdout:6/458: rename d5/db/d2e/l18 to d5/d30/l9d 0 2024-04-22T15:49:57.396 INFO:tasks.workunit.client.1.smithi027.stdout:7/416: creat d0/d11/d3a/d46/f82 x:0 0 0 2024-04-22T15:49:57.398 INFO:tasks.workunit.client.1.smithi027.stdout:1/420: unlink d2/d5f/d64/c8f 0 2024-04-22T15:49:57.400 INFO:tasks.workunit.client.1.smithi027.stdout:4/468: rename d5/d11/d1d/d54/l57 to d5/d45/lb7 0 2024-04-22T15:49:57.401 INFO:tasks.workunit.client.1.smithi027.stdout:7/417: write d0/d26/f42 [1359111,16437] 0 2024-04-22T15:49:57.403 INFO:tasks.workunit.client.1.smithi027.stdout:7/418: truncate d0/d3f/f7b 984431 0 2024-04-22T15:49:57.406 INFO:tasks.workunit.client.1.smithi027.stdout:6/459: dread d5/dc/d20/d2b/d34/f65 [0,4194304] 0 2024-04-22T15:49:57.408 INFO:tasks.workunit.client.1.smithi027.stdout:7/419: dread d0/d11/d1c/d2c/f40 [0,4194304] 0 2024-04-22T15:49:57.409 INFO:tasks.workunit.client.1.smithi027.stdout:9/443: link d0/c6f d0/d51/c8f 0 2024-04-22T15:49:57.409 INFO:tasks.workunit.client.1.smithi027.stdout:1/421: fsync d2/d36/f4b 0 2024-04-22T15:49:57.412 INFO:tasks.workunit.client.1.smithi027.stdout:8/423: rename d1/d3/d1c/d28/c2c to d1/d3/c91 0 2024-04-22T15:49:57.413 INFO:tasks.workunit.client.1.smithi027.stdout:4/469: creat d5/da8/fb8 x:0 0 0 2024-04-22T15:49:57.415 INFO:tasks.workunit.client.1.smithi027.stdout:8/424: write d1/d3/d1c/d32/d41/d52/f60 [1609757,104461] 0 2024-04-22T15:49:57.415 INFO:tasks.workunit.client.1.smithi027.stdout:7/420: dwrite d0/d11/f5c [0,4194304] 0 2024-04-22T15:49:57.418 INFO:tasks.workunit.client.1.smithi027.stdout:7/421: stat d0/d11/c7f 0 2024-04-22T15:49:57.421 INFO:tasks.workunit.client.1.smithi027.stdout:4/470: chown d5/l31 11101539 1 2024-04-22T15:49:57.423 INFO:tasks.workunit.client.1.smithi027.stdout:2/462: sync 2024-04-22T15:49:57.423 INFO:tasks.workunit.client.1.smithi027.stdout:3/519: sync 2024-04-22T15:49:57.423 INFO:tasks.workunit.client.1.smithi027.stdout:5/477: sync 2024-04-22T15:49:57.423 INFO:tasks.workunit.client.1.smithi027.stdout:6/460: truncate d5/dc/d20/f3f 3088991 0 2024-04-22T15:49:57.423 INFO:tasks.workunit.client.1.smithi027.stdout:0/485: sync 2024-04-22T15:49:57.425 INFO:tasks.workunit.client.1.smithi027.stdout:0/486: stat d0/c69 0 2024-04-22T15:49:57.426 INFO:tasks.workunit.client.1.smithi027.stdout:9/444: dwrite d0/d8/f11 [4194304,4194304] 0 2024-04-22T15:49:57.427 INFO:tasks.workunit.client.1.smithi027.stdout:9/445: chown d0/dc/de 4992137 1 2024-04-22T15:49:57.438 INFO:tasks.workunit.client.1.smithi027.stdout:3/520: rmdir d28 39 2024-04-22T15:49:57.442 INFO:tasks.workunit.client.1.smithi027.stdout:4/471: dwrite d5/d45/d5a/f82 [0,4194304] 0 2024-04-22T15:49:57.446 INFO:tasks.workunit.client.1.smithi027.stdout:6/461: mknod d5/dc/dd/d79/d7b/c9e 0 2024-04-22T15:49:57.449 INFO:tasks.workunit.client.1.smithi027.stdout:6/462: dread - d5/dc/d20/d2b/f97 zero size 2024-04-22T15:49:57.449 INFO:tasks.workunit.client.1.smithi027.stdout:4/472: dwrite d5/d45/d47/f62 [0,4194304] 0 2024-04-22T15:49:57.450 INFO:tasks.workunit.client.1.smithi027.stdout:4/473: chown d5 12671 1 2024-04-22T15:49:57.451 INFO:tasks.workunit.client.1.smithi027.stdout:2/463: rename dd/df/f18 to dd/df/d1b/d38/f91 0 2024-04-22T15:49:57.451 INFO:tasks.workunit.client.1.smithi027.stdout:2/464: stat dd/df/d1b/d1f/f30 0 2024-04-22T15:49:57.456 INFO:tasks.workunit.client.1.smithi027.stdout:7/422: mknod d0/d16/d52/d72/c83 0 2024-04-22T15:49:57.460 INFO:tasks.workunit.client.1.smithi027.stdout:7/423: dread d0/d11/d1c/d2c/f38 [0,4194304] 0 2024-04-22T15:49:57.461 INFO:tasks.workunit.client.1.smithi027.stdout:7/424: readlink d0/d11/d1c/d31/l43 0 2024-04-22T15:49:57.461 INFO:tasks.workunit.client.1.smithi027.stdout:8/425: creat d1/f92 x:0 0 0 2024-04-22T15:49:57.462 INFO:tasks.workunit.client.1.smithi027.stdout:7/425: chown d0/d11/c7f 8386509 1 2024-04-22T15:49:57.465 INFO:tasks.workunit.client.1.smithi027.stdout:5/478: symlink d0/d97/l9e 0 2024-04-22T15:49:57.469 INFO:tasks.workunit.client.1.smithi027.stdout:7/426: dwrite d0/f39 [0,4194304] 0 2024-04-22T15:49:57.471 INFO:tasks.workunit.client.1.smithi027.stdout:7/427: write d0/fb [1244743,29442] 0 2024-04-22T15:49:57.471 INFO:tasks.workunit.client.1.smithi027.stdout:6/463: symlink d5/db/d2e/d59/l9f 0 2024-04-22T15:49:57.475 INFO:tasks.workunit.client.1.smithi027.stdout:5/479: dwrite d0/d1a/d22/d26/d32/d79/f87 [0,4194304] 0 2024-04-22T15:49:57.483 INFO:tasks.workunit.client.1.smithi027.stdout:1/422: getdents d2/d5f/d64 0 2024-04-22T15:49:57.484 INFO:tasks.workunit.client.1.smithi027.stdout:9/446: dwrite d0/dc/d12/d30/f39 [4194304,4194304] 0 2024-04-22T15:49:57.488 INFO:tasks.workunit.client.1.smithi027.stdout:3/521: fdatasync d28/d69/f82 0 2024-04-22T15:49:57.493 INFO:tasks.workunit.client.1.smithi027.stdout:8/426: rmdir d1/d3 39 2024-04-22T15:49:57.501 INFO:tasks.workunit.client.1.smithi027.stdout:7/428: creat d0/d49/f84 x:0 0 0 2024-04-22T15:49:57.502 INFO:tasks.workunit.client.1.smithi027.stdout:7/429: chown d0/d11/d1c/d31/f47 19769 1 2024-04-22T15:49:57.504 INFO:tasks.workunit.client.1.smithi027.stdout:4/474: write d5/de/d2a/f9a [538,24346] 0 2024-04-22T15:49:57.508 INFO:tasks.workunit.client.1.smithi027.stdout:2/465: symlink dd/df/d70/l92 0 2024-04-22T15:49:57.508 INFO:tasks.workunit.client.1.smithi027.stdout:2/466: dread - dd/df/d1b/f41 zero size 2024-04-22T15:49:57.508 INFO:tasks.workunit.client.1.smithi027.stdout:2/467: write dd/df/f49 [1027894,62563] 0 2024-04-22T15:49:57.514 INFO:tasks.workunit.client.1.smithi027.stdout:8/427: chown d1/d3/d1c/d32/d41/l81 26174 1 2024-04-22T15:49:57.515 INFO:tasks.workunit.client.1.smithi027.stdout:6/464: creat d5/dc/dd/d79/d7b/d9a/fa0 x:0 0 0 2024-04-22T15:49:57.516 INFO:tasks.workunit.client.1.smithi027.stdout:7/430: mknod d0/d16/d2f/c85 0 2024-04-22T15:49:57.519 INFO:tasks.workunit.client.1.smithi027.stdout:5/480: dwrite d0/df/de/d24/f35 [0,4194304] 0 2024-04-22T15:49:57.524 INFO:tasks.workunit.client.1.smithi027.stdout:2/468: creat dd/d47/f93 x:0 0 0 2024-04-22T15:49:57.525 INFO:tasks.workunit.client.1.smithi027.stdout:3/522: dwrite d28/d4b/d5c/f6f [0,4194304] 0 2024-04-22T15:49:57.527 INFO:tasks.workunit.client.1.smithi027.stdout:3/523: dread - d28/d69/f85 zero size 2024-04-22T15:49:57.528 INFO:tasks.workunit.client.1.smithi027.stdout:6/465: stat d5/dc/c8f 0 2024-04-22T15:49:57.530 INFO:tasks.workunit.client.1.smithi027.stdout:4/475: dwrite d5/f13 [0,4194304] 0 2024-04-22T15:49:57.531 INFO:tasks.workunit.client.1.smithi027.stdout:8/428: mknod d1/d1e/d47/d53/c93 0 2024-04-22T15:49:57.535 INFO:tasks.workunit.client.1.smithi027.stdout:7/431: mknod d0/d11/d3a/c86 0 2024-04-22T15:49:57.535 INFO:tasks.workunit.client.1.smithi027.stdout:1/423: link d2/d1d/l48 d2/d1d/d56/l99 0 2024-04-22T15:49:57.536 INFO:tasks.workunit.client.1.smithi027.stdout:2/469: mkdir dd/df/d1b/d94 0 2024-04-22T15:49:57.537 INFO:tasks.workunit.client.1.smithi027.stdout:4/476: symlink d5/da8/lb9 0 2024-04-22T15:49:57.538 INFO:tasks.workunit.client.1.smithi027.stdout:5/481: dread d0/d1a/d22/d26/d32/f54 [0,4194304] 0 2024-04-22T15:49:57.538 INFO:tasks.workunit.client.1.smithi027.stdout:8/429: mkdir d1/d3/d1c/d49/d94 0 2024-04-22T15:49:57.540 INFO:tasks.workunit.client.1.smithi027.stdout:9/447: getdents d0/dc/d12/d30/d5b 0 2024-04-22T15:49:57.540 INFO:tasks.workunit.client.1.smithi027.stdout:1/424: rmdir d2/d30 39 2024-04-22T15:49:57.541 INFO:tasks.workunit.client.1.smithi027.stdout:7/432: symlink d0/d11/d1c/d31/l87 0 2024-04-22T15:49:57.543 INFO:tasks.workunit.client.1.smithi027.stdout:4/477: symlink d5/d11/d6c/lba 0 2024-04-22T15:49:57.544 INFO:tasks.workunit.client.1.smithi027.stdout:2/470: read dd/df/d1b/d38/f91 [226474,50018] 0 2024-04-22T15:49:57.544 INFO:tasks.workunit.client.1.smithi027.stdout:1/425: fdatasync d2/d1d/f91 0 2024-04-22T15:49:57.547 INFO:tasks.workunit.client.1.smithi027.stdout:5/482: creat d0/d1a/d22/d26/d32/d88/d90/f9f x:0 0 0 2024-04-22T15:49:57.548 INFO:tasks.workunit.client.1.smithi027.stdout:4/478: write d5/d45/f94 [790498,75665] 0 2024-04-22T15:49:57.549 INFO:tasks.workunit.client.1.smithi027.stdout:8/430: rename d1/d3/d1c/d56/d62 to d1/d3/d95 0 2024-04-22T15:49:57.549 INFO:tasks.workunit.client.1.smithi027.stdout:7/433: fsync d0/d16/f3b 0 2024-04-22T15:49:57.551 INFO:tasks.workunit.client.1.smithi027.stdout:6/466: dwrite d5/dc/d26/d3d/f7c [0,4194304] 0 2024-04-22T15:49:57.552 INFO:tasks.workunit.client.1.smithi027.stdout:4/479: readlink d5/d11/d6c/l9d 0 2024-04-22T15:49:57.552 INFO:tasks.workunit.client.1.smithi027.stdout:7/434: stat d0/d3f/f7b 0 2024-04-22T15:49:57.553 INFO:tasks.workunit.client.1.smithi027.stdout:5/483: mknod d0/df/de/d24/d1c/d3e/ca0 0 2024-04-22T15:49:57.559 INFO:tasks.workunit.client.1.smithi027.stdout:8/431: read d1/d3/f46 [126421,16082] 0 2024-04-22T15:49:57.564 INFO:tasks.workunit.client.1.smithi027.stdout:8/432: read d1/d1e/f33 [833637,109474] 0 2024-04-22T15:49:57.564 INFO:tasks.workunit.client.1.smithi027.stdout:4/480: read d5/d11/f7a [3474665,106480] 0 2024-04-22T15:49:57.567 INFO:tasks.workunit.client.1.smithi027.stdout:2/471: dwrite dd/df/d70/d4b/f68 [0,4194304] 0 2024-04-22T15:49:57.569 INFO:tasks.workunit.client.1.smithi027.stdout:5/484: symlink d0/df/de/d40/d89/la1 0 2024-04-22T15:49:57.570 INFO:tasks.workunit.client.1.smithi027.stdout:9/448: dread d0/dc/d12/d30/f4f [0,4194304] 0 2024-04-22T15:49:57.571 INFO:tasks.workunit.client.1.smithi027.stdout:4/481: write d5/d7/f1f [198486,27682] 0 2024-04-22T15:49:57.575 INFO:tasks.workunit.client.1.smithi027.stdout:7/435: dwrite d0/d11/d1c/d2c/f4f [0,4194304] 0 2024-04-22T15:49:57.576 INFO:tasks.workunit.client.1.smithi027.stdout:1/426: dread d2/d1d/f32 [0,4194304] 0 2024-04-22T15:49:57.580 INFO:tasks.workunit.client.1.smithi027.stdout:6/467: getdents d5/dc/d99 0 2024-04-22T15:49:57.581 INFO:tasks.workunit.client.1.smithi027.stdout:2/472: fdatasync dd/df/d1b/f41 0 2024-04-22T15:49:57.587 INFO:tasks.workunit.client.1.smithi027.stdout:5/485: rename d0/df/d3d/d8c to d0/df/de/d24/d1c/d3e/da2 0 2024-04-22T15:49:57.590 INFO:tasks.workunit.client.1.smithi027.stdout:6/468: creat d5/dc/d17/d53/fa1 x:0 0 0 2024-04-22T15:49:57.592 INFO:tasks.workunit.client.1.smithi027.stdout:2/473: creat dd/d43/f95 x:0 0 0 2024-04-22T15:49:57.602 INFO:tasks.workunit.client.1.smithi027.stdout:7/436: link d0/f37 d0/d16/f88 0 2024-04-22T15:49:57.603 INFO:tasks.workunit.client.1.smithi027.stdout:5/486: unlink d0/df/de/d24/d1c/f34 0 2024-04-22T15:49:57.604 INFO:tasks.workunit.client.1.smithi027.stdout:2/474: dwrite dd/df/d70/d5a/f77 [0,4194304] 0 2024-04-22T15:49:57.607 INFO:tasks.workunit.client.1.smithi027.stdout:4/482: truncate d5/d11/d6c/f8b 3347854 0 2024-04-22T15:49:57.610 INFO:tasks.workunit.client.1.smithi027.stdout:9/449: creat d0/f90 x:0 0 0 2024-04-22T15:49:57.611 INFO:tasks.workunit.client.1.smithi027.stdout:4/483: readlink d5/l31 0 2024-04-22T15:49:57.611 INFO:tasks.workunit.client.1.smithi027.stdout:8/433: getdents d1/d4f 0 2024-04-22T15:49:57.611 INFO:tasks.workunit.client.1.smithi027.stdout:6/469: dread d5/f7a [0,4194304] 0 2024-04-22T15:49:57.614 INFO:tasks.workunit.client.1.smithi027.stdout:7/437: dread d0/d16/f23 [4194304,4194304] 0 2024-04-22T15:49:57.615 INFO:tasks.workunit.client.1.smithi027.stdout:9/450: truncate d0/d64/f7c 166853 0 2024-04-22T15:49:57.616 INFO:tasks.workunit.client.1.smithi027.stdout:5/487: dwrite d0/df/d10/f60 [0,4194304] 0 2024-04-22T15:49:57.622 INFO:tasks.workunit.client.1.smithi027.stdout:6/470: dwrite d5/dc/d17/d53/f91 [0,4194304] 0 2024-04-22T15:49:57.623 INFO:tasks.workunit.client.1.smithi027.stdout:7/438: dread d0/d26/f42 [0,4194304] 0 2024-04-22T15:49:57.625 INFO:tasks.workunit.client.1.smithi027.stdout:7/439: write d0/d26/f42 [1263279,98183] 0 2024-04-22T15:49:57.634 INFO:tasks.workunit.client.1.smithi027.stdout:2/475: mkdir dd/df/d1b/d96 0 2024-04-22T15:49:57.637 INFO:tasks.workunit.client.1.smithi027.stdout:6/471: dwrite d5/dc/dd/d79/d7b/d5d/f6c [0,4194304] 0 2024-04-22T15:49:57.639 INFO:tasks.workunit.client.1.smithi027.stdout:4/484: unlink d5/de/f43 0 2024-04-22T15:49:57.645 INFO:tasks.workunit.client.1.smithi027.stdout:5/488: chown d0/f8 0 1 2024-04-22T15:49:57.645 INFO:tasks.workunit.client.1.smithi027.stdout:9/451: mkdir d0/dc/d12/d30/d5b/d91 0 2024-04-22T15:49:57.647 INFO:tasks.workunit.client.1.smithi027.stdout:7/440: creat d0/d16/d52/f89 x:0 0 0 2024-04-22T15:49:57.648 INFO:tasks.workunit.client.1.smithi027.stdout:7/441: write d0/d49/f71 [969470,54703] 0 2024-04-22T15:49:57.649 INFO:tasks.workunit.client.1.smithi027.stdout:1/427: getdents d2/d36/d74 0 2024-04-22T15:49:57.653 INFO:tasks.workunit.client.1.smithi027.stdout:6/472: creat d5/dc/dd/fa2 x:0 0 0 2024-04-22T15:49:57.654 INFO:tasks.workunit.client.1.smithi027.stdout:4/485: unlink d5/d11/d89/f3c 0 2024-04-22T15:49:57.655 INFO:tasks.workunit.client.1.smithi027.stdout:4/486: write d5/f9 [1408209,83056] 0 2024-04-22T15:49:57.658 INFO:tasks.workunit.client.1.smithi027.stdout:8/434: mkdir d1/d3/d1c/d28/d96 0 2024-04-22T15:49:57.668 INFO:tasks.workunit.client.1.smithi027.stdout:1/428: dread d2/f3 [0,4194304] 0 2024-04-22T15:49:57.671 INFO:tasks.workunit.client.1.smithi027.stdout:7/442: dread d0/d11/d1c/d31/f47 [0,4194304] 0 2024-04-22T15:49:57.677 INFO:tasks.workunit.client.1.smithi027.stdout:1/429: dwrite d2/f1f [0,4194304] 0 2024-04-22T15:49:57.678 INFO:tasks.workunit.client.1.smithi027.stdout:2/476: dwrite dd/df/d1b/f25 [0,4194304] 0 2024-04-22T15:49:57.679 INFO:tasks.workunit.client.1.smithi027.stdout:2/477: fsync dd/f11 0 2024-04-22T15:49:57.681 INFO:tasks.workunit.client.1.smithi027.stdout:8/435: unlink d1/d4f/f8d 0 2024-04-22T15:49:57.686 INFO:tasks.workunit.client.1.smithi027.stdout:2/478: dread dd/df/d1b/f25 [0,4194304] 0 2024-04-22T15:49:57.702 INFO:tasks.workunit.client.1.smithi027.stdout:9/452: dwrite d0/d64/f7b [0,4194304] 0 2024-04-22T15:49:57.704 INFO:tasks.workunit.client.1.smithi027.stdout:6/473: mkdir d5/dc/d20/da3 0 2024-04-22T15:49:57.709 INFO:tasks.workunit.client.1.smithi027.stdout:0/487: sync 2024-04-22T15:49:57.709 INFO:tasks.workunit.client.1.smithi027.stdout:3/524: sync 2024-04-22T15:49:57.724 INFO:tasks.workunit.client.1.smithi027.stdout:1/430: dwrite d2/f70 [4194304,4194304] 0 2024-04-22T15:49:57.727 INFO:tasks.workunit.client.1.smithi027.stdout:2/479: rmdir dd/df/d70/d4b 39 2024-04-22T15:49:57.728 INFO:tasks.workunit.client.1.smithi027.stdout:5/489: link d0/df/de/d24/d1c/c50 d0/d1a/d22/d26/ca3 0 2024-04-22T15:49:57.731 INFO:tasks.workunit.client.1.smithi027.stdout:2/480: dread dd/f6f [0,4194304] 0 2024-04-22T15:49:57.735 INFO:tasks.workunit.client.1.smithi027.stdout:8/436: dread d1/d3/fa [0,4194304] 0 2024-04-22T15:49:57.736 INFO:tasks.workunit.client.1.smithi027.stdout:9/453: chown d0/d8/ld 737 1 2024-04-22T15:49:57.738 INFO:tasks.workunit.client.1.smithi027.stdout:6/474: symlink d5/dc/dd/d79/d7b/la4 0 2024-04-22T15:49:57.740 INFO:tasks.workunit.client.1.smithi027.stdout:6/475: truncate d5/dc/d20/d2b/f78 398055 0 2024-04-22T15:49:57.740 INFO:tasks.workunit.client.1.smithi027.stdout:3/525: stat d28/d4b/d52/c91 0 2024-04-22T15:49:57.741 INFO:tasks.workunit.client.1.smithi027.stdout:6/476: write d5/dc/d20/d2b/f97 [894311,24509] 0 2024-04-22T15:49:57.748 INFO:tasks.workunit.client.1.smithi027.stdout:1/431: rename d2/f72 to d2/d1d/d56/f9a 0 2024-04-22T15:49:57.752 INFO:tasks.workunit.client.1.smithi027.stdout:2/481: rmdir dd/df 39 2024-04-22T15:49:57.753 INFO:tasks.workunit.client.1.smithi027.stdout:7/443: rmdir d0/d26/d78 0 2024-04-22T15:49:57.755 INFO:tasks.workunit.client.1.smithi027.stdout:7/444: write d0/d26/f69 [888569,62088] 0 2024-04-22T15:49:57.756 INFO:tasks.workunit.client.1.smithi027.stdout:7/445: chown d0/d16/d52/f89 121 1 2024-04-22T15:49:57.756 INFO:tasks.workunit.client.1.smithi027.stdout:8/437: unlink d1/l17 0 2024-04-22T15:49:57.757 INFO:tasks.workunit.client.1.smithi027.stdout:9/454: rmdir d0/d64 39 2024-04-22T15:49:57.759 INFO:tasks.workunit.client.1.smithi027.stdout:4/487: getdents d5/d11/d89 0 2024-04-22T15:49:57.760 INFO:tasks.workunit.client.1.smithi027.stdout:0/488: mknod d0/d2a/d38/d89/c99 0 2024-04-22T15:49:57.764 INFO:tasks.workunit.client.1.smithi027.stdout:3/526: creat d28/d5d/d62/faa x:0 0 0 2024-04-22T15:49:57.766 INFO:tasks.workunit.client.1.smithi027.stdout:6/477: stat d5/dc/d17/d8a/l9c 0 2024-04-22T15:49:57.769 INFO:tasks.workunit.client.1.smithi027.stdout:1/432: chown d2/d5f/d64/d81/d3d/d44/c4a 3 1 2024-04-22T15:49:57.770 INFO:tasks.workunit.client.1.smithi027.stdout:9/455: dread d0/dc/d12/d58/f8a [0,4194304] 0 2024-04-22T15:49:57.772 INFO:tasks.workunit.client.1.smithi027.stdout:2/482: write dd/df/d1b/f26 [4439925,124021] 0 2024-04-22T15:49:57.776 INFO:tasks.workunit.client.1.smithi027.stdout:0/489: creat d0/d2a/d38/d70/f9a x:0 0 0 2024-04-22T15:49:57.778 INFO:tasks.workunit.client.1.smithi027.stdout:0/490: write d0/d34/d59/f62 [2445230,82345] 0 2024-04-22T15:49:57.778 INFO:tasks.workunit.client.1.smithi027.stdout:3/527: creat d28/d4b/d52/d9a/fab x:0 0 0 2024-04-22T15:49:57.781 INFO:tasks.workunit.client.1.smithi027.stdout:7/446: dwrite d0/d3f/f48 [4194304,4194304] 0 2024-04-22T15:49:57.781 INFO:tasks.workunit.client.1.smithi027.stdout:1/433: creat d2/d5f/d64/d81/d69/f9b x:0 0 0 2024-04-22T15:49:57.783 INFO:tasks.workunit.client.1.smithi027.stdout:9/456: creat d0/dc/d12/d58/d6b/d6e/f92 x:0 0 0 2024-04-22T15:49:57.786 INFO:tasks.workunit.client.1.smithi027.stdout:8/438: creat d1/d3/d36/d88/f97 x:0 0 0 2024-04-22T15:49:57.790 INFO:tasks.workunit.client.1.smithi027.stdout:6/478: mkdir d5/db/da5 0 2024-04-22T15:49:57.790 INFO:tasks.workunit.client.1.smithi027.stdout:0/491: dwrite d0/d34/f81 [0,4194304] 0 2024-04-22T15:49:57.791 INFO:tasks.workunit.client.1.smithi027.stdout:0/492: chown d0/d2a/d38/f4a 0 1 2024-04-22T15:49:57.792 INFO:tasks.workunit.client.1.smithi027.stdout:0/493: dread - d0/d28/d63/f7d zero size 2024-04-22T15:49:57.797 INFO:tasks.workunit.client.1.smithi027.stdout:2/483: symlink dd/df/d1b/d96/l97 0 2024-04-22T15:49:57.799 INFO:tasks.workunit.client.1.smithi027.stdout:2/484: dread - dd/df/d1b/d38/d60/f83 zero size 2024-04-22T15:49:57.800 INFO:tasks.workunit.client.1.smithi027.stdout:2/485: write dd/df/d70/f6b [223130,47596] 0 2024-04-22T15:49:57.801 INFO:tasks.workunit.client.1.smithi027.stdout:2/486: dread - dd/df/d1b/f62 zero size 2024-04-22T15:49:57.808 INFO:tasks.workunit.client.1.smithi027.stdout:8/439: mknod d1/d3/d1c/d43/d5f/c98 0 2024-04-22T15:49:57.809 INFO:tasks.workunit.client.1.smithi027.stdout:3/528: symlink d28/d5d/lac 0 2024-04-22T15:49:57.809 INFO:tasks.workunit.client.1.smithi027.stdout:8/440: read d1/f18 [2722188,70012] 0 2024-04-22T15:49:57.814 INFO:tasks.workunit.client.1.smithi027.stdout:1/434: fdatasync d2/d36/f55 0 2024-04-22T15:49:57.819 INFO:tasks.workunit.client.1.smithi027.stdout:4/488: link d5/f26 d5/de/fbb 0 2024-04-22T15:49:57.821 INFO:tasks.workunit.client.1.smithi027.stdout:2/487: truncate dd/df/d70/d4b/f68 4501515 0 2024-04-22T15:49:57.823 INFO:tasks.workunit.client.1.smithi027.stdout:9/457: write d0/dc/d12/f23 [2471884,19871] 0 2024-04-22T15:49:57.826 INFO:tasks.workunit.client.1.smithi027.stdout:8/441: creat d1/d3/f99 x:0 0 0 2024-04-22T15:49:57.827 INFO:tasks.workunit.client.1.smithi027.stdout:7/447: fsync d0/d3f/f7b 0 2024-04-22T15:49:57.828 INFO:tasks.workunit.client.1.smithi027.stdout:8/442: chown d1/d3/d1c/d56/f7f 208 1 2024-04-22T15:49:57.831 INFO:tasks.workunit.client.1.smithi027.stdout:0/494: creat d0/d2a/d58/d66/f9b x:0 0 0 2024-04-22T15:49:57.832 INFO:tasks.workunit.client.1.smithi027.stdout:0/495: chown d0/d34/d4e 7378954 1 2024-04-22T15:49:57.840 INFO:tasks.workunit.client.1.smithi027.stdout:0/496: dwrite d0/d2a/d58/d66/f9b [0,4194304] 0 2024-04-22T15:49:57.840 INFO:tasks.workunit.client.1.smithi027.stdout:0/497: readlink d0/d28/d37/l46 0 2024-04-22T15:49:57.843 INFO:tasks.workunit.client.1.smithi027.stdout:7/448: mkdir d0/d11/d1c/d31/d8a 0 2024-04-22T15:49:57.844 INFO:tasks.workunit.client.1.smithi027.stdout:7/449: dread - d0/d11/d3a/f45 zero size 2024-04-22T15:49:57.844 INFO:tasks.workunit.client.1.smithi027.stdout:6/479: creat d5/dc/fa6 x:0 0 0 2024-04-22T15:49:57.845 INFO:tasks.workunit.client.1.smithi027.stdout:1/435: truncate d2/d5f/d64/d81/d3d/d44/f73 1551634 0 2024-04-22T15:49:57.847 INFO:tasks.workunit.client.1.smithi027.stdout:7/450: fdatasync d0/d11/d1c/d31/f70 0 2024-04-22T15:49:57.849 INFO:tasks.workunit.client.1.smithi027.stdout:9/458: stat d0/d64/l81 0 2024-04-22T15:49:57.850 INFO:tasks.workunit.client.1.smithi027.stdout:0/498: dwrite d0/f14 [0,4194304] 0 2024-04-22T15:49:57.855 INFO:tasks.workunit.client.1.smithi027.stdout:1/436: write d2/d5f/d64/d81/d69/f9b [109925,101037] 0 2024-04-22T15:49:57.856 INFO:tasks.workunit.client.1.smithi027.stdout:9/459: fdatasync d0/dc/d12/d30/f67 0 2024-04-22T15:49:57.857 INFO:tasks.workunit.client.1.smithi027.stdout:3/529: creat d28/d4b/fad x:0 0 0 2024-04-22T15:49:57.857 INFO:tasks.workunit.client.1.smithi027.stdout:8/443: symlink d1/l9a 0 2024-04-22T15:49:57.858 INFO:tasks.workunit.client.1.smithi027.stdout:0/499: chown d0/d2a/d38/f6b 1 1 2024-04-22T15:49:57.858 INFO:tasks.workunit.client.1.smithi027.stdout:4/489: dwrite d5/d11/d1d/f5e [0,4194304] 0 2024-04-22T15:49:57.860 INFO:tasks.workunit.client.1.smithi027.stdout:2/488: rmdir dd/df/d1b/d94 0 2024-04-22T15:49:57.864 INFO:tasks.workunit.client.1.smithi027.stdout:0/500: read - d0/d28/d63/f8d zero size 2024-04-22T15:49:57.864 INFO:tasks.workunit.client.1.smithi027.stdout:2/489: readlink dd/df/d1b/d38/l50 0 2024-04-22T15:49:57.864 INFO:tasks.workunit.client.1.smithi027.stdout:7/451: write d0/d11/d1c/d31/f47 [793426,116580] 0 2024-04-22T15:49:57.866 INFO:tasks.workunit.client.1.smithi027.stdout:9/460: chown d0/dc/d12/d30/c35 52 1 2024-04-22T15:49:57.868 INFO:tasks.workunit.client.1.smithi027.stdout:4/490: write d5/de/d30/d5d/f9f [108164,52014] 0 2024-04-22T15:49:57.869 INFO:tasks.workunit.client.1.smithi027.stdout:1/437: mkdir d2/d5f/d64/d81/d3d/d44/d9c 0 2024-04-22T15:49:57.873 INFO:tasks.workunit.client.1.smithi027.stdout:0/501: dread d0/d34/d59/f62 [4194304,4194304] 0 2024-04-22T15:49:57.874 INFO:tasks.workunit.client.1.smithi027.stdout:4/491: write d5/d11/d89/f84 [1411884,50716] 0 2024-04-22T15:49:57.875 INFO:tasks.workunit.client.1.smithi027.stdout:6/480: symlink d5/d30/d40/la7 0 2024-04-22T15:49:57.878 INFO:tasks.workunit.client.1.smithi027.stdout:2/490: fdatasync dd/d43/f55 0 2024-04-22T15:49:57.879 INFO:tasks.workunit.client.1.smithi027.stdout:2/491: dread - dd/df/d70/d4b/d4e/f56 zero size 2024-04-22T15:49:57.886 INFO:tasks.workunit.client.1.smithi027.stdout:8/444: dread d1/f18 [0,4194304] 0 2024-04-22T15:49:57.888 INFO:tasks.workunit.client.1.smithi027.stdout:7/452: rename d0/d16/d52 to d0/d4c/d8b 0 2024-04-22T15:49:57.888 INFO:tasks.workunit.client.1.smithi027.stdout:2/492: dwrite dd/df/d70/d4b/d4e/f56 [0,4194304] 0 2024-04-22T15:49:57.892 INFO:tasks.workunit.client.1.smithi027.stdout:1/438: rmdir d2/d65 39 2024-04-22T15:49:57.897 INFO:tasks.workunit.client.1.smithi027.stdout:0/502: rmdir d0/d34/d59 39 2024-04-22T15:49:57.901 INFO:tasks.workunit.client.1.smithi027.stdout:3/530: write d28/d5d/f8b [1799971,126611] 0 2024-04-22T15:49:57.901 INFO:tasks.workunit.client.1.smithi027.stdout:5/490: sync 2024-04-22T15:49:57.902 INFO:tasks.workunit.client.1.smithi027.stdout:6/481: mknod d5/dc/d99/ca8 0 2024-04-22T15:49:57.904 INFO:tasks.workunit.client.1.smithi027.stdout:3/531: write d28/f3c [8138369,89493] 0 2024-04-22T15:49:57.911 INFO:tasks.workunit.client.1.smithi027.stdout:3/532: dwrite f11 [4194304,4194304] 0 2024-04-22T15:49:57.912 INFO:tasks.workunit.client.1.smithi027.stdout:9/461: rename d0/dc/d12/d58/d6b/d6e/c82 to d0/dc/d12/d30/c93 0 2024-04-22T15:49:57.920 INFO:tasks.workunit.client.1.smithi027.stdout:1/439: unlink d2/c9 0 2024-04-22T15:49:57.927 INFO:tasks.workunit.client.1.smithi027.stdout:7/453: write d0/f37 [1755051,58521] 0 2024-04-22T15:49:57.929 INFO:tasks.workunit.client.1.smithi027.stdout:7/454: read d0/f13 [1114896,34512] 0 2024-04-22T15:49:57.929 INFO:tasks.workunit.client.1.smithi027.stdout:5/491: mknod d0/d1a/d22/d26/d5d/ca4 0 2024-04-22T15:49:57.931 INFO:tasks.workunit.client.1.smithi027.stdout:5/492: write d0/df/d10/f14 [3422331,115209] 0 2024-04-22T15:49:57.935 INFO:tasks.workunit.client.1.smithi027.stdout:6/482: fsync d5/dc/d20/d2b/d34/f65 0 2024-04-22T15:49:57.938 INFO:tasks.workunit.client.1.smithi027.stdout:4/492: write d5/f48 [246866,97383] 0 2024-04-22T15:49:57.943 INFO:tasks.workunit.client.1.smithi027.stdout:4/493: dread d5/d45/f94 [0,4194304] 0 2024-04-22T15:49:57.949 INFO:tasks.workunit.client.1.smithi027.stdout:8/445: unlink d1/d3/d36/c8e 0 2024-04-22T15:49:57.950 INFO:tasks.workunit.client.1.smithi027.stdout:8/446: read - d1/d1e/d47/d53/f77 zero size 2024-04-22T15:49:57.950 INFO:tasks.workunit.client.1.smithi027.stdout:3/533: mkdir d28/d4b/d52/d94/dae 0 2024-04-22T15:49:57.951 INFO:tasks.workunit.client.1.smithi027.stdout:8/447: readlink d1/d3/d1c/d56/l67 0 2024-04-22T15:49:57.953 INFO:tasks.workunit.client.1.smithi027.stdout:2/493: rename dd/df/d1b/d38/f91 to dd/df/d1b/d96/f98 0 2024-04-22T15:49:57.953 INFO:tasks.workunit.client.1.smithi027.stdout:2/494: chown dd/df/d70/f61 11101495 1 2024-04-22T15:49:57.954 INFO:tasks.workunit.client.1.smithi027.stdout:9/462: creat d0/dc/d12/f94 x:0 0 0 2024-04-22T15:49:57.962 INFO:tasks.workunit.client.1.smithi027.stdout:9/463: dwrite d0/dc/d12/f23 [0,4194304] 0 2024-04-22T15:49:57.965 INFO:tasks.workunit.client.1.smithi027.stdout:5/493: unlink d0/df/de/d24/d1c/d3e/f9b 0 2024-04-22T15:49:57.978 INFO:tasks.workunit.client.1.smithi027.stdout:8/448: creat d1/d3/f9b x:0 0 0 2024-04-22T15:49:57.985 INFO:tasks.workunit.client.1.smithi027.stdout:1/440: rename d2/d5f/d64/d81/d69/f9b to d2/d65/f9d 0 2024-04-22T15:49:57.989 INFO:tasks.workunit.client.1.smithi027.stdout:0/503: rename d0/d34/f42 to d0/d34/f9c 0 2024-04-22T15:49:57.990 INFO:tasks.workunit.client.1.smithi027.stdout:0/504: chown d0/d28/d63/d67/l6d 9800 1 2024-04-22T15:49:57.991 INFO:tasks.workunit.client.1.smithi027.stdout:0/505: chown d0/d2a/l4f 0 1 2024-04-22T15:49:57.994 INFO:tasks.workunit.client.1.smithi027.stdout:3/534: write d28/f3f [823574,106114] 0 2024-04-22T15:49:58.001 INFO:tasks.workunit.client.1.smithi027.stdout:9/464: symlink d0/dc/d12/d30/l95 0 2024-04-22T15:49:58.007 INFO:tasks.workunit.client.1.smithi027.stdout:4/494: symlink d5/de/da9/lbc 0 2024-04-22T15:49:58.011 INFO:tasks.workunit.client.1.smithi027.stdout:2/495: mknod dd/df/d1b/d1f/c99 0 2024-04-22T15:49:58.012 INFO:tasks.workunit.client.1.smithi027.stdout:4/495: dread d5/d11/d1d/f5e [0,4194304] 0 2024-04-22T15:49:58.016 INFO:tasks.workunit.client.1.smithi027.stdout:2/496: dread dd/f35 [0,4194304] 0 2024-04-22T15:49:58.017 INFO:tasks.workunit.client.1.smithi027.stdout:5/494: rename d0/d1a/d22/d26/d5d/c6c to d0/d1a/d22/d72/ca5 0 2024-04-22T15:49:58.019 INFO:tasks.workunit.client.1.smithi027.stdout:1/441: creat d2/d65/f9e x:0 0 0 2024-04-22T15:49:58.019 INFO:tasks.workunit.client.1.smithi027.stdout:5/495: chown d0/df/de/d24/d1c/d3e 1 1 2024-04-22T15:49:58.022 INFO:tasks.workunit.client.1.smithi027.stdout:9/465: fsync d0/f47 0 2024-04-22T15:49:58.024 INFO:tasks.workunit.client.1.smithi027.stdout:6/483: creat d5/db/fa9 x:0 0 0 2024-04-22T15:49:58.027 INFO:tasks.workunit.client.1.smithi027.stdout:8/449: mknod d1/d3/d1c/d56/c9c 0 2024-04-22T15:49:58.031 INFO:tasks.workunit.client.1.smithi027.stdout:4/496: chown d5/de/d2a/c56 415457 1 2024-04-22T15:49:58.031 INFO:tasks.workunit.client.1.smithi027.stdout:9/466: dwrite d0/dc/d12/d30/f7a [0,4194304] 0 2024-04-22T15:49:58.033 INFO:tasks.workunit.client.1.smithi027.stdout:4/497: write d5/d7/f1f [1121478,12001] 0 2024-04-22T15:49:58.033 INFO:tasks.workunit.client.1.smithi027.stdout:2/497: write dd/df/d1b/d96/f98 [1538408,127470] 0 2024-04-22T15:49:58.034 INFO:tasks.workunit.client.1.smithi027.stdout:2/498: readlink dd/df/d1b/d38/l50 0 2024-04-22T15:49:58.037 INFO:tasks.workunit.client.1.smithi027.stdout:4/498: write d5/d7/f8f [403496,124703] 0 2024-04-22T15:49:58.037 INFO:tasks.workunit.client.1.smithi027.stdout:2/499: write dd/df/d70/f6e [4617644,81569] 0 2024-04-22T15:49:58.038 INFO:tasks.workunit.client.1.smithi027.stdout:0/506: rename d0/d2a/d38/d70 to d0/d9d 0 2024-04-22T15:49:58.042 INFO:tasks.workunit.client.1.smithi027.stdout:9/467: dread d0/dc/d12/d30/f4f [0,4194304] 0 2024-04-22T15:49:58.045 INFO:tasks.workunit.client.1.smithi027.stdout:1/442: unlink d2/f1f 0 2024-04-22T15:49:58.047 INFO:tasks.workunit.client.1.smithi027.stdout:7/455: getdents d0/d11/d1c/d31 0 2024-04-22T15:49:58.052 INFO:tasks.workunit.client.1.smithi027.stdout:8/450: mkdir d1/d1e/d47/d9d 0 2024-04-22T15:49:58.053 INFO:tasks.workunit.client.1.smithi027.stdout:5/496: dwrite d0/f3c [0,4194304] 0 2024-04-22T15:49:58.054 INFO:tasks.workunit.client.1.smithi027.stdout:8/451: readlink d1/d3/d1c/d32/d41/l50 0 2024-04-22T15:49:58.055 INFO:tasks.workunit.client.1.smithi027.stdout:5/497: write d0/df/d10/f64 [998298,125101] 0 2024-04-22T15:49:58.066 INFO:tasks.workunit.client.1.smithi027.stdout:4/499: rename d5/d11/l4b to d5/d45/db3/lbd 0 2024-04-22T15:49:58.068 INFO:tasks.workunit.client.1.smithi027.stdout:0/507: dread d0/d28/d63/d67/f82 [0,4194304] 0 2024-04-22T15:49:58.069 INFO:tasks.workunit.client.1.smithi027.stdout:2/500: mkdir dd/df/d70/d4b/d52/d9a 0 2024-04-22T15:49:58.078 INFO:tasks.workunit.client.1.smithi027.stdout:0/508: dwrite d0/d2a/d38/f6b [0,4194304] 0 2024-04-22T15:49:58.081 INFO:tasks.workunit.client.1.smithi027.stdout:8/452: dread d1/d3/d1c/d32/d41/d52/f60 [0,4194304] 0 2024-04-22T15:49:58.082 INFO:tasks.workunit.client.1.smithi027.stdout:6/484: symlink d5/laa 0 2024-04-22T15:49:58.083 INFO:tasks.workunit.client.1.smithi027.stdout:8/453: chown d1/d3/d1c/d32 1847 1 2024-04-22T15:49:58.083 INFO:tasks.workunit.client.1.smithi027.stdout:0/509: chown d0/d9d/f9a 6912 1 2024-04-22T15:49:58.085 INFO:tasks.workunit.client.1.smithi027.stdout:0/510: fdatasync d0/d2a/d38/f55 0 2024-04-22T15:49:58.085 INFO:tasks.workunit.client.1.smithi027.stdout:5/498: mknod d0/d1a/ca6 0 2024-04-22T15:49:58.086 INFO:tasks.workunit.client.1.smithi027.stdout:0/511: chown d0/d34/f80 130 1 2024-04-22T15:49:58.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: pgmap v41: 65 pgs: 65 active+clean; 2.8 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 62 MiB/s rd, 211 MiB/s wr, 440 op/s 2024-04-22T15:49:58.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:58.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:58.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:49:58.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:49:58.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:58.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:49:58.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mgr fail", "who": "smithi027.ukdgzs"}]: dispatch 2024-04-22T15:49:58.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mgr fail", "who": "smithi027.ukdgzs"}]: dispatch 2024-04-22T15:49:58.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: osdmap e48: 6 total, 6 up, 6 in 2024-04-22T15:49:58.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' cmd='[{"prefix": "mgr fail", "who": "smithi027.ukdgzs"}]': finished 2024-04-22T15:49:58.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: mgrmap e27: smithi016.rnsnnr(active, starting, since 0.00564429s) 2024-04-22T15:49:58.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-04-22T15:49:58.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:49:58.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.pgvphu"}]: dispatch 2024-04-22T15:49:58.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.fkkejm"}]: dispatch 2024-04-22T15:49:58.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.iobjds"}]: dispatch 2024-04-22T15:49:58.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.dgqyes"}]: dispatch 2024-04-22T15:49:58.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr metadata", "who": "smithi016.rnsnnr", "id": "smithi016.rnsnnr"}]: dispatch 2024-04-22T15:49:58.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:49:58.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:49:58.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:49:58.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:49:58.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:49:58.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:49:58.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T15:49:58.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T15:49:58.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:57 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T15:49:58.094 INFO:tasks.workunit.client.1.smithi027.stdout:4/500: creat d5/d45/d47/fbe x:0 0 0 2024-04-22T15:49:58.096 INFO:tasks.workunit.client.1.smithi027.stdout:9/468: link d0/dc/d12/d30/f32 d0/d51/f96 0 2024-04-22T15:49:58.098 INFO:tasks.workunit.client.1.smithi027.stdout:7/456: dwrite d0/d16/f88 [4194304,4194304] 0 2024-04-22T15:49:58.098 INFO:tasks.workunit.client.1.smithi027.stdout:1/443: mkdir d2/d5f/d64/d81/d3d/d44/d9c/d9f 0 2024-04-22T15:49:58.100 INFO:tasks.workunit.client.1.smithi027.stdout:6/485: rmdir d5/dc/d17 39 2024-04-22T15:49:58.102 INFO:tasks.workunit.client.1.smithi027.stdout:6/486: chown d5/dc/c8f 24970434 1 2024-04-22T15:49:58.102 INFO:tasks.workunit.client.1.smithi027.stdout:1/444: write d2/d1d/f91 [505488,22590] 0 2024-04-22T15:49:58.104 INFO:tasks.workunit.client.1.smithi027.stdout:7/457: chown d0/d11/d1c/d2c/c51 12380 1 2024-04-22T15:49:58.112 INFO:tasks.workunit.client.1.smithi027.stdout:0/512: mkdir d0/d2a/d6a/d6c/d83/d9e 0 2024-04-22T15:49:58.112 INFO:tasks.workunit.client.1.smithi027.stdout:7/458: dwrite d0/d49/f71 [0,4194304] 0 2024-04-22T15:49:58.118 INFO:tasks.workunit.client.1.smithi027.stdout:4/501: write d5/d11/d1d/d7b/f85 [106638,89577] 0 2024-04-22T15:49:58.119 INFO:tasks.workunit.client.1.smithi027.stdout:7/459: dread d0/d11/f5c [0,4194304] 0 2024-04-22T15:49:58.121 INFO:tasks.workunit.client.1.smithi027.stdout:9/469: symlink d0/dc/d12/l97 0 2024-04-22T15:49:58.121 INFO:tasks.workunit.client.1.smithi027.stdout:7/460: fsync d0/d4c/d4e/f57 0 2024-04-22T15:49:58.122 INFO:tasks.workunit.client.1.smithi027.stdout:9/470: fsync d0/f44 0 2024-04-22T15:49:58.130 INFO:tasks.workunit.client.1.smithi027.stdout:0/513: truncate d0/d34/d59/f5c 769426 0 2024-04-22T15:49:58.133 INFO:tasks.workunit.client.1.smithi027.stdout:4/502: dwrite d5/d42/f9e [0,4194304] 0 2024-04-22T15:49:58.143 INFO:tasks.workunit.client.1.smithi027.stdout:8/454: dread d1/d3/f13 [0,4194304] 0 2024-04-22T15:49:58.144 INFO:tasks.workunit.client.1.smithi027.stdout:8/455: write d1/d3/f9b [488833,13033] 0 2024-04-22T15:49:58.147 INFO:tasks.workunit.client.1.smithi027.stdout:6/487: write d5/db/f11 [1963945,99287] 0 2024-04-22T15:49:58.160 INFO:tasks.workunit.client.1.smithi027.stdout:4/503: unlink d5/de/d30/l91 0 2024-04-22T15:49:58.163 INFO:tasks.workunit.client.1.smithi027.stdout:7/461: symlink d0/d16/d65/l8c 0 2024-04-22T15:49:58.164 INFO:tasks.workunit.client.1.smithi027.stdout:9/471: mknod d0/d8b/c98 0 2024-04-22T15:49:58.165 INFO:tasks.workunit.client.1.smithi027.stdout:7/462: chown d0/d16/f88 1445 1 2024-04-22T15:49:58.166 INFO:tasks.workunit.client.1.smithi027.stdout:9/472: chown d0/dc/d12/d58/d6b/d6e/l3c 2 1 2024-04-22T15:49:58.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: pgmap v41: 65 pgs: 65 active+clean; 2.8 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 62 MiB/s rd, 211 MiB/s wr, 440 op/s 2024-04-22T15:49:58.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:58.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:58.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:49:58.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:49:58.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' 2024-04-22T15:49:58.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:49:58.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: from='mgr.24395 172.21.15.27:0/1703022158' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mgr fail", "who": "smithi027.ukdgzs"}]: dispatch 2024-04-22T15:49:58.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "mgr fail", "who": "smithi027.ukdgzs"}]: dispatch 2024-04-22T15:49:58.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: osdmap e48: 6 total, 6 up, 6 in 2024-04-22T15:49:58.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: from='mgr.24395 ' entity='mgr.smithi027.ukdgzs' cmd='[{"prefix": "mgr fail", "who": "smithi027.ukdgzs"}]': finished 2024-04-22T15:49:58.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: mgrmap e27: smithi016.rnsnnr(active, starting, since 0.00564429s) 2024-04-22T15:49:58.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-04-22T15:49:58.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:49:58.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.pgvphu"}]: dispatch 2024-04-22T15:49:58.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.fkkejm"}]: dispatch 2024-04-22T15:49:58.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.iobjds"}]: dispatch 2024-04-22T15:49:58.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.dgqyes"}]: dispatch 2024-04-22T15:49:58.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr metadata", "who": "smithi016.rnsnnr", "id": "smithi016.rnsnnr"}]: dispatch 2024-04-22T15:49:58.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:49:58.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:49:58.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:49:58.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:49:58.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:49:58.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:49:58.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T15:49:58.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T15:49:58.169 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:57 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T15:49:58.169 INFO:tasks.workunit.client.1.smithi027.stdout:6/488: creat d5/dc/d99/fab x:0 0 0 2024-04-22T15:49:58.169 INFO:tasks.workunit.client.1.smithi027.stdout:6/489: write d5/d30/f47 [3049536,18514] 0 2024-04-22T15:49:58.173 INFO:tasks.workunit.client.1.smithi027.stdout:5/499: getdents d0/df 0 2024-04-22T15:49:58.174 INFO:tasks.workunit.client.1.smithi027.stdout:0/514: creat d0/d2a/d5e/d8b/f9f x:0 0 0 2024-04-22T15:49:58.177 INFO:tasks.workunit.client.1.smithi027.stdout:0/515: write d0/d34/f71 [121312,103109] 0 2024-04-22T15:49:58.178 INFO:tasks.workunit.client.1.smithi027.stdout:9/473: chown d0/d8/l4e 3 1 2024-04-22T15:49:58.182 INFO:tasks.workunit.client.1.smithi027.stdout:6/490: dwrite d5/dc/dd/d79/d7b/d5d/f6c [0,4194304] 0 2024-04-22T15:49:58.182 INFO:tasks.workunit.client.1.smithi027.stdout:8/456: dread d1/d3/f10 [0,4194304] 0 2024-04-22T15:49:58.191 INFO:tasks.workunit.client.1.smithi027.stdout:4/504: link d5/d45/d5a/f96 d5/de/d2a/d58/fbf 0 2024-04-22T15:49:58.191 INFO:tasks.workunit.client.1.smithi027.stdout:6/491: dwrite d5/dc/dd/fa2 [0,4194304] 0 2024-04-22T15:49:58.195 INFO:tasks.workunit.client.1.smithi027.stdout:7/463: creat d0/d11/d1c/d31/d62/f8d x:0 0 0 2024-04-22T15:49:58.196 INFO:tasks.workunit.client.1.smithi027.stdout:0/516: chown d0/f1 32906 1 2024-04-22T15:49:58.197 INFO:tasks.workunit.client.1.smithi027.stdout:4/505: dread d5/de/d2a/d58/f67 [0,4194304] 0 2024-04-22T15:49:58.202 INFO:tasks.workunit.client.1.smithi027.stdout:5/500: rename d0/d1a/d22/d26/d5d/f9a to d0/d1a/d22/d26/d32/d79/fa7 0 2024-04-22T15:49:58.203 INFO:tasks.workunit.client.1.smithi027.stdout:9/474: mkdir d0/dc/d12/d30/d99 0 2024-04-22T15:49:58.203 INFO:tasks.workunit.client.1.smithi027.stdout:9/475: read - d0/dc/f8e zero size 2024-04-22T15:49:58.207 INFO:tasks.workunit.client.1.smithi027.stdout:1/445: link d2/d30/l5c d2/d5f/d64/d81/d3d/d44/d9c/d9f/la0 0 2024-04-22T15:49:58.210 INFO:tasks.workunit.client.1.smithi027.stdout:8/457: creat d1/d3/d1c/d43/d5f/f9e x:0 0 0 2024-04-22T15:49:58.211 INFO:tasks.workunit.client.1.smithi027.stdout:5/501: dwrite d0/f3c [0,4194304] 0 2024-04-22T15:49:58.214 INFO:tasks.workunit.client.1.smithi027.stdout:3/535: sync 2024-04-22T15:49:58.219 INFO:tasks.workunit.client.1.smithi027.stdout:7/464: mknod d0/d11/d1c/d31/d62/c8e 0 2024-04-22T15:49:58.219 INFO:tasks.workunit.client.1.smithi027.stdout:0/517: truncate d0/d2a/f48 1791708 0 2024-04-22T15:49:58.223 INFO:tasks.workunit.client.1.smithi027.stdout:6/492: dread d5/d30/d54/f7f [0,4194304] 0 2024-04-22T15:49:58.226 INFO:tasks.workunit.client.1.smithi027.stdout:7/465: dread d0/d11/d1c/d31/f47 [0,4194304] 0 2024-04-22T15:49:58.230 INFO:tasks.workunit.client.1.smithi027.stdout:3/536: rmdir d28/d4b/d52/d94 39 2024-04-22T15:49:58.241 INFO:tasks.workunit.client.1.smithi027.stdout:3/537: dwrite d28/d69/f85 [0,4194304] 0 2024-04-22T15:49:58.247 INFO:tasks.workunit.client.1.smithi027.stdout:8/458: dread d1/d1e/f4c [0,4194304] 0 2024-04-22T15:49:58.248 INFO:tasks.workunit.client.1.smithi027.stdout:4/506: dread d5/de/fbb [0,4194304] 0 2024-04-22T15:49:58.252 INFO:tasks.workunit.client.1.smithi027.stdout:5/502: dread d0/d1a/d22/d26/f2a [0,4194304] 0 2024-04-22T15:49:58.256 INFO:tasks.workunit.client.1.smithi027.stdout:7/466: symlink d0/d4c/d4e/l8f 0 2024-04-22T15:49:58.256 INFO:tasks.workunit.client.1.smithi027.stdout:1/446: creat d2/d5f/d64/d81/d7e/fa1 x:0 0 0 2024-04-22T15:49:58.259 INFO:tasks.workunit.client.1.smithi027.stdout:1/447: chown d2/c8c 12005 1 2024-04-22T15:49:58.262 INFO:tasks.workunit.client.1.smithi027.stdout:7/467: chown d0/d4c/d8b/l6e 1 1 2024-04-22T15:49:58.262 INFO:tasks.workunit.client.1.smithi027.stdout:3/538: dread d28/d4b/f53 [0,4194304] 0 2024-04-22T15:49:58.262 INFO:tasks.workunit.client.1.smithi027.stdout:1/448: write d2/f75 [1235602,68248] 0 2024-04-22T15:49:58.263 INFO:tasks.workunit.client.1.smithi027.stdout:5/503: dwrite d0/df/d10/f14 [0,4194304] 0 2024-04-22T15:49:58.267 INFO:tasks.workunit.client.1.smithi027.stdout:4/507: creat d5/de/d2a/d58/fc0 x:0 0 0 2024-04-22T15:49:58.269 INFO:tasks.workunit.client.1.smithi027.stdout:4/508: truncate d5/de/d2a/d58/fc0 353285 0 2024-04-22T15:49:58.275 INFO:tasks.workunit.client.1.smithi027.stdout:2/501: sync 2024-04-22T15:49:58.280 INFO:tasks.workunit.client.1.smithi027.stdout:9/476: getdents d0/dc/d12/d30/d5b 0 2024-04-22T15:49:58.281 INFO:tasks.workunit.client.1.smithi027.stdout:3/539: stat d28/d4b/d52/l99 0 2024-04-22T15:49:58.281 INFO:tasks.workunit.client.1.smithi027.stdout:6/493: symlink d5/dc/d17/d53/d8c/lac 0 2024-04-22T15:49:58.283 INFO:tasks.workunit.client.1.smithi027.stdout:1/449: mknod d2/d1d/ca2 0 2024-04-22T15:49:58.286 INFO:tasks.workunit.client.1.smithi027.stdout:3/540: dwrite d28/f45 [0,4194304] 0 2024-04-22T15:49:58.288 INFO:tasks.workunit.client.1.smithi027.stdout:8/459: mknod d1/d3/d1c/d32/d41/d52/d61/d8f/c9f 0 2024-04-22T15:49:58.289 INFO:tasks.workunit.client.1.smithi027.stdout:4/509: mkdir d5/d11/dc1 0 2024-04-22T15:49:58.291 INFO:tasks.workunit.client.1.smithi027.stdout:7/468: dwrite d0/d11/d1c/d31/f47 [0,4194304] 0 2024-04-22T15:49:58.291 INFO:tasks.workunit.client.1.smithi027.stdout:4/510: readlink d5/d11/d6c/lba 0 2024-04-22T15:49:58.296 INFO:tasks.workunit.client.1.smithi027.stdout:9/477: mknod d0/d8b/c9a 0 2024-04-22T15:49:58.297 INFO:tasks.workunit.client.1.smithi027.stdout:0/518: getdents d0/d28 0 2024-04-22T15:49:58.300 INFO:tasks.workunit.client.1.smithi027.stdout:6/494: creat d5/d30/d40/fad x:0 0 0 2024-04-22T15:49:58.301 INFO:tasks.workunit.client.1.smithi027.stdout:3/541: dread d28/f38 [0,4194304] 0 2024-04-22T15:49:58.301 INFO:tasks.workunit.client.1.smithi027.stdout:6/495: chown d5/dc/d99 13853 1 2024-04-22T15:49:58.302 INFO:tasks.workunit.client.1.smithi027.stdout:3/542: chown d28/d5d/d79 974 1 2024-04-22T15:49:58.302 INFO:tasks.workunit.client.1.smithi027.stdout:7/469: rmdir d0/d49 39 2024-04-22T15:49:58.304 INFO:tasks.workunit.client.1.smithi027.stdout:2/502: symlink dd/df/d70/d4b/d84/d8d/l9b 0 2024-04-22T15:49:58.305 INFO:tasks.workunit.client.1.smithi027.stdout:6/496: write d5/dc/d26/d3d/f83 [2944583,127032] 0 2024-04-22T15:49:58.307 INFO:tasks.workunit.client.1.smithi027.stdout:4/511: symlink d5/d45/db3/lc2 0 2024-04-22T15:49:58.308 INFO:tasks.workunit.client.1.smithi027.stdout:9/478: dwrite d0/d64/f7b [0,4194304] 0 2024-04-22T15:49:58.310 INFO:tasks.workunit.client.1.smithi027.stdout:9/479: write d0/dc/d12/d58/f6a [1036710,42823] 0 2024-04-22T15:49:58.312 INFO:tasks.workunit.client.1.smithi027.stdout:9/480: chown d0/c52 133330106 1 2024-04-22T15:49:58.314 INFO:tasks.workunit.client.1.smithi027.stdout:5/504: rename d0/df/d3d/f44 to d0/df/de/d24/fa8 0 2024-04-22T15:49:58.314 INFO:tasks.workunit.client.1.smithi027.stdout:9/481: write d0/dc/d12/d58/d6b/d6e/f31 [3505671,70297] 0 2024-04-22T15:49:58.317 INFO:tasks.workunit.client.1.smithi027.stdout:0/519: creat d0/d2a/d6a/fa0 x:0 0 0 2024-04-22T15:49:58.318 INFO:tasks.workunit.client.1.smithi027.stdout:6/497: dwrite d5/dc/dd/d52/f7e [0,4194304] 0 2024-04-22T15:49:58.321 INFO:tasks.workunit.client.1.smithi027.stdout:8/460: creat d1/d3/d1c/d28/d96/fa0 x:0 0 0 2024-04-22T15:49:58.322 INFO:tasks.workunit.client.1.smithi027.stdout:7/470: mknod d0/d26/c90 0 2024-04-22T15:49:58.326 INFO:tasks.workunit.client.1.smithi027.stdout:2/503: unlink dd/l71 0 2024-04-22T15:49:58.330 INFO:tasks.workunit.client.1.smithi027.stdout:4/512: creat d5/de/d2a/fc3 x:0 0 0 2024-04-22T15:49:58.341 INFO:tasks.workunit.client.1.smithi027.stdout:6/498: readlink d5/dc/dd/d79/d7b/d5d/l6b 0 2024-04-22T15:49:58.342 INFO:tasks.workunit.client.1.smithi027.stdout:5/505: dwrite d0/df/de/d24/fa8 [0,4194304] 0 2024-04-22T15:49:58.344 INFO:tasks.workunit.client.1.smithi027.stdout:6/499: write d5/db/d2e/f62 [4817216,29827] 0 2024-04-22T15:49:58.345 INFO:tasks.workunit.client.1.smithi027.stdout:5/506: write d0/df/de/d24/d1c/d3e/d55/f9c [946430,66038] 0 2024-04-22T15:49:58.346 INFO:tasks.workunit.client.1.smithi027.stdout:9/482: mknod d0/dc/d12/d58/d6b/d6e/c9b 0 2024-04-22T15:49:58.348 INFO:tasks.workunit.client.1.smithi027.stdout:6/500: write d5/dc/d20/d2b/f78 [1373645,117250] 0 2024-04-22T15:49:58.351 INFO:tasks.workunit.client.1.smithi027.stdout:8/461: creat d1/d3/d1c/d32/d41/d52/fa1 x:0 0 0 2024-04-22T15:49:58.353 INFO:tasks.workunit.client.1.smithi027.stdout:7/471: truncate d0/d11/d1c/d2c/f38 1262779 0 2024-04-22T15:49:58.358 INFO:tasks.workunit.client.1.smithi027.stdout:6/501: dwrite d5/dc/d17/d53/f81 [0,4194304] 0 2024-04-22T15:49:58.359 INFO:tasks.workunit.client.1.smithi027.stdout:4/513: mknod d5/d45/d47/cc4 0 2024-04-22T15:49:58.366 INFO:tasks.workunit.client.1.smithi027.stdout:3/543: creat d28/d4b/faf x:0 0 0 2024-04-22T15:49:58.368 INFO:tasks.workunit.client.1.smithi027.stdout:5/507: write d0/f6 [5418719,1385] 0 2024-04-22T15:49:58.370 INFO:tasks.workunit.client.1.smithi027.stdout:9/483: creat d0/d64/f9c x:0 0 0 2024-04-22T15:49:58.372 INFO:tasks.workunit.client.1.smithi027.stdout:3/544: dread d28/f3b [0,4194304] 0 2024-04-22T15:49:58.373 INFO:tasks.workunit.client.1.smithi027.stdout:8/462: creat d1/d3/d1c/d32/d41/d52/d61/fa2 x:0 0 0 2024-04-22T15:49:58.378 INFO:tasks.workunit.client.1.smithi027.stdout:4/514: mkdir d5/de/dc5 0 2024-04-22T15:49:58.378 INFO:tasks.workunit.client.1.smithi027.stdout:4/515: chown d5/d11/f18 8 1 2024-04-22T15:49:58.381 INFO:tasks.workunit.client.1.smithi027.stdout:0/520: write d0/f1 [1114668,94857] 0 2024-04-22T15:49:58.382 INFO:tasks.workunit.client.1.smithi027.stdout:9/484: mkdir d0/d51/d9d 0 2024-04-22T15:49:58.386 INFO:tasks.workunit.client.1.smithi027.stdout:7/472: link d0/d16/f81 d0/d11/d1c/d31/d62/f91 0 2024-04-22T15:49:58.395 INFO:tasks.workunit.client.1.smithi027.stdout:9/485: dwrite d0/dc/d12/d58/d6b/d6e/f61 [0,4194304] 0 2024-04-22T15:49:58.395 INFO:tasks.workunit.client.1.smithi027.stdout:0/521: dwrite d0/d2a/f35 [0,4194304] 0 2024-04-22T15:49:58.397 INFO:tasks.workunit.client.1.smithi027.stdout:6/502: mkdir d5/dc/dae 0 2024-04-22T15:49:58.402 INFO:tasks.workunit.client.1.smithi027.stdout:7/473: dwrite d0/f30 [0,4194304] 0 2024-04-22T15:49:58.411 INFO:tasks.workunit.client.1.smithi027.stdout:9/486: write d0/dc/de/f6c [110130,32150] 0 2024-04-22T15:49:58.411 INFO:tasks.workunit.client.1.smithi027.stdout:3/545: mkdir d28/d5d/db0 0 2024-04-22T15:49:58.411 INFO:tasks.workunit.client.1.smithi027.stdout:9/487: fdatasync d0/dc/f3d 0 2024-04-22T15:49:58.414 INFO:tasks.workunit.client.1.smithi027.stdout:0/522: dwrite d0/f1 [0,4194304] 0 2024-04-22T15:49:58.418 INFO:tasks.workunit.client.1.smithi027.stdout:2/504: dwrite dd/df/d1b/d38/f6c [0,4194304] 0 2024-04-22T15:49:58.418 INFO:tasks.workunit.client.1.smithi027.stdout:5/508: write d0/df/d3d/f8e [1212992,42233] 0 2024-04-22T15:49:58.418 INFO:tasks.workunit.client.1.smithi027.stdout:4/516: rmdir d5 39 2024-04-22T15:49:58.420 INFO:tasks.workunit.client.1.smithi027.stdout:2/505: dread - dd/df/d1b/f62 zero size 2024-04-22T15:49:58.423 INFO:tasks.workunit.client.1.smithi027.stdout:5/509: fsync d0/df/de/d24/d62/f7f 0 2024-04-22T15:49:58.423 INFO:tasks.workunit.client.1.smithi027.stdout:2/506: write dd/f8e [726207,127396] 0 2024-04-22T15:49:58.426 INFO:tasks.workunit.client.1.smithi027.stdout:5/510: chown d0/df/de/f38 2 1 2024-04-22T15:49:58.428 INFO:tasks.workunit.client.1.smithi027.stdout:7/474: mkdir d0/d4c/d8b/d92 0 2024-04-22T15:49:58.429 INFO:tasks.workunit.client.1.smithi027.stdout:6/503: creat d5/dc/dd/d79/d7b/d9a/faf x:0 0 0 2024-04-22T15:49:58.432 INFO:tasks.workunit.client.1.smithi027.stdout:3/546: creat d28/d5d/d62/fb1 x:0 0 0 2024-04-22T15:49:58.434 INFO:tasks.workunit.client.1.smithi027.stdout:9/488: symlink d0/dc/d12/d58/d6b/l9e 0 2024-04-22T15:49:58.437 INFO:tasks.workunit.client.1.smithi027.stdout:4/517: fsync d5/d42/f61 0 2024-04-22T15:49:58.438 INFO:tasks.workunit.client.1.smithi027.stdout:5/511: symlink d0/d1a/d22/d72/la9 0 2024-04-22T15:49:58.439 INFO:tasks.workunit.client.1.smithi027.stdout:2/507: symlink dd/df/d70/d4b/d84/l9c 0 2024-04-22T15:49:58.439 INFO:tasks.workunit.client.1.smithi027.stdout:5/512: chown d0/d1a/d22 5869 1 2024-04-22T15:49:58.444 INFO:tasks.workunit.client.1.smithi027.stdout:7/475: rename d0/d16/d2f to d0/d11/d1c/d2c/d93 0 2024-04-22T15:49:58.446 INFO:tasks.workunit.client.1.smithi027.stdout:9/489: dwrite d0/dc/d12/d58/d6b/d6e/f31 [0,4194304] 0 2024-04-22T15:49:58.446 INFO:tasks.workunit.client.1.smithi027.stdout:5/513: truncate d0/df/de/d24/d1c/d8f/f94 1193061 0 2024-04-22T15:49:58.448 INFO:tasks.workunit.client.1.smithi027.stdout:8/463: getdents d1/d1e/d47/d53/d76 0 2024-04-22T15:49:58.449 INFO:tasks.workunit.client.1.smithi027.stdout:5/514: chown d0/df/de/d24/d62/f7f 27 1 2024-04-22T15:49:58.449 INFO:tasks.workunit.client.1.smithi027.stdout:0/523: dread d0/fc [0,4194304] 0 2024-04-22T15:49:58.451 INFO:tasks.workunit.client.1.smithi027.stdout:0/524: read d0/d28/d63/d67/f82 [359496,71904] 0 2024-04-22T15:49:58.460 INFO:tasks.workunit.client.1.smithi027.stdout:2/508: creat dd/df/d1b/d96/f9d x:0 0 0 2024-04-22T15:49:58.465 INFO:tasks.workunit.client.1.smithi027.stdout:2/509: dread dd/df/d1b/d38/d60/f83 [0,4194304] 0 2024-04-22T15:49:58.466 INFO:tasks.workunit.client.1.smithi027.stdout:5/515: creat d0/df/de/d24/d1c/d3e/da2/faa x:0 0 0 2024-04-22T15:49:58.474 INFO:tasks.workunit.client.1.smithi027.stdout:9/490: creat d0/dc/d12/d30/d5b/d91/f9f x:0 0 0 2024-04-22T15:49:58.476 INFO:tasks.workunit.client.1.smithi027.stdout:2/510: rmdir dd/df/d1b/d38/d60 39 2024-04-22T15:49:58.485 INFO:tasks.workunit.client.1.smithi027.stdout:3/547: rename d28/d4b/d5c to d28/d4b/d52/db2 0 2024-04-22T15:49:58.488 INFO:tasks.workunit.client.1.smithi027.stdout:0/525: dwrite d0/f22 [0,4194304] 0 2024-04-22T15:49:58.492 INFO:tasks.workunit.client.1.smithi027.stdout:2/511: symlink dd/df/d70/d4b/l9e 0 2024-04-22T15:49:58.494 INFO:tasks.workunit.client.1.smithi027.stdout:2/512: write dd/df/f23 [764985,114719] 0 2024-04-22T15:49:58.496 INFO:tasks.workunit.client.1.smithi027.stdout:4/518: getdents d5/de 0 2024-04-22T15:49:58.498 INFO:tasks.workunit.client.1.smithi027.stdout:5/516: dread d0/d1a/d22/d26/f48 [0,4194304] 0 2024-04-22T15:49:58.499 INFO:tasks.workunit.client.1.smithi027.stdout:9/491: symlink d0/dc/d12/d58/la0 0 2024-04-22T15:49:58.502 INFO:tasks.workunit.client.1.smithi027.stdout:7/476: rename d0/d11/d1c/d54 to d0/d11/d3a/d46/d94 0 2024-04-22T15:49:58.503 INFO:tasks.workunit.client.1.smithi027.stdout:9/492: chown d0/dc/d12/d30/c4a 1483364225 1 2024-04-22T15:49:58.505 INFO:tasks.workunit.client.1.smithi027.stdout:0/526: mkdir d0/d2a/d91/da1 0 2024-04-22T15:49:58.508 INFO:tasks.workunit.client.1.smithi027.stdout:2/513: truncate dd/df/d1b/d1f/f36 880005 0 2024-04-22T15:49:58.511 INFO:tasks.workunit.client.1.smithi027.stdout:3/548: rename d28/d4b/c4f to d28/d69/cb3 0 2024-04-22T15:49:58.512 INFO:tasks.workunit.client.1.smithi027.stdout:4/519: dwrite d5/d11/d6c/f95 [4194304,4194304] 0 2024-04-22T15:49:58.512 INFO:tasks.workunit.client.1.smithi027.stdout:5/517: dwrite d0/d1a/f82 [0,4194304] 0 2024-04-22T15:49:58.517 INFO:tasks.workunit.client.1.smithi027.stdout:2/514: dread dd/df/d70/f6b [0,4194304] 0 2024-04-22T15:49:58.518 INFO:tasks.workunit.client.1.smithi027.stdout:7/477: mknod d0/d11/d3a/d46/d94/c95 0 2024-04-22T15:49:58.518 INFO:tasks.workunit.client.1.smithi027.stdout:9/493: truncate d0/f68 925636 0 2024-04-22T15:49:58.518 INFO:tasks.workunit.client.1.smithi027.stdout:5/518: truncate d0/df/d10/f64 1935546 0 2024-04-22T15:49:58.520 INFO:tasks.workunit.client.1.smithi027.stdout:9/494: truncate d0/d64/f7c 528199 0 2024-04-22T15:49:58.521 INFO:tasks.workunit.client.1.smithi027.stdout:9/495: chown d0/d51 316723 1 2024-04-22T15:49:58.521 INFO:tasks.workunit.client.1.smithi027.stdout:7/478: write d0/d11/d1c/d31/d62/f8d [687760,19257] 0 2024-04-22T15:49:58.523 INFO:tasks.workunit.client.1.smithi027.stdout:9/496: fdatasync d0/d64/f7c 0 2024-04-22T15:49:58.524 INFO:tasks.workunit.client.1.smithi027.stdout:4/520: mkdir d5/d7/dc6 0 2024-04-22T15:49:58.525 INFO:tasks.workunit.client.1.smithi027.stdout:3/549: rename d28/d5d/l5e to d28/d69/lb4 0 2024-04-22T15:49:58.533 INFO:tasks.workunit.client.1.smithi027.stdout:5/519: symlink d0/d1a/d22/d26/d32/lab 0 2024-04-22T15:49:58.534 INFO:tasks.workunit.client.1.smithi027.stdout:7/479: unlink d0/d4c/d8b/f77 0 2024-04-22T15:49:58.536 INFO:tasks.workunit.client.1.smithi027.stdout:9/497: unlink d0/d8/ld 0 2024-04-22T15:49:58.537 INFO:tasks.workunit.client.1.smithi027.stdout:7/480: stat d0/d11/d1c/d31/d56/f58 0 2024-04-22T15:49:58.537 INFO:tasks.workunit.client.1.smithi027.stdout:4/521: dwrite d5/d11/f80 [0,4194304] 0 2024-04-22T15:49:58.539 INFO:tasks.workunit.client.1.smithi027.stdout:3/550: dwrite d28/f48 [0,4194304] 0 2024-04-22T15:49:58.540 INFO:tasks.workunit.client.1.smithi027.stdout:9/498: readlink d0/dc/d12/l19 0 2024-04-22T15:49:58.542 INFO:tasks.workunit.client.1.smithi027.stdout:9/499: fsync d0/dc/d12/d58/f6a 0 2024-04-22T15:49:58.542 INFO:tasks.workunit.client.1.smithi027.stdout:7/481: mkdir d0/d11/d1c/d2c/d96 0 2024-04-22T15:49:58.543 INFO:tasks.workunit.client.1.smithi027.stdout:5/520: mkdir d0/d1a/d22/d72/dac 0 2024-04-22T15:49:58.544 INFO:tasks.workunit.client.1.smithi027.stdout:9/500: fdatasync d0/dc/d12/d30/f67 0 2024-04-22T15:49:58.545 INFO:tasks.workunit.client.1.smithi027.stdout:4/522: creat d5/de/da9/fc7 x:0 0 0 2024-04-22T15:49:58.546 INFO:tasks.workunit.client.1.smithi027.stdout:3/551: read - d28/f6c zero size 2024-04-22T15:49:58.549 INFO:tasks.workunit.client.1.smithi027.stdout:7/482: write d0/d11/f5c [3610347,38470] 0 2024-04-22T15:49:58.551 INFO:tasks.workunit.client.1.smithi027.stdout:5/521: fdatasync d0/d1a/d22/d26/f2a 0 2024-04-22T15:49:58.553 INFO:tasks.workunit.client.1.smithi027.stdout:9/501: symlink d0/dc/d12/d58/d6b/d6e/la1 0 2024-04-22T15:49:58.558 INFO:tasks.workunit.client.1.smithi027.stdout:4/523: mknod d5/d11/da6/cc8 0 2024-04-22T15:49:58.560 INFO:tasks.workunit.client.1.smithi027.stdout:7/483: dwrite d0/d4c/d8b/f89 [0,4194304] 0 2024-04-22T15:49:58.562 INFO:tasks.workunit.client.1.smithi027.stdout:4/524: truncate d5/de/da9/fc7 197113 0 2024-04-22T15:49:58.566 INFO:tasks.workunit.client.1.smithi027.stdout:0/527: truncate d0/d2a/d38/f65 7889517 0 2024-04-22T15:49:58.575 INFO:tasks.workunit.client.1.smithi027.stdout:1/450: sync 2024-04-22T15:49:58.575 INFO:tasks.workunit.client.1.smithi027.stdout:1/451: readlink d2/l2b 0 2024-04-22T15:49:58.577 INFO:tasks.workunit.client.1.smithi027.stdout:5/522: mknod d0/d1a/d22/d72/dac/cad 0 2024-04-22T15:49:58.578 INFO:tasks.workunit.client.1.smithi027.stdout:0/528: rmdir d0/d2a/d5e/d8b 39 2024-04-22T15:49:58.580 INFO:tasks.workunit.client.1.smithi027.stdout:9/502: symlink d0/la2 0 2024-04-22T15:49:58.582 INFO:tasks.workunit.client.1.smithi027.stdout:3/552: creat d28/fb5 x:0 0 0 2024-04-22T15:49:58.584 INFO:tasks.workunit.client.1.smithi027.stdout:1/452: dwrite d2/d1d/f5b [0,4194304] 0 2024-04-22T15:49:58.584 INFO:tasks.workunit.client.1.smithi027.stdout:3/553: chown d28/d4b/d52/db2/l70 0 1 2024-04-22T15:49:58.591 INFO:tasks.workunit.client.1.smithi027.stdout:5/523: mkdir d0/d1a/d22/d26/d5d/dae 0 2024-04-22T15:49:58.592 INFO:tasks.workunit.client.1.smithi027.stdout:7/484: write d0/d16/f81 [824310,76968] 0 2024-04-22T15:49:58.593 INFO:tasks.workunit.client.1.smithi027.stdout:5/524: stat d0/f8 0 2024-04-22T15:49:58.596 INFO:tasks.workunit.client.1.smithi027.stdout:0/529: mknod d0/d2a/d38/d89/ca2 0 2024-04-22T15:49:58.598 INFO:tasks.workunit.client.1.smithi027.stdout:4/525: mknod d5/d11/d1d/d54/d8c/cc9 0 2024-04-22T15:49:58.601 INFO:tasks.workunit.client.1.smithi027.stdout:1/453: truncate d2/f3 2735100 0 2024-04-22T15:49:58.610 INFO:tasks.workunit.client.1.smithi027.stdout:4/526: dread d5/f13 [0,4194304] 0 2024-04-22T15:49:58.640 INFO:tasks.workunit.client.1.smithi027.stdout:9/503: creat d0/dc/fa3 x:0 0 0 2024-04-22T15:49:58.643 INFO:tasks.workunit.client.1.smithi027.stdout:0/530: unlink d0/d2a/l30 0 2024-04-22T15:49:58.647 INFO:tasks.workunit.client.1.smithi027.stdout:7/485: link d0/d11/d3a/d46/l76 d0/d11/d1c/d2c/l97 0 2024-04-22T15:49:58.647 INFO:tasks.workunit.client.1.smithi027.stdout:7/486: stat d0/d11/d1c/d31/f5f 0 2024-04-22T15:49:58.652 INFO:tasks.workunit.client.1.smithi027.stdout:0/531: mkdir d0/da3 0 2024-04-22T15:49:58.654 INFO:tasks.workunit.client.1.smithi027.stdout:0/532: truncate d0/d34/f9c 9138805 0 2024-04-22T15:49:58.655 INFO:tasks.workunit.client.1.smithi027.stdout:7/487: creat d0/d4c/d8b/f98 x:0 0 0 2024-04-22T15:49:58.656 INFO:tasks.workunit.client.1.smithi027.stdout:3/554: link d28/d5d/d79/la5 d28/d4b/d52/db2/d95/lb6 0 2024-04-22T15:49:58.658 INFO:tasks.workunit.client.1.smithi027.stdout:1/454: rename d2/d1d/f1c to d2/d30/fa3 0 2024-04-22T15:49:58.660 INFO:tasks.workunit.client.1.smithi027.stdout:7/488: dread d0/f13 [0,4194304] 0 2024-04-22T15:49:58.661 INFO:tasks.workunit.client.1.smithi027.stdout:7/489: write d0/d16/f81 [159126,71268] 0 2024-04-22T15:49:58.664 INFO:tasks.workunit.client.1.smithi027.stdout:3/555: symlink d28/d5d/d62/lb7 0 2024-04-22T15:49:58.669 INFO:tasks.workunit.client.1.smithi027.stdout:1/455: creat d2/d5f/d64/d81/d3d/d82/d85/fa4 x:0 0 0 2024-04-22T15:49:58.671 INFO:tasks.workunit.client.1.smithi027.stdout:4/527: getdents d5/d42 0 2024-04-22T15:49:58.673 INFO:tasks.workunit.client.1.smithi027.stdout:4/528: chown d5/de/da9/lbc 201 1 2024-04-22T15:49:58.676 INFO:tasks.workunit.client.1.smithi027.stdout:5/525: dwrite d0/df/d10/f33 [0,4194304] 0 2024-04-22T15:49:58.676 INFO:tasks.workunit.client.1.smithi027.stdout:7/490: mknod d0/d3f/c99 0 2024-04-22T15:49:58.678 INFO:tasks.workunit.client.1.smithi027.stdout:5/526: readlink d0/df/de/d40/d89/la1 0 2024-04-22T15:49:58.680 INFO:tasks.workunit.client.1.smithi027.stdout:5/527: write d0/d1a/d22/d26/d32/d88/d90/f9f [985669,115386] 0 2024-04-22T15:49:58.690 INFO:tasks.workunit.client.1.smithi027.stdout:0/533: link d0/d34/l84 d0/d28/d37/la4 0 2024-04-22T15:49:58.692 INFO:tasks.workunit.client.1.smithi027.stdout:4/529: mkdir d5/d11/d6c/dca 0 2024-04-22T15:49:58.692 INFO:tasks.workunit.client.1.smithi027.stdout:3/556: dwrite d28/d4b/f53 [4194304,4194304] 0 2024-04-22T15:49:58.694 INFO:tasks.workunit.client.1.smithi027.stdout:7/491: rename d0/d11/c7f to d0/d16/d65/c9a 0 2024-04-22T15:49:58.696 INFO:tasks.workunit.client.1.smithi027.stdout:0/534: chown d0/d34/f3a 1 1 2024-04-22T15:49:58.698 INFO:tasks.workunit.client.1.smithi027.stdout:5/528: fdatasync d0/df/d3d/f69 0 2024-04-22T15:49:58.700 INFO:tasks.workunit.client.1.smithi027.stdout:9/504: dwrite d0/dc/de/f3a [0,4194304] 0 2024-04-22T15:49:58.700 INFO:tasks.workunit.client.1.smithi027.stdout:4/530: write d5/d42/f9e [3652228,39075] 0 2024-04-22T15:49:58.702 INFO:tasks.workunit.client.1.smithi027.stdout:1/456: truncate d2/d36/f55 4032553 0 2024-04-22T15:49:58.704 INFO:tasks.workunit.client.1.smithi027.stdout:4/531: chown d5/de/d30/d5d/f99 649969321 1 2024-04-22T15:49:58.706 INFO:tasks.workunit.client.1.smithi027.stdout:4/532: dread - d5/d45/d47/f8a zero size 2024-04-22T15:49:58.707 INFO:tasks.workunit.client.1.smithi027.stdout:3/557: fdatasync f10 0 2024-04-22T15:49:58.710 INFO:tasks.workunit.client.1.smithi027.stdout:1/457: rmdir d2/d5f/d64/d81 39 2024-04-22T15:49:58.713 INFO:tasks.workunit.client.1.smithi027.stdout:0/535: creat d0/d2a/d5e/d8b/fa5 x:0 0 0 2024-04-22T15:49:58.715 INFO:tasks.workunit.client.1.smithi027.stdout:5/529: dwrite d0/df/de/d24/d1c/d3e/d55/f7d [0,4194304] 0 2024-04-22T15:49:58.715 INFO:tasks.workunit.client.1.smithi027.stdout:0/536: stat d0/d9d 0 2024-04-22T15:49:58.717 INFO:tasks.workunit.client.1.smithi027.stdout:5/530: readlink d0/df/de/d40/d89/la1 0 2024-04-22T15:49:58.718 INFO:tasks.workunit.client.1.smithi027.stdout:3/558: rmdir d28/d4b/d52/db2/d95 39 2024-04-22T15:49:58.720 INFO:tasks.workunit.client.1.smithi027.stdout:1/458: readlink d2/d30/l97 0 2024-04-22T15:49:58.728 INFO:tasks.workunit.client.1.smithi027.stdout:3/559: dwrite d28/d5d/d79/fa1 [0,4194304] 0 2024-04-22T15:49:58.728 INFO:tasks.workunit.client.1.smithi027.stdout:4/533: creat d5/de/dc5/fcb x:0 0 0 2024-04-22T15:49:58.738 INFO:tasks.workunit.client.1.smithi027.stdout:3/560: dwrite d28/d5d/fa8 [0,4194304] 0 2024-04-22T15:49:58.750 INFO:tasks.workunit.client.1.smithi027.stdout:5/531: rename d0/df/de/d40/d89/la1 to d0/d1a/d22/d26/d5d/d99/laf 0 2024-04-22T15:49:58.751 INFO:tasks.workunit.client.1.smithi027.stdout:1/459: dread d2/d65/f67 [0,4194304] 0 2024-04-22T15:49:58.753 INFO:tasks.workunit.client.1.smithi027.stdout:7/492: dread d0/d11/d1c/d31/d56/f58 [0,4194304] 0 2024-04-22T15:49:58.754 INFO:tasks.workunit.client.1.smithi027.stdout:7/493: readlink d0/d16/d65/l8c 0 2024-04-22T15:49:58.756 INFO:tasks.workunit.client.1.smithi027.stdout:6/504: sync 2024-04-22T15:49:58.756 INFO:tasks.workunit.client.1.smithi027.stdout:8/464: sync 2024-04-22T15:49:58.758 INFO:tasks.workunit.client.1.smithi027.stdout:6/505: chown d5/dc/d17/d8a/c95 374798 1 2024-04-22T15:49:58.758 INFO:tasks.workunit.client.1.smithi027.stdout:8/465: dread - d1/d3/d1c/d32/d41/d52/fa1 zero size 2024-04-22T15:49:58.759 INFO:tasks.workunit.client.1.smithi027.stdout:4/534: symlink d5/de/dc5/lcc 0 2024-04-22T15:49:58.762 INFO:tasks.workunit.client.1.smithi027.stdout:4/535: chown d5/de/d2a/d58/caa 1 1 2024-04-22T15:49:58.763 INFO:tasks.workunit.client.1.smithi027.stdout:4/536: fdatasync d5/de/da9/fc7 0 2024-04-22T15:49:58.765 INFO:tasks.workunit.client.1.smithi027.stdout:7/494: dwrite d0/d16/f81 [0,4194304] 0 2024-04-22T15:49:58.766 INFO:tasks.workunit.client.1.smithi027.stdout:7/495: fsync d0/f3e 0 2024-04-22T15:49:58.768 INFO:tasks.workunit.client.1.smithi027.stdout:5/532: unlink d0/df/f56 0 2024-04-22T15:49:58.771 INFO:tasks.workunit.client.1.smithi027.stdout:9/505: link d0/dc/d12/d58/c7e d0/d51/ca4 0 2024-04-22T15:49:58.772 INFO:tasks.workunit.client.1.smithi027.stdout:9/506: write d0/dc/de/f24 [4840361,117646] 0 2024-04-22T15:49:58.782 INFO:tasks.workunit.client.1.smithi027.stdout:6/506: creat d5/dc/d20/d2b/d34/fb0 x:0 0 0 2024-04-22T15:49:58.787 INFO:tasks.workunit.client.1.smithi027.stdout:3/561: link d28/d5d/d79/f87 d28/d5d/d79/fb8 0 2024-04-22T15:49:58.788 INFO:tasks.workunit.client.1.smithi027.stdout:8/466: creat d1/d3/d1c/d43/d5f/d7d/fa3 x:0 0 0 2024-04-22T15:49:58.789 INFO:tasks.workunit.client.1.smithi027.stdout:3/562: write d28/d5d/f8b [1939564,60106] 0 2024-04-22T15:49:58.791 INFO:tasks.workunit.client.1.smithi027.stdout:4/537: mkdir d5/d45/d5a/dcd 0 2024-04-22T15:49:58.792 INFO:tasks.workunit.client.1.smithi027.stdout:4/538: fsync d5/d45/d47/f8a 0 2024-04-22T15:49:58.797 INFO:tasks.workunit.client.1.smithi027.stdout:7/496: creat d0/d16/d65/f9b x:0 0 0 2024-04-22T15:49:58.798 INFO:tasks.workunit.client.1.smithi027.stdout:7/497: chown d0/d4c/d8b/f98 3456 1 2024-04-22T15:49:58.800 INFO:tasks.workunit.client.1.smithi027.stdout:9/507: symlink d0/dc/d12/d30/la5 0 2024-04-22T15:49:58.801 INFO:tasks.workunit.client.1.smithi027.stdout:9/508: chown d0/dc/de 29 1 2024-04-22T15:49:58.801 INFO:tasks.workunit.client.1.smithi027.stdout:0/537: getdents d0/d2a/d6a/d6c 0 2024-04-22T15:49:58.802 INFO:tasks.workunit.client.1.smithi027.stdout:9/509: fsync d0/d51/f7f 0 2024-04-22T15:49:58.815 INFO:tasks.workunit.client.1.smithi027.stdout:4/539: creat d5/d11/d1d/d7b/fce x:0 0 0 2024-04-22T15:49:58.816 INFO:tasks.workunit.client.1.smithi027.stdout:4/540: write d5/d11/f80 [760445,79259] 0 2024-04-22T15:49:58.818 INFO:tasks.workunit.client.1.smithi027.stdout:2/515: sync 2024-04-22T15:49:58.822 INFO:tasks.workunit.client.1.smithi027.stdout:0/538: mkdir d0/d2a/d6a/da6 0 2024-04-22T15:49:58.835 INFO:tasks.workunit.client.1.smithi027.stdout:6/507: write d5/dc/dd/f16 [3849032,32548] 0 2024-04-22T15:49:58.839 INFO:tasks.workunit.client.1.smithi027.stdout:5/533: read d0/df/d3d/f69 [56146,97496] 0 2024-04-22T15:49:58.840 INFO:tasks.workunit.client.1.smithi027.stdout:8/467: truncate d1/d3/f40 3771869 0 2024-04-22T15:49:58.842 INFO:tasks.workunit.client.1.smithi027.stdout:8/468: readlink d1/d3/d1c/l3e 0 2024-04-22T15:49:58.843 INFO:tasks.workunit.client.1.smithi027.stdout:1/460: dwrite d2/d5f/d64/d81/f3a [0,4194304] 0 2024-04-22T15:49:58.849 INFO:tasks.workunit.client.1.smithi027.stdout:7/498: link d0/d11/d3a/f45 d0/d11/d3a/d46/f9c 0 2024-04-22T15:49:58.851 INFO:tasks.workunit.client.1.smithi027.stdout:9/510: mkdir d0/dc/d12/d58/da6 0 2024-04-22T15:49:58.852 INFO:tasks.workunit.client.1.smithi027.stdout:3/563: creat d28/fb9 x:0 0 0 2024-04-22T15:49:58.853 INFO:tasks.workunit.client.1.smithi027.stdout:3/564: truncate d28/d4b/f59 953169 0 2024-04-22T15:49:58.853 INFO:tasks.workunit.client.1.smithi027.stdout:6/508: creat d5/db/d2e/d59/fb1 x:0 0 0 2024-04-22T15:49:58.857 INFO:tasks.workunit.client.1.smithi027.stdout:1/461: readlink d2/l25 0 2024-04-22T15:49:58.861 INFO:tasks.workunit.client.1.smithi027.stdout:2/516: mkdir dd/df/d1b/d38/d60/d75/d9f 0 2024-04-22T15:49:58.862 INFO:tasks.workunit.client.1.smithi027.stdout:7/499: creat d0/d16/d65/f9d x:0 0 0 2024-04-22T15:49:58.871 INFO:tasks.workunit.client.1.smithi027.stdout:5/534: rmdir d0/df/de/d24/d1c/d3e/da2 39 2024-04-22T15:49:58.875 INFO:tasks.workunit.client.1.smithi027.stdout:8/469: link d1/d3/f99 d1/d3/d1c/d43/d5f/fa4 0 2024-04-22T15:49:58.878 INFO:tasks.workunit.client.1.smithi027.stdout:1/462: rmdir d2/d5f/d64/d81/d69 39 2024-04-22T15:49:58.882 INFO:tasks.workunit.client.1.smithi027.stdout:2/517: symlink dd/df/d70/d4b/d84/la0 0 2024-04-22T15:49:58.883 INFO:tasks.workunit.client.1.smithi027.stdout:4/541: dwrite d5/d45/f59 [0,4194304] 0 2024-04-22T15:49:58.884 INFO:tasks.workunit.client.1.smithi027.stdout:2/518: write dd/d47/f7f [595097,66217] 0 2024-04-22T15:49:58.887 INFO:tasks.workunit.client.1.smithi027.stdout:0/539: creat d0/d2a/fa7 x:0 0 0 2024-04-22T15:49:58.887 INFO:tasks.workunit.client.1.smithi027.stdout:2/519: chown dd/df/d1b/c4c 1361750 1 2024-04-22T15:49:58.891 INFO:tasks.workunit.client.1.smithi027.stdout:9/511: mkdir d0/dc/d12/d30/d99/da7 0 2024-04-22T15:49:58.892 INFO:tasks.workunit.client.1.smithi027.stdout:3/565: creat d28/d5d/db0/fba x:0 0 0 2024-04-22T15:49:58.898 INFO:tasks.workunit.client.1.smithi027.stdout:1/463: symlink d2/d5f/la5 0 2024-04-22T15:49:58.902 INFO:tasks.workunit.client.1.smithi027.stdout:4/542: rename d5/d45/fad to d5/d11/dc1/fcf 0 2024-04-22T15:49:58.908 INFO:tasks.workunit.client.1.smithi027.stdout:4/543: dwrite d5/de/d2a/d58/f68 [0,4194304] 0 2024-04-22T15:49:58.909 INFO:tasks.workunit.client.1.smithi027.stdout:7/500: unlink d0/d11/d3a/d46/l76 0 2024-04-22T15:49:58.910 INFO:tasks.workunit.client.1.smithi027.stdout:3/566: write d28/f49 [311626,24590] 0 2024-04-22T15:49:58.914 INFO:tasks.workunit.client.1.smithi027.stdout:5/535: getdents d0/df/d3d/d81 0 2024-04-22T15:49:58.917 INFO:tasks.workunit.client.1.smithi027.stdout:1/464: fdatasync d2/d30/fa3 0 2024-04-22T15:49:58.919 INFO:tasks.workunit.client.1.smithi027.stdout:9/512: rename d0/dc/d12/d58/d6b/l9e to d0/dc/d12/d30/d99/da7/la8 0 2024-04-22T15:49:58.923 INFO:tasks.workunit.client.1.smithi027.stdout:4/544: readlink d5/de/d2a/d58/la5 0 2024-04-22T15:49:58.924 INFO:tasks.workunit.client.1.smithi027.stdout:4/545: truncate d5/de/dc5/fcb 616121 0 2024-04-22T15:49:58.926 INFO:tasks.workunit.client.1.smithi027.stdout:3/567: dread - d28/d69/f82 zero size 2024-04-22T15:49:58.927 INFO:tasks.workunit.client.1.smithi027.stdout:6/509: link d5/db/d2e/l58 d5/dc/d20/lb2 0 2024-04-22T15:49:58.931 INFO:tasks.workunit.client.1.smithi027.stdout:5/536: creat d0/d1a/d22/d26/d32/d88/fb0 x:0 0 0 2024-04-22T15:49:58.933 INFO:tasks.workunit.client.1.smithi027.stdout:1/465: write d2/d36/f62 [291484,48120] 0 2024-04-22T15:49:58.936 INFO:tasks.workunit.client.1.smithi027.stdout:9/513: mkdir d0/dc/d12/d30/d5b/d91/da9 0 2024-04-22T15:49:58.937 INFO:tasks.workunit.client.1.smithi027.stdout:9/514: truncate d0/dc/fa3 987854 0 2024-04-22T15:49:58.939 INFO:tasks.workunit.client.1.smithi027.stdout:4/546: mkdir d5/d11/da6/dd0 0 2024-04-22T15:49:58.944 INFO:tasks.workunit.client.1.smithi027.stdout:8/470: getdents d1/d1e/d47 0 2024-04-22T15:49:58.947 INFO:tasks.workunit.client.1.smithi027.stdout:6/510: mknod d5/dc/d17/d53/d8c/cb3 0 2024-04-22T15:49:58.950 INFO:tasks.workunit.client.1.smithi027.stdout:2/520: getdents dd/d47 0 2024-04-22T15:49:58.952 INFO:tasks.workunit.client.1.smithi027.stdout:0/540: getdents d0/d2a/d5e 0 2024-04-22T15:49:58.952 INFO:tasks.workunit.client.1.smithi027.stdout:8/471: dwrite d1/d3/d1c/d43/d5f/f9e [0,4194304] 0 2024-04-22T15:49:58.953 INFO:tasks.workunit.client.1.smithi027.stdout:1/466: chown d2/d36/f55 32 1 2024-04-22T15:49:58.954 INFO:tasks.workunit.client.1.smithi027.stdout:0/541: read d0/f1 [2309367,53478] 0 2024-04-22T15:49:58.956 INFO:tasks.workunit.client.1.smithi027.stdout:5/537: dread d0/df/f5e [0,4194304] 0 2024-04-22T15:49:58.958 INFO:tasks.workunit.client.1.smithi027.stdout:8/472: dread - d1/d3/d1c/d32/d41/d52/d61/fa2 zero size 2024-04-22T15:49:58.959 INFO:tasks.workunit.client.1.smithi027.stdout:9/515: mknod d0/dc/d12/d30/d5b/caa 0 2024-04-22T15:49:58.959 INFO:tasks.workunit.client.1.smithi027.stdout:8/473: chown d1/d3/d36/c4e 128522332 1 2024-04-22T15:49:58.959 INFO:tasks.workunit.client.1.smithi027.stdout:4/547: symlink d5/de/ld1 0 2024-04-22T15:49:58.961 INFO:tasks.workunit.client.1.smithi027.stdout:3/568: creat d28/d4b/d52/d94/fbb x:0 0 0 2024-04-22T15:49:58.962 INFO:tasks.workunit.client.1.smithi027.stdout:4/548: write d5/de/d30/d5d/f9f [971404,76686] 0 2024-04-22T15:49:58.964 INFO:tasks.workunit.client.1.smithi027.stdout:3/569: stat d28/d5d/c73 0 2024-04-22T15:49:58.969 INFO:tasks.workunit.client.1.smithi027.stdout:1/467: symlink d2/d36/d74/la6 0 2024-04-22T15:49:58.971 INFO:tasks.workunit.client.1.smithi027.stdout:8/474: dread d1/d3/d1c/f34 [0,4194304] 0 2024-04-22T15:49:58.972 INFO:tasks.workunit.client.1.smithi027.stdout:2/521: rename dd/df/d70/c69 to dd/df/d70/d4b/d84/ca1 0 2024-04-22T15:49:58.974 INFO:tasks.workunit.client.1.smithi027.stdout:0/542: write d0/d2a/f2b [4266672,97501] 0 2024-04-22T15:49:58.977 INFO:tasks.workunit.client.1.smithi027.stdout:8/475: dread d1/d3/d1c/d43/d5f/f9e [0,4194304] 0 2024-04-22T15:49:58.979 INFO:tasks.workunit.client.1.smithi027.stdout:3/570: dread d28/d4b/f7d [0,4194304] 0 2024-04-22T15:49:58.979 INFO:tasks.workunit.client.1.smithi027.stdout:7/501: getdents d0/d11/d1c 0 2024-04-22T15:49:58.980 INFO:tasks.workunit.client.1.smithi027.stdout:7/502: stat d0/d11/c2b 0 2024-04-22T15:49:58.985 INFO:tasks.workunit.client.1.smithi027.stdout:9/516: write d0/dc/d12/f23 [296901,32602] 0 2024-04-22T15:49:58.990 INFO:tasks.workunit.client.1.smithi027.stdout:4/549: creat d5/d11/dc1/fd2 x:0 0 0 2024-04-22T15:49:58.991 INFO:tasks.workunit.client.1.smithi027.stdout:6/511: dwrite d5/dc/d20/d2b/f24 [0,4194304] 0 2024-04-22T15:49:58.995 INFO:tasks.workunit.client.1.smithi027.stdout:1/468: rmdir d2/d5f/d64/d81/d3d/d44/d9c/d9f 39 2024-04-22T15:49:59.001 INFO:tasks.workunit.client.1.smithi027.stdout:1/469: dread d2/d5f/d64/d81/d3d/d44/f6d [0,4194304] 0 2024-04-22T15:49:59.002 INFO:tasks.workunit.client.1.smithi027.stdout:3/571: rename d28/d4b/l4d to d28/d5d/d79/lbc 0 2024-04-22T15:49:59.004 INFO:tasks.workunit.client.1.smithi027.stdout:5/538: mknod d0/df/cb1 0 2024-04-22T15:49:59.006 INFO:tasks.workunit.client.1.smithi027.stdout:5/539: chown d0/d1a/d22/d26/d32/f4f 0 1 2024-04-22T15:49:59.012 INFO:tasks.workunit.client.1.smithi027.stdout:6/512: mknod d5/dc/d99/cb4 0 2024-04-22T15:49:59.014 INFO:tasks.workunit.client.1.smithi027.stdout:2/522: dread dd/df/d1b/d1f/f2c [0,4194304] 0 2024-04-22T15:49:59.015 INFO:tasks.workunit.client.1.smithi027.stdout:2/523: chown dd/d43/f95 100 1 2024-04-22T15:49:59.017 INFO:tasks.workunit.client.1.smithi027.stdout:0/543: readlink d0/d28/d37/l4c 0 2024-04-22T15:49:59.023 INFO:tasks.workunit.client.1.smithi027.stdout:9/517: write d0/dc/f80 [602285,34954] 0 2024-04-22T15:49:59.028 INFO:tasks.workunit.client.1.smithi027.stdout:6/513: mkdir d5/db/d2e/d59/db5 0 2024-04-22T15:49:59.029 INFO:tasks.workunit.client.1.smithi027.stdout:4/550: dwrite d5/de/f37 [0,4194304] 0 2024-04-22T15:49:59.035 INFO:tasks.workunit.client.1.smithi027.stdout:0/544: rmdir d0/d2a 39 2024-04-22T15:49:59.041 INFO:tasks.workunit.client.1.smithi027.stdout:5/540: creat d0/df/de/d40/fb2 x:0 0 0 2024-04-22T15:49:59.043 INFO:tasks.workunit.client.1.smithi027.stdout:9/518: creat d0/dc/d12/d30/d99/fab x:0 0 0 2024-04-22T15:49:59.044 INFO:tasks.workunit.client.1.smithi027.stdout:9/519: read - d0/dc/d12/d30/d5b/f65 zero size 2024-04-22T15:49:59.045 INFO:tasks.workunit.client.1.smithi027.stdout:9/520: fdatasync d0/dc/d12/d30/f7a 0 2024-04-22T15:49:59.052 INFO:tasks.workunit.client.1.smithi027.stdout:4/551: dwrite d5/d11/d1d/f5e [0,4194304] 0 2024-04-22T15:49:59.055 INFO:tasks.workunit.client.1.smithi027.stdout:1/470: rename d2/d65 to d2/d5f/d64/d81/da7 0 2024-04-22T15:49:59.057 INFO:tasks.workunit.client.1.smithi027.stdout:1/471: write d2/d5f/d64/d81/d3d/d82/d85/f95 [402047,50532] 0 2024-04-22T15:49:59.059 INFO:tasks.workunit.client.1.smithi027.stdout:1/472: truncate d2/d36/f7c 724254 0 2024-04-22T15:49:59.061 INFO:tasks.workunit.client.1.smithi027.stdout:3/572: getdents d28/d5d 0 2024-04-22T15:49:59.062 INFO:tasks.workunit.client.1.smithi027.stdout:4/552: creat d5/d11/d1d/d7b/fd3 x:0 0 0 2024-04-22T15:49:59.067 INFO:tasks.workunit.client.1.smithi027.stdout:2/524: rename dd/df/f49 to dd/df/d1b/d38/d60/fa2 0 2024-04-22T15:49:59.068 INFO:tasks.workunit.client.1.smithi027.stdout:3/573: dwrite d28/d4b/d52/db2/f6f [0,4194304] 0 2024-04-22T15:49:59.070 INFO:tasks.workunit.client.1.smithi027.stdout:5/541: mknod d0/cb3 0 2024-04-22T15:49:59.070 INFO:tasks.workunit.client.1.smithi027.stdout:2/525: dread - dd/df/d1b/d1f/f85 zero size 2024-04-22T15:49:59.072 INFO:tasks.workunit.client.1.smithi027.stdout:5/542: fdatasync d0/d1a/d22/d26/d5d/f8b 0 2024-04-22T15:49:59.080 INFO:tasks.workunit.client.1.smithi027.stdout:0/545: creat d0/fa8 x:0 0 0 2024-04-22T15:49:59.087 INFO:tasks.workunit.client.1.smithi027.stdout:4/553: creat d5/de/d2a/d58/fd4 x:0 0 0 2024-04-22T15:49:59.090 INFO:tasks.workunit.client.1.smithi027.stdout:6/514: getdents d5/dc/d26 0 2024-04-22T15:49:59.097 INFO:tasks.workunit.client.1.smithi027.stdout:3/574: creat d28/d5d/db0/fbd x:0 0 0 2024-04-22T15:49:59.100 INFO:tasks.workunit.client.1.smithi027.stdout:5/543: dwrite d0/df/d10/f60 [0,4194304] 0 2024-04-22T15:49:59.108 INFO:tasks.workunit.client.1.smithi027.stdout:6/515: dread d5/dc/dd/d79/d7b/f4e [0,4194304] 0 2024-04-22T15:49:59.110 INFO:tasks.workunit.client.1.smithi027.stdout:6/516: write d5/dc/d20/d2b/f78 [321247,25918] 0 2024-04-22T15:49:59.111 INFO:tasks.workunit.client.1.smithi027.stdout:2/526: creat dd/df/d70/d4b/d52/d9a/fa3 x:0 0 0 2024-04-22T15:49:59.116 INFO:tasks.workunit.client.1.smithi027.stdout:1/473: creat d2/d5f/d64/d81/d69/d7d/fa8 x:0 0 0 2024-04-22T15:49:59.118 INFO:tasks.workunit.client.1.smithi027.stdout:2/527: dread dd/df/d1b/d38/d60/fa2 [0,4194304] 0 2024-04-22T15:49:59.119 INFO:tasks.workunit.client.1.smithi027.stdout:9/521: getdents d0/dc 0 2024-04-22T15:49:59.119 INFO:tasks.workunit.client.1.smithi027.stdout:3/575: mknod d28/d5d/d79/cbe 0 2024-04-22T15:49:59.125 INFO:tasks.workunit.client.1.smithi027.stdout:0/546: creat d0/d2a/d38/d89/fa9 x:0 0 0 2024-04-22T15:49:59.133 INFO:tasks.workunit.client.1.smithi027.stdout:3/576: unlink d28/d4b/c67 0 2024-04-22T15:49:59.134 INFO:tasks.workunit.client.1.smithi027.stdout:2/528: dread dd/d47/f48 [0,4194304] 0 2024-04-22T15:49:59.137 INFO:tasks.workunit.client.1.smithi027.stdout:4/554: creat d5/fd5 x:0 0 0 2024-04-22T15:49:59.141 INFO:tasks.workunit.client.1.smithi027.stdout:6/517: mknod d5/db/cb6 0 2024-04-22T15:49:59.142 INFO:tasks.workunit.client.1.smithi027.stdout:6/518: fdatasync d5/dc/dd/d79/d7b/d5d/f70 0 2024-04-22T15:49:59.143 INFO:tasks.workunit.client.1.smithi027.stdout:6/519: write d5/f75 [2163101,126268] 0 2024-04-22T15:49:59.144 INFO:tasks.workunit.client.1.smithi027.stdout:6/520: chown d5/dc/d20/d2b/d34/c41 52 1 2024-04-22T15:49:59.146 INFO:tasks.workunit.client.1.smithi027.stdout:9/522: symlink d0/dc/d12/d58/lac 0 2024-04-22T15:49:59.149 INFO:tasks.workunit.client.1.smithi027.stdout:2/529: mkdir dd/df/d70/d4b/d52/d9a/da4 0 2024-04-22T15:49:59.152 INFO:tasks.workunit.client.1.smithi027.stdout:2/530: chown dd/df/d1b/d96/f98 33 1 2024-04-22T15:49:59.153 INFO:tasks.workunit.client.1.smithi027.stdout:6/521: dwrite d5/dc/d99/fab [0,4194304] 0 2024-04-22T15:49:59.156 INFO:tasks.workunit.client.1.smithi027.stdout:6/522: chown d5/f75 134 1 2024-04-22T15:49:59.157 INFO:tasks.workunit.client.1.smithi027.stdout:3/577: dread d28/f36 [0,4194304] 0 2024-04-22T15:49:59.160 INFO:tasks.workunit.client.1.smithi027.stdout:3/578: stat d28/d4b/f8e 0 2024-04-22T15:49:59.160 INFO:tasks.workunit.client.1.smithi027.stdout:5/544: getdents d0/df 0 2024-04-22T15:49:59.160 INFO:tasks.workunit.client.1.smithi027.stdout:3/579: dread - d28/fb5 zero size 2024-04-22T15:49:59.161 INFO:tasks.workunit.client.1.smithi027.stdout:2/531: mknod dd/d47/ca5 0 2024-04-22T15:49:59.162 INFO:tasks.workunit.client.1.smithi027.stdout:3/580: fsync d28/d4b/f59 0 2024-04-22T15:49:59.163 INFO:tasks.workunit.client.1.smithi027.stdout:9/523: creat d0/d51/d9d/fad x:0 0 0 2024-04-22T15:49:59.163 INFO:tasks.workunit.client.1.smithi027.stdout:4/555: fdatasync d5/d11/d1d/d24/f9c 0 2024-04-22T15:49:59.166 INFO:tasks.workunit.client.1.smithi027.stdout:6/523: rename d5/dc/d99/c71 to d5/dc/dd/cb7 0 2024-04-22T15:49:59.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:58 smithi016 ceph-mon[26620]: Manager daemon smithi016.rnsnnr is now available 2024-04-22T15:49:59.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:58 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:49:59.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:58 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:49:59.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:58 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi016.rnsnnr/mirror_snapshot_schedule"}]: dispatch 2024-04-22T15:49:59.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:58 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi016.rnsnnr/trash_purge_schedule"}]: dispatch 2024-04-22T15:49:59.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:58 smithi016 ceph-mon[26620]: [22/Apr/2024:15:49:58] ENGINE Bus STARTING 2024-04-22T15:49:59.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:58 smithi016 ceph-mon[26620]: [22/Apr/2024:15:49:58] ENGINE Serving on https://172.21.15.16:7150 2024-04-22T15:49:59.167 INFO:tasks.workunit.client.1.smithi027.stdout:0/547: getdents d0 0 2024-04-22T15:49:59.168 INFO:tasks.workunit.client.1.smithi027.stdout:2/532: write dd/df/d1b/d96/f9d [225501,74958] 0 2024-04-22T15:49:59.169 INFO:tasks.workunit.client.1.smithi027.stdout:6/524: read - d5/dc/d17/d53/fa1 zero size 2024-04-22T15:49:59.182 INFO:tasks.workunit.client.1.smithi027.stdout:5/545: dread d0/d1a/d22/d26/d32/d79/fa7 [0,4194304] 0 2024-04-22T15:49:59.183 INFO:tasks.workunit.client.1.smithi027.stdout:5/546: stat d0/d1a/d22/d26/d32/d88/d90 0 2024-04-22T15:49:59.207 INFO:tasks.workunit.client.1.smithi027.stdout:4/556: unlink d5/d45/f64 0 2024-04-22T15:49:59.210 INFO:tasks.workunit.client.1.smithi027.stdout:2/533: rename dd/df/d70/d5a/f77 to dd/df/d1b/d38/d60/d75/fa6 0 2024-04-22T15:49:59.213 INFO:tasks.workunit.client.1.smithi027.stdout:5/547: mknod d0/df/cb4 0 2024-04-22T15:49:59.214 INFO:tasks.workunit.client.1.smithi027.stdout:3/581: mkdir d28/d4b/d52/db2/d95/dbf 0 2024-04-22T15:49:59.215 INFO:tasks.workunit.client.1.smithi027.stdout:1/474: truncate d2/d30/f3c 1634533 0 2024-04-22T15:49:59.215 INFO:tasks.workunit.client.1.smithi027.stdout:1/475: chown d2/fa 2413 1 2024-04-22T15:49:59.216 INFO:tasks.workunit.client.1.smithi027.stdout:9/524: creat d0/dc/d12/d58/da6/fae x:0 0 0 2024-04-22T15:49:59.217 INFO:tasks.workunit.client.1.smithi027.stdout:1/476: chown d2/d5f/d64/d81/d3d/d44 0 1 2024-04-22T15:49:59.228 INFO:tasks.workunit.client.1.smithi027.stdout:2/534: write dd/df/d1b/f25 [1094774,36290] 0 2024-04-22T15:49:59.235 INFO:tasks.workunit.client.1.smithi027.stdout:3/582: unlink d28/c8c 0 2024-04-22T15:49:59.235 INFO:tasks.workunit.client.1.smithi027.stdout:0/548: truncate d0/d2a/f35 476892 0 2024-04-22T15:49:59.238 INFO:tasks.workunit.client.1.smithi027.stdout:9/525: mkdir d0/dc/d12/d30/d5b/d91/daf 0 2024-04-22T15:49:59.242 INFO:tasks.workunit.client.1.smithi027.stdout:1/477: stat d2/d30/l46 0 2024-04-22T15:49:59.242 INFO:tasks.workunit.client.1.smithi027.stdout:9/526: read d0/dc/d12/d30/f39 [1244881,37673] 0 2024-04-22T15:49:59.242 INFO:tasks.workunit.client.1.smithi027.stdout:6/525: truncate d5/db/f11 221901 0 2024-04-22T15:49:59.242 INFO:tasks.workunit.client.1.smithi027.stdout:6/526: write d5/db/f4d [5019885,127467] 0 2024-04-22T15:49:59.242 INFO:tasks.workunit.client.1.smithi027.stdout:4/557: rename d5/de/d2a/f3a to d5/d11/da6/fd6 0 2024-04-22T15:49:59.245 INFO:tasks.workunit.client.1.smithi027.stdout:2/535: creat dd/d43/fa7 x:0 0 0 2024-04-22T15:49:59.251 INFO:tasks.workunit.client.1.smithi027.stdout:9/527: symlink d0/dc/d12/d30/d99/lb0 0 2024-04-22T15:49:59.253 INFO:tasks.workunit.client.1.smithi027.stdout:4/558: read - d5/d11/d1d/d7b/f83 zero size 2024-04-22T15:49:59.253 INFO:tasks.workunit.client.1.smithi027.stdout:5/548: dread d0/df/de/d24/d1c/d8f/f94 [0,4194304] 0 2024-04-22T15:49:59.254 INFO:tasks.workunit.client.1.smithi027.stdout:9/528: chown d0/dc/d12/d30/d5b/c75 221343295 1 2024-04-22T15:49:59.254 INFO:tasks.workunit.client.1.smithi027.stdout:3/583: mknod d28/d4b/d52/db2/cc0 0 2024-04-22T15:49:59.255 INFO:tasks.workunit.client.1.smithi027.stdout:3/584: dread - d28/fb5 zero size 2024-04-22T15:49:59.256 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:58 smithi027 ceph-mon[34922]: Manager daemon smithi016.rnsnnr is now available 2024-04-22T15:49:59.256 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:58 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:49:59.256 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:58 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:49:59.256 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:58 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi016.rnsnnr/mirror_snapshot_schedule"}]: dispatch 2024-04-22T15:49:59.256 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:58 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi016.rnsnnr/trash_purge_schedule"}]: dispatch 2024-04-22T15:49:59.256 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:58 smithi027 ceph-mon[34922]: [22/Apr/2024:15:49:58] ENGINE Bus STARTING 2024-04-22T15:49:59.256 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:58 smithi027 ceph-mon[34922]: [22/Apr/2024:15:49:58] ENGINE Serving on https://172.21.15.16:7150 2024-04-22T15:49:59.257 INFO:tasks.workunit.client.1.smithi027.stdout:9/529: read - d0/dc/d12/d30/d5b/f83 zero size 2024-04-22T15:49:59.257 INFO:tasks.workunit.client.1.smithi027.stdout:2/536: creat dd/df/d70/d4b/d84/d8d/fa8 x:0 0 0 2024-04-22T15:49:59.258 INFO:tasks.workunit.client.1.smithi027.stdout:2/537: stat dd/df/d1b/f26 0 2024-04-22T15:49:59.259 INFO:tasks.workunit.client.1.smithi027.stdout:6/527: dread d5/dc/d20/d2b/f27 [0,4194304] 0 2024-04-22T15:49:59.260 INFO:tasks.workunit.client.1.smithi027.stdout:2/538: read - dd/df/d1b/f62 zero size 2024-04-22T15:49:59.266 INFO:tasks.workunit.client.1.smithi027.stdout:4/559: symlink d5/d11/d6c/da3/ld7 0 2024-04-22T15:49:59.269 INFO:tasks.workunit.client.1.smithi027.stdout:3/585: mkdir d28/d4b/d52/db2/d95/d9c/dc1 0 2024-04-22T15:49:59.270 INFO:tasks.workunit.client.1.smithi027.stdout:6/528: dwrite d5/dc/d20/d2b/d34/f65 [0,4194304] 0 2024-04-22T15:49:59.271 INFO:tasks.workunit.client.1.smithi027.stdout:9/530: unlink d0/dc/d12/d30/l95 0 2024-04-22T15:49:59.277 INFO:tasks.workunit.client.1.smithi027.stdout:1/478: write d2/d1d/f32 [1945408,110706] 0 2024-04-22T15:49:59.280 INFO:tasks.workunit.client.1.smithi027.stdout:5/549: creat d0/df/d3d/fb5 x:0 0 0 2024-04-22T15:49:59.285 INFO:tasks.workunit.client.1.smithi027.stdout:3/586: creat d28/d4b/d52/d94/fc2 x:0 0 0 2024-04-22T15:49:59.287 INFO:tasks.workunit.client.1.smithi027.stdout:9/531: chown d0/d8/c1b 217330 1 2024-04-22T15:49:59.288 INFO:tasks.workunit.client.1.smithi027.stdout:9/532: chown d0/dc/de/f49 1053083 1 2024-04-22T15:49:59.290 INFO:tasks.workunit.client.1.smithi027.stdout:6/529: mknod d5/dc/d26/d3d/cb8 0 2024-04-22T15:49:59.291 INFO:tasks.workunit.client.1.smithi027.stdout:0/549: getdents d0/d2a/d5e/d8b 0 2024-04-22T15:49:59.292 INFO:tasks.workunit.client.1.smithi027.stdout:1/479: dread d2/d36/d74/f83 [0,4194304] 0 2024-04-22T15:49:59.293 INFO:tasks.workunit.client.1.smithi027.stdout:1/480: fdatasync d2/f75 0 2024-04-22T15:49:59.300 INFO:tasks.workunit.client.1.smithi027.stdout:9/533: mknod d0/d51/cb1 0 2024-04-22T15:49:59.307 INFO:tasks.workunit.client.1.smithi027.stdout:0/550: symlink d0/d2a/d38/d89/laa 0 2024-04-22T15:49:59.309 INFO:tasks.workunit.client.1.smithi027.stdout:6/530: dwrite d5/dc/d20/d2b/f27 [0,4194304] 0 2024-04-22T15:49:59.311 INFO:tasks.workunit.client.1.smithi027.stdout:6/531: write d5/dc/d20/d2b/f27 [4697230,127248] 0 2024-04-22T15:49:59.313 INFO:tasks.workunit.client.1.smithi027.stdout:5/550: mkdir d0/db6 0 2024-04-22T15:49:59.315 INFO:tasks.workunit.client.1.smithi027.stdout:2/539: link dd/l3a dd/df/d70/la9 0 2024-04-22T15:49:59.315 INFO:tasks.workunit.client.1.smithi027.stdout:0/551: dread d0/d34/f3a [0,4194304] 0 2024-04-22T15:49:59.317 INFO:tasks.workunit.client.1.smithi027.stdout:5/551: chown d0/d1a/f82 0 1 2024-04-22T15:49:59.318 INFO:tasks.workunit.client.1.smithi027.stdout:6/532: truncate d5/dc/dd/d52/f8e 4235583 0 2024-04-22T15:49:59.322 INFO:tasks.workunit.client.1.smithi027.stdout:5/552: readlink d0/d1a/d22/l7e 0 2024-04-22T15:49:59.322 INFO:tasks.workunit.client.1.smithi027.stdout:9/534: mknod d0/dc/d12/d30/d5b/d91/da9/cb2 0 2024-04-22T15:49:59.323 INFO:tasks.workunit.client.1.smithi027.stdout:4/560: getdents d5/d45/d47 0 2024-04-22T15:49:59.326 INFO:tasks.workunit.client.1.smithi027.stdout:3/587: dwrite d28/d4b/d52/d94/fa4 [0,4194304] 0 2024-04-22T15:49:59.332 INFO:tasks.workunit.client.1.smithi027.stdout:6/533: unlink d5/db/f1e 0 2024-04-22T15:49:59.336 INFO:tasks.workunit.client.1.smithi027.stdout:4/561: symlink d5/d11/d1d/d7b/ld8 0 2024-04-22T15:49:59.341 INFO:tasks.workunit.client.1.smithi027.stdout:2/540: dread dd/f1e [0,4194304] 0 2024-04-22T15:49:59.345 INFO:tasks.workunit.client.1.smithi027.stdout:9/535: truncate d0/dc/d12/d58/f8a 3644070 0 2024-04-22T15:49:59.346 INFO:tasks.workunit.client.1.smithi027.stdout:5/553: symlink d0/db6/lb7 0 2024-04-22T15:49:59.346 INFO:tasks.workunit.client.1.smithi027.stdout:9/536: chown d0/dc/d12/d58 6519 1 2024-04-22T15:49:59.348 INFO:tasks.workunit.client.1.smithi027.stdout:9/537: write d0/d51/d9d/fad [960539,90676] 0 2024-04-22T15:49:59.349 INFO:tasks.workunit.client.1.smithi027.stdout:9/538: dread - d0/dc/d12/f94 zero size 2024-04-22T15:49:59.350 INFO:tasks.workunit.client.1.smithi027.stdout:6/534: rmdir d5/d30/d40 39 2024-04-22T15:49:59.350 INFO:tasks.workunit.client.1.smithi027.stdout:9/539: fdatasync d0/dc/d12/d30/f4f 0 2024-04-22T15:49:59.353 INFO:tasks.workunit.client.1.smithi027.stdout:9/540: fdatasync d0/dc/de/f24 0 2024-04-22T15:49:59.361 INFO:tasks.workunit.client.1.smithi027.stdout:0/552: dwrite d0/d34/d59/f5d [0,4194304] 0 2024-04-22T15:49:59.362 INFO:tasks.workunit.client.1.smithi027.stdout:0/553: chown d0/d2a/d6a/d6c/d83 14152054 1 2024-04-22T15:49:59.366 INFO:tasks.workunit.client.1.smithi027.stdout:3/588: dwrite d28/d4b/f5b [0,4194304] 0 2024-04-22T15:49:59.366 INFO:tasks.workunit.client.1.smithi027.stdout:5/554: fsync d0/d1a/d22/d26/d32/f54 0 2024-04-22T15:49:59.367 INFO:tasks.workunit.client.1.smithi027.stdout:6/535: write d5/f7a [3896791,62805] 0 2024-04-22T15:49:59.368 INFO:tasks.workunit.client.1.smithi027.stdout:6/536: stat d5/dc/d26/d3d/f83 0 2024-04-22T15:49:59.371 INFO:tasks.workunit.client.1.smithi027.stdout:9/541: fdatasync d0/dc/d12/d30/f32 0 2024-04-22T15:49:59.372 INFO:tasks.workunit.client.1.smithi027.stdout:9/542: readlink d0/d64/l70 0 2024-04-22T15:49:59.374 INFO:tasks.workunit.client.1.smithi027.stdout:9/543: write d0/dc/f80 [631211,120031] 0 2024-04-22T15:49:59.378 INFO:tasks.workunit.client.1.smithi027.stdout:6/537: dwrite d5/dc/d17/d53/f91 [0,4194304] 0 2024-04-22T15:49:59.380 INFO:tasks.workunit.client.1.smithi027.stdout:2/541: symlink dd/df/d70/laa 0 2024-04-22T15:49:59.388 INFO:tasks.workunit.client.1.smithi027.stdout:3/589: symlink d28/d4b/d52/d94/lc3 0 2024-04-22T15:49:59.388 INFO:tasks.workunit.client.1.smithi027.stdout:0/554: creat d0/d34/d4e/fab x:0 0 0 2024-04-22T15:49:59.393 INFO:tasks.workunit.client.1.smithi027.stdout:9/544: creat d0/dc/d12/fb3 x:0 0 0 2024-04-22T15:49:59.394 INFO:tasks.workunit.client.1.smithi027.stdout:6/538: rename d5/db/d2e/d59/f85 to d5/dc/dd/d79/d7b/d5d/fb9 0 2024-04-22T15:49:59.395 INFO:tasks.workunit.client.1.smithi027.stdout:6/539: write d5/f7a [970259,39466] 0 2024-04-22T15:49:59.397 INFO:tasks.workunit.client.1.smithi027.stdout:2/542: creat dd/df/d1b/fab x:0 0 0 2024-04-22T15:49:59.399 INFO:tasks.workunit.client.1.smithi027.stdout:2/543: write dd/df/d1b/d1f/f85 [509746,52575] 0 2024-04-22T15:49:59.403 INFO:tasks.workunit.client.1.smithi027.stdout:0/555: mknod d0/d2a/d5e/cac 0 2024-04-22T15:49:59.405 INFO:tasks.workunit.client.1.smithi027.stdout:0/556: fdatasync d0/d28/f1c 0 2024-04-22T15:49:59.405 INFO:tasks.workunit.client.1.smithi027.stdout:0/557: chown d0/d9d 28 1 2024-04-22T15:49:59.405 INFO:tasks.workunit.client.1.smithi027.stdout:3/590: mkdir d28/d4b/d52/d94/dc4 0 2024-04-22T15:49:59.405 INFO:tasks.workunit.client.1.smithi027.stdout:0/558: dread - d0/d2a/d5e/d8b/f9f zero size 2024-04-22T15:49:59.409 INFO:tasks.workunit.client.1.smithi027.stdout:4/562: getdents d5/de/d30 0 2024-04-22T15:49:59.410 INFO:tasks.workunit.client.1.smithi027.stdout:9/545: mkdir d0/dc/d12/d30/d99/db4 0 2024-04-22T15:49:59.412 INFO:tasks.workunit.client.1.smithi027.stdout:0/559: dwrite d0/f12 [0,4194304] 0 2024-04-22T15:49:59.413 INFO:tasks.workunit.client.1.smithi027.stdout:0/560: fdatasync d0/d34/f81 0 2024-04-22T15:49:59.413 INFO:tasks.workunit.client.1.smithi027.stdout:6/540: mkdir d5/dc/dd/d79/d7b/d5d/dba 0 2024-04-22T15:49:59.414 INFO:tasks.workunit.client.1.smithi027.stdout:6/541: chown d5/dc/d20/c96 214 1 2024-04-22T15:49:59.416 INFO:tasks.workunit.client.1.smithi027.stdout:2/544: mknod dd/df/d70/d4b/d52/d9a/cac 0 2024-04-22T15:49:59.419 INFO:tasks.workunit.client.1.smithi027.stdout:3/591: rename d28/c50 to d28/d4b/d52/d94/dae/cc5 0 2024-04-22T15:49:59.424 INFO:tasks.workunit.client.1.smithi027.stdout:9/546: unlink d0/d51/f96 0 2024-04-22T15:49:59.424 INFO:tasks.workunit.client.1.smithi027.stdout:2/545: dwrite dd/df/d1b/d38/d60/f65 [0,4194304] 0 2024-04-22T15:49:59.431 INFO:tasks.workunit.client.1.smithi027.stdout:2/546: dread - dd/df/d1b/d1f/f78 zero size 2024-04-22T15:49:59.432 INFO:tasks.workunit.client.1.smithi027.stdout:0/561: creat d0/d2a/d38/fad x:0 0 0 2024-04-22T15:49:59.433 INFO:tasks.workunit.client.1.smithi027.stdout:6/542: creat d5/dc/d17/d8a/fbb x:0 0 0 2024-04-22T15:49:59.436 INFO:tasks.workunit.client.1.smithi027.stdout:4/563: dwrite d5/d11/dc1/fcf [0,4194304] 0 2024-04-22T15:49:59.436 INFO:tasks.workunit.client.1.smithi027.stdout:9/547: mkdir d0/dc/de/db5 0 2024-04-22T15:49:59.443 INFO:tasks.workunit.client.1.smithi027.stdout:2/547: rmdir dd/df/d1b/d96 39 2024-04-22T15:49:59.443 INFO:tasks.workunit.client.1.smithi027.stdout:3/592: mkdir d28/d4b/dc6 0 2024-04-22T15:49:59.451 INFO:tasks.workunit.client.1.smithi027.stdout:6/543: rename d5/db/d2e/c50 to d5/dc/d17/d8a/cbc 0 2024-04-22T15:49:59.454 INFO:tasks.workunit.client.1.smithi027.stdout:3/593: creat d28/d4b/d52/db2/d95/fc7 x:0 0 0 2024-04-22T15:49:59.456 INFO:tasks.workunit.client.1.smithi027.stdout:3/594: write d28/d5d/f96 [2589946,60136] 0 2024-04-22T15:49:59.459 INFO:tasks.workunit.client.1.smithi027.stdout:5/555: truncate d0/df/d10/f60 153566 0 2024-04-22T15:49:59.460 INFO:tasks.workunit.client.1.smithi027.stdout:0/562: truncate d0/d34/f81 3853599 0 2024-04-22T15:49:59.461 INFO:tasks.workunit.client.1.smithi027.stdout:3/595: read d28/f3f [564318,121937] 0 2024-04-22T15:49:59.462 INFO:tasks.workunit.client.1.smithi027.stdout:3/596: fsync d28/d5d/d62/f8f 0 2024-04-22T15:49:59.463 INFO:tasks.workunit.client.1.smithi027.stdout:3/597: truncate d28/d5d/db0/fbd 993511 0 2024-04-22T15:49:59.464 INFO:tasks.workunit.client.1.smithi027.stdout:3/598: fdatasync d28/f88 0 2024-04-22T15:49:59.464 INFO:tasks.workunit.client.1.smithi027.stdout:6/544: rename d5/d30/f47 to d5/db/d2e/d59/fbd 0 2024-04-22T15:49:59.466 INFO:tasks.workunit.client.1.smithi027.stdout:3/599: write d28/d5d/d62/fb1 [188730,44529] 0 2024-04-22T15:49:59.466 INFO:tasks.workunit.client.1.smithi027.stdout:6/545: dread - d5/dc/dd/d79/d7b/d9a/fa0 zero size 2024-04-22T15:49:59.469 INFO:tasks.workunit.client.1.smithi027.stdout:9/548: creat d0/fb6 x:0 0 0 2024-04-22T15:49:59.470 INFO:tasks.workunit.client.1.smithi027.stdout:9/549: dread - d0/dc/d12/d30/d5b/f65 zero size 2024-04-22T15:49:59.484 INFO:tasks.workunit.client.1.smithi027.stdout:4/564: getdents d5/de/d2a/d58 0 2024-04-22T15:49:59.486 INFO:tasks.workunit.client.1.smithi027.stdout:0/563: mkdir d0/da3/dae 0 2024-04-22T15:49:59.486 INFO:tasks.workunit.client.1.smithi027.stdout:0/564: chown d0/d2a/c51 20 1 2024-04-22T15:49:59.487 INFO:tasks.workunit.client.1.smithi027.stdout:2/548: getdents dd/df/d70/d4b/d84 0 2024-04-22T15:49:59.488 INFO:tasks.workunit.client.1.smithi027.stdout:0/565: write d0/d2a/d38/d89/fa9 [280274,70094] 0 2024-04-22T15:49:59.489 INFO:tasks.workunit.client.1.smithi027.stdout:6/546: mkdir d5/d30/dbe 0 2024-04-22T15:49:59.489 INFO:tasks.workunit.client.1.smithi027.stdout:3/600: mknod d28/d4b/d52/db2/d95/d9c/dc1/cc8 0 2024-04-22T15:49:59.491 INFO:tasks.workunit.client.1.smithi027.stdout:3/601: chown d28/d69/l9e 924580 1 2024-04-22T15:49:59.493 INFO:tasks.workunit.client.1.smithi027.stdout:5/556: rename d0/df/de/f13 to d0/df/de/d24/fb8 0 2024-04-22T15:49:59.495 INFO:tasks.workunit.client.1.smithi027.stdout:6/547: unlink d5/dc/d20/d2b/d34/c41 0 2024-04-22T15:49:59.496 INFO:tasks.workunit.client.1.smithi027.stdout:9/550: creat d0/dc/d12/d58/fb7 x:0 0 0 2024-04-22T15:49:59.497 INFO:tasks.workunit.client.1.smithi027.stdout:4/565: symlink d5/d11/da6/dd0/ld9 0 2024-04-22T15:49:59.498 INFO:tasks.workunit.client.1.smithi027.stdout:3/602: mknod d28/d4b/d52/db2/d95/d9c/dc1/cc9 0 2024-04-22T15:49:59.498 INFO:tasks.workunit.client.1.smithi027.stdout:4/566: write d5/f48 [126936,40213] 0 2024-04-22T15:49:59.499 INFO:tasks.workunit.client.1.smithi027.stdout:0/566: mknod d0/d2a/d6a/da6/caf 0 2024-04-22T15:49:59.501 INFO:tasks.workunit.client.1.smithi027.stdout:0/567: write d0/d34/f7a [2400530,39007] 0 2024-04-22T15:49:59.502 INFO:tasks.workunit.client.1.smithi027.stdout:0/568: stat d0/d28/f1c 0 2024-04-22T15:49:59.503 INFO:tasks.workunit.client.1.smithi027.stdout:2/549: rename dd/df/d70/d4b/d84 to dd/df/d70/d4b/d52/d9a/da4/dad 0 2024-04-22T15:49:59.505 INFO:tasks.workunit.client.1.smithi027.stdout:5/557: creat d0/d1a/d22/d26/d5d/d99/fb9 x:0 0 0 2024-04-22T15:49:59.509 INFO:tasks.workunit.client.1.smithi027.stdout:9/551: mkdir d0/dc/d12/d30/d5b/db8 0 2024-04-22T15:49:59.519 INFO:tasks.workunit.client.1.smithi027.stdout:5/558: dread d0/d1a/d22/d26/d32/f4f [0,4194304] 0 2024-04-22T15:49:59.521 INFO:tasks.workunit.client.1.smithi027.stdout:0/569: rename d0/d34/c74 to d0/d34/d4e/d85/cb0 0 2024-04-22T15:49:59.523 INFO:tasks.workunit.client.1.smithi027.stdout:2/550: read dd/d43/f55 [2933209,60519] 0 2024-04-22T15:49:59.524 INFO:tasks.workunit.client.1.smithi027.stdout:2/551: write dd/d47/f93 [874047,67485] 0 2024-04-22T15:49:59.529 INFO:tasks.workunit.client.1.smithi027.stdout:9/552: mknod d0/d51/d9d/cb9 0 2024-04-22T15:49:59.530 INFO:tasks.workunit.client.1.smithi027.stdout:9/553: chown d0/f90 86273 1 2024-04-22T15:49:59.530 INFO:tasks.workunit.client.1.smithi027.stdout:3/603: mknod d28/d4b/d52/db2/cca 0 2024-04-22T15:49:59.538 INFO:tasks.workunit.client.1.smithi027.stdout:9/554: dwrite d0/dc/d12/d30/f4f [0,4194304] 0 2024-04-22T15:49:59.540 INFO:tasks.workunit.client.1.smithi027.stdout:3/604: dwrite d28/d4b/f4c [0,4194304] 0 2024-04-22T15:49:59.542 INFO:tasks.workunit.client.1.smithi027.stdout:0/570: symlink d0/d2a/d38/lb1 0 2024-04-22T15:49:59.542 INFO:tasks.workunit.client.1.smithi027.stdout:9/555: readlink d0/dc/d12/d58/la0 0 2024-04-22T15:49:59.544 INFO:tasks.workunit.client.1.smithi027.stdout:3/605: chown d28/f49 22534 1 2024-04-22T15:49:59.547 INFO:tasks.workunit.client.1.smithi027.stdout:4/567: truncate d5/d11/dc1/fcf 1880299 0 2024-04-22T15:49:59.548 INFO:tasks.workunit.client.1.smithi027.stdout:0/571: truncate d0/d34/f80 768607 0 2024-04-22T15:49:59.549 INFO:tasks.workunit.client.1.smithi027.stdout:6/548: dwrite d5/dc/d20/f3f [0,4194304] 0 2024-04-22T15:49:59.552 INFO:tasks.workunit.client.1.smithi027.stdout:5/559: rename d0/df/cb1 to d0/df/de/d24/d1c/cba 0 2024-04-22T15:49:59.553 INFO:tasks.workunit.client.1.smithi027.stdout:2/552: mkdir dd/dae 0 2024-04-22T15:49:59.555 INFO:tasks.workunit.client.1.smithi027.stdout:3/606: rmdir d28/d5d/d62 39 2024-04-22T15:49:59.557 INFO:tasks.workunit.client.1.smithi027.stdout:4/568: dwrite d5/fd5 [0,4194304] 0 2024-04-22T15:49:59.558 INFO:tasks.workunit.client.1.smithi027.stdout:0/572: write d0/f1 [5218352,71037] 0 2024-04-22T15:49:59.558 INFO:tasks.workunit.client.1.smithi027.stdout:8/476: sync 2024-04-22T15:49:59.559 INFO:tasks.workunit.client.1.smithi027.stdout:0/573: chown d0/d9d 207848 1 2024-04-22T15:49:59.561 INFO:tasks.workunit.client.1.smithi027.stdout:8/477: readlink d1/d3/d1c/d43/l6f 0 2024-04-22T15:49:59.564 INFO:tasks.workunit.client.1.smithi027.stdout:7/503: sync 2024-04-22T15:49:59.564 INFO:tasks.workunit.client.1.smithi027.stdout:1/481: sync 2024-04-22T15:49:59.564 INFO:tasks.workunit.client.1.smithi027.stdout:6/549: creat d5/dc/d20/d2b/d34/fbf x:0 0 0 2024-04-22T15:49:59.565 INFO:tasks.workunit.client.1.smithi027.stdout:5/560: mkdir d0/df/de/dbb 0 2024-04-22T15:49:59.565 INFO:tasks.workunit.client.1.smithi027.stdout:8/478: readlink d1/d3/d1c/d43/l6f 0 2024-04-22T15:49:59.566 INFO:tasks.workunit.client.1.smithi027.stdout:4/569: read d5/d11/f28 [4034488,68675] 0 2024-04-22T15:49:59.567 INFO:tasks.workunit.client.1.smithi027.stdout:1/482: chown d2/d1d/f32 142 1 2024-04-22T15:49:59.568 INFO:tasks.workunit.client.1.smithi027.stdout:3/607: dwrite d28/d4b/f7c [0,4194304] 0 2024-04-22T15:49:59.570 INFO:tasks.workunit.client.1.smithi027.stdout:4/570: fdatasync d5/d42/f9e 0 2024-04-22T15:49:59.571 INFO:tasks.workunit.client.1.smithi027.stdout:4/571: readlink d5/de/ld1 0 2024-04-22T15:49:59.573 INFO:tasks.workunit.client.1.smithi027.stdout:2/553: readlink dd/df/d70/la9 0 2024-04-22T15:49:59.576 INFO:tasks.workunit.client.1.smithi027.stdout:9/556: mknod d0/dc/d12/d30/d5b/db8/cba 0 2024-04-22T15:49:59.577 INFO:tasks.workunit.client.1.smithi027.stdout:0/574: symlink d0/d9d/lb2 0 2024-04-22T15:49:59.582 INFO:tasks.workunit.client.1.smithi027.stdout:5/561: creat d0/df/d3d/d81/fbc x:0 0 0 2024-04-22T15:49:59.583 INFO:tasks.workunit.client.1.smithi027.stdout:7/504: mknod d0/d26/c9e 0 2024-04-22T15:49:59.584 INFO:tasks.workunit.client.1.smithi027.stdout:7/505: write d0/d11/d3a/d46/f6c [682789,71938] 0 2024-04-22T15:49:59.587 INFO:tasks.workunit.client.1.smithi027.stdout:8/479: fdatasync d1/d3/d1c/d28/f51 0 2024-04-22T15:49:59.594 INFO:tasks.workunit.client.1.smithi027.stdout:6/550: rename d5/dc/dd/d79/d7b/la4 to d5/db/d2e/d59/db5/lc0 0 2024-04-22T15:49:59.598 INFO:tasks.workunit.client.1.smithi027.stdout:1/483: mkdir d2/d5f/d64/d81/d3d/da9 0 2024-04-22T15:49:59.599 INFO:tasks.workunit.client.1.smithi027.stdout:4/572: rename d5/d45/f59 to d5/d45/fda 0 2024-04-22T15:49:59.600 INFO:tasks.workunit.client.1.smithi027.stdout:8/480: dread d1/d3/fa [0,4194304] 0 2024-04-22T15:49:59.604 INFO:tasks.workunit.client.1.smithi027.stdout:2/554: mkdir dd/df/d70/d4b/d4e/daf 0 2024-04-22T15:49:59.605 INFO:tasks.workunit.client.1.smithi027.stdout:9/557: chown d0/d64/c79 252137957 1 2024-04-22T15:49:59.637 INFO:tasks.workunit.client.1.smithi027.stdout:6/551: dwrite d5/dc/d26/f2d [0,4194304] 0 2024-04-22T15:49:59.650 INFO:tasks.workunit.client.1.smithi027.stdout:9/558: mkdir d0/dc/d12/d58/da6/dbb 0 2024-04-22T15:49:59.652 INFO:tasks.workunit.client.1.smithi027.stdout:9/559: write d0/dc/d12/fb3 [597350,35047] 0 2024-04-22T15:49:59.653 INFO:tasks.workunit.client.1.smithi027.stdout:5/562: fsync d0/df/f12 0 2024-04-22T15:49:59.655 INFO:tasks.workunit.client.1.smithi027.stdout:9/560: dread d0/dc/d12/d58/d6b/f62 [0,4194304] 0 2024-04-22T15:49:59.655 INFO:tasks.workunit.client.1.smithi027.stdout:5/563: stat d0/df/de/d24/d1c/d3e/d55/l80 0 2024-04-22T15:49:59.657 INFO:tasks.workunit.client.1.smithi027.stdout:9/561: write d0/dc/d12/d58/f59 [1365448,99370] 0 2024-04-22T15:49:59.657 INFO:tasks.workunit.client.1.smithi027.stdout:7/506: rename d0/d16/d65 to d0/d49/d9f 0 2024-04-22T15:49:59.658 INFO:tasks.workunit.client.1.smithi027.stdout:9/562: stat d0/dc/d12/d30/c35 0 2024-04-22T15:49:59.659 INFO:tasks.workunit.client.1.smithi027.stdout:1/484: mkdir d2/d36/daa 0 2024-04-22T15:49:59.661 INFO:tasks.workunit.client.1.smithi027.stdout:4/573: creat d5/d7/dc6/fdb x:0 0 0 2024-04-22T15:49:59.661 INFO:tasks.workunit.client.1.smithi027.stdout:4/574: dread - d5/d11/d1d/d7b/f92 zero size 2024-04-22T15:49:59.662 INFO:tasks.workunit.client.1.smithi027.stdout:2/555: link dd/df/d70/f45 dd/df/fb0 0 2024-04-22T15:49:59.663 INFO:tasks.workunit.client.1.smithi027.stdout:2/556: readlink dd/df/d70/d4b/l9e 0 2024-04-22T15:49:59.665 INFO:tasks.workunit.client.1.smithi027.stdout:6/552: unlink d5/db/d2e/c9b 0 2024-04-22T15:49:59.665 INFO:tasks.workunit.client.1.smithi027.stdout:6/553: stat d5/dc/dd/d79/d7b/c9e 0 2024-04-22T15:49:59.666 INFO:tasks.workunit.client.1.smithi027.stdout:6/554: chown d5/db/d2e/f62 415 1 2024-04-22T15:49:59.674 INFO:tasks.workunit.client.1.smithi027.stdout:8/481: rename d1/d3/d1c/c1f to d1/d3/d1c/d28/d42/ca5 0 2024-04-22T15:49:59.675 INFO:tasks.workunit.client.1.smithi027.stdout:3/608: write d28/d5d/d79/f7e [255350,104926] 0 2024-04-22T15:49:59.676 INFO:tasks.workunit.client.1.smithi027.stdout:0/575: dwrite d0/d34/d59/f62 [0,4194304] 0 2024-04-22T15:49:59.677 INFO:tasks.workunit.client.1.smithi027.stdout:8/482: readlink d1/d3/d1c/d32/d41/l50 0 2024-04-22T15:49:59.681 INFO:tasks.workunit.client.1.smithi027.stdout:9/563: rename d0/dc/d12/d30/d5b/db8 to d0/dc/d12/d30/d5b/dbc 0 2024-04-22T15:49:59.681 INFO:tasks.workunit.client.1.smithi027.stdout:0/576: dread d0/d28/f1c [0,4194304] 0 2024-04-22T15:49:59.682 INFO:tasks.workunit.client.1.smithi027.stdout:4/575: unlink d5/d11/da6/dd0/ld9 0 2024-04-22T15:49:59.687 INFO:tasks.workunit.client.1.smithi027.stdout:6/555: creat d5/dc/dd/d79/d7b/fc1 x:0 0 0 2024-04-22T15:49:59.694 INFO:tasks.workunit.client.1.smithi027.stdout:1/485: symlink d2/d5f/d64/d81/lab 0 2024-04-22T15:49:59.697 INFO:tasks.workunit.client.1.smithi027.stdout:9/564: rmdir d0/dc 39 2024-04-22T15:49:59.698 INFO:tasks.workunit.client.1.smithi027.stdout:3/609: dwrite d28/f38 [0,4194304] 0 2024-04-22T15:49:59.700 INFO:tasks.workunit.client.1.smithi027.stdout:5/564: truncate d0/df/de/d24/f2e 2787298 0 2024-04-22T15:49:59.701 INFO:tasks.workunit.client.1.smithi027.stdout:5/565: readlink d0/db6/lb7 0 2024-04-22T15:49:59.701 INFO:tasks.workunit.client.1.smithi027.stdout:4/576: mkdir d5/d45/ddc 0 2024-04-22T15:49:59.703 INFO:tasks.workunit.client.1.smithi027.stdout:2/557: mknod dd/df/d1b/cb1 0 2024-04-22T15:49:59.705 INFO:tasks.workunit.client.1.smithi027.stdout:6/556: fsync d5/dc/dd/d79/d7b/f4e 0 2024-04-22T15:49:59.707 INFO:tasks.workunit.client.1.smithi027.stdout:6/557: stat d5/dc/d17/d53/c64 0 2024-04-22T15:49:59.708 INFO:tasks.workunit.client.1.smithi027.stdout:6/558: write d5/dc/d26/f88 [321151,101945] 0 2024-04-22T15:49:59.709 INFO:tasks.workunit.client.1.smithi027.stdout:8/483: link d1/d3/d1c/d43/f71 d1/d3/d1c/d32/d41/d52/d61/fa6 0 2024-04-22T15:49:59.709 INFO:tasks.workunit.client.1.smithi027.stdout:4/577: dread d5/d11/f18 [0,4194304] 0 2024-04-22T15:49:59.719 INFO:tasks.workunit.client.1.smithi027.stdout:4/578: dread d5/d11/d89/f84 [0,4194304] 0 2024-04-22T15:49:59.723 INFO:tasks.workunit.client.1.smithi027.stdout:5/566: creat d0/df/de/d24/d1c/d8f/fbd x:0 0 0 2024-04-22T15:49:59.726 INFO:tasks.workunit.client.1.smithi027.stdout:7/507: creat d0/fa0 x:0 0 0 2024-04-22T15:49:59.732 INFO:tasks.workunit.client.1.smithi027.stdout:7/508: dread d0/d26/f42 [0,4194304] 0 2024-04-22T15:49:59.734 INFO:tasks.workunit.client.1.smithi027.stdout:6/559: truncate d5/dc/d20/d2b/f35 4704013 0 2024-04-22T15:49:59.746 INFO:tasks.workunit.client.1.smithi027.stdout:4/579: unlink d5/de/d2a/f49 0 2024-04-22T15:49:59.747 INFO:tasks.workunit.client.1.smithi027.stdout:9/565: write d0/f68 [836911,130378] 0 2024-04-22T15:49:59.750 INFO:tasks.workunit.client.1.smithi027.stdout:9/566: write d0/d64/f7b [4122538,128247] 0 2024-04-22T15:49:59.750 INFO:tasks.workunit.client.1.smithi027.stdout:2/558: truncate dd/df/f17 7037745 0 2024-04-22T15:49:59.757 INFO:tasks.workunit.client.1.smithi027.stdout:7/509: creat d0/d11/d1c/d31/d56/fa1 x:0 0 0 2024-04-22T15:49:59.760 INFO:tasks.workunit.client.1.smithi027.stdout:1/486: link d2/d36/f55 d2/d5f/d64/d81/d3d/d44/d9c/d9f/fac 0 2024-04-22T15:49:59.763 INFO:tasks.workunit.client.1.smithi027.stdout:6/560: dwrite d5/dc/d20/d2b/f12 [4194304,4194304] 0 2024-04-22T15:49:59.765 INFO:tasks.workunit.client.1.smithi027.stdout:0/577: getdents d0/d34 0 2024-04-22T15:49:59.766 INFO:tasks.workunit.client.1.smithi027.stdout:6/561: truncate d5/db/fa9 937635 0 2024-04-22T15:49:59.766 INFO:tasks.workunit.client.1.smithi027.stdout:0/578: fdatasync d0/d2a/d38/f6b 0 2024-04-22T15:49:59.770 INFO:tasks.workunit.client.1.smithi027.stdout:4/580: rename d5/d45/d47/f62 to d5/d11/d1d/d7b/fdd 0 2024-04-22T15:49:59.771 INFO:tasks.workunit.client.1.smithi027.stdout:4/581: chown d5/d45/d47/c87 14507104 1 2024-04-22T15:49:59.863 INFO:tasks.workunit.client.1.smithi027.stdout:8/484: link d1/c2a d1/d3/d1c/d43/d82/ca7 0 2024-04-22T15:49:59.865 INFO:tasks.workunit.client.1.smithi027.stdout:8/485: dread d1/d3/d1c/f34 [0,4194304] 0 2024-04-22T15:49:59.866 INFO:tasks.workunit.client.1.smithi027.stdout:7/510: creat d0/d49/fa2 x:0 0 0 2024-04-22T15:49:59.868 INFO:tasks.workunit.client.1.smithi027.stdout:1/487: mknod d2/d5f/d64/d81/d3d/d82/d85/cad 0 2024-04-22T15:49:59.871 INFO:tasks.workunit.client.1.smithi027.stdout:3/610: getdents d28/d4b/d52/d94/dae 0 2024-04-22T15:49:59.876 INFO:tasks.workunit.client.1.smithi027.stdout:3/611: dread d28/d4b/f7c [0,4194304] 0 2024-04-22T15:49:59.879 INFO:tasks.workunit.client.1.smithi027.stdout:5/567: creat d0/df/de/d24/d1c/d3e/fbe x:0 0 0 2024-04-22T15:49:59.880 INFO:tasks.workunit.client.1.smithi027.stdout:4/582: creat d5/d11/d1d/d54/d8c/fde x:0 0 0 2024-04-22T15:49:59.881 INFO:tasks.workunit.client.1.smithi027.stdout:4/583: readlink d5/de/dc5/lcc 0 2024-04-22T15:49:59.882 INFO:tasks.workunit.client.1.smithi027.stdout:9/567: creat d0/dc/d12/d58/d6b/d6e/fbd x:0 0 0 2024-04-22T15:49:59.885 INFO:tasks.workunit.client.1.smithi027.stdout:1/488: mknod d2/d5f/d64/d81/d69/cae 0 2024-04-22T15:49:59.893 INFO:tasks.workunit.client.1.smithi027.stdout:8/486: symlink d1/d3/d1c/d56/la8 0 2024-04-22T15:49:59.895 INFO:tasks.workunit.client.1.smithi027.stdout:7/511: mkdir d0/d4c/d8b/d92/da3 0 2024-04-22T15:49:59.902 INFO:tasks.workunit.client.1.smithi027.stdout:4/584: symlink d5/d45/d5a/dcd/ldf 0 2024-04-22T15:49:59.902 INFO:tasks.workunit.client.1.smithi027.stdout:2/559: link dd/df/d70/d4b/d4e/c4f dd/df/d70/cb2 0 2024-04-22T15:49:59.902 INFO:tasks.workunit.client.1.smithi027.stdout:9/568: dread d0/dc/d12/d30/f32 [0,4194304] 0 2024-04-22T15:49:59.904 INFO:tasks.workunit.client.1.smithi027.stdout:9/569: readlink d0/dc/d12/d58/d6b/d6e/l56 0 2024-04-22T15:49:59.905 INFO:tasks.workunit.client.1.smithi027.stdout:9/570: fdatasync d0/dc/d12/f94 0 2024-04-22T15:49:59.905 INFO:tasks.workunit.client.1.smithi027.stdout:8/487: truncate d1/d3/d1c/d43/d5f/f9e 3981670 0 2024-04-22T15:49:59.907 INFO:tasks.workunit.client.1.smithi027.stdout:3/612: creat d28/d5d/fcb x:0 0 0 2024-04-22T15:49:59.908 INFO:tasks.workunit.client.1.smithi027.stdout:8/488: dread d1/d3/d1c/f34 [0,4194304] 0 2024-04-22T15:49:59.908 INFO:tasks.workunit.client.1.smithi027.stdout:3/613: readlink d28/d4b/l5a 0 2024-04-22T15:49:59.909 INFO:tasks.workunit.client.1.smithi027.stdout:4/585: creat d5/de/da9/fe0 x:0 0 0 2024-04-22T15:49:59.911 INFO:tasks.workunit.client.1.smithi027.stdout:2/560: fsync dd/df/d70/f61 0 2024-04-22T15:49:59.911 INFO:tasks.workunit.client.1.smithi027.stdout:4/586: chown d5/de/d2a/d58/faf 114932 1 2024-04-22T15:49:59.912 INFO:tasks.workunit.client.1.smithi027.stdout:4/587: readlink d5/de/dc5/lcc 0 2024-04-22T15:49:59.919 INFO:tasks.workunit.client.1.smithi027.stdout:1/489: link d2/d5f/d64/d81/d3d/d44/d9c/d9f/la0 d2/d5f/d64/d81/d69/d7d/laf 0 2024-04-22T15:49:59.920 INFO:tasks.workunit.client.1.smithi027.stdout:1/490: stat d2/f35 0 2024-04-22T15:49:59.923 INFO:tasks.workunit.client.1.smithi027.stdout:2/561: rename dd/d47/f93 to dd/df/d1b/d38/d60/d75/fb3 0 2024-04-22T15:49:59.925 INFO:tasks.workunit.client.1.smithi027.stdout:2/562: readlink dd/df/d70/d4b/d52/d9a/da4/dad/d8d/l9b 0 2024-04-22T15:49:59.932 INFO:tasks.workunit.client.1.smithi027.stdout:3/614: link d28/d4b/fad d28/d4b/d52/d94/dae/fcc 0 2024-04-22T15:49:59.936 INFO:tasks.workunit.client.1.smithi027.stdout:3/615: write d28/d69/f85 [4552695,37935] 0 2024-04-22T15:49:59.937 INFO:tasks.workunit.client.1.smithi027.stdout:1/491: rename d2/d1d/c61 to d2/d36/d74/cb0 0 2024-04-22T15:49:59.938 INFO:tasks.workunit.client.1.smithi027.stdout:3/616: truncate d28/d4b/fad 32604 0 2024-04-22T15:49:59.939 INFO:tasks.workunit.client.1.smithi027.stdout:2/563: creat dd/d43/fb4 x:0 0 0 2024-04-22T15:49:59.941 INFO:tasks.workunit.client.1.smithi027.stdout:2/564: write dd/df/d1b/d1f/f82 [523911,12358] 0 2024-04-22T15:49:59.941 INFO:tasks.workunit.client.1.smithi027.stdout:3/617: creat d28/d5d/d79/fcd x:0 0 0 2024-04-22T15:49:59.947 INFO:tasks.workunit.client.1.smithi027.stdout:6/562: write d5/dc/d20/d2b/f35 [988819,122382] 0 2024-04-22T15:49:59.957 INFO:tasks.workunit.client.1.smithi027.stdout:6/563: getdents d5/dc/d20 0 2024-04-22T15:49:59.958 INFO:tasks.workunit.client.1.smithi027.stdout:6/564: chown d5/f89 676520 1 2024-04-22T15:49:59.961 INFO:tasks.workunit.client.1.smithi027.stdout:6/565: fdatasync d5/dc/dd/d79/d7b/fc1 0 2024-04-22T15:49:59.964 INFO:tasks.workunit.client.1.smithi027.stdout:8/489: write d1/d3/d36/f85 [3931745,66863] 0 2024-04-22T15:49:59.965 INFO:tasks.workunit.client.1.smithi027.stdout:7/512: dwrite d0/d11/d3a/f45 [0,4194304] 0 2024-04-22T15:49:59.967 INFO:tasks.workunit.client.1.smithi027.stdout:7/513: write d0/d16/f81 [3941232,63170] 0 2024-04-22T15:49:59.970 INFO:tasks.workunit.client.1.smithi027.stdout:9/571: truncate d0/f36 528696 0 2024-04-22T15:49:59.973 INFO:tasks.workunit.client.1.smithi027.stdout:8/490: mkdir d1/d3/d36/d88/da9 0 2024-04-22T15:49:59.984 INFO:tasks.workunit.client.1.smithi027.stdout:9/572: rename d0/dc/de/f49 to d0/dc/d12/d30/d5b/d91/da9/fbe 0 2024-04-22T15:49:59.989 INFO:tasks.workunit.client.1.smithi027.stdout:8/491: fsync d1/d3/d1c/d32/d41/d52/d61/fa6 0 2024-04-22T15:49:59.989 INFO:tasks.workunit.client.1.smithi027.stdout:8/492: fsync d1/d3/f9b 0 2024-04-22T15:49:59.996 INFO:tasks.workunit.client.1.smithi027.stdout:9/573: dwrite d0/dc/d12/d30/d5b/f65 [0,4194304] 0 2024-04-22T15:50:00.001 INFO:tasks.workunit.client.1.smithi027.stdout:1/492: write d2/d36/f55 [4959990,52383] 0 2024-04-22T15:50:00.002 INFO:tasks.workunit.client.1.smithi027.stdout:9/574: dread d0/dc/de/f3a [0,4194304] 0 2024-04-22T15:50:00.003 INFO:tasks.workunit.client.1.smithi027.stdout:3/618: write d28/f6c [644110,112851] 0 2024-04-22T15:50:00.007 INFO:tasks.workunit.client.1.smithi027.stdout:3/619: read f1d [1727706,76323] 0 2024-04-22T15:50:00.018 INFO:tasks.workunit.client.1.smithi027.stdout:6/566: dwrite d5/dc/d17/f39 [0,4194304] 0 2024-04-22T15:50:00.019 INFO:tasks.workunit.client.1.smithi027.stdout:6/567: stat d5/l7 0 2024-04-22T15:50:00.020 INFO:tasks.workunit.client.1.smithi027.stdout:6/568: fsync d5/dc/d20/d2b/d34/fb0 0 2024-04-22T15:50:00.022 INFO:tasks.workunit.client.1.smithi027.stdout:7/514: link d0/d11/d1c/f7e d0/d11/d1c/d31/d8a/fa4 0 2024-04-22T15:50:00.041 INFO:tasks.workunit.client.1.smithi027.stdout:6/569: dread d5/dc/dd/fa2 [0,4194304] 0 2024-04-22T15:50:00.052 INFO:tasks.workunit.client.1.smithi027.stdout:8/493: creat d1/d1e/d47/d9d/faa x:0 0 0 2024-04-22T15:50:00.058 INFO:tasks.workunit.client.1.smithi027.stdout:9/575: mkdir d0/dc/de/db5/dbf 0 2024-04-22T15:50:00.061 INFO:tasks.workunit.client.1.smithi027.stdout:6/570: mknod d5/dc/d17/d8a/cc2 0 2024-04-22T15:50:00.062 INFO:tasks.workunit.client.1.smithi027.stdout:6/571: dread d5/db/fa9 [0,4194304] 0 2024-04-22T15:50:00.066 INFO:tasks.workunit.client.1.smithi027.stdout:3/620: mkdir d28/d5d/db0/dce 0 2024-04-22T15:50:00.068 INFO:tasks.workunit.client.1.smithi027.stdout:7/515: symlink d0/d11/d3a/d46/d94/d5a/la5 0 2024-04-22T15:50:00.071 INFO:tasks.workunit.client.1.smithi027.stdout:7/516: truncate d0/d49/d9f/f9b 591627 0 2024-04-22T15:50:00.073 INFO:tasks.workunit.client.1.smithi027.stdout:8/494: truncate d1/d3/f40 2921507 0 2024-04-22T15:50:00.073 INFO:tasks.workunit.client.1.smithi027.stdout:3/621: dwrite d28/d4b/d52/f61 [0,4194304] 0 2024-04-22T15:50:00.074 INFO:tasks.workunit.client.1.smithi027.stdout:3/622: readlink d28/d5d/d79/l98 0 2024-04-22T15:50:00.075 INFO:tasks.workunit.client.1.smithi027.stdout:9/576: link d0/dc/d12/d30/d5b/d91/da9/fbe d0/d8b/fc0 0 2024-04-22T15:50:00.081 INFO:tasks.workunit.client.1.smithi027.stdout:7/517: symlink d0/d3f/la6 0 2024-04-22T15:50:00.082 INFO:tasks.workunit.client.1.smithi027.stdout:8/495: dwrite d1/d3/d1c/d28/d96/fa0 [0,4194304] 0 2024-04-22T15:50:00.084 INFO:tasks.workunit.client.1.smithi027.stdout:7/518: readlink d0/d11/d1c/d31/l43 0 2024-04-22T15:50:00.084 INFO:tasks.workunit.client.1.smithi027.stdout:3/623: truncate d28/d5d/d62/faa 927023 0 2024-04-22T15:50:00.085 INFO:tasks.workunit.client.1.smithi027.stdout:6/572: link d5/dc/l21 d5/db/da5/lc3 0 2024-04-22T15:50:00.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:59 smithi027 ceph-mon[34922]: [22/Apr/2024:15:49:58] ENGINE Serving on http://172.21.15.16:8765 2024-04-22T15:50:00.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:59 smithi027 ceph-mon[34922]: [22/Apr/2024:15:49:58] ENGINE Bus STARTED 2024-04-22T15:50:00.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:59 smithi027 ceph-mon[34922]: mgrmap e28: smithi016.rnsnnr(active, since 1.0471s) 2024-04-22T15:50:00.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:49:59 smithi027 ceph-mon[34922]: pgmap v3: 65 pgs: 65 active+clean; 2.9 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail 2024-04-22T15:50:00.088 INFO:tasks.workunit.client.1.smithi027.stdout:9/577: rename d0/dc/de/l28 to d0/dc/d12/d58/d6b/d6e/lc1 0 2024-04-22T15:50:00.098 INFO:tasks.workunit.client.1.smithi027.stdout:7/519: unlink d0/d49/fa2 0 2024-04-22T15:50:00.101 INFO:tasks.workunit.client.1.smithi027.stdout:6/573: mknod d5/db/d2e/d59/db5/cc4 0 2024-04-22T15:50:00.112 INFO:tasks.workunit.client.1.smithi027.stdout:3/624: dwrite d28/d4b/fad [0,4194304] 0 2024-04-22T15:50:00.116 INFO:tasks.workunit.client.1.smithi027.stdout:9/578: write d0/dc/de/f41 [1996819,39323] 0 2024-04-22T15:50:00.118 INFO:tasks.workunit.client.1.smithi027.stdout:7/520: unlink d0/d16/f79 0 2024-04-22T15:50:00.120 INFO:tasks.workunit.client.1.smithi027.stdout:8/496: truncate d1/f7 992808 0 2024-04-22T15:50:00.126 INFO:tasks.workunit.client.1.smithi027.stdout:3/625: write d28/d5d/f96 [676149,19708] 0 2024-04-22T15:50:00.126 INFO:tasks.workunit.client.1.smithi027.stdout:8/497: dwrite d1/f4 [0,4194304] 0 2024-04-22T15:50:00.132 INFO:tasks.workunit.client.1.smithi027.stdout:9/579: fdatasync d0/dc/d12/f23 0 2024-04-22T15:50:00.134 INFO:tasks.workunit.client.1.smithi027.stdout:7/521: creat d0/d11/d3a/fa7 x:0 0 0 2024-04-22T15:50:00.134 INFO:tasks.workunit.client.1.smithi027.stdout:6/574: mkdir d5/dc/dae/dc5 0 2024-04-22T15:50:00.139 INFO:tasks.workunit.client.1.smithi027.stdout:3/626: fsync d28/f34 0 2024-04-22T15:50:00.142 INFO:tasks.workunit.client.1.smithi027.stdout:8/498: creat d1/d3/d1c/d32/d41/fab x:0 0 0 2024-04-22T15:50:00.152 INFO:tasks.workunit.client.1.smithi027.stdout:7/522: rename d0/d16 to d0/d4c/d8b/d92/da3/da8 0 2024-04-22T15:50:00.153 INFO:tasks.workunit.client.1.smithi027.stdout:9/580: dread d0/dc/f72 [0,4194304] 0 2024-04-22T15:50:00.154 INFO:tasks.workunit.client.1.smithi027.stdout:9/581: chown d0/dc/d12/d58/d6b/d6e/f84 17338 1 2024-04-22T15:50:00.158 INFO:tasks.workunit.client.1.smithi027.stdout:3/627: dread d28/f37 [0,4194304] 0 2024-04-22T15:50:00.159 INFO:tasks.workunit.client.1.smithi027.stdout:3/628: chown d28/d5d/d62/c76 7 1 2024-04-22T15:50:00.165 INFO:tasks.workunit.client.1.smithi027.stdout:3/629: dread d28/d5d/d79/fb8 [4194304,4194304] 0 2024-04-22T15:50:00.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:59 smithi016 ceph-mon[26620]: [22/Apr/2024:15:49:58] ENGINE Serving on http://172.21.15.16:8765 2024-04-22T15:50:00.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:59 smithi016 ceph-mon[26620]: [22/Apr/2024:15:49:58] ENGINE Bus STARTED 2024-04-22T15:50:00.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:59 smithi016 ceph-mon[26620]: mgrmap e28: smithi016.rnsnnr(active, since 1.0471s) 2024-04-22T15:50:00.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:49:59 smithi016 ceph-mon[26620]: pgmap v3: 65 pgs: 65 active+clean; 2.9 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail 2024-04-22T15:50:00.171 INFO:tasks.workunit.client.1.smithi027.stdout:6/575: link d5/dc/d26/f2d d5/d30/dbe/fc6 0 2024-04-22T15:50:00.174 INFO:tasks.workunit.client.1.smithi027.stdout:7/523: dread d0/d4c/d8b/d92/da3/da8/f4a [4194304,4194304] 0 2024-04-22T15:50:00.174 INFO:tasks.workunit.client.1.smithi027.stdout:9/582: dread d0/dc/d12/d58/d6b/d6e/f31 [0,4194304] 0 2024-04-22T15:50:00.174 INFO:tasks.workunit.client.1.smithi027.stdout:3/630: dwrite d28/d4b/f59 [0,4194304] 0 2024-04-22T15:50:00.185 INFO:tasks.workunit.client.1.smithi027.stdout:7/524: read d0/d11/d1c/d2c/f40 [1609180,81957] 0 2024-04-22T15:50:00.185 INFO:tasks.workunit.client.1.smithi027.stdout:9/583: creat d0/dc/de/fc2 x:0 0 0 2024-04-22T15:50:00.189 INFO:tasks.workunit.client.1.smithi027.stdout:6/576: mkdir d5/d30/dc7 0 2024-04-22T15:50:00.198 INFO:tasks.workunit.client.1.smithi027.stdout:8/499: dwrite d1/d3/d1c/d43/d5f/f9e [4194304,4194304] 0 2024-04-22T15:50:00.199 INFO:tasks.workunit.client.1.smithi027.stdout:8/500: dread - d1/d3/d1c/d43/d5f/fa4 zero size 2024-04-22T15:50:00.201 INFO:tasks.workunit.client.1.smithi027.stdout:6/577: link d5/db/d2e/d59/fb1 d5/dc/fc8 0 2024-04-22T15:50:00.202 INFO:tasks.workunit.client.1.smithi027.stdout:3/631: write d28/f4e [195942,106920] 0 2024-04-22T15:50:00.206 INFO:tasks.workunit.client.1.smithi027.stdout:8/501: rmdir d1/d3/d1c/d32/d41/d52/d61/d8f 39 2024-04-22T15:50:00.209 INFO:tasks.workunit.client.1.smithi027.stdout:6/578: dread d5/dc/d20/f3f [0,4194304] 0 2024-04-22T15:50:00.211 INFO:tasks.workunit.client.1.smithi027.stdout:6/579: fsync d5/dc/dd/d79/f8b 0 2024-04-22T15:50:00.212 INFO:tasks.workunit.client.1.smithi027.stdout:3/632: mkdir d28/d5d/d62/dcf 0 2024-04-22T15:50:00.212 INFO:tasks.workunit.client.1.smithi027.stdout:7/525: dwrite d0/d3f/f5b [0,4194304] 0 2024-04-22T15:50:00.213 INFO:tasks.workunit.client.1.smithi027.stdout:3/633: chown d28/d4b/d52/d94/dae 1918 1 2024-04-22T15:50:00.215 INFO:tasks.workunit.client.1.smithi027.stdout:7/526: truncate d0/d11/d1c/d31/f70 604522 0 2024-04-22T15:50:00.216 INFO:tasks.workunit.client.1.smithi027.stdout:8/502: fsync d1/f18 0 2024-04-22T15:50:00.216 INFO:tasks.workunit.client.1.smithi027.stdout:3/634: dread - d28/d4b/faf zero size 2024-04-22T15:50:00.217 INFO:tasks.workunit.client.1.smithi027.stdout:7/527: stat d0/d11/d1c/d31/f70 0 2024-04-22T15:50:00.222 INFO:tasks.workunit.client.1.smithi027.stdout:3/635: unlink l17 0 2024-04-22T15:50:00.223 INFO:tasks.workunit.client.1.smithi027.stdout:8/503: unlink d1/d3/c65 0 2024-04-22T15:50:00.226 INFO:tasks.workunit.client.1.smithi027.stdout:3/636: symlink d28/d5d/db0/ld0 0 2024-04-22T15:50:00.229 INFO:tasks.workunit.client.1.smithi027.stdout:3/637: fdatasync d28/d4b/d52/d94/dae/fcc 0 2024-04-22T15:50:00.230 INFO:tasks.workunit.client.1.smithi027.stdout:7/528: fdatasync d0/d11/d1c/d2c/f38 0 2024-04-22T15:50:00.234 INFO:tasks.workunit.client.1.smithi027.stdout:3/638: truncate d28/d4b/d52/d94/fbb 244324 0 2024-04-22T15:50:00.235 INFO:tasks.workunit.client.1.smithi027.stdout:6/580: dread d5/f75 [0,4194304] 0 2024-04-22T15:50:00.236 INFO:tasks.workunit.client.1.smithi027.stdout:8/504: dwrite d1/d1e/d47/d53/f78 [0,4194304] 0 2024-04-22T15:50:00.238 INFO:tasks.workunit.client.1.smithi027.stdout:6/581: write d5/dc/d26/d3d/f7c [5116947,93046] 0 2024-04-22T15:50:00.241 INFO:tasks.workunit.client.1.smithi027.stdout:7/529: dwrite d0/f30 [0,4194304] 0 2024-04-22T15:50:00.243 INFO:tasks.workunit.client.1.smithi027.stdout:7/530: chown d0/d4c/d4e/l8f 1567758789 1 2024-04-22T15:50:00.246 INFO:tasks.workunit.client.1.smithi027.stdout:7/531: symlink d0/d4c/d8b/la9 0 2024-04-22T15:50:00.249 INFO:tasks.workunit.client.1.smithi027.stdout:6/582: mkdir d5/dc/d20/da3/dc9 0 2024-04-22T15:50:00.252 INFO:tasks.workunit.client.1.smithi027.stdout:7/532: unlink d0/d11/d3a/d46/d94/l6a 0 2024-04-22T15:50:00.254 INFO:tasks.workunit.client.1.smithi027.stdout:6/583: rmdir d5/dc/d17/d8a 39 2024-04-22T15:50:00.256 INFO:tasks.workunit.client.1.smithi027.stdout:3/639: dwrite f16 [0,4194304] 0 2024-04-22T15:50:00.260 INFO:tasks.workunit.client.1.smithi027.stdout:6/584: rmdir d5/dc/dd/d52 39 2024-04-22T15:50:00.262 INFO:tasks.workunit.client.1.smithi027.stdout:6/585: chown d5/dc/l5f 878 1 2024-04-22T15:50:00.266 INFO:tasks.workunit.client.1.smithi027.stdout:7/533: dwrite d0/d3f/f48 [0,4194304] 0 2024-04-22T15:50:00.268 INFO:tasks.workunit.client.1.smithi027.stdout:7/534: creat d0/d11/d1c/d31/faa x:0 0 0 2024-04-22T15:50:00.269 INFO:tasks.workunit.client.1.smithi027.stdout:7/535: readlink d0/l20 0 2024-04-22T15:50:00.271 INFO:tasks.workunit.client.1.smithi027.stdout:6/586: fdatasync d5/dc/dd/d79/d7b/f4e 0 2024-04-22T15:50:00.273 INFO:tasks.workunit.client.1.smithi027.stdout:6/587: write d5/dc/d20/d2b/d34/f65 [1628209,29718] 0 2024-04-22T15:50:00.274 INFO:tasks.workunit.client.1.smithi027.stdout:6/588: chown d5/dc/d20/d2b/d34 554385 1 2024-04-22T15:50:00.275 INFO:tasks.workunit.client.1.smithi027.stdout:6/589: chown d5/d30 21704723 1 2024-04-22T15:50:00.276 INFO:tasks.workunit.client.1.smithi027.stdout:3/640: truncate d28/f45 3936504 0 2024-04-22T15:50:00.281 INFO:tasks.workunit.client.1.smithi027.stdout:6/590: rename d5/dc/d17/d53/f91 to d5/d30/dbe/fca 0 2024-04-22T15:50:00.285 INFO:tasks.workunit.client.1.smithi027.stdout:3/641: dwrite d28/d5d/d62/f8f [0,4194304] 0 2024-04-22T15:50:00.287 INFO:tasks.workunit.client.1.smithi027.stdout:3/642: dread d28/d5d/db0/fbd [0,4194304] 0 2024-04-22T15:50:00.294 INFO:tasks.workunit.client.1.smithi027.stdout:7/536: dwrite d0/d11/d1c/d31/f47 [0,4194304] 0 2024-04-22T15:50:00.296 INFO:tasks.workunit.client.1.smithi027.stdout:7/537: write d0/d11/f4d [4316346,128022] 0 2024-04-22T15:50:00.300 INFO:tasks.workunit.client.1.smithi027.stdout:3/643: rmdir d28/d69 39 2024-04-22T15:50:00.302 INFO:tasks.workunit.client.1.smithi027.stdout:3/644: truncate d28/f4e 1053838 0 2024-04-22T15:50:00.305 INFO:tasks.workunit.client.1.smithi027.stdout:7/538: rmdir d0/d11/d1c/d31/d56 39 2024-04-22T15:50:00.310 INFO:tasks.workunit.client.1.smithi027.stdout:3/645: unlink d28/d4b/d52/f54 0 2024-04-22T15:50:00.311 INFO:tasks.workunit.client.1.smithi027.stdout:3/646: dread - d28/d4b/d52/db2/fa6 zero size 2024-04-22T15:50:00.314 INFO:tasks.workunit.client.1.smithi027.stdout:7/539: mknod d0/d11/d1c/d2c/d93/cab 0 2024-04-22T15:50:00.315 INFO:tasks.workunit.client.1.smithi027.stdout:7/540: chown d0/d4c/d4e/f67 910 1 2024-04-22T15:50:00.319 INFO:tasks.workunit.client.1.smithi027.stdout:6/591: truncate d5/dc/dd/d79/d7b/d5d/f6c 1175004 0 2024-04-22T15:50:00.325 INFO:tasks.workunit.client.1.smithi027.stdout:7/541: dwrite d0/d11/d1c/d2c/f55 [0,4194304] 0 2024-04-22T15:50:00.326 INFO:tasks.workunit.client.1.smithi027.stdout:7/542: dread - d0/fa0 zero size 2024-04-22T15:50:00.328 INFO:tasks.workunit.client.1.smithi027.stdout:6/592: symlink d5/dc/d20/da3/dc9/lcb 0 2024-04-22T15:50:00.329 INFO:tasks.workunit.client.1.smithi027.stdout:3/647: truncate d28/d4b/d52/f61 2833426 0 2024-04-22T15:50:00.335 INFO:tasks.workunit.client.1.smithi027.stdout:7/543: creat d0/d26/fac x:0 0 0 2024-04-22T15:50:00.336 INFO:tasks.workunit.client.1.smithi027.stdout:7/544: chown d0/d49/d9f 27057289 1 2024-04-22T15:50:00.337 INFO:tasks.workunit.client.1.smithi027.stdout:6/593: creat d5/dc/d20/da3/fcc x:0 0 0 2024-04-22T15:50:00.341 INFO:tasks.workunit.client.1.smithi027.stdout:7/545: rename d0/d3f/f75 to d0/d4c/d4e/fad 0 2024-04-22T15:50:00.343 INFO:tasks.workunit.client.1.smithi027.stdout:6/594: rmdir d5/dc/d26 39 2024-04-22T15:50:00.347 INFO:tasks.workunit.client.1.smithi027.stdout:7/546: creat d0/d4c/d8b/fae x:0 0 0 2024-04-22T15:50:00.351 INFO:tasks.workunit.client.1.smithi027.stdout:7/547: rename d0/d11/d3a/d46/f6c to d0/d11/d1c/d31/d62/faf 0 2024-04-22T15:50:00.353 INFO:tasks.workunit.client.1.smithi027.stdout:6/595: dwrite d5/f75 [0,4194304] 0 2024-04-22T15:50:00.448 INFO:tasks.workunit.client.1.smithi027.stdout:6/596: unlink d5/db/d2e/d59/db5/cc4 0 2024-04-22T15:50:00.449 INFO:tasks.workunit.client.1.smithi027.stdout:6/597: dread - d5/dc/dd/d79/d7b/fc1 zero size 2024-04-22T15:50:00.450 INFO:tasks.workunit.client.1.smithi027.stdout:7/548: mknod d0/d11/d1c/d2c/cb0 0 2024-04-22T15:50:00.452 INFO:tasks.workunit.client.1.smithi027.stdout:6/598: mkdir d5/db/dcd 0 2024-04-22T15:50:00.453 INFO:tasks.workunit.client.1.smithi027.stdout:6/599: stat d5/d30/dc7 0 2024-04-22T15:50:00.454 INFO:tasks.workunit.client.1.smithi027.stdout:7/549: dread d0/d11/d1c/d31/d62/faf [0,4194304] 0 2024-04-22T15:50:00.456 INFO:tasks.workunit.client.1.smithi027.stdout:6/600: creat d5/d30/d40/fce x:0 0 0 2024-04-22T15:50:00.457 INFO:tasks.workunit.client.1.smithi027.stdout:7/550: creat d0/d11/d1c/fb1 x:0 0 0 2024-04-22T15:50:00.461 INFO:tasks.workunit.client.1.smithi027.stdout:7/551: rename d0/d26 to d0/d4c/d8b/d72/db2 0 2024-04-22T15:50:00.463 INFO:tasks.workunit.client.1.smithi027.stdout:6/601: link d5/dc/c8f d5/dc/d17/ccf 0 2024-04-22T15:50:00.465 INFO:tasks.workunit.client.1.smithi027.stdout:6/602: rmdir d5/db/da5 39 2024-04-22T15:50:00.480 INFO:tasks.workunit.client.1.smithi027.stdout:6/603: dread d5/dc/dd/d79/d7b/d5d/f70 [0,4194304] 0 2024-04-22T15:50:00.480 INFO:tasks.workunit.client.1.smithi027.stdout:6/604: chown d5/d30/d40/l4c 1757802 1 2024-04-22T15:50:00.481 INFO:tasks.workunit.client.1.smithi027.stdout:6/605: chown d5/dc/d99 267142799 1 2024-04-22T15:50:00.490 INFO:tasks.workunit.client.1.smithi027.stdout:0/579: sync 2024-04-22T15:50:00.493 INFO:tasks.workunit.client.1.smithi027.stdout:6/606: dread d5/db/d2e/ff [0,4194304] 0 2024-04-22T15:50:00.603 INFO:tasks.workunit.client.1.smithi027.stdout:6/607: rename d5/dc/d20/d2b to d5/dc/dd/dd0 0 2024-04-22T15:50:00.607 INFO:tasks.workunit.client.1.smithi027.stdout:6/608: symlink d5/dc/dd/d52/ld1 0 2024-04-22T15:50:00.608 INFO:tasks.workunit.client.1.smithi027.stdout:6/609: write d5/dc/dd/dd0/d34/f65 [4235382,81155] 0 2024-04-22T15:50:00.724 INFO:tasks.workunit.client.1.smithi027.stdout:6/610: creat d5/d30/d54/fd2 x:0 0 0 2024-04-22T15:50:00.731 INFO:tasks.workunit.client.1.smithi027.stdout:6/611: link d5/dc/l82 d5/dc/d17/d8a/ld3 0 2024-04-22T15:50:00.787 INFO:tasks.workunit.client.1.smithi027.stdout:6/612: link d5/d30/d40/la7 d5/dc/dd/ld4 0 2024-04-22T15:50:00.790 INFO:tasks.workunit.client.1.smithi027.stdout:6/613: creat d5/dc/dd/dd0/d34/fd5 x:0 0 0 2024-04-22T15:50:00.798 INFO:tasks.workunit.client.1.smithi027.stdout:5/568: sync 2024-04-22T15:50:00.800 INFO:tasks.workunit.client.1.smithi027.stdout:5/569: write d0/df/d3d/f8e [741310,2074] 0 2024-04-22T15:50:00.819 INFO:tasks.workunit.client.1.smithi027.stdout:6/614: dwrite d5/dc/dd/d79/d7b/f4e [0,4194304] 0 2024-04-22T15:50:00.824 INFO:tasks.workunit.client.1.smithi027.stdout:6/615: mkdir d5/db/d2e/d59/dd6 0 2024-04-22T15:50:00.834 INFO:tasks.workunit.client.1.smithi027.stdout:5/570: dread d0/d1a/d22/d26/d5d/f8b [0,4194304] 0 2024-04-22T15:50:00.835 INFO:tasks.workunit.client.1.smithi027.stdout:5/571: truncate d0/d1a/f82 4382367 0 2024-04-22T15:50:00.838 INFO:tasks.workunit.client.1.smithi027.stdout:5/572: fsync d0/d1a/d22/d26/d32/f4f 0 2024-04-22T15:50:00.845 INFO:tasks.workunit.client.1.smithi027.stdout:6/616: dwrite d5/dc/fc8 [0,4194304] 0 2024-04-22T15:50:00.846 INFO:tasks.workunit.client.1.smithi027.stdout:6/617: fsync d5/dc/dd/d79/f8b 0 2024-04-22T15:50:00.848 INFO:tasks.workunit.client.1.smithi027.stdout:5/573: truncate d0/df/d10/f14 3799031 0 2024-04-22T15:50:00.849 INFO:tasks.workunit.client.1.smithi027.stdout:5/574: chown d0/df/d10/f33 62890 1 2024-04-22T15:50:00.851 INFO:tasks.workunit.client.1.smithi027.stdout:6/618: unlink c1 0 2024-04-22T15:50:00.854 INFO:tasks.workunit.client.1.smithi027.stdout:5/575: truncate d0/d1a/d22/d26/d32/d88/d90/f91 508154 0 2024-04-22T15:50:00.856 INFO:tasks.workunit.client.1.smithi027.stdout:5/576: write d0/d1a/d22/d26/d32/d88/d90/f9f [1839977,23352] 0 2024-04-22T15:50:00.858 INFO:tasks.workunit.client.1.smithi027.stdout:5/577: chown d0/df/de/d24/d1c/c50 0 1 2024-04-22T15:50:00.859 INFO:tasks.workunit.client.1.smithi027.stdout:6/619: dread d5/dc/d26/d3d/f7c [0,4194304] 0 2024-04-22T15:50:00.860 INFO:tasks.workunit.client.1.smithi027.stdout:6/620: chown d5/dc/dd/d52 110325 1 2024-04-22T15:50:00.862 INFO:tasks.workunit.client.1.smithi027.stdout:5/578: rename d0/df/de/d24/d83 to d0/d97/dbf 0 2024-04-22T15:50:00.866 INFO:tasks.workunit.client.1.smithi027.stdout:5/579: truncate d0/d1a/d22/d72/d86/f75 694697 0 2024-04-22T15:50:00.867 INFO:tasks.workunit.client.1.smithi027.stdout:6/621: unlink d5/dc/c8f 0 2024-04-22T15:50:00.868 INFO:tasks.workunit.client.1.smithi027.stdout:5/580: write d0/df/de/d24/d1c/d3e/fbe [142894,16868] 0 2024-04-22T15:50:00.877 INFO:tasks.workunit.client.1.smithi027.stdout:5/581: dread d0/d1a/d22/d26/d32/d88/d90/f9f [0,4194304] 0 2024-04-22T15:50:00.882 INFO:tasks.workunit.client.1.smithi027.stdout:5/582: dread d0/d1a/d22/d26/d32/f4f [0,4194304] 0 2024-04-22T15:50:00.885 INFO:tasks.workunit.client.1.smithi027.stdout:5/583: dread d0/d1a/d22/d26/d32/d88/d90/f9f [0,4194304] 0 2024-04-22T15:50:00.887 INFO:tasks.workunit.client.1.smithi027.stdout:6/622: dread d5/dc/dd/d52/f7e [0,4194304] 0 2024-04-22T15:50:00.888 INFO:tasks.workunit.client.1.smithi027.stdout:6/623: chown d5/dc/dd/dd0/d34/fd5 153039063 1 2024-04-22T15:50:00.890 INFO:tasks.workunit.client.1.smithi027.stdout:6/624: write d5/dc/dd/f16 [1520434,65176] 0 2024-04-22T15:50:00.899 INFO:tasks.workunit.client.1.smithi027.stdout:6/625: chown d5/dc/d20/da3/dc9/lcb 2871 1 2024-04-22T15:50:00.901 INFO:tasks.workunit.client.1.smithi027.stdout:6/626: dread - d5/f89 zero size 2024-04-22T15:50:00.902 INFO:tasks.workunit.client.1.smithi027.stdout:5/584: read d0/df/de/f38 [381869,118464] 0 2024-04-22T15:50:00.902 INFO:tasks.workunit.client.1.smithi027.stdout:6/627: stat d5/dc/dd/d79 0 2024-04-22T15:50:00.909 INFO:tasks.workunit.client.1.smithi027.stdout:6/628: dwrite d5/dc/dd/d79/f8b [0,4194304] 0 2024-04-22T15:50:00.915 INFO:tasks.workunit.client.1.smithi027.stdout:6/629: dread d5/db/fa9 [0,4194304] 0 2024-04-22T15:50:00.917 INFO:tasks.workunit.client.1.smithi027.stdout:6/630: write d5/dc/d20/da3/fcc [194585,126340] 0 2024-04-22T15:50:00.918 INFO:tasks.workunit.client.1.smithi027.stdout:5/585: dread d0/d1a/d22/d72/f7b [0,4194304] 0 2024-04-22T15:50:00.919 INFO:tasks.workunit.client.1.smithi027.stdout:5/586: stat d0/d1a/d22/l96 0 2024-04-22T15:50:00.920 INFO:tasks.workunit.client.1.smithi027.stdout:5/587: readlink d0/db6/lb7 0 2024-04-22T15:50:00.921 INFO:tasks.workunit.client.1.smithi027.stdout:6/631: rmdir d5/dc/d17/d53 39 2024-04-22T15:50:00.924 INFO:tasks.workunit.client.1.smithi027.stdout:6/632: creat d5/dc/d26/fd7 x:0 0 0 2024-04-22T15:50:00.929 INFO:tasks.workunit.client.1.smithi027.stdout:5/588: dwrite d0/df/f45 [4194304,4194304] 0 2024-04-22T15:50:00.933 INFO:tasks.workunit.client.1.smithi027.stdout:6/633: write d5/db/fa9 [390249,33819] 0 2024-04-22T15:50:00.934 INFO:tasks.workunit.client.1.smithi027.stdout:6/634: chown d5/dc/d26 12488262 1 2024-04-22T15:50:00.936 INFO:tasks.workunit.client.1.smithi027.stdout:6/635: truncate d5/dc/dd/dd0/f35 5025825 0 2024-04-22T15:50:00.936 INFO:tasks.workunit.client.1.smithi027.stdout:5/589: truncate d0/df/d3d/f51 528123 0 2024-04-22T15:50:00.940 INFO:tasks.workunit.client.1.smithi027.stdout:6/636: mkdir d5/d30/d54/dd8 0 2024-04-22T15:50:00.941 INFO:tasks.workunit.client.1.smithi027.stdout:6/637: chown d5/dc/dd/d79/d7b/d5d/fb9 1934999 1 2024-04-22T15:50:00.941 INFO:tasks.workunit.client.1.smithi027.stdout:5/590: creat d0/d1a/d22/d72/dac/fc0 x:0 0 0 2024-04-22T15:50:00.945 INFO:tasks.workunit.client.1.smithi027.stdout:6/638: mknod d5/dc/d17/d53/d8c/cd9 0 2024-04-22T15:50:00.951 INFO:tasks.workunit.client.1.smithi027.stdout:6/639: creat d5/fda x:0 0 0 2024-04-22T15:50:00.952 INFO:tasks.workunit.client.1.smithi027.stdout:5/591: write d0/df/d10/f14 [3080026,12629] 0 2024-04-22T15:50:00.953 INFO:tasks.workunit.client.1.smithi027.stdout:6/640: mknod d5/dc/d20/da3/dc9/cdb 0 2024-04-22T15:50:00.956 INFO:tasks.workunit.client.1.smithi027.stdout:6/641: creat d5/dc/d17/d8a/fdc x:0 0 0 2024-04-22T15:50:00.957 INFO:tasks.workunit.client.1.smithi027.stdout:6/642: symlink d5/dc/dae/ldd 0 2024-04-22T15:50:00.960 INFO:tasks.workunit.client.1.smithi027.stdout:6/643: unlink d5/dc/d26/d3d/c7d 0 2024-04-22T15:50:00.967 INFO:tasks.workunit.client.1.smithi027.stdout:6/644: dwrite d5/dc/dd/dd0/d34/fbf [0,4194304] 0 2024-04-22T15:50:00.978 INFO:tasks.workunit.client.1.smithi027.stdout:6/645: dwrite d5/dc/dd/d79/d7b/d5d/f70 [0,4194304] 0 2024-04-22T15:50:00.979 INFO:tasks.workunit.client.1.smithi027.stdout:6/646: write d5/dc/f74 [47688,28648] 0 2024-04-22T15:50:00.987 INFO:tasks.workunit.client.1.smithi027.stdout:6/647: mkdir d5/dc/d20/da3/dde 0 2024-04-22T15:50:00.990 INFO:tasks.workunit.client.1.smithi027.stdout:6/648: mkdir d5/db/dcd/ddf 0 2024-04-22T15:50:00.998 INFO:tasks.workunit.client.1.smithi027.stdout:6/649: symlink d5/dc/dd/d52/le0 0 2024-04-22T15:50:01.006 INFO:tasks.workunit.client.1.smithi027.stdout:6/650: dwrite d5/db/d2e/ff [0,4194304] 0 2024-04-22T15:50:01.012 INFO:tasks.workunit.client.1.smithi027.stdout:6/651: rename d5/dc/d17/f39 to d5/db/d2e/d59/db5/fe1 0 2024-04-22T15:50:01.029 INFO:tasks.workunit.client.1.smithi027.stdout:6/652: truncate d5/d30/d54/f7f 2078881 0 2024-04-22T15:50:01.031 INFO:tasks.workunit.client.1.smithi027.stdout:6/653: symlink d5/dc/le2 0 2024-04-22T15:50:01.032 INFO:tasks.workunit.client.1.smithi027.stdout:6/654: fsync d5/db/d2e/ff 0 2024-04-22T15:50:01.034 INFO:tasks.workunit.client.1.smithi027.stdout:6/655: truncate d5/dc/dd/dd0/d34/f65 4917948 0 2024-04-22T15:50:01.036 INFO:tasks.workunit.client.1.smithi027.stdout:6/656: rmdir d5/dc/dd/dd0 39 2024-04-22T15:50:01.038 INFO:tasks.workunit.client.1.smithi027.stdout:6/657: creat d5/dc/d20/da3/fe3 x:0 0 0 2024-04-22T15:50:01.041 INFO:tasks.workunit.client.1.smithi027.stdout:6/658: dread d5/dc/dd/fa2 [0,4194304] 0 2024-04-22T15:50:01.049 INFO:tasks.workunit.client.1.smithi027.stdout:6/659: dwrite d5/dc/dd/d79/d7b/f4e [4194304,4194304] 0 2024-04-22T15:50:01.058 INFO:tasks.workunit.client.1.smithi027.stdout:6/660: unlink d5/dc/dd/d52/ld1 0 2024-04-22T15:50:01.060 INFO:tasks.workunit.client.1.smithi027.stdout:6/661: truncate d5/d30/dbe/fc6 224250 0 2024-04-22T15:50:01.062 INFO:tasks.workunit.client.1.smithi027.stdout:6/662: unlink d5/dc/le2 0 2024-04-22T15:50:01.065 INFO:tasks.workunit.client.1.smithi027.stdout:6/663: truncate d5/dc/d20/f3f 4895989 0 2024-04-22T15:50:01.069 INFO:tasks.workunit.client.1.smithi027.stdout:6/664: getdents d5/dc/d17/d8a 0 2024-04-22T15:50:01.077 INFO:tasks.workunit.client.1.smithi027.stdout:6/665: rename d5/db/da5/lc3 to d5/dc/d17/d8a/le4 0 2024-04-22T15:50:01.095 INFO:tasks.workunit.client.1.smithi027.stdout:6/666: dwrite d5/f89 [0,4194304] 0 2024-04-22T15:50:01.099 INFO:tasks.workunit.client.1.smithi027.stdout:6/667: creat d5/dc/dd/d79/d7b/d9a/fe5 x:0 0 0 2024-04-22T15:50:01.105 INFO:tasks.workunit.client.1.smithi027.stdout:4/588: sync 2024-04-22T15:50:01.105 INFO:tasks.workunit.client.1.smithi027.stdout:2/565: sync 2024-04-22T15:50:01.111 INFO:tasks.workunit.client.1.smithi027.stdout:4/589: rename d5/de/c20 to d5/d45/db3/ce1 0 2024-04-22T15:50:01.116 INFO:tasks.workunit.client.1.smithi027.stdout:2/566: getdents dd/df/d1b/d38/d60/d75 0 2024-04-22T15:50:01.118 INFO:tasks.workunit.client.1.smithi027.stdout:6/668: dwrite d5/d30/dbe/fc6 [0,4194304] 0 2024-04-22T15:50:01.120 INFO:tasks.workunit.client.1.smithi027.stdout:6/669: fdatasync d5/dc/dd/d79/d7b/d9a/fa0 0 2024-04-22T15:50:01.126 INFO:tasks.workunit.client.1.smithi027.stdout:2/567: rename dd/le to dd/df/d70/d4b/d52/lb5 0 2024-04-22T15:50:01.131 INFO:tasks.workunit.client.1.smithi027.stdout:2/568: rename dd/df/d70/d4b/d52/d9a/da4/dad/d8d/fa8 to dd/df/d70/d4b/d52/fb6 0 2024-04-22T15:50:01.133 INFO:tasks.workunit.client.1.smithi027.stdout:6/670: dwrite d5/dc/dd/fa2 [0,4194304] 0 2024-04-22T15:50:01.140 INFO:tasks.workunit.client.1.smithi027.stdout:6/671: dwrite d5/db/d2e/f62 [0,4194304] 0 2024-04-22T15:50:01.144 INFO:tasks.workunit.client.1.smithi027.stdout:6/672: dread d5/dc/d20/da3/fcc [0,4194304] 0 2024-04-22T15:50:01.145 INFO:tasks.workunit.client.1.smithi027.stdout:6/673: write d5/dc/d26/f88 [1087167,44506] 0 2024-04-22T15:50:01.153 INFO:tasks.workunit.client.1.smithi027.stdout:2/569: dread dd/df/d1b/d38/d60/f65 [0,4194304] 0 2024-04-22T15:50:01.154 INFO:tasks.workunit.client.1.smithi027.stdout:6/674: mknod d5/d30/ce6 0 2024-04-22T15:50:01.158 INFO:tasks.workunit.client.1.smithi027.stdout:6/675: dread d5/dc/d26/f88 [0,4194304] 0 2024-04-22T15:50:01.159 INFO:tasks.workunit.client.1.smithi027.stdout:6/676: chown d5/dc/d26 1593019393 1 2024-04-22T15:50:01.163 INFO:tasks.workunit.client.1.smithi027.stdout:6/677: link d5/dc/dd/d52/le0 d5/d30/dbe/le7 0 2024-04-22T15:50:01.165 INFO:tasks.workunit.client.1.smithi027.stdout:6/678: fsync d5/dc/dd/dd0/d34/fb0 0 2024-04-22T15:50:01.166 INFO:tasks.workunit.client.1.smithi027.stdout:6/679: write d5/dc/dd/dd0/f97 [1489495,97251] 0 2024-04-22T15:50:01.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:00 smithi016 ceph-mon[26620]: pgmap v4: 65 pgs: 65 active+clean; 2.9 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail 2024-04-22T15:50:01.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:00 smithi016 ceph-mon[26620]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:50:01.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:00 smithi016 ceph-mon[26620]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:50:01.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:00 smithi016 ceph-mon[26620]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:50:01.177 INFO:tasks.workunit.client.1.smithi027.stdout:6/680: truncate d5/dc/dd/dd0/f35 2476153 0 2024-04-22T15:50:01.179 INFO:tasks.workunit.client.1.smithi027.stdout:6/681: symlink d5/dc/d99/le8 0 2024-04-22T15:50:01.180 INFO:tasks.workunit.client.1.smithi027.stdout:6/682: symlink d5/dc/d99/le9 0 2024-04-22T15:50:01.181 INFO:tasks.workunit.client.1.smithi027.stdout:6/683: chown d5/l32 2217 1 2024-04-22T15:50:01.184 INFO:tasks.workunit.client.1.smithi027.stdout:6/684: link d5/dc/dd/d79/d7b/fc1 d5/d30/d54/fea 0 2024-04-22T15:50:01.185 INFO:tasks.workunit.client.1.smithi027.stdout:6/685: dread - d5/dc/dd/d79/d7b/fc1 zero size 2024-04-22T15:50:01.189 INFO:tasks.workunit.client.1.smithi027.stdout:6/686: creat d5/dc/d26/d3d/feb x:0 0 0 2024-04-22T15:50:01.190 INFO:tasks.workunit.client.1.smithi027.stdout:6/687: chown d5/dc/dd/dd0/d34/fb0 18626 1 2024-04-22T15:50:01.193 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:00 smithi027 ceph-mon[34922]: pgmap v4: 65 pgs: 65 active+clean; 2.9 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail 2024-04-22T15:50:01.193 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:00 smithi027 ceph-mon[34922]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:50:01.193 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:00 smithi027 ceph-mon[34922]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:50:01.193 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:00 smithi027 ceph-mon[34922]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:50:01.197 INFO:tasks.workunit.client.1.smithi027.stdout:6/688: dwrite d5/dc/f74 [0,4194304] 0 2024-04-22T15:50:01.212 INFO:tasks.workunit.client.1.smithi027.stdout:6/689: write d5/dc/dd/d79/d7b/d5d/f6c [1826676,97688] 0 2024-04-22T15:50:01.213 INFO:tasks.workunit.client.1.smithi027.stdout:6/690: chown d5/dc/dd/d79/d7b/fc1 356706027 1 2024-04-22T15:50:01.216 INFO:tasks.workunit.client.1.smithi027.stdout:6/691: mkdir d5/dc/d17/d53/dec 0 2024-04-22T15:50:01.281 INFO:tasks.workunit.client.1.smithi027.stdout:1/493: sync 2024-04-22T15:50:01.281 INFO:tasks.workunit.client.1.smithi027.stdout:9/584: sync 2024-04-22T15:50:01.281 INFO:tasks.workunit.client.1.smithi027.stdout:5/592: sync 2024-04-22T15:50:01.281 INFO:tasks.workunit.client.1.smithi027.stdout:8/505: sync 2024-04-22T15:50:01.281 INFO:tasks.workunit.client.1.smithi027.stdout:7/552: sync 2024-04-22T15:50:01.281 INFO:tasks.workunit.client.1.smithi027.stdout:3/648: sync 2024-04-22T15:50:01.281 INFO:tasks.workunit.client.1.smithi027.stdout:0/580: sync 2024-04-22T15:50:01.282 INFO:tasks.workunit.client.1.smithi027.stdout:7/553: fsync d0/d3f/f5b 0 2024-04-22T15:50:01.285 INFO:tasks.workunit.client.1.smithi027.stdout:8/506: mkdir d1/d3/d1c/d43/d82/dac 0 2024-04-22T15:50:01.285 INFO:tasks.workunit.client.1.smithi027.stdout:3/649: dread d28/d5d/d62/faa [0,4194304] 0 2024-04-22T15:50:01.285 INFO:tasks.workunit.client.1.smithi027.stdout:8/507: chown d1/d1e/d47/d53 0 1 2024-04-22T15:50:01.378 INFO:tasks.workunit.client.1.smithi027.stdout:0/581: mkdir d0/d34/d4e/d85/db3 0 2024-04-22T15:50:01.378 INFO:tasks.workunit.client.1.smithi027.stdout:1/494: truncate d2/f3 3081403 0 2024-04-22T15:50:01.379 INFO:tasks.workunit.client.1.smithi027.stdout:5/593: rename d0/df/de/d24/d62/c71 to d0/df/de/d24/d1c/d8f/cc1 0 2024-04-22T15:50:01.379 INFO:tasks.workunit.client.1.smithi027.stdout:1/495: readlink d2/d1d/l2a 0 2024-04-22T15:50:01.380 INFO:tasks.workunit.client.1.smithi027.stdout:3/650: creat d28/d4b/d52/d9a/fd1 x:0 0 0 2024-04-22T15:50:01.381 INFO:tasks.workunit.client.1.smithi027.stdout:0/582: mknod d0/d2a/d38/cb4 0 2024-04-22T15:50:01.382 INFO:tasks.workunit.client.1.smithi027.stdout:0/583: fdatasync d0/d34/f71 0 2024-04-22T15:50:01.382 INFO:tasks.workunit.client.1.smithi027.stdout:0/584: readlink d0/l16 0 2024-04-22T15:50:01.384 INFO:tasks.workunit.client.1.smithi027.stdout:8/508: rename d1/d3/d1c/d43/d5f/d7d/f90 to d1/d1e/d47/d53/d76/fad 0 2024-04-22T15:50:01.387 INFO:tasks.workunit.client.1.smithi027.stdout:8/509: truncate d1/f92 824687 0 2024-04-22T15:50:01.389 INFO:tasks.workunit.client.1.smithi027.stdout:1/496: dwrite d2/d5f/d64/d81/d3d/d44/f86 [0,4194304] 0 2024-04-22T15:50:01.390 INFO:tasks.workunit.client.1.smithi027.stdout:8/510: read - d1/d3/d1c/d32/d41/f83 zero size 2024-04-22T15:50:01.391 INFO:tasks.workunit.client.1.smithi027.stdout:3/651: dread d28/f88 [0,4194304] 0 2024-04-22T15:50:01.392 INFO:tasks.workunit.client.1.smithi027.stdout:1/497: readlink d2/d5f/d64/d81/da7/l6a 0 2024-04-22T15:50:01.398 INFO:tasks.workunit.client.1.smithi027.stdout:3/652: dread d28/f6c [0,4194304] 0 2024-04-22T15:50:01.402 INFO:tasks.workunit.client.1.smithi027.stdout:0/585: getdents d0/d2a/d6a/d6c/d83/d9e 0 2024-04-22T15:50:01.407 INFO:tasks.workunit.client.1.smithi027.stdout:0/586: dwrite d0/d34/f80 [0,4194304] 0 2024-04-22T15:50:01.408 INFO:tasks.workunit.client.1.smithi027.stdout:0/587: fdatasync d0/d28/d63/f8d 0 2024-04-22T15:50:01.410 INFO:tasks.workunit.client.1.smithi027.stdout:1/498: write d2/d5f/d64/d81/d3d/d44/f92 [12630,78954] 0 2024-04-22T15:50:01.413 INFO:tasks.workunit.client.1.smithi027.stdout:1/499: write d2/d5f/d64/d81/d7e/fa1 [37409,91585] 0 2024-04-22T15:50:01.415 INFO:tasks.workunit.client.1.smithi027.stdout:7/554: mkdir d0/d11/d1c/d2c/d96/db3 0 2024-04-22T15:50:01.418 INFO:tasks.workunit.client.1.smithi027.stdout:3/653: creat d28/d4b/d52/db2/d95/d9c/fd2 x:0 0 0 2024-04-22T15:50:01.422 INFO:tasks.workunit.client.1.smithi027.stdout:8/511: creat d1/d3/d1c/d49/d94/fae x:0 0 0 2024-04-22T15:50:01.424 INFO:tasks.workunit.client.1.smithi027.stdout:5/594: dwrite d0/df/de/d24/f2e [0,4194304] 0 2024-04-22T15:50:01.427 INFO:tasks.workunit.client.1.smithi027.stdout:0/588: creat d0/d2a/d5e/d8b/fb5 x:0 0 0 2024-04-22T15:50:01.428 INFO:tasks.workunit.client.1.smithi027.stdout:3/654: dwrite d28/d4b/fad [0,4194304] 0 2024-04-22T15:50:01.428 INFO:tasks.workunit.client.1.smithi027.stdout:0/589: readlink d0/d2a/d38/l79 0 2024-04-22T15:50:01.435 INFO:tasks.workunit.client.1.smithi027.stdout:8/512: unlink d1/d3/d1c/d56/c79 0 2024-04-22T15:50:01.439 INFO:tasks.workunit.client.1.smithi027.stdout:5/595: creat d0/db6/fc2 x:0 0 0 2024-04-22T15:50:01.441 INFO:tasks.workunit.client.1.smithi027.stdout:5/596: write d0/db6/fc2 [780838,52202] 0 2024-04-22T15:50:01.445 INFO:tasks.workunit.client.1.smithi027.stdout:0/590: mkdir d0/da3/db6 0 2024-04-22T15:50:01.449 INFO:tasks.workunit.client.1.smithi027.stdout:8/513: mknod d1/d3/d1c/d49/caf 0 2024-04-22T15:50:01.451 INFO:tasks.workunit.client.1.smithi027.stdout:3/655: rename d28/d5d/d79/f7e to d28/d69/fd3 0 2024-04-22T15:50:01.452 INFO:tasks.workunit.client.1.smithi027.stdout:3/656: truncate d28/d5d/d62/fb1 1094156 0 2024-04-22T15:50:01.454 INFO:tasks.workunit.client.1.smithi027.stdout:5/597: mkdir d0/d97/dbf/dc3 0 2024-04-22T15:50:01.455 INFO:tasks.workunit.client.1.smithi027.stdout:5/598: truncate d0/df/d3d/d81/f98 478031 0 2024-04-22T15:50:01.456 INFO:tasks.workunit.client.1.smithi027.stdout:4/590: sync 2024-04-22T15:50:01.461 INFO:tasks.workunit.client.1.smithi027.stdout:5/599: dread d0/df/de/d24/d1c/d3e/d55/f9c [0,4194304] 0 2024-04-22T15:50:01.462 INFO:tasks.workunit.client.1.smithi027.stdout:1/500: write d2/f27 [2122298,15961] 0 2024-04-22T15:50:01.463 INFO:tasks.workunit.client.1.smithi027.stdout:5/600: chown d0/d1a/d22/d26/d5d/ca4 299588 1 2024-04-22T15:50:01.464 INFO:tasks.workunit.client.1.smithi027.stdout:4/591: dwrite d5/de/d2a/fc3 [0,4194304] 0 2024-04-22T15:50:01.464 INFO:tasks.workunit.client.1.smithi027.stdout:1/501: fdatasync d2/f35 0 2024-04-22T15:50:01.470 INFO:tasks.workunit.client.1.smithi027.stdout:0/591: rename d0/d2a/d38/d89 to d0/d9d/db7 0 2024-04-22T15:50:01.471 INFO:tasks.workunit.client.1.smithi027.stdout:0/592: truncate d0/fa8 291460 0 2024-04-22T15:50:01.471 INFO:tasks.workunit.client.1.smithi027.stdout:0/593: stat d0/d28/c29 0 2024-04-22T15:50:01.484 INFO:tasks.workunit.client.1.smithi027.stdout:4/592: mkdir d5/d11/d89/de2 0 2024-04-22T15:50:01.485 INFO:tasks.workunit.client.1.smithi027.stdout:4/593: chown d5/d42/l78 949546 1 2024-04-22T15:50:01.487 INFO:tasks.workunit.client.1.smithi027.stdout:8/514: write d1/d3/f40 [3114123,28617] 0 2024-04-22T15:50:01.488 INFO:tasks.workunit.client.1.smithi027.stdout:8/515: stat d1/d3/d1c 0 2024-04-22T15:50:01.490 INFO:tasks.workunit.client.1.smithi027.stdout:1/502: write d2/d30/fa3 [4246433,38202] 0 2024-04-22T15:50:01.491 INFO:tasks.workunit.client.1.smithi027.stdout:1/503: dread - d2/d5f/d64/d81/da7/f9e zero size 2024-04-22T15:50:01.492 INFO:tasks.workunit.client.1.smithi027.stdout:0/594: creat d0/d34/d4e/d85/db3/fb8 x:0 0 0 2024-04-22T15:50:01.493 INFO:tasks.workunit.client.1.smithi027.stdout:3/657: truncate d28/d5d/d79/fa1 620792 0 2024-04-22T15:50:01.494 INFO:tasks.workunit.client.1.smithi027.stdout:3/658: fdatasync d28/f38 0 2024-04-22T15:50:01.494 INFO:tasks.workunit.client.1.smithi027.stdout:5/601: dread d0/df/de/d40/f6a [0,4194304] 0 2024-04-22T15:50:01.498 INFO:tasks.workunit.client.1.smithi027.stdout:0/595: fsync d0/fc 0 2024-04-22T15:50:01.498 INFO:tasks.workunit.client.1.smithi027.stdout:1/504: write d2/d30/f41 [1457383,85624] 0 2024-04-22T15:50:01.499 INFO:tasks.workunit.client.1.smithi027.stdout:0/596: readlink d0/d2a/d38/lb1 0 2024-04-22T15:50:01.502 INFO:tasks.workunit.client.1.smithi027.stdout:1/505: dread - d2/d1d/f96 zero size 2024-04-22T15:50:01.502 INFO:tasks.workunit.client.1.smithi027.stdout:0/597: write d0/d34/f5a [179821,17642] 0 2024-04-22T15:50:01.502 INFO:tasks.workunit.client.1.smithi027.stdout:5/602: creat d0/d1a/d22/d26/fc4 x:0 0 0 2024-04-22T15:50:01.503 INFO:tasks.workunit.client.1.smithi027.stdout:5/603: stat d0/df/de/l1f 0 2024-04-22T15:50:01.504 INFO:tasks.workunit.client.1.smithi027.stdout:4/594: dread d5/d11/d1d/f5c [0,4194304] 0 2024-04-22T15:50:01.507 INFO:tasks.workunit.client.1.smithi027.stdout:8/516: link d1/d1e/d47/d53/f77 d1/d3/d1c/d32/fb0 0 2024-04-22T15:50:01.513 INFO:tasks.workunit.client.1.smithi027.stdout:3/659: truncate d28/f44 860408 0 2024-04-22T15:50:01.517 INFO:tasks.workunit.client.1.smithi027.stdout:1/506: dread d2/d36/f55 [0,4194304] 0 2024-04-22T15:50:01.517 INFO:tasks.workunit.client.1.smithi027.stdout:4/595: mkdir d5/de/d30/de3 0 2024-04-22T15:50:01.520 INFO:tasks.workunit.client.1.smithi027.stdout:0/598: dread d0/d2a/f2b [0,4194304] 0 2024-04-22T15:50:01.520 INFO:tasks.workunit.client.1.smithi027.stdout:5/604: rename d0/d1a/d22/d26/ca3 to d0/d1a/d22/d72/cc5 0 2024-04-22T15:50:01.522 INFO:tasks.workunit.client.1.smithi027.stdout:8/517: rmdir d1/d1e/d47/d9d 39 2024-04-22T15:50:01.523 INFO:tasks.workunit.client.1.smithi027.stdout:0/599: write d0/d2a/f48 [3297691,46348] 0 2024-04-22T15:50:01.529 INFO:tasks.workunit.client.1.smithi027.stdout:4/596: dwrite d5/d42/f61 [0,4194304] 0 2024-04-22T15:50:01.533 INFO:tasks.workunit.client.1.smithi027.stdout:0/600: dread d0/d34/f80 [0,4194304] 0 2024-04-22T15:50:01.537 INFO:tasks.workunit.client.1.smithi027.stdout:5/605: creat d0/d1a/d22/fc6 x:0 0 0 2024-04-22T15:50:01.537 INFO:tasks.workunit.client.1.smithi027.stdout:0/601: write d0/d28/d63/f8d [844579,32122] 0 2024-04-22T15:50:01.541 INFO:tasks.workunit.client.1.smithi027.stdout:8/518: creat d1/d3/d1c/d28/d42/fb1 x:0 0 0 2024-04-22T15:50:01.541 INFO:tasks.workunit.client.1.smithi027.stdout:3/660: mkdir d28/d5d/db0/dce/dd4 0 2024-04-22T15:50:01.542 INFO:tasks.workunit.client.1.smithi027.stdout:4/597: dwrite d5/d11/f80 [0,4194304] 0 2024-04-22T15:50:01.542 INFO:tasks.workunit.client.1.smithi027.stdout:5/606: dread - d0/df/de/d24/d62/f7f zero size 2024-04-22T15:50:01.544 INFO:tasks.workunit.client.1.smithi027.stdout:1/507: dread d2/f68 [0,4194304] 0 2024-04-22T15:50:01.544 INFO:tasks.workunit.client.1.smithi027.stdout:3/661: chown d28/d69/f78 15883 1 2024-04-22T15:50:01.545 INFO:tasks.workunit.client.1.smithi027.stdout:4/598: chown d5/d45/d47/c50 4011 1 2024-04-22T15:50:01.546 INFO:tasks.workunit.client.1.smithi027.stdout:0/602: mkdir d0/da3/db9 0 2024-04-22T15:50:01.553 INFO:tasks.workunit.client.1.smithi027.stdout:5/607: fsync d0/df/de/d24/d62/f68 0 2024-04-22T15:50:01.553 INFO:tasks.workunit.client.1.smithi027.stdout:1/508: dwrite d2/d36/f4b [0,4194304] 0 2024-04-22T15:50:01.555 INFO:tasks.workunit.client.1.smithi027.stdout:5/608: dread - d0/df/de/d24/d62/f7f zero size 2024-04-22T15:50:01.558 INFO:tasks.workunit.client.1.smithi027.stdout:0/603: creat d0/d2a/d58/fba x:0 0 0 2024-04-22T15:50:01.565 INFO:tasks.workunit.client.1.smithi027.stdout:5/609: fsync d0/df/de/d24/d1c/d3e/d55/f9c 0 2024-04-22T15:50:01.567 INFO:tasks.workunit.client.1.smithi027.stdout:8/519: dread d1/d3/d1c/d32/f3d [0,4194304] 0 2024-04-22T15:50:01.574 INFO:tasks.workunit.client.1.smithi027.stdout:3/662: link l8 d28/d4b/ld5 0 2024-04-22T15:50:01.580 INFO:tasks.workunit.client.1.smithi027.stdout:1/509: write d2/f60 [353716,121962] 0 2024-04-22T15:50:01.581 INFO:tasks.workunit.client.1.smithi027.stdout:3/663: mknod d28/d4b/d52/db2/d95/cd6 0 2024-04-22T15:50:01.582 INFO:tasks.workunit.client.1.smithi027.stdout:3/664: fsync f11 0 2024-04-22T15:50:01.587 INFO:tasks.workunit.client.1.smithi027.stdout:5/610: getdents d0/df/de/d24 0 2024-04-22T15:50:01.587 INFO:tasks.workunit.client.1.smithi027.stdout:0/604: dwrite d0/d34/f81 [0,4194304] 0 2024-04-22T15:50:01.588 INFO:tasks.workunit.client.1.smithi027.stdout:5/611: chown d0/df/d10/f14 130160 1 2024-04-22T15:50:01.590 INFO:tasks.workunit.client.1.smithi027.stdout:8/520: getdents d1/d3/d1c/d56 0 2024-04-22T15:50:01.590 INFO:tasks.workunit.client.1.smithi027.stdout:5/612: stat d0/d1a/d22/d26/d32/d79/fa7 0 2024-04-22T15:50:01.593 INFO:tasks.workunit.client.1.smithi027.stdout:1/510: symlink d2/lb1 0 2024-04-22T15:50:01.593 INFO:tasks.workunit.client.1.smithi027.stdout:5/613: read d0/df/de/d24/f35 [699662,41920] 0 2024-04-22T15:50:01.595 INFO:tasks.workunit.client.1.smithi027.stdout:0/605: rename d0/l3 to d0/d2a/d6a/lbb 0 2024-04-22T15:50:01.603 INFO:tasks.workunit.client.1.smithi027.stdout:5/614: symlink d0/d1a/d22/d26/d32/d79/lc7 0 2024-04-22T15:50:01.606 INFO:tasks.workunit.client.1.smithi027.stdout:3/665: rmdir d28/d4b/d52/db2/d95/dbf 0 2024-04-22T15:50:01.607 INFO:tasks.workunit.client.1.smithi027.stdout:5/615: dread d0/d1a/d22/d72/f7b [0,4194304] 0 2024-04-22T15:50:01.614 INFO:tasks.workunit.client.1.smithi027.stdout:8/521: creat d1/d3/d1c/d32/d41/d52/d61/fb2 x:0 0 0 2024-04-22T15:50:01.622 INFO:tasks.workunit.client.1.smithi027.stdout:0/606: dread d0/d34/f7a [0,4194304] 0 2024-04-22T15:50:01.622 INFO:tasks.workunit.client.1.smithi027.stdout:0/607: chown d0/d2a/d91/da1 51519858 1 2024-04-22T15:50:01.627 INFO:tasks.workunit.client.1.smithi027.stdout:3/666: dread f5 [0,4194304] 0 2024-04-22T15:50:01.629 INFO:tasks.workunit.client.1.smithi027.stdout:0/608: dwrite d0/d34/f5a [0,4194304] 0 2024-04-22T15:50:01.643 INFO:tasks.workunit.client.1.smithi027.stdout:6/692: sync 2024-04-22T15:50:01.643 INFO:tasks.workunit.client.1.smithi027.stdout:2/570: sync 2024-04-22T15:50:01.666 INFO:tasks.workunit.client.1.smithi027.stdout:1/511: getdents d2/d1d 0 2024-04-22T15:50:01.668 INFO:tasks.workunit.client.1.smithi027.stdout:5/616: creat d0/df/de/d24/d1c/d3e/fc8 x:0 0 0 2024-04-22T15:50:01.669 INFO:tasks.workunit.client.1.smithi027.stdout:7/555: sync 2024-04-22T15:50:01.669 INFO:tasks.workunit.client.1.smithi027.stdout:9/585: sync 2024-04-22T15:50:01.669 INFO:tasks.workunit.client.1.smithi027.stdout:4/599: sync 2024-04-22T15:50:01.674 INFO:tasks.workunit.client.1.smithi027.stdout:8/522: symlink d1/d3/d1c/d32/d41/d52/d61/d8f/lb3 0 2024-04-22T15:50:01.675 INFO:tasks.workunit.client.1.smithi027.stdout:1/512: creat d2/d5f/d64/d81/d3d/d44/fb2 x:0 0 0 2024-04-22T15:50:01.679 INFO:tasks.workunit.client.1.smithi027.stdout:9/586: symlink d0/dc/d12/d58/d6b/d6e/lc3 0 2024-04-22T15:50:01.680 INFO:tasks.workunit.client.1.smithi027.stdout:4/600: dread - d5/d11/d89/f70 zero size 2024-04-22T15:50:01.682 INFO:tasks.workunit.client.1.smithi027.stdout:3/667: mkdir d28/d5d/dd7 0 2024-04-22T15:50:01.683 INFO:tasks.workunit.client.1.smithi027.stdout:5/617: truncate d0/df/d3d/f69 1039052 0 2024-04-22T15:50:01.686 INFO:tasks.workunit.client.1.smithi027.stdout:6/693: symlink d5/db/led 0 2024-04-22T15:50:01.689 INFO:tasks.workunit.client.1.smithi027.stdout:3/668: dwrite f10 [0,4194304] 0 2024-04-22T15:50:01.691 INFO:tasks.workunit.client.1.smithi027.stdout:8/523: dread d1/d3/d1c/d28/d96/fa0 [0,4194304] 0 2024-04-22T15:50:01.691 INFO:tasks.workunit.client.1.smithi027.stdout:1/513: read d2/f8 [1226327,50335] 0 2024-04-22T15:50:01.698 INFO:tasks.workunit.client.1.smithi027.stdout:7/556: mknod d0/d11/d1c/d31/cb4 0 2024-04-22T15:50:01.701 INFO:tasks.workunit.client.1.smithi027.stdout:3/669: rename d28/d4b/d52/d94/dae/cc5 to d28/d4b/d52/d94/dae/cd8 0 2024-04-22T15:50:01.703 INFO:tasks.workunit.client.1.smithi027.stdout:8/524: symlink d1/d3/d1c/d32/d41/d52/d61/d8f/lb4 0 2024-04-22T15:50:01.704 INFO:tasks.workunit.client.1.smithi027.stdout:6/694: dread d5/d30/dbe/fca [0,4194304] 0 2024-04-22T15:50:01.705 INFO:tasks.workunit.client.1.smithi027.stdout:8/525: chown d1/d3/d1c/d32/d41/d52/d61/fb2 344 1 2024-04-22T15:50:01.710 INFO:tasks.workunit.client.1.smithi027.stdout:0/609: getdents d0/d28/d63/d67 0 2024-04-22T15:50:01.711 INFO:tasks.workunit.client.1.smithi027.stdout:2/571: dwrite dd/df/d1b/f5f [0,4194304] 0 2024-04-22T15:50:01.712 INFO:tasks.workunit.client.1.smithi027.stdout:2/572: stat dd/f6f 0 2024-04-22T15:50:01.719 INFO:tasks.workunit.client.1.smithi027.stdout:5/618: creat d0/fc9 x:0 0 0 2024-04-22T15:50:01.720 INFO:tasks.workunit.client.1.smithi027.stdout:4/601: write d5/d11/d89/f70 [632838,57250] 0 2024-04-22T15:50:01.722 INFO:tasks.workunit.client.1.smithi027.stdout:6/695: mkdir d5/dc/d26/dee 0 2024-04-22T15:50:01.731 INFO:tasks.workunit.client.1.smithi027.stdout:0/610: symlink d0/d28/d63/lbc 0 2024-04-22T15:50:01.733 INFO:tasks.workunit.client.1.smithi027.stdout:7/557: dwrite d0/d49/f63 [4194304,4194304] 0 2024-04-22T15:50:01.736 INFO:tasks.workunit.client.1.smithi027.stdout:0/611: write d0/f2e [789731,98700] 0 2024-04-22T15:50:01.738 INFO:tasks.workunit.client.1.smithi027.stdout:9/587: getdents d0/d8 0 2024-04-22T15:50:01.743 INFO:tasks.workunit.client.1.smithi027.stdout:9/588: read - d0/dc/d12/f94 zero size 2024-04-22T15:50:01.746 INFO:tasks.workunit.client.1.smithi027.stdout:1/514: dwrite d2/d30/f47 [0,4194304] 0 2024-04-22T15:50:01.748 INFO:tasks.workunit.client.1.smithi027.stdout:4/602: dread d5/de/d2a/d58/f67 [0,4194304] 0 2024-04-22T15:50:01.749 INFO:tasks.workunit.client.1.smithi027.stdout:7/558: stat d0/d11/d1c/d31/d56 0 2024-04-22T15:50:01.751 INFO:tasks.workunit.client.1.smithi027.stdout:0/612: creat d0/d2a/d5e/fbd x:0 0 0 2024-04-22T15:50:01.751 INFO:tasks.workunit.client.1.smithi027.stdout:3/670: dread d28/d4b/f7c [0,4194304] 0 2024-04-22T15:50:01.755 INFO:tasks.workunit.client.1.smithi027.stdout:2/573: dwrite dd/df/d1b/d38/d60/d75/fb3 [0,4194304] 0 2024-04-22T15:50:01.756 INFO:tasks.workunit.client.1.smithi027.stdout:6/696: dread d5/db/f4d [0,4194304] 0 2024-04-22T15:50:01.761 INFO:tasks.workunit.client.1.smithi027.stdout:8/526: rename f0 to d1/d3/d1c/d28/fb5 0 2024-04-22T15:50:01.763 INFO:tasks.workunit.client.1.smithi027.stdout:5/619: symlink d0/df/de/d24/d1c/d3e/da2/lca 0 2024-04-22T15:50:01.764 INFO:tasks.workunit.client.1.smithi027.stdout:9/589: mknod d0/d8/cc4 0 2024-04-22T15:50:01.768 INFO:tasks.workunit.client.1.smithi027.stdout:2/574: fsync dd/df/d70/f6b 0 2024-04-22T15:50:01.769 INFO:tasks.workunit.client.1.smithi027.stdout:4/603: dwrite d5/d11/d1d/d7b/fce [0,4194304] 0 2024-04-22T15:50:01.770 INFO:tasks.workunit.client.1.smithi027.stdout:0/613: mknod d0/d34/d4e/d85/db3/cbe 0 2024-04-22T15:50:01.772 INFO:tasks.workunit.client.1.smithi027.stdout:0/614: chown d0/d2a/d38/lb1 2264 1 2024-04-22T15:50:01.774 INFO:tasks.workunit.client.1.smithi027.stdout:6/697: dwrite d5/dc/dd/dd0/f97 [0,4194304] 0 2024-04-22T15:50:01.781 INFO:tasks.workunit.client.1.smithi027.stdout:1/515: rename d2/d30/l97 to d2/d5f/d64/d81/d69/d7d/lb3 0 2024-04-22T15:50:01.782 INFO:tasks.workunit.client.1.smithi027.stdout:3/671: rename d28/d4b/d52/db2/d95/d9c to d28/d4b/d52/db2/d95/d9c/dd9 22 2024-04-22T15:50:01.783 INFO:tasks.workunit.client.1.smithi027.stdout:0/615: dread d0/d9d/db7/fa9 [0,4194304] 0 2024-04-22T15:50:01.787 INFO:tasks.workunit.client.1.smithi027.stdout:9/590: creat d0/dc/de/fc5 x:0 0 0 2024-04-22T15:50:01.793 INFO:tasks.workunit.client.1.smithi027.stdout:8/527: dwrite d1/d1e/d47/d53/d76/fad [0,4194304] 0 2024-04-22T15:50:01.795 INFO:tasks.workunit.client.1.smithi027.stdout:8/528: stat d1/d3/d1c/d32/d41/d52 0 2024-04-22T15:50:01.796 INFO:tasks.workunit.client.1.smithi027.stdout:2/575: truncate dd/df/d1b/d96/f98 5797661 0 2024-04-22T15:50:01.797 INFO:tasks.workunit.client.1.smithi027.stdout:4/604: creat d5/d45/d81/fe4 x:0 0 0 2024-04-22T15:50:01.799 INFO:tasks.workunit.client.1.smithi027.stdout:1/516: dread d2/d5f/d64/d81/da7/f67 [0,4194304] 0 2024-04-22T15:50:01.802 INFO:tasks.workunit.client.1.smithi027.stdout:0/616: rmdir d0/d2a/d5e/d8b 39 2024-04-22T15:50:01.805 INFO:tasks.workunit.client.1.smithi027.stdout:7/559: write d0/d11/d1c/d2c/f38 [629219,72608] 0 2024-04-22T15:50:01.808 INFO:tasks.workunit.client.1.smithi027.stdout:9/591: rename d0/dc/d12/d30/c88 to d0/dc/d12/d30/d99/cc6 0 2024-04-22T15:50:01.808 INFO:tasks.workunit.client.1.smithi027.stdout:0/617: dread d0/d34/f7a [0,4194304] 0 2024-04-22T15:50:01.809 INFO:tasks.workunit.client.1.smithi027.stdout:9/592: write d0/dc/de/f40 [1124926,13234] 0 2024-04-22T15:50:01.810 INFO:tasks.workunit.client.1.smithi027.stdout:0/618: read d0/d28/f1f [234446,61240] 0 2024-04-22T15:50:01.813 INFO:tasks.workunit.client.1.smithi027.stdout:6/698: symlink d5/db/dcd/ddf/lef 0 2024-04-22T15:50:01.818 INFO:tasks.workunit.client.1.smithi027.stdout:7/560: rmdir d0/d11/d1c/d31/d8a 39 2024-04-22T15:50:01.819 INFO:tasks.workunit.client.1.smithi027.stdout:2/576: dread dd/df/d1b/d96/f98 [0,4194304] 0 2024-04-22T15:50:01.824 INFO:tasks.workunit.client.1.smithi027.stdout:4/605: rename d5/d11/d1d/d54/fa1 to d5/d11/fe5 0 2024-04-22T15:50:01.828 INFO:tasks.workunit.client.1.smithi027.stdout:0/619: mkdir d0/d2a/d58/dbf 0 2024-04-22T15:50:01.830 INFO:tasks.workunit.client.1.smithi027.stdout:9/593: mknod d0/dc/d12/d30/d5b/d91/cc7 0 2024-04-22T15:50:01.833 INFO:tasks.workunit.client.1.smithi027.stdout:6/699: mkdir d5/db/d2e/d59/df0 0 2024-04-22T15:50:01.834 INFO:tasks.workunit.client.1.smithi027.stdout:2/577: dread dd/df/f59 [4194304,4194304] 0 2024-04-22T15:50:01.836 INFO:tasks.workunit.client.1.smithi027.stdout:5/620: dwrite d0/d1a/d22/d26/d32/f46 [0,4194304] 0 2024-04-22T15:50:01.837 INFO:tasks.workunit.client.1.smithi027.stdout:5/621: write d0/d1a/d22/d26/d32/f46 [1029169,97549] 0 2024-04-22T15:50:01.842 INFO:tasks.workunit.client.1.smithi027.stdout:7/561: write d0/d11/d1c/d31/d56/f58 [4891619,62646] 0 2024-04-22T15:50:01.850 INFO:tasks.workunit.client.1.smithi027.stdout:8/529: write d1/d3/d1c/d32/fb0 [742017,36645] 0 2024-04-22T15:50:01.853 INFO:tasks.workunit.client.1.smithi027.stdout:5/622: read - d0/d1a/d22/d26/f57 zero size 2024-04-22T15:50:01.855 INFO:tasks.workunit.client.1.smithi027.stdout:3/672: getdents d28/d5d 0 2024-04-22T15:50:01.861 INFO:tasks.workunit.client.1.smithi027.stdout:0/620: rename d0/d28/d37/f3b to d0/d9d/fc0 0 2024-04-22T15:50:01.865 INFO:tasks.workunit.client.1.smithi027.stdout:6/700: creat d5/dc/d17/d53/dec/ff1 x:0 0 0 2024-04-22T15:50:01.875 INFO:tasks.workunit.client.1.smithi027.stdout:5/623: creat d0/df/de/d24/d1c/fcb x:0 0 0 2024-04-22T15:50:01.875 INFO:tasks.workunit.client.1.smithi027.stdout:1/517: getdents d2/d5f 0 2024-04-22T15:50:01.876 INFO:tasks.workunit.client.1.smithi027.stdout:8/530: dread d1/d3/d1c/d43/d5f/f9e [4194304,4194304] 0 2024-04-22T15:50:01.879 INFO:tasks.workunit.client.1.smithi027.stdout:8/531: truncate d1/d3/f9b 969773 0 2024-04-22T15:50:01.881 INFO:tasks.workunit.client.1.smithi027.stdout:3/673: read d28/f63 [637777,25025] 0 2024-04-22T15:50:01.883 INFO:tasks.workunit.client.1.smithi027.stdout:8/532: read d1/d1e/d47/f70 [7819824,54198] 0 2024-04-22T15:50:01.883 INFO:tasks.workunit.client.1.smithi027.stdout:1/518: dread d2/d5f/d64/d81/d3d/d82/d85/f95 [0,4194304] 0 2024-04-22T15:50:01.888 INFO:tasks.workunit.client.1.smithi027.stdout:0/621: creat d0/d34/d59/fc1 x:0 0 0 2024-04-22T15:50:01.889 INFO:tasks.workunit.client.1.smithi027.stdout:0/622: chown d0/f2e 136 1 2024-04-22T15:50:01.895 INFO:tasks.workunit.client.1.smithi027.stdout:3/674: mkdir d28/d5d/db0/dce/dda 0 2024-04-22T15:50:01.897 INFO:tasks.workunit.client.1.smithi027.stdout:5/624: dread d0/d1a/d22/d72/d86/f77 [0,4194304] 0 2024-04-22T15:50:01.898 INFO:tasks.workunit.client.1.smithi027.stdout:7/562: rename d0/d11/d1c/d2c/f33 to d0/d49/d9f/fb5 0 2024-04-22T15:50:01.898 INFO:tasks.workunit.client.1.smithi027.stdout:5/625: dread - d0/df/d3d/d81/fbc zero size 2024-04-22T15:50:01.899 INFO:tasks.workunit.client.1.smithi027.stdout:8/533: creat d1/d3/d1c/d32/d41/d52/fb6 x:0 0 0 2024-04-22T15:50:01.900 INFO:tasks.workunit.client.1.smithi027.stdout:7/563: write d0/d11/d1c/d31/d62/f91 [713056,8122] 0 2024-04-22T15:50:01.900 INFO:tasks.workunit.client.1.smithi027.stdout:1/519: truncate d2/f18 1260374 0 2024-04-22T15:50:01.902 INFO:tasks.workunit.client.1.smithi027.stdout:0/623: unlink d0/d2a/d5e/cac 0 2024-04-22T15:50:01.903 INFO:tasks.workunit.client.1.smithi027.stdout:6/701: mkdir d5/dc/dd/d79/d7b/d5d/dba/df2 0 2024-04-22T15:50:01.904 INFO:tasks.workunit.client.1.smithi027.stdout:2/578: link dd/df/l21 dd/df/d70/lb7 0 2024-04-22T15:50:01.906 INFO:tasks.workunit.client.1.smithi027.stdout:3/675: mkdir d28/d69/ddb 0 2024-04-22T15:50:01.912 INFO:tasks.workunit.client.1.smithi027.stdout:5/626: dread d0/d1a/d22/d26/d32/f54 [0,4194304] 0 2024-04-22T15:50:01.912 INFO:tasks.workunit.client.1.smithi027.stdout:8/534: creat d1/d1e/d47/d53/fb7 x:0 0 0 2024-04-22T15:50:01.918 INFO:tasks.workunit.client.1.smithi027.stdout:7/564: dread d0/d11/d1c/d2c/f40 [0,4194304] 0 2024-04-22T15:50:01.918 INFO:tasks.workunit.client.1.smithi027.stdout:7/565: stat d0/d4c/d8b/f89 0 2024-04-22T15:50:01.918 INFO:tasks.workunit.client.1.smithi027.stdout:2/579: symlink dd/df/lb8 0 2024-04-22T15:50:01.919 INFO:tasks.workunit.client.1.smithi027.stdout:2/580: fdatasync dd/d43/f95 0 2024-04-22T15:50:01.921 INFO:tasks.workunit.client.1.smithi027.stdout:0/624: getdents d0/da3/dae 0 2024-04-22T15:50:01.922 INFO:tasks.workunit.client.1.smithi027.stdout:4/606: write d5/d11/f28 [7329831,78323] 0 2024-04-22T15:50:01.926 INFO:tasks.workunit.client.1.smithi027.stdout:5/627: rename d0/df/de/d24/d1c/d3e/da2/faa to d0/df/de/d40/d89/fcc 0 2024-04-22T15:50:01.926 INFO:tasks.workunit.client.1.smithi027.stdout:9/594: write d0/dc/d12/d58/f8a [3944954,37793] 0 2024-04-22T15:50:01.929 INFO:tasks.workunit.client.1.smithi027.stdout:5/628: chown d0/d1a/d22/d72/d86/c85 4 1 2024-04-22T15:50:01.929 INFO:tasks.workunit.client.1.smithi027.stdout:2/581: truncate dd/df/d1b/f5e 40135 0 2024-04-22T15:50:01.929 INFO:tasks.workunit.client.1.smithi027.stdout:5/629: fdatasync d0/df/de/f38 0 2024-04-22T15:50:01.937 INFO:tasks.workunit.client.1.smithi027.stdout:0/625: rename d0/d28/l4b to d0/d2a/d58/d66/lc2 0 2024-04-22T15:50:01.938 INFO:tasks.workunit.client.1.smithi027.stdout:9/595: chown d0/dc/d12/d30/c93 264 1 2024-04-22T15:50:01.939 INFO:tasks.workunit.client.1.smithi027.stdout:9/596: write d0/dc/f4c [1858994,95631] 0 2024-04-22T15:50:01.943 INFO:tasks.workunit.client.1.smithi027.stdout:8/535: creat d1/d3/d1c/d32/d41/d52/d61/fb8 x:0 0 0 2024-04-22T15:50:01.945 INFO:tasks.workunit.client.1.smithi027.stdout:5/630: mknod d0/df/de/d24/d1c/d8f/ccd 0 2024-04-22T15:50:01.947 INFO:tasks.workunit.client.1.smithi027.stdout:5/631: write d0/df/de/d24/d1c/d3e/fc8 [614244,83060] 0 2024-04-22T15:50:01.948 INFO:tasks.workunit.client.1.smithi027.stdout:5/632: chown d0/df/de/d24/d1c/fcb 323183 1 2024-04-22T15:50:01.950 INFO:tasks.workunit.client.1.smithi027.stdout:5/633: write d0/fc9 [114178,95387] 0 2024-04-22T15:50:01.950 INFO:tasks.workunit.client.1.smithi027.stdout:9/597: creat d0/dc/d12/d30/d5b/d91/da9/fc8 x:0 0 0 2024-04-22T15:50:01.953 INFO:tasks.workunit.client.1.smithi027.stdout:8/536: mknod d1/d3/d1c/d32/d41/d52/d61/d8f/cb9 0 2024-04-22T15:50:01.956 INFO:tasks.workunit.client.1.smithi027.stdout:4/607: link d5/de/d2a/d58/f67 d5/d45/d5a/fe6 0 2024-04-22T15:50:01.957 INFO:tasks.workunit.client.1.smithi027.stdout:0/626: dwrite d0/d2a/d5e/d8b/f9f [0,4194304] 0 2024-04-22T15:50:01.959 INFO:tasks.workunit.client.1.smithi027.stdout:1/520: write d2/d30/f3c [2166422,33070] 0 2024-04-22T15:50:01.960 INFO:tasks.workunit.client.1.smithi027.stdout:5/634: mkdir d0/df/de/d24/d1c/d8f/dce 0 2024-04-22T15:50:01.965 INFO:tasks.workunit.client.1.smithi027.stdout:9/598: dwrite d0/dc/d12/d58/fb7 [0,4194304] 0 2024-04-22T15:50:01.967 INFO:tasks.workunit.client.1.smithi027.stdout:7/566: truncate d0/d49/f71 2349771 0 2024-04-22T15:50:01.969 INFO:tasks.workunit.client.1.smithi027.stdout:3/676: dwrite d28/f3f [0,4194304] 0 2024-04-22T15:50:01.970 INFO:tasks.workunit.client.1.smithi027.stdout:8/537: symlink d1/d3/d36/d88/lba 0 2024-04-22T15:50:01.973 INFO:tasks.workunit.client.1.smithi027.stdout:2/582: getdents dd/df/d70/d5a 0 2024-04-22T15:50:01.978 INFO:tasks.workunit.client.1.smithi027.stdout:5/635: fdatasync d0/df/f12 0 2024-04-22T15:50:01.980 INFO:tasks.workunit.client.1.smithi027.stdout:1/521: creat d2/d5f/d64/d81/d7e/fb4 x:0 0 0 2024-04-22T15:50:01.981 INFO:tasks.workunit.client.1.smithi027.stdout:1/522: fdatasync d2/d1d/f32 0 2024-04-22T15:50:01.982 INFO:tasks.workunit.client.1.smithi027.stdout:9/599: rmdir d0/dc/d12/d30/d5b/dbc 39 2024-04-22T15:50:01.982 INFO:tasks.workunit.client.1.smithi027.stdout:1/523: chown d2/d36/daa 0 1 2024-04-22T15:50:01.984 INFO:tasks.workunit.client.1.smithi027.stdout:7/567: creat d0/d11/d3a/d46/d94/d5a/fb6 x:0 0 0 2024-04-22T15:50:01.989 INFO:tasks.workunit.client.1.smithi027.stdout:2/583: truncate dd/f6f 562345 0 2024-04-22T15:50:01.991 INFO:tasks.workunit.client.1.smithi027.stdout:4/608: mknod d5/d11/d89/de2/ce7 0 2024-04-22T15:50:01.991 INFO:tasks.workunit.client.1.smithi027.stdout:7/568: dwrite d0/d4c/f7a [0,4194304] 0 2024-04-22T15:50:01.995 INFO:tasks.workunit.client.1.smithi027.stdout:5/636: creat d0/db6/fcf x:0 0 0 2024-04-22T15:50:02.004 INFO:tasks.workunit.client.1.smithi027.stdout:3/677: mknod d28/d5d/db0/dce/dd4/cdc 0 2024-04-22T15:50:02.005 INFO:tasks.workunit.client.1.smithi027.stdout:3/678: stat d28/d5d/ca7 0 2024-04-22T15:50:02.006 INFO:tasks.workunit.client.1.smithi027.stdout:3/679: chown d28/d4b/d52/d94/dae/fcc 36240 1 2024-04-22T15:50:02.007 INFO:tasks.workunit.client.1.smithi027.stdout:3/680: readlink d28/d5d/db0/ld0 0 2024-04-22T15:50:02.009 INFO:tasks.workunit.client.1.smithi027.stdout:7/569: chown d0/d11/d1c/l34 389064 1 2024-04-22T15:50:02.009 INFO:tasks.workunit.client.1.smithi027.stdout:7/570: readlink d0/l10 0 2024-04-22T15:50:02.012 INFO:tasks.workunit.client.1.smithi027.stdout:5/637: mkdir d0/df/de/d40/d89/dd0 0 2024-04-22T15:50:02.012 INFO:tasks.workunit.client.1.smithi027.stdout:8/538: dwrite d1/d1e/f33 [0,4194304] 0 2024-04-22T15:50:02.013 INFO:tasks.workunit.client.1.smithi027.stdout:5/638: chown d0/df/de/d24/d62/l65 27026455 1 2024-04-22T15:50:02.014 INFO:tasks.workunit.client.1.smithi027.stdout:0/627: rename d0/d34/f81 to d0/d28/d37/fc3 0 2024-04-22T15:50:02.016 INFO:tasks.workunit.client.1.smithi027.stdout:0/628: dread d0/fa8 [0,4194304] 0 2024-04-22T15:50:02.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.013+0000 7f16ebcae700 1 -- 172.21.15.16:0/669451795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16e40ffbc0 msgr2=0x7f16e40fffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:02.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.013+0000 7f16ebcae700 1 --2- 172.21.15.16:0/669451795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16e40ffbc0 0x7f16e40fffe0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f16e0009910 tx=0x7f16e0009c20 comp rx=0 tx=0).stop 2024-04-22T15:50:02.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.013+0000 7f16ebcae700 1 -- 172.21.15.16:0/669451795 shutdown_connections 2024-04-22T15:50:02.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.013+0000 7f16ebcae700 1 --2- 172.21.15.16:0/669451795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16e4100dc0 0x7f16e4101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:02.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.013+0000 7f16ebcae700 1 --2- 172.21.15.16:0/669451795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16e40ffbc0 0x7f16e40fffe0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:02.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.013+0000 7f16ebcae700 1 -- 172.21.15.16:0/669451795 >> 172.21.15.16:0/669451795 conn(0x7f16e40fb160 msgr2=0x7f16e40fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:02.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.013+0000 7f16ebcae700 1 -- 172.21.15.16:0/669451795 shutdown_connections 2024-04-22T15:50:02.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.013+0000 7f16ebcae700 1 -- 172.21.15.16:0/669451795 wait complete. 2024-04-22T15:50:02.017 INFO:tasks.workunit.client.1.smithi027.stdout:9/600: link d0/f44 d0/dc/d12/d58/da6/fc9 0 2024-04-22T15:50:02.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.014+0000 7f16ebcae700 1 Processor -- start 2024-04-22T15:50:02.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.014+0000 7f16ebcae700 1 -- start start 2024-04-22T15:50:02.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.014+0000 7f16ebcae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16e40ffbc0 0x7f16e4069970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:02.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.014+0000 7f16ebcae700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16e4100dc0 0x7f16e419b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:02.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.014+0000 7f16ebcae700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e419bb90 con 0x7f16e40ffbc0 2024-04-22T15:50:02.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.015+0000 7f16ebcae700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e419bcd0 con 0x7f16e4100dc0 2024-04-22T15:50:02.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.015+0000 7f16e9a4a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16e40ffbc0 0x7f16e4069970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:02.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.015+0000 7f16e9a4a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16e40ffbc0 0x7f16e4069970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58376/0 (socket says 172.21.15.16:58376) 2024-04-22T15:50:02.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.015+0000 7f16e9a4a700 1 -- 172.21.15.16:0/1024331935 learned_addr learned my addr 172.21.15.16:0/1024331935 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:50:02.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.015+0000 7f16e9a4a700 1 -- 172.21.15.16:0/1024331935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16e4100dc0 msgr2=0x7f16e419b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T15:50:02.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.015+0000 7f16e9a4a700 1 --2- 172.21.15.16:0/1024331935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16e4100dc0 0x7f16e419b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:02.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.015+0000 7f16e9a4a700 1 -- 172.21.15.16:0/1024331935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e00095f0 con 0x7f16e40ffbc0 2024-04-22T15:50:02.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.016+0000 7f16e9a4a700 1 --2- 172.21.15.16:0/1024331935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16e40ffbc0 0x7f16e4069970 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f16e000bda0 tx=0x7f16e000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:02.020 INFO:tasks.workunit.client.1.smithi027.stdout:9/601: chown d0 1 1 2024-04-22T15:50:02.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.016+0000 7f16daffd700 1 -- 172.21.15.16:0/1024331935 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16e001d070 con 0x7f16e40ffbc0 2024-04-22T15:50:02.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.016+0000 7f16daffd700 1 -- 172.21.15.16:0/1024331935 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f16e0004c50 con 0x7f16e40ffbc0 2024-04-22T15:50:02.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.016+0000 7f16ebcae700 1 -- 172.21.15.16:0/1024331935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16e419bf50 con 0x7f16e40ffbc0 2024-04-22T15:50:02.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.016+0000 7f16daffd700 1 -- 172.21.15.16:0/1024331935 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16e000f690 con 0x7f16e40ffbc0 2024-04-22T15:50:02.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.016+0000 7f16ebcae700 1 -- 172.21.15.16:0/1024331935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16e419c410 con 0x7f16e40ffbc0 2024-04-22T15:50:02.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.018+0000 7f16ebcae700 1 -- 172.21.15.16:0/1024331935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16e418f5b0 con 0x7f16e40ffbc0 2024-04-22T15:50:02.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.018+0000 7f16daffd700 1 -- 172.21.15.16:0/1024331935 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 29) v1 ==== 46697+0+0 (secure 0 0 0) 0x7f16e000f810 con 0x7f16e40ffbc0 2024-04-22T15:50:02.021 INFO:tasks.workunit.client.1.smithi027.stdout:2/584: creat dd/df/d70/d4b/d4e/daf/fb9 x:0 0 0 2024-04-22T15:50:02.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.018+0000 7f16daffd700 1 --2- 172.21.15.16:0/1024331935 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f16d00398c0 0x7f16d003bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:02.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.018+0000 7f16daffd700 1 -- 172.21.15.16:0/1024331935 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f16e004f660 con 0x7f16e40ffbc0 2024-04-22T15:50:02.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.018+0000 7f16e9249700 1 --2- 172.21.15.16:0/1024331935 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f16d00398c0 0x7f16d003bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:02.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.019+0000 7f16e9249700 1 --2- 172.21.15.16:0/1024331935 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f16d00398c0 0x7f16d003bd80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f16e4069ef0 tx=0x7f16d4006cb0 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:02.025 INFO:tasks.workunit.client.1.smithi027.stdout:7/571: creat d0/d4c/d8b/d72/db2/fb7 x:0 0 0 2024-04-22T15:50:02.025 INFO:tasks.workunit.client.1.smithi027.stdout:1/524: dwrite d2/d5f/f7a [0,4194304] 0 2024-04-22T15:50:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.025+0000 7f16daffd700 1 -- 172.21.15.16:0/1024331935 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f16e0020780 con 0x7f16e40ffbc0 2024-04-22T15:50:02.028 INFO:tasks.workunit.client.1.smithi027.stdout:8/539: symlink d1/d3/d1c/d32/d41/d52/d61/d8f/lbb 0 2024-04-22T15:50:02.033 INFO:tasks.workunit.client.1.smithi027.stdout:2/585: dread dd/df/d1b/d96/f9d [0,4194304] 0 2024-04-22T15:50:02.038 INFO:tasks.workunit.client.1.smithi027.stdout:3/681: dread d28/d5d/d62/f66 [0,4194304] 0 2024-04-22T15:50:02.040 INFO:tasks.workunit.client.1.smithi027.stdout:2/586: dread dd/df/d1b/d38/d60/fa2 [0,4194304] 0 2024-04-22T15:50:02.041 INFO:tasks.workunit.client.1.smithi027.stdout:2/587: fsync f7 0 2024-04-22T15:50:02.046 INFO:tasks.workunit.client.1.smithi027.stdout:9/602: rename d0/dc/d12/d58/d6b/d6e/f84 to d0/dc/d12/d30/d5b/d91/da9/fca 0 2024-04-22T15:50:02.047 INFO:tasks.workunit.client.1.smithi027.stdout:2/588: dwrite dd/d43/fa7 [0,4194304] 0 2024-04-22T15:50:02.049 INFO:tasks.workunit.client.1.smithi027.stdout:2/589: read dd/df/d1b/f5f [191062,94838] 0 2024-04-22T15:50:02.051 INFO:tasks.workunit.client.1.smithi027.stdout:1/525: creat d2/d5f/d64/d81/d3d/d44/d9c/d9f/fb5 x:0 0 0 2024-04-22T15:50:02.054 INFO:tasks.workunit.client.1.smithi027.stdout:9/603: dwrite d0/dc/d12/d30/f7a [0,4194304] 0 2024-04-22T15:50:02.055 INFO:tasks.workunit.client.1.smithi027.stdout:9/604: dread - d0/dc/d12/d58/f89 zero size 2024-04-22T15:50:02.063 INFO:tasks.workunit.client.1.smithi027.stdout:1/526: dwrite d2/f7f [0,4194304] 0 2024-04-22T15:50:02.066 INFO:tasks.workunit.client.1.smithi027.stdout:0/629: fdatasync d0/d34/d4e/d85/db3/fb8 0 2024-04-22T15:50:02.070 INFO:tasks.workunit.client.1.smithi027.stdout:1/527: read d2/d5f/d64/d81/f3a [806190,118376] 0 2024-04-22T15:50:02.073 INFO:tasks.workunit.client.1.smithi027.stdout:7/572: mknod d0/d11/cb8 0 2024-04-22T15:50:02.074 INFO:tasks.workunit.client.1.smithi027.stdout:2/590: symlink dd/df/d70/d4b/d52/d9a/da4/dad/lba 0 2024-04-22T15:50:02.075 INFO:tasks.workunit.client.1.smithi027.stdout:2/591: readlink dd/df/d70/la9 0 2024-04-22T15:50:02.075 INFO:tasks.workunit.client.1.smithi027.stdout:9/605: stat d0/dc/d12/d58/d6b/d6e/l1a 0 2024-04-22T15:50:02.082 INFO:tasks.workunit.client.1.smithi027.stdout:3/682: mknod d28/d5d/d62/dcf/cdd 0 2024-04-22T15:50:02.083 INFO:tasks.workunit.client.1.smithi027.stdout:7/573: dwrite d0/d4c/d8b/d72/db2/fac [0,4194304] 0 2024-04-22T15:50:02.085 INFO:tasks.workunit.client.1.smithi027.stdout:1/528: rmdir d2/d5f/d64/d81/d69 39 2024-04-22T15:50:02.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:01 smithi027 ceph-mon[34922]: mgrmap e29: smithi016.rnsnnr(active, since 3s) 2024-04-22T15:50:02.087 INFO:tasks.workunit.client.1.smithi027.stdout:2/592: dwrite f7 [0,4194304] 0 2024-04-22T15:50:02.093 INFO:tasks.workunit.client.1.smithi027.stdout:2/593: dread dd/df/d1b/d38/d60/fa2 [0,4194304] 0 2024-04-22T15:50:02.094 INFO:tasks.workunit.client.1.smithi027.stdout:5/639: write d0/d1a/d22/d72/d86/f75 [1119853,64224] 0 2024-04-22T15:50:02.097 INFO:tasks.workunit.client.1.smithi027.stdout:0/630: creat d0/da3/dae/fc4 x:0 0 0 2024-04-22T15:50:02.101 INFO:tasks.workunit.client.1.smithi027.stdout:8/540: dwrite d1/d3/fa [0,4194304] 0 2024-04-22T15:50:02.105 INFO:tasks.workunit.client.1.smithi027.stdout:8/541: truncate d1/d3/d1c/d28/d42/fb1 167436 0 2024-04-22T15:50:02.107 INFO:tasks.workunit.client.1.smithi027.stdout:9/606: dread d0/dc/de/f24 [0,4194304] 0 2024-04-22T15:50:02.109 INFO:tasks.workunit.client.1.smithi027.stdout:2/594: creat dd/df/d1b/d38/d60/d75/fbb x:0 0 0 2024-04-22T15:50:02.110 INFO:tasks.workunit.client.1.smithi027.stdout:9/607: fdatasync d0/dc/d12/d30/d5b/f83 0 2024-04-22T15:50:02.111 INFO:tasks.workunit.client.1.smithi027.stdout:0/631: chown d0/d28/d37/c47 627538180 1 2024-04-22T15:50:02.111 INFO:tasks.workunit.client.1.smithi027.stdout:8/542: chown d1/d3/d1c/d32/d41/d52/d61/d8f/lbb 16736705 1 2024-04-22T15:50:02.112 INFO:tasks.workunit.client.1.smithi027.stdout:1/529: rmdir d2/d5f/d64/d81/d69 39 2024-04-22T15:50:02.118 INFO:tasks.workunit.client.1.smithi027.stdout:3/683: dread d28/d5d/fa8 [0,4194304] 0 2024-04-22T15:50:02.118 INFO:tasks.workunit.client.1.smithi027.stdout:1/530: write d2/d30/fa3 [2805072,74592] 0 2024-04-22T15:50:02.118 INFO:tasks.workunit.client.1.smithi027.stdout:3/684: write d28/d4b/f5b [1737581,31734] 0 2024-04-22T15:50:02.118 INFO:tasks.workunit.client.1.smithi027.stdout:7/574: rename d0/d3f/c53 to d0/d4c/d8b/d72/db2/cb9 0 2024-04-22T15:50:02.119 INFO:tasks.workunit.client.1.smithi027.stdout:2/595: symlink dd/df/d70/d4b/lbc 0 2024-04-22T15:50:02.120 INFO:tasks.workunit.client.1.smithi027.stdout:8/543: mkdir d1/d1e/dbc 0 2024-04-22T15:50:02.121 INFO:tasks.workunit.client.1.smithi027.stdout:9/608: mknod d0/dc/d12/d30/d5b/d91/ccb 0 2024-04-22T15:50:02.122 INFO:tasks.workunit.client.1.smithi027.stdout:9/609: fsync d0/dc/f8e 0 2024-04-22T15:50:02.124 INFO:tasks.workunit.client.1.smithi027.stdout:8/544: dread d1/d3/d1c/d43/d5f/f9e [4194304,4194304] 0 2024-04-22T15:50:02.124 INFO:tasks.workunit.client.1.smithi027.stdout:9/610: chown d0/dc/d12/d58/d6b/d6e/fbd 430057 1 2024-04-22T15:50:02.126 INFO:tasks.workunit.client.1.smithi027.stdout:7/575: fdatasync d0/f14 0 2024-04-22T15:50:02.131 INFO:tasks.workunit.client.1.smithi027.stdout:1/531: symlink d2/d30/lb6 0 2024-04-22T15:50:02.131 INFO:tasks.workunit.client.1.smithi027.stdout:7/576: dread d0/d11/d1c/d2c/f38 [0,4194304] 0 2024-04-22T15:50:02.133 INFO:tasks.workunit.client.1.smithi027.stdout:7/577: chown d0/d4c/d8b/d92/da3/da8/f23 13030373 1 2024-04-22T15:50:02.133 INFO:tasks.workunit.client.1.smithi027.stdout:7/578: fsync d0/f13 0 2024-04-22T15:50:02.134 INFO:tasks.workunit.client.1.smithi027.stdout:8/545: creat d1/d3/d36/d57/fbd x:0 0 0 2024-04-22T15:50:02.135 INFO:tasks.workunit.client.1.smithi027.stdout:9/611: readlink d0/dc/d12/d58/d6b/d6e/lc1 0 2024-04-22T15:50:02.137 INFO:tasks.workunit.client.1.smithi027.stdout:0/632: link d0/d9d/db7/laa d0/d34/d4e/lc5 0 2024-04-22T15:50:02.140 INFO:tasks.workunit.client.1.smithi027.stdout:5/640: rename d0/df/de/d24/d1c/d8f/c92 to d0/df/de/d24/d1c/d3e/cd1 0 2024-04-22T15:50:02.143 INFO:tasks.workunit.client.1.smithi027.stdout:8/546: read d1/d1e/d47/d53/d76/fad [2959360,100660] 0 2024-04-22T15:50:02.144 INFO:tasks.workunit.client.1.smithi027.stdout:7/579: creat d0/d11/d3a/d46/d94/fba x:0 0 0 2024-04-22T15:50:02.146 INFO:tasks.workunit.client.1.smithi027.stdout:3/685: rmdir d28/d5d/dd7 0 2024-04-22T15:50:02.151 INFO:tasks.workunit.client.1.smithi027.stdout:1/532: symlink d2/d36/daa/lb7 0 2024-04-22T15:50:02.158 INFO:tasks.workunit.client.1.smithi027.stdout:8/547: mkdir d1/d3/d95/dbe 0 2024-04-22T15:50:02.159 INFO:tasks.workunit.client.1.smithi027.stdout:0/633: dwrite d0/d9d/fc0 [0,4194304] 0 2024-04-22T15:50:02.162 INFO:tasks.workunit.client.1.smithi027.stdout:7/580: dread d0/d11/d1c/d31/f47 [0,4194304] 0 2024-04-22T15:50:02.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:01 smithi016 ceph-mon[26620]: mgrmap e29: smithi016.rnsnnr(active, since 3s) 2024-04-22T15:50:02.168 INFO:tasks.workunit.client.1.smithi027.stdout:2/596: getdents dd/df/d70/d4b/d4e/daf 0 2024-04-22T15:50:02.170 INFO:tasks.workunit.client.1.smithi027.stdout:2/597: chown dd/df/d1b/d38/f6c 397 1 2024-04-22T15:50:02.171 INFO:tasks.workunit.client.1.smithi027.stdout:2/598: write dd/d43/f95 [959440,37601] 0 2024-04-22T15:50:02.175 INFO:tasks.workunit.client.1.smithi027.stdout:5/641: fdatasync d0/df/d10/f60 0 2024-04-22T15:50:02.176 INFO:tasks.workunit.client.1.smithi027.stdout:5/642: readlink d0/d1a/d22/l96 0 2024-04-22T15:50:02.179 INFO:tasks.workunit.client.1.smithi027.stdout:8/548: creat d1/d3/d1c/d32/d41/d52/fbf x:0 0 0 2024-04-22T15:50:02.179 INFO:tasks.workunit.client.1.smithi027.stdout:3/686: write d28/d4b/f57 [1177322,19499] 0 2024-04-22T15:50:02.187 INFO:tasks.workunit.client.1.smithi027.stdout:9/612: rmdir d0/dc/d12/d58/da6/dbb 0 2024-04-22T15:50:02.190 INFO:tasks.workunit.client.1.smithi027.stdout:1/533: dwrite d2/d5f/d64/d81/d3d/d44/f63 [0,4194304] 0 2024-04-22T15:50:02.196 INFO:tasks.workunit.client.1.smithi027.stdout:0/634: truncate d0/fc 3335714 0 2024-04-22T15:50:02.196 INFO:tasks.workunit.client.1.smithi027.stdout:3/687: dread d28/f48 [0,4194304] 0 2024-04-22T15:50:02.199 INFO:tasks.workunit.client.1.smithi027.stdout:7/581: mknod d0/d11/d1c/cbb 0 2024-04-22T15:50:02.199 INFO:tasks.workunit.client.1.smithi027.stdout:3/688: chown d28/d4b/d52/f7a 165901 1 2024-04-22T15:50:02.201 INFO:tasks.workunit.client.1.smithi027.stdout:7/582: stat d0/d4c/d8b/d72/db2 0 2024-04-22T15:50:02.202 INFO:tasks.workunit.client.1.smithi027.stdout:1/534: dread d2/d30/f41 [0,4194304] 0 2024-04-22T15:50:02.205 INFO:tasks.workunit.client.1.smithi027.stdout:5/643: dwrite d0/d1a/d22/d26/f48 [0,4194304] 0 2024-04-22T15:50:02.205 INFO:tasks.workunit.client.1.smithi027.stdout:1/535: fdatasync d2/d5f/d64/d81/d7e/fa1 0 2024-04-22T15:50:02.207 INFO:tasks.workunit.client.1.smithi027.stdout:2/599: unlink dd/df/d70/d4b/d52/lb5 0 2024-04-22T15:50:02.208 INFO:tasks.workunit.client.1.smithi027.stdout:5/644: chown d0/db6/lb7 133273899 1 2024-04-22T15:50:02.210 INFO:tasks.workunit.client.1.smithi027.stdout:5/645: chown d0/d1a/d22/d26/d32/d79/f87 448 1 2024-04-22T15:50:02.216 INFO:tasks.workunit.client.1.smithi027.stdout:8/549: mknod d1/d1e/d47/d9d/cc0 0 2024-04-22T15:50:02.220 INFO:tasks.workunit.client.1.smithi027.stdout:3/689: mkdir d28/d4b/d52/d94/dde 0 2024-04-22T15:50:02.221 INFO:tasks.workunit.client.1.smithi027.stdout:7/583: mknod d0/d49/d9f/cbc 0 2024-04-22T15:50:02.227 INFO:tasks.workunit.client.1.smithi027.stdout:1/536: creat d2/d36/d74/fb8 x:0 0 0 2024-04-22T15:50:02.227 INFO:tasks.workunit.client.1.smithi027.stdout:8/550: dwrite d1/d1e/d47/d53/f78 [0,4194304] 0 2024-04-22T15:50:02.229 INFO:tasks.workunit.client.1.smithi027.stdout:2/600: creat dd/df/d1b/d96/fbd x:0 0 0 2024-04-22T15:50:02.230 INFO:tasks.workunit.client.1.smithi027.stdout:8/551: write d1/f21 [1272092,5639] 0 2024-04-22T15:50:02.231 INFO:tasks.workunit.client.1.smithi027.stdout:1/537: fsync d2/f75 0 2024-04-22T15:50:02.233 INFO:tasks.workunit.client.1.smithi027.stdout:5/646: mknod d0/df/de/d24/d1c/d3e/d55/cd2 0 2024-04-22T15:50:02.235 INFO:tasks.workunit.client.1.smithi027.stdout:9/613: creat d0/dc/d12/d30/d99/db4/fcc x:0 0 0 2024-04-22T15:50:02.236 INFO:tasks.workunit.client.1.smithi027.stdout:1/538: dread - d2/d5f/d64/d81/d3d/d82/d85/fa4 zero size 2024-04-22T15:50:02.240 INFO:tasks.workunit.client.1.smithi027.stdout:3/690: dwrite d28/d4b/f7d [4194304,4194304] 0 2024-04-22T15:50:02.241 INFO:tasks.workunit.client.1.smithi027.stdout:1/539: fsync d2/d36/f4b 0 2024-04-22T15:50:02.246 INFO:tasks.workunit.client.1.smithi027.stdout:0/635: creat d0/d2a/fc6 x:0 0 0 2024-04-22T15:50:02.247 INFO:tasks.workunit.client.1.smithi027.stdout:0/636: readlink d0/d28/d5f/l88 0 2024-04-22T15:50:02.251 INFO:tasks.workunit.client.1.smithi027.stdout:8/552: unlink d1/d3/d1c/d28/d42/c44 0 2024-04-22T15:50:02.256 INFO:tasks.workunit.client.1.smithi027.stdout:9/614: rmdir d0/dc/de/db5 39 2024-04-22T15:50:02.258 INFO:tasks.workunit.client.1.smithi027.stdout:7/584: truncate d0/f39 934272 0 2024-04-22T15:50:02.273 INFO:tasks.workunit.client.1.smithi027.stdout:0/637: symlink d0/d2a/d91/lc7 0 2024-04-22T15:50:02.277 INFO:tasks.workunit.client.1.smithi027.stdout:2/601: write dd/df/d70/f61 [197580,101149] 0 2024-04-22T15:50:02.278 INFO:tasks.workunit.client.1.smithi027.stdout:7/585: symlink d0/d4c/d8b/d92/da3/lbd 0 2024-04-22T15:50:02.280 INFO:tasks.workunit.client.1.smithi027.stdout:7/586: write d0/d11/d1c/d31/d56/f58 [737354,29337] 0 2024-04-22T15:50:02.281 INFO:tasks.workunit.client.1.smithi027.stdout:1/540: mkdir d2/db9 0 2024-04-22T15:50:02.285 INFO:tasks.workunit.client.1.smithi027.stdout:0/638: symlink d0/d2a/d91/lc8 0 2024-04-22T15:50:02.289 INFO:tasks.workunit.client.1.smithi027.stdout:7/587: dwrite d0/d4c/d8b/d92/da3/da8/f23 [0,4194304] 0 2024-04-22T15:50:02.290 INFO:tasks.workunit.client.1.smithi027.stdout:7/588: write d0/d49/d9f/f9b [1454017,114962] 0 2024-04-22T15:50:02.295 INFO:tasks.workunit.client.1.smithi027.stdout:9/615: symlink d0/dc/d12/d58/lcd 0 2024-04-22T15:50:02.297 INFO:tasks.workunit.client.1.smithi027.stdout:2/602: creat dd/df/d70/d4b/d52/fbe x:0 0 0 2024-04-22T15:50:02.300 INFO:tasks.workunit.client.1.smithi027.stdout:5/647: dwrite d0/df/de/f18 [0,4194304] 0 2024-04-22T15:50:02.302 INFO:tasks.workunit.client.1.smithi027.stdout:8/553: rename d1/d3/d1c/d43/d82/dac to d1/d1e/d47/dc1 0 2024-04-22T15:50:02.303 INFO:tasks.workunit.client.1.smithi027.stdout:8/554: chown d1/d3/d1c/d32/d41/fab 8 1 2024-04-22T15:50:02.305 INFO:tasks.workunit.client.1.smithi027.stdout:0/639: dread d0/d9d/db7/fa9 [0,4194304] 0 2024-04-22T15:50:02.306 INFO:tasks.workunit.client.1.smithi027.stdout:7/589: mknod d0/d49/d9f/cbe 0 2024-04-22T15:50:02.308 INFO:tasks.workunit.client.1.smithi027.stdout:2/603: read - dd/df/d1b/d1f/f30 zero size 2024-04-22T15:50:02.309 INFO:tasks.workunit.client.1.smithi027.stdout:1/541: symlink d2/d5f/d64/d81/d69/lba 0 2024-04-22T15:50:02.310 INFO:tasks.workunit.client.1.smithi027.stdout:5/648: symlink d0/db6/ld3 0 2024-04-22T15:50:02.315 INFO:tasks.workunit.client.1.smithi027.stdout:2/604: readlink dd/df/l79 0 2024-04-22T15:50:02.315 INFO:tasks.workunit.client.1.smithi027.stdout:1/542: mknod d2/d5f/d64/d81/d69/cbb 0 2024-04-22T15:50:02.318 INFO:tasks.workunit.client.1.smithi027.stdout:1/543: creat d2/d36/d74/fbc x:0 0 0 2024-04-22T15:50:02.320 INFO:tasks.workunit.client.1.smithi027.stdout:1/544: chown d2/d5f/d64/d81/f3a 493997238 1 2024-04-22T15:50:02.322 INFO:tasks.workunit.client.1.smithi027.stdout:2/605: unlink dd/df/l73 0 2024-04-22T15:50:02.327 INFO:tasks.workunit.client.1.smithi027.stdout:1/545: rename d2/d36/d74/c90 to d2/d5f/d64/d81/d7e/cbd 0 2024-04-22T15:50:02.330 INFO:tasks.workunit.client.1.smithi027.stdout:2/606: mknod dd/df/d70/d4b/d52/d9a/da4/cbf 0 2024-04-22T15:50:02.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.328+0000 7f16ebcae700 1 -- 172.21.15.16:0/1024331935 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f16e40611d0 con 0x7f16d00398c0 2024-04-22T15:50:02.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.330+0000 7f16daffd700 1 -- 172.21.15.16:0/1024331935 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f16e40611d0 con 0x7f16d00398c0 2024-04-22T15:50:02.336 INFO:tasks.workunit.client.1.smithi027.stdout:1/546: write d2/d5f/d64/d81/d3d/d44/d9c/d9f/fac [1271621,129814] 0 2024-04-22T15:50:02.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.333+0000 7f16ebcae700 1 -- 172.21.15.16:0/1024331935 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f16d00398c0 msgr2=0x7f16d003bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:02.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.333+0000 7f16ebcae700 1 --2- 172.21.15.16:0/1024331935 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f16d00398c0 0x7f16d003bd80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f16e4069ef0 tx=0x7f16d4006cb0 comp rx=0 tx=0).stop 2024-04-22T15:50:02.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.333+0000 7f16ebcae700 1 -- 172.21.15.16:0/1024331935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16e40ffbc0 msgr2=0x7f16e4069970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:02.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.333+0000 7f16ebcae700 1 --2- 172.21.15.16:0/1024331935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16e40ffbc0 0x7f16e4069970 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f16e000bda0 tx=0x7f16e000be80 comp rx=0 tx=0).stop 2024-04-22T15:50:02.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.333+0000 7f16ebcae700 1 -- 172.21.15.16:0/1024331935 shutdown_connections 2024-04-22T15:50:02.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.333+0000 7f16ebcae700 1 --2- 172.21.15.16:0/1024331935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16e40ffbc0 0x7f16e4069970 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:02.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.333+0000 7f16ebcae700 1 --2- 172.21.15.16:0/1024331935 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f16d00398c0 0x7f16d003bd80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:02.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.333+0000 7f16ebcae700 1 --2- 172.21.15.16:0/1024331935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16e4100dc0 0x7f16e419b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:02.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.333+0000 7f16ebcae700 1 -- 172.21.15.16:0/1024331935 >> 172.21.15.16:0/1024331935 conn(0x7f16e40fb160 msgr2=0x7f16e41040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:02.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.333+0000 7f16ebcae700 1 -- 172.21.15.16:0/1024331935 shutdown_connections 2024-04-22T15:50:02.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.333+0000 7f16ebcae700 1 -- 172.21.15.16:0/1024331935 wait complete. 2024-04-22T15:50:02.338 INFO:tasks.workunit.client.1.smithi027.stdout:2/607: unlink dd/df/fb0 0 2024-04-22T15:50:02.339 INFO:tasks.workunit.client.1.smithi027.stdout:1/547: rmdir d2/d5f/d64/d81/d3d/d82 39 2024-04-22T15:50:02.341 INFO:tasks.workunit.client.1.smithi027.stdout:7/590: write d0/d4c/d8b/d92/da3/da8/f3d [48518,56910] 0 2024-04-22T15:50:02.341 INFO:tasks.workunit.client.1.smithi027.stdout:0/640: write d0/d28/d5f/f73 [388337,110079] 0 2024-04-22T15:50:02.343 INFO:tasks.workunit.client.1.smithi027.stdout:2/608: creat dd/df/d70/d4b/d52/d9a/da4/dad/fc0 x:0 0 0 2024-04-22T15:50:02.343 INFO:tasks.workunit.client.1.smithi027.stdout:5/649: write d0/d1a/d22/d26/d32/f67 [1944205,45798] 0 2024-04-22T15:50:02.347 INFO:tasks.workunit.client.1.smithi027.stdout:0/641: write d0/d34/d4e/fab [112191,122516] 0 2024-04-22T15:50:02.349 INFO:tasks.workunit.client.1.smithi027.stdout:8/555: dwrite d1/d3/d1c/d43/f71 [0,4194304] 0 2024-04-22T15:50:02.350 INFO:tasks.workunit.client.1.smithi027.stdout:0/642: dread - d0/d34/d4e/d85/db3/fb8 zero size 2024-04-22T15:50:02.351 INFO:tasks.workunit.client.1.smithi027.stdout:0/643: chown d0/d28/f5b 16034 1 2024-04-22T15:50:02.351 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:50:02.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.481+0000 7f61cccc1700 1 -- 172.21.15.16:0/2436582423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c80ffa70 msgr2=0x7f61c80ffe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:02.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.481+0000 7f61cccc1700 1 --2- 172.21.15.16:0/2436582423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c80ffa70 0x7f61c80ffe90 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f61b0009910 tx=0x7f61b0009c20 comp rx=0 tx=0).stop 2024-04-22T15:50:02.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.481+0000 7f61cccc1700 1 -- 172.21.15.16:0/2436582423 shutdown_connections 2024-04-22T15:50:02.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.481+0000 7f61cccc1700 1 --2- 172.21.15.16:0/2436582423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61c8100c70 0x7f61c81010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:02.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.481+0000 7f61cccc1700 1 --2- 172.21.15.16:0/2436582423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c80ffa70 0x7f61c80ffe90 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:02.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.481+0000 7f61cccc1700 1 -- 172.21.15.16:0/2436582423 >> 172.21.15.16:0/2436582423 conn(0x7f61c80fafd0 msgr2=0x7f61c80fd450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:02.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.481+0000 7f61cccc1700 1 -- 172.21.15.16:0/2436582423 shutdown_connections 2024-04-22T15:50:02.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.481+0000 7f61cccc1700 1 -- 172.21.15.16:0/2436582423 wait complete. 2024-04-22T15:50:02.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.482+0000 7f61cccc1700 1 Processor -- start 2024-04-22T15:50:02.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.482+0000 7f61cccc1700 1 -- start start 2024-04-22T15:50:02.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.482+0000 7f61cccc1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61c80ffa70 0x7f61c8103a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:02.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.482+0000 7f61cccc1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c8100c70 0x7f61c8102050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:02.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.482+0000 7f61cccc1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c8102590 con 0x7f61c8100c70 2024-04-22T15:50:02.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.482+0000 7f61cccc1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c81026d0 con 0x7f61c80ffa70 2024-04-22T15:50:02.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.482+0000 7f61c5d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c8100c70 0x7f61c8102050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:02.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.482+0000 7f61c5d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c8100c70 0x7f61c8102050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58396/0 (socket says 172.21.15.16:58396) 2024-04-22T15:50:02.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.482+0000 7f61c5d9b700 1 -- 172.21.15.16:0/2736880068 learned_addr learned my addr 172.21.15.16:0/2736880068 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:50:02.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.483+0000 7f61c659c700 1 --2- 172.21.15.16:0/2736880068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61c80ffa70 0x7f61c8103a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:02.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.483+0000 7f61c5d9b700 1 -- 172.21.15.16:0/2736880068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61c80ffa70 msgr2=0x7f61c8103a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:02.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.483+0000 7f61c5d9b700 1 --2- 172.21.15.16:0/2736880068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61c80ffa70 0x7f61c8103a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:02.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.483+0000 7f61c5d9b700 1 -- 172.21.15.16:0/2736880068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b00095f0 con 0x7f61c8100c70 2024-04-22T15:50:02.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.483+0000 7f61c5d9b700 1 --2- 172.21.15.16:0/2736880068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c8100c70 0x7f61c8102050 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f61b800d900 tx=0x7f61b800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:02.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.483+0000 7f61bf7fe700 1 -- 172.21.15.16:0/2736880068 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61b80041f0 con 0x7f61c8100c70 2024-04-22T15:50:02.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.483+0000 7f61bf7fe700 1 -- 172.21.15.16:0/2736880068 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f61b8004ce0 con 0x7f61c8100c70 2024-04-22T15:50:02.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.483+0000 7f61cccc1700 1 -- 172.21.15.16:0/2736880068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61c81029b0 con 0x7f61c8100c70 2024-04-22T15:50:02.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.483+0000 7f61bf7fe700 1 -- 172.21.15.16:0/2736880068 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61b800b750 con 0x7f61c8100c70 2024-04-22T15:50:02.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.483+0000 7f61cccc1700 1 -- 172.21.15.16:0/2736880068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c819e380 con 0x7f61c8100c70 2024-04-22T15:50:02.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.484+0000 7f61bf7fe700 1 -- 172.21.15.16:0/2736880068 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 29) v1 ==== 46697+0+0 (secure 0 0 0) 0x7f61b800b9c0 con 0x7f61c8100c70 2024-04-22T15:50:02.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.484+0000 7f61cccc1700 1 -- 172.21.15.16:0/2736880068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61a8005320 con 0x7f61c8100c70 2024-04-22T15:50:02.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.484+0000 7f61bf7fe700 1 --2- 172.21.15.16:0/2736880068 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f61b40398c0 0x7f61b403bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:02.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.484+0000 7f61bf7fe700 1 -- 172.21.15.16:0/2736880068 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f61b804dba0 con 0x7f61c8100c70 2024-04-22T15:50:02.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.485+0000 7f61c659c700 1 --2- 172.21.15.16:0/2736880068 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f61b40398c0 0x7f61b403bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:02.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.485+0000 7f61c659c700 1 --2- 172.21.15.16:0/2736880068 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f61b40398c0 0x7f61b403bd80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f61b0006010 tx=0x7f61b0005ec0 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:02.490 INFO:tasks.workunit.client.1.smithi027.stdout:5/650: creat d0/d1a/d22/d26/d32/d88/d90/fd4 x:0 0 0 2024-04-22T15:50:02.491 INFO:tasks.workunit.client.1.smithi027.stdout:8/556: creat d1/d1e/d47/d53/fc2 x:0 0 0 2024-04-22T15:50:02.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.490+0000 7f61bf7fe700 1 -- 172.21.15.16:0/2736880068 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f61b8022710 con 0x7f61c8100c70 2024-04-22T15:50:02.499 INFO:tasks.workunit.client.1.smithi027.stdout:7/591: mknod d0/d11/d1c/d2c/d96/db3/cbf 0 2024-04-22T15:50:02.501 INFO:tasks.workunit.client.1.smithi027.stdout:8/557: write d1/d3/f59 [4827882,33904] 0 2024-04-22T15:50:02.505 INFO:tasks.workunit.client.1.smithi027.stdout:1/548: rename d2/d36/c94 to d2/d5f/d64/d81/d3d/d82/cbe 0 2024-04-22T15:50:02.507 INFO:tasks.workunit.client.1.smithi027.stdout:2/609: creat dd/df/d1b/fc1 x:0 0 0 2024-04-22T15:50:02.509 INFO:tasks.workunit.client.1.smithi027.stdout:5/651: mkdir d0/df/de/d24/d1c/d3e/dd5 0 2024-04-22T15:50:02.510 INFO:tasks.workunit.client.1.smithi027.stdout:8/558: symlink d1/d3/d1c/d28/d96/lc3 0 2024-04-22T15:50:02.513 INFO:tasks.workunit.client.1.smithi027.stdout:1/549: rename d2/d5f/d64/d81/d3d/d44/f63 to d2/d36/daa/fbf 0 2024-04-22T15:50:02.514 INFO:tasks.workunit.client.1.smithi027.stdout:7/592: symlink d0/d11/lc0 0 2024-04-22T15:50:02.515 INFO:tasks.workunit.client.1.smithi027.stdout:2/610: creat dd/df/d70/d4b/d52/d9a/fc2 x:0 0 0 2024-04-22T15:50:02.518 INFO:tasks.workunit.client.1.smithi027.stdout:0/644: getdents d0/d2a/d38 0 2024-04-22T15:50:02.519 INFO:tasks.workunit.client.1.smithi027.stdout:8/559: fsync d1/d3/d1c/d32/d41/f7a 0 2024-04-22T15:50:02.530 INFO:tasks.workunit.client.1.smithi027.stdout:2/611: write dd/f1c [4859411,37170] 0 2024-04-22T15:50:02.530 INFO:tasks.workunit.client.1.smithi027.stdout:2/612: fdatasync dd/f1c 0 2024-04-22T15:50:02.533 INFO:tasks.workunit.client.1.smithi027.stdout:5/652: rmdir d0/d97/dbf/dc3 0 2024-04-22T15:50:02.535 INFO:tasks.workunit.client.1.smithi027.stdout:8/560: dread d1/d3/d1c/d28/f51 [0,4194304] 0 2024-04-22T15:50:02.538 INFO:tasks.workunit.client.1.smithi027.stdout:7/593: dwrite d0/f13 [0,4194304] 0 2024-04-22T15:50:02.539 INFO:tasks.workunit.client.1.smithi027.stdout:2/613: rmdir dd/df/d1b/d1f 39 2024-04-22T15:50:02.540 INFO:tasks.workunit.client.1.smithi027.stdout:2/614: readlink dd/l42 0 2024-04-22T15:50:02.541 INFO:tasks.workunit.client.1.smithi027.stdout:0/645: getdents d0/d34 0 2024-04-22T15:50:02.543 INFO:tasks.workunit.client.1.smithi027.stdout:0/646: dread - d0/d2a/d58/fba zero size 2024-04-22T15:50:02.546 INFO:tasks.workunit.client.1.smithi027.stdout:8/561: unlink d1/d3/f99 0 2024-04-22T15:50:02.548 INFO:tasks.workunit.client.1.smithi027.stdout:7/594: symlink d0/d4c/d8b/lc1 0 2024-04-22T15:50:02.548 INFO:tasks.workunit.client.1.smithi027.stdout:0/647: dwrite d0/f12 [0,4194304] 0 2024-04-22T15:50:02.551 INFO:tasks.workunit.client.1.smithi027.stdout:2/615: creat dd/df/d1b/d96/fc3 x:0 0 0 2024-04-22T15:50:02.552 INFO:tasks.workunit.client.1.smithi027.stdout:2/616: stat dd/df/d70/d4b/d52/d9a/da4/dad/fc0 0 2024-04-22T15:50:02.553 INFO:tasks.workunit.client.1.smithi027.stdout:2/617: write dd/d43/fa7 [4127731,3495] 0 2024-04-22T15:50:02.558 INFO:tasks.workunit.client.1.smithi027.stdout:5/653: write d0/df/de/d24/d1c/d3e/d55/f9c [1104677,68845] 0 2024-04-22T15:50:02.576 INFO:tasks.workunit.client.1.smithi027.stdout:2/618: write dd/df/d1b/d38/d60/f65 [2974466,126691] 0 2024-04-22T15:50:02.576 INFO:tasks.workunit.client.1.smithi027.stdout:5/654: write d0/df/de/d40/f6a [5082552,35642] 0 2024-04-22T15:50:02.580 INFO:tasks.workunit.client.1.smithi027.stdout:5/655: mkdir d0/d1a/d22/d26/d32/d79/dd6 0 2024-04-22T15:50:02.582 INFO:tasks.workunit.client.1.smithi027.stdout:5/656: truncate d0/df/de/d24/d1c/fcb 494693 0 2024-04-22T15:50:02.586 INFO:tasks.workunit.client.1.smithi027.stdout:7/595: rename d0/d11/d1c/d2c/f4f to d0/d11/d1c/d2c/fc2 0 2024-04-22T15:50:02.587 INFO:tasks.workunit.client.1.smithi027.stdout:0/648: getdents d0/d2a/d5e/d8b 0 2024-04-22T15:50:02.589 INFO:tasks.workunit.client.1.smithi027.stdout:5/657: mkdir d0/d97/dd7 0 2024-04-22T15:50:02.589 INFO:tasks.workunit.client.1.smithi027.stdout:5/658: readlink d0/df/d10/l42 0 2024-04-22T15:50:02.596 INFO:tasks.workunit.client.1.smithi027.stdout:7/596: rename d0/d3f/f7b to d0/d11/d1c/d31/d62/fc3 0 2024-04-22T15:50:02.597 INFO:tasks.workunit.client.1.smithi027.stdout:0/649: creat d0/d34/fc9 x:0 0 0 2024-04-22T15:50:02.606 INFO:tasks.workunit.client.1.smithi027.stdout:5/659: dwrite d0/df/de/d40/d89/fcc [0,4194304] 0 2024-04-22T15:50:02.610 INFO:tasks.workunit.client.1.smithi027.stdout:2/619: getdents dd/df/d70/d4b/d52/d9a 0 2024-04-22T15:50:02.612 INFO:tasks.workunit.client.1.smithi027.stdout:5/660: mknod d0/d1a/d22/d72/cd8 0 2024-04-22T15:50:02.614 INFO:tasks.workunit.client.1.smithi027.stdout:7/597: rename d0/f39 to d0/d11/d1c/d2c/fc4 0 2024-04-22T15:50:02.618 INFO:tasks.workunit.client.1.smithi027.stdout:0/650: rename d0/d9d/lb2 to d0/d34/d59/lca 0 2024-04-22T15:50:02.620 INFO:tasks.workunit.client.1.smithi027.stdout:7/598: symlink d0/d11/d3a/d46/d94/d5a/lc5 0 2024-04-22T15:50:02.625 INFO:tasks.workunit.client.1.smithi027.stdout:0/651: creat d0/d2a/d6a/fcb x:0 0 0 2024-04-22T15:50:02.626 INFO:tasks.workunit.client.1.smithi027.stdout:2/620: dwrite dd/df/d1b/d38/f6c [0,4194304] 0 2024-04-22T15:50:02.628 INFO:tasks.workunit.client.1.smithi027.stdout:2/621: dread - dd/df/d1b/fab zero size 2024-04-22T15:50:02.629 INFO:tasks.workunit.client.1.smithi027.stdout:2/622: readlink dd/df/lb8 0 2024-04-22T15:50:02.651 INFO:tasks.workunit.client.1.smithi027.stdout:7/599: rmdir d0/d4c/d8b/d92/da3 39 2024-04-22T15:50:02.656 INFO:tasks.workunit.client.1.smithi027.stdout:7/600: creat d0/d11/d1c/d31/d56/fc6 x:0 0 0 2024-04-22T15:50:02.656 INFO:tasks.workunit.client.1.smithi027.stdout:0/652: link d0/d2a/c51 d0/d2a/d6a/d8e/ccc 0 2024-04-22T15:50:02.658 INFO:tasks.workunit.client.1.smithi027.stdout:4/609: sync 2024-04-22T15:50:02.658 INFO:tasks.workunit.client.1.smithi027.stdout:6/702: sync 2024-04-22T15:50:02.658 INFO:tasks.workunit.client.1.smithi027.stdout:3/691: sync 2024-04-22T15:50:02.665 INFO:tasks.workunit.client.1.smithi027.stdout:2/623: dread dd/f19 [0,4194304] 0 2024-04-22T15:50:02.672 INFO:tasks.workunit.client.1.smithi027.stdout:2/624: symlink dd/df/d1b/d38/d60/lc4 0 2024-04-22T15:50:02.675 INFO:tasks.workunit.client.1.smithi027.stdout:7/601: getdents d0/d11/d1c 0 2024-04-22T15:50:02.679 INFO:tasks.workunit.client.1.smithi027.stdout:7/602: rmdir d0/d4c/d8b/d92/da3/da8 39 2024-04-22T15:50:02.679 INFO:tasks.workunit.client.1.smithi027.stdout:2/625: dread dd/df/d1b/d1f/f36 [0,4194304] 0 2024-04-22T15:50:02.683 INFO:tasks.workunit.client.1.smithi027.stdout:7/603: dread d0/d4c/f7a [0,4194304] 0 2024-04-22T15:50:02.683 INFO:tasks.workunit.client.1.smithi027.stdout:2/626: mkdir dd/df/d1b/d96/dc5 0 2024-04-22T15:50:02.685 INFO:tasks.workunit.client.1.smithi027.stdout:4/610: write d5/d45/d5a/fb0 [595754,54079] 0 2024-04-22T15:50:02.687 INFO:tasks.workunit.client.1.smithi027.stdout:7/604: mkdir d0/d4c/d8b/dc7 0 2024-04-22T15:50:02.688 INFO:tasks.workunit.client.1.smithi027.stdout:6/703: truncate d5/dc/dd/d79/d7b/f4e 4120416 0 2024-04-22T15:50:02.692 INFO:tasks.workunit.client.1.smithi027.stdout:2/627: rename dd/df/d1b/d38/f6c to dd/df/d70/d4b/d52/d9a/da4/fc6 0 2024-04-22T15:50:02.692 INFO:tasks.workunit.client.1.smithi027.stdout:0/653: truncate d0/d34/f5a 3039116 0 2024-04-22T15:50:02.694 INFO:tasks.workunit.client.1.smithi027.stdout:4/611: creat d5/d45/db3/fe8 x:0 0 0 2024-04-22T15:50:02.695 INFO:tasks.workunit.client.1.smithi027.stdout:4/612: fdatasync d5/d11/d89/f70 0 2024-04-22T15:50:02.697 INFO:tasks.workunit.client.1.smithi027.stdout:6/704: creat d5/db/dcd/ff3 x:0 0 0 2024-04-22T15:50:02.698 INFO:tasks.workunit.client.1.smithi027.stdout:6/705: chown d5/laa 453352285 1 2024-04-22T15:50:02.698 INFO:tasks.workunit.client.1.smithi027.stdout:6/706: dread - d5/db/dcd/ff3 zero size 2024-04-22T15:50:02.700 INFO:tasks.workunit.client.1.smithi027.stdout:0/654: symlink d0/da3/lcd 0 2024-04-22T15:50:02.702 INFO:tasks.workunit.client.1.smithi027.stdout:2/628: mkdir dd/df/d70/d4b/d52/d9a/da4/dc7 0 2024-04-22T15:50:02.702 INFO:tasks.workunit.client.1.smithi027.stdout:0/655: read d0/d34/d59/f5d [3349080,100987] 0 2024-04-22T15:50:02.703 INFO:tasks.workunit.client.1.smithi027.stdout:2/629: dread - dd/df/d70/d5a/f7e zero size 2024-04-22T15:50:02.711 INFO:tasks.workunit.client.1.smithi027.stdout:7/605: write d0/d11/d1c/d31/f5f [52805,57342] 0 2024-04-22T15:50:02.714 INFO:tasks.workunit.client.1.smithi027.stdout:6/707: mknod d5/cf4 0 2024-04-22T15:50:02.715 INFO:tasks.workunit.client.1.smithi027.stdout:6/708: chown d5/dc/d99/c93 224082 1 2024-04-22T15:50:02.718 INFO:tasks.workunit.client.1.smithi027.stdout:7/606: symlink d0/d4c/d4e/lc8 0 2024-04-22T15:50:02.722 INFO:tasks.workunit.client.1.smithi027.stdout:0/656: rename d0/d2a/d38/l87 to d0/d34/d4e/d85/db3/lce 0 2024-04-22T15:50:02.722 INFO:tasks.workunit.client.1.smithi027.stdout:4/613: write d5/d45/d81/f98 [1354298,48633] 0 2024-04-22T15:50:02.724 INFO:tasks.workunit.client.1.smithi027.stdout:7/607: dread d0/d11/d1c/d31/d56/f58 [4194304,4194304] 0 2024-04-22T15:50:02.724 INFO:tasks.workunit.client.1.smithi027.stdout:0/657: fdatasync d0/d28/d63/f8d 0 2024-04-22T15:50:02.728 INFO:tasks.workunit.client.1.smithi027.stdout:6/709: mknod d5/dc/d20/cf5 0 2024-04-22T15:50:02.731 INFO:tasks.workunit.client.1.smithi027.stdout:4/614: mkdir d5/d11/d1d/d7b/de9 0 2024-04-22T15:50:02.733 INFO:tasks.workunit.client.1.smithi027.stdout:0/658: mknod d0/d2a/d58/d66/ccf 0 2024-04-22T15:50:02.736 INFO:tasks.workunit.client.1.smithi027.stdout:7/608: link d0/d3f/f48 d0/d4c/fc9 0 2024-04-22T15:50:02.737 INFO:tasks.workunit.client.1.smithi027.stdout:0/659: creat d0/d28/d5f/fd0 x:0 0 0 2024-04-22T15:50:02.739 INFO:tasks.workunit.client.1.smithi027.stdout:6/710: symlink d5/dc/d20/lf6 0 2024-04-22T15:50:02.740 INFO:tasks.workunit.client.1.smithi027.stdout:6/711: chown d5/dc/d17/d53/c64 235513 1 2024-04-22T15:50:02.741 INFO:tasks.workunit.client.1.smithi027.stdout:4/615: mkdir d5/dea 0 2024-04-22T15:50:02.743 INFO:tasks.workunit.client.1.smithi027.stdout:7/609: unlink d0/d49/d9f/f9d 0 2024-04-22T15:50:02.751 INFO:tasks.workunit.client.1.smithi027.stdout:7/610: creat d0/d3f/fca x:0 0 0 2024-04-22T15:50:02.752 INFO:tasks.workunit.client.1.smithi027.stdout:6/712: dwrite d5/dc/dd/d79/d7b/d5d/fb9 [0,4194304] 0 2024-04-22T15:50:02.754 INFO:tasks.workunit.client.1.smithi027.stdout:6/713: readlink d5/dc/l21 0 2024-04-22T15:50:02.755 INFO:tasks.workunit.client.1.smithi027.stdout:6/714: dread - d5/dc/d17/d8a/fdc zero size 2024-04-22T15:50:02.762 INFO:tasks.workunit.client.1.smithi027.stdout:0/660: dread d0/d34/f80 [0,4194304] 0 2024-04-22T15:50:02.765 INFO:tasks.workunit.client.1.smithi027.stdout:0/661: truncate d0/d34/d4e/d85/f8f 4544503 0 2024-04-22T15:50:02.766 INFO:tasks.workunit.client.1.smithi027.stdout:0/662: truncate d0/d2a/d6a/fcb 702590 0 2024-04-22T15:50:02.770 INFO:tasks.workunit.client.1.smithi027.stdout:6/715: dread d5/dc/d26/d3d/f83 [0,4194304] 0 2024-04-22T15:50:02.771 INFO:tasks.workunit.client.1.smithi027.stdout:6/716: write d5/f89 [3081856,127837] 0 2024-04-22T15:50:02.777 INFO:tasks.workunit.client.1.smithi027.stdout:0/663: symlink d0/d2a/d5e/ld1 0 2024-04-22T15:50:02.779 INFO:tasks.workunit.client.1.smithi027.stdout:4/616: write d5/d7/f10 [3466760,64059] 0 2024-04-22T15:50:02.780 INFO:tasks.workunit.client.1.smithi027.stdout:4/617: fsync d5/de/d2a/d58/fc0 0 2024-04-22T15:50:02.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.778+0000 7f61cccc1700 1 -- 172.21.15.16:0/2736880068 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f61a8000bf0 con 0x7f61b40398c0 2024-04-22T15:50:02.781 INFO:tasks.workunit.client.1.smithi027.stdout:6/717: symlink d5/dc/dd/d79/d7b/d5d/dba/lf7 0 2024-04-22T15:50:02.782 INFO:tasks.workunit.client.1.smithi027.stdout:6/718: chown d5/d30/d40/l4c 160023060 1 2024-04-22T15:50:02.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.779+0000 7f61bf7fe700 1 -- 172.21.15.16:0/2736880068 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f61a8000bf0 con 0x7f61b40398c0 2024-04-22T15:50:02.783 INFO:tasks.workunit.client.1.smithi027.stdout:6/719: chown d5/dc/dd/dd0/c94 938491 1 2024-04-22T15:50:02.784 INFO:tasks.workunit.client.1.smithi027.stdout:6/720: chown d5/dc/d17/d53 1025 1 2024-04-22T15:50:02.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.782+0000 7f61cccc1700 1 -- 172.21.15.16:0/2736880068 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f61b40398c0 msgr2=0x7f61b403bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:02.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.782+0000 7f61cccc1700 1 --2- 172.21.15.16:0/2736880068 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f61b40398c0 0x7f61b403bd80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f61b0006010 tx=0x7f61b0005ec0 comp rx=0 tx=0).stop 2024-04-22T15:50:02.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.782+0000 7f61cccc1700 1 -- 172.21.15.16:0/2736880068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c8100c70 msgr2=0x7f61c8102050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:02.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.782+0000 7f61cccc1700 1 --2- 172.21.15.16:0/2736880068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c8100c70 0x7f61c8102050 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f61b800d900 tx=0x7f61b800dc10 comp rx=0 tx=0).stop 2024-04-22T15:50:02.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.782+0000 7f61cccc1700 1 -- 172.21.15.16:0/2736880068 shutdown_connections 2024-04-22T15:50:02.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.782+0000 7f61cccc1700 1 --2- 172.21.15.16:0/2736880068 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f61b40398c0 0x7f61b403bd80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:02.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.782+0000 7f61cccc1700 1 --2- 172.21.15.16:0/2736880068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61c80ffa70 0x7f61c8103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:02.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.782+0000 7f61cccc1700 1 --2- 172.21.15.16:0/2736880068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c8100c70 0x7f61c8102050 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:02.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.782+0000 7f61cccc1700 1 -- 172.21.15.16:0/2736880068 >> 172.21.15.16:0/2736880068 conn(0x7f61c80fafd0 msgr2=0x7f61c8068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:02.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.782+0000 7f61cccc1700 1 -- 172.21.15.16:0/2736880068 shutdown_connections 2024-04-22T15:50:02.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.782+0000 7f61cccc1700 1 -- 172.21.15.16:0/2736880068 wait complete. 2024-04-22T15:50:02.792 INFO:tasks.workunit.client.1.smithi027.stdout:6/721: symlink d5/db/dcd/ddf/lf8 0 2024-04-22T15:50:02.794 INFO:tasks.workunit.client.1.smithi027.stdout:4/618: dread d5/d7/f2c [0,4194304] 0 2024-04-22T15:50:02.794 INFO:tasks.workunit.client.1.smithi027.stdout:6/722: truncate d5/dc/d17/d53/fa1 869999 0 2024-04-22T15:50:02.797 INFO:tasks.workunit.client.1.smithi027.stdout:4/619: symlink d5/de/leb 0 2024-04-22T15:50:02.800 INFO:tasks.workunit.client.1.smithi027.stdout:0/664: getdents d0/d34/d4e 0 2024-04-22T15:50:02.801 INFO:tasks.workunit.client.1.smithi027.stdout:0/665: chown d0/d28/f1c 158 1 2024-04-22T15:50:02.802 INFO:tasks.workunit.client.1.smithi027.stdout:4/620: creat d5/d11/d89/fec x:0 0 0 2024-04-22T15:50:02.803 INFO:tasks.workunit.client.1.smithi027.stdout:6/723: dwrite d5/dc/dd/d79/d7b/d9a/fe5 [0,4194304] 0 2024-04-22T15:50:02.803 INFO:tasks.workunit.client.1.smithi027.stdout:4/621: chown d5/de/da9 7350 1 2024-04-22T15:50:02.807 INFO:tasks.workunit.client.1.smithi027.stdout:0/666: mkdir d0/d28/dd2 0 2024-04-22T15:50:02.808 INFO:tasks.workunit.client.1.smithi027.stdout:0/667: dread - d0/da3/dae/fc4 zero size 2024-04-22T15:50:02.809 INFO:tasks.workunit.client.1.smithi027.stdout:9/616: sync 2024-04-22T15:50:02.809 INFO:tasks.workunit.client.1.smithi027.stdout:1/550: sync 2024-04-22T15:50:02.810 INFO:tasks.workunit.client.1.smithi027.stdout:1/551: chown d2/d5f/f7a 1765559 1 2024-04-22T15:50:02.812 INFO:tasks.workunit.client.1.smithi027.stdout:6/724: rename d5/db/d2e/d59/l9f to d5/dc/dd/dd0/lf9 0 2024-04-22T15:50:02.816 INFO:tasks.workunit.client.1.smithi027.stdout:1/552: rmdir d2/d5f/d64/d81/d69/d7d 39 2024-04-22T15:50:02.822 INFO:tasks.workunit.client.1.smithi027.stdout:1/553: readlink d2/d30/l89 0 2024-04-22T15:50:02.823 INFO:tasks.workunit.client.1.smithi027.stdout:1/554: truncate d2/d1d/d56/f9a 700784 0 2024-04-22T15:50:02.824 INFO:tasks.workunit.client.1.smithi027.stdout:1/555: chown d2/f60 0 1 2024-04-22T15:50:02.826 INFO:tasks.workunit.client.1.smithi027.stdout:1/556: write d2/d5f/d64/d81/d3d/d44/d9c/d9f/fb5 [938180,108842] 0 2024-04-22T15:50:02.826 INFO:tasks.workunit.client.1.smithi027.stdout:6/725: creat d5/dc/dd/ffa x:0 0 0 2024-04-22T15:50:02.827 INFO:tasks.workunit.client.1.smithi027.stdout:1/557: read - d2/d5f/f76 zero size 2024-04-22T15:50:02.828 INFO:tasks.workunit.client.1.smithi027.stdout:1/558: chown d2/d5f/d64/d81/f50 470184272 1 2024-04-22T15:50:02.831 INFO:tasks.workunit.client.1.smithi027.stdout:6/726: creat d5/dc/d26/d49/d86/ffb x:0 0 0 2024-04-22T15:50:02.832 INFO:tasks.workunit.client.1.smithi027.stdout:6/727: write d5/dc/d20/da3/fe3 [118332,89921] 0 2024-04-22T15:50:02.834 INFO:tasks.workunit.client.1.smithi027.stdout:1/559: rmdir d2/d5f/d64/d81/d3d/d44 39 2024-04-22T15:50:02.840 INFO:tasks.workunit.client.1.smithi027.stdout:1/560: mkdir d2/d1d/d56/dc0 0 2024-04-22T15:50:02.854 INFO:tasks.workunit.client.1.smithi027.stdout:4/622: dwrite d5/d11/d89/f84 [0,4194304] 0 2024-04-22T15:50:02.855 INFO:tasks.workunit.client.1.smithi027.stdout:4/623: chown d5/d45/d81 26032 1 2024-04-22T15:50:02.859 INFO:tasks.workunit.client.1.smithi027.stdout:1/561: dread d2/f27 [0,4194304] 0 2024-04-22T15:50:02.861 INFO:tasks.workunit.client.1.smithi027.stdout:6/728: dread d5/d30/dbe/fc6 [0,4194304] 0 2024-04-22T15:50:02.866 INFO:tasks.workunit.client.1.smithi027.stdout:6/729: rmdir d5/db 39 2024-04-22T15:50:02.868 INFO:tasks.workunit.client.1.smithi027.stdout:1/562: dwrite d2/d5f/d64/d81/da7/f9e [0,4194304] 0 2024-04-22T15:50:02.868 INFO:tasks.workunit.client.1.smithi027.stdout:9/617: write d0/d8/f11 [1762161,68423] 0 2024-04-22T15:50:02.874 INFO:tasks.workunit.client.1.smithi027.stdout:1/563: symlink d2/d36/daa/lc1 0 2024-04-22T15:50:02.875 INFO:tasks.workunit.client.1.smithi027.stdout:9/618: unlink d0/c52 0 2024-04-22T15:50:02.877 INFO:tasks.workunit.client.1.smithi027.stdout:6/730: mkdir d5/d30/d54/dd8/dfc 0 2024-04-22T15:50:02.879 INFO:tasks.workunit.client.1.smithi027.stdout:1/564: mkdir d2/d5f/d64/d81/d3d/d82/dc2 0 2024-04-22T15:50:02.880 INFO:tasks.workunit.client.1.smithi027.stdout:9/619: creat d0/dc/d12/d58/da6/fce x:0 0 0 2024-04-22T15:50:02.882 INFO:tasks.workunit.client.1.smithi027.stdout:6/731: mknod d5/d30/d54/dd8/cfd 0 2024-04-22T15:50:02.883 INFO:tasks.workunit.client.1.smithi027.stdout:6/732: fdatasync d5/dc/d17/d8a/fdc 0 2024-04-22T15:50:02.885 INFO:tasks.workunit.client.1.smithi027.stdout:6/733: fsync d5/dc/dd/f16 0 2024-04-22T15:50:02.890 INFO:tasks.workunit.client.1.smithi027.stdout:1/565: dwrite d2/d5f/d64/d81/d3d/d44/d9c/d9f/fac [0,4194304] 0 2024-04-22T15:50:02.891 INFO:tasks.workunit.client.1.smithi027.stdout:6/734: write d5/dc/dd/d52/f8e [4414676,60109] 0 2024-04-22T15:50:02.893 INFO:tasks.workunit.client.1.smithi027.stdout:4/624: dwrite d5/d11/da6/fd6 [0,4194304] 0 2024-04-22T15:50:02.893 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:02 smithi027 ceph-mon[34922]: pgmap v5: 65 pgs: 65 active+clean; 2.9 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail 2024-04-22T15:50:02.893 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:02 smithi027 ceph-mon[34922]: from='client.14702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:02.898 INFO:tasks.workunit.client.1.smithi027.stdout:6/735: read d5/d30/dbe/fca [3452247,11563] 0 2024-04-22T15:50:02.903 INFO:tasks.workunit.client.1.smithi027.stdout:6/736: dread d5/dc/d20/f3f [4194304,4194304] 0 2024-04-22T15:50:02.903 INFO:tasks.workunit.client.1.smithi027.stdout:4/625: dread d5/d11/d89/f70 [0,4194304] 0 2024-04-22T15:50:02.906 INFO:tasks.workunit.client.1.smithi027.stdout:4/626: dread d5/de/dc5/fcb [0,4194304] 0 2024-04-22T15:50:02.907 INFO:tasks.workunit.client.1.smithi027.stdout:1/566: dwrite d2/d1d/f32 [4194304,4194304] 0 2024-04-22T15:50:02.908 INFO:tasks.workunit.client.1.smithi027.stdout:4/627: dread - d5/d11/d1d/d7b/f92 zero size 2024-04-22T15:50:02.916 INFO:tasks.workunit.client.1.smithi027.stdout:1/567: dwrite d2/d36/f7c [0,4194304] 0 2024-04-22T15:50:02.916 INFO:tasks.workunit.client.1.smithi027.stdout:1/568: readlink d2/d1d/l2a 0 2024-04-22T15:50:02.917 INFO:tasks.workunit.client.1.smithi027.stdout:1/569: chown d2 0 1 2024-04-22T15:50:02.919 INFO:tasks.workunit.client.1.smithi027.stdout:4/628: mknod d5/de/ced 0 2024-04-22T15:50:02.920 INFO:tasks.workunit.client.1.smithi027.stdout:4/629: chown d5/de 162488037 1 2024-04-22T15:50:02.924 INFO:tasks.workunit.client.1.smithi027.stdout:1/570: fsync d2/d5f/d64/d81/d3d/d82/d85/f95 0 2024-04-22T15:50:02.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.922+0000 7f436cb18700 1 -- 172.21.15.16:0/3111410887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4368103fb0 msgr2=0x7f4368104410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:02.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.922+0000 7f436cb18700 1 --2- 172.21.15.16:0/3111410887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4368103fb0 0x7f4368104410 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f435c009910 tx=0x7f435c009c20 comp rx=0 tx=0).stop 2024-04-22T15:50:02.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.922+0000 7f436cb18700 1 -- 172.21.15.16:0/3111410887 shutdown_connections 2024-04-22T15:50:02.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.922+0000 7f436cb18700 1 --2- 172.21.15.16:0/3111410887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4368103fb0 0x7f4368104410 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:02.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.922+0000 7f436cb18700 1 --2- 172.21.15.16:0/3111410887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4368102db0 0x7f43681031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:02.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.922+0000 7f436cb18700 1 -- 172.21.15.16:0/3111410887 >> 172.21.15.16:0/3111410887 conn(0x7f43680fe330 msgr2=0x7f4368100790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:02.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.922+0000 7f436cb18700 1 -- 172.21.15.16:0/3111410887 shutdown_connections 2024-04-22T15:50:02.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.923+0000 7f436cb18700 1 -- 172.21.15.16:0/3111410887 wait complete. 2024-04-22T15:50:02.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.924+0000 7f436cb18700 1 Processor -- start 2024-04-22T15:50:02.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.924+0000 7f436cb18700 1 -- start start 2024-04-22T15:50:02.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.924+0000 7f436cb18700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4368102db0 0x7f4368198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:02.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.924+0000 7f436cb18700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4368103fb0 0x7f4368198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:02.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.924+0000 7f436cb18700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4368199090 con 0x7f4368103fb0 2024-04-22T15:50:02.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.924+0000 7f436cb18700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43681991d0 con 0x7f4368102db0 2024-04-22T15:50:02.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.924+0000 7f4365d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4368103fb0 0x7f4368198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:02.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.924+0000 7f4365d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4368103fb0 0x7f4368198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58418/0 (socket says 172.21.15.16:58418) 2024-04-22T15:50:02.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.924+0000 7f4365d9b700 1 -- 172.21.15.16:0/1043576464 learned_addr learned my addr 172.21.15.16:0/1043576464 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:50:02.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.924+0000 7f436659c700 1 --2- 172.21.15.16:0/1043576464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4368102db0 0x7f4368198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:02.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.925+0000 7f4365d9b700 1 -- 172.21.15.16:0/1043576464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4368102db0 msgr2=0x7f4368198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:02.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.925+0000 7f4365d9b700 1 --2- 172.21.15.16:0/1043576464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4368102db0 0x7f4368198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:02.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.925+0000 7f4365d9b700 1 -- 172.21.15.16:0/1043576464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435c0095f0 con 0x7f4368103fb0 2024-04-22T15:50:02.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.925+0000 7f4365d9b700 1 --2- 172.21.15.16:0/1043576464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4368103fb0 0x7f4368198a70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f435c000c00 tx=0x7f435c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:02.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.925+0000 7f43577fe700 1 -- 172.21.15.16:0/1043576464 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f435c01d070 con 0x7f4368103fb0 2024-04-22T15:50:02.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.925+0000 7f436cb18700 1 -- 172.21.15.16:0/1043576464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f436819dc20 con 0x7f4368103fb0 2024-04-22T15:50:02.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.925+0000 7f43577fe700 1 -- 172.21.15.16:0/1043576464 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f435c005dc0 con 0x7f4368103fb0 2024-04-22T15:50:02.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.925+0000 7f436cb18700 1 -- 172.21.15.16:0/1043576464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f436819e110 con 0x7f4368103fb0 2024-04-22T15:50:02.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.926+0000 7f43577fe700 1 -- 172.21.15.16:0/1043576464 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f435c0226a0 con 0x7f4368103fb0 2024-04-22T15:50:02.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.926+0000 7f43577fe700 1 -- 172.21.15.16:0/1043576464 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 29) v1 ==== 46697+0+0 (secure 0 0 0) 0x7f435c022800 con 0x7f4368103fb0 2024-04-22T15:50:02.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.926+0000 7f43577fe700 1 --2- 172.21.15.16:0/1043576464 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f43500398c0 0x7f435003bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:02.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.926+0000 7f43577fe700 1 -- 172.21.15.16:0/1043576464 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f435c050230 con 0x7f4368103fb0 2024-04-22T15:50:02.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.927+0000 7f436cb18700 1 -- 172.21.15.16:0/1043576464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4348005320 con 0x7f4368103fb0 2024-04-22T15:50:02.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.927+0000 7f436659c700 1 --2- 172.21.15.16:0/1043576464 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f43500398c0 0x7f435003bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:02.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.927+0000 7f436659c700 1 --2- 172.21.15.16:0/1043576464 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f43500398c0 0x7f435003bd80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4368103e10 tx=0x7f4358009260 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:02.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:02.933+0000 7f43577fe700 1 -- 172.21.15.16:0/1043576464 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f435c003e60 con 0x7f4368103fb0 2024-04-22T15:50:02.936 INFO:tasks.workunit.client.1.smithi027.stdout:6/737: getdents d5/dc/d26/d49 0 2024-04-22T15:50:02.942 INFO:tasks.workunit.client.1.smithi027.stdout:6/738: creat d5/db/d2e/d59/ffe x:0 0 0 2024-04-22T15:50:02.942 INFO:tasks.workunit.client.1.smithi027.stdout:1/571: truncate d2/d1d/f32 4732071 0 2024-04-22T15:50:02.948 INFO:tasks.workunit.client.1.smithi027.stdout:1/572: dwrite d2/d5f/d64/d81/d3d/d44/fb2 [0,4194304] 0 2024-04-22T15:50:02.955 INFO:tasks.workunit.client.1.smithi027.stdout:1/573: dread d2/f7f [0,4194304] 0 2024-04-22T15:50:02.955 INFO:tasks.workunit.client.1.smithi027.stdout:1/574: chown d2/d30/fa3 1 1 2024-04-22T15:50:02.956 INFO:tasks.workunit.client.1.smithi027.stdout:1/575: write d2/d1d/f91 [593773,90754] 0 2024-04-22T15:50:02.957 INFO:tasks.workunit.client.1.smithi027.stdout:6/739: dread d5/f7a [0,4194304] 0 2024-04-22T15:50:02.959 INFO:tasks.workunit.client.1.smithi027.stdout:6/740: fdatasync d5/d30/d40/fce 0 2024-04-22T15:50:02.962 INFO:tasks.workunit.client.1.smithi027.stdout:6/741: symlink d5/d30/d40/lff 0 2024-04-22T15:50:02.964 INFO:tasks.workunit.client.1.smithi027.stdout:6/742: write d5/dc/dd/dd0/f12 [11066698,5083] 0 2024-04-22T15:50:02.966 INFO:tasks.workunit.client.1.smithi027.stdout:5/661: sync 2024-04-22T15:50:02.967 INFO:tasks.workunit.client.1.smithi027.stdout:8/562: sync 2024-04-22T15:50:02.967 INFO:tasks.workunit.client.1.smithi027.stdout:6/743: unlink d5/d30/d40/lff 0 2024-04-22T15:50:02.972 INFO:tasks.workunit.client.1.smithi027.stdout:6/744: write d5/d30/dbe/fca [2465139,109363] 0 2024-04-22T15:50:02.972 INFO:tasks.workunit.client.1.smithi027.stdout:5/662: dread d0/d1a/d22/d72/d86/f75 [0,4194304] 0 2024-04-22T15:50:02.974 INFO:tasks.workunit.client.1.smithi027.stdout:8/563: mkdir d1/d3/d1c/d28/d42/dc4 0 2024-04-22T15:50:02.975 INFO:tasks.workunit.client.1.smithi027.stdout:5/663: chown d0/d1a/d22/d26/d32/d88/fb0 1337517396 1 2024-04-22T15:50:02.976 INFO:tasks.workunit.client.1.smithi027.stdout:5/664: write d0/df/d3d/f8e [728567,84865] 0 2024-04-22T15:50:02.977 INFO:tasks.workunit.client.1.smithi027.stdout:1/576: truncate d2/d5f/d64/d81/d3d/d44/fb2 2409455 0 2024-04-22T15:50:02.978 INFO:tasks.workunit.client.1.smithi027.stdout:1/577: stat d2/d36/daa/lc1 0 2024-04-22T15:50:02.981 INFO:tasks.workunit.client.1.smithi027.stdout:6/745: symlink d5/d30/dc7/l100 0 2024-04-22T15:50:02.984 INFO:tasks.workunit.client.1.smithi027.stdout:6/746: truncate d5/dc/dd/d79/d7b/d9a/fa0 121677 0 2024-04-22T15:50:02.985 INFO:tasks.workunit.client.1.smithi027.stdout:5/665: symlink d0/df/de/d24/d1c/d3e/ld9 0 2024-04-22T15:50:02.987 INFO:tasks.workunit.client.1.smithi027.stdout:8/564: rename d1/d3/f13 to d1/d3/d1c/d32/fc5 0 2024-04-22T15:50:02.991 INFO:tasks.workunit.client.1.smithi027.stdout:1/578: dwrite d2/d5f/d64/d81/da7/f9d [0,4194304] 0 2024-04-22T15:50:02.997 INFO:tasks.workunit.client.1.smithi027.stdout:1/579: creat d2/d5f/fc3 x:0 0 0 2024-04-22T15:50:02.998 INFO:tasks.workunit.client.1.smithi027.stdout:6/747: dread d5/db/d2e/ff [0,4194304] 0 2024-04-22T15:50:02.999 INFO:tasks.workunit.client.1.smithi027.stdout:6/748: fdatasync d5/dc/fa6 0 2024-04-22T15:50:03.006 INFO:tasks.workunit.client.1.smithi027.stdout:6/749: mknod d5/dc/d26/dee/c101 0 2024-04-22T15:50:03.006 INFO:tasks.workunit.client.1.smithi027.stdout:6/750: chown d5/d30/dbe/fca 96199 1 2024-04-22T15:50:03.010 INFO:tasks.workunit.client.1.smithi027.stdout:5/666: write d0/f8 [1201609,37756] 0 2024-04-22T15:50:03.013 INFO:tasks.workunit.client.1.smithi027.stdout:5/667: creat d0/d1a/d22/d26/d5d/fda x:0 0 0 2024-04-22T15:50:03.014 INFO:tasks.workunit.client.1.smithi027.stdout:5/668: chown d0/d97/dbf 898921 1 2024-04-22T15:50:03.016 INFO:tasks.workunit.client.1.smithi027.stdout:6/751: dwrite d5/d30/dbe/fc6 [4194304,4194304] 0 2024-04-22T15:50:03.016 INFO:tasks.workunit.client.1.smithi027.stdout:8/565: write d1/d3/f46 [239008,11608] 0 2024-04-22T15:50:03.020 INFO:tasks.workunit.client.1.smithi027.stdout:8/566: creat d1/d3/fc6 x:0 0 0 2024-04-22T15:50:03.022 INFO:tasks.workunit.client.1.smithi027.stdout:8/567: dread - d1/d1e/d47/d53/fc2 zero size 2024-04-22T15:50:03.023 INFO:tasks.workunit.client.1.smithi027.stdout:6/752: mknod d5/dc/dd/d52/c102 0 2024-04-22T15:50:03.028 INFO:tasks.workunit.client.1.smithi027.stdout:5/669: getdents d0/df 0 2024-04-22T15:50:03.039 INFO:tasks.workunit.client.1.smithi027.stdout:5/670: symlink d0/d1a/d22/d26/d32/d79/dd6/ldb 0 2024-04-22T15:50:03.039 INFO:tasks.workunit.client.1.smithi027.stdout:5/671: write d0/db6/fcf [618640,85745] 0 2024-04-22T15:50:03.041 INFO:tasks.workunit.client.1.smithi027.stdout:6/753: creat d5/dc/dd/d52/f103 x:0 0 0 2024-04-22T15:50:03.041 INFO:tasks.workunit.client.1.smithi027.stdout:5/672: write d0/df/de/d40/fb2 [61467,54153] 0 2024-04-22T15:50:03.043 INFO:tasks.workunit.client.1.smithi027.stdout:8/568: write d1/d3/d1c/d32/fc5 [475965,109018] 0 2024-04-22T15:50:03.044 INFO:tasks.workunit.client.1.smithi027.stdout:6/754: creat d5/dc/dae/f104 x:0 0 0 2024-04-22T15:50:03.047 INFO:tasks.workunit.client.1.smithi027.stdout:8/569: creat d1/d3/fc7 x:0 0 0 2024-04-22T15:50:03.049 INFO:tasks.workunit.client.1.smithi027.stdout:6/755: mknod d5/dc/d20/da3/c105 0 2024-04-22T15:50:03.051 INFO:tasks.workunit.client.1.smithi027.stdout:5/673: rename d0/df/d3d/f69 to d0/d1a/d22/d26/d32/d79/dd6/fdc 0 2024-04-22T15:50:03.054 INFO:tasks.workunit.client.1.smithi027.stdout:8/570: dwrite d1/d3/d1c/d32/d41/d52/d61/fa6 [0,4194304] 0 2024-04-22T15:50:03.054 INFO:tasks.workunit.client.1.smithi027.stdout:6/756: creat d5/dc/dd/dd0/d34/f106 x:0 0 0 2024-04-22T15:50:03.055 INFO:tasks.workunit.client.1.smithi027.stdout:5/674: mknod d0/d1a/d22/d72/dac/cdd 0 2024-04-22T15:50:03.056 INFO:tasks.workunit.client.1.smithi027.stdout:6/757: write d5/db/d2e/d59/fb1 [4070340,5211] 0 2024-04-22T15:50:03.057 INFO:tasks.workunit.client.1.smithi027.stdout:5/675: readlink d0/d1a/d22/d26/d32/lab 0 2024-04-22T15:50:03.058 INFO:tasks.workunit.client.1.smithi027.stdout:6/758: chown d5/dc/d17/d53/c64 219914245 1 2024-04-22T15:50:03.058 INFO:tasks.workunit.client.1.smithi027.stdout:5/676: readlink d0/db6/ld3 0 2024-04-22T15:50:03.060 INFO:tasks.workunit.client.1.smithi027.stdout:8/571: creat d1/d3/d1c/d32/d41/d52/d61/d8f/fc8 x:0 0 0 2024-04-22T15:50:03.061 INFO:tasks.workunit.client.1.smithi027.stdout:8/572: dread - d1/d3/d1c/d32/d41/d52/fbf zero size 2024-04-22T15:50:03.069 INFO:tasks.workunit.client.1.smithi027.stdout:8/573: dwrite d1/d3/d1c/d32/fb0 [0,4194304] 0 2024-04-22T15:50:03.071 INFO:tasks.workunit.client.1.smithi027.stdout:6/759: link d5/dc/dd/d52/f103 d5/dc/dd/f107 0 2024-04-22T15:50:03.077 INFO:tasks.workunit.client.1.smithi027.stdout:8/574: getdents d1/d3/d1c/d56 0 2024-04-22T15:50:03.077 INFO:tasks.workunit.client.1.smithi027.stdout:6/760: mkdir d5/dc/d17/d108 0 2024-04-22T15:50:03.080 INFO:tasks.workunit.client.1.smithi027.stdout:6/761: dread - d5/dc/dd/d79/d7b/fc1 zero size 2024-04-22T15:50:03.081 INFO:tasks.workunit.client.1.smithi027.stdout:8/575: truncate d1/d1e/f4c 5164299 0 2024-04-22T15:50:03.082 INFO:tasks.workunit.client.1.smithi027.stdout:5/677: dwrite d0/d1a/d22/d26/d32/f70 [0,4194304] 0 2024-04-22T15:50:03.085 INFO:tasks.workunit.client.1.smithi027.stdout:8/576: mkdir d1/d3/d1c/d56/dc9 0 2024-04-22T15:50:03.086 INFO:tasks.workunit.client.1.smithi027.stdout:6/762: creat d5/dc/dd/d79/d7b/d5d/f109 x:0 0 0 2024-04-22T15:50:03.090 INFO:tasks.workunit.client.1.smithi027.stdout:5/678: mkdir d0/d1a/d22/d72/dac/dde 0 2024-04-22T15:50:03.095 INFO:tasks.workunit.client.1.smithi027.stdout:5/679: truncate d0/d1a/d22/d26/d32/d79/f87 24083 0 2024-04-22T15:50:03.097 INFO:tasks.workunit.client.1.smithi027.stdout:5/680: chown d0/df/d10/f33 1218131 1 2024-04-22T15:50:03.099 INFO:tasks.workunit.client.1.smithi027.stdout:6/763: dwrite d5/db/f4d [0,4194304] 0 2024-04-22T15:50:03.100 INFO:tasks.workunit.client.1.smithi027.stdout:8/577: rename d1/d3/d1c/d43/d5f/d7d/f8c to d1/d3/d1c/d32/fca 0 2024-04-22T15:50:03.101 INFO:tasks.workunit.client.1.smithi027.stdout:8/578: chown d1/d3/l8 9336845 1 2024-04-22T15:50:03.105 INFO:tasks.workunit.client.1.smithi027.stdout:6/764: chown d5/dc/dd/d52/f7e 50849 1 2024-04-22T15:50:03.111 INFO:tasks.workunit.client.1.smithi027.stdout:5/681: getdents d0/d1a/d22/d72 0 2024-04-22T15:50:03.112 INFO:tasks.workunit.client.1.smithi027.stdout:8/579: creat d1/d3/d1c/d32/fcb x:0 0 0 2024-04-22T15:50:03.115 INFO:tasks.workunit.client.1.smithi027.stdout:8/580: fdatasync d1/d3/d1c/d49/f63 0 2024-04-22T15:50:03.121 INFO:tasks.workunit.client.1.smithi027.stdout:6/765: write d5/db/f11 [605888,38538] 0 2024-04-22T15:50:03.123 INFO:tasks.workunit.client.1.smithi027.stdout:6/766: fdatasync d5/dc/dd/d79/d7b/d9a/fe5 0 2024-04-22T15:50:03.127 INFO:tasks.workunit.client.1.smithi027.stdout:5/682: dwrite d0/d1a/d22/d72/d86/f77 [0,4194304] 0 2024-04-22T15:50:03.134 INFO:tasks.workunit.client.1.smithi027.stdout:5/683: dwrite d0/d1a/d22/d26/d32/d88/fb0 [0,4194304] 0 2024-04-22T15:50:03.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:02 smithi016 ceph-mon[26620]: pgmap v5: 65 pgs: 65 active+clean; 2.9 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail 2024-04-22T15:50:03.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:02 smithi016 ceph-mon[26620]: from='client.14702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:03.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.202+0000 7f436cb18700 1 -- 172.21.15.16:0/1043576464 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4348000bf0 con 0x7f43500398c0 2024-04-22T15:50:03.206 INFO:tasks.workunit.client.1.smithi027.stdout:5/684: rename d0/d1a/d22/d26/d32/d88/fb0 to d0/d1a/d22/d26/d32/d79/dd6/fdf 0 2024-04-22T15:50:03.207 INFO:tasks.workunit.client.1.smithi027.stdout:8/581: dwrite d1/d3/d1c/d43/d82/f84 [0,4194304] 0 2024-04-22T15:50:03.208 INFO:tasks.workunit.client.1.smithi027.stdout:8/582: stat d1/d3/f9b 0 2024-04-22T15:50:03.211 INFO:tasks.workunit.client.1.smithi027.stdout:6/767: creat d5/dc/d17/d108/f10a x:0 0 0 2024-04-22T15:50:03.216 INFO:tasks.workunit.client.1.smithi027.stdout:5/685: mknod d0/d1a/d22/d26/d5d/ce0 0 2024-04-22T15:50:03.217 INFO:tasks.workunit.client.1.smithi027.stdout:5/686: chown d0/df/de/d24/d1c 34179 1 2024-04-22T15:50:03.218 INFO:tasks.workunit.client.1.smithi027.stdout:6/768: fsync d5/dc/dd/d79/d7b/fc1 0 2024-04-22T15:50:03.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.215+0000 7f43577fe700 1 -- 172.21.15.16:0/1043576464 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f4348000bf0 con 0x7f43500398c0 2024-04-22T15:50:03.218 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:50:03.219 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.smithi016 smithi016 *:9093,9094 running (7m) 10s ago 8m 24.8M - 0.25.0 c8568f914cd2 fab40987f1b0 2024-04-22T15:50:03.219 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi016 smithi016 running (9m) 10s ago 9m 7680k - 18.2.1 5be31c24972a f8b627d3f41e 2024-04-22T15:50:03.219 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi027 smithi027 running (8m) 55s ago 8m 7604k - 18.2.1 5be31c24972a f00fd40b4220 2024-04-22T15:50:03.219 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi016 smithi016 running (9m) 10s ago 9m 6656k - 18.2.1 5be31c24972a 6b93db1fa0d9 2024-04-22T15:50:03.219 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi027 smithi027 running (8m) 55s ago 8m 6665k - 18.2.1 5be31c24972a d3bd439f30a0 2024-04-22T15:50:03.219 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.smithi016 smithi016 *:3000 running (7m) 10s ago 8m 85.9M - 9.4.7 954c08fa6188 a85bdbf5073d 2024-04-22T15:50:03.219 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.fkkejm smithi016 running (3m) 10s ago 3m 13.3M - 18.2.1 5be31c24972a 4153ac94cea0 2024-04-22T15:50:03.219 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.pgvphu smithi016 running (3m) 10s ago 3m 246M - 18.2.1 5be31c24972a 93dc072068a2 2024-04-22T15:50:03.219 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.dgqyes smithi027 running (3m) 55s ago 3m 15.9M - 18.2.1 5be31c24972a 41a9220430fd 2024-04-22T15:50:03.219 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.iobjds smithi027 running (3m) 55s ago 3m 98.2M - 18.2.1 5be31c24972a c1c8d6770f02 2024-04-22T15:50:03.219 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi016.rnsnnr smithi016 *:8443,9283,8765 running (25s) 10s ago 10m 70.5M - 19.0.0-3223-g5883d494 3d983c26e278 09b4fe285bc4 2024-04-22T15:50:03.219 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi027.ukdgzs smithi027 *:8443,9283,8765 running (92s) 55s ago 7m 491M - 19.0.0-3223-g5883d494 3d983c26e278 8dfb23a29706 2024-04-22T15:50:03.220 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi016 smithi016 running (10m) 10s ago 10m 53.1M 2048M 18.2.1 5be31c24972a 72af40873cb6 2024-04-22T15:50:03.220 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi027 smithi027 running (7m) 55s ago 7m 40.5M 2048M 18.2.1 5be31c24972a 9f21ce7fa7ef 2024-04-22T15:50:03.220 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi016 smithi016 *:9100 running (9m) 10s ago 9m 16.1M - 1.5.0 0da6a335fe13 1937a0bc1959 2024-04-22T15:50:03.220 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi027 smithi027 *:9100 running (8m) 55s ago 8m 15.7M - 1.5.0 0da6a335fe13 00440c991b3f 2024-04-22T15:50:03.220 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (7m) 10s ago 7m 144M 1217M 18.2.1 5be31c24972a 3c6ebc4ea041 2024-04-22T15:50:03.220 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (6m) 10s ago 6m 147M 1217M 18.2.1 5be31c24972a 6aa2e6796784 2024-04-22T15:50:03.220 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (6m) 10s ago 6m 118M 1217M 18.2.1 5be31c24972a 3d9d310aa9eb 2024-04-22T15:50:03.220 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi027 running (5m) 55s ago 5m 67.5M 2240M 18.2.1 5be31c24972a cdf9b7aaf865 2024-04-22T15:50:03.220 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (5m) 55s ago 5m 61.5M 2240M 18.2.1 5be31c24972a 629397d49c76 2024-04-22T15:50:03.220 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi027 running (4m) 55s ago 4m 64.3M 2240M 18.2.1 5be31c24972a 23adcf675b04 2024-04-22T15:50:03.220 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.smithi016 smithi016 *:9095 running (42s) 10s ago 8m 44.6M - 2.43.0 a07b618ecd1d 07fdbecf03d2 2024-04-22T15:50:03.221 INFO:tasks.workunit.client.1.smithi027.stdout:5/687: write d0/d1a/d22/d26/d5d/fda [679833,60423] 0 2024-04-22T15:50:03.221 INFO:tasks.workunit.client.1.smithi027.stdout:8/583: rename d1/f7 to d1/d4f/fcc 0 2024-04-22T15:50:03.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.218+0000 7f436cb18700 1 -- 172.21.15.16:0/1043576464 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f43500398c0 msgr2=0x7f435003bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.218+0000 7f436cb18700 1 --2- 172.21.15.16:0/1043576464 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f43500398c0 0x7f435003bd80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4368103e10 tx=0x7f4358009260 comp rx=0 tx=0).stop 2024-04-22T15:50:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.218+0000 7f436cb18700 1 -- 172.21.15.16:0/1043576464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4368103fb0 msgr2=0x7f4368198a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.218+0000 7f436cb18700 1 --2- 172.21.15.16:0/1043576464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4368103fb0 0x7f4368198a70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f435c000c00 tx=0x7f435c00f690 comp rx=0 tx=0).stop 2024-04-22T15:50:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.218+0000 7f436cb18700 1 -- 172.21.15.16:0/1043576464 shutdown_connections 2024-04-22T15:50:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.218+0000 7f436cb18700 1 --2- 172.21.15.16:0/1043576464 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f43500398c0 0x7f435003bd80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.218+0000 7f436cb18700 1 --2- 172.21.15.16:0/1043576464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4368102db0 0x7f4368198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.218+0000 7f436cb18700 1 --2- 172.21.15.16:0/1043576464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4368103fb0 0x7f4368198a70 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.218+0000 7f436cb18700 1 -- 172.21.15.16:0/1043576464 >> 172.21.15.16:0/1043576464 conn(0x7f43680fe330 msgr2=0x7f43681071e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.218+0000 7f436cb18700 1 -- 172.21.15.16:0/1043576464 shutdown_connections 2024-04-22T15:50:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.218+0000 7f436cb18700 1 -- 172.21.15.16:0/1043576464 wait complete. 2024-04-22T15:50:03.223 INFO:tasks.workunit.client.1.smithi027.stdout:6/769: creat d5/d30/d54/dd8/f10b x:0 0 0 2024-04-22T15:50:03.223 INFO:tasks.workunit.client.1.smithi027.stdout:6/770: dread - d5/d30/d40/fad zero size 2024-04-22T15:50:03.224 INFO:tasks.workunit.client.1.smithi027.stdout:6/771: chown d5/dc/dd/dd0/d34/f65 296 1 2024-04-22T15:50:03.226 INFO:tasks.workunit.client.1.smithi027.stdout:8/584: rename d1/d3/d1c/d28/d42/l74 to d1/d3/d1c/d28/d42/lcd 0 2024-04-22T15:50:03.232 INFO:tasks.workunit.client.1.smithi027.stdout:6/772: dwrite d5/db/d2e/d59/fb1 [0,4194304] 0 2024-04-22T15:50:03.236 INFO:tasks.workunit.client.1.smithi027.stdout:6/773: creat d5/dc/d26/d49/f10c x:0 0 0 2024-04-22T15:50:03.236 INFO:tasks.workunit.client.1.smithi027.stdout:8/585: dwrite d1/d3/d1c/d32/d41/f83 [0,4194304] 0 2024-04-22T15:50:03.237 INFO:tasks.workunit.client.1.smithi027.stdout:8/586: chown d1/d3/d36/d57/l58 3 1 2024-04-22T15:50:03.257 INFO:tasks.workunit.client.1.smithi027.stdout:8/587: dwrite d1/d3/d1c/d32/d41/f7a [0,4194304] 0 2024-04-22T15:50:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.341+0000 7fdbc1cdf700 1 -- 172.21.15.16:0/3110615681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbbc101eb0 msgr2=0x7fdbbc102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.341+0000 7fdbc1cdf700 1 --2- 172.21.15.16:0/3110615681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbbc101eb0 0x7fdbbc102310 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fdba4009910 tx=0x7fdba4009c20 comp rx=0 tx=0).stop 2024-04-22T15:50:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.341+0000 7fdbc1cdf700 1 -- 172.21.15.16:0/3110615681 shutdown_connections 2024-04-22T15:50:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.341+0000 7fdbc1cdf700 1 --2- 172.21.15.16:0/3110615681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbbc101eb0 0x7fdbbc102310 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.341+0000 7fdbc1cdf700 1 --2- 172.21.15.16:0/3110615681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbbc100cb0 0x7fdbbc1010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.341+0000 7fdbc1cdf700 1 -- 172.21.15.16:0/3110615681 >> 172.21.15.16:0/3110615681 conn(0x7fdbbc0fc230 msgr2=0x7fdbbc0fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.341+0000 7fdbc1cdf700 1 -- 172.21.15.16:0/3110615681 shutdown_connections 2024-04-22T15:50:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.341+0000 7fdbc1cdf700 1 -- 172.21.15.16:0/3110615681 wait complete. 2024-04-22T15:50:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.342+0000 7fdbc1cdf700 1 Processor -- start 2024-04-22T15:50:03.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.342+0000 7fdbc1cdf700 1 -- start start 2024-04-22T15:50:03.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.342+0000 7fdbc1cdf700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbbc100cb0 0x7fdbbc1964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:03.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.342+0000 7fdbc1cdf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbbc101eb0 0x7fdbbc196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:03.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.342+0000 7fdbc1cdf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc197040 con 0x7fdbbc101eb0 2024-04-22T15:50:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.342+0000 7fdbc1cdf700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc197180 con 0x7fdbbc100cb0 2024-04-22T15:50:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.342+0000 7fdbbaffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbbc101eb0 0x7fdbbc196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.342+0000 7fdbbb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbbc100cb0 0x7fdbbc1964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.342+0000 7fdbbaffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbbc101eb0 0x7fdbbc196a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58442/0 (socket says 172.21.15.16:58442) 2024-04-22T15:50:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.342+0000 7fdbbb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbbc100cb0 0x7fdbbc1964e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:36276/0 (socket says 172.21.15.16:36276) 2024-04-22T15:50:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.342+0000 7fdbbaffd700 1 -- 172.21.15.16:0/2043994193 learned_addr learned my addr 172.21.15.16:0/2043994193 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:50:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.343+0000 7fdbbb7fe700 1 -- 172.21.15.16:0/2043994193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbbc101eb0 msgr2=0x7fdbbc196a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.343+0000 7fdbbb7fe700 1 --2- 172.21.15.16:0/2043994193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbbc101eb0 0x7fdbbc196a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.343+0000 7fdbbb7fe700 1 -- 172.21.15.16:0/2043994193 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba40095f0 con 0x7fdbbc100cb0 2024-04-22T15:50:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.343+0000 7fdbbb7fe700 1 --2- 172.21.15.16:0/2043994193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbbc100cb0 0x7fdbbc1964e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00eee0 tx=0x7fdbac00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.343+0000 7fdbb8ff9700 1 -- 172.21.15.16:0/2043994193 <== mon.1 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbac018070 con 0x7fdbbc100cb0 2024-04-22T15:50:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.343+0000 7fdbc1cdf700 1 -- 172.21.15.16:0/2043994193 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbbc19bbd0 con 0x7fdbbc100cb0 2024-04-22T15:50:03.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.343+0000 7fdbb8ff9700 1 -- 172.21.15.16:0/2043994193 <== mon.1 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdbac00f040 con 0x7fdbbc100cb0 2024-04-22T15:50:03.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.343+0000 7fdbb8ff9700 1 -- 172.21.15.16:0/2043994193 <== mon.1 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbac0044f0 con 0x7fdbbc100cb0 2024-04-22T15:50:03.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.343+0000 7fdbc1cdf700 1 -- 172.21.15.16:0/2043994193 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbbc19c120 con 0x7fdbbc100cb0 2024-04-22T15:50:03.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.344+0000 7fdbb8ff9700 1 -- 172.21.15.16:0/2043994193 <== mon.1 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 29) v1 ==== 46697+0+0 (secure 0 0 0) 0x7fdbac004760 con 0x7fdbbc100cb0 2024-04-22T15:50:03.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.344+0000 7fdbc1cdf700 1 -- 172.21.15.16:0/2043994193 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbbc109810 con 0x7fdbbc100cb0 2024-04-22T15:50:03.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.345+0000 7fdbb8ff9700 1 --2- 172.21.15.16:0/2043994193 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fdba80398c0 0x7fdba803bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:03.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.345+0000 7fdbb8ff9700 1 -- 172.21.15.16:0/2043994193 <== mon.1 v2:172.21.15.27:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fdbac04dff0 con 0x7fdbbc100cb0 2024-04-22T15:50:03.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.345+0000 7fdbbaffd700 1 --2- 172.21.15.16:0/2043994193 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fdba80398c0 0x7fdba803bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:03.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.345+0000 7fdbbaffd700 1 --2- 172.21.15.16:0/2043994193 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fdba80398c0 0x7fdba803bd80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdba4009fd0 tx=0x7fdba4005940 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:03.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.350+0000 7fdbb8ff9700 1 -- 172.21.15.16:0/2043994193 <== mon.1 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7fdbac007500 con 0x7fdbbc100cb0 2024-04-22T15:50:03.363 INFO:tasks.workunit.client.1.smithi027.stdout:3/692: sync 2024-04-22T15:50:03.370 INFO:tasks.workunit.client.1.smithi027.stdout:3/693: truncate f10 1690917 0 2024-04-22T15:50:03.373 INFO:tasks.workunit.client.1.smithi027.stdout:3/694: symlink d28/d4b/d52/db2/ldf 0 2024-04-22T15:50:03.380 INFO:tasks.workunit.client.1.smithi027.stdout:3/695: getdents d28/d4b/d52/db2/d95 0 2024-04-22T15:50:03.382 INFO:tasks.workunit.client.1.smithi027.stdout:3/696: creat d28/d4b/d52/d94/fe0 x:0 0 0 2024-04-22T15:50:03.383 INFO:tasks.workunit.client.1.smithi027.stdout:3/697: write d28/fb9 [615223,115041] 0 2024-04-22T15:50:03.389 INFO:tasks.workunit.client.1.smithi027.stdout:3/698: rmdir d28/d4b/d52 39 2024-04-22T15:50:03.395 INFO:tasks.workunit.client.1.smithi027.stdout:3/699: truncate d28/d4b/f4c 2152219 0 2024-04-22T15:50:03.401 INFO:tasks.workunit.client.1.smithi027.stdout:3/700: rename d28/d4b/d52/db2/d95/d9c/dc1/cc8 to d28/ce1 0 2024-04-22T15:50:03.410 INFO:tasks.workunit.client.1.smithi027.stdout:3/701: write d28/d69/f86 [103683,83569] 0 2024-04-22T15:50:03.414 INFO:tasks.workunit.client.1.smithi027.stdout:3/702: fdatasync d28/d5d/fcb 0 2024-04-22T15:50:03.416 INFO:tasks.workunit.client.1.smithi027.stdout:3/703: write d28/d4b/d52/db2/d95/fc7 [166669,115105] 0 2024-04-22T15:50:03.420 INFO:tasks.workunit.client.1.smithi027.stdout:3/704: symlink d28/d5d/db0/dce/le2 0 2024-04-22T15:50:03.423 INFO:tasks.workunit.client.1.smithi027.stdout:3/705: unlink d28/d4b/d52/d94/fa4 0 2024-04-22T15:50:03.431 INFO:tasks.workunit.client.1.smithi027.stdout:3/706: write f1d [4548453,10548] 0 2024-04-22T15:50:03.434 INFO:tasks.workunit.client.1.smithi027.stdout:3/707: mknod d28/d4b/ce3 0 2024-04-22T15:50:03.443 INFO:tasks.workunit.client.1.smithi027.stdout:3/708: rename d28/d69/c74 to d28/d4b/d52/ce4 0 2024-04-22T15:50:03.448 INFO:tasks.workunit.client.1.smithi027.stdout:3/709: rename d28/f88 to d28/d4b/d52/db2/d95/fe5 0 2024-04-22T15:50:03.491 INFO:tasks.workunit.client.1.smithi027.stdout:7/611: sync 2024-04-22T15:50:03.491 INFO:tasks.workunit.client.1.smithi027.stdout:2/630: sync 2024-04-22T15:50:03.491 INFO:tasks.workunit.client.1.smithi027.stdout:0/668: sync 2024-04-22T15:50:03.492 INFO:tasks.workunit.client.1.smithi027.stdout:0/669: chown d0/c1b 28742 1 2024-04-22T15:50:03.493 INFO:tasks.workunit.client.1.smithi027.stdout:7/612: chown d0/d11/d1c/d2c/f55 268 1 2024-04-22T15:50:03.493 INFO:tasks.workunit.client.1.smithi027.stdout:2/631: chown dd/df/d70/d4b/d52/d9a/da4 3 1 2024-04-22T15:50:03.494 INFO:tasks.workunit.client.1.smithi027.stdout:2/632: fdatasync dd/d43/fb4 0 2024-04-22T15:50:03.496 INFO:tasks.workunit.client.1.smithi027.stdout:2/633: fsync dd/df/d1b/d96/fc3 0 2024-04-22T15:50:03.501 INFO:tasks.workunit.client.1.smithi027.stdout:7/613: dwrite d0/d3f/f5b [0,4194304] 0 2024-04-22T15:50:03.502 INFO:tasks.workunit.client.1.smithi027.stdout:0/670: dwrite d0/d28/d5f/f73 [0,4194304] 0 2024-04-22T15:50:03.509 INFO:tasks.workunit.client.1.smithi027.stdout:7/614: rename d0/d4c/d4e/lc8 to d0/d11/d1c/d31/d56/lcb 0 2024-04-22T15:50:03.509 INFO:tasks.workunit.client.1.smithi027.stdout:2/634: dwrite dd/df/d1b/d38/d60/d75/fa6 [0,4194304] 0 2024-04-22T15:50:03.511 INFO:tasks.workunit.client.1.smithi027.stdout:2/635: chown dd/df/d70/d5a/c66 954174 1 2024-04-22T15:50:03.595 INFO:tasks.workunit.client.1.smithi027.stdout:7/615: truncate d0/d11/d1c/d2c/d93/f59 5036667 0 2024-04-22T15:50:03.596 INFO:tasks.workunit.client.1.smithi027.stdout:0/671: getdents d0/d34/d4e 0 2024-04-22T15:50:03.598 INFO:tasks.workunit.client.1.smithi027.stdout:2/636: symlink dd/dae/lc8 0 2024-04-22T15:50:03.599 INFO:tasks.workunit.client.1.smithi027.stdout:0/672: creat d0/d28/d63/d67/fd3 x:0 0 0 2024-04-22T15:50:03.605 INFO:tasks.workunit.client.1.smithi027.stdout:7/616: creat d0/fcc x:0 0 0 2024-04-22T15:50:03.607 INFO:tasks.workunit.client.1.smithi027.stdout:2/637: dwrite dd/df/d1b/fc1 [0,4194304] 0 2024-04-22T15:50:03.612 INFO:tasks.workunit.client.1.smithi027.stdout:0/673: dread d0/d2a/f2b [0,4194304] 0 2024-04-22T15:50:03.613 INFO:tasks.workunit.client.1.smithi027.stdout:7/617: creat d0/fcd x:0 0 0 2024-04-22T15:50:03.615 INFO:tasks.workunit.client.1.smithi027.stdout:2/638: mkdir dd/df/d1b/dc9 0 2024-04-22T15:50:03.616 INFO:tasks.workunit.client.1.smithi027.stdout:0/674: creat d0/da3/dae/fd4 x:0 0 0 2024-04-22T15:50:03.621 INFO:tasks.workunit.client.1.smithi027.stdout:0/675: creat d0/d34/d4e/d85/fd5 x:0 0 0 2024-04-22T15:50:03.623 INFO:tasks.workunit.client.1.smithi027.stdout:0/676: chown d0/d2a/d6a/d6c/f93 126 1 2024-04-22T15:50:03.623 INFO:tasks.workunit.client.1.smithi027.stdout:2/639: truncate dd/df/f17 4536419 0 2024-04-22T15:50:03.624 INFO:tasks.workunit.client.1.smithi027.stdout:2/640: chown dd/df/d70/d4b/d52/d9a/da4/dad/d8d 513386374 1 2024-04-22T15:50:03.631 INFO:tasks.workunit.client.1.smithi027.stdout:0/677: rmdir d0/d28 39 2024-04-22T15:50:03.638 INFO:tasks.workunit.client.1.smithi027.stdout:7/618: dwrite d0/fb [0,4194304] 0 2024-04-22T15:50:03.639 INFO:tasks.workunit.client.1.smithi027.stdout:2/641: dwrite dd/df/d1b/d38/d60/d75/fbb [0,4194304] 0 2024-04-22T15:50:03.684 INFO:tasks.workunit.client.1.smithi027.stdout:0/678: unlink d0/d34/d4e/d85/f8f 0 2024-04-22T15:50:03.685 INFO:tasks.workunit.client.1.smithi027.stdout:0/679: dread - d0/d2a/d5e/d8b/fa5 zero size 2024-04-22T15:50:03.686 INFO:tasks.workunit.client.1.smithi027.stdout:7/619: creat d0/d11/d1c/d31/d8a/fce x:0 0 0 2024-04-22T15:50:03.688 INFO:tasks.workunit.client.1.smithi027.stdout:7/620: write d0/f3e [801035,31022] 0 2024-04-22T15:50:03.693 INFO:tasks.workunit.client.1.smithi027.stdout:0/680: fsync d0/d34/f5a 0 2024-04-22T15:50:03.693 INFO:tasks.workunit.client.1.smithi027.stdout:0/681: readlink d0/d2a/d6a/l78 0 2024-04-22T15:50:03.694 INFO:tasks.workunit.client.1.smithi027.stdout:7/621: symlink d0/d11/d1c/d2c/d96/db3/lcf 0 2024-04-22T15:50:03.699 INFO:tasks.workunit.client.1.smithi027.stdout:0/682: creat d0/d2a/d38/fd6 x:0 0 0 2024-04-22T15:50:03.700 INFO:tasks.workunit.client.1.smithi027.stdout:7/622: creat d0/d4c/d8b/d72/db2/fd0 x:0 0 0 2024-04-22T15:50:03.700 INFO:tasks.workunit.client.1.smithi027.stdout:7/623: readlink d0/d3f/l41 0 2024-04-22T15:50:03.702 INFO:tasks.workunit.client.1.smithi027.stdout:2/642: getdents dd/df/d70 0 2024-04-22T15:50:03.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.706+0000 7fdbc1cdf700 1 -- 172.21.15.16:0/2043994193 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fdbbc19c730 con 0x7fdbbc100cb0 2024-04-22T15:50:03.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.707+0000 7fdbb8ff9700 1 -- 172.21.15.16:0/2043994193 <== mon.1 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7fdbac005000 con 0x7fdbbc100cb0 2024-04-22T15:50:03.710 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:50:03.710 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-22T15:50:03.710 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T15:50:03.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:50:03.711 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-22T15:50:03.711 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 1 2024-04-22T15:50:03.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:50:03.711 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-22T15:50:03.711 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-22T15:50:03.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:50:03.711 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-22T15:50:03.711 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T15:50:03.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:50:03.711 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-22T15:50:03.711 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-04-22T15:50:03.711 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 1 2024-04-22T15:50:03.712 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-22T15:50:03.712 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:50:03.712 INFO:tasks.workunit.client.1.smithi027.stdout:0/683: dwrite d0/d34/f71 [0,4194304] 0 2024-04-22T15:50:03.712 INFO:tasks.workunit.client.1.smithi027.stdout:2/643: truncate dd/df/d1b/f5e 51204 0 2024-04-22T15:50:03.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.710+0000 7fdbc1cdf700 1 -- 172.21.15.16:0/2043994193 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fdba80398c0 msgr2=0x7fdba803bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:03.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.710+0000 7fdbc1cdf700 1 --2- 172.21.15.16:0/2043994193 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fdba80398c0 0x7fdba803bd80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdba4009fd0 tx=0x7fdba4005940 comp rx=0 tx=0).stop 2024-04-22T15:50:03.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.710+0000 7fdbc1cdf700 1 -- 172.21.15.16:0/2043994193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbbc100cb0 msgr2=0x7fdbbc1964e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:03.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.710+0000 7fdbc1cdf700 1 --2- 172.21.15.16:0/2043994193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbbc100cb0 0x7fdbbc1964e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00eee0 tx=0x7fdbac00c3a0 comp rx=0 tx=0).stop 2024-04-22T15:50:03.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.710+0000 7fdbc1cdf700 1 -- 172.21.15.16:0/2043994193 shutdown_connections 2024-04-22T15:50:03.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.710+0000 7fdbc1cdf700 1 --2- 172.21.15.16:0/2043994193 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fdba80398c0 0x7fdba803bd80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:03.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.710+0000 7fdbc1cdf700 1 --2- 172.21.15.16:0/2043994193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbbc100cb0 0x7fdbbc1964e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:03.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.710+0000 7fdbc1cdf700 1 --2- 172.21.15.16:0/2043994193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbbc101eb0 0x7fdbbc196a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:03.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.710+0000 7fdbc1cdf700 1 -- 172.21.15.16:0/2043994193 >> 172.21.15.16:0/2043994193 conn(0x7fdbbc0fc230 msgr2=0x7fdbbc1050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:03.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.710+0000 7fdbc1cdf700 1 -- 172.21.15.16:0/2043994193 shutdown_connections 2024-04-22T15:50:03.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.710+0000 7fdbc1cdf700 1 -- 172.21.15.16:0/2043994193 wait complete. 2024-04-22T15:50:03.714 INFO:tasks.workunit.client.1.smithi027.stdout:7/624: dwrite d0/d11/d1c/d31/f47 [4194304,4194304] 0 2024-04-22T15:50:03.715 INFO:tasks.workunit.client.1.smithi027.stdout:2/644: write dd/df/d1b/d38/d60/f65 [2127267,92435] 0 2024-04-22T15:50:03.717 INFO:tasks.workunit.client.1.smithi027.stdout:2/645: fsync dd/df/d1b/d38/d60/d75/fbb 0 2024-04-22T15:50:03.721 INFO:tasks.workunit.client.1.smithi027.stdout:0/684: mknod d0/d34/d4e/cd7 0 2024-04-22T15:50:03.725 INFO:tasks.workunit.client.1.smithi027.stdout:7/625: dwrite d0/d11/d3a/fa7 [0,4194304] 0 2024-04-22T15:50:03.726 INFO:tasks.workunit.client.1.smithi027.stdout:7/626: read d0/d3f/f5b [324691,26688] 0 2024-04-22T15:50:03.727 INFO:tasks.workunit.client.1.smithi027.stdout:7/627: dread - d0/d3f/fca zero size 2024-04-22T15:50:03.731 INFO:tasks.workunit.client.1.smithi027.stdout:0/685: getdents d0/d34 0 2024-04-22T15:50:03.738 INFO:tasks.workunit.client.1.smithi027.stdout:0/686: mknod d0/d2a/d58/dbf/cd8 0 2024-04-22T15:50:03.740 INFO:tasks.workunit.client.1.smithi027.stdout:2/646: dread dd/df/d1b/d38/d60/f80 [0,4194304] 0 2024-04-22T15:50:03.742 INFO:tasks.workunit.client.1.smithi027.stdout:0/687: creat d0/d2a/d58/d66/fd9 x:0 0 0 2024-04-22T15:50:03.742 INFO:tasks.workunit.client.1.smithi027.stdout:0/688: fsync d0/d2a/fa7 0 2024-04-22T15:50:03.748 INFO:tasks.workunit.client.1.smithi027.stdout:7/628: write d0/d11/d1c/d31/d56/f58 [537154,129256] 0 2024-04-22T15:50:03.754 INFO:tasks.workunit.client.1.smithi027.stdout:2/647: write dd/df/d1b/f76 [202268,4858] 0 2024-04-22T15:50:03.755 INFO:tasks.workunit.client.1.smithi027.stdout:2/648: dread - dd/df/d70/d4b/d52/fb6 zero size 2024-04-22T15:50:03.756 INFO:tasks.workunit.client.1.smithi027.stdout:2/649: chown dd/df/d1b/d38/d60/d75 142 1 2024-04-22T15:50:03.757 INFO:tasks.workunit.client.1.smithi027.stdout:2/650: readlink dd/df/l79 0 2024-04-22T15:50:03.757 INFO:tasks.workunit.client.1.smithi027.stdout:0/689: link d0/d28/d37/l46 d0/d2a/d38/lda 0 2024-04-22T15:50:03.758 INFO:tasks.workunit.client.1.smithi027.stdout:7/629: getdents d0/d11/d1c/d31/d56 0 2024-04-22T15:50:03.759 INFO:tasks.workunit.client.1.smithi027.stdout:0/690: stat d0/d34/f9c 0 2024-04-22T15:50:03.762 INFO:tasks.workunit.client.1.smithi027.stdout:0/691: creat d0/d2a/d58/dbf/fdb x:0 0 0 2024-04-22T15:50:03.766 INFO:tasks.workunit.client.1.smithi027.stdout:2/651: dwrite dd/df/d70/d4b/d52/d9a/fc2 [0,4194304] 0 2024-04-22T15:50:03.767 INFO:tasks.workunit.client.1.smithi027.stdout:7/630: link d0/d4c/d4e/f67 d0/d4c/fd1 0 2024-04-22T15:50:03.775 INFO:tasks.workunit.client.1.smithi027.stdout:7/631: dwrite d0/d11/fe [0,4194304] 0 2024-04-22T15:50:03.779 INFO:tasks.workunit.client.1.smithi027.stdout:0/692: dread d0/d34/f5a [0,4194304] 0 2024-04-22T15:50:03.784 INFO:tasks.workunit.client.1.smithi027.stdout:2/652: mkdir dd/df/d1b/dc9/dca 0 2024-04-22T15:50:03.788 INFO:tasks.workunit.client.1.smithi027.stdout:0/693: mknod d0/cdc 0 2024-04-22T15:50:03.788 INFO:tasks.workunit.client.1.smithi027.stdout:2/653: dread - dd/df/d1b/d1f/f74 zero size 2024-04-22T15:50:03.800 INFO:tasks.workunit.client.1.smithi027.stdout:7/632: truncate d0/d11/d3a/f45 2152281 0 2024-04-22T15:50:03.805 INFO:tasks.workunit.client.1.smithi027.stdout:7/633: unlink d0/d11/d1c/d31/d62/fc3 0 2024-04-22T15:50:03.806 INFO:tasks.workunit.client.1.smithi027.stdout:4/630: sync 2024-04-22T15:50:03.806 INFO:tasks.workunit.client.1.smithi027.stdout:9/620: sync 2024-04-22T15:50:03.809 INFO:tasks.workunit.client.1.smithi027.stdout:4/631: symlink d5/d11/d6c/lee 0 2024-04-22T15:50:03.810 INFO:tasks.workunit.client.1.smithi027.stdout:9/621: symlink d0/d51/d9d/lcf 0 2024-04-22T15:50:03.811 INFO:tasks.workunit.client.1.smithi027.stdout:0/694: write d0/f22 [4738608,11455] 0 2024-04-22T15:50:03.813 INFO:tasks.workunit.client.1.smithi027.stdout:0/695: fdatasync d0/f12 0 2024-04-22T15:50:03.814 INFO:tasks.workunit.client.1.smithi027.stdout:0/696: chown d0/d28/d63/f7d 16952326 1 2024-04-22T15:50:03.815 INFO:tasks.workunit.client.1.smithi027.stdout:4/632: dwrite d5/d11/f80 [0,4194304] 0 2024-04-22T15:50:03.817 INFO:tasks.workunit.client.1.smithi027.stdout:9/622: mkdir d0/dc/de/dd0 0 2024-04-22T15:50:03.817 INFO:tasks.workunit.client.1.smithi027.stdout:2/654: write dd/f11 [1925394,34404] 0 2024-04-22T15:50:03.818 INFO:tasks.workunit.client.1.smithi027.stdout:9/623: stat d0/dc/f8e 0 2024-04-22T15:50:03.818 INFO:tasks.workunit.client.1.smithi027.stdout:0/697: mkdir d0/d34/d4e/d85/ddd 0 2024-04-22T15:50:03.820 INFO:tasks.workunit.client.1.smithi027.stdout:4/633: creat d5/da8/fef x:0 0 0 2024-04-22T15:50:03.821 INFO:tasks.workunit.client.1.smithi027.stdout:2/655: mkdir dd/df/d70/d4b/d4e/dcb 0 2024-04-22T15:50:03.824 INFO:tasks.workunit.client.1.smithi027.stdout:9/624: mknod d0/dc/d12/d30/d99/da7/cd1 0 2024-04-22T15:50:03.825 INFO:tasks.workunit.client.1.smithi027.stdout:4/634: creat d5/d11/d1d/d54/d8c/ff0 x:0 0 0 2024-04-22T15:50:03.828 INFO:tasks.workunit.client.1.smithi027.stdout:0/698: truncate d0/d2a/f35 761101 0 2024-04-22T15:50:03.834 INFO:tasks.workunit.client.1.smithi027.stdout:4/635: symlink d5/lf1 0 2024-04-22T15:50:03.836 INFO:tasks.workunit.client.1.smithi027.stdout:9/625: dread d0/dc/d12/d58/f6a [0,4194304] 0 2024-04-22T15:50:03.838 INFO:tasks.workunit.client.1.smithi027.stdout:0/699: getdents d0/d2a/d6a/d6c/d83 0 2024-04-22T15:50:03.844 INFO:tasks.workunit.client.1.smithi027.stdout:0/700: mknod d0/d28/d5f/cde 0 2024-04-22T15:50:03.845 INFO:tasks.workunit.client.1.smithi027.stdout:0/701: dread - d0/d2a/d38/fad zero size 2024-04-22T15:50:03.846 INFO:tasks.workunit.client.1.smithi027.stdout:0/702: dread - d0/d28/d5f/fd0 zero size 2024-04-22T15:50:03.848 INFO:tasks.workunit.client.1.smithi027.stdout:5/688: sync 2024-04-22T15:50:03.848 INFO:tasks.workunit.client.1.smithi027.stdout:1/580: sync 2024-04-22T15:50:03.848 INFO:tasks.workunit.client.1.smithi027.stdout:6/774: sync 2024-04-22T15:50:03.848 INFO:tasks.workunit.client.1.smithi027.stdout:3/710: sync 2024-04-22T15:50:03.848 INFO:tasks.workunit.client.1.smithi027.stdout:8/588: sync 2024-04-22T15:50:03.850 INFO:tasks.workunit.client.1.smithi027.stdout:7/634: truncate d0/d11/d1c/d2c/fc4 523408 0 2024-04-22T15:50:03.850 INFO:tasks.workunit.client.1.smithi027.stdout:3/711: write d28/f36 [6225341,14597] 0 2024-04-22T15:50:03.852 INFO:tasks.workunit.client.1.smithi027.stdout:4/636: dread d5/de/d2a/d58/faf [0,4194304] 0 2024-04-22T15:50:03.853 INFO:tasks.workunit.client.1.smithi027.stdout:7/635: readlink d0/d4c/d8b/la9 0 2024-04-22T15:50:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.850+0000 7fa7b0523700 1 -- 172.21.15.16:0/2090600755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a8101800 msgr2=0x7fa7a8103bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.850+0000 7fa7b0523700 1 --2- 172.21.15.16:0/2090600755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a8101800 0x7fa7a8103bf0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4009910 tx=0x7fa7a4009c20 comp rx=0 tx=0).stop 2024-04-22T15:50:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.851+0000 7fa7b0523700 1 -- 172.21.15.16:0/2090600755 shutdown_connections 2024-04-22T15:50:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.851+0000 7fa7b0523700 1 --2- 172.21.15.16:0/2090600755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7a8104130 0x7fa7a8106520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.851+0000 7fa7b0523700 1 --2- 172.21.15.16:0/2090600755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a8101800 0x7fa7a8103bf0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.851+0000 7fa7b0523700 1 -- 172.21.15.16:0/2090600755 >> 172.21.15.16:0/2090600755 conn(0x7fa7a80fb190 msgr2=0x7fa7a80fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:03.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.851+0000 7fa7b0523700 1 -- 172.21.15.16:0/2090600755 shutdown_connections 2024-04-22T15:50:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.851+0000 7fa7b0523700 1 -- 172.21.15.16:0/2090600755 wait complete. 2024-04-22T15:50:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.852+0000 7fa7b0523700 1 Processor -- start 2024-04-22T15:50:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.852+0000 7fa7b0523700 1 -- start start 2024-04-22T15:50:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.852+0000 7fa7b0523700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a8101800 0x7fa7a8196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.852+0000 7fa7b0523700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7a8104130 0x7fa7a81969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.852+0000 7fa7b0523700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a8196fe0 con 0x7fa7a8101800 2024-04-22T15:50:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.852+0000 7fa7b0523700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a8197120 con 0x7fa7a8104130 2024-04-22T15:50:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.852+0000 7fa7ae2bf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a8101800 0x7fa7a8196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.852+0000 7fa7ae2bf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a8101800 0x7fa7a8196480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58466/0 (socket says 172.21.15.16:58466) 2024-04-22T15:50:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.852+0000 7fa7ae2bf700 1 -- 172.21.15.16:0/3284195683 learned_addr learned my addr 172.21.15.16:0/3284195683 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:50:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.853+0000 7fa7adabe700 1 --2- 172.21.15.16:0/3284195683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7a8104130 0x7fa7a81969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.853+0000 7fa7ae2bf700 1 -- 172.21.15.16:0/3284195683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7a8104130 msgr2=0x7fa7a81969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.853+0000 7fa7ae2bf700 1 --2- 172.21.15.16:0/3284195683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7a8104130 0x7fa7a81969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.853+0000 7fa7ae2bf700 1 -- 172.21.15.16:0/3284195683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a40095f0 con 0x7fa7a8101800 2024-04-22T15:50:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.853+0000 7fa7ae2bf700 1 --2- 172.21.15.16:0/3284195683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a8101800 0x7fa7a8196480 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4006010 tx=0x7fa7a400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:03.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.853+0000 7fa79f7fe700 1 -- 172.21.15.16:0/3284195683 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7a401d070 con 0x7fa7a8101800 2024-04-22T15:50:03.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.853+0000 7fa7b0523700 1 -- 172.21.15.16:0/3284195683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7a819bb70 con 0x7fa7a8101800 2024-04-22T15:50:03.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.853+0000 7fa79f7fe700 1 -- 172.21.15.16:0/3284195683 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa7a400f460 con 0x7fa7a8101800 2024-04-22T15:50:03.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.853+0000 7fa79f7fe700 1 -- 172.21.15.16:0/3284195683 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7a4021640 con 0x7fa7a8101800 2024-04-22T15:50:03.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.853+0000 7fa7b0523700 1 -- 172.21.15.16:0/3284195683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a819c060 con 0x7fa7a8101800 2024-04-22T15:50:03.858 INFO:tasks.workunit.client.1.smithi027.stdout:1/581: mkdir d2/d1d/d56/dc4 0 2024-04-22T15:50:03.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.855+0000 7fa7b0523700 1 -- 172.21.15.16:0/3284195683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7a8190610 con 0x7fa7a8101800 2024-04-22T15:50:03.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.855+0000 7fa79f7fe700 1 -- 172.21.15.16:0/3284195683 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 29) v1 ==== 46697+0+0 (secure 0 0 0) 0x7fa7a400fab0 con 0x7fa7a8101800 2024-04-22T15:50:03.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.855+0000 7fa79f7fe700 1 --2- 172.21.15.16:0/3284195683 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fa7940398c0 0x7fa79403bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:03.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.855+0000 7fa79f7fe700 1 -- 172.21.15.16:0/3284195683 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fa7a404eeb0 con 0x7fa7a8101800 2024-04-22T15:50:03.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.856+0000 7fa7adabe700 1 --2- 172.21.15.16:0/3284195683 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fa7940398c0 0x7fa79403bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:03.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.856+0000 7fa7adabe700 1 --2- 172.21.15.16:0/3284195683 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fa7940398c0 0x7fa79403bd80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa798005950 tx=0x7fa79800b410 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:03.860 INFO:tasks.workunit.client.1.smithi027.stdout:1/582: write d2/d30/fa3 [4644097,118626] 0 2024-04-22T15:50:03.861 INFO:tasks.workunit.client.1.smithi027.stdout:3/712: unlink d28/f37 0 2024-04-22T15:50:03.862 INFO:tasks.workunit.client.1.smithi027.stdout:8/589: dwrite d1/d3/d1c/d32/d41/d52/fbf [0,4194304] 0 2024-04-22T15:50:03.863 INFO:tasks.workunit.client.1.smithi027.stdout:0/703: getdents d0/da3 0 2024-04-22T15:50:03.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:03.861+0000 7fa79f7fe700 1 -- 172.21.15.16:0/3284195683 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7fa7a40298f0 con 0x7fa7a8101800 2024-04-22T15:50:03.871 INFO:tasks.workunit.client.1.smithi027.stdout:7/636: unlink d0/d11/d1c/d31/f47 0 2024-04-22T15:50:03.877 INFO:tasks.workunit.client.1.smithi027.stdout:5/689: creat d0/df/de/dbb/fe1 x:0 0 0 2024-04-22T15:50:03.880 INFO:tasks.workunit.client.1.smithi027.stdout:1/583: write d2/d5f/d64/d81/d69/d7d/fa8 [35829,120136] 0 2024-04-22T15:50:03.880 INFO:tasks.workunit.client.1.smithi027.stdout:5/690: chown d0/df/de/d24/d1c/l95 4819 1 2024-04-22T15:50:03.883 INFO:tasks.workunit.client.1.smithi027.stdout:3/713: creat d28/d4b/d52/db2/d95/d9c/fe6 x:0 0 0 2024-04-22T15:50:03.883 INFO:tasks.workunit.client.1.smithi027.stdout:8/590: dread d1/d3/d1c/d32/fc5 [0,4194304] 0 2024-04-22T15:50:03.885 INFO:tasks.workunit.client.1.smithi027.stdout:3/714: chown l25 588 1 2024-04-22T15:50:03.885 INFO:tasks.workunit.client.1.smithi027.stdout:0/704: mkdir d0/da3/ddf 0 2024-04-22T15:50:03.886 INFO:tasks.workunit.client.1.smithi027.stdout:2/656: dwrite dd/f35 [0,4194304] 0 2024-04-22T15:50:03.895 INFO:tasks.workunit.client.1.smithi027.stdout:8/591: dwrite d1/d3/d1c/d43/d82/f84 [0,4194304] 0 2024-04-22T15:50:03.904 INFO:tasks.workunit.client.1.smithi027.stdout:7/637: mknod d0/d4c/cd2 0 2024-04-22T15:50:03.912 INFO:tasks.workunit.client.1.smithi027.stdout:5/691: stat d0/d1a/d22/d26/d32/d79/f87 0 2024-04-22T15:50:03.918 INFO:tasks.workunit.client.1.smithi027.stdout:3/715: unlink d28/c3d 0 2024-04-22T15:50:03.924 INFO:tasks.workunit.client.1.smithi027.stdout:8/592: dread d1/d3/f40 [0,4194304] 0 2024-04-22T15:50:03.928 INFO:tasks.workunit.client.1.smithi027.stdout:9/626: dwrite d0/dc/d12/d58/d6b/f62 [0,4194304] 0 2024-04-22T15:50:03.932 INFO:tasks.workunit.client.1.smithi027.stdout:9/627: read d0/dc/d12/d58/f6a [112812,14483] 0 2024-04-22T15:50:03.933 INFO:tasks.workunit.client.1.smithi027.stdout:2/657: creat dd/df/d70/d4b/d52/d9a/da4/fcc x:0 0 0 2024-04-22T15:50:03.936 INFO:tasks.workunit.client.1.smithi027.stdout:2/658: chown la 64836 1 2024-04-22T15:50:03.937 INFO:tasks.workunit.client.1.smithi027.stdout:1/584: dread d2/f70 [0,4194304] 0 2024-04-22T15:50:03.941 INFO:tasks.workunit.client.1.smithi027.stdout:7/638: creat d0/d11/d1c/fd3 x:0 0 0 2024-04-22T15:50:03.948 INFO:tasks.workunit.client.1.smithi027.stdout:3/716: creat d28/d4b/d52/db2/d95/d9c/dc1/fe7 x:0 0 0 2024-04-22T15:50:03.952 INFO:tasks.workunit.client.1.smithi027.stdout:4/637: dwrite d5/de/f16 [4194304,4194304] 0 2024-04-22T15:50:03.963 INFO:tasks.workunit.client.1.smithi027.stdout:0/705: mkdir d0/d2a/de0 0 2024-04-22T15:50:03.965 INFO:tasks.workunit.client.1.smithi027.stdout:9/628: creat d0/dc/d12/d58/da6/fd2 x:0 0 0 2024-04-22T15:50:03.969 INFO:tasks.workunit.client.1.smithi027.stdout:0/706: write d0/d34/d4e/fab [828377,31982] 0 2024-04-22T15:50:03.969 INFO:tasks.workunit.client.1.smithi027.stdout:9/629: dread - d0/dc/d12/d58/d6b/d6e/fbd zero size 2024-04-22T15:50:03.970 INFO:tasks.workunit.client.1.smithi027.stdout:0/707: read d0/d2a/f2b [3049594,53773] 0 2024-04-22T15:50:03.983 INFO:tasks.workunit.client.1.smithi027.stdout:4/638: rename d5/d11/d89 to d5/de/df2 0 2024-04-22T15:50:03.983 INFO:tasks.workunit.client.1.smithi027.stdout:6/775: truncate d5/dc/d26/d3d/f83 3481177 0 2024-04-22T15:50:03.984 INFO:tasks.workunit.client.1.smithi027.stdout:9/630: dread d0/d51/d9d/fad [0,4194304] 0 2024-04-22T15:50:03.987 INFO:tasks.workunit.client.1.smithi027.stdout:6/776: creat d5/dc/d26/dee/f10d x:0 0 0 2024-04-22T15:50:03.993 INFO:tasks.workunit.client.1.smithi027.stdout:4/639: symlink d5/d45/ddc/lf3 0 2024-04-22T15:50:03.995 INFO:tasks.workunit.client.1.smithi027.stdout:6/777: read d5/dc/d17/d53/f81 [4131627,62540] 0 2024-04-22T15:50:03.996 INFO:tasks.workunit.client.1.smithi027.stdout:0/708: creat d0/fe1 x:0 0 0 2024-04-22T15:50:03.998 INFO:tasks.workunit.client.1.smithi027.stdout:3/717: getdents d28/d4b 0 2024-04-22T15:50:03.999 INFO:tasks.workunit.client.1.smithi027.stdout:4/640: dread d5/d45/d5a/fb0 [0,4194304] 0 2024-04-22T15:50:04.003 INFO:tasks.workunit.client.1.smithi027.stdout:2/659: rmdir dd/df/d70/d4b/d52 39 2024-04-22T15:50:04.004 INFO:tasks.workunit.client.1.smithi027.stdout:0/709: symlink d0/d28/le2 0 2024-04-22T15:50:04.006 INFO:tasks.workunit.client.1.smithi027.stdout:3/718: rmdir d28/d4b/d52/db2 39 2024-04-22T15:50:04.008 INFO:tasks.workunit.client.1.smithi027.stdout:4/641: symlink d5/de/d30/d5d/lf4 0 2024-04-22T15:50:04.015 INFO:tasks.workunit.client.1.smithi027.stdout:5/692: dwrite d0/df/de/d24/fb8 [0,4194304] 0 2024-04-22T15:50:04.015 INFO:tasks.workunit.client.1.smithi027.stdout:6/778: mkdir d5/dc/d17/d10e 0 2024-04-22T15:50:04.017 INFO:tasks.workunit.client.1.smithi027.stdout:6/779: fdatasync d5/dc/dd/fa2 0 2024-04-22T15:50:04.019 INFO:tasks.workunit.client.1.smithi027.stdout:8/593: dwrite d1/d3/d1c/d32/d41/d52/f60 [0,4194304] 0 2024-04-22T15:50:04.020 INFO:tasks.workunit.client.1.smithi027.stdout:5/693: write d0/df/de/d24/f2e [4592102,3100] 0 2024-04-22T15:50:04.020 INFO:tasks.workunit.client.1.smithi027.stdout:6/780: fsync d5/db/dcd/ff3 0 2024-04-22T15:50:04.023 INFO:tasks.workunit.client.1.smithi027.stdout:6/781: write d5/dc/dd/dd0/d34/fd5 [1023980,128496] 0 2024-04-22T15:50:04.023 INFO:tasks.workunit.client.1.smithi027.stdout:4/642: unlink d5/f93 0 2024-04-22T15:50:04.033 INFO:tasks.workunit.client.1.smithi027.stdout:0/710: mknod d0/d2a/de0/ce3 0 2024-04-22T15:50:04.038 INFO:tasks.workunit.client.1.smithi027.stdout:5/694: symlink d0/d1a/d22/d26/d32/le2 0 2024-04-22T15:50:04.039 INFO:tasks.workunit.client.1.smithi027.stdout:9/631: truncate d0/dc/d12/fb3 201473 0 2024-04-22T15:50:04.039 INFO:tasks.workunit.client.1.smithi027.stdout:5/695: dread - d0/df/d3d/d81/fbc zero size 2024-04-22T15:50:04.042 INFO:tasks.workunit.client.1.smithi027.stdout:6/782: creat d5/d30/dbe/f10f x:0 0 0 2024-04-22T15:50:04.048 INFO:tasks.workunit.client.1.smithi027.stdout:0/711: mkdir d0/d2a/d58/d66/de4 0 2024-04-22T15:50:04.050 INFO:tasks.workunit.client.1.smithi027.stdout:6/783: dwrite d5/d30/d40/fad [0,4194304] 0 2024-04-22T15:50:04.052 INFO:tasks.workunit.client.1.smithi027.stdout:2/660: write dd/f19 [1162857,80627] 0 2024-04-22T15:50:04.059 INFO:tasks.workunit.client.1.smithi027.stdout:5/696: dread d0/df/f45 [0,4194304] 0 2024-04-22T15:50:04.060 INFO:tasks.workunit.client.1.smithi027.stdout:7/639: fsync d0/d11/d1c/d2c/d93/f59 0 2024-04-22T15:50:04.064 INFO:tasks.workunit.client.1.smithi027.stdout:3/719: link d28/c31 d28/d5d/ce8 0 2024-04-22T15:50:04.068 INFO:tasks.workunit.client.1.smithi027.stdout:0/712: mkdir d0/d2a/d91/de5 0 2024-04-22T15:50:04.073 INFO:tasks.workunit.client.1.smithi027.stdout:6/784: fsync d5/d30/d54/fea 0 2024-04-22T15:50:04.075 INFO:tasks.workunit.client.1.smithi027.stdout:4/643: dwrite d5/d7/f5b [0,4194304] 0 2024-04-22T15:50:04.080 INFO:tasks.workunit.client.1.smithi027.stdout:4/644: truncate d5/d11/d1d/d7b/f92 708656 0 2024-04-22T15:50:04.080 INFO:tasks.workunit.client.1.smithi027.stdout:0/713: dwrite d0/d34/d59/fc1 [0,4194304] 0 2024-04-22T15:50:04.082 INFO:tasks.workunit.client.1.smithi027.stdout:4/645: fdatasync d5/d11/f80 0 2024-04-22T15:50:04.089 INFO:tasks.workunit.client.1.smithi027.stdout:7/640: creat d0/d4c/fd4 x:0 0 0 2024-04-22T15:50:04.091 INFO:tasks.workunit.client.1.smithi027.stdout:8/594: creat d1/d3/d1c/d32/d41/d52/d61/fce x:0 0 0 2024-04-22T15:50:04.094 INFO:tasks.workunit.client.1.smithi027.stdout:2/661: write dd/df/d1b/f2e [19538,119687] 0 2024-04-22T15:50:04.098 INFO:tasks.workunit.client.1.smithi027.stdout:9/632: mkdir d0/dc/de/db5/dbf/dd3 0 2024-04-22T15:50:04.101 INFO:tasks.workunit.client.1.smithi027.stdout:8/595: dwrite d1/d3/d1c/d43/f71 [0,4194304] 0 2024-04-22T15:50:04.103 INFO:tasks.workunit.client.1.smithi027.stdout:5/697: dread d0/df/de/d24/f35 [0,4194304] 0 2024-04-22T15:50:04.108 INFO:tasks.workunit.client.1.smithi027.stdout:4/646: rename d5/d11/d1d/f5c to d5/d45/ddc/ff5 0 2024-04-22T15:50:04.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:03 smithi016 ceph-mon[26620]: from='client.14706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:04.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:03 smithi016 ceph-mon[26620]: from='client.14710 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:04.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:03 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/2043994193' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:50:04.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:04.200+0000 7fa7b0523700 1 -- 172.21.15.16:0/3284195683 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7a80611d0 con 0x7fa7940398c0 2024-04-22T15:50:04.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:04.201+0000 7fa79f7fe700 1 -- 172.21.15.16:0/3284195683 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7fa7a80611d0 con 0x7fa7940398c0 2024-04-22T15:50:04.204 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:50:04.204 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:50:04.204 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-22T15:50:04.204 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T15:50:04.204 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-22T15:50:04.204 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-04-22T15:50:04.204 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-22T15:50:04.204 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "2/2 daemons upgraded", 2024-04-22T15:50:04.204 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "", 2024-04-22T15:50:04.204 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-22T15:50:04.205 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:50:04.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:04.203+0000 7fa7b0523700 1 -- 172.21.15.16:0/3284195683 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fa7940398c0 msgr2=0x7fa79403bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:04.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:04.203+0000 7fa7b0523700 1 --2- 172.21.15.16:0/3284195683 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fa7940398c0 0x7fa79403bd80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa798005950 tx=0x7fa79800b410 comp rx=0 tx=0).stop 2024-04-22T15:50:04.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:04.203+0000 7fa7b0523700 1 -- 172.21.15.16:0/3284195683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a8101800 msgr2=0x7fa7a8196480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:04.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:04.203+0000 7fa7b0523700 1 --2- 172.21.15.16:0/3284195683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a8101800 0x7fa7a8196480 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4006010 tx=0x7fa7a400bd70 comp rx=0 tx=0).stop 2024-04-22T15:50:04.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:04.203+0000 7fa7b0523700 1 -- 172.21.15.16:0/3284195683 shutdown_connections 2024-04-22T15:50:04.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:04.203+0000 7fa7b0523700 1 --2- 172.21.15.16:0/3284195683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a8101800 0x7fa7a8196480 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:04.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:04.203+0000 7fa7b0523700 1 --2- 172.21.15.16:0/3284195683 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fa7940398c0 0x7fa79403bd80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:04.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:04.203+0000 7fa7b0523700 1 --2- 172.21.15.16:0/3284195683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7a8104130 0x7fa7a81969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:04.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:04.203+0000 7fa7b0523700 1 -- 172.21.15.16:0/3284195683 >> 172.21.15.16:0/3284195683 conn(0x7fa7a80fb190 msgr2=0x7fa7a80fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:04.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:04.203+0000 7fa7b0523700 1 -- 172.21.15.16:0/3284195683 shutdown_connections 2024-04-22T15:50:04.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:04.203+0000 7fa7b0523700 1 -- 172.21.15.16:0/3284195683 wait complete. 2024-04-22T15:50:04.207 INFO:tasks.workunit.client.1.smithi027.stdout:3/720: chown d28/d5d/d79/fa1 10600343 1 2024-04-22T15:50:04.210 INFO:tasks.workunit.client.1.smithi027.stdout:7/641: write d0/d3f/f48 [1023108,68308] 0 2024-04-22T15:50:04.215 INFO:tasks.workunit.client.1.smithi027.stdout:9/633: mkdir d0/dc/d12/d30/dd4 0 2024-04-22T15:50:04.215 INFO:tasks.workunit.client.1.smithi027.stdout:7/642: dread d0/d11/fe [0,4194304] 0 2024-04-22T15:50:04.216 INFO:tasks.workunit.client.1.smithi027.stdout:0/714: truncate d0/d28/d37/f40 2118022 0 2024-04-22T15:50:04.217 INFO:tasks.workunit.client.1.smithi027.stdout:7/643: dread - d0/d4c/d8b/d72/db2/fb7 zero size 2024-04-22T15:50:04.219 INFO:tasks.workunit.client.1.smithi027.stdout:8/596: rename d1/d3/d1c/d32/d41/d52/d61/d8f/lb4 to d1/d3/lcf 0 2024-04-22T15:50:04.221 INFO:tasks.workunit.client.1.smithi027.stdout:3/721: chown c1a 11445040 1 2024-04-22T15:50:04.223 INFO:tasks.workunit.client.1.smithi027.stdout:6/785: creat d5/dc/dd/f110 x:0 0 0 2024-04-22T15:50:04.224 INFO:tasks.workunit.client.1.smithi027.stdout:6/786: dread - d5/d30/d54/fd2 zero size 2024-04-22T15:50:04.226 INFO:tasks.workunit.client.1.smithi027.stdout:2/662: mknod dd/df/d70/d4b/d52/d9a/da4/dad/ccd 0 2024-04-22T15:50:04.233 INFO:tasks.workunit.client.1.smithi027.stdout:8/597: creat d1/d3/d1c/d32/d41/d52/fd0 x:0 0 0 2024-04-22T15:50:04.237 INFO:tasks.workunit.client.1.smithi027.stdout:4/647: dwrite d5/de/dc5/fcb [0,4194304] 0 2024-04-22T15:50:04.237 INFO:tasks.workunit.client.1.smithi027.stdout:8/598: truncate d1/d3/d1c/d43/d5f/d7d/fa3 349496 0 2024-04-22T15:50:04.242 INFO:tasks.workunit.client.1.smithi027.stdout:6/787: chown d5/dc/dd/dd0/f35 7533 1 2024-04-22T15:50:04.249 INFO:tasks.workunit.client.1.smithi027.stdout:3/722: mkdir d28/d4b/d52/db2/d95/d9c/de9 0 2024-04-22T15:50:04.251 INFO:tasks.workunit.client.1.smithi027.stdout:0/715: link d0/d28/d63/d67/l6d d0/da3/db6/le6 0 2024-04-22T15:50:04.279 INFO:tasks.workunit.client.1.smithi027.stdout:9/634: getdents d0 0 2024-04-22T15:50:04.279 INFO:tasks.workunit.client.1.smithi027.stdout:6/788: link d5/dc/dd/dd0/f27 d5/dc/d17/d10e/f111 0 2024-04-22T15:50:04.280 INFO:tasks.workunit.client.1.smithi027.stdout:2/663: write dd/df/d1b/f7a [842476,69684] 0 2024-04-22T15:50:04.285 INFO:tasks.workunit.client.1.smithi027.stdout:6/789: rmdir d5/dc/d20 39 2024-04-22T15:50:04.285 INFO:tasks.workunit.client.1.smithi027.stdout:9/635: dread - d0/dc/d12/d30/d5b/d91/da9/fc8 zero size 2024-04-22T15:50:04.286 INFO:tasks.workunit.client.1.smithi027.stdout:0/716: write d0/d34/d59/f5c [75228,63442] 0 2024-04-22T15:50:04.288 INFO:tasks.workunit.client.1.smithi027.stdout:3/723: dwrite d28/d69/f85 [4194304,4194304] 0 2024-04-22T15:50:04.290 INFO:tasks.workunit.client.1.smithi027.stdout:9/636: fdatasync d0/dc/de/fc5 0 2024-04-22T15:50:04.294 INFO:tasks.workunit.client.1.smithi027.stdout:7/644: dwrite d0/d11/d1c/d2c/fc4 [0,4194304] 0 2024-04-22T15:50:04.297 INFO:tasks.workunit.client.1.smithi027.stdout:8/599: dwrite d1/d3/d1c/d32/f3d [0,4194304] 0 2024-04-22T15:50:04.300 INFO:tasks.workunit.client.1.smithi027.stdout:6/790: rename d5/db/dcd to d5/d30/dbe/d112 0 2024-04-22T15:50:04.300 INFO:tasks.workunit.client.1.smithi027.stdout:2/664: truncate dd/df/d70/d4b/d52/f8f 226502 0 2024-04-22T15:50:04.300 INFO:tasks.workunit.client.1.smithi027.stdout:6/791: chown d5/f89 30805571 1 2024-04-22T15:50:04.306 INFO:tasks.workunit.client.1.smithi027.stdout:0/717: unlink d0/d34/d4e/c54 0 2024-04-22T15:50:04.307 INFO:tasks.workunit.client.1.smithi027.stdout:8/600: truncate d1/d3/d1c/d32/d41/f86 817224 0 2024-04-22T15:50:04.314 INFO:tasks.workunit.client.1.smithi027.stdout:0/718: stat d0/d2a/d6a/d8e/ccc 0 2024-04-22T15:50:04.316 INFO:tasks.workunit.client.1.smithi027.stdout:8/601: unlink d1/l9a 0 2024-04-22T15:50:04.318 INFO:tasks.workunit.client.1.smithi027.stdout:0/719: dread d0/d28/f1c [0,4194304] 0 2024-04-22T15:50:04.321 INFO:tasks.workunit.client.1.smithi027.stdout:0/720: dread - d0/d28/d5f/fd0 zero size 2024-04-22T15:50:04.322 INFO:tasks.workunit.client.1.smithi027.stdout:9/637: write d0/dc/d12/d30/f39 [3281197,23654] 0 2024-04-22T15:50:04.324 INFO:tasks.workunit.client.1.smithi027.stdout:8/602: creat d1/fd1 x:0 0 0 2024-04-22T15:50:04.326 INFO:tasks.workunit.client.1.smithi027.stdout:7/645: write d0/d4c/d8b/d72/db2/f69 [1805375,105080] 0 2024-04-22T15:50:04.328 INFO:tasks.workunit.client.1.smithi027.stdout:9/638: read d0/dc/d12/d58/d6b/f62 [3531672,91471] 0 2024-04-22T15:50:04.328 INFO:tasks.workunit.client.1.smithi027.stdout:0/721: mknod d0/da3/ddf/ce7 0 2024-04-22T15:50:04.328 INFO:tasks.workunit.client.1.smithi027.stdout:3/724: dwrite d28/f45 [4194304,4194304] 0 2024-04-22T15:50:04.332 INFO:tasks.workunit.client.1.smithi027.stdout:0/722: write d0/d34/d59/f5c [182237,27334] 0 2024-04-22T15:50:04.333 INFO:tasks.workunit.client.1.smithi027.stdout:3/725: write d28/d69/f86 [185596,7950] 0 2024-04-22T15:50:04.334 INFO:tasks.workunit.client.1.smithi027.stdout:2/665: dread dd/f19 [4194304,4194304] 0 2024-04-22T15:50:04.334 INFO:tasks.workunit.client.1.smithi027.stdout:0/723: chown d0/d2a/fa7 11416646 1 2024-04-22T15:50:04.335 INFO:tasks.workunit.client.1.smithi027.stdout:9/639: chown d0/d8/c50 127 1 2024-04-22T15:50:04.336 INFO:tasks.workunit.client.1.smithi027.stdout:2/666: chown dd/df/d70/d4b/d4e 3 1 2024-04-22T15:50:04.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:03 smithi027 ceph-mon[34922]: from='client.14706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:04.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:03 smithi027 ceph-mon[34922]: from='client.14710 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:04.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:03 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/2043994193' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:50:04.341 INFO:tasks.workunit.client.1.smithi027.stdout:8/603: dread d1/d1e/f2f [0,4194304] 0 2024-04-22T15:50:04.344 INFO:tasks.workunit.client.1.smithi027.stdout:0/724: creat d0/d34/d4e/d85/fe8 x:0 0 0 2024-04-22T15:50:04.345 INFO:tasks.workunit.client.1.smithi027.stdout:9/640: dread - d0/d64/f87 zero size 2024-04-22T15:50:04.347 INFO:tasks.workunit.client.1.smithi027.stdout:3/726: rmdir d28/d5d/d62 39 2024-04-22T15:50:04.349 INFO:tasks.workunit.client.1.smithi027.stdout:0/725: unlink d0/d28/d63/f7d 0 2024-04-22T15:50:04.350 INFO:tasks.workunit.client.1.smithi027.stdout:9/641: dread - d0/dc/d12/d30/d5b/d91/f9f zero size 2024-04-22T15:50:04.350 INFO:tasks.workunit.client.1.smithi027.stdout:0/726: chown d0/d2a/d6a/fcb 1 1 2024-04-22T15:50:04.352 INFO:tasks.workunit.client.1.smithi027.stdout:3/727: chown d28/d5d/c73 9 1 2024-04-22T15:50:04.353 INFO:tasks.workunit.client.1.smithi027.stdout:9/642: symlink d0/d8b/ld5 0 2024-04-22T15:50:04.355 INFO:tasks.workunit.client.1.smithi027.stdout:2/667: getdents dd/df/d70 0 2024-04-22T15:50:04.355 INFO:tasks.workunit.client.1.smithi027.stdout:9/643: chown d0/dc/d12/d58/d6b/d6e/f92 111 1 2024-04-22T15:50:04.359 INFO:tasks.workunit.client.1.smithi027.stdout:7/646: rename d0/d11/d1c/d2c/l97 to d0/d11/d1c/d31/ld5 0 2024-04-22T15:50:04.361 INFO:tasks.workunit.client.1.smithi027.stdout:0/727: creat d0/da3/db6/fe9 x:0 0 0 2024-04-22T15:50:04.362 INFO:tasks.workunit.client.1.smithi027.stdout:9/644: stat d0/d8/l4e 0 2024-04-22T15:50:04.366 INFO:tasks.workunit.client.1.smithi027.stdout:3/728: dread d28/d4b/f59 [0,4194304] 0 2024-04-22T15:50:04.367 INFO:tasks.workunit.client.1.smithi027.stdout:7/647: truncate d0/d4c/d4e/f57 1951245 0 2024-04-22T15:50:04.369 INFO:tasks.workunit.client.1.smithi027.stdout:0/728: fdatasync d0/d34/d4e/f68 0 2024-04-22T15:50:04.374 INFO:tasks.workunit.client.1.smithi027.stdout:3/729: rename f15 to d28/d4b/d52/db2/d95/fea 0 2024-04-22T15:50:04.378 INFO:tasks.workunit.client.1.smithi027.stdout:7/648: unlink d0/d49/d9f/l8c 0 2024-04-22T15:50:04.378 INFO:tasks.workunit.client.1.smithi027.stdout:7/649: stat d0/d11/c18 0 2024-04-22T15:50:04.379 INFO:tasks.workunit.client.1.smithi027.stdout:2/668: dread dd/df/d70/d4b/d4e/f56 [0,4194304] 0 2024-04-22T15:50:04.380 INFO:tasks.workunit.client.1.smithi027.stdout:0/729: creat d0/da3/dae/fea x:0 0 0 2024-04-22T15:50:04.382 INFO:tasks.workunit.client.1.smithi027.stdout:9/645: getdents d0/dc/d12/d30/d5b/d91/daf 0 2024-04-22T15:50:04.384 INFO:tasks.workunit.client.1.smithi027.stdout:9/646: write d0/dc/f25 [2946388,79543] 0 2024-04-22T15:50:04.385 INFO:tasks.workunit.client.1.smithi027.stdout:7/650: dwrite d0/d49/f63 [0,4194304] 0 2024-04-22T15:50:04.391 INFO:tasks.workunit.client.1.smithi027.stdout:3/730: write d28/d4b/d52/f65 [4759640,102753] 0 2024-04-22T15:50:04.392 INFO:tasks.workunit.client.1.smithi027.stdout:0/730: stat d0/d28/d37/la4 0 2024-04-22T15:50:04.394 INFO:tasks.workunit.client.1.smithi027.stdout:3/731: write d28/d4b/d52/d9a/fd1 [720677,35117] 0 2024-04-22T15:50:04.398 INFO:tasks.workunit.client.1.smithi027.stdout:9/647: chown d0/dc/d12/d58/c7e 0 1 2024-04-22T15:50:04.400 INFO:tasks.workunit.client.1.smithi027.stdout:2/669: link dd/df/d70/d4b/l9e dd/df/d70/d4b/d4e/lce 0 2024-04-22T15:50:04.403 INFO:tasks.workunit.client.1.smithi027.stdout:9/648: mknod d0/dc/de/db5/dbf/cd6 0 2024-04-22T15:50:04.404 INFO:tasks.workunit.client.1.smithi027.stdout:0/731: dwrite d0/da3/dae/fea [0,4194304] 0 2024-04-22T15:50:04.405 INFO:tasks.workunit.client.1.smithi027.stdout:3/732: mkdir d28/d4b/d52/db2/deb 0 2024-04-22T15:50:04.405 INFO:tasks.workunit.client.1.smithi027.stdout:0/732: readlink d0/d28/d5f/l6e 0 2024-04-22T15:50:04.405 INFO:tasks.workunit.client.1.smithi027.stdout:9/649: chown d0/dc/d12/c3b 7190 1 2024-04-22T15:50:04.409 INFO:tasks.workunit.client.1.smithi027.stdout:7/651: dread d0/d11/d3a/f45 [0,4194304] 0 2024-04-22T15:50:04.411 INFO:tasks.workunit.client.1.smithi027.stdout:3/733: fsync d28/f63 0 2024-04-22T15:50:04.412 INFO:tasks.workunit.client.1.smithi027.stdout:3/734: chown d28/d4b/d52/d94/fc2 2241 1 2024-04-22T15:50:04.413 INFO:tasks.workunit.client.1.smithi027.stdout:0/733: fdatasync d0/d2a/f3c 0 2024-04-22T15:50:04.416 INFO:tasks.workunit.client.1.smithi027.stdout:9/650: creat d0/dc/d12/d58/d6b/d6e/fd7 x:0 0 0 2024-04-22T15:50:04.418 INFO:tasks.workunit.client.1.smithi027.stdout:7/652: chown d0/d4c/d8b/d92/da3/da8/f81 177530979 1 2024-04-22T15:50:04.424 INFO:tasks.workunit.client.1.smithi027.stdout:7/653: unlink d0/d4c/d8b/d72/db2/c90 0 2024-04-22T15:50:04.425 INFO:tasks.workunit.client.1.smithi027.stdout:3/735: mkdir d28/d5d/d62/dcf/dec 0 2024-04-22T15:50:04.426 INFO:tasks.workunit.client.1.smithi027.stdout:2/670: write dd/df/d1b/f5f [2305153,108649] 0 2024-04-22T15:50:04.426 INFO:tasks.workunit.client.1.smithi027.stdout:3/736: read - d28/d4b/d52/db2/fa6 zero size 2024-04-22T15:50:04.431 INFO:tasks.workunit.client.1.smithi027.stdout:9/651: dread d0/dc/d12/d30/d5b/d91/da9/fca [0,4194304] 0 2024-04-22T15:50:04.433 INFO:tasks.workunit.client.1.smithi027.stdout:9/652: write d0/dc/d12/d30/d99/fab [193719,8466] 0 2024-04-22T15:50:04.433 INFO:tasks.workunit.client.1.smithi027.stdout:7/654: link d0/d11/d1c/d31/d8a/fce d0/d11/d3a/d46/fd6 0 2024-04-22T15:50:04.434 INFO:tasks.workunit.client.1.smithi027.stdout:3/737: dwrite d28/f36 [0,4194304] 0 2024-04-22T15:50:04.437 INFO:tasks.workunit.client.1.smithi027.stdout:0/734: write d0/f3e [4780595,57570] 0 2024-04-22T15:50:04.438 INFO:tasks.workunit.client.1.smithi027.stdout:9/653: symlink d0/d8b/ld8 0 2024-04-22T15:50:04.440 INFO:tasks.workunit.client.1.smithi027.stdout:3/738: creat d28/d4b/d52/d94/dae/fed x:0 0 0 2024-04-22T15:50:04.441 INFO:tasks.workunit.client.1.smithi027.stdout:3/739: write d28/fb5 [913487,51097] 0 2024-04-22T15:50:04.443 INFO:tasks.workunit.client.1.smithi027.stdout:7/655: rename d0/d4c/d8b/d92/da3/lbd to d0/d49/d9f/ld7 0 2024-04-22T15:50:04.446 INFO:tasks.workunit.client.1.smithi027.stdout:0/735: creat d0/da3/db9/feb x:0 0 0 2024-04-22T15:50:04.449 INFO:tasks.workunit.client.1.smithi027.stdout:3/740: rename d28/d69/ddb to d28/d5d/db0/dce/dee 0 2024-04-22T15:50:04.451 INFO:tasks.workunit.client.1.smithi027.stdout:0/736: dread d0/d28/f5b [0,4194304] 0 2024-04-22T15:50:04.453 INFO:tasks.workunit.client.1.smithi027.stdout:7/656: dwrite d0/d11/d1c/d31/faa [0,4194304] 0 2024-04-22T15:50:04.454 INFO:tasks.workunit.client.1.smithi027.stdout:1/585: sync 2024-04-22T15:50:04.455 INFO:tasks.workunit.client.1.smithi027.stdout:0/737: dread d0/d28/f1c [0,4194304] 0 2024-04-22T15:50:04.456 INFO:tasks.workunit.client.1.smithi027.stdout:9/654: dread d0/dc/f25 [4194304,4194304] 0 2024-04-22T15:50:04.464 INFO:tasks.workunit.client.1.smithi027.stdout:7/657: dwrite d0/d49/d9f/f9b [0,4194304] 0 2024-04-22T15:50:04.465 INFO:tasks.workunit.client.1.smithi027.stdout:0/738: symlink d0/d2a/d58/lec 0 2024-04-22T15:50:04.466 INFO:tasks.workunit.client.1.smithi027.stdout:1/586: dread d2/d5f/d64/d81/d3d/d44/d9c/d9f/fb5 [0,4194304] 0 2024-04-22T15:50:04.468 INFO:tasks.workunit.client.1.smithi027.stdout:7/658: truncate d0/d11/d1c/fb1 1042280 0 2024-04-22T15:50:04.474 INFO:tasks.workunit.client.1.smithi027.stdout:0/739: mkdir d0/d2a/d58/d66/ded 0 2024-04-22T15:50:04.476 INFO:tasks.workunit.client.1.smithi027.stdout:9/655: unlink d0/d8/f18 0 2024-04-22T15:50:04.479 INFO:tasks.workunit.client.1.smithi027.stdout:7/659: creat d0/d4c/d8b/d92/da3/da8/fd8 x:0 0 0 2024-04-22T15:50:04.489 INFO:tasks.workunit.client.1.smithi027.stdout:9/656: creat d0/d8b/fd9 x:0 0 0 2024-04-22T15:50:04.489 INFO:tasks.workunit.client.1.smithi027.stdout:3/741: dwrite d28/d69/f77 [0,4194304] 0 2024-04-22T15:50:04.490 INFO:tasks.workunit.client.1.smithi027.stdout:0/740: dread d0/f1 [0,4194304] 0 2024-04-22T15:50:04.491 INFO:tasks.workunit.client.1.smithi027.stdout:9/657: dread - d0/fb6 zero size 2024-04-22T15:50:04.500 INFO:tasks.workunit.client.1.smithi027.stdout:0/741: mkdir d0/d2a/d58/dee 0 2024-04-22T15:50:04.517 INFO:tasks.workunit.client.1.smithi027.stdout:7/660: rename d0/d11/d1c/d31/d62/c8e to d0/d4c/d8b/d92/da3/cd9 0 2024-04-22T15:50:04.519 INFO:tasks.workunit.client.1.smithi027.stdout:9/658: dwrite d0/dc/d12/d58/d6b/d6e/f61 [4194304,4194304] 0 2024-04-22T15:50:04.520 INFO:tasks.workunit.client.1.smithi027.stdout:1/587: link d2/d5f/d64/l6e d2/d36/lc5 0 2024-04-22T15:50:04.522 INFO:tasks.workunit.client.1.smithi027.stdout:1/588: read d2/f68 [7562515,117098] 0 2024-04-22T15:50:04.527 INFO:tasks.workunit.client.1.smithi027.stdout:3/742: link d28/d4b/d52/db2/c9b d28/d4b/d52/cef 0 2024-04-22T15:50:04.528 INFO:tasks.workunit.client.1.smithi027.stdout:0/742: dwrite d0/d2a/d58/d66/f9b [0,4194304] 0 2024-04-22T15:50:04.529 INFO:tasks.workunit.client.1.smithi027.stdout:9/659: truncate d0/dc/d12/d58/d6b/d6e/fbd 620823 0 2024-04-22T15:50:04.529 INFO:tasks.workunit.client.1.smithi027.stdout:1/589: unlink d2/d30/f41 0 2024-04-22T15:50:04.531 INFO:tasks.workunit.client.1.smithi027.stdout:7/661: dwrite d0/d3f/f48 [0,4194304] 0 2024-04-22T15:50:04.532 INFO:tasks.workunit.client.1.smithi027.stdout:7/662: truncate d0/fa0 406670 0 2024-04-22T15:50:04.539 INFO:tasks.workunit.client.1.smithi027.stdout:7/663: creat d0/d11/d3a/d46/d94/fda x:0 0 0 2024-04-22T15:50:04.548 INFO:tasks.workunit.client.1.smithi027.stdout:9/660: dwrite d0/dc/de/f3a [0,4194304] 0 2024-04-22T15:50:04.551 INFO:tasks.workunit.client.1.smithi027.stdout:7/664: dwrite d0/d4c/f7a [0,4194304] 0 2024-04-22T15:50:04.558 INFO:tasks.workunit.client.1.smithi027.stdout:0/743: write d0/d2a/f61 [5032868,70001] 0 2024-04-22T15:50:04.561 INFO:tasks.workunit.client.1.smithi027.stdout:3/743: dread d28/d4b/d52/d94/dae/fcc [0,4194304] 0 2024-04-22T15:50:04.562 INFO:tasks.workunit.client.1.smithi027.stdout:3/744: stat d28/f56 0 2024-04-22T15:50:04.562 INFO:tasks.workunit.client.1.smithi027.stdout:7/665: symlink d0/ldb 0 2024-04-22T15:50:04.568 INFO:tasks.workunit.client.1.smithi027.stdout:7/666: chown d0/d49/f71 217017136 1 2024-04-22T15:50:04.569 INFO:tasks.workunit.client.1.smithi027.stdout:7/667: fsync d0/f37 0 2024-04-22T15:50:04.570 INFO:tasks.workunit.client.1.smithi027.stdout:0/744: write d0/d9d/db7/fa9 [339686,16414] 0 2024-04-22T15:50:04.571 INFO:tasks.workunit.client.1.smithi027.stdout:1/590: dwrite d2/d5f/d64/d81/f3a [0,4194304] 0 2024-04-22T15:50:04.574 INFO:tasks.workunit.client.1.smithi027.stdout:7/668: truncate d0/d4c/fd4 264465 0 2024-04-22T15:50:04.577 INFO:tasks.workunit.client.1.smithi027.stdout:9/661: getdents d0 0 2024-04-22T15:50:04.582 INFO:tasks.workunit.client.1.smithi027.stdout:9/662: truncate d0/dc/de/f41 2204863 0 2024-04-22T15:50:04.582 INFO:tasks.workunit.client.1.smithi027.stdout:1/591: dread d2/f68 [0,4194304] 0 2024-04-22T15:50:04.582 INFO:tasks.workunit.client.1.smithi027.stdout:3/745: dread d28/d5d/d62/f66 [0,4194304] 0 2024-04-22T15:50:04.589 INFO:tasks.workunit.client.1.smithi027.stdout:1/592: unlink d2/d5f/d64/d81/da7/f66 0 2024-04-22T15:50:04.591 INFO:tasks.workunit.client.1.smithi027.stdout:0/745: dread d0/f12 [0,4194304] 0 2024-04-22T15:50:04.592 INFO:tasks.workunit.client.1.smithi027.stdout:1/593: fsync d2/d5f/d64/d81/d3d/d44/d9c/d9f/fac 0 2024-04-22T15:50:04.592 INFO:tasks.workunit.client.1.smithi027.stdout:0/746: chown d0/da3/db9 112360464 1 2024-04-22T15:50:04.593 INFO:tasks.workunit.client.1.smithi027.stdout:7/669: rename d0/d4c/d8b/dc7 to d0/d11/d1c/ddc 0 2024-04-22T15:50:04.593 INFO:tasks.workunit.client.1.smithi027.stdout:0/747: stat d0/d2a/d38/f92 0 2024-04-22T15:50:04.596 INFO:tasks.workunit.client.1.smithi027.stdout:3/746: write f1c [619075,90972] 0 2024-04-22T15:50:04.598 INFO:tasks.workunit.client.1.smithi027.stdout:3/747: write d28/d5d/d62/fb1 [1860246,58436] 0 2024-04-22T15:50:04.599 INFO:tasks.workunit.client.1.smithi027.stdout:1/594: creat d2/d5f/fc6 x:0 0 0 2024-04-22T15:50:04.601 INFO:tasks.workunit.client.1.smithi027.stdout:9/663: getdents d0/d8 0 2024-04-22T15:50:04.602 INFO:tasks.workunit.client.1.smithi027.stdout:7/670: symlink d0/d11/d3a/d46/d94/d5a/ldd 0 2024-04-22T15:50:04.603 INFO:tasks.workunit.client.1.smithi027.stdout:9/664: read d0/d8/f11 [2881111,112497] 0 2024-04-22T15:50:04.604 INFO:tasks.workunit.client.1.smithi027.stdout:0/748: rename d0/d34/d59/fc1 to d0/d28/fef 0 2024-04-22T15:50:04.604 INFO:tasks.workunit.client.1.smithi027.stdout:9/665: chown d0/d8/c1b 570 1 2024-04-22T15:50:04.605 INFO:tasks.workunit.client.1.smithi027.stdout:0/749: write d0/da3/db6/fe9 [855767,102007] 0 2024-04-22T15:50:04.606 INFO:tasks.workunit.client.1.smithi027.stdout:3/748: mknod d28/d4b/d52/db2/d95/d9c/cf0 0 2024-04-22T15:50:04.607 INFO:tasks.workunit.client.1.smithi027.stdout:9/666: write d0/dc/d12/d30/d99/db4/fcc [196617,111069] 0 2024-04-22T15:50:04.621 INFO:tasks.workunit.client.1.smithi027.stdout:5/698: sync 2024-04-22T15:50:04.621 INFO:tasks.workunit.client.1.smithi027.stdout:4/648: sync 2024-04-22T15:50:04.622 INFO:tasks.workunit.client.1.smithi027.stdout:4/649: fsync d5/d45/d81/f98 0 2024-04-22T15:50:04.622 INFO:tasks.workunit.client.1.smithi027.stdout:9/667: fdatasync d0/dc/d12/d30/f67 0 2024-04-22T15:50:04.624 INFO:tasks.workunit.client.1.smithi027.stdout:7/671: symlink d0/d11/d1c/d31/d62/lde 0 2024-04-22T15:50:04.640 INFO:tasks.workunit.client.1.smithi027.stdout:0/750: unlink d0/l23 0 2024-04-22T15:50:04.640 INFO:tasks.workunit.client.1.smithi027.stdout:1/595: dwrite d2/d5f/d64/d81/d3d/d44/f6d [0,4194304] 0 2024-04-22T15:50:04.641 INFO:tasks.workunit.client.1.smithi027.stdout:5/699: unlink d0/df/de/l4d 0 2024-04-22T15:50:04.642 INFO:tasks.workunit.client.1.smithi027.stdout:5/700: chown d0/fc9 72 1 2024-04-22T15:50:04.644 INFO:tasks.workunit.client.1.smithi027.stdout:4/650: mknod d5/de/df2/de2/cf6 0 2024-04-22T15:50:04.645 INFO:tasks.workunit.client.1.smithi027.stdout:9/668: creat d0/dc/d12/d30/d99/da7/fda x:0 0 0 2024-04-22T15:50:04.647 INFO:tasks.workunit.client.1.smithi027.stdout:5/701: dread - d0/df/d3d/d81/fbc zero size 2024-04-22T15:50:04.653 INFO:tasks.workunit.client.1.smithi027.stdout:3/749: dwrite d28/d5d/f80 [0,4194304] 0 2024-04-22T15:50:04.657 INFO:tasks.workunit.client.1.smithi027.stdout:0/751: dwrite d0/f3e [4194304,4194304] 0 2024-04-22T15:50:04.657 INFO:tasks.workunit.client.1.smithi027.stdout:3/750: read - d28/d5d/d79/fcd zero size 2024-04-22T15:50:04.662 INFO:tasks.workunit.client.1.smithi027.stdout:3/751: write d28/d69/f85 [4985726,49298] 0 2024-04-22T15:50:04.668 INFO:tasks.workunit.client.1.smithi027.stdout:1/596: dread d2/d5f/d64/d81/da7/f9e [0,4194304] 0 2024-04-22T15:50:04.707 INFO:tasks.workunit.client.1.smithi027.stdout:7/672: mknod d0/d4c/d8b/d92/da3/cdf 0 2024-04-22T15:50:04.708 INFO:tasks.workunit.client.1.smithi027.stdout:7/673: fdatasync d0/d4c/d8b/d92/da3/da8/f23 0 2024-04-22T15:50:04.711 INFO:tasks.workunit.client.1.smithi027.stdout:5/702: rename d0/df/de/d24/d1c to d0/d1a/d22/d26/d5d/de3 0 2024-04-22T15:50:04.716 INFO:tasks.workunit.client.1.smithi027.stdout:0/752: truncate d0/d2a/d38/f65 8225917 0 2024-04-22T15:50:04.725 INFO:tasks.workunit.client.1.smithi027.stdout:4/651: symlink d5/d11/d1d/lf7 0 2024-04-22T15:50:04.731 INFO:tasks.workunit.client.1.smithi027.stdout:5/703: creat d0/d97/dbf/fe4 x:0 0 0 2024-04-22T15:50:04.733 INFO:tasks.workunit.client.1.smithi027.stdout:5/704: dread d0/d1a/d22/d26/d32/d79/f87 [0,4194304] 0 2024-04-22T15:50:04.734 INFO:tasks.workunit.client.1.smithi027.stdout:3/752: fdatasync d28/d4b/d52/db2/d95/fea 0 2024-04-22T15:50:04.737 INFO:tasks.workunit.client.1.smithi027.stdout:1/597: symlink d2/d5f/d64/d81/lc7 0 2024-04-22T15:50:04.741 INFO:tasks.workunit.client.1.smithi027.stdout:4/652: rename d5/de/c4f to d5/d11/dc1/cf8 0 2024-04-22T15:50:04.742 INFO:tasks.workunit.client.1.smithi027.stdout:4/653: chown d5/de/d30/d5d/c66 12185 1 2024-04-22T15:50:04.745 INFO:tasks.workunit.client.1.smithi027.stdout:7/674: write d0/d11/d3a/d46/f82 [796962,100372] 0 2024-04-22T15:50:04.756 INFO:tasks.workunit.client.1.smithi027.stdout:1/598: rename d2/d36/f55 to d2/d1d/d56/dc4/fc8 0 2024-04-22T15:50:04.756 INFO:tasks.workunit.client.1.smithi027.stdout:1/599: fsync d2/d1d/f96 0 2024-04-22T15:50:04.765 INFO:tasks.workunit.client.1.smithi027.stdout:0/753: chown d0/d2a/d58/d66/lc2 6458 1 2024-04-22T15:50:04.770 INFO:tasks.workunit.client.1.smithi027.stdout:5/705: fsync d0/df/de/d24/f35 0 2024-04-22T15:50:04.772 INFO:tasks.workunit.client.1.smithi027.stdout:7/675: mknod d0/d4c/d8b/d92/da3/ce0 0 2024-04-22T15:50:04.772 INFO:tasks.workunit.client.1.smithi027.stdout:4/654: dwrite d5/d11/d1d/d24/f9c [0,4194304] 0 2024-04-22T15:50:04.773 INFO:tasks.workunit.client.1.smithi027.stdout:0/754: creat d0/d9d/db7/ff0 x:0 0 0 2024-04-22T15:50:04.782 INFO:tasks.workunit.client.1.smithi027.stdout:3/753: getdents d28/d5d/d62/dcf 0 2024-04-22T15:50:04.787 INFO:tasks.workunit.client.1.smithi027.stdout:5/706: dread - d0/d1a/d22/d26/d5d/d99/fb9 zero size 2024-04-22T15:50:04.788 INFO:tasks.workunit.client.1.smithi027.stdout:5/707: dread - d0/d1a/d22/d72/dac/fc0 zero size 2024-04-22T15:50:04.790 INFO:tasks.workunit.client.1.smithi027.stdout:4/655: dread d5/d11/d1d/d24/f51 [0,4194304] 0 2024-04-22T15:50:04.791 INFO:tasks.workunit.client.1.smithi027.stdout:1/600: creat d2/d5f/d64/d81/fc9 x:0 0 0 2024-04-22T15:50:04.797 INFO:tasks.workunit.client.1.smithi027.stdout:7/676: unlink d0/d4c/d4e/f57 0 2024-04-22T15:50:04.798 INFO:tasks.workunit.client.1.smithi027.stdout:7/677: fsync d0/d11/d3a/fa7 0 2024-04-22T15:50:04.801 INFO:tasks.workunit.client.1.smithi027.stdout:3/754: truncate d28/d4b/d52/f61 1966044 0 2024-04-22T15:50:04.802 INFO:tasks.workunit.client.1.smithi027.stdout:5/708: getdents d0/df/de/d40/d89/dd0 0 2024-04-22T15:50:04.805 INFO:tasks.workunit.client.1.smithi027.stdout:0/755: dread d0/d34/f9c [0,4194304] 0 2024-04-22T15:50:04.806 INFO:tasks.workunit.client.1.smithi027.stdout:1/601: getdents d2/db9 0 2024-04-22T15:50:04.816 INFO:tasks.workunit.client.1.smithi027.stdout:1/602: dwrite d2/d5f/d64/d81/d69/d7d/fa8 [0,4194304] 0 2024-04-22T15:50:04.817 INFO:tasks.workunit.client.1.smithi027.stdout:4/656: dread d5/d11/f7a [0,4194304] 0 2024-04-22T15:50:04.818 INFO:tasks.workunit.client.1.smithi027.stdout:4/657: chown d5/d11/da6/dd0 396563563 1 2024-04-22T15:50:04.823 INFO:tasks.workunit.client.1.smithi027.stdout:4/658: read - d5/d45/d47/fac zero size 2024-04-22T15:50:04.824 INFO:tasks.workunit.client.1.smithi027.stdout:1/603: dwrite d2/d1d/f91 [0,4194304] 0 2024-04-22T15:50:04.824 INFO:tasks.workunit.client.1.smithi027.stdout:4/659: dread - d5/de/df2/fec zero size 2024-04-22T15:50:04.845 INFO:tasks.workunit.client.1.smithi027.stdout:6/792: sync 2024-04-22T15:50:04.850 INFO:tasks.workunit.client.1.smithi027.stdout:6/793: mknod d5/db/d2e/d59/c113 0 2024-04-22T15:50:04.851 INFO:tasks.workunit.client.1.smithi027.stdout:6/794: dread - d5/d30/d40/fce zero size 2024-04-22T15:50:04.853 INFO:tasks.workunit.client.1.smithi027.stdout:4/660: getdents d5/d45/d81 0 2024-04-22T15:50:04.859 INFO:tasks.workunit.client.1.smithi027.stdout:4/661: mknod d5/d11/d1d/d54/cf9 0 2024-04-22T15:50:04.860 INFO:tasks.workunit.client.1.smithi027.stdout:4/662: readlink d5/de/d30/d5d/lf4 0 2024-04-22T15:50:04.861 INFO:tasks.workunit.client.1.smithi027.stdout:4/663: fdatasync d5/de/d2a/d58/fc0 0 2024-04-22T15:50:04.867 INFO:tasks.workunit.client.1.smithi027.stdout:6/795: creat d5/dc/f114 x:0 0 0 2024-04-22T15:50:04.870 INFO:tasks.workunit.client.1.smithi027.stdout:7/678: truncate d0/d11/d1c/d31/d56/f58 116552 0 2024-04-22T15:50:04.873 INFO:tasks.workunit.client.1.smithi027.stdout:7/679: truncate d0/f3e 1418179 0 2024-04-22T15:50:04.874 INFO:tasks.workunit.client.1.smithi027.stdout:3/755: dwrite d28/d4b/d52/d94/fc2 [0,4194304] 0 2024-04-22T15:50:04.884 INFO:tasks.workunit.client.1.smithi027.stdout:0/756: truncate d0/f3e 6584644 0 2024-04-22T15:50:04.888 INFO:tasks.workunit.client.1.smithi027.stdout:4/664: creat d5/d11/d1d/ffa x:0 0 0 2024-04-22T15:50:04.890 INFO:tasks.workunit.client.1.smithi027.stdout:3/756: symlink d28/d4b/d52/db2/d95/lf1 0 2024-04-22T15:50:04.893 INFO:tasks.workunit.client.1.smithi027.stdout:0/757: dwrite d0/d2a/d5e/d8b/f9f [0,4194304] 0 2024-04-22T15:50:04.894 INFO:tasks.workunit.client.1.smithi027.stdout:0/758: read d0/d28/f1f [825545,73883] 0 2024-04-22T15:50:04.894 INFO:tasks.workunit.client.1.smithi027.stdout:6/796: dread d5/dc/dd/f16 [0,4194304] 0 2024-04-22T15:50:04.897 INFO:tasks.workunit.client.1.smithi027.stdout:4/665: rmdir d5/de/dc5 39 2024-04-22T15:50:04.897 INFO:tasks.workunit.client.1.smithi027.stdout:6/797: write d5/db/d2e/f62 [5694650,37615] 0 2024-04-22T15:50:04.901 INFO:tasks.workunit.client.1.smithi027.stdout:3/757: unlink d28/d4b/d52/d94/dae/fed 0 2024-04-22T15:50:04.904 INFO:tasks.workunit.client.1.smithi027.stdout:7/680: creat d0/d4c/d8b/d92/da3/fe1 x:0 0 0 2024-04-22T15:50:04.904 INFO:tasks.workunit.client.1.smithi027.stdout:1/604: write d2/d1d/f32 [284844,129371] 0 2024-04-22T15:50:04.905 INFO:tasks.workunit.client.1.smithi027.stdout:0/759: dread - d0/d34/d4e/f68 zero size 2024-04-22T15:50:04.906 INFO:tasks.workunit.client.1.smithi027.stdout:1/605: write d2/d36/f7c [2498770,87929] 0 2024-04-22T15:50:04.907 INFO:tasks.workunit.client.1.smithi027.stdout:7/681: write d0/d4c/d8b/d72/db2/f69 [337831,88722] 0 2024-04-22T15:50:04.909 INFO:tasks.workunit.client.1.smithi027.stdout:4/666: mknod d5/d45/d5a/dcd/cfb 0 2024-04-22T15:50:04.910 INFO:tasks.workunit.client.1.smithi027.stdout:4/667: chown d5/d45/f7d 448185899 1 2024-04-22T15:50:04.911 INFO:tasks.workunit.client.1.smithi027.stdout:4/668: fsync d5/d45/d81/fe4 0 2024-04-22T15:50:04.912 INFO:tasks.workunit.client.1.smithi027.stdout:4/669: chown d5/d11/d1d/d54/cb4 1092879 1 2024-04-22T15:50:04.914 INFO:tasks.workunit.client.1.smithi027.stdout:4/670: chown d5/de/da9/fe0 298003 1 2024-04-22T15:50:04.915 INFO:tasks.workunit.client.1.smithi027.stdout:4/671: write d5/d45/f7d [868369,49112] 0 2024-04-22T15:50:04.921 INFO:tasks.workunit.client.1.smithi027.stdout:7/682: unlink d0/f30 0 2024-04-22T15:50:04.922 INFO:tasks.workunit.client.1.smithi027.stdout:3/758: unlink d28/d69/cb3 0 2024-04-22T15:50:04.932 INFO:tasks.workunit.client.1.smithi027.stdout:3/759: write d28/f48 [699924,119435] 0 2024-04-22T15:50:04.934 INFO:tasks.workunit.client.1.smithi027.stdout:4/672: mkdir d5/d11/d1d/d24/dfc 0 2024-04-22T15:50:04.938 INFO:tasks.workunit.client.1.smithi027.stdout:3/760: fdatasync d28/f3b 0 2024-04-22T15:50:04.939 INFO:tasks.workunit.client.1.smithi027.stdout:3/761: chown d28/d5d/fcb 0 1 2024-04-22T15:50:04.943 INFO:tasks.workunit.client.1.smithi027.stdout:1/606: getdents d2 0 2024-04-22T15:50:04.944 INFO:tasks.workunit.client.1.smithi027.stdout:1/607: write d2/d5f/d64/d81/d7e/fb4 [871163,93150] 0 2024-04-22T15:50:04.947 INFO:tasks.workunit.client.1.smithi027.stdout:4/673: creat d5/d11/d1d/d7b/de9/ffd x:0 0 0 2024-04-22T15:50:04.949 INFO:tasks.workunit.client.1.smithi027.stdout:7/683: getdents d0/d11/d3a/d46/d94 0 2024-04-22T15:50:04.955 INFO:tasks.workunit.client.1.smithi027.stdout:1/608: dwrite d2/d36/daa/fbf [0,4194304] 0 2024-04-22T15:50:04.956 INFO:tasks.workunit.client.1.smithi027.stdout:1/609: chown d2/l2b 430 1 2024-04-22T15:50:04.957 INFO:tasks.workunit.client.1.smithi027.stdout:4/674: creat d5/de/dc5/ffe x:0 0 0 2024-04-22T15:50:04.957 INFO:tasks.workunit.client.1.smithi027.stdout:0/760: write d0/d2a/d6a/d6c/f93 [781258,15701] 0 2024-04-22T15:50:04.960 INFO:tasks.workunit.client.1.smithi027.stdout:0/761: dread - d0/d9d/db7/ff0 zero size 2024-04-22T15:50:04.961 INFO:tasks.workunit.client.1.smithi027.stdout:0/762: readlink d0/d2a/d91/lc7 0 2024-04-22T15:50:04.962 INFO:tasks.workunit.client.1.smithi027.stdout:7/684: mknod d0/d11/d1c/d2c/d96/ce2 0 2024-04-22T15:50:04.966 INFO:tasks.workunit.client.1.smithi027.stdout:3/762: getdents d28/d5d/db0 0 2024-04-22T15:50:04.970 INFO:tasks.workunit.client.1.smithi027.stdout:0/763: write d0/d2a/d6a/fcb [282,112468] 0 2024-04-22T15:50:04.977 INFO:tasks.workunit.client.1.smithi027.stdout:0/764: creat d0/d2a/d6a/d6c/ff1 x:0 0 0 2024-04-22T15:50:04.980 INFO:tasks.workunit.client.1.smithi027.stdout:7/685: link d0/d4c/fc9 d0/d11/d3a/d46/fe3 0 2024-04-22T15:50:04.984 INFO:tasks.workunit.client.1.smithi027.stdout:3/763: rename d28/d4b/d52/db2/c9b to d28/d4b/d52/db2/d95/cf2 0 2024-04-22T15:50:04.984 INFO:tasks.workunit.client.1.smithi027.stdout:3/764: readlink d28/l30 0 2024-04-22T15:50:04.985 INFO:tasks.workunit.client.1.smithi027.stdout:1/610: link d2/d30/l46 d2/d5f/d64/d81/d3d/d44/d9c/d9f/lca 0 2024-04-22T15:50:04.988 INFO:tasks.workunit.client.1.smithi027.stdout:7/686: creat d0/d11/d1c/d2c/d96/db3/fe4 x:0 0 0 2024-04-22T15:50:04.990 INFO:tasks.workunit.client.1.smithi027.stdout:4/675: getdents d5/d7 0 2024-04-22T15:50:04.990 INFO:tasks.workunit.client.1.smithi027.stdout:4/676: stat d5/d11/d6c/da3/ld7 0 2024-04-22T15:50:04.993 INFO:tasks.workunit.client.1.smithi027.stdout:0/765: read d0/f3e [498642,56585] 0 2024-04-22T15:50:04.997 INFO:tasks.workunit.client.1.smithi027.stdout:7/687: mkdir d0/d4c/de5 0 2024-04-22T15:50:04.999 INFO:tasks.workunit.client.1.smithi027.stdout:3/765: mkdir d28/d4b/d52/db2/deb/df3 0 2024-04-22T15:50:05.003 INFO:tasks.workunit.client.1.smithi027.stdout:4/677: symlink d5/d11/d1d/d24/lff 0 2024-04-22T15:50:05.004 INFO:tasks.workunit.client.1.smithi027.stdout:4/678: write d5/d11/f80 [2831572,119784] 0 2024-04-22T15:50:05.006 INFO:tasks.workunit.client.1.smithi027.stdout:1/611: rename d2/d1d/l2a to d2/d5f/d64/d81/d69/lcb 0 2024-04-22T15:50:05.007 INFO:tasks.workunit.client.1.smithi027.stdout:0/766: symlink d0/lf2 0 2024-04-22T15:50:05.008 INFO:tasks.workunit.client.1.smithi027.stdout:1/612: write d2/d5f/d64/d81/d3d/d44/f86 [4519543,92471] 0 2024-04-22T15:50:05.009 INFO:tasks.workunit.client.1.smithi027.stdout:3/766: symlink d28/d5d/d79/lf4 0 2024-04-22T15:50:05.011 INFO:tasks.workunit.client.1.smithi027.stdout:7/688: write d0/d4c/fd1 [3110293,25033] 0 2024-04-22T15:50:05.016 INFO:tasks.workunit.client.1.smithi027.stdout:0/767: dwrite d0/d2a/fc6 [0,4194304] 0 2024-04-22T15:50:05.018 INFO:tasks.workunit.client.1.smithi027.stdout:0/768: stat d0/d2a/de0/ce3 0 2024-04-22T15:50:05.020 INFO:tasks.workunit.client.1.smithi027.stdout:3/767: dwrite d28/d4b/f7d [0,4194304] 0 2024-04-22T15:50:05.028 INFO:tasks.workunit.client.1.smithi027.stdout:4/679: rename d5/d11/d1d/c4d to d5/de/d30/c100 0 2024-04-22T15:50:05.032 INFO:tasks.workunit.client.1.smithi027.stdout:1/613: creat d2/d5f/d64/d81/d3d/fcc x:0 0 0 2024-04-22T15:50:05.033 INFO:tasks.workunit.client.1.smithi027.stdout:4/680: fdatasync d5/d42/f9e 0 2024-04-22T15:50:05.033 INFO:tasks.workunit.client.1.smithi027.stdout:1/614: chown d2/l25 2563 1 2024-04-22T15:50:05.035 INFO:tasks.workunit.client.1.smithi027.stdout:0/769: getdents d0/d28/dd2 0 2024-04-22T15:50:05.040 INFO:tasks.workunit.client.1.smithi027.stdout:1/615: creat d2/d5f/d64/d81/d7e/fcd x:0 0 0 2024-04-22T15:50:05.048 INFO:tasks.workunit.client.1.smithi027.stdout:0/770: chown d0/d2a/d58/lec 276 1 2024-04-22T15:50:05.049 INFO:tasks.workunit.client.1.smithi027.stdout:7/689: link d0/d49/d9f/fb5 d0/d4c/d8b/d92/fe6 0 2024-04-22T15:50:05.053 INFO:tasks.workunit.client.1.smithi027.stdout:1/616: dread d2/d5f/d64/d81/d3d/d44/fb2 [0,4194304] 0 2024-04-22T15:50:05.054 INFO:tasks.workunit.client.1.smithi027.stdout:3/768: dwrite d28/d5d/db0/fba [0,4194304] 0 2024-04-22T15:50:05.060 INFO:tasks.workunit.client.1.smithi027.stdout:4/681: dwrite d5/d11/d1d/f5e [4194304,4194304] 0 2024-04-22T15:50:05.061 INFO:tasks.workunit.client.1.smithi027.stdout:1/617: creat d2/d1d/d56/dc4/fce x:0 0 0 2024-04-22T15:50:05.063 INFO:tasks.workunit.client.1.smithi027.stdout:3/769: symlink d28/d5d/db0/dce/dd4/lf5 0 2024-04-22T15:50:05.066 INFO:tasks.workunit.client.1.smithi027.stdout:0/771: link d0/d28/d37/l4c d0/da3/db9/lf3 0 2024-04-22T15:50:05.066 INFO:tasks.workunit.client.1.smithi027.stdout:4/682: mknod d5/d45/c101 0 2024-04-22T15:50:05.072 INFO:tasks.workunit.client.1.smithi027.stdout:3/770: unlink lb 0 2024-04-22T15:50:05.074 INFO:tasks.workunit.client.1.smithi027.stdout:3/771: mknod d28/d4b/d52/db2/d95/cf6 0 2024-04-22T15:50:05.084 INFO:tasks.workunit.client.1.smithi027.stdout:7/690: dread d0/d4c/d8b/f89 [0,4194304] 0 2024-04-22T15:50:05.087 INFO:tasks.workunit.client.1.smithi027.stdout:7/691: mkdir d0/d4c/d8b/d72/db2/de7 0 2024-04-22T15:50:05.088 INFO:tasks.workunit.client.1.smithi027.stdout:7/692: write d0/d4c/f7a [2749765,121818] 0 2024-04-22T15:50:05.092 INFO:tasks.workunit.client.1.smithi027.stdout:7/693: creat d0/d4c/d8b/fe8 x:0 0 0 2024-04-22T15:50:05.111 INFO:tasks.workunit.client.1.smithi027.stdout:8/604: sync 2024-04-22T15:50:05.114 INFO:tasks.workunit.client.1.smithi027.stdout:7/694: read - d0/d4c/d8b/d92/fe6 zero size 2024-04-22T15:50:05.115 INFO:tasks.workunit.client.1.smithi027.stdout:0/772: dwrite d0/d28/d63/d67/f82 [0,4194304] 0 2024-04-22T15:50:05.121 INFO:tasks.workunit.client.1.smithi027.stdout:8/605: creat d1/d3/d1c/d43/d5f/d7d/fd2 x:0 0 0 2024-04-22T15:50:05.122 INFO:tasks.workunit.client.1.smithi027.stdout:3/772: dwrite d28/d4b/d52/f61 [0,4194304] 0 2024-04-22T15:50:05.129 INFO:tasks.workunit.client.1.smithi027.stdout:7/695: dread d0/d11/d1c/d31/d56/f58 [0,4194304] 0 2024-04-22T15:50:05.130 INFO:tasks.workunit.client.1.smithi027.stdout:0/773: dwrite d0/d34/f71 [0,4194304] 0 2024-04-22T15:50:05.133 INFO:tasks.workunit.client.1.smithi027.stdout:0/774: fdatasync d0/da3/db6/fe9 0 2024-04-22T15:50:05.138 INFO:tasks.workunit.client.1.smithi027.stdout:8/606: dwrite d1/d3/d1c/d32/fcb [0,4194304] 0 2024-04-22T15:50:05.140 INFO:tasks.workunit.client.1.smithi027.stdout:3/773: rename d28/d69/lb4 to d28/d4b/dc6/lf7 0 2024-04-22T15:50:05.144 INFO:tasks.workunit.client.1.smithi027.stdout:7/696: mkdir d0/d11/d1c/d31/d62/de9 0 2024-04-22T15:50:05.146 INFO:tasks.workunit.client.1.smithi027.stdout:0/775: dwrite d0/f2e [0,4194304] 0 2024-04-22T15:50:05.148 INFO:tasks.workunit.client.1.smithi027.stdout:7/697: chown d0/d11/d3a/d46/fe3 396876 1 2024-04-22T15:50:05.149 INFO:tasks.workunit.client.1.smithi027.stdout:7/698: read - d0/d11/d1c/d31/d56/fc6 zero size 2024-04-22T15:50:05.150 INFO:tasks.workunit.client.1.smithi027.stdout:8/607: mkdir d1/d3/d1c/d32/d41/d52/d61/d8f/dd3 0 2024-04-22T15:50:05.153 INFO:tasks.workunit.client.1.smithi027.stdout:8/608: dread - d1/d1e/d47/d9d/faa zero size 2024-04-22T15:50:05.157 INFO:tasks.workunit.client.1.smithi027.stdout:0/776: rename d0/d34/l84 to d0/d9d/lf4 0 2024-04-22T15:50:05.157 INFO:tasks.workunit.client.1.smithi027.stdout:7/699: symlink d0/d4c/d8b/d72/db2/lea 0 2024-04-22T15:50:05.159 INFO:tasks.workunit.client.1.smithi027.stdout:3/774: truncate d28/f44 1329988 0 2024-04-22T15:50:05.160 INFO:tasks.workunit.client.1.smithi027.stdout:8/609: stat d1/d3/d1c/d43/d5f/f9e 0 2024-04-22T15:50:05.164 INFO:tasks.workunit.client.1.smithi027.stdout:0/777: creat d0/d2a/d6a/d6c/d83/d9e/ff5 x:0 0 0 2024-04-22T15:50:05.165 INFO:tasks.workunit.client.1.smithi027.stdout:3/775: chown d28/d4b/d52/db2/d95/lb6 25633 1 2024-04-22T15:50:05.166 INFO:tasks.workunit.client.1.smithi027.stdout:3/776: truncate d28/d4b/f59 4572239 0 2024-04-22T15:50:05.167 INFO:tasks.workunit.client.1.smithi027.stdout:7/700: unlink d0/d11/d1c/d2c/c51 0 2024-04-22T15:50:05.170 INFO:tasks.workunit.client.1.smithi027.stdout:3/777: creat d28/d5d/db0/dce/dd4/ff8 x:0 0 0 2024-04-22T15:50:05.171 INFO:tasks.workunit.client.1.smithi027.stdout:0/778: symlink d0/d2a/d91/da1/lf6 0 2024-04-22T15:50:05.172 INFO:tasks.workunit.client.1.smithi027.stdout:0/779: dread - d0/d2a/d58/d66/fd9 zero size 2024-04-22T15:50:05.175 INFO:tasks.workunit.client.1.smithi027.stdout:8/610: dread d1/f18 [0,4194304] 0 2024-04-22T15:50:05.177 INFO:tasks.workunit.client.1.smithi027.stdout:3/778: rename d28/d4b/d52/db2/d95/fea to d28/d69/ff9 0 2024-04-22T15:50:05.180 INFO:tasks.workunit.client.1.smithi027.stdout:7/701: rmdir d0/d11/d1c/d31 39 2024-04-22T15:50:05.181 INFO:tasks.workunit.client.1.smithi027.stdout:8/611: read d1/d3/d1c/d43/d82/f8b [3296,125006] 0 2024-04-22T15:50:05.183 INFO:tasks.workunit.client.1.smithi027.stdout:8/612: chown d1/d3/d1c/d28/f51 46775 1 2024-04-22T15:50:05.184 INFO:tasks.workunit.client.1.smithi027.stdout:7/702: fdatasync d0/d49/d9f/fb5 0 2024-04-22T15:50:05.186 INFO:tasks.workunit.client.1.smithi027.stdout:8/613: chown d1/d3/d1c/l3e 492 1 2024-04-22T15:50:05.186 INFO:tasks.workunit.client.1.smithi027.stdout:3/779: dwrite d28/d69/f86 [0,4194304] 0 2024-04-22T15:50:05.193 INFO:tasks.workunit.client.1.smithi027.stdout:7/703: creat d0/d11/d3a/d46/d94/d5a/feb x:0 0 0 2024-04-22T15:50:05.203 INFO:tasks.workunit.client.1.smithi027.stdout:7/704: dwrite d0/fcc [0,4194304] 0 2024-04-22T15:50:05.203 INFO:tasks.workunit.client.1.smithi027.stdout:3/780: mknod d28/d4b/d52/d94/dde/cfa 0 2024-04-22T15:50:05.203 INFO:tasks.workunit.client.1.smithi027.stdout:7/705: chown d0/d11/d3a/d46 594068055 1 2024-04-22T15:50:05.203 INFO:tasks.workunit.client.1.smithi027.stdout:0/780: symlink d0/d34/d4e/d85/ddd/lf7 0 2024-04-22T15:50:05.210 INFO:tasks.workunit.client.1.smithi027.stdout:8/614: getdents d1/d1e/d47/d53 0 2024-04-22T15:50:05.211 INFO:tasks.workunit.client.1.smithi027.stdout:8/615: chown d1/d3/d1c/d32/d41/d52/fbf 44 1 2024-04-22T15:50:05.214 INFO:tasks.workunit.client.1.smithi027.stdout:3/781: write d28/d5d/d62/f8f [2317954,64870] 0 2024-04-22T15:50:05.218 INFO:tasks.workunit.client.1.smithi027.stdout:0/781: write d0/d34/f7a [3677324,101858] 0 2024-04-22T15:50:05.219 INFO:tasks.workunit.client.1.smithi027.stdout:8/616: mknod d1/d4f/cd4 0 2024-04-22T15:50:05.222 INFO:tasks.workunit.client.1.smithi027.stdout:7/706: creat d0/fec x:0 0 0 2024-04-22T15:50:05.229 INFO:tasks.workunit.client.1.smithi027.stdout:8/617: creat d1/d1e/d47/d9d/fd5 x:0 0 0 2024-04-22T15:50:05.230 INFO:tasks.workunit.client.1.smithi027.stdout:7/707: symlink d0/d11/d3a/d46/d94/led 0 2024-04-22T15:50:05.236 INFO:tasks.workunit.client.1.smithi027.stdout:0/782: creat d0/d34/ff8 x:0 0 0 2024-04-22T15:50:05.238 INFO:tasks.workunit.client.1.smithi027.stdout:3/782: dwrite d28/f56 [0,4194304] 0 2024-04-22T15:50:05.244 INFO:tasks.workunit.client.1.smithi027.stdout:7/708: symlink d0/d11/d1c/d31/lee 0 2024-04-22T15:50:05.248 INFO:tasks.workunit.client.1.smithi027.stdout:8/618: dwrite d1/d3/d1c/d32/fb0 [4194304,4194304] 0 2024-04-22T15:50:05.257 INFO:tasks.workunit.client.1.smithi027.stdout:7/709: dwrite d0/d4c/d8b/d92/da3/fe1 [0,4194304] 0 2024-04-22T15:50:05.261 INFO:tasks.workunit.client.1.smithi027.stdout:7/710: fsync d0/d4c/d8b/d92/da3/da8/f3d 0 2024-04-22T15:50:05.265 INFO:tasks.workunit.client.1.smithi027.stdout:0/783: dread d0/d2a/f2b [0,4194304] 0 2024-04-22T15:50:05.265 INFO:tasks.workunit.client.1.smithi027.stdout:8/619: dwrite d1/d3/d1c/d32/d41/d52/fbf [0,4194304] 0 2024-04-22T15:50:05.276 INFO:tasks.workunit.client.1.smithi027.stdout:7/711: dwrite d0/d11/d3a/d46/d94/d5a/fb6 [0,4194304] 0 2024-04-22T15:50:05.278 INFO:tasks.workunit.client.1.smithi027.stdout:7/712: readlink d0/d4c/d8b/l6e 0 2024-04-22T15:50:05.279 INFO:tasks.workunit.client.1.smithi027.stdout:3/783: dread f11 [0,4194304] 0 2024-04-22T15:50:05.281 INFO:tasks.workunit.client.1.smithi027.stdout:7/713: rmdir d0/d11/d1c/d2c/d96/db3 39 2024-04-22T15:50:05.282 INFO:tasks.workunit.client.1.smithi027.stdout:8/620: write d1/d3/d1c/d43/d82/f8b [191180,6978] 0 2024-04-22T15:50:05.284 INFO:tasks.workunit.client.1.smithi027.stdout:8/621: write d1/d3/f46 [20430,3503] 0 2024-04-22T15:50:05.284 INFO:tasks.workunit.client.1.smithi027.stdout:3/784: dread d28/d69/f77 [0,4194304] 0 2024-04-22T15:50:05.287 INFO:tasks.workunit.client.1.smithi027.stdout:7/714: dread d0/d11/d3a/d46/f82 [0,4194304] 0 2024-04-22T15:50:05.301 INFO:tasks.workunit.client.1.smithi027.stdout:2/671: sync 2024-04-22T15:50:05.304 INFO:tasks.workunit.client.1.smithi027.stdout:2/672: dread - dd/df/d70/d4b/d4e/daf/fb9 zero size 2024-04-22T15:50:05.311 INFO:tasks.workunit.client.1.smithi027.stdout:8/622: chown d1/d1e/l29 354500 1 2024-04-22T15:50:05.312 INFO:tasks.workunit.client.1.smithi027.stdout:8/623: chown d1/d3/d1c/d32/d41/d52 15036 1 2024-04-22T15:50:05.318 INFO:tasks.workunit.client.1.smithi027.stdout:8/624: dwrite d1/d3/d1c/d32/fb0 [0,4194304] 0 2024-04-22T15:50:05.318 INFO:tasks.workunit.client.1.smithi027.stdout:2/673: symlink dd/df/d70/d4b/d4e/lcf 0 2024-04-22T15:50:05.320 INFO:tasks.workunit.client.1.smithi027.stdout:8/625: truncate d1/d3/d1c/d32/d41/d52/fb6 129923 0 2024-04-22T15:50:05.322 INFO:tasks.workunit.client.1.smithi027.stdout:3/785: link d28/d4b/f57 d28/d5d/d79/ffb 0 2024-04-22T15:50:05.324 INFO:tasks.workunit.client.1.smithi027.stdout:2/674: symlink dd/df/d70/d4b/d4e/daf/ld0 0 2024-04-22T15:50:05.326 INFO:tasks.workunit.client.1.smithi027.stdout:3/786: dread - d28/d4b/d52/d9a/fab zero size 2024-04-22T15:50:05.327 INFO:tasks.workunit.client.1.smithi027.stdout:7/715: link d0/d11/d1c/d31/d8a/fa4 d0/d4c/d8b/d92/da3/da8/fef 0 2024-04-22T15:50:05.331 INFO:tasks.workunit.client.1.smithi027.stdout:2/675: mkdir dd/df/d1b/d96/dc5/dd1 0 2024-04-22T15:50:05.331 INFO:tasks.workunit.client.1.smithi027.stdout:8/626: link d1/d1e/d47/d53/d76/c7e d1/d3/d1c/d32/d41/d52/cd6 0 2024-04-22T15:50:05.333 INFO:tasks.workunit.client.1.smithi027.stdout:3/787: mkdir d28/d4b/d52/d94/dc4/dfc 0 2024-04-22T15:50:05.333 INFO:tasks.workunit.client.1.smithi027.stdout:2/676: rmdir dd/dae 39 2024-04-22T15:50:05.334 INFO:tasks.workunit.client.1.smithi027.stdout:8/627: rmdir d1/d3/d1c/d43/d82 39 2024-04-22T15:50:05.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:05 smithi027 ceph-mon[34922]: pgmap v6: 65 pgs: 65 active+clean; 2.9 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail 2024-04-22T15:50:05.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:05 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:05.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:05 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:05.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:05 smithi027 ceph-mon[34922]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:05.337 INFO:tasks.workunit.client.1.smithi027.stdout:3/788: symlink d28/d4b/d52/d94/dc4/dfc/lfd 0 2024-04-22T15:50:05.338 INFO:tasks.workunit.client.1.smithi027.stdout:2/677: unlink dd/df/d70/lb7 0 2024-04-22T15:50:05.340 INFO:tasks.workunit.client.1.smithi027.stdout:8/628: rename d1/d3/d36/d57/c7c to d1/d4f/cd7 0 2024-04-22T15:50:05.342 INFO:tasks.workunit.client.1.smithi027.stdout:8/629: chown d1/d3/d36/d57/l58 3551404 1 2024-04-22T15:50:05.345 INFO:tasks.workunit.client.1.smithi027.stdout:3/789: creat d28/d4b/d52/d9a/ffe x:0 0 0 2024-04-22T15:50:05.346 INFO:tasks.workunit.client.1.smithi027.stdout:2/678: creat dd/df/d1b/d96/dc5/dd1/fd2 x:0 0 0 2024-04-22T15:50:05.350 INFO:tasks.workunit.client.1.smithi027.stdout:2/679: dwrite dd/df/d1b/f5f [0,4194304] 0 2024-04-22T15:50:05.351 INFO:tasks.workunit.client.1.smithi027.stdout:8/630: getdents d1/d3/d95 0 2024-04-22T15:50:05.351 INFO:tasks.workunit.client.1.smithi027.stdout:2/680: chown dd/df/d1b/d38/f5c 27478794 1 2024-04-22T15:50:05.355 INFO:tasks.workunit.client.1.smithi027.stdout:2/681: write dd/f1c [4227744,31594] 0 2024-04-22T15:50:05.355 INFO:tasks.workunit.client.1.smithi027.stdout:8/631: creat d1/d1e/d47/dc1/fd8 x:0 0 0 2024-04-22T15:50:05.356 INFO:tasks.workunit.client.1.smithi027.stdout:8/632: read d1/f18 [1803279,22700] 0 2024-04-22T15:50:05.360 INFO:tasks.workunit.client.1.smithi027.stdout:2/682: mknod dd/df/d1b/d38/d60/d75/cd3 0 2024-04-22T15:50:05.361 INFO:tasks.workunit.client.1.smithi027.stdout:2/683: chown dd/df/d1b/f2e 91290 1 2024-04-22T15:50:05.361 INFO:tasks.workunit.client.1.smithi027.stdout:8/633: write d1/d3/d1c/d43/d82/f8b [970849,53234] 0 2024-04-22T15:50:05.367 INFO:tasks.workunit.client.1.smithi027.stdout:2/684: mkdir dd/dae/dd4 0 2024-04-22T15:50:05.369 INFO:tasks.workunit.client.1.smithi027.stdout:8/634: dwrite d1/d1e/d47/d53/f77 [4194304,4194304] 0 2024-04-22T15:50:05.373 INFO:tasks.workunit.client.1.smithi027.stdout:2/685: mkdir dd/df/d1b/d38/dd5 0 2024-04-22T15:50:05.374 INFO:tasks.workunit.client.1.smithi027.stdout:2/686: chown dd/df/d1b/d38/d60/fa2 102 1 2024-04-22T15:50:05.376 INFO:tasks.workunit.client.1.smithi027.stdout:8/635: rename d1/d3/f10 to d1/d3/d1c/d32/d41/fd9 0 2024-04-22T15:50:05.385 INFO:tasks.workunit.client.1.smithi027.stdout:8/636: dwrite d1/d3/d1c/d28/d96/fa0 [0,4194304] 0 2024-04-22T15:50:05.388 INFO:tasks.workunit.client.1.smithi027.stdout:8/637: rmdir d1/d3/d1c/d49 39 2024-04-22T15:50:05.392 INFO:tasks.workunit.client.1.smithi027.stdout:8/638: symlink d1/d3/d1c/d56/dc9/lda 0 2024-04-22T15:50:05.395 INFO:tasks.workunit.client.1.smithi027.stdout:8/639: creat d1/d1e/d47/dc1/fdb x:0 0 0 2024-04-22T15:50:05.397 INFO:tasks.workunit.client.1.smithi027.stdout:8/640: chown d1/d1e/c48 3147321 1 2024-04-22T15:50:05.399 INFO:tasks.workunit.client.1.smithi027.stdout:8/641: chown d1/d3/d1c/d32/d41/d52/d61/d8f/c9f 1936 1 2024-04-22T15:50:05.415 INFO:tasks.workunit.client.1.smithi027.stdout:2/687: dwrite dd/df/d1b/f5e [0,4194304] 0 2024-04-22T15:50:05.416 INFO:tasks.workunit.client.1.smithi027.stdout:8/642: write d1/d3/f40 [2229293,3856] 0 2024-04-22T15:50:05.421 INFO:tasks.workunit.client.1.smithi027.stdout:2/688: creat dd/df/d1b/d96/dc5/fd6 x:0 0 0 2024-04-22T15:50:05.423 INFO:tasks.workunit.client.1.smithi027.stdout:8/643: creat d1/d1e/d47/d53/fdc x:0 0 0 2024-04-22T15:50:05.425 INFO:tasks.workunit.client.1.smithi027.stdout:2/689: symlink dd/df/d70/d4b/d4e/daf/ld7 0 2024-04-22T15:50:05.426 INFO:tasks.workunit.client.1.smithi027.stdout:2/690: fdatasync dd/df/d1b/fc1 0 2024-04-22T15:50:05.427 INFO:tasks.workunit.client.1.smithi027.stdout:8/644: rename d1/d3/d95 to d1/d3/d1c/d32/d41/d52/d61/d8f/ddd 0 2024-04-22T15:50:05.441 INFO:tasks.workunit.client.1.smithi027.stdout:8/645: dread d1/d3/fa [0,4194304] 0 2024-04-22T15:50:05.442 INFO:tasks.workunit.client.1.smithi027.stdout:2/691: write dd/f6f [515964,95846] 0 2024-04-22T15:50:05.445 INFO:tasks.workunit.client.1.smithi027.stdout:8/646: unlink d1/d1e/d47/d9d/fd5 0 2024-04-22T15:50:05.448 INFO:tasks.workunit.client.1.smithi027.stdout:2/692: dwrite dd/d43/fa7 [0,4194304] 0 2024-04-22T15:50:05.449 INFO:tasks.workunit.client.1.smithi027.stdout:2/693: chown dd/d47/f48 205 1 2024-04-22T15:50:05.451 INFO:tasks.workunit.client.1.smithi027.stdout:2/694: write dd/df/d70/d4b/d52/d9a/da4/fcc [372555,60639] 0 2024-04-22T15:50:05.455 INFO:tasks.workunit.client.1.smithi027.stdout:2/695: mkdir dd/df/d70/d4b/d4e/daf/dd8 0 2024-04-22T15:50:05.504 INFO:tasks.workunit.client.1.smithi027.stdout:1/618: sync 2024-04-22T15:50:05.504 INFO:tasks.workunit.client.1.smithi027.stdout:5/709: sync 2024-04-22T15:50:05.505 INFO:tasks.workunit.client.1.smithi027.stdout:6/798: sync 2024-04-22T15:50:05.505 INFO:tasks.workunit.client.1.smithi027.stdout:9/669: sync 2024-04-22T15:50:05.505 INFO:tasks.workunit.client.1.smithi027.stdout:4/683: sync 2024-04-22T15:50:05.554 INFO:tasks.workunit.client.1.smithi027.stdout:8/647: dwrite d1/d3/d1c/d32/fca [0,4194304] 0 2024-04-22T15:50:05.555 INFO:tasks.workunit.client.1.smithi027.stdout:8/648: stat d1/d1e/f2f 0 2024-04-22T15:50:05.558 INFO:tasks.workunit.client.1.smithi027.stdout:9/670: mkdir d0/dc/d12/d58/ddb 0 2024-04-22T15:50:05.628 INFO:tasks.workunit.client.1.smithi027.stdout:6/799: mknod d5/dc/d26/d49/c115 0 2024-04-22T15:50:05.628 INFO:tasks.workunit.client.1.smithi027.stdout:1/619: mkdir d2/d5f/d64/d81/dcf 0 2024-04-22T15:50:05.629 INFO:tasks.workunit.client.1.smithi027.stdout:2/696: dread dd/df/f86 [0,4194304] 0 2024-04-22T15:50:05.629 INFO:tasks.workunit.client.1.smithi027.stdout:5/710: symlink d0/df/de/d40/d89/dd0/le5 0 2024-04-22T15:50:05.636 INFO:tasks.workunit.client.1.smithi027.stdout:2/697: chown dd/df/d1b/d38/d60/d75/d9f 893866 1 2024-04-22T15:50:05.638 INFO:tasks.workunit.client.1.smithi027.stdout:9/671: dwrite d0/dc/d12/f94 [0,4194304] 0 2024-04-22T15:50:05.646 INFO:tasks.workunit.client.1.smithi027.stdout:6/800: dread d5/dc/d17/d10e/f111 [4194304,4194304] 0 2024-04-22T15:50:05.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:05 smithi016 ceph-mon[26620]: pgmap v6: 65 pgs: 65 active+clean; 2.9 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail 2024-04-22T15:50:05.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:05 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:05.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:05 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:05.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:05 smithi016 ceph-mon[26620]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:05.688 INFO:tasks.workunit.client.1.smithi027.stdout:0/784: sync 2024-04-22T15:50:05.690 INFO:tasks.workunit.client.1.smithi027.stdout:5/711: rename d0/db6/fc2 to d0/db6/fe6 0 2024-04-22T15:50:05.753 INFO:tasks.workunit.client.1.smithi027.stdout:7/716: sync 2024-04-22T15:50:05.753 INFO:tasks.workunit.client.1.smithi027.stdout:3/790: sync 2024-04-22T15:50:05.755 INFO:tasks.workunit.client.1.smithi027.stdout:3/791: fsync d28/d4b/d52/d9a/fd1 0 2024-04-22T15:50:05.756 INFO:tasks.workunit.client.1.smithi027.stdout:3/792: write d28/f48 [4243226,17242] 0 2024-04-22T15:50:05.762 INFO:tasks.workunit.client.1.smithi027.stdout:0/785: mkdir d0/da3/dae/df9 0 2024-04-22T15:50:05.762 INFO:tasks.workunit.client.1.smithi027.stdout:0/786: stat d0/d2a/d38/f55 0 2024-04-22T15:50:05.765 INFO:tasks.workunit.client.1.smithi027.stdout:2/698: symlink dd/df/d1b/d38/ld9 0 2024-04-22T15:50:05.774 INFO:tasks.workunit.client.1.smithi027.stdout:7/717: truncate d0/f29 2348646 0 2024-04-22T15:50:05.775 INFO:tasks.workunit.client.1.smithi027.stdout:7/718: stat d0/d3f/l41 0 2024-04-22T15:50:05.779 INFO:tasks.workunit.client.1.smithi027.stdout:2/699: mkdir dd/df/d70/d4b/d52/dda 0 2024-04-22T15:50:05.791 INFO:tasks.workunit.client.1.smithi027.stdout:6/801: getdents d5/dc/dd/d79 0 2024-04-22T15:50:05.791 INFO:tasks.workunit.client.1.smithi027.stdout:7/719: truncate d0/d11/d1c/d2c/d96/db3/fe4 128419 0 2024-04-22T15:50:05.792 INFO:tasks.workunit.client.1.smithi027.stdout:6/802: write d5/dc/d26/dee/f10d [143360,25206] 0 2024-04-22T15:50:05.793 INFO:tasks.workunit.client.1.smithi027.stdout:2/700: mkdir dd/dae/dd4/ddb 0 2024-04-22T15:50:05.795 INFO:tasks.workunit.client.1.smithi027.stdout:4/684: sync 2024-04-22T15:50:05.795 INFO:tasks.workunit.client.1.smithi027.stdout:2/701: dread dd/df/d70/d5a/f87 [0,4194304] 0 2024-04-22T15:50:05.796 INFO:tasks.workunit.client.1.smithi027.stdout:6/803: chown d5/dc/dd/d79/d7b/d5d/dba/lf7 10 1 2024-04-22T15:50:05.797 INFO:tasks.workunit.client.1.smithi027.stdout:7/720: write d0/d11/d1c/d31/d62/faf [721669,50704] 0 2024-04-22T15:50:05.799 INFO:tasks.workunit.client.1.smithi027.stdout:7/721: stat d0/d4c/d8b/d92/da3/ce0 0 2024-04-22T15:50:05.800 INFO:tasks.workunit.client.1.smithi027.stdout:4/685: creat d5/d11/d1d/d54/d8c/f102 x:0 0 0 2024-04-22T15:50:05.802 INFO:tasks.workunit.client.1.smithi027.stdout:2/702: mknod dd/df/d70/cdc 0 2024-04-22T15:50:05.802 INFO:tasks.workunit.client.1.smithi027.stdout:4/686: write d5/d45/db3/fe8 [857237,58064] 0 2024-04-22T15:50:05.804 INFO:tasks.workunit.client.1.smithi027.stdout:2/703: chown dd/df/d1b/f26 127 1 2024-04-22T15:50:05.806 INFO:tasks.workunit.client.1.smithi027.stdout:2/704: read dd/df/d1b/d38/d60/f83 [4016907,125209] 0 2024-04-22T15:50:05.807 INFO:tasks.workunit.client.1.smithi027.stdout:7/722: chown d0/d4c/d8b/d92/da3/cd9 26981 1 2024-04-22T15:50:05.811 INFO:tasks.workunit.client.1.smithi027.stdout:6/804: link d5/f75 d5/dc/d26/d49/d86/f116 0 2024-04-22T15:50:05.814 INFO:tasks.workunit.client.1.smithi027.stdout:2/705: unlink dd/df/d1b/d1f/f82 0 2024-04-22T15:50:05.814 INFO:tasks.workunit.client.1.smithi027.stdout:2/706: fsync dd/f1c 0 2024-04-22T15:50:05.815 INFO:tasks.workunit.client.1.smithi027.stdout:7/723: rmdir d0/d11/d1c/d2c 39 2024-04-22T15:50:05.826 INFO:tasks.workunit.client.1.smithi027.stdout:7/724: getdents d0/d11/d1c/ddc 0 2024-04-22T15:50:05.830 INFO:tasks.workunit.client.1.smithi027.stdout:7/725: unlink d0/d11/d3a/d46/d94/c5d 0 2024-04-22T15:50:05.834 INFO:tasks.workunit.client.1.smithi027.stdout:7/726: write d0/d11/d1c/d2c/f55 [2129225,72438] 0 2024-04-22T15:50:05.835 INFO:tasks.workunit.client.1.smithi027.stdout:6/805: mknod d5/dc/d20/da3/c117 0 2024-04-22T15:50:05.843 INFO:tasks.workunit.client.1.smithi027.stdout:1/620: dwrite d2/d36/d74/f79 [0,4194304] 0 2024-04-22T15:50:05.849 INFO:tasks.workunit.client.1.smithi027.stdout:7/727: link d0/d4c/d8b/d72/db2/f69 d0/d4c/d4e/ff0 0 2024-04-22T15:50:05.854 INFO:tasks.workunit.client.1.smithi027.stdout:6/806: dread d5/dc/dd/d79/d7b/d5d/f70 [0,4194304] 0 2024-04-22T15:50:05.858 INFO:tasks.workunit.client.1.smithi027.stdout:6/807: creat d5/dc/dae/f118 x:0 0 0 2024-04-22T15:50:05.863 INFO:tasks.workunit.client.1.smithi027.stdout:6/808: creat d5/dc/dd/d79/d7b/d9a/f119 x:0 0 0 2024-04-22T15:50:05.892 INFO:tasks.workunit.client.1.smithi027.stdout:9/672: write d0/dc/d12/d30/d5b/d91/da9/fbe [5097828,32506] 0 2024-04-22T15:50:05.900 INFO:tasks.workunit.client.1.smithi027.stdout:8/649: rename d1/d3/d1c/d43/f80 to d1/d3/d1c/d32/d41/d52/d61/d8f/ddd/dbe/fde 0 2024-04-22T15:50:05.902 INFO:tasks.workunit.client.1.smithi027.stdout:8/650: fsync d1/d3/d1c/d32/d41/f83 0 2024-04-22T15:50:05.905 INFO:tasks.workunit.client.1.smithi027.stdout:8/651: write d1/d3/d1c/d32/d41/d52/d61/fce [46497,4956] 0 2024-04-22T15:50:05.914 INFO:tasks.workunit.client.1.smithi027.stdout:8/652: creat d1/d1e/d47/d53/d76/fdf x:0 0 0 2024-04-22T15:50:05.916 INFO:tasks.workunit.client.1.smithi027.stdout:3/793: truncate d28/d4b/d52/db2/d95/fe5 3002408 0 2024-04-22T15:50:05.917 INFO:tasks.workunit.client.1.smithi027.stdout:9/673: dread d0/dc/d12/d30/d5b/f65 [0,4194304] 0 2024-04-22T15:50:05.920 INFO:tasks.workunit.client.1.smithi027.stdout:8/653: dread d1/d3/d1c/d32/fcb [0,4194304] 0 2024-04-22T15:50:05.924 INFO:tasks.workunit.client.1.smithi027.stdout:3/794: unlink d28/d69/f77 0 2024-04-22T15:50:05.926 INFO:tasks.workunit.client.1.smithi027.stdout:9/674: creat d0/dc/d12/d30/d5b/d91/da9/fdc x:0 0 0 2024-04-22T15:50:05.928 INFO:tasks.workunit.client.1.smithi027.stdout:9/675: fsync d0/dc/d12/d58/d6b/d6e/f61 0 2024-04-22T15:50:05.940 INFO:tasks.workunit.client.1.smithi027.stdout:4/687: write d5/de/d2a/d58/f68 [2800083,59949] 0 2024-04-22T15:50:05.943 INFO:tasks.workunit.client.1.smithi027.stdout:9/676: rmdir d0/dc/de/db5/dbf/dd3 0 2024-04-22T15:50:05.945 INFO:tasks.workunit.client.1.smithi027.stdout:9/677: read - d0/dc/d12/d30/d99/da7/fda zero size 2024-04-22T15:50:05.947 INFO:tasks.workunit.client.1.smithi027.stdout:4/688: truncate d5/da8/fb8 382659 0 2024-04-22T15:50:05.950 INFO:tasks.workunit.client.1.smithi027.stdout:0/787: rename d0/d2a/c51 to d0/d2a/d58/d66/ded/cfa 0 2024-04-22T15:50:05.953 INFO:tasks.workunit.client.1.smithi027.stdout:9/678: dwrite d0/dc/de/f3a [0,4194304] 0 2024-04-22T15:50:05.958 INFO:tasks.workunit.client.1.smithi027.stdout:1/621: rename d2/d5f/fc6 to d2/d5f/d64/d81/d3d/d44/d9c/d9f/fd0 0 2024-04-22T15:50:05.959 INFO:tasks.workunit.client.1.smithi027.stdout:1/622: readlink d2/d36/daa/lb7 0 2024-04-22T15:50:05.959 INFO:tasks.workunit.client.1.smithi027.stdout:0/788: symlink d0/d2a/d6a/d6c/d83/d9e/lfb 0 2024-04-22T15:50:05.965 INFO:tasks.workunit.client.1.smithi027.stdout:6/809: rename d5/dc/d26/d49/d86/f116 to d5/db/d2e/d59/df0/f11a 0 2024-04-22T15:50:05.968 INFO:tasks.workunit.client.1.smithi027.stdout:6/810: truncate d5/dc/dd/d79/d7b/d9a/fa0 780371 0 2024-04-22T15:50:05.968 INFO:tasks.workunit.client.1.smithi027.stdout:1/623: creat d2/d5f/fd1 x:0 0 0 2024-04-22T15:50:05.969 INFO:tasks.workunit.client.1.smithi027.stdout:2/707: truncate dd/df/d70/f61 75047 0 2024-04-22T15:50:05.971 INFO:tasks.workunit.client.1.smithi027.stdout:0/789: symlink d0/d2a/d6a/d6c/lfc 0 2024-04-22T15:50:05.974 INFO:tasks.workunit.client.1.smithi027.stdout:6/811: creat d5/db/d2e/d59/f11b x:0 0 0 2024-04-22T15:50:05.975 INFO:tasks.workunit.client.1.smithi027.stdout:6/812: dread - d5/d30/dbe/d112/ff3 zero size 2024-04-22T15:50:05.977 INFO:tasks.workunit.client.1.smithi027.stdout:6/813: fdatasync d5/dc/dd/f16 0 2024-04-22T15:50:05.978 INFO:tasks.workunit.client.1.smithi027.stdout:2/708: creat dd/df/d70/d4b/d52/d9a/da4/dad/fdd x:0 0 0 2024-04-22T15:50:05.980 INFO:tasks.workunit.client.1.smithi027.stdout:0/790: creat d0/d34/d4e/ffd x:0 0 0 2024-04-22T15:50:05.982 INFO:tasks.workunit.client.1.smithi027.stdout:6/814: mkdir d5/dc/d17/d11c 0 2024-04-22T15:50:05.986 INFO:tasks.workunit.client.1.smithi027.stdout:0/791: write d0/d28/d37/fc3 [4248497,22396] 0 2024-04-22T15:50:05.986 INFO:tasks.workunit.client.1.smithi027.stdout:6/815: chown d5/dc/d17/d8a/le4 33725 1 2024-04-22T15:50:05.988 INFO:tasks.workunit.client.1.smithi027.stdout:9/679: unlink d0/dc/f4c 0 2024-04-22T15:50:05.989 INFO:tasks.workunit.client.1.smithi027.stdout:2/709: dwrite dd/df/d1b/d96/fbd [0,4194304] 0 2024-04-22T15:50:05.990 INFO:tasks.workunit.client.1.smithi027.stdout:6/816: dread - d5/dc/dd/ffa zero size 2024-04-22T15:50:05.992 INFO:tasks.workunit.client.1.smithi027.stdout:3/795: dread d28/d4b/d52/f6a [0,4194304] 0 2024-04-22T15:50:05.992 INFO:tasks.workunit.client.1.smithi027.stdout:7/728: write d0/d4c/d8b/d92/da3/da8/f25 [2222913,55810] 0 2024-04-22T15:50:05.994 INFO:tasks.workunit.client.1.smithi027.stdout:0/792: mknod d0/d28/d37/cfe 0 2024-04-22T15:50:05.998 INFO:tasks.workunit.client.1.smithi027.stdout:3/796: creat d28/d4b/dc6/fff x:0 0 0 2024-04-22T15:50:05.999 INFO:tasks.workunit.client.1.smithi027.stdout:2/710: fdatasync dd/df/d1b/d1f/f36 0 2024-04-22T15:50:06.001 INFO:tasks.workunit.client.1.smithi027.stdout:2/711: write dd/d43/f95 [681135,41961] 0 2024-04-22T15:50:06.002 INFO:tasks.workunit.client.1.smithi027.stdout:6/817: rename d5/dc/dd/d52 to d5/dc/d17/d11c/d11d 0 2024-04-22T15:50:06.004 INFO:tasks.workunit.client.1.smithi027.stdout:9/680: dwrite d0/dc/d12/d30/d5b/f83 [0,4194304] 0 2024-04-22T15:50:06.006 INFO:tasks.workunit.client.1.smithi027.stdout:9/681: fdatasync d0/dc/d12/d58/fb7 0 2024-04-22T15:50:06.007 INFO:tasks.workunit.client.1.smithi027.stdout:0/793: unlink d0/d2a/d38/fd6 0 2024-04-22T15:50:06.018 INFO:tasks.workunit.client.1.smithi027.stdout:6/818: mknod d5/d30/d54/c11e 0 2024-04-22T15:50:06.029 INFO:tasks.workunit.client.1.smithi027.stdout:0/794: dwrite d0/d34/d59/f5c [0,4194304] 0 2024-04-22T15:50:06.031 INFO:tasks.workunit.client.1.smithi027.stdout:0/795: write d0/d2a/d5e/d8b/f9f [2520948,13919] 0 2024-04-22T15:50:06.031 INFO:tasks.workunit.client.1.smithi027.stdout:3/797: link d28/d4b/d52/d94/fc2 d28/d5d/f100 0 2024-04-22T15:50:06.032 INFO:tasks.workunit.client.1.smithi027.stdout:3/798: chown d28/d4b/d52 63 1 2024-04-22T15:50:06.033 INFO:tasks.workunit.client.1.smithi027.stdout:3/799: dread - d28/d4b/d52/d94/fe0 zero size 2024-04-22T15:50:06.036 INFO:tasks.workunit.client.1.smithi027.stdout:5/712: mknod d0/d1a/d22/d26/d5d/ce7 0 2024-04-22T15:50:06.046 INFO:tasks.workunit.client.1.smithi027.stdout:5/713: dwrite d0/df/de/d40/d89/fcc [0,4194304] 0 2024-04-22T15:50:06.046 INFO:tasks.workunit.client.1.smithi027.stdout:3/800: stat d28/d4b/d52/db2/d95/ca0 0 2024-04-22T15:50:06.049 INFO:tasks.workunit.client.1.smithi027.stdout:2/712: getdents dd/df/d70/d4b/d52 0 2024-04-22T15:50:06.050 INFO:tasks.workunit.client.1.smithi027.stdout:0/796: symlink d0/lff 0 2024-04-22T15:50:06.051 INFO:tasks.workunit.client.1.smithi027.stdout:0/797: dread - d0/d2a/d6a/d6c/d83/d9e/ff5 zero size 2024-04-22T15:50:06.052 INFO:tasks.workunit.client.1.smithi027.stdout:0/798: read d0/d28/d63/d67/f82 [1061349,96399] 0 2024-04-22T15:50:06.053 INFO:tasks.workunit.client.1.smithi027.stdout:6/819: getdents d5/d30/dbe 0 2024-04-22T15:50:06.054 INFO:tasks.workunit.client.1.smithi027.stdout:5/714: mkdir d0/d1a/d22/d72/de8 0 2024-04-22T15:50:06.056 INFO:tasks.workunit.client.1.smithi027.stdout:6/820: truncate d5/dc/d26/d3d/feb 192024 0 2024-04-22T15:50:06.058 INFO:tasks.workunit.client.1.smithi027.stdout:2/713: fdatasync dd/df/d1b/f25 0 2024-04-22T15:50:06.064 INFO:tasks.workunit.client.1.smithi027.stdout:3/801: mkdir d28/d4b/d52/db2/d101 0 2024-04-22T15:50:06.066 INFO:tasks.workunit.client.1.smithi027.stdout:6/821: dwrite d5/d30/d40/fce [0,4194304] 0 2024-04-22T15:50:06.070 INFO:tasks.workunit.client.1.smithi027.stdout:8/654: getdents d1/d3/d1c/d43 0 2024-04-22T15:50:06.076 INFO:tasks.workunit.client.1.smithi027.stdout:6/822: dwrite d5/dc/d26/d49/d86/ffb [0,4194304] 0 2024-04-22T15:50:06.087 INFO:tasks.workunit.client.1.smithi027.stdout:2/714: creat dd/df/d1b/d96/dc5/dd1/fde x:0 0 0 2024-04-22T15:50:06.100 INFO:tasks.workunit.client.1.smithi027.stdout:2/715: mkdir dd/df/d1b/d38/d60/d75/ddf 0 2024-04-22T15:50:06.104 INFO:tasks.workunit.client.1.smithi027.stdout:4/689: dwrite d5/d45/d47/f4a [4194304,4194304] 0 2024-04-22T15:50:06.111 INFO:tasks.workunit.client.1.smithi027.stdout:8/655: dread d1/f4 [0,4194304] 0 2024-04-22T15:50:06.113 INFO:tasks.workunit.client.1.smithi027.stdout:1/624: write d2/fa [1075768,87534] 0 2024-04-22T15:50:06.113 INFO:tasks.workunit.client.1.smithi027.stdout:8/656: dread - d1/d3/d1c/d43/d5f/d7d/fd2 zero size 2024-04-22T15:50:06.115 INFO:tasks.workunit.client.1.smithi027.stdout:1/625: stat d2/d30/l89 0 2024-04-22T15:50:06.115 INFO:tasks.workunit.client.1.smithi027.stdout:6/823: dread d5/dc/f74 [0,4194304] 0 2024-04-22T15:50:06.116 INFO:tasks.workunit.client.1.smithi027.stdout:4/690: rename d5/d11/da6/lb5 to d5/d11/d1d/d54/l103 0 2024-04-22T15:50:06.117 INFO:tasks.workunit.client.1.smithi027.stdout:1/626: chown d2/f27 29 1 2024-04-22T15:50:06.119 INFO:tasks.workunit.client.1.smithi027.stdout:4/691: write d5/d11/f80 [5151530,50230] 0 2024-04-22T15:50:06.122 INFO:tasks.workunit.client.1.smithi027.stdout:8/657: symlink d1/d3/d1c/d32/d41/d52/d61/d8f/ddd/le0 0 2024-04-22T15:50:06.127 INFO:tasks.workunit.client.1.smithi027.stdout:1/627: creat d2/d5f/d64/d81/d7e/fd2 x:0 0 0 2024-04-22T15:50:06.127 INFO:tasks.workunit.client.1.smithi027.stdout:1/628: stat d2/d30/f3c 0 2024-04-22T15:50:06.130 INFO:tasks.workunit.client.1.smithi027.stdout:8/658: dread - d1/d3/d36/d88/f97 zero size 2024-04-22T15:50:06.132 INFO:tasks.workunit.client.1.smithi027.stdout:1/629: mknod d2/d5f/d64/d81/d3d/d44/d9c/d9f/cd3 0 2024-04-22T15:50:06.136 INFO:tasks.workunit.client.1.smithi027.stdout:4/692: link d5/d45/f94 d5/de/d2a/d58/f104 0 2024-04-22T15:50:06.139 INFO:tasks.workunit.client.1.smithi027.stdout:7/729: dwrite d0/d4c/d8b/d72/db2/f69 [0,4194304] 0 2024-04-22T15:50:06.143 INFO:tasks.workunit.client.1.smithi027.stdout:9/682: write d0/dc/d12/d58/f7d [364786,95564] 0 2024-04-22T15:50:06.143 INFO:tasks.workunit.client.1.smithi027.stdout:1/630: dread d2/d5f/d64/d81/d3d/d44/d9c/d9f/fb5 [0,4194304] 0 2024-04-22T15:50:06.145 INFO:tasks.workunit.client.1.smithi027.stdout:7/730: mknod d0/d11/d1c/d31/d8a/cf1 0 2024-04-22T15:50:06.146 INFO:tasks.workunit.client.1.smithi027.stdout:4/693: getdents d5/de/d30 0 2024-04-22T15:50:06.156 INFO:tasks.workunit.client.1.smithi027.stdout:7/731: unlink d0/d11/d3a/d46/d94/fda 0 2024-04-22T15:50:06.162 INFO:tasks.workunit.client.1.smithi027.stdout:1/631: truncate d2/f75 3044689 0 2024-04-22T15:50:06.163 INFO:tasks.workunit.client.1.smithi027.stdout:9/683: symlink d0/dc/de/ldd 0 2024-04-22T15:50:06.168 INFO:tasks.workunit.client.1.smithi027.stdout:7/732: unlink d0/fec 0 2024-04-22T15:50:06.171 INFO:tasks.workunit.client.1.smithi027.stdout:9/684: creat d0/dc/d12/d30/d99/db4/fde x:0 0 0 2024-04-22T15:50:06.175 INFO:tasks.workunit.client.1.smithi027.stdout:9/685: symlink d0/d8/ldf 0 2024-04-22T15:50:06.176 INFO:tasks.workunit.client.1.smithi027.stdout:7/733: mknod d0/d11/cf2 0 2024-04-22T15:50:06.179 INFO:tasks.workunit.client.1.smithi027.stdout:0/799: write d0/d34/f80 [1869720,41582] 0 2024-04-22T15:50:06.182 INFO:tasks.workunit.client.1.smithi027.stdout:5/715: write d0/df/de/d24/f9d [413969,27159] 0 2024-04-22T15:50:06.186 INFO:tasks.workunit.client.1.smithi027.stdout:3/802: truncate d28/d5d/f8b 3382736 0 2024-04-22T15:50:06.188 INFO:tasks.workunit.client.1.smithi027.stdout:3/803: read d28/d4b/f7c [3746829,70293] 0 2024-04-22T15:50:06.191 INFO:tasks.workunit.client.1.smithi027.stdout:3/804: chown d28/d4b/d52/db2/d95/cf6 24 1 2024-04-22T15:50:06.192 INFO:tasks.workunit.client.1.smithi027.stdout:2/716: write dd/df/f17 [4972365,46726] 0 2024-04-22T15:50:06.196 INFO:tasks.workunit.client.1.smithi027.stdout:7/734: dread d0/d11/f4d [0,4194304] 0 2024-04-22T15:50:06.199 INFO:tasks.workunit.client.1.smithi027.stdout:9/686: dread d0/dc/d12/d58/d6b/d6e/f61 [4194304,4194304] 0 2024-04-22T15:50:06.200 INFO:tasks.workunit.client.1.smithi027.stdout:2/717: stat dd/df/d70/d4b/d52/d9a/da4/dad/fc0 0 2024-04-22T15:50:06.200 INFO:tasks.workunit.client.1.smithi027.stdout:3/805: creat d28/d4b/d52/d94/dc4/dfc/f102 x:0 0 0 2024-04-22T15:50:06.200 INFO:tasks.workunit.client.1.smithi027.stdout:7/735: chown d0/d11/d1c/d31/d56 164191320 1 2024-04-22T15:50:06.200 INFO:tasks.workunit.client.1.smithi027.stdout:0/800: dread d0/d28/fef [0,4194304] 0 2024-04-22T15:50:06.201 INFO:tasks.workunit.client.1.smithi027.stdout:5/716: getdents d0/d1a/d22/d26/d5d/d99 0 2024-04-22T15:50:06.205 INFO:tasks.workunit.client.1.smithi027.stdout:6/824: dwrite d5/f75 [0,4194304] 0 2024-04-22T15:50:06.205 INFO:tasks.workunit.client.1.smithi027.stdout:0/801: write d0/d34/f80 [1172996,16539] 0 2024-04-22T15:50:06.212 INFO:tasks.workunit.client.1.smithi027.stdout:5/717: truncate d0/d1a/d22/d26/d5d/de3/d3e/fbe 922731 0 2024-04-22T15:50:06.212 INFO:tasks.workunit.client.1.smithi027.stdout:3/806: dwrite d28/fb9 [0,4194304] 0 2024-04-22T15:50:06.212 INFO:tasks.workunit.client.1.smithi027.stdout:4/694: write d5/d11/d1d/d24/f79 [676966,103216] 0 2024-04-22T15:50:06.213 INFO:tasks.workunit.client.1.smithi027.stdout:7/736: mknod d0/d4c/d8b/d72/cf3 0 2024-04-22T15:50:06.219 INFO:tasks.workunit.client.1.smithi027.stdout:0/802: mkdir d0/d2a/d6a/da6/d100 0 2024-04-22T15:50:06.220 INFO:tasks.workunit.client.1.smithi027.stdout:0/803: truncate d0/d2a/d5e/fbd 903177 0 2024-04-22T15:50:06.221 INFO:tasks.workunit.client.1.smithi027.stdout:3/807: dread d28/d5d/d79/fa1 [0,4194304] 0 2024-04-22T15:50:06.223 INFO:tasks.workunit.client.1.smithi027.stdout:5/718: fdatasync d0/df/d10/f33 0 2024-04-22T15:50:06.224 INFO:tasks.workunit.client.1.smithi027.stdout:4/695: dread - d5/d45/d47/f8a zero size 2024-04-22T15:50:06.229 INFO:tasks.workunit.client.1.smithi027.stdout:1/632: dwrite d2/f18 [0,4194304] 0 2024-04-22T15:50:06.244 INFO:tasks.workunit.client.1.smithi027.stdout:0/804: fdatasync d0/d34/d4e/f68 0 2024-04-22T15:50:06.248 INFO:tasks.workunit.client.1.smithi027.stdout:9/687: rmdir d0/d8 39 2024-04-22T15:50:06.248 INFO:tasks.workunit.client.1.smithi027.stdout:7/737: dread d0/d49/f63 [0,4194304] 0 2024-04-22T15:50:06.254 INFO:tasks.workunit.client.1.smithi027.stdout:5/719: creat d0/d1a/d22/d26/d32/d79/dd6/fe9 x:0 0 0 2024-04-22T15:50:06.256 INFO:tasks.workunit.client.1.smithi027.stdout:5/720: dread d0/d1a/d22/d26/d32/f54 [0,4194304] 0 2024-04-22T15:50:06.258 INFO:tasks.workunit.client.1.smithi027.stdout:2/718: rmdir dd/df/d70/d4b/d4e/daf/dd8 0 2024-04-22T15:50:06.259 INFO:tasks.workunit.client.1.smithi027.stdout:3/808: symlink d28/d4b/d52/db2/deb/df3/l103 0 2024-04-22T15:50:06.259 INFO:tasks.workunit.client.1.smithi027.stdout:3/809: readlink d28/l47 0 2024-04-22T15:50:06.263 INFO:tasks.workunit.client.1.smithi027.stdout:0/805: mkdir d0/d2a/d91/da1/d101 0 2024-04-22T15:50:06.265 INFO:tasks.workunit.client.1.smithi027.stdout:9/688: creat d0/dc/d12/d30/d5b/d91/fe0 x:0 0 0 2024-04-22T15:50:06.267 INFO:tasks.workunit.client.1.smithi027.stdout:5/721: unlink d0/d1a/d22/l96 0 2024-04-22T15:50:06.268 INFO:tasks.workunit.client.1.smithi027.stdout:2/719: truncate dd/f4d 1945024 0 2024-04-22T15:50:06.269 INFO:tasks.workunit.client.1.smithi027.stdout:3/810: rmdir d28/d4b/d52/db2/d95/d9c 39 2024-04-22T15:50:06.270 INFO:tasks.workunit.client.1.smithi027.stdout:7/738: symlink d0/lf4 0 2024-04-22T15:50:06.272 INFO:tasks.workunit.client.1.smithi027.stdout:0/806: mkdir d0/d9d/d102 0 2024-04-22T15:50:06.273 INFO:tasks.workunit.client.1.smithi027.stdout:0/807: chown d0/l7f 23914683 1 2024-04-22T15:50:06.274 INFO:tasks.workunit.client.1.smithi027.stdout:6/825: truncate d5/db/fa9 29515 0 2024-04-22T15:50:06.275 INFO:tasks.workunit.client.1.smithi027.stdout:1/633: dread d2/d5f/d64/d81/d3d/d44/d9c/d9f/fac [0,4194304] 0 2024-04-22T15:50:06.275 INFO:tasks.workunit.client.1.smithi027.stdout:3/811: read d28/d4b/f53 [6103665,50156] 0 2024-04-22T15:50:06.277 INFO:tasks.workunit.client.1.smithi027.stdout:7/739: unlink d0/d4c/d8b/d92/da3/cdf 0 2024-04-22T15:50:06.280 INFO:tasks.workunit.client.1.smithi027.stdout:7/740: dread d0/d4c/d8b/d72/db2/f36 [0,4194304] 0 2024-04-22T15:50:06.280 INFO:tasks.workunit.client.1.smithi027.stdout:5/722: symlink d0/df/de/d24/lea 0 2024-04-22T15:50:06.284 INFO:tasks.workunit.client.1.smithi027.stdout:3/812: mkdir d28/d4b/d52/d94/dae/d104 0 2024-04-22T15:50:06.285 INFO:tasks.workunit.client.1.smithi027.stdout:4/696: write d5/f26 [2050002,49848] 0 2024-04-22T15:50:06.287 INFO:tasks.workunit.client.1.smithi027.stdout:7/741: mkdir d0/d4c/d4e/df5 0 2024-04-22T15:50:06.288 INFO:tasks.workunit.client.1.smithi027.stdout:7/742: fdatasync d0/fcc 0 2024-04-22T15:50:06.293 INFO:tasks.workunit.client.1.smithi027.stdout:3/813: mkdir d28/d4b/d52/d94/d105 0 2024-04-22T15:50:06.295 INFO:tasks.workunit.client.1.smithi027.stdout:3/814: read d28/d4b/d52/db2/d95/fc7 [208960,80606] 0 2024-04-22T15:50:06.296 INFO:tasks.workunit.client.1.smithi027.stdout:4/697: symlink d5/de/d2a/l105 0 2024-04-22T15:50:06.309 INFO:tasks.workunit.client.1.smithi027.stdout:1/634: link d2/d1d/f32 d2/d5f/d64/d81/d69/fd4 0 2024-04-22T15:50:06.313 INFO:tasks.workunit.client.1.smithi027.stdout:0/808: write d0/d28/f5b [457786,128427] 0 2024-04-22T15:50:06.315 INFO:tasks.workunit.client.1.smithi027.stdout:9/689: dwrite d0/dc/d12/fb3 [0,4194304] 0 2024-04-22T15:50:06.328 INFO:tasks.workunit.client.1.smithi027.stdout:5/723: dwrite d0/d1a/d22/d26/d5d/de3/d8f/fbd [0,4194304] 0 2024-04-22T15:50:06.337 INFO:tasks.workunit.client.1.smithi027.stdout:0/809: dread d0/d2a/f48 [0,4194304] 0 2024-04-22T15:50:06.338 INFO:tasks.workunit.client.1.smithi027.stdout:1/635: dread d2/f7f [0,4194304] 0 2024-04-22T15:50:06.342 INFO:tasks.workunit.client.1.smithi027.stdout:5/724: creat d0/df/de/feb x:0 0 0 2024-04-22T15:50:06.344 INFO:tasks.workunit.client.1.smithi027.stdout:3/815: write d28/d4b/faf [978217,52067] 0 2024-04-22T15:50:06.352 INFO:tasks.workunit.client.1.smithi027.stdout:4/698: dwrite d5/de/d30/d5d/f99 [0,4194304] 0 2024-04-22T15:50:06.353 INFO:tasks.workunit.client.1.smithi027.stdout:4/699: stat d5/d7/dc6/fdb 0 2024-04-22T15:50:06.354 INFO:tasks.workunit.client.1.smithi027.stdout:9/690: write d0/d64/f7b [1216936,92321] 0 2024-04-22T15:50:06.355 INFO:tasks.workunit.client.1.smithi027.stdout:4/700: chown d5/d45/d47/cc4 793 1 2024-04-22T15:50:06.360 INFO:tasks.workunit.client.1.smithi027.stdout:5/725: dread d0/d1a/d22/d26/d32/f4f [0,4194304] 0 2024-04-22T15:50:06.361 INFO:tasks.workunit.client.1.smithi027.stdout:7/743: link d0/d4c/d8b/d72/db2/f36 d0/d11/d1c/d31/ff6 0 2024-04-22T15:50:06.366 INFO:tasks.workunit.client.1.smithi027.stdout:3/816: symlink d28/d4b/d52/d9a/l106 0 2024-04-22T15:50:06.366 INFO:tasks.workunit.client.1.smithi027.stdout:0/810: write d0/d28/f1f [1221110,97788] 0 2024-04-22T15:50:06.371 INFO:tasks.workunit.client.1.smithi027.stdout:7/744: dwrite d0/d4c/d8b/d72/db2/fd0 [0,4194304] 0 2024-04-22T15:50:06.372 INFO:tasks.workunit.client.1.smithi027.stdout:4/701: unlink d5/d45/d5a/fb0 0 2024-04-22T15:50:06.377 INFO:tasks.workunit.client.1.smithi027.stdout:5/726: mknod d0/d1a/d22/d26/d5d/de3/d3e/d55/cec 0 2024-04-22T15:50:06.383 INFO:tasks.workunit.client.1.smithi027.stdout:7/745: symlink d0/d4c/d8b/d72/lf7 0 2024-04-22T15:50:06.385 INFO:tasks.workunit.client.1.smithi027.stdout:4/702: creat d5/d11/d1d/d54/d8c/f106 x:0 0 0 2024-04-22T15:50:06.385 INFO:tasks.workunit.client.1.smithi027.stdout:1/636: write d2/f75 [2746160,71001] 0 2024-04-22T15:50:06.386 INFO:tasks.workunit.client.1.smithi027.stdout:1/637: write d2/f35 [1407434,40529] 0 2024-04-22T15:50:06.387 INFO:tasks.workunit.client.1.smithi027.stdout:4/703: chown d5/d11/f18 31348 1 2024-04-22T15:50:06.389 INFO:tasks.workunit.client.1.smithi027.stdout:1/638: write d2/d5f/d64/d81/d7e/fa1 [498408,124006] 0 2024-04-22T15:50:06.390 INFO:tasks.workunit.client.1.smithi027.stdout:1/639: chown d2/d5f/d64/d81/d7e 127982 1 2024-04-22T15:50:06.391 INFO:tasks.workunit.client.1.smithi027.stdout:5/727: unlink d0/d1a/d22/d26/d5d/f8b 0 2024-04-22T15:50:06.395 INFO:tasks.workunit.client.1.smithi027.stdout:3/817: mkdir d28/d4b/d52/d107 0 2024-04-22T15:50:06.396 INFO:tasks.workunit.client.1.smithi027.stdout:3/818: stat d28/d4b/d52/d94/d105 0 2024-04-22T15:50:06.399 INFO:tasks.workunit.client.1.smithi027.stdout:4/704: creat d5/de/df2/f107 x:0 0 0 2024-04-22T15:50:06.403 INFO:tasks.workunit.client.1.smithi027.stdout:5/728: unlink d0/d1a/d22/d26/f57 0 2024-04-22T15:50:06.405 INFO:tasks.workunit.client.1.smithi027.stdout:0/811: creat d0/d2a/f103 x:0 0 0 2024-04-22T15:50:06.406 INFO:tasks.workunit.client.1.smithi027.stdout:0/812: write d0/f2e [3987376,39362] 0 2024-04-22T15:50:06.407 INFO:tasks.workunit.client.1.smithi027.stdout:3/819: rmdir d28/d4b/d52/db2/deb 39 2024-04-22T15:50:06.411 INFO:tasks.workunit.client.1.smithi027.stdout:7/746: link d0/d49/d9f/f9b d0/d11/d1c/d31/d8a/ff8 0 2024-04-22T15:50:06.417 INFO:tasks.workunit.client.1.smithi027.stdout:5/729: read - d0/df/de/d24/d62/f7f zero size 2024-04-22T15:50:06.420 INFO:tasks.workunit.client.1.smithi027.stdout:3/820: dread d28/d4b/d52/d94/dae/fcc [0,4194304] 0 2024-04-22T15:50:06.422 INFO:tasks.workunit.client.1.smithi027.stdout:1/640: dwrite d2/f57 [0,4194304] 0 2024-04-22T15:50:06.423 INFO:tasks.workunit.client.1.smithi027.stdout:1/641: read d2/d36/d74/f79 [1457411,92031] 0 2024-04-22T15:50:06.424 INFO:tasks.workunit.client.1.smithi027.stdout:7/747: symlink d0/lf9 0 2024-04-22T15:50:06.430 INFO:tasks.workunit.client.1.smithi027.stdout:0/813: rmdir d0/d2a/d91/da1/d101 0 2024-04-22T15:50:06.434 INFO:tasks.workunit.client.1.smithi027.stdout:1/642: creat d2/d36/daa/fd5 x:0 0 0 2024-04-22T15:50:06.437 INFO:tasks.workunit.client.1.smithi027.stdout:4/705: getdents d5/d11/d1d 0 2024-04-22T15:50:06.438 INFO:tasks.workunit.client.1.smithi027.stdout:5/730: creat d0/d1a/d22/d26/d5d/de3/d3e/dd5/fed x:0 0 0 2024-04-22T15:50:06.441 INFO:tasks.workunit.client.1.smithi027.stdout:3/821: mknod d28/d4b/d52/db2/deb/c108 0 2024-04-22T15:50:06.442 INFO:tasks.workunit.client.1.smithi027.stdout:3/822: chown d28/d4b/faf 23 1 2024-04-22T15:50:06.443 INFO:tasks.workunit.client.1.smithi027.stdout:7/748: mkdir d0/d11/dfa 0 2024-04-22T15:50:06.445 INFO:tasks.workunit.client.1.smithi027.stdout:4/706: rename d5/d45/d47/f4a to d5/d11/d6c/f108 0 2024-04-22T15:50:06.446 INFO:tasks.workunit.client.1.smithi027.stdout:5/731: mknod d0/d1a/d22/cee 0 2024-04-22T15:50:06.449 INFO:tasks.workunit.client.1.smithi027.stdout:3/823: rmdir d28/d4b/d52/d94/dae 39 2024-04-22T15:50:06.456 INFO:tasks.workunit.client.1.smithi027.stdout:4/707: dread - d5/d11/fe5 zero size 2024-04-22T15:50:06.458 INFO:tasks.workunit.client.1.smithi027.stdout:0/814: symlink d0/d2a/d58/dee/l104 0 2024-04-22T15:50:06.461 INFO:tasks.workunit.client.1.smithi027.stdout:5/732: dread d0/df/d10/f33 [0,4194304] 0 2024-04-22T15:50:06.461 INFO:tasks.workunit.client.1.smithi027.stdout:7/749: symlink d0/d4c/de5/lfb 0 2024-04-22T15:50:06.463 INFO:tasks.workunit.client.1.smithi027.stdout:3/824: rmdir d28/d4b/d52/db2/d95 39 2024-04-22T15:50:06.465 INFO:tasks.workunit.client.1.smithi027.stdout:3/825: truncate d28/d5d/d79/fcd 263538 0 2024-04-22T15:50:06.465 INFO:tasks.workunit.client.1.smithi027.stdout:3/826: stat d28/d4b/d52/db2/c6d 0 2024-04-22T15:50:06.469 INFO:tasks.workunit.client.1.smithi027.stdout:0/815: mknod d0/d2a/d58/d66/ded/c105 0 2024-04-22T15:50:06.471 INFO:tasks.workunit.client.1.smithi027.stdout:7/750: rmdir d0/d3f 39 2024-04-22T15:50:06.472 INFO:tasks.workunit.client.1.smithi027.stdout:4/708: dread d5/f26 [0,4194304] 0 2024-04-22T15:50:06.474 INFO:tasks.workunit.client.1.smithi027.stdout:5/733: unlink d0/d1a/d22/d26/d5d/de3/d3e/fc8 0 2024-04-22T15:50:06.476 INFO:tasks.workunit.client.1.smithi027.stdout:1/643: link d2/d5f/d64/d81/d3d/d44/f73 d2/d5f/fd6 0 2024-04-22T15:50:06.478 INFO:tasks.workunit.client.1.smithi027.stdout:7/751: rmdir d0/d11/d1c/d2c/d96 39 2024-04-22T15:50:06.479 INFO:tasks.workunit.client.1.smithi027.stdout:5/734: mkdir d0/df/de/d24/d62/def 0 2024-04-22T15:50:06.486 INFO:tasks.workunit.client.1.smithi027.stdout:5/735: dwrite d0/d1a/d22/d26/d5d/de3/d8f/fbd [0,4194304] 0 2024-04-22T15:50:06.489 INFO:tasks.workunit.client.1.smithi027.stdout:4/709: symlink d5/dea/l109 0 2024-04-22T15:50:06.492 INFO:tasks.workunit.client.1.smithi027.stdout:1/644: symlink d2/d5f/d64/d81/d3d/da9/ld7 0 2024-04-22T15:50:06.496 INFO:tasks.workunit.client.1.smithi027.stdout:5/736: dread d0/df/d3d/f51 [0,4194304] 0 2024-04-22T15:50:06.497 INFO:tasks.workunit.client.1.smithi027.stdout:4/710: dread - d5/d11/d6c/f9b zero size 2024-04-22T15:50:06.498 INFO:tasks.workunit.client.1.smithi027.stdout:0/816: truncate d0/d28/f1f 137341 0 2024-04-22T15:50:06.499 INFO:tasks.workunit.client.1.smithi027.stdout:1/645: stat d2/d5f/d64/d81/d3d/d44/c6f 0 2024-04-22T15:50:06.502 INFO:tasks.workunit.client.1.smithi027.stdout:4/711: creat d5/d45/d5a/f10a x:0 0 0 2024-04-22T15:50:06.505 INFO:tasks.workunit.client.1.smithi027.stdout:0/817: unlink d0/da3/dae/fc4 0 2024-04-22T15:50:06.506 INFO:tasks.workunit.client.1.smithi027.stdout:1/646: creat d2/d5f/d64/d81/d3d/d44/fd8 x:0 0 0 2024-04-22T15:50:06.514 INFO:tasks.workunit.client.1.smithi027.stdout:0/818: mknod d0/d2a/d6a/d6c/d83/d9e/c106 0 2024-04-22T15:50:06.520 INFO:tasks.workunit.client.1.smithi027.stdout:0/819: symlink d0/d2a/de0/l107 0 2024-04-22T15:50:06.526 INFO:tasks.workunit.client.1.smithi027.stdout:5/737: truncate d0/df/de/d40/f6a 4136862 0 2024-04-22T15:50:06.528 INFO:tasks.workunit.client.1.smithi027.stdout:0/820: creat d0/d28/d37/f108 x:0 0 0 2024-04-22T15:50:06.529 INFO:tasks.workunit.client.1.smithi027.stdout:5/738: rmdir d0/d97 39 2024-04-22T15:50:06.536 INFO:tasks.workunit.client.1.smithi027.stdout:4/712: dwrite d5/f48 [0,4194304] 0 2024-04-22T15:50:06.538 INFO:tasks.workunit.client.1.smithi027.stdout:5/739: dwrite d0/df/d10/f33 [0,4194304] 0 2024-04-22T15:50:06.549 INFO:tasks.workunit.client.1.smithi027.stdout:4/713: mkdir d5/d11/d6c/da3/d10b 0 2024-04-22T15:50:06.552 INFO:tasks.workunit.client.1.smithi027.stdout:5/740: write d0/d1a/d22/d26/d32/d79/fa7 [473033,21621] 0 2024-04-22T15:50:06.553 INFO:tasks.workunit.client.1.smithi027.stdout:5/741: readlink d0/df/de/l3f 0 2024-04-22T15:50:06.553 INFO:tasks.workunit.client.1.smithi027.stdout:5/742: chown d0/df/d3d 314 1 2024-04-22T15:50:06.554 INFO:tasks.workunit.client.1.smithi027.stdout:0/821: write d0/d2a/f48 [3516984,43754] 0 2024-04-22T15:50:06.560 INFO:tasks.workunit.client.1.smithi027.stdout:5/743: mkdir d0/d97/dbf/df0 0 2024-04-22T15:50:06.569 INFO:tasks.workunit.client.1.smithi027.stdout:5/744: dwrite d0/d1a/d22/d26/d32/f70 [0,4194304] 0 2024-04-22T15:50:06.569 INFO:tasks.workunit.client.1.smithi027.stdout:4/714: dread d5/f13 [0,4194304] 0 2024-04-22T15:50:06.571 INFO:tasks.workunit.client.1.smithi027.stdout:0/822: chown d0/d28/f1f 15539745 1 2024-04-22T15:50:06.575 INFO:tasks.workunit.client.1.smithi027.stdout:5/745: write d0/df/de/d24/f2e [1985286,51272] 0 2024-04-22T15:50:06.579 INFO:tasks.workunit.client.1.smithi027.stdout:4/715: write d5/d11/f7a [201665,62221] 0 2024-04-22T15:50:06.581 INFO:tasks.workunit.client.1.smithi027.stdout:4/716: dread - d5/de/df2/f107 zero size 2024-04-22T15:50:06.582 INFO:tasks.workunit.client.1.smithi027.stdout:4/717: dread - d5/da8/fef zero size 2024-04-22T15:50:06.582 INFO:tasks.workunit.client.1.smithi027.stdout:5/746: symlink d0/d1a/d22/d26/d5d/de3/d3e/lf1 0 2024-04-22T15:50:06.584 INFO:tasks.workunit.client.1.smithi027.stdout:4/718: chown d5/d11/d1d/d7b/de9 43 1 2024-04-22T15:50:06.585 INFO:tasks.workunit.client.1.smithi027.stdout:0/823: dwrite d0/d2a/f2b [0,4194304] 0 2024-04-22T15:50:06.589 INFO:tasks.workunit.client.1.smithi027.stdout:5/747: creat d0/d1a/d22/d26/d5d/de3/d3e/dd5/ff2 x:0 0 0 2024-04-22T15:50:06.594 INFO:tasks.workunit.client.1.smithi027.stdout:4/719: dwrite d5/de/df2/fec [0,4194304] 0 2024-04-22T15:50:06.595 INFO:tasks.workunit.client.1.smithi027.stdout:0/824: unlink d0/d2a/d38/f65 0 2024-04-22T15:50:06.599 INFO:tasks.workunit.client.1.smithi027.stdout:0/825: write d0/d2a/d38/fad [1036438,128578] 0 2024-04-22T15:50:06.600 INFO:tasks.workunit.client.1.smithi027.stdout:4/720: dread - d5/de/d2a/d58/fd4 zero size 2024-04-22T15:50:06.604 INFO:tasks.workunit.client.1.smithi027.stdout:0/826: creat d0/d2a/d38/f109 x:0 0 0 2024-04-22T15:50:06.608 INFO:tasks.workunit.client.1.smithi027.stdout:0/827: chown d0/d2a/d38/lda 0 1 2024-04-22T15:50:06.610 INFO:tasks.workunit.client.1.smithi027.stdout:0/828: unlink d0/d2a/d38/f4a 0 2024-04-22T15:50:06.611 INFO:tasks.workunit.client.1.smithi027.stdout:0/829: rmdir d0/d2a/d5e 39 2024-04-22T15:50:06.614 INFO:tasks.workunit.client.1.smithi027.stdout:0/830: chown d0/d2a/d5e/ld1 114310123 1 2024-04-22T15:50:06.614 INFO:tasks.workunit.client.1.smithi027.stdout:5/748: write d0/df/d10/f60 [463920,22813] 0 2024-04-22T15:50:06.620 INFO:tasks.workunit.client.1.smithi027.stdout:0/831: symlink d0/d34/d4e/d85/db3/l10a 0 2024-04-22T15:50:06.622 INFO:tasks.workunit.client.1.smithi027.stdout:4/721: dwrite d5/de/d2a/d58/fbf [0,4194304] 0 2024-04-22T15:50:06.624 INFO:tasks.workunit.client.1.smithi027.stdout:5/749: dread d0/d1a/d22/d26/d32/f46 [0,4194304] 0 2024-04-22T15:50:06.625 INFO:tasks.workunit.client.1.smithi027.stdout:0/832: mkdir d0/d2a/de0/d10b 0 2024-04-22T15:50:06.628 INFO:tasks.workunit.client.1.smithi027.stdout:4/722: fdatasync d5/d45/d47/fbe 0 2024-04-22T15:50:06.630 INFO:tasks.workunit.client.1.smithi027.stdout:4/723: dread d5/de/d2a/f9a [0,4194304] 0 2024-04-22T15:50:06.633 INFO:tasks.workunit.client.1.smithi027.stdout:0/833: rename d0/d28/le2 to d0/da3/dae/l10c 0 2024-04-22T15:50:06.639 INFO:tasks.workunit.client.1.smithi027.stdout:5/750: creat d0/d97/dd7/ff3 x:0 0 0 2024-04-22T15:50:06.639 INFO:tasks.workunit.client.1.smithi027.stdout:0/834: dwrite d0/d2a/d38/f109 [0,4194304] 0 2024-04-22T15:50:06.646 INFO:tasks.workunit.client.1.smithi027.stdout:0/835: unlink d0/f1 0 2024-04-22T15:50:06.651 INFO:tasks.workunit.client.1.smithi027.stdout:0/836: stat d0/d28/d63/lbc 0 2024-04-22T15:50:06.656 INFO:tasks.workunit.client.1.smithi027.stdout:4/724: dwrite d5/d45/d47/fac [0,4194304] 0 2024-04-22T15:50:06.673 INFO:tasks.workunit.client.1.smithi027.stdout:4/725: dread d5/d45/f7d [0,4194304] 0 2024-04-22T15:50:06.685 INFO:tasks.workunit.client.1.smithi027.stdout:0/837: mkdir d0/da3/dae/df9/d10d 0 2024-04-22T15:50:06.689 INFO:tasks.workunit.client.1.smithi027.stdout:0/838: symlink d0/d9d/db7/l10e 0 2024-04-22T15:50:06.697 INFO:tasks.workunit.client.1.smithi027.stdout:0/839: dwrite d0/d2a/d5e/d8b/f9f [4194304,4194304] 0 2024-04-22T15:50:06.698 INFO:tasks.workunit.client.1.smithi027.stdout:0/840: write d0/d28/d63/f8d [716058,81430] 0 2024-04-22T15:50:06.705 INFO:tasks.workunit.client.1.smithi027.stdout:4/726: dwrite d5/d45/d47/f76 [0,4194304] 0 2024-04-22T15:50:06.708 INFO:tasks.workunit.client.1.smithi027.stdout:0/841: getdents d0/d28/d37 0 2024-04-22T15:50:06.712 INFO:tasks.workunit.client.1.smithi027.stdout:0/842: mknod d0/d2a/d6a/d6c/c10f 0 2024-04-22T15:50:06.715 INFO:tasks.workunit.client.1.smithi027.stdout:4/727: dwrite d5/d11/d1d/f5e [0,4194304] 0 2024-04-22T15:50:06.716 INFO:tasks.workunit.client.1.smithi027.stdout:4/728: stat d5/d45/c2e 0 2024-04-22T15:50:06.718 INFO:tasks.workunit.client.1.smithi027.stdout:4/729: write d5/d45/d47/f76 [2242287,61736] 0 2024-04-22T15:50:06.724 INFO:tasks.workunit.client.1.smithi027.stdout:4/730: dread d5/da8/fb8 [0,4194304] 0 2024-04-22T15:50:06.733 INFO:tasks.workunit.client.1.smithi027.stdout:0/843: dread d0/d34/f71 [0,4194304] 0 2024-04-22T15:50:06.733 INFO:tasks.workunit.client.1.smithi027.stdout:0/844: stat d0/d9d/db7 0 2024-04-22T15:50:06.734 INFO:tasks.workunit.client.1.smithi027.stdout:0/845: fdatasync d0/d2a/f48 0 2024-04-22T15:50:06.735 INFO:tasks.workunit.client.1.smithi027.stdout:0/846: readlink d0/d2a/d6a/l78 0 2024-04-22T15:50:06.735 INFO:tasks.workunit.client.1.smithi027.stdout:0/847: chown d0/c69 21794 1 2024-04-22T15:50:06.737 INFO:tasks.workunit.client.1.smithi027.stdout:0/848: write d0/d28/f1c [1394606,22396] 0 2024-04-22T15:50:06.739 INFO:tasks.workunit.client.1.smithi027.stdout:0/849: read d0/d34/d4e/fab [707981,71811] 0 2024-04-22T15:50:06.746 INFO:tasks.workunit.client.1.smithi027.stdout:0/850: write d0/d34/d4e/f68 [223011,111510] 0 2024-04-22T15:50:06.748 INFO:tasks.workunit.client.1.smithi027.stdout:4/731: dread d5/de/d2a/d58/f68 [0,4194304] 0 2024-04-22T15:50:06.753 INFO:tasks.workunit.client.1.smithi027.stdout:0/851: dwrite d0/d28/f1c [0,4194304] 0 2024-04-22T15:50:06.753 INFO:tasks.workunit.client.1.smithi027.stdout:4/732: rename d5/de/fab to d5/d45/d5a/f10c 0 2024-04-22T15:50:06.755 INFO:tasks.workunit.client.1.smithi027.stdout:4/733: chown d5/de/c3b 14088209 1 2024-04-22T15:50:06.755 INFO:tasks.workunit.client.1.smithi027.stdout:0/852: dread - d0/d34/ff8 zero size 2024-04-22T15:50:06.755 INFO:tasks.workunit.client.1.smithi027.stdout:4/734: dread - d5/d11/fe5 zero size 2024-04-22T15:50:06.757 INFO:tasks.workunit.client.1.smithi027.stdout:0/853: creat d0/d34/d4e/d85/db3/f110 x:0 0 0 2024-04-22T15:50:06.764 INFO:tasks.workunit.client.1.smithi027.stdout:4/735: creat d5/de/f10d x:0 0 0 2024-04-22T15:50:06.771 INFO:tasks.workunit.client.1.smithi027.stdout:4/736: dwrite d5/d11/da6/fd6 [4194304,4194304] 0 2024-04-22T15:50:06.778 INFO:tasks.workunit.client.1.smithi027.stdout:0/854: dread d0/f12 [0,4194304] 0 2024-04-22T15:50:06.782 INFO:tasks.workunit.client.1.smithi027.stdout:0/855: rename d0/d2a/fa7 to d0/da3/db6/f111 0 2024-04-22T15:50:06.783 INFO:tasks.workunit.client.1.smithi027.stdout:0/856: write d0/d34/d4e/d85/fe8 [1006775,27870] 0 2024-04-22T15:50:06.791 INFO:tasks.workunit.client.1.smithi027.stdout:4/737: write d5/d42/f9e [4465750,24708] 0 2024-04-22T15:50:06.794 INFO:tasks.workunit.client.1.smithi027.stdout:0/857: dwrite d0/da3/db6/f111 [0,4194304] 0 2024-04-22T15:50:06.799 INFO:tasks.workunit.client.1.smithi027.stdout:4/738: write d5/de/d30/d5d/f99 [426759,86078] 0 2024-04-22T15:50:06.800 INFO:tasks.workunit.client.1.smithi027.stdout:4/739: fdatasync d5/d45/d5a/f96 0 2024-04-22T15:50:06.805 INFO:tasks.workunit.client.1.smithi027.stdout:4/740: creat d5/da8/f10e x:0 0 0 2024-04-22T15:50:06.806 INFO:tasks.workunit.client.1.smithi027.stdout:4/741: truncate d5/de/dc5/ffe 117579 0 2024-04-22T15:50:06.808 INFO:tasks.workunit.client.1.smithi027.stdout:4/742: mkdir d5/d42/d10f 0 2024-04-22T15:50:06.816 INFO:tasks.workunit.client.1.smithi027.stdout:0/858: dread d0/d28/d5f/f73 [0,4194304] 0 2024-04-22T15:50:06.819 INFO:tasks.workunit.client.1.smithi027.stdout:0/859: mknod d0/d28/d37/c112 0 2024-04-22T15:50:06.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:06 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:06.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:06 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:06.840 INFO:tasks.workunit.client.1.smithi027.stdout:0/860: chown d0/d28/d37/cfe 2 1 2024-04-22T15:50:06.843 INFO:tasks.workunit.client.1.smithi027.stdout:0/861: readlink d0/d34/d59/lca 0 2024-04-22T15:50:06.845 INFO:tasks.workunit.client.1.smithi027.stdout:0/862: dread - d0/d2a/d5e/d8b/fb5 zero size 2024-04-22T15:50:06.848 INFO:tasks.workunit.client.1.smithi027.stdout:0/863: symlink d0/d2a/d6a/d6c/d83/l113 0 2024-04-22T15:50:06.851 INFO:tasks.workunit.client.1.smithi027.stdout:0/864: fsync d0/fc 0 2024-04-22T15:50:06.860 INFO:tasks.workunit.client.1.smithi027.stdout:0/865: rename d0/d34/f5a to d0/d34/d4e/f114 0 2024-04-22T15:50:06.866 INFO:tasks.workunit.client.1.smithi027.stdout:0/866: dread d0/d2a/d38/fad [0,4194304] 0 2024-04-22T15:50:06.866 INFO:tasks.workunit.client.1.smithi027.stdout:0/867: fdatasync d0/d34/d59/f5c 0 2024-04-22T15:50:06.896 INFO:tasks.workunit.client.1.smithi027.stdout:0/868: dwrite d0/d28/d37/f40 [0,4194304] 0 2024-04-22T15:50:06.903 INFO:tasks.workunit.client.1.smithi027.stdout:0/869: rename d0/da3/dae/l10c to d0/da3/db6/l115 0 2024-04-22T15:50:06.905 INFO:tasks.workunit.client.1.smithi027.stdout:0/870: rmdir d0/da3 39 2024-04-22T15:50:06.910 INFO:tasks.workunit.client.1.smithi027.stdout:0/871: dwrite d0/d9d/fc0 [0,4194304] 0 2024-04-22T15:50:06.911 INFO:tasks.workunit.client.1.smithi027.stdout:0/872: stat d0/d9d/db7/c99 0 2024-04-22T15:50:06.914 INFO:tasks.workunit.client.1.smithi027.stdout:0/873: mkdir d0/d28/d63/d116 0 2024-04-22T15:50:06.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:06 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:06.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:06 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:06.926 INFO:tasks.workunit.client.1.smithi027.stdout:0/874: dwrite d0/d28/f1f [0,4194304] 0 2024-04-22T15:50:06.933 INFO:tasks.workunit.client.1.smithi027.stdout:0/875: dwrite d0/d2a/f61 [4194304,4194304] 0 2024-04-22T15:50:06.941 INFO:tasks.workunit.client.1.smithi027.stdout:0/876: dread d0/d2a/d6a/fcb [0,4194304] 0 2024-04-22T15:50:06.945 INFO:tasks.workunit.client.1.smithi027.stdout:0/877: creat d0/d2a/d5e/f117 x:0 0 0 2024-04-22T15:50:06.951 INFO:tasks.workunit.client.1.smithi027.stdout:0/878: getdents d0/d34 0 2024-04-22T15:50:06.951 INFO:tasks.workunit.client.1.smithi027.stdout:0/879: chown d0/d2a/de0/l107 52355979 1 2024-04-22T15:50:06.954 INFO:tasks.workunit.client.1.smithi027.stdout:0/880: dread d0/f12 [0,4194304] 0 2024-04-22T15:50:06.957 INFO:tasks.workunit.client.1.smithi027.stdout:0/881: dread d0/fa8 [0,4194304] 0 2024-04-22T15:50:06.959 INFO:tasks.workunit.client.1.smithi027.stdout:0/882: creat d0/d34/d4e/d85/f118 x:0 0 0 2024-04-22T15:50:06.966 INFO:tasks.workunit.client.1.smithi027.stdout:0/883: dwrite d0/d2a/f48 [0,4194304] 0 2024-04-22T15:50:06.967 INFO:tasks.workunit.client.1.smithi027.stdout:0/884: stat d0/d34/d4e/d85/ddd/lf7 0 2024-04-22T15:50:06.973 INFO:tasks.workunit.client.1.smithi027.stdout:0/885: rmdir d0/d2a/d6a/da6/d100 0 2024-04-22T15:50:06.977 INFO:tasks.workunit.client.1.smithi027.stdout:0/886: rmdir d0/d2a/d6a/d6c 39 2024-04-22T15:50:06.982 INFO:tasks.workunit.client.1.smithi027.stdout:0/887: mkdir d0/d2a/d58/d119 0 2024-04-22T15:50:06.985 INFO:tasks.workunit.client.1.smithi027.stdout:0/888: rename d0/d2a/d6a/l97 to d0/d34/d4e/d85/l11a 0 2024-04-22T15:50:06.986 INFO:tasks.workunit.client.1.smithi027.stdout:0/889: mknod d0/d34/c11b 0 2024-04-22T15:50:06.988 INFO:tasks.workunit.client.1.smithi027.stdout:0/890: fdatasync d0/d9d/f9a 0 2024-04-22T15:50:07.017 INFO:tasks.workunit.client.1.smithi027.stdout:0/891: readlink d0/d34/d4e/d85/ddd/lf7 0 2024-04-22T15:50:07.023 INFO:tasks.workunit.client.1.smithi027.stdout:0/892: dwrite d0/d28/fef [4194304,4194304] 0 2024-04-22T15:50:07.032 INFO:tasks.workunit.client.1.smithi027.stdout:0/893: dwrite d0/d34/d4e/d85/db3/fb8 [0,4194304] 0 2024-04-22T15:50:07.039 INFO:tasks.workunit.client.1.smithi027.stdout:0/894: mknod d0/d2a/d6a/d8e/c11c 0 2024-04-22T15:50:07.044 INFO:tasks.workunit.client.1.smithi027.stdout:0/895: rename d0/d34/d4e/d85/db3/cbe to d0/d2a/d58/d66/de4/c11d 0 2024-04-22T15:50:07.049 INFO:tasks.workunit.client.1.smithi027.stdout:0/896: creat d0/da3/dae/f11e x:0 0 0 2024-04-22T15:50:07.049 INFO:tasks.workunit.client.1.smithi027.stdout:0/897: dread - d0/d2a/d58/d66/fd9 zero size 2024-04-22T15:50:07.050 INFO:tasks.workunit.client.1.smithi027.stdout:0/898: chown d0/d28/d63/lbc 50270 1 2024-04-22T15:50:07.054 INFO:tasks.workunit.client.1.smithi027.stdout:0/899: chown d0/d28/d37/l46 25803767 1 2024-04-22T15:50:07.056 INFO:tasks.workunit.client.1.smithi027.stdout:0/900: mknod d0/d28/d63/d67/c11f 0 2024-04-22T15:50:07.070 INFO:tasks.workunit.client.1.smithi027.stdout:0/901: dread d0/d2a/d38/f92 [0,4194304] 0 2024-04-22T15:50:07.072 INFO:tasks.workunit.client.1.smithi027.stdout:0/902: creat d0/d2a/d6a/da6/f120 x:0 0 0 2024-04-22T15:50:07.075 INFO:tasks.workunit.client.1.smithi027.stdout:0/903: dread d0/d2a/f2b [4194304,4194304] 0 2024-04-22T15:50:07.075 INFO:tasks.workunit.client.1.smithi027.stdout:0/904: fsync d0/d28/d5f/f73 0 2024-04-22T15:50:07.077 INFO:tasks.workunit.client.1.smithi027.stdout:0/905: mknod d0/d9d/d102/c121 0 2024-04-22T15:50:07.078 INFO:tasks.workunit.client.1.smithi027.stdout:0/906: write d0/f2e [2354089,78563] 0 2024-04-22T15:50:07.081 INFO:tasks.workunit.client.1.smithi027.stdout:0/907: symlink d0/d34/d59/l122 0 2024-04-22T15:50:07.100 INFO:tasks.workunit.client.1.smithi027.stdout:0/908: write d0/d2a/d5e/d8b/fb5 [167960,72456] 0 2024-04-22T15:50:07.102 INFO:tasks.workunit.client.1.smithi027.stdout:0/909: creat d0/da3/dae/f123 x:0 0 0 2024-04-22T15:50:07.170 INFO:tasks.workunit.client.1.smithi027.stdout:8/659: sync 2024-04-22T15:50:07.177 INFO:tasks.workunit.client.1.smithi027.stdout:8/660: link d1/d3/d1c/d43/d5f/fa4 d1/d3/d36/d57/fe1 0 2024-04-22T15:50:07.190 INFO:tasks.workunit.client.1.smithi027.stdout:8/661: dwrite d1/d3/f1a [4194304,4194304] 0 2024-04-22T15:50:07.191 INFO:tasks.workunit.client.1.smithi027.stdout:8/662: fsync d1/d1e/d47/d53/fc2 0 2024-04-22T15:50:07.192 INFO:tasks.workunit.client.1.smithi027.stdout:8/663: dread - d1/d1e/d47/d53/fdc zero size 2024-04-22T15:50:07.295 INFO:tasks.workunit.client.1.smithi027.stdout:8/664: symlink d1/d3/d1c/d56/le2 0 2024-04-22T15:50:07.296 INFO:tasks.workunit.client.1.smithi027.stdout:8/665: write d1/d3/d1c/d28/d96/fa0 [1117043,28541] 0 2024-04-22T15:50:07.299 INFO:tasks.workunit.client.1.smithi027.stdout:8/666: symlink d1/d3/d1c/d56/dc9/le3 0 2024-04-22T15:50:07.303 INFO:tasks.workunit.client.1.smithi027.stdout:8/667: rename d1/d3/d1c/d49/d94/fae to d1/d3/d1c/d32/d41/fe4 0 2024-04-22T15:50:07.307 INFO:tasks.workunit.client.1.smithi027.stdout:8/668: creat d1/d3/d1c/fe5 x:0 0 0 2024-04-22T15:50:07.310 INFO:tasks.workunit.client.1.smithi027.stdout:8/669: creat d1/d3/d1c/d49/fe6 x:0 0 0 2024-04-22T15:50:07.311 INFO:tasks.workunit.client.1.smithi027.stdout:8/670: readlink d1/d3/d36/l3a 0 2024-04-22T15:50:07.315 INFO:tasks.workunit.client.1.smithi027.stdout:8/671: mknod d1/d3/d1c/d32/d41/d52/d61/ce7 0 2024-04-22T15:50:07.316 INFO:tasks.workunit.client.1.smithi027.stdout:8/672: fdatasync d1/d1e/d47/d53/fdc 0 2024-04-22T15:50:07.320 INFO:tasks.workunit.client.1.smithi027.stdout:8/673: creat d1/d1e/d47/d53/fe8 x:0 0 0 2024-04-22T15:50:07.335 INFO:tasks.workunit.client.1.smithi027.stdout:8/674: truncate d1/d1e/d47/d53/f77 2464418 0 2024-04-22T15:50:07.336 INFO:tasks.workunit.client.1.smithi027.stdout:2/720: sync 2024-04-22T15:50:07.340 INFO:tasks.workunit.client.1.smithi027.stdout:2/721: truncate dd/df/d1b/d1f/f74 664432 0 2024-04-22T15:50:07.340 INFO:tasks.workunit.client.1.smithi027.stdout:2/722: chown dd/df/d1b/d96 4261 1 2024-04-22T15:50:07.342 INFO:tasks.workunit.client.1.smithi027.stdout:2/723: truncate dd/df/d1b/f7a 1230979 0 2024-04-22T15:50:07.343 INFO:tasks.workunit.client.1.smithi027.stdout:2/724: chown dd/df/d70/d4b 7288 1 2024-04-22T15:50:07.343 INFO:tasks.workunit.client.1.smithi027.stdout:8/675: link d1/d1e/d47/d53/fc2 d1/d3/d1c/d43/d5f/fe9 0 2024-04-22T15:50:07.345 INFO:tasks.workunit.client.1.smithi027.stdout:8/676: dread - d1/d3/d1c/d43/d5f/fe9 zero size 2024-04-22T15:50:07.350 INFO:tasks.workunit.client.1.smithi027.stdout:2/725: creat dd/df/d70/d4b/d4e/daf/fe0 x:0 0 0 2024-04-22T15:50:07.352 INFO:tasks.workunit.client.1.smithi027.stdout:2/726: dread - dd/df/d70/d4b/d4e/daf/fb9 zero size 2024-04-22T15:50:07.354 INFO:tasks.workunit.client.1.smithi027.stdout:8/677: unlink d1/d3/d1c/d49/c5a 0 2024-04-22T15:50:07.355 INFO:tasks.workunit.client.1.smithi027.stdout:8/678: chown d1/d3/d1c/d43/f71 278602 1 2024-04-22T15:50:07.357 INFO:tasks.workunit.client.1.smithi027.stdout:2/727: creat dd/df/d70/fe1 x:0 0 0 2024-04-22T15:50:07.363 INFO:tasks.workunit.client.1.smithi027.stdout:8/679: mknod d1/d3/d1c/d56/cea 0 2024-04-22T15:50:07.366 INFO:tasks.workunit.client.1.smithi027.stdout:8/680: mkdir d1/d1e/deb 0 2024-04-22T15:50:07.370 INFO:tasks.workunit.client.1.smithi027.stdout:8/681: write d1/d3/fa [3566059,1706] 0 2024-04-22T15:50:07.372 INFO:tasks.workunit.client.1.smithi027.stdout:8/682: chown d1/d3/d1c/d28/d96/fa0 373582 1 2024-04-22T15:50:07.374 INFO:tasks.workunit.client.1.smithi027.stdout:8/683: read d1/d1e/f2f [718597,60621] 0 2024-04-22T15:50:07.375 INFO:tasks.workunit.client.1.smithi027.stdout:2/728: dread dd/df/d1b/d38/d60/d75/fb3 [0,4194304] 0 2024-04-22T15:50:07.376 INFO:tasks.workunit.client.1.smithi027.stdout:2/729: dread - dd/df/d1b/d1f/f30 zero size 2024-04-22T15:50:07.379 INFO:tasks.workunit.client.1.smithi027.stdout:8/684: unlink d1/d3/d1c/d32/fca 0 2024-04-22T15:50:07.382 INFO:tasks.workunit.client.1.smithi027.stdout:2/730: symlink dd/d47/le2 0 2024-04-22T15:50:07.394 INFO:tasks.workunit.client.1.smithi027.stdout:8/685: dread d1/d3/d1c/d43/d82/f84 [0,4194304] 0 2024-04-22T15:50:07.397 INFO:tasks.workunit.client.1.smithi027.stdout:8/686: mkdir d1/d3/d1c/d32/d41/d52/dec 0 2024-04-22T15:50:07.400 INFO:tasks.workunit.client.1.smithi027.stdout:2/731: dwrite dd/df/d1b/d38/d60/f80 [0,4194304] 0 2024-04-22T15:50:07.400 INFO:tasks.workunit.client.1.smithi027.stdout:2/732: chown dd/l3a 9228 1 2024-04-22T15:50:07.401 INFO:tasks.workunit.client.1.smithi027.stdout:2/733: stat dd/dae/dd4/ddb 0 2024-04-22T15:50:07.401 INFO:tasks.workunit.client.1.smithi027.stdout:6/826: sync 2024-04-22T15:50:07.402 INFO:tasks.workunit.client.1.smithi027.stdout:8/687: creat d1/d3/d1c/d32/d41/fed x:0 0 0 2024-04-22T15:50:07.406 INFO:tasks.workunit.client.1.smithi027.stdout:8/688: creat d1/d3/d1c/d43/d82/fee x:0 0 0 2024-04-22T15:50:07.408 INFO:tasks.workunit.client.1.smithi027.stdout:8/689: dread - d1/d3/d36/d88/f97 zero size 2024-04-22T15:50:07.409 INFO:tasks.workunit.client.1.smithi027.stdout:8/690: chown d1/d1e/d47/d9d/cc0 33354380 1 2024-04-22T15:50:07.410 INFO:tasks.workunit.client.1.smithi027.stdout:6/827: link d5/db/d2e/d59/df0/f11a d5/dc/d99/f11f 0 2024-04-22T15:50:07.412 INFO:tasks.workunit.client.1.smithi027.stdout:8/691: stat d1/d3/d1c/d28/d42/lcd 0 2024-04-22T15:50:07.413 INFO:tasks.workunit.client.1.smithi027.stdout:6/828: dread d5/dc/d20/f3f [4194304,4194304] 0 2024-04-22T15:50:07.415 INFO:tasks.workunit.client.1.smithi027.stdout:6/829: write d5/f7a [2272441,100101] 0 2024-04-22T15:50:07.416 INFO:tasks.workunit.client.1.smithi027.stdout:6/830: chown d5/dc/dd/d79/d7b/d5d/f70 778 1 2024-04-22T15:50:07.418 INFO:tasks.workunit.client.1.smithi027.stdout:2/734: dread dd/d43/f55 [0,4194304] 0 2024-04-22T15:50:07.419 INFO:tasks.workunit.client.1.smithi027.stdout:8/692: getdents d1/d3/d1c/d49/d94 0 2024-04-22T15:50:07.423 INFO:tasks.workunit.client.1.smithi027.stdout:2/735: dread dd/df/d1b/d38/d60/f80 [0,4194304] 0 2024-04-22T15:50:07.425 INFO:tasks.workunit.client.1.smithi027.stdout:2/736: write dd/df/d70/fe1 [322035,28022] 0 2024-04-22T15:50:07.426 INFO:tasks.workunit.client.1.smithi027.stdout:8/693: unlink d1/d3/d1c/d32/f3d 0 2024-04-22T15:50:07.438 INFO:tasks.workunit.client.1.smithi027.stdout:2/737: creat dd/df/d70/d4b/d52/d9a/da4/fe3 x:0 0 0 2024-04-22T15:50:07.439 INFO:tasks.workunit.client.1.smithi027.stdout:2/738: chown dd/df/d70/f6e 8132 1 2024-04-22T15:50:07.439 INFO:tasks.workunit.client.1.smithi027.stdout:8/694: truncate d1/d3/d1c/d49/f63 879217 0 2024-04-22T15:50:07.439 INFO:tasks.workunit.client.1.smithi027.stdout:8/695: chown d1/d3/d1c/d56 152 1 2024-04-22T15:50:07.440 INFO:tasks.workunit.client.1.smithi027.stdout:2/739: chown dd/df/d1b/d38/d60/d75/cd3 0 1 2024-04-22T15:50:07.443 INFO:tasks.workunit.client.1.smithi027.stdout:8/696: mkdir d1/d3/d1c/d28/d42/def 0 2024-04-22T15:50:07.444 INFO:tasks.workunit.client.1.smithi027.stdout:8/697: read - d1/d1e/d47/d53/fdc zero size 2024-04-22T15:50:07.444 INFO:tasks.workunit.client.1.smithi027.stdout:8/698: chown d1/d1e/d47/d53/l55 24 1 2024-04-22T15:50:07.445 INFO:tasks.workunit.client.1.smithi027.stdout:8/699: stat d1/d3/d1c/d32/c75 0 2024-04-22T15:50:07.445 INFO:tasks.workunit.client.1.smithi027.stdout:2/740: unlink dd/df/d1b/c4c 0 2024-04-22T15:50:07.451 INFO:tasks.workunit.client.1.smithi027.stdout:2/741: rename dd/df/d1b/l31 to dd/df/d1b/d1f/le4 0 2024-04-22T15:50:07.452 INFO:tasks.workunit.client.1.smithi027.stdout:8/700: dwrite d1/d3/d1c/d43/d5f/d7d/fd2 [0,4194304] 0 2024-04-22T15:50:07.454 INFO:tasks.workunit.client.1.smithi027.stdout:8/701: chown d1/d1e/c48 1224 1 2024-04-22T15:50:07.457 INFO:tasks.workunit.client.1.smithi027.stdout:2/742: creat dd/df/d1b/d96/dc5/dd1/fe5 x:0 0 0 2024-04-22T15:50:07.459 INFO:tasks.workunit.client.1.smithi027.stdout:8/702: mkdir d1/d3/d36/d57/df0 0 2024-04-22T15:50:07.460 INFO:tasks.workunit.client.1.smithi027.stdout:8/703: stat d1/d3/c23 0 2024-04-22T15:50:07.470 INFO:tasks.workunit.client.1.smithi027.stdout:8/704: creat d1/d3/d1c/d32/d41/d52/d61/d8f/dd3/ff1 x:0 0 0 2024-04-22T15:50:07.471 INFO:tasks.workunit.client.1.smithi027.stdout:8/705: write d1/d1e/d47/dc1/fd8 [316101,117860] 0 2024-04-22T15:50:07.476 INFO:tasks.workunit.client.1.smithi027.stdout:8/706: fdatasync d1/d3/d1c/d32/d41/d52/d61/fa2 0 2024-04-22T15:50:07.483 INFO:tasks.workunit.client.1.smithi027.stdout:2/743: rmdir dd/df/d1b/d96/dc5 39 2024-04-22T15:50:07.485 INFO:tasks.workunit.client.1.smithi027.stdout:2/744: creat dd/df/fe6 x:0 0 0 2024-04-22T15:50:07.491 INFO:tasks.workunit.client.1.smithi027.stdout:2/745: dwrite dd/df/d1b/fab [0,4194304] 0 2024-04-22T15:50:07.496 INFO:tasks.workunit.client.1.smithi027.stdout:2/746: creat dd/df/d70/d5a/fe7 x:0 0 0 2024-04-22T15:50:07.500 INFO:tasks.workunit.client.1.smithi027.stdout:2/747: fsync dd/df/f23 0 2024-04-22T15:50:07.501 INFO:tasks.workunit.client.1.smithi027.stdout:2/748: fsync dd/d43/fa7 0 2024-04-22T15:50:07.505 INFO:tasks.workunit.client.1.smithi027.stdout:2/749: dread dd/df/d1b/d1f/f74 [0,4194304] 0 2024-04-22T15:50:07.513 INFO:tasks.workunit.client.1.smithi027.stdout:2/750: dwrite dd/f6f [0,4194304] 0 2024-04-22T15:50:07.516 INFO:tasks.workunit.client.1.smithi027.stdout:2/751: mkdir dd/df/d1b/d38/de8 0 2024-04-22T15:50:07.517 INFO:tasks.workunit.client.1.smithi027.stdout:2/752: write dd/f11 [3821772,15655] 0 2024-04-22T15:50:07.520 INFO:tasks.workunit.client.1.smithi027.stdout:2/753: fsync dd/df/d1b/f32 0 2024-04-22T15:50:07.531 INFO:tasks.workunit.client.1.smithi027.stdout:2/754: getdents dd/df/d1b/d96 0 2024-04-22T15:50:07.538 INFO:tasks.workunit.client.1.smithi027.stdout:2/755: dwrite dd/df/d1b/d38/d60/d75/fbb [0,4194304] 0 2024-04-22T15:50:07.551 INFO:tasks.workunit.client.1.smithi027.stdout:2/756: dread dd/df/d1b/d38/d60/fa2 [0,4194304] 0 2024-04-22T15:50:07.553 INFO:tasks.workunit.client.1.smithi027.stdout:2/757: write dd/df/d70/d4b/d4e/daf/fe0 [809073,62179] 0 2024-04-22T15:50:07.561 INFO:tasks.workunit.client.1.smithi027.stdout:2/758: write dd/df/d1b/d1f/f36 [281750,5137] 0 2024-04-22T15:50:07.569 INFO:tasks.workunit.client.1.smithi027.stdout:2/759: dwrite dd/df/d1b/d96/dc5/dd1/fde [0,4194304] 0 2024-04-22T15:50:07.573 INFO:tasks.workunit.client.1.smithi027.stdout:2/760: truncate dd/df/d70/d4b/d52/f8f 254172 0 2024-04-22T15:50:07.580 INFO:tasks.workunit.client.1.smithi027.stdout:2/761: dwrite dd/df/d1b/d96/dc5/dd1/fd2 [0,4194304] 0 2024-04-22T15:50:07.582 INFO:tasks.workunit.client.1.smithi027.stdout:2/762: chown dd/df/d70/la9 89366 1 2024-04-22T15:50:07.590 INFO:tasks.workunit.client.1.smithi027.stdout:2/763: dwrite dd/df/f59 [0,4194304] 0 2024-04-22T15:50:07.595 INFO:tasks.workunit.client.1.smithi027.stdout:2/764: unlink dd/df/d70/c39 0 2024-04-22T15:50:07.626 INFO:tasks.workunit.client.1.smithi027.stdout:9/691: sync 2024-04-22T15:50:07.626 INFO:tasks.workunit.client.1.smithi027.stdout:1/647: sync 2024-04-22T15:50:07.626 INFO:tasks.workunit.client.1.smithi027.stdout:7/752: sync 2024-04-22T15:50:07.626 INFO:tasks.workunit.client.1.smithi027.stdout:0/910: sync 2024-04-22T15:50:07.626 INFO:tasks.workunit.client.1.smithi027.stdout:4/743: sync 2024-04-22T15:50:07.626 INFO:tasks.workunit.client.1.smithi027.stdout:5/751: sync 2024-04-22T15:50:07.626 INFO:tasks.workunit.client.1.smithi027.stdout:3/827: sync 2024-04-22T15:50:07.627 INFO:tasks.workunit.client.1.smithi027.stdout:9/692: stat d0/f44 0 2024-04-22T15:50:07.628 INFO:tasks.workunit.client.1.smithi027.stdout:4/744: fsync d5/d45/d5a/f10a 0 2024-04-22T15:50:07.629 INFO:tasks.workunit.client.1.smithi027.stdout:7/753: dread - d0/d11/d1c/d31/d56/fc6 zero size 2024-04-22T15:50:07.630 INFO:tasks.workunit.client.1.smithi027.stdout:5/752: chown d0/d1a/d22/d26/d5d/de3/l95 7 1 2024-04-22T15:50:07.631 INFO:tasks.workunit.client.1.smithi027.stdout:1/648: creat d2/d1d/fd9 x:0 0 0 2024-04-22T15:50:07.631 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:07 smithi016 ceph-mon[26620]: pgmap v7: 65 pgs: 65 active+clean; 3.0 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 38 MiB/s rd, 80 MiB/s wr, 243 op/s 2024-04-22T15:50:07.633 INFO:tasks.workunit.client.1.smithi027.stdout:7/754: chown d0/d4c/d4e/ff0 12439700 1 2024-04-22T15:50:07.635 INFO:tasks.workunit.client.1.smithi027.stdout:0/911: dread d0/d2a/d38/fad [0,4194304] 0 2024-04-22T15:50:07.635 INFO:tasks.workunit.client.1.smithi027.stdout:3/828: write d28/f49 [465891,122653] 0 2024-04-22T15:50:07.638 INFO:tasks.workunit.client.1.smithi027.stdout:4/745: unlink d5/de/df2/f70 0 2024-04-22T15:50:07.640 INFO:tasks.workunit.client.1.smithi027.stdout:0/912: dread d0/d34/d59/f62 [0,4194304] 0 2024-04-22T15:50:07.641 INFO:tasks.workunit.client.1.smithi027.stdout:1/649: symlink d2/d5f/d64/d81/da7/lda 0 2024-04-22T15:50:07.647 INFO:tasks.workunit.client.1.smithi027.stdout:9/693: getdents d0/dc/d12/d58/ddb 0 2024-04-22T15:50:07.657 INFO:tasks.workunit.client.1.smithi027.stdout:0/913: symlink d0/d2a/d58/d66/de4/l124 0 2024-04-22T15:50:07.660 INFO:tasks.workunit.client.1.smithi027.stdout:0/914: write d0/d2a/d58/d66/f9b [2899180,60979] 0 2024-04-22T15:50:07.663 INFO:tasks.workunit.client.1.smithi027.stdout:5/753: dwrite d0/d1a/d22/d26/d5d/d99/fb9 [0,4194304] 0 2024-04-22T15:50:07.701 INFO:tasks.workunit.client.1.smithi027.stdout:4/746: unlink d5/c1c 0 2024-04-22T15:50:07.702 INFO:tasks.workunit.client.1.smithi027.stdout:7/755: rmdir d0/d11/dfa 0 2024-04-22T15:50:07.718 INFO:tasks.workunit.client.1.smithi027.stdout:9/694: rmdir d0/dc/de/dd0 0 2024-04-22T15:50:07.721 INFO:tasks.workunit.client.1.smithi027.stdout:0/915: creat d0/d2a/d6a/d6c/d83/d9e/f125 x:0 0 0 2024-04-22T15:50:07.727 INFO:tasks.workunit.client.1.smithi027.stdout:4/747: creat d5/d11/d6c/da3/d10b/f110 x:0 0 0 2024-04-22T15:50:07.728 INFO:tasks.workunit.client.1.smithi027.stdout:0/916: dwrite d0/d34/d4e/d85/f118 [0,4194304] 0 2024-04-22T15:50:07.729 INFO:tasks.workunit.client.1.smithi027.stdout:7/756: mkdir d0/d11/d1c/d2c/dfc 0 2024-04-22T15:50:07.731 INFO:tasks.workunit.client.1.smithi027.stdout:7/757: readlink d0/d4c/d8b/la9 0 2024-04-22T15:50:07.731 INFO:tasks.workunit.client.1.smithi027.stdout:7/758: readlink d0/ldb 0 2024-04-22T15:50:07.732 INFO:tasks.workunit.client.1.smithi027.stdout:5/754: link d0/d1a/d22/d26/d5d/ce0 d0/d97/dd7/cf4 0 2024-04-22T15:50:07.738 INFO:tasks.workunit.client.1.smithi027.stdout:5/755: dread d0/d1a/d22/d26/d5d/d99/fb9 [0,4194304] 0 2024-04-22T15:50:07.740 INFO:tasks.workunit.client.1.smithi027.stdout:0/917: rename d0/d2a/d91/da1 to d0/d9d/d126 0 2024-04-22T15:50:07.741 INFO:tasks.workunit.client.1.smithi027.stdout:0/918: fsync d0/d9d/db7/ff0 0 2024-04-22T15:50:07.749 INFO:tasks.workunit.client.1.smithi027.stdout:0/919: dwrite d0/d9d/fc0 [4194304,4194304] 0 2024-04-22T15:50:07.759 INFO:tasks.workunit.client.1.smithi027.stdout:5/756: dread d0/d1a/d22/d26/d32/d88/d90/f9f [0,4194304] 0 2024-04-22T15:50:07.761 INFO:tasks.workunit.client.1.smithi027.stdout:0/920: creat d0/d28/f127 x:0 0 0 2024-04-22T15:50:07.772 INFO:tasks.workunit.client.1.smithi027.stdout:5/757: symlink d0/d1a/d22/d26/d5d/lf5 0 2024-04-22T15:50:07.778 INFO:tasks.workunit.client.1.smithi027.stdout:4/748: getdents d5/d11 0 2024-04-22T15:50:07.778 INFO:tasks.workunit.client.1.smithi027.stdout:7/759: link d0/d11/d1c/d31/d8a/fa4 d0/d11/d1c/ddc/ffd 0 2024-04-22T15:50:07.779 INFO:tasks.workunit.client.1.smithi027.stdout:0/921: dread d0/da3/dae/fea [0,4194304] 0 2024-04-22T15:50:07.800 INFO:tasks.workunit.client.1.smithi027.stdout:4/749: truncate d5/d11/d6c/f8b 2175303 0 2024-04-22T15:50:07.807 INFO:tasks.workunit.client.1.smithi027.stdout:4/750: creat d5/d45/d5a/dcd/f111 x:0 0 0 2024-04-22T15:50:07.809 INFO:tasks.workunit.client.1.smithi027.stdout:4/751: dread - d5/d45/d47/f8a zero size 2024-04-22T15:50:07.810 INFO:tasks.workunit.client.1.smithi027.stdout:0/922: rename d0/d28/d63/d67/c11f to d0/d28/d37/c128 0 2024-04-22T15:50:07.810 INFO:tasks.workunit.client.1.smithi027.stdout:3/829: dwrite d28/d4b/d52/db2/d95/fe5 [0,4194304] 0 2024-04-22T15:50:07.812 INFO:tasks.workunit.client.1.smithi027.stdout:3/830: dread - d28/d4b/d52/d9a/fab zero size 2024-04-22T15:50:07.813 INFO:tasks.workunit.client.1.smithi027.stdout:3/831: dread - d28/d4b/d52/d9a/ffe zero size 2024-04-22T15:50:07.816 INFO:tasks.workunit.client.1.smithi027.stdout:0/923: creat d0/d2a/d6a/d6c/f129 x:0 0 0 2024-04-22T15:50:07.820 INFO:tasks.workunit.client.1.smithi027.stdout:4/752: dwrite d5/de/d2a/d58/faf [0,4194304] 0 2024-04-22T15:50:07.832 INFO:tasks.workunit.client.1.smithi027.stdout:9/695: truncate d0/dc/d12/d30/d5b/f83 698229 0 2024-04-22T15:50:07.833 INFO:tasks.workunit.client.1.smithi027.stdout:9/696: fdatasync d0/dc/d12/d58/da6/fce 0 2024-04-22T15:50:07.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:07 smithi027 ceph-mon[34922]: pgmap v7: 65 pgs: 65 active+clean; 3.0 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 38 MiB/s rd, 80 MiB/s wr, 243 op/s 2024-04-22T15:50:07.836 INFO:tasks.workunit.client.1.smithi027.stdout:4/753: rename d5/d11/f36 to d5/d11/d6c/f112 0 2024-04-22T15:50:07.840 INFO:tasks.workunit.client.1.smithi027.stdout:0/924: write d0/d34/f71 [3907909,90316] 0 2024-04-22T15:50:07.843 INFO:tasks.workunit.client.1.smithi027.stdout:9/697: mknod d0/dc/de/db5/dbf/ce1 0 2024-04-22T15:50:07.848 INFO:tasks.workunit.client.1.smithi027.stdout:0/925: rmdir d0/d34/d4e/d85 39 2024-04-22T15:50:07.849 INFO:tasks.workunit.client.1.smithi027.stdout:0/926: chown d0/d28/d63/d67 215 1 2024-04-22T15:50:07.851 INFO:tasks.workunit.client.1.smithi027.stdout:9/698: rename d0/dc/l66 to d0/dc/d12/d30/dd4/le2 0 2024-04-22T15:50:07.852 INFO:tasks.workunit.client.1.smithi027.stdout:3/832: link d28/d5d/ca7 d28/d4b/d52/db2/deb/df3/c109 0 2024-04-22T15:50:07.854 INFO:tasks.workunit.client.1.smithi027.stdout:9/699: truncate d0/dc/d12/d30/d5b/d91/fe0 936148 0 2024-04-22T15:50:07.856 INFO:tasks.workunit.client.1.smithi027.stdout:0/927: creat d0/da3/dae/f12a x:0 0 0 2024-04-22T15:50:07.857 INFO:tasks.workunit.client.1.smithi027.stdout:7/760: write d0/d11/d1c/d31/f70 [1509430,123540] 0 2024-04-22T15:50:07.859 INFO:tasks.workunit.client.1.smithi027.stdout:3/833: chown d28/d4b/d52/db2/d95/d9c/dc1 29 1 2024-04-22T15:50:07.861 INFO:tasks.workunit.client.1.smithi027.stdout:5/758: dwrite d0/df/d10/f64 [0,4194304] 0 2024-04-22T15:50:07.874 INFO:tasks.workunit.client.1.smithi027.stdout:7/761: symlink d0/d11/d3a/d46/d94/d5a/lfe 0 2024-04-22T15:50:07.881 INFO:tasks.workunit.client.1.smithi027.stdout:5/759: dwrite d0/d1a/d22/d72/d86/f75 [0,4194304] 0 2024-04-22T15:50:07.881 INFO:tasks.workunit.client.1.smithi027.stdout:3/834: unlink d28/d5d/l72 0 2024-04-22T15:50:07.882 INFO:tasks.workunit.client.1.smithi027.stdout:3/835: truncate f1c 4383155 0 2024-04-22T15:50:07.888 INFO:tasks.workunit.client.1.smithi027.stdout:0/928: rename d0/d34/f7a to d0/d34/f12b 0 2024-04-22T15:50:07.889 INFO:tasks.workunit.client.1.smithi027.stdout:3/836: mkdir d28/d4b/d52/d94/dc4/d10a 0 2024-04-22T15:50:07.890 INFO:tasks.workunit.client.1.smithi027.stdout:7/762: link d0/d11/d1c/d31/lee d0/d4c/d8b/d92/da3/da8/lff 0 2024-04-22T15:50:07.907 INFO:tasks.workunit.client.1.smithi027.stdout:7/763: link d0/d4c/d8b/d92/da3/da8/f3b d0/d4c/de5/f100 0 2024-04-22T15:50:07.965 INFO:tasks.workunit.client.1.smithi027.stdout:7/764: dwrite d0/fb [0,4194304] 0 2024-04-22T15:50:07.965 INFO:tasks.workunit.client.1.smithi027.stdout:7/765: rename d0/d11/d1c/d2c/d96/db3/cbf to d0/d4c/d4e/c101 0 2024-04-22T15:50:07.965 INFO:tasks.workunit.client.1.smithi027.stdout:7/766: unlink d0/d4c/d8b/d92/da3/da8/c1f 0 2024-04-22T15:50:07.965 INFO:tasks.workunit.client.1.smithi027.stdout:7/767: readlink d0/d11/d1c/d31/d56/l7d 0 2024-04-22T15:50:07.965 INFO:tasks.workunit.client.1.smithi027.stdout:9/700: dwrite d0/d51/d9d/fad [0,4194304] 0 2024-04-22T15:50:07.966 INFO:tasks.workunit.client.1.smithi027.stdout:9/701: symlink d0/dc/d12/d30/d99/le3 0 2024-04-22T15:50:07.966 INFO:tasks.workunit.client.1.smithi027.stdout:7/768: mknod d0/d11/c102 0 2024-04-22T15:50:07.966 INFO:tasks.workunit.client.1.smithi027.stdout:7/769: chown d0/d4c/d4e/fad 347 1 2024-04-22T15:50:07.966 INFO:tasks.workunit.client.1.smithi027.stdout:9/702: mknod d0/d8/ce4 0 2024-04-22T15:50:07.966 INFO:tasks.workunit.client.1.smithi027.stdout:7/770: creat d0/d11/d1c/f103 x:0 0 0 2024-04-22T15:50:07.966 INFO:tasks.workunit.client.1.smithi027.stdout:7/771: write d0/fa0 [470984,64602] 0 2024-04-22T15:50:07.976 INFO:tasks.workunit.client.1.smithi027.stdout:0/929: write d0/d2a/d6a/fa0 [897904,41338] 0 2024-04-22T15:50:07.983 INFO:tasks.workunit.client.1.smithi027.stdout:0/930: dwrite d0/fe1 [0,4194304] 0 2024-04-22T15:50:08.049 INFO:tasks.workunit.client.1.smithi027.stdout:7/772: write d0/d11/d3a/d46/d94/fba [633599,44086] 0 2024-04-22T15:50:08.050 INFO:tasks.workunit.client.1.smithi027.stdout:9/703: dwrite d0/dc/d12/d58/d6b/d6e/f92 [0,4194304] 0 2024-04-22T15:50:08.053 INFO:tasks.workunit.client.1.smithi027.stdout:7/773: rename d0/d11/d1c to d0/d11/d1c/d2c/d93/d104 22 2024-04-22T15:50:08.055 INFO:tasks.workunit.client.1.smithi027.stdout:9/704: symlink d0/d64/le5 0 2024-04-22T15:50:08.061 INFO:tasks.workunit.client.1.smithi027.stdout:0/931: dwrite d0/d34/fc9 [0,4194304] 0 2024-04-22T15:50:08.063 INFO:tasks.workunit.client.1.smithi027.stdout:9/705: mkdir d0/d51/de6 0 2024-04-22T15:50:08.068 INFO:tasks.workunit.client.1.smithi027.stdout:7/774: creat d0/d4c/d8b/d72/db2/de7/f105 x:0 0 0 2024-04-22T15:50:08.071 INFO:tasks.workunit.client.1.smithi027.stdout:9/706: fdatasync d0/f90 0 2024-04-22T15:50:08.078 INFO:tasks.workunit.client.1.smithi027.stdout:7/775: mknod d0/d4c/d8b/d72/db2/de7/c106 0 2024-04-22T15:50:08.079 INFO:tasks.workunit.client.1.smithi027.stdout:7/776: rmdir d0/d11/d3a/d46 39 2024-04-22T15:50:08.081 INFO:tasks.workunit.client.1.smithi027.stdout:0/932: getdents d0/d28/d5f 0 2024-04-22T15:50:08.083 INFO:tasks.workunit.client.1.smithi027.stdout:9/707: symlink d0/dc/d12/d30/d5b/le7 0 2024-04-22T15:50:08.087 INFO:tasks.workunit.client.1.smithi027.stdout:9/708: mkdir d0/dc/d12/d58/da6/de8 0 2024-04-22T15:50:08.092 INFO:tasks.workunit.client.1.smithi027.stdout:0/933: getdents d0/d9d 0 2024-04-22T15:50:08.094 INFO:tasks.workunit.client.1.smithi027.stdout:9/709: write d0/d64/f9c [727880,87939] 0 2024-04-22T15:50:08.094 INFO:tasks.workunit.client.1.smithi027.stdout:7/777: dwrite d0/d11/f4d [0,4194304] 0 2024-04-22T15:50:08.097 INFO:tasks.workunit.client.1.smithi027.stdout:0/934: mknod d0/d2a/d6a/d6c/d83/c12c 0 2024-04-22T15:50:08.097 INFO:tasks.workunit.client.1.smithi027.stdout:9/710: write d0/dc/d12/d30/d5b/d91/da9/fdc [660816,130065] 0 2024-04-22T15:50:08.101 INFO:tasks.workunit.client.1.smithi027.stdout:7/778: rename d0/fb to d0/d4c/d4e/df5/f107 0 2024-04-22T15:50:08.103 INFO:tasks.workunit.client.1.smithi027.stdout:0/935: dread d0/fe1 [0,4194304] 0 2024-04-22T15:50:08.106 INFO:tasks.workunit.client.1.smithi027.stdout:7/779: creat d0/d4c/d8b/f108 x:0 0 0 2024-04-22T15:50:08.106 INFO:tasks.workunit.client.1.smithi027.stdout:7/780: chown d0/d4c/f7a 1 1 2024-04-22T15:50:08.109 INFO:tasks.workunit.client.1.smithi027.stdout:9/711: symlink d0/dc/d12/d30/d5b/le9 0 2024-04-22T15:50:08.111 INFO:tasks.workunit.client.1.smithi027.stdout:0/936: creat d0/da3/dae/df9/d10d/f12d x:0 0 0 2024-04-22T15:50:08.112 INFO:tasks.workunit.client.1.smithi027.stdout:0/937: read d0/d2a/d38/f92 [1744643,125242] 0 2024-04-22T15:50:08.114 INFO:tasks.workunit.client.1.smithi027.stdout:0/938: creat d0/d28/f12e x:0 0 0 2024-04-22T15:50:08.118 INFO:tasks.workunit.client.1.smithi027.stdout:7/781: dwrite d0/d11/d1c/d2c/fc2 [0,4194304] 0 2024-04-22T15:50:08.124 INFO:tasks.workunit.client.1.smithi027.stdout:7/782: mknod d0/d11/d3a/d46/d94/c109 0 2024-04-22T15:50:08.131 INFO:tasks.workunit.client.1.smithi027.stdout:7/783: rename d0/d11/f5c to d0/d11/d1c/d2c/d96/f10a 0 2024-04-22T15:50:08.132 INFO:tasks.workunit.client.1.smithi027.stdout:0/939: dread d0/d28/d63/d67/f82 [0,4194304] 0 2024-04-22T15:50:08.133 INFO:tasks.workunit.client.1.smithi027.stdout:0/940: dread - d0/d2a/d6a/d6c/f129 zero size 2024-04-22T15:50:08.139 INFO:tasks.workunit.client.1.smithi027.stdout:0/941: truncate d0/d34/f3a 1758682 0 2024-04-22T15:50:08.141 INFO:tasks.workunit.client.1.smithi027.stdout:0/942: unlink d0/d2a/d6a/d6c/f129 0 2024-04-22T15:50:08.146 INFO:tasks.workunit.client.1.smithi027.stdout:0/943: write d0/d34/d4e/d85/fe8 [1571612,33400] 0 2024-04-22T15:50:08.150 INFO:tasks.workunit.client.1.smithi027.stdout:0/944: mknod d0/d2a/d58/c12f 0 2024-04-22T15:50:08.151 INFO:tasks.workunit.client.1.smithi027.stdout:0/945: write d0/d34/d4e/f68 [231183,96658] 0 2024-04-22T15:50:08.159 INFO:tasks.workunit.client.1.smithi027.stdout:0/946: mknod d0/d34/d4e/c130 0 2024-04-22T15:50:08.162 INFO:tasks.workunit.client.1.smithi027.stdout:0/947: readlink d0/d28/d37/la4 0 2024-04-22T15:50:08.163 INFO:tasks.workunit.client.1.smithi027.stdout:0/948: write d0/d28/d63/f8d [1391876,108736] 0 2024-04-22T15:50:08.170 INFO:tasks.workunit.client.1.smithi027.stdout:6/831: sync 2024-04-22T15:50:08.171 INFO:tasks.workunit.client.1.smithi027.stdout:7/784: dwrite d0/d11/d1c/d2c/d96/f10a [0,4194304] 0 2024-04-22T15:50:08.174 INFO:tasks.workunit.client.1.smithi027.stdout:0/949: dwrite d0/da3/dae/fea [0,4194304] 0 2024-04-22T15:50:08.176 INFO:tasks.workunit.client.1.smithi027.stdout:7/785: dread d0/d4c/fd1 [4194304,4194304] 0 2024-04-22T15:50:08.180 INFO:tasks.workunit.client.1.smithi027.stdout:6/832: symlink d5/dc/dae/dc5/l120 0 2024-04-22T15:50:08.182 INFO:tasks.workunit.client.1.smithi027.stdout:6/833: write d5/dc/dd/d79/d7b/d9a/fa0 [1048147,129885] 0 2024-04-22T15:50:08.182 INFO:tasks.workunit.client.1.smithi027.stdout:0/950: fdatasync d0/d2a/f3c 0 2024-04-22T15:50:08.190 INFO:tasks.workunit.client.1.smithi027.stdout:6/834: rmdir d5/dc/d20/da3 39 2024-04-22T15:50:08.191 INFO:tasks.workunit.client.1.smithi027.stdout:0/951: dwrite d0/d2a/f61 [4194304,4194304] 0 2024-04-22T15:50:08.196 INFO:tasks.workunit.client.1.smithi027.stdout:0/952: creat d0/d2a/d58/d66/de4/f131 x:0 0 0 2024-04-22T15:50:08.209 INFO:tasks.workunit.client.1.smithi027.stdout:6/835: link d5/db/d2e/d59/db5/lc0 d5/db/d2e/d59/db5/l121 0 2024-04-22T15:50:08.214 INFO:tasks.workunit.client.1.smithi027.stdout:7/786: dwrite d0/d4c/d8b/d92/da3/da8/f4a [0,4194304] 0 2024-04-22T15:50:08.215 INFO:tasks.workunit.client.1.smithi027.stdout:7/787: chown d0/d11/d1c/ddc 1 1 2024-04-22T15:50:08.215 INFO:tasks.workunit.client.1.smithi027.stdout:7/788: chown d0/d4c 88079 1 2024-04-22T15:50:08.221 INFO:tasks.workunit.client.1.smithi027.stdout:0/953: dread d0/f2e [0,4194304] 0 2024-04-22T15:50:08.222 INFO:tasks.workunit.client.1.smithi027.stdout:0/954: readlink d0/d2a/d58/d66/de4/l124 0 2024-04-22T15:50:08.224 INFO:tasks.workunit.client.1.smithi027.stdout:7/789: rename d0/d11/c18 to d0/d4c/d8b/d92/c10b 0 2024-04-22T15:50:08.226 INFO:tasks.workunit.client.1.smithi027.stdout:0/955: creat d0/d2a/d6a/da6/f132 x:0 0 0 2024-04-22T15:50:08.226 INFO:tasks.workunit.client.1.smithi027.stdout:7/790: write d0/d11/d1c/d31/faa [4183710,110193] 0 2024-04-22T15:50:08.228 INFO:tasks.workunit.client.1.smithi027.stdout:7/791: dread d0/d11/d1c/fb1 [0,4194304] 0 2024-04-22T15:50:08.233 INFO:tasks.workunit.client.1.smithi027.stdout:0/956: rename d0/d2a/d6a/d6c/d83 to d0/d28/d63/d116/d133 0 2024-04-22T15:50:08.239 INFO:tasks.workunit.client.1.smithi027.stdout:6/836: truncate d5/dc/f74 3494232 0 2024-04-22T15:50:08.240 INFO:tasks.workunit.client.1.smithi027.stdout:7/792: symlink d0/l10c 0 2024-04-22T15:50:08.241 INFO:tasks.workunit.client.1.smithi027.stdout:0/957: dwrite d0/d2a/d6a/d6c/ff1 [0,4194304] 0 2024-04-22T15:50:08.242 INFO:tasks.workunit.client.1.smithi027.stdout:0/958: stat d0/fe1 0 2024-04-22T15:50:08.244 INFO:tasks.workunit.client.1.smithi027.stdout:0/959: write d0/d28/d63/d116/d133/d9e/ff5 [467925,116548] 0 2024-04-22T15:50:08.249 INFO:tasks.workunit.client.1.smithi027.stdout:6/837: truncate d5/dc/d20/da3/fe3 805612 0 2024-04-22T15:50:08.253 INFO:tasks.workunit.client.1.smithi027.stdout:0/960: dwrite d0/f22 [0,4194304] 0 2024-04-22T15:50:08.253 INFO:tasks.workunit.client.1.smithi027.stdout:7/793: read d0/d4c/d8b/d92/da3/da8/fef [3301741,129800] 0 2024-04-22T15:50:08.262 INFO:tasks.workunit.client.1.smithi027.stdout:6/838: dwrite d5/dc/d26/f2d [0,4194304] 0 2024-04-22T15:50:08.266 INFO:tasks.workunit.client.1.smithi027.stdout:0/961: dwrite d0/d34/d4e/d85/fd5 [0,4194304] 0 2024-04-22T15:50:08.268 INFO:tasks.workunit.client.1.smithi027.stdout:0/962: fdatasync d0/d2a/d58/d66/f9b 0 2024-04-22T15:50:08.273 INFO:tasks.workunit.client.1.smithi027.stdout:6/839: mknod d5/dc/d20/da3/dc9/c122 0 2024-04-22T15:50:08.277 INFO:tasks.workunit.client.1.smithi027.stdout:7/794: symlink d0/l10d 0 2024-04-22T15:50:08.284 INFO:tasks.workunit.client.1.smithi027.stdout:7/795: mkdir d0/d11/d1c/d2c/d96/db3/d10e 0 2024-04-22T15:50:08.289 INFO:tasks.workunit.client.1.smithi027.stdout:7/796: dread - d0/d49/d9f/fb5 zero size 2024-04-22T15:50:08.289 INFO:tasks.workunit.client.1.smithi027.stdout:4/754: sync 2024-04-22T15:50:08.289 INFO:tasks.workunit.client.1.smithi027.stdout:8/707: sync 2024-04-22T15:50:08.290 INFO:tasks.workunit.client.1.smithi027.stdout:5/760: sync 2024-04-22T15:50:08.290 INFO:tasks.workunit.client.1.smithi027.stdout:3/837: sync 2024-04-22T15:50:08.290 INFO:tasks.workunit.client.1.smithi027.stdout:2/765: sync 2024-04-22T15:50:08.290 INFO:tasks.workunit.client.1.smithi027.stdout:1/650: sync 2024-04-22T15:50:08.290 INFO:tasks.workunit.client.1.smithi027.stdout:0/963: link d0/d2a/l2d d0/d2a/d58/dee/l134 0 2024-04-22T15:50:08.295 INFO:tasks.workunit.client.1.smithi027.stdout:7/797: dread d0/d4c/d4e/df5/f107 [0,4194304] 0 2024-04-22T15:50:08.309 INFO:tasks.workunit.client.1.smithi027.stdout:0/964: dwrite d0/d2a/f2b [4194304,4194304] 0 2024-04-22T15:50:08.311 INFO:tasks.workunit.client.1.smithi027.stdout:1/651: getdents d2/db9 0 2024-04-22T15:50:08.313 INFO:tasks.workunit.client.1.smithi027.stdout:1/652: fdatasync d2/d1d/d56/dc4/fce 0 2024-04-22T15:50:08.314 INFO:tasks.workunit.client.1.smithi027.stdout:7/798: mkdir d0/d11/d1c/d31/d56/d10f 0 2024-04-22T15:50:08.318 INFO:tasks.workunit.client.1.smithi027.stdout:6/840: dwrite d5/dc/dd/f107 [0,4194304] 0 2024-04-22T15:50:08.324 INFO:tasks.workunit.client.1.smithi027.stdout:4/755: write d5/d11/d6c/f9b [924711,4359] 0 2024-04-22T15:50:08.324 INFO:tasks.workunit.client.1.smithi027.stdout:5/761: mknod d0/d1a/d22/d26/d5d/de3/d8f/dce/cf6 0 2024-04-22T15:50:08.331 INFO:tasks.workunit.client.1.smithi027.stdout:7/799: dwrite d0/d11/d1c/d31/d62/faf [0,4194304] 0 2024-04-22T15:50:08.333 INFO:tasks.workunit.client.1.smithi027.stdout:0/965: rmdir d0/d2a/d6a/d6c 39 2024-04-22T15:50:08.338 INFO:tasks.workunit.client.1.smithi027.stdout:5/762: dwrite d0/df/de/d24/f2e [0,4194304] 0 2024-04-22T15:50:08.340 INFO:tasks.workunit.client.1.smithi027.stdout:6/841: mknod d5/db/d2e/d59/df0/c123 0 2024-04-22T15:50:08.342 INFO:tasks.workunit.client.1.smithi027.stdout:7/800: write d0/d4c/d8b/d92/da3/da8/f25 [2367134,111573] 0 2024-04-22T15:50:08.344 INFO:tasks.workunit.client.1.smithi027.stdout:4/756: mkdir d5/d45/d113 0 2024-04-22T15:50:08.344 INFO:tasks.workunit.client.1.smithi027.stdout:0/966: chown d0/d2a/d58/d66/lc2 46 1 2024-04-22T15:50:08.348 INFO:tasks.workunit.client.1.smithi027.stdout:3/838: dwrite d28/f36 [0,4194304] 0 2024-04-22T15:50:08.348 INFO:tasks.workunit.client.1.smithi027.stdout:4/757: chown d5/d11/d1d/d24/dfc 2660 1 2024-04-22T15:50:08.349 INFO:tasks.workunit.client.1.smithi027.stdout:2/766: getdents dd/df/d70/d4b/d4e/daf 0 2024-04-22T15:50:08.352 INFO:tasks.workunit.client.1.smithi027.stdout:2/767: dread - dd/df/d1b/f41 zero size 2024-04-22T15:50:08.353 INFO:tasks.workunit.client.1.smithi027.stdout:5/763: truncate d0/fc9 109061 0 2024-04-22T15:50:08.357 INFO:tasks.workunit.client.1.smithi027.stdout:7/801: mkdir d0/d4c/d8b/d72/db2/de7/d110 0 2024-04-22T15:50:08.359 INFO:tasks.workunit.client.1.smithi027.stdout:0/967: dwrite d0/da3/dae/f123 [0,4194304] 0 2024-04-22T15:50:08.361 INFO:tasks.workunit.client.1.smithi027.stdout:6/842: mknod d5/dc/d17/c124 0 2024-04-22T15:50:08.361 INFO:tasks.workunit.client.1.smithi027.stdout:1/653: dread d2/d5f/d64/d81/f3a [0,4194304] 0 2024-04-22T15:50:08.362 INFO:tasks.workunit.client.1.smithi027.stdout:6/843: dread - d5/d30/dbe/f10f zero size 2024-04-22T15:50:08.368 INFO:tasks.workunit.client.1.smithi027.stdout:3/839: dread d28/f44 [0,4194304] 0 2024-04-22T15:50:08.370 INFO:tasks.workunit.client.1.smithi027.stdout:3/840: stat d28/d4b/d52/db2 0 2024-04-22T15:50:08.370 INFO:tasks.workunit.client.1.smithi027.stdout:5/764: creat d0/df/de/d40/ff7 x:0 0 0 2024-04-22T15:50:08.374 INFO:tasks.workunit.client.1.smithi027.stdout:6/844: dwrite d5/dc/d26/d3d/f51 [0,4194304] 0 2024-04-22T15:50:08.378 INFO:tasks.workunit.client.1.smithi027.stdout:4/758: mkdir d5/de/d114 0 2024-04-22T15:50:08.386 INFO:tasks.workunit.client.1.smithi027.stdout:1/654: mkdir d2/d5f/ddb 0 2024-04-22T15:50:08.390 INFO:tasks.workunit.client.1.smithi027.stdout:0/968: mknod d0/c135 0 2024-04-22T15:50:08.392 INFO:tasks.workunit.client.1.smithi027.stdout:0/969: truncate d0/da3/dae/fd4 306295 0 2024-04-22T15:50:08.394 INFO:tasks.workunit.client.1.smithi027.stdout:1/655: truncate d2/f3 1946104 0 2024-04-22T15:50:08.395 INFO:tasks.workunit.client.1.smithi027.stdout:7/802: creat d0/d11/d1c/d31/f111 x:0 0 0 2024-04-22T15:50:08.397 INFO:tasks.workunit.client.1.smithi027.stdout:0/970: symlink d0/d2a/d6a/d8e/l136 0 2024-04-22T15:50:08.398 INFO:tasks.workunit.client.1.smithi027.stdout:0/971: readlink d0/d28/l27 0 2024-04-22T15:50:08.402 INFO:tasks.workunit.client.1.smithi027.stdout:2/768: write dd/df/d70/d4b/d4e/f51 [755663,31827] 0 2024-04-22T15:50:08.405 INFO:tasks.workunit.client.1.smithi027.stdout:3/841: getdents d28/d4b/d52/db2/d95 0 2024-04-22T15:50:08.407 INFO:tasks.workunit.client.1.smithi027.stdout:4/759: dread d5/d45/f7d [0,4194304] 0 2024-04-22T15:50:08.414 INFO:tasks.workunit.client.1.smithi027.stdout:9/712: sync 2024-04-22T15:50:08.415 INFO:tasks.workunit.client.1.smithi027.stdout:4/760: mkdir d5/de/df2/de2/d115 0 2024-04-22T15:50:08.415 INFO:tasks.workunit.client.1.smithi027.stdout:7/803: creat d0/d4c/d8b/d92/f112 x:0 0 0 2024-04-22T15:50:08.417 INFO:tasks.workunit.client.1.smithi027.stdout:5/765: dread d0/f6 [0,4194304] 0 2024-04-22T15:50:08.417 INFO:tasks.workunit.client.1.smithi027.stdout:2/769: dwrite dd/df/d1b/d38/d60/d75/fbb [0,4194304] 0 2024-04-22T15:50:08.418 INFO:tasks.workunit.client.1.smithi027.stdout:9/713: stat d0/dc 0 2024-04-22T15:50:08.421 INFO:tasks.workunit.client.1.smithi027.stdout:7/804: write d0/d4c/d8b/d92/f112 [1034448,70064] 0 2024-04-22T15:50:08.421 INFO:tasks.workunit.client.1.smithi027.stdout:4/761: chown d5/d11/d1d/d24/l3e 416627 1 2024-04-22T15:50:08.422 INFO:tasks.workunit.client.1.smithi027.stdout:3/842: dwrite d28/fb9 [0,4194304] 0 2024-04-22T15:50:08.425 INFO:tasks.workunit.client.1.smithi027.stdout:1/656: link d2/d36/d74/cb0 d2/d5f/d64/d81/d3d/d82/d85/cdc 0 2024-04-22T15:50:08.428 INFO:tasks.workunit.client.1.smithi027.stdout:5/766: mknod d0/d1a/d22/d26/d32/d79/cf8 0 2024-04-22T15:50:08.431 INFO:tasks.workunit.client.1.smithi027.stdout:3/843: write d28/d4b/f5b [4450821,58190] 0 2024-04-22T15:50:08.437 INFO:tasks.workunit.client.1.smithi027.stdout:4/762: symlink d5/d7/l116 0 2024-04-22T15:50:08.441 INFO:tasks.workunit.client.1.smithi027.stdout:9/714: rename d0/dc/d12/d58/d6b/d6e/fd7 to d0/dc/d12/d30/fea 0 2024-04-22T15:50:08.441 INFO:tasks.workunit.client.1.smithi027.stdout:9/715: write d0/dc/d12/d30/f39 [4170950,127613] 0 2024-04-22T15:50:08.458 INFO:tasks.workunit.client.1.smithi027.stdout:5/767: write d0/d1a/d22/d26/d5d/de3/fcb [560459,30635] 0 2024-04-22T15:50:08.458 INFO:tasks.workunit.client.1.smithi027.stdout:9/716: fdatasync d0/dc/d12/d58/d6b/f62 0 2024-04-22T15:50:08.460 INFO:tasks.workunit.client.1.smithi027.stdout:1/657: dread d2/d1d/d56/dc4/fc8 [0,4194304] 0 2024-04-22T15:50:08.463 INFO:tasks.workunit.client.1.smithi027.stdout:5/768: truncate d0/d1a/d22/d26/d32/d79/fa7 5276141 0 2024-04-22T15:50:08.464 INFO:tasks.workunit.client.1.smithi027.stdout:2/770: rmdir dd/df/d70/d4b/d52/d9a/da4/dc7 0 2024-04-22T15:50:08.469 INFO:tasks.workunit.client.1.smithi027.stdout:9/717: fsync d0/f44 0 2024-04-22T15:50:08.473 INFO:tasks.workunit.client.1.smithi027.stdout:4/763: truncate d5/d7/f5b 1763575 0 2024-04-22T15:50:08.475 INFO:tasks.workunit.client.1.smithi027.stdout:5/769: read - d0/df/d3d/fb5 zero size 2024-04-22T15:50:08.479 INFO:tasks.workunit.client.1.smithi027.stdout:1/658: dwrite d2/d1d/d56/f9a [0,4194304] 0 2024-04-22T15:50:08.479 INFO:tasks.workunit.client.1.smithi027.stdout:9/718: truncate d0/d51/f7f 709322 0 2024-04-22T15:50:08.483 INFO:tasks.workunit.client.1.smithi027.stdout:1/659: chown d2/d5f/d64/d81/d7e/fb4 7288921 1 2024-04-22T15:50:08.487 INFO:tasks.workunit.client.1.smithi027.stdout:4/764: chown d5/d45/lb7 1629593 1 2024-04-22T15:50:08.488 INFO:tasks.workunit.client.1.smithi027.stdout:1/660: rename d2/d5f/fc3 to d2/d5f/d64/d81/d7e/fdd 0 2024-04-22T15:50:08.492 INFO:tasks.workunit.client.1.smithi027.stdout:9/719: link d0/fb6 d0/dc/d12/d58/da6/feb 0 2024-04-22T15:50:08.501 INFO:tasks.workunit.client.1.smithi027.stdout:4/765: dread - d5/d11/d1d/d54/d8c/fde zero size 2024-04-22T15:50:08.502 INFO:tasks.workunit.client.1.smithi027.stdout:2/771: dwrite dd/df/d1b/f62 [0,4194304] 0 2024-04-22T15:50:08.510 INFO:tasks.workunit.client.1.smithi027.stdout:2/772: dwrite dd/df/d70/d4b/f58 [0,4194304] 0 2024-04-22T15:50:08.512 INFO:tasks.workunit.client.1.smithi027.stdout:2/773: chown dd/df/d70/d4b/f68 0 1 2024-04-22T15:50:08.513 INFO:tasks.workunit.client.1.smithi027.stdout:2/774: chown dd/df/d70/d4b/d4e/dcb 253595210 1 2024-04-22T15:50:08.521 INFO:tasks.workunit.client.1.smithi027.stdout:5/770: dwrite d0/d1a/d22/d26/d32/d88/d90/fd4 [0,4194304] 0 2024-04-22T15:50:08.525 INFO:tasks.workunit.client.1.smithi027.stdout:4/766: chown d5/d11/d6c/la2 448391436 1 2024-04-22T15:50:08.530 INFO:tasks.workunit.client.1.smithi027.stdout:9/720: creat d0/dc/d12/d58/ddb/fec x:0 0 0 2024-04-22T15:50:08.531 INFO:tasks.workunit.client.1.smithi027.stdout:2/775: mknod dd/df/d1b/d96/dc5/ce9 0 2024-04-22T15:50:08.532 INFO:tasks.workunit.client.1.smithi027.stdout:9/721: write d0/dc/d12/d30/d5b/d91/da9/fbe [2285036,18458] 0 2024-04-22T15:50:08.534 INFO:tasks.workunit.client.1.smithi027.stdout:9/722: dread - d0/d64/f87 zero size 2024-04-22T15:50:08.538 INFO:tasks.workunit.client.1.smithi027.stdout:4/767: rename d5/d45/db3/lc2 to d5/d45/d5a/dcd/l117 0 2024-04-22T15:50:08.539 INFO:tasks.workunit.client.1.smithi027.stdout:4/768: write d5/d45/db3/fe8 [33501,126127] 0 2024-04-22T15:50:08.543 INFO:tasks.workunit.client.1.smithi027.stdout:2/776: dread dd/df/d1b/d96/f9d [0,4194304] 0 2024-04-22T15:50:08.544 INFO:tasks.workunit.client.1.smithi027.stdout:1/661: truncate d2/f57 4091102 0 2024-04-22T15:50:08.546 INFO:tasks.workunit.client.1.smithi027.stdout:5/771: creat d0/d1a/d22/d72/de8/ff9 x:0 0 0 2024-04-22T15:50:08.547 INFO:tasks.workunit.client.1.smithi027.stdout:1/662: truncate d2/d5f/d64/d81/d7e/fdd 914528 0 2024-04-22T15:50:08.549 INFO:tasks.workunit.client.1.smithi027.stdout:4/769: rmdir d5/d11/d1d/d24 39 2024-04-22T15:50:08.550 INFO:tasks.workunit.client.1.smithi027.stdout:4/770: readlink d5/da8/lb9 0 2024-04-22T15:50:08.551 INFO:tasks.workunit.client.1.smithi027.stdout:9/723: mknod d0/dc/d12/d30/d5b/ced 0 2024-04-22T15:50:08.552 INFO:tasks.workunit.client.1.smithi027.stdout:2/777: symlink dd/df/d1b/d96/dc5/lea 0 2024-04-22T15:50:08.554 INFO:tasks.workunit.client.1.smithi027.stdout:5/772: rmdir d0/d97 39 2024-04-22T15:50:08.565 INFO:tasks.workunit.client.1.smithi027.stdout:4/771: creat d5/d11/da6/dd0/f118 x:0 0 0 2024-04-22T15:50:08.567 INFO:tasks.workunit.client.1.smithi027.stdout:1/663: symlink d2/d5f/d64/d81/dcf/lde 0 2024-04-22T15:50:08.568 INFO:tasks.workunit.client.1.smithi027.stdout:5/773: creat d0/d1a/d22/d26/d5d/de3/ffa x:0 0 0 2024-04-22T15:50:08.569 INFO:tasks.workunit.client.1.smithi027.stdout:1/664: write d2/d5f/d64/d81/d3d/d44/f86 [4167806,121210] 0 2024-04-22T15:50:08.573 INFO:tasks.workunit.client.1.smithi027.stdout:2/778: getdents dd/df/d1b/d38/de8 0 2024-04-22T15:50:08.576 INFO:tasks.workunit.client.1.smithi027.stdout:5/774: dread d0/d1a/d22/d26/d5d/de3/d3e/fbe [0,4194304] 0 2024-04-22T15:50:08.579 INFO:tasks.workunit.client.1.smithi027.stdout:1/665: symlink d2/d5f/d64/ldf 0 2024-04-22T15:50:08.582 INFO:tasks.workunit.client.1.smithi027.stdout:2/779: creat dd/df/d1b/dc9/feb x:0 0 0 2024-04-22T15:50:08.585 INFO:tasks.workunit.client.1.smithi027.stdout:1/666: truncate d2/d5f/d64/d81/d7e/fcd 28707 0 2024-04-22T15:50:08.585 INFO:tasks.workunit.client.1.smithi027.stdout:5/775: rename d0/d1a/d22/d72/d86 to d0/d1a/d22/d72/dac/dde/dfb 0 2024-04-22T15:50:08.592 INFO:tasks.workunit.client.1.smithi027.stdout:2/780: mkdir dd/df/d70/d4b/dec 0 2024-04-22T15:50:08.602 INFO:tasks.workunit.client.1.smithi027.stdout:5/776: rename d0/d1a/d22/d26/d5d/ce0 to d0/d1a/d22/d26/d5d/dae/cfc 0 2024-04-22T15:50:08.603 INFO:tasks.workunit.client.1.smithi027.stdout:2/781: dread dd/f19 [4194304,4194304] 0 2024-04-22T15:50:08.603 INFO:tasks.workunit.client.1.smithi027.stdout:9/724: write d0/f36 [308069,2812] 0 2024-04-22T15:50:08.605 INFO:tasks.workunit.client.1.smithi027.stdout:2/782: chown dd/df/d1b/d38/d60/f65 12 1 2024-04-22T15:50:08.607 INFO:tasks.workunit.client.1.smithi027.stdout:5/777: write d0/d1a/d22/d26/d5d/de3/d3e/fbe [1410436,9876] 0 2024-04-22T15:50:08.608 INFO:tasks.workunit.client.1.smithi027.stdout:9/725: mkdir d0/dc/d12/dee 0 2024-04-22T15:50:08.614 INFO:tasks.workunit.client.1.smithi027.stdout:2/783: symlink dd/df/d70/d4b/d52/d9a/da4/dad/led 0 2024-04-22T15:50:08.615 INFO:tasks.workunit.client.1.smithi027.stdout:5/778: fsync d0/d1a/d22/d26/d32/d79/dd6/fdf 0 2024-04-22T15:50:08.620 INFO:tasks.workunit.client.1.smithi027.stdout:9/726: fsync d0/dc/de/f24 0 2024-04-22T15:50:08.620 INFO:tasks.workunit.client.1.smithi027.stdout:1/667: dwrite d2/f3 [0,4194304] 0 2024-04-22T15:50:08.628 INFO:tasks.workunit.client.1.smithi027.stdout:5/779: fsync d0/d1a/d22/d26/d32/d79/f87 0 2024-04-22T15:50:08.642 INFO:tasks.workunit.client.1.smithi027.stdout:5/780: truncate d0/d1a/d22/d26/f2a 4190494 0 2024-04-22T15:50:08.646 INFO:tasks.workunit.client.1.smithi027.stdout:2/784: rename dd/df/d70/cb2 to dd/df/d1b/d1f/cee 0 2024-04-22T15:50:08.649 INFO:tasks.workunit.client.1.smithi027.stdout:9/727: link d0/dc/d12/d30/d5b/le7 d0/dc/d12/d58/da6/de8/lef 0 2024-04-22T15:50:08.650 INFO:tasks.workunit.client.1.smithi027.stdout:5/781: dwrite d0/df/d3d/d81/fbc [0,4194304] 0 2024-04-22T15:50:08.654 INFO:tasks.workunit.client.1.smithi027.stdout:1/668: getdents d2/d5f/d64 0 2024-04-22T15:50:08.669 INFO:tasks.workunit.client.1.smithi027.stdout:2/785: write dd/df/f88 [664182,51514] 0 2024-04-22T15:50:08.671 INFO:tasks.workunit.client.1.smithi027.stdout:9/728: write d0/dc/f8e [695493,5261] 0 2024-04-22T15:50:08.676 INFO:tasks.workunit.client.1.smithi027.stdout:5/782: dwrite d0/df/d10/f14 [0,4194304] 0 2024-04-22T15:50:08.677 INFO:tasks.workunit.client.1.smithi027.stdout:5/783: dread - d0/df/de/feb zero size 2024-04-22T15:50:08.711 INFO:tasks.workunit.client.1.smithi027.stdout:2/786: symlink dd/df/d70/d4b/d52/d9a/da4/dad/lef 0 2024-04-22T15:50:08.725 INFO:tasks.workunit.client.1.smithi027.stdout:9/729: rename d0/d8/c50 to d0/dc/d12/d58/da6/cf0 0 2024-04-22T15:50:08.729 INFO:tasks.workunit.client.1.smithi027.stdout:9/730: symlink d0/d8b/lf1 0 2024-04-22T15:50:08.732 INFO:tasks.workunit.client.1.smithi027.stdout:9/731: creat d0/dc/d12/d58/ddb/ff2 x:0 0 0 2024-04-22T15:50:08.744 INFO:tasks.workunit.client.1.smithi027.stdout:8/708: sync 2024-04-22T15:50:08.747 INFO:tasks.workunit.client.1.smithi027.stdout:8/709: symlink d1/d3/d1c/d32/d41/d52/d61/d8f/ddd/lf2 0 2024-04-22T15:50:08.752 INFO:tasks.workunit.client.1.smithi027.stdout:8/710: mkdir d1/d3/d1c/d28/d42/dc4/df3 0 2024-04-22T15:50:08.754 INFO:tasks.workunit.client.1.smithi027.stdout:1/669: dwrite d2/d5f/d64/d81/d3d/d44/f92 [0,4194304] 0 2024-04-22T15:50:08.759 INFO:tasks.workunit.client.1.smithi027.stdout:8/711: symlink d1/d3/d1c/d28/d42/lf4 0 2024-04-22T15:50:08.762 INFO:tasks.workunit.client.1.smithi027.stdout:2/787: truncate dd/df/d1b/f2e 224023 0 2024-04-22T15:50:08.763 INFO:tasks.workunit.client.1.smithi027.stdout:1/670: truncate d2/d5f/d64/d81/d3d/d44/f73 2581052 0 2024-04-22T15:50:08.765 INFO:tasks.workunit.client.1.smithi027.stdout:5/784: dwrite d0/fc9 [0,4194304] 0 2024-04-22T15:50:08.766 INFO:tasks.workunit.client.1.smithi027.stdout:2/788: write dd/df/d1b/f5f [3380518,100533] 0 2024-04-22T15:50:08.770 INFO:tasks.workunit.client.1.smithi027.stdout:8/712: mknod d1/d3/d1c/d32/d41/cf5 0 2024-04-22T15:50:08.778 INFO:tasks.workunit.client.1.smithi027.stdout:2/789: mkdir dd/df/d70/d4b/d52/d9a/da4/dad/d8d/df0 0 2024-04-22T15:50:08.779 INFO:tasks.workunit.client.1.smithi027.stdout:2/790: dread - dd/df/d1b/d1f/f30 zero size 2024-04-22T15:50:08.789 INFO:tasks.workunit.client.1.smithi027.stdout:1/671: write d2/d5f/d64/d81/da7/f67 [3509923,33666] 0 2024-04-22T15:50:08.791 INFO:tasks.workunit.client.1.smithi027.stdout:8/713: creat d1/d3/d1c/ff6 x:0 0 0 2024-04-22T15:50:08.798 INFO:tasks.workunit.client.1.smithi027.stdout:1/672: mknod d2/d5f/d64/d81/d3d/d82/d85/ce0 0 2024-04-22T15:50:08.800 INFO:tasks.workunit.client.1.smithi027.stdout:8/714: chown d1/d3/d1c/d32/d41/d52/d61/d8f/ddd/dbe/fde 133651543 1 2024-04-22T15:50:08.806 INFO:tasks.workunit.client.1.smithi027.stdout:2/791: getdents dd/df/d70/d5a 0 2024-04-22T15:50:08.808 INFO:tasks.workunit.client.1.smithi027.stdout:1/673: rmdir d2/d36/d74 39 2024-04-22T15:50:08.819 INFO:tasks.workunit.client.1.smithi027.stdout:8/715: rename d1/d4f/cd7 to d1/d3/d36/cf7 0 2024-04-22T15:50:08.821 INFO:tasks.workunit.client.1.smithi027.stdout:8/716: chown d1/d3/f59 14660577 1 2024-04-22T15:50:08.821 INFO:tasks.workunit.client.1.smithi027.stdout:2/792: dread dd/df/d1b/d96/dc5/dd1/fd2 [0,4194304] 0 2024-04-22T15:50:08.822 INFO:tasks.workunit.client.1.smithi027.stdout:2/793: rename dd to dd/df/d1b/d38/d60/d75/d9f/df1 22 2024-04-22T15:50:08.827 INFO:tasks.workunit.client.1.smithi027.stdout:2/794: read f7 [2192131,32960] 0 2024-04-22T15:50:08.827 INFO:tasks.workunit.client.1.smithi027.stdout:8/717: creat d1/d3/d36/d88/da9/ff8 x:0 0 0 2024-04-22T15:50:08.831 INFO:tasks.workunit.client.1.smithi027.stdout:8/718: truncate d1/d3/d1c/d56/f6a 1717489 0 2024-04-22T15:50:08.833 INFO:tasks.workunit.client.1.smithi027.stdout:1/674: dwrite d2/d1d/f5b [0,4194304] 0 2024-04-22T15:50:08.835 INFO:tasks.workunit.client.1.smithi027.stdout:2/795: rename dd/df/d70/d4b/d52/d9a/da4/fc6 to dd/df/d70/d4b/d52/d9a/da4/ff2 0 2024-04-22T15:50:08.836 INFO:tasks.workunit.client.1.smithi027.stdout:2/796: dread - dd/df/d70/d5a/f81 zero size 2024-04-22T15:50:08.836 INFO:tasks.workunit.client.1.smithi027.stdout:8/719: creat d1/d3/d1c/d32/d41/d52/ff9 x:0 0 0 2024-04-22T15:50:08.837 INFO:tasks.workunit.client.1.smithi027.stdout:8/720: readlink d1/d3/l12 0 2024-04-22T15:50:08.847 INFO:tasks.workunit.client.1.smithi027.stdout:2/797: creat dd/df/d1b/d38/d60/d75/ddf/ff3 x:0 0 0 2024-04-22T15:50:08.857 INFO:tasks.workunit.client.1.smithi027.stdout:2/798: dwrite dd/df/d70/d4b/d52/fb6 [0,4194304] 0 2024-04-22T15:50:08.865 INFO:tasks.workunit.client.1.smithi027.stdout:2/799: mknod dd/d47/cf4 0 2024-04-22T15:50:08.867 INFO:tasks.workunit.client.1.smithi027.stdout:2/800: chown dd/df/d1b/d96/fc3 1 1 2024-04-22T15:50:08.876 INFO:tasks.workunit.client.1.smithi027.stdout:2/801: creat dd/df/d70/d4b/d52/d9a/da4/dad/ff5 x:0 0 0 2024-04-22T15:50:08.880 INFO:tasks.workunit.client.1.smithi027.stdout:8/721: dread d1/d3/d36/f85 [0,4194304] 0 2024-04-22T15:50:08.881 INFO:tasks.workunit.client.1.smithi027.stdout:2/802: mkdir dd/df/d1b/d38/d60/d75/ddf/df6 0 2024-04-22T15:50:08.885 INFO:tasks.workunit.client.1.smithi027.stdout:1/675: dwrite d2/d30/f47 [0,4194304] 0 2024-04-22T15:50:08.892 INFO:tasks.workunit.client.1.smithi027.stdout:2/803: dread dd/df/d1b/d38/d60/fa2 [0,4194304] 0 2024-04-22T15:50:08.893 INFO:tasks.workunit.client.1.smithi027.stdout:8/722: mknod d1/d3/d1c/d32/d41/cfa 0 2024-04-22T15:50:08.894 INFO:tasks.workunit.client.1.smithi027.stdout:1/676: dwrite d2/d1d/fd9 [0,4194304] 0 2024-04-22T15:50:08.895 INFO:tasks.workunit.client.1.smithi027.stdout:2/804: dread - dd/df/d1b/d1f/f30 zero size 2024-04-22T15:50:08.895 INFO:tasks.workunit.client.1.smithi027.stdout:8/723: readlink d1/d1e/d47/d53/l5b 0 2024-04-22T15:50:08.896 INFO:tasks.workunit.client.1.smithi027.stdout:2/805: chown dd 0 1 2024-04-22T15:50:08.898 INFO:tasks.workunit.client.1.smithi027.stdout:2/806: write dd/df/fe6 [633276,75743] 0 2024-04-22T15:50:08.903 INFO:tasks.workunit.client.1.smithi027.stdout:8/724: read d1/d3/d1c/d43/d82/f84 [1823345,65548] 0 2024-04-22T15:50:08.907 INFO:tasks.workunit.client.1.smithi027.stdout:1/677: link d2/fa d2/d5f/d64/d81/d3d/d44/d9c/fe1 0 2024-04-22T15:50:08.909 INFO:tasks.workunit.client.1.smithi027.stdout:1/678: chown d2/d5f/d64/d81/d69/d7d/fa8 2217 1 2024-04-22T15:50:08.912 INFO:tasks.workunit.client.1.smithi027.stdout:2/807: creat dd/df/d1b/d38/dd5/ff7 x:0 0 0 2024-04-22T15:50:08.914 INFO:tasks.workunit.client.1.smithi027.stdout:8/725: rename d1/d3/d1c/d32/d41/d52/fb6 to d1/d3/d1c/d32/d41/d52/d61/d8f/ddd/ffb 0 2024-04-22T15:50:08.916 INFO:tasks.workunit.client.1.smithi027.stdout:1/679: creat d2/d5f/d64/d81/d3d/d44/fe2 x:0 0 0 2024-04-22T15:50:08.927 INFO:tasks.workunit.client.1.smithi027.stdout:8/726: rename d1/d3/d1c/d32/d41/d52/fbf to d1/d1e/d47/d9d/ffc 0 2024-04-22T15:50:08.929 INFO:tasks.workunit.client.1.smithi027.stdout:1/680: creat d2/d5f/d64/d81/d3d/da9/fe3 x:0 0 0 2024-04-22T15:50:08.931 INFO:tasks.workunit.client.1.smithi027.stdout:2/808: getdents dd/df/d70/d5a 0 2024-04-22T15:50:08.939 INFO:tasks.workunit.client.1.smithi027.stdout:2/809: unlink dd/df/l3d 0 2024-04-22T15:50:08.946 INFO:tasks.workunit.client.1.smithi027.stdout:8/727: truncate d1/f18 4157360 0 2024-04-22T15:50:08.947 INFO:tasks.workunit.client.1.smithi027.stdout:8/728: chown d1/d1e/d47/d53/f78 107681 1 2024-04-22T15:50:08.947 INFO:tasks.workunit.client.1.smithi027.stdout:8/729: stat d1/d1e/d47/d53/c5e 0 2024-04-22T15:50:08.948 INFO:tasks.workunit.client.1.smithi027.stdout:1/681: mknod d2/d5f/d64/d81/ce4 0 2024-04-22T15:50:08.948 INFO:tasks.workunit.client.1.smithi027.stdout:8/730: dread - d1/d3/d1c/d49/fe6 zero size 2024-04-22T15:50:08.951 INFO:tasks.workunit.client.1.smithi027.stdout:8/731: fsync d1/d3/d1c/f34 0 2024-04-22T15:50:08.992 INFO:tasks.workunit.client.1.smithi027.stdout:6/845: sync 2024-04-22T15:50:08.995 INFO:tasks.workunit.client.1.smithi027.stdout:8/732: creat d1/d3/d36/d57/df0/ffd x:0 0 0 2024-04-22T15:50:08.996 INFO:tasks.workunit.client.1.smithi027.stdout:2/810: dwrite dd/df/d70/d4b/d52/d9a/da4/ff2 [0,4194304] 0 2024-04-22T15:50:08.996 INFO:tasks.workunit.client.1.smithi027.stdout:8/733: stat d1/d3/d1c/ff6 0 2024-04-22T15:50:08.997 INFO:tasks.workunit.client.1.smithi027.stdout:6/846: mknod d5/d30/d54/c125 0 2024-04-22T15:50:09.000 INFO:tasks.workunit.client.1.smithi027.stdout:2/811: chown dd/df/d70/d4b/d4e/lce 782730 1 2024-04-22T15:50:09.002 INFO:tasks.workunit.client.1.smithi027.stdout:8/734: unlink d1/fd1 0 2024-04-22T15:50:09.014 INFO:tasks.workunit.client.1.smithi027.stdout:8/735: fsync d1/d3/d36/f85 0 2024-04-22T15:50:09.021 INFO:tasks.workunit.client.1.smithi027.stdout:2/812: symlink dd/df/d70/d4b/d52/dda/lf8 0 2024-04-22T15:50:09.027 INFO:tasks.workunit.client.1.smithi027.stdout:2/813: creat dd/dae/dd4/ff9 x:0 0 0 2024-04-22T15:50:09.028 INFO:tasks.workunit.client.1.smithi027.stdout:6/847: dwrite d5/d30/d54/f7f [0,4194304] 0 2024-04-22T15:50:09.036 INFO:tasks.workunit.client.1.smithi027.stdout:8/736: dwrite d1/d3/d36/d88/f97 [0,4194304] 0 2024-04-22T15:50:09.036 INFO:tasks.workunit.client.1.smithi027.stdout:6/848: symlink d5/l126 0 2024-04-22T15:50:09.042 INFO:tasks.workunit.client.1.smithi027.stdout:6/849: rename d5/dc/d99/le8 to d5/dc/d20/da3/dc9/l127 0 2024-04-22T15:50:09.045 INFO:tasks.workunit.client.1.smithi027.stdout:6/850: symlink d5/dc/dd/d79/d7b/d5d/l128 0 2024-04-22T15:50:09.050 INFO:tasks.workunit.client.1.smithi027.stdout:6/851: read - d5/dc/fa6 zero size 2024-04-22T15:50:09.052 INFO:tasks.workunit.client.1.smithi027.stdout:2/814: dwrite dd/f8e [0,4194304] 0 2024-04-22T15:50:09.058 INFO:tasks.workunit.client.1.smithi027.stdout:2/815: unlink dd/df/d1b/d96/f98 0 2024-04-22T15:50:09.059 INFO:tasks.workunit.client.1.smithi027.stdout:8/737: dread d1/f18 [0,4194304] 0 2024-04-22T15:50:09.063 INFO:tasks.workunit.client.1.smithi027.stdout:2/816: dread dd/df/d70/d4b/d52/d9a/da4/ff2 [0,4194304] 0 2024-04-22T15:50:09.069 INFO:tasks.workunit.client.1.smithi027.stdout:8/738: fdatasync d1/d1e/f33 0 2024-04-22T15:50:09.070 INFO:tasks.workunit.client.1.smithi027.stdout:6/852: dread d5/db/d2e/d59/df0/f11a [0,4194304] 0 2024-04-22T15:50:09.071 INFO:tasks.workunit.client.1.smithi027.stdout:2/817: creat dd/df/d1b/dc9/dca/ffa x:0 0 0 2024-04-22T15:50:09.074 INFO:tasks.workunit.client.1.smithi027.stdout:6/853: creat d5/dc/d26/d3d/f129 x:0 0 0 2024-04-22T15:50:09.082 INFO:tasks.workunit.client.1.smithi027.stdout:8/739: write d1/d3/d1c/d32/d41/f7b [202165,21225] 0 2024-04-22T15:50:09.086 INFO:tasks.workunit.client.1.smithi027.stdout:6/854: write d5/dc/d17/d8a/fdc [1042758,84279] 0 2024-04-22T15:50:09.090 INFO:tasks.workunit.client.1.smithi027.stdout:6/855: read d5/db/fa9 [9701,116522] 0 2024-04-22T15:50:09.091 INFO:tasks.workunit.client.1.smithi027.stdout:8/740: link d1/d3/d1c/ff6 d1/d3/d1c/d28/d42/ffe 0 2024-04-22T15:50:09.094 INFO:tasks.workunit.client.1.smithi027.stdout:6/856: unlink d5/dc/d17/d53/dec/ff1 0 2024-04-22T15:50:09.094 INFO:tasks.workunit.client.1.smithi027.stdout:2/818: dwrite dd/df/d1b/d96/dc5/dd1/fd2 [0,4194304] 0 2024-04-22T15:50:09.096 INFO:tasks.workunit.client.1.smithi027.stdout:8/741: unlink d1/d3/d1c/d43/d5f/fe9 0 2024-04-22T15:50:09.099 INFO:tasks.workunit.client.1.smithi027.stdout:2/819: stat dd/df/d1b/d96/fbd 0 2024-04-22T15:50:09.099 INFO:tasks.workunit.client.1.smithi027.stdout:6/857: symlink d5/d30/d54/dd8/dfc/l12a 0 2024-04-22T15:50:09.100 INFO:tasks.workunit.client.1.smithi027.stdout:8/742: creat d1/d3/d1c/d32/d41/fff x:0 0 0 2024-04-22T15:50:09.100 INFO:tasks.workunit.client.1.smithi027.stdout:2/820: fdatasync dd/df/d1b/d38/d60/d75/ddf/ff3 0 2024-04-22T15:50:09.102 INFO:tasks.workunit.client.1.smithi027.stdout:2/821: chown dd/df/d1b/d38/d60/d75/c89 696400 1 2024-04-22T15:50:09.103 INFO:tasks.workunit.client.1.smithi027.stdout:6/858: mknod d5/dc/dd/d79/d7b/d9a/c12b 0 2024-04-22T15:50:09.103 INFO:tasks.workunit.client.1.smithi027.stdout:6/859: read - d5/dc/fa6 zero size 2024-04-22T15:50:09.105 INFO:tasks.workunit.client.1.smithi027.stdout:8/743: creat d1/d1e/dbc/f100 x:0 0 0 2024-04-22T15:50:09.105 INFO:tasks.workunit.client.1.smithi027.stdout:2/822: read - dd/df/d70/d4b/d52/d9a/da4/dad/fc0 zero size 2024-04-22T15:50:09.109 INFO:tasks.workunit.client.1.smithi027.stdout:8/744: creat d1/d3/d1c/d56/f101 x:0 0 0 2024-04-22T15:50:09.110 INFO:tasks.workunit.client.1.smithi027.stdout:8/745: write d1/d3/f1a [5466028,50456] 0 2024-04-22T15:50:09.111 INFO:tasks.workunit.client.1.smithi027.stdout:2/823: rename dd/df/d70/d4b/d52/d9a/da4/dad/fc0 to dd/df/d70/d4b/d4e/daf/ffb 0 2024-04-22T15:50:09.117 INFO:tasks.workunit.client.1.smithi027.stdout:2/824: creat dd/df/d1b/ffc x:0 0 0 2024-04-22T15:50:09.123 INFO:tasks.workunit.client.1.smithi027.stdout:2/825: rename dd/df/d70/d5a/fe7 to dd/df/d1b/d1f/ffd 0 2024-04-22T15:50:09.126 INFO:tasks.workunit.client.1.smithi027.stdout:2/826: unlink dd/df/d70/d4b/d52/f8a 0 2024-04-22T15:50:09.127 INFO:tasks.workunit.client.1.smithi027.stdout:2/827: readlink dd/df/d70/d4b/d52/d9a/da4/dad/led 0 2024-04-22T15:50:09.130 INFO:tasks.workunit.client.1.smithi027.stdout:2/828: creat dd/dae/ffe x:0 0 0 2024-04-22T15:50:09.132 INFO:tasks.workunit.client.1.smithi027.stdout:2/829: rmdir dd/df/d70/d4b/d52/d9a/da4/dad/d8d 39 2024-04-22T15:50:09.135 INFO:tasks.workunit.client.1.smithi027.stdout:2/830: unlink dd/df/d70/d4b/d4e/lce 0 2024-04-22T15:50:09.135 INFO:tasks.workunit.client.1.smithi027.stdout:8/746: dwrite d1/d3/d1c/d32/d41/f86 [0,4194304] 0 2024-04-22T15:50:09.140 INFO:tasks.workunit.client.1.smithi027.stdout:8/747: creat d1/d3/d1c/d32/d41/d52/f102 x:0 0 0 2024-04-22T15:50:09.142 INFO:tasks.workunit.client.1.smithi027.stdout:8/748: chown d1/d3/d1c/d32/d41/d52/f102 7359 1 2024-04-22T15:50:09.142 INFO:tasks.workunit.client.1.smithi027.stdout:2/831: fsync dd/df/d1b/d1f/f85 0 2024-04-22T15:50:09.152 INFO:tasks.workunit.client.1.smithi027.stdout:8/749: creat d1/d3/d1c/d32/f103 x:0 0 0 2024-04-22T15:50:09.154 INFO:tasks.workunit.client.1.smithi027.stdout:8/750: mknod d1/d3/c104 0 2024-04-22T15:50:09.155 INFO:tasks.workunit.client.1.smithi027.stdout:8/751: chown d1/d3/d1c/d32/fcb 9 1 2024-04-22T15:50:09.156 INFO:tasks.workunit.client.1.smithi027.stdout:8/752: write d1/d3/d36/d88/da9/ff8 [818185,119473] 0 2024-04-22T15:50:09.158 INFO:tasks.workunit.client.1.smithi027.stdout:2/832: dread dd/f6f [0,4194304] 0 2024-04-22T15:50:09.158 INFO:tasks.workunit.client.1.smithi027.stdout:8/753: fdatasync d1/d3/d1c/d32/f69 0 2024-04-22T15:50:09.162 INFO:tasks.workunit.client.1.smithi027.stdout:2/833: read dd/df/d70/f45 [2286657,27654] 0 2024-04-22T15:50:09.168 INFO:tasks.workunit.client.1.smithi027.stdout:8/754: write d1/d3/d1c/d32/d41/d52/d61/d8f/ddd/dbe/fde [80916,56618] 0 2024-04-22T15:50:09.170 INFO:tasks.workunit.client.1.smithi027.stdout:8/755: chown d1/d3/d1c/d32/d41/d52/fa1 14359 1 2024-04-22T15:50:09.174 INFO:tasks.workunit.client.1.smithi027.stdout:2/834: dwrite dd/df/d1b/d38/d60/d75/fa6 [4194304,4194304] 0 2024-04-22T15:50:09.177 INFO:tasks.workunit.client.1.smithi027.stdout:8/756: link d1/d3/f40 d1/d3/d1c/d49/d94/f105 0 2024-04-22T15:50:09.180 INFO:tasks.workunit.client.1.smithi027.stdout:2/835: stat dd/df/d1b/d96/fc3 0 2024-04-22T15:50:09.182 INFO:tasks.workunit.client.1.smithi027.stdout:8/757: mkdir d1/d3/d1c/d28/d42/dc4/d106 0 2024-04-22T15:50:09.186 INFO:tasks.workunit.client.1.smithi027.stdout:2/836: symlink dd/df/d70/d4b/lff 0 2024-04-22T15:50:09.188 INFO:tasks.workunit.client.1.smithi027.stdout:8/758: fsync d1/d3/d1c/d32/fb0 0 2024-04-22T15:50:09.190 INFO:tasks.workunit.client.1.smithi027.stdout:2/837: dread dd/f6f [0,4194304] 0 2024-04-22T15:50:09.194 INFO:tasks.workunit.client.1.smithi027.stdout:2/838: dread dd/f19 [4194304,4194304] 0 2024-04-22T15:50:09.194 INFO:tasks.workunit.client.1.smithi027.stdout:0/972: sync 2024-04-22T15:50:09.194 INFO:tasks.workunit.client.1.smithi027.stdout:7/805: sync 2024-04-22T15:50:09.198 INFO:tasks.workunit.client.1.smithi027.stdout:2/839: rmdir dd/df/d1b/d96/dc5 39 2024-04-22T15:50:09.201 INFO:tasks.workunit.client.1.smithi027.stdout:8/759: dread d1/d3/d1c/d49/d94/f105 [0,4194304] 0 2024-04-22T15:50:09.206 INFO:tasks.workunit.client.1.smithi027.stdout:8/760: symlink d1/d3/l107 0 2024-04-22T15:50:09.208 INFO:tasks.workunit.client.1.smithi027.stdout:7/806: dwrite d0/d11/d1c/d31/d56/fa1 [0,4194304] 0 2024-04-22T15:50:09.211 INFO:tasks.workunit.client.1.smithi027.stdout:2/840: rename dd/d43/fa7 to dd/df/d1b/d96/dc5/f100 0 2024-04-22T15:50:09.212 INFO:tasks.workunit.client.1.smithi027.stdout:7/807: truncate d0/d11/d1c/d31/d8a/fce 538048 0 2024-04-22T15:50:09.213 INFO:tasks.workunit.client.1.smithi027.stdout:8/761: fdatasync d1/f4 0 2024-04-22T15:50:09.215 INFO:tasks.workunit.client.1.smithi027.stdout:8/762: read - d1/d1e/d47/dc1/fdb zero size 2024-04-22T15:50:09.216 INFO:tasks.workunit.client.1.smithi027.stdout:8/763: chown d1/d3/d36/f85 307503 1 2024-04-22T15:50:09.216 INFO:tasks.workunit.client.1.smithi027.stdout:0/973: dread d0/d34/d59/f5d [0,4194304] 0 2024-04-22T15:50:09.222 INFO:tasks.workunit.client.1.smithi027.stdout:2/841: stat dd/df/d1b/d96/f9d 0 2024-04-22T15:50:09.223 INFO:tasks.workunit.client.1.smithi027.stdout:7/808: write d0/f14 [1073172,128225] 0 2024-04-22T15:50:09.223 INFO:tasks.workunit.client.1.smithi027.stdout:2/842: chown dd/c6a 3358 1 2024-04-22T15:50:09.224 INFO:tasks.workunit.client.1.smithi027.stdout:8/764: readlink d1/d1e/l24 0 2024-04-22T15:50:09.225 INFO:tasks.workunit.client.1.smithi027.stdout:8/765: readlink d1/d3/l9 0 2024-04-22T15:50:09.227 INFO:tasks.workunit.client.1.smithi027.stdout:7/809: creat d0/d11/d1c/ddc/f113 x:0 0 0 2024-04-22T15:50:09.228 INFO:tasks.workunit.client.1.smithi027.stdout:2/843: mknod dd/df/d1b/d38/d60/d75/ddf/c101 0 2024-04-22T15:50:09.230 INFO:tasks.workunit.client.1.smithi027.stdout:8/766: dread d1/d3/d1c/f34 [0,4194304] 0 2024-04-22T15:50:09.230 INFO:tasks.workunit.client.1.smithi027.stdout:7/810: creat d0/d49/f114 x:0 0 0 2024-04-22T15:50:09.232 INFO:tasks.workunit.client.1.smithi027.stdout:0/974: link d0/d34/ff8 d0/d2a/d58/dee/f137 0 2024-04-22T15:50:09.232 INFO:tasks.workunit.client.1.smithi027.stdout:0/975: fsync d0/d28/f1f 0 2024-04-22T15:50:09.233 INFO:tasks.workunit.client.1.smithi027.stdout:2/844: truncate dd/df/d1b/d1f/f74 79451 0 2024-04-22T15:50:09.236 INFO:tasks.workunit.client.1.smithi027.stdout:7/811: symlink d0/d4c/d8b/d72/db2/l115 0 2024-04-22T15:50:09.245 INFO:tasks.workunit.client.1.smithi027.stdout:2/845: rename dd/df/d70/d4b/l9e to dd/dae/dd4/ddb/l102 0 2024-04-22T15:50:09.246 INFO:tasks.workunit.client.1.smithi027.stdout:0/976: dwrite d0/d2a/d58/dbf/fdb [0,4194304] 0 2024-04-22T15:50:09.251 INFO:tasks.workunit.client.1.smithi027.stdout:8/767: dread d1/d3/d1c/d32/d41/fd9 [0,4194304] 0 2024-04-22T15:50:09.256 INFO:tasks.workunit.client.1.smithi027.stdout:2/846: mknod dd/df/d1b/d38/dd5/c103 0 2024-04-22T15:50:09.257 INFO:tasks.workunit.client.1.smithi027.stdout:8/768: mknod d1/d3/d36/d57/c108 0 2024-04-22T15:50:09.258 INFO:tasks.workunit.client.1.smithi027.stdout:7/812: dread d0/d11/f4d [0,4194304] 0 2024-04-22T15:50:09.258 INFO:tasks.workunit.client.1.smithi027.stdout:0/977: dwrite d0/da3/dae/f12a [0,4194304] 0 2024-04-22T15:50:09.261 INFO:tasks.workunit.client.1.smithi027.stdout:0/978: dread - d0/d2a/d58/d66/fd9 zero size 2024-04-22T15:50:09.263 INFO:tasks.workunit.client.1.smithi027.stdout:2/847: rmdir dd/df/d1b/d38/d60/d75 39 2024-04-22T15:50:09.267 INFO:tasks.workunit.client.1.smithi027.stdout:7/813: creat d0/d4c/d8b/f116 x:0 0 0 2024-04-22T15:50:09.276 INFO:tasks.workunit.client.1.smithi027.stdout:8/769: link d1/d3/d1c/d32/d41/d52/d61/d8f/lbb d1/d3/d1c/d56/l109 0 2024-04-22T15:50:09.277 INFO:tasks.workunit.client.1.smithi027.stdout:2/848: creat dd/df/d1b/f104 x:0 0 0 2024-04-22T15:50:09.278 INFO:tasks.workunit.client.1.smithi027.stdout:0/979: dread d0/d34/fc9 [0,4194304] 0 2024-04-22T15:50:09.278 INFO:tasks.workunit.client.1.smithi027.stdout:0/980: readlink d0/d2a/l4f 0 2024-04-22T15:50:09.278 INFO:tasks.workunit.client.1.smithi027.stdout:2/849: chown dd/df/d1b/d1f/c99 279 1 2024-04-22T15:50:09.279 INFO:tasks.workunit.client.1.smithi027.stdout:8/770: unlink d1/c3b 0 2024-04-22T15:50:09.284 INFO:tasks.workunit.client.1.smithi027.stdout:8/771: creat d1/d4f/f10a x:0 0 0 2024-04-22T15:50:09.285 INFO:tasks.workunit.client.1.smithi027.stdout:2/850: dwrite dd/df/d1b/f7a [0,4194304] 0 2024-04-22T15:50:09.287 INFO:tasks.workunit.client.1.smithi027.stdout:8/772: write d1/d3/d1c/d32/d41/d52/f102 [407791,82073] 0 2024-04-22T15:50:09.287 INFO:tasks.workunit.client.1.smithi027.stdout:0/981: rename d0/d28/dd2 to d0/d34/d4e/d138 0 2024-04-22T15:50:09.291 INFO:tasks.workunit.client.1.smithi027.stdout:2/851: mkdir dd/df/d70/d4b/d52/d9a/d105 0 2024-04-22T15:50:09.292 INFO:tasks.workunit.client.1.smithi027.stdout:7/814: dwrite d0/f13 [0,4194304] 0 2024-04-22T15:50:09.294 INFO:tasks.workunit.client.1.smithi027.stdout:8/773: dread - d1/d3/d1c/d32/d41/d52/d61/fb2 zero size 2024-04-22T15:50:09.295 INFO:tasks.workunit.client.1.smithi027.stdout:0/982: creat d0/d2a/d38/f139 x:0 0 0 2024-04-22T15:50:09.296 INFO:tasks.workunit.client.1.smithi027.stdout:8/774: dread - d1/d1e/dbc/f100 zero size 2024-04-22T15:50:09.301 INFO:tasks.workunit.client.1.smithi027.stdout:0/983: write d0/d28/d37/f108 [67012,17250] 0 2024-04-22T15:50:09.303 INFO:tasks.workunit.client.1.smithi027.stdout:2/852: rename dd/df/d1b/f25 to dd/df/d1b/f106 0 2024-04-22T15:50:09.304 INFO:tasks.workunit.client.1.smithi027.stdout:7/815: dwrite d0/d11/d1c/d2c/fc4 [0,4194304] 0 2024-04-22T15:50:09.305 INFO:tasks.workunit.client.1.smithi027.stdout:8/775: dread d1/d3/d1c/d49/d94/f105 [0,4194304] 0 2024-04-22T15:50:09.305 INFO:tasks.workunit.client.1.smithi027.stdout:0/984: write d0/d9d/db7/ff0 [352871,35579] 0 2024-04-22T15:50:09.308 INFO:tasks.workunit.client.1.smithi027.stdout:0/985: write d0/d34/d4e/d85/db3/f110 [674803,17418] 0 2024-04-22T15:50:09.318 INFO:tasks.workunit.client.1.smithi027.stdout:2/853: dwrite dd/df/d1b/d96/dc5/dd1/fe5 [0,4194304] 0 2024-04-22T15:50:09.323 INFO:tasks.workunit.client.1.smithi027.stdout:0/986: dwrite d0/d28/d37/f40 [0,4194304] 0 2024-04-22T15:50:09.327 INFO:tasks.workunit.client.1.smithi027.stdout:7/816: write d0/d4c/d8b/d72/db2/fac [2688687,64719] 0 2024-04-22T15:50:09.330 INFO:tasks.workunit.client.1.smithi027.stdout:0/987: symlink d0/d28/l13a 0 2024-04-22T15:50:09.331 INFO:tasks.workunit.client.1.smithi027.stdout:2/854: dread dd/df/f23 [0,4194304] 0 2024-04-22T15:50:09.332 INFO:tasks.workunit.client.1.smithi027.stdout:0/988: read - d0/d9d/f9a zero size 2024-04-22T15:50:09.333 INFO:tasks.workunit.client.1.smithi027.stdout:2/855: write dd/df/d1b/fc1 [3316721,49916] 0 2024-04-22T15:50:09.335 INFO:tasks.workunit.client.1.smithi027.stdout:7/817: dread d0/d11/d1c/d31/ff6 [0,4194304] 0 2024-04-22T15:50:09.337 INFO:tasks.workunit.client.1.smithi027.stdout:8/776: dread d1/d3/d1c/d49/f5d [4194304,4194304] 0 2024-04-22T15:50:09.339 INFO:tasks.workunit.client.1.smithi027.stdout:2/856: dread dd/f6f [0,4194304] 0 2024-04-22T15:50:09.340 INFO:tasks.workunit.client.1.smithi027.stdout:0/989: symlink d0/d2a/d6a/l13b 0 2024-04-22T15:50:09.343 INFO:tasks.workunit.client.1.smithi027.stdout:7/818: rename d0/d3f/l41 to d0/d4c/d8b/d92/da3/da8/l117 0 2024-04-22T15:50:09.344 INFO:tasks.workunit.client.1.smithi027.stdout:2/857: creat dd/df/d70/d4b/d4e/f107 x:0 0 0 2024-04-22T15:50:09.345 INFO:tasks.workunit.client.1.smithi027.stdout:0/990: mknod d0/da3/dae/df9/c13c 0 2024-04-22T15:50:09.349 INFO:tasks.workunit.client.1.smithi027.stdout:0/991: creat d0/d9d/d126/f13d x:0 0 0 2024-04-22T15:50:09.350 INFO:tasks.workunit.client.1.smithi027.stdout:7/819: creat d0/d11/f118 x:0 0 0 2024-04-22T15:50:09.354 INFO:tasks.workunit.client.1.smithi027.stdout:8/777: dread d1/d3/d1c/d32/d41/f83 [0,4194304] 0 2024-04-22T15:50:09.354 INFO:tasks.workunit.client.1.smithi027.stdout:2/858: dwrite dd/f6f [0,4194304] 0 2024-04-22T15:50:09.359 INFO:tasks.workunit.client.1.smithi027.stdout:2/859: rmdir dd/df/d1b/dc9 39 2024-04-22T15:50:09.362 INFO:tasks.workunit.client.1.smithi027.stdout:7/820: dwrite d0/d11/d1c/d31/ff6 [0,4194304] 0 2024-04-22T15:50:09.365 INFO:tasks.workunit.client.1.smithi027.stdout:7/821: truncate d0/d4c/d8b/fe8 825357 0 2024-04-22T15:50:09.366 INFO:tasks.workunit.client.1.smithi027.stdout:2/860: dread dd/df/d1b/d1f/f85 [0,4194304] 0 2024-04-22T15:50:09.366 INFO:tasks.workunit.client.1.smithi027.stdout:7/822: chown d0/d11/d1c/fb1 297 1 2024-04-22T15:50:09.367 INFO:tasks.workunit.client.1.smithi027.stdout:2/861: stat dd/df/d70/d4b/d52/d9a 0 2024-04-22T15:50:09.367 INFO:tasks.workunit.client.1.smithi027.stdout:7/823: chown d0/d4c/d8b/lc1 1 1 2024-04-22T15:50:09.376 INFO:tasks.workunit.client.1.smithi027.stdout:7/824: unlink d0/d4c/d8b/d92/da3/cd9 0 2024-04-22T15:50:09.377 INFO:tasks.workunit.client.1.smithi027.stdout:0/992: dwrite d0/d2a/d38/fad [0,4194304] 0 2024-04-22T15:50:09.378 INFO:tasks.workunit.client.1.smithi027.stdout:2/862: mknod dd/df/d70/d4b/c108 0 2024-04-22T15:50:09.379 INFO:tasks.workunit.client.1.smithi027.stdout:8/778: link d1/d3/d1c/d32/d41/d52/cd6 d1/d3/d1c/d43/c10b 0 2024-04-22T15:50:09.382 INFO:tasks.workunit.client.1.smithi027.stdout:0/993: mknod d0/d2a/d6a/da6/c13e 0 2024-04-22T15:50:09.389 INFO:tasks.workunit.client.1.smithi027.stdout:7/825: dwrite d0/d11/d1c/fd3 [0,4194304] 0 2024-04-22T15:50:09.390 INFO:tasks.workunit.client.1.smithi027.stdout:7/826: fsync d0/fcc 0 2024-04-22T15:50:09.391 INFO:tasks.workunit.client.1.smithi027.stdout:0/994: symlink d0/d2a/d91/de5/l13f 0 2024-04-22T15:50:09.395 INFO:tasks.workunit.client.1.smithi027.stdout:8/779: dwrite d1/d3/d1c/d32/d41/d52/d61/d8f/dd3/ff1 [0,4194304] 0 2024-04-22T15:50:09.396 INFO:tasks.workunit.client.1.smithi027.stdout:7/827: stat d0/d4c/d8b/d72/db2/fd0 0 2024-04-22T15:50:09.396 INFO:tasks.workunit.client.1.smithi027.stdout:0/995: chown d0/d28/d63/d116/d133 699584472 1 2024-04-22T15:50:09.397 INFO:tasks.workunit.client.1.smithi027.stdout:0/996: write d0/d9d/db7/ff0 [1333699,43161] 0 2024-04-22T15:50:09.400 INFO:tasks.workunit.client.1.smithi027.stdout:0/997: read d0/d9d/db7/fa9 [39055,61181] 0 2024-04-22T15:50:09.402 INFO:tasks.workunit.client.1.smithi027.stdout:8/780: rename d1/d3/d1c/d32/d41/l81 to d1/d3/l10c 0 2024-04-22T15:50:09.406 INFO:tasks.workunit.client.1.smithi027.stdout:0/998: mkdir d0/d34/d4e/d85/d140 0 2024-04-22T15:50:09.411 INFO:tasks.workunit.client.1.smithi027.stdout:7/828: rename d0/d11/d1c/d2c/d96/db3 to d0/d11/d3a/d46/d94/d5a/d119 0 2024-04-22T15:50:09.412 INFO:tasks.workunit.client.1.smithi027.stdout:8/781: write d1/d3/d1c/d43/f71 [3054528,23776] 0 2024-04-22T15:50:09.413 INFO:tasks.workunit.client.1.smithi027.stdout:7/829: write d0/d4c/d8b/d92/f112 [1323965,85240] 0 2024-04-22T15:50:09.416 INFO:tasks.workunit.client.1.smithi027.stdout:2/863: dread dd/df/d70/d4b/f68 [0,4194304] 0 2024-04-22T15:50:09.420 INFO:tasks.workunit.client.1.smithi027.stdout:0/999: rename d0/d9d/db7 to d0/da3/dae/d141 0 2024-04-22T15:50:09.425 INFO:tasks.workunit.client.1.smithi027.stdout:8/782: creat d1/d3/d1c/d49/d94/f10d x:0 0 0 2024-04-22T15:50:09.434 INFO:tasks.workunit.client.1.smithi027.stdout:8/783: creat d1/d3/f10e x:0 0 0 2024-04-22T15:50:09.435 INFO:tasks.workunit.client.1.smithi027.stdout:8/784: write d1/d3/d1c/d32/d41/d52/f60 [4102072,105245] 0 2024-04-22T15:50:09.439 INFO:tasks.workunit.client.1.smithi027.stdout:2/864: rename dd/df/d1b/d96/dc5/fd6 to dd/df/d70/d4b/d52/d9a/da4/dad/d8d/f109 0 2024-04-22T15:50:09.442 INFO:tasks.workunit.client.1.smithi027.stdout:8/785: unlink d1/d3/f59 0 2024-04-22T15:50:09.443 INFO:tasks.workunit.client.1.smithi027.stdout:7/830: write d0/d49/d9f/f9b [5188705,96634] 0 2024-04-22T15:50:09.445 INFO:tasks.workunit.client.1.smithi027.stdout:2/865: mknod dd/df/d1b/d1f/c10a 0 2024-04-22T15:50:09.446 INFO:tasks.workunit.client.1.smithi027.stdout:8/786: symlink d1/d3/d1c/d32/d41/d52/d61/l10f 0 2024-04-22T15:50:09.447 INFO:tasks.workunit.client.1.smithi027.stdout:8/787: write d1/d3/d36/d88/da9/ff8 [1297810,41048] 0 2024-04-22T15:50:09.450 INFO:tasks.workunit.client.1.smithi027.stdout:7/831: mknod d0/d11/d3a/d46/d94/d5a/c11a 0 2024-04-22T15:50:09.450 INFO:tasks.workunit.client.1.smithi027.stdout:7/832: dread - d0/d11/d1c/d31/f111 zero size 2024-04-22T15:50:09.453 INFO:tasks.workunit.client.1.smithi027.stdout:8/788: rename d1/d3/d1c/d32/d41/c4a to d1/d1e/dbc/c110 0 2024-04-22T15:50:09.461 INFO:tasks.workunit.client.1.smithi027.stdout:7/833: dread d0/d4c/d8b/d72/db2/f42 [0,4194304] 0 2024-04-22T15:50:09.465 INFO:tasks.workunit.client.1.smithi027.stdout:7/834: rename d0/d4c/d8b/d72/cf3 to d0/d4c/d8b/c11b 0 2024-04-22T15:50:09.471 INFO:tasks.workunit.client.1.smithi027.stdout:2/866: write dd/df/d70/d4b/d4e/daf/ffb [349498,114081] 0 2024-04-22T15:50:09.472 INFO:tasks.workunit.client.1.smithi027.stdout:7/835: dwrite d0/d4c/d8b/d72/db2/fac [0,4194304] 0 2024-04-22T15:50:09.474 INFO:tasks.workunit.client.1.smithi027.stdout:2/867: fsync dd/df/d1b/fab 0 2024-04-22T15:50:09.481 INFO:tasks.workunit.client.1.smithi027.stdout:8/789: dwrite d1/d3/d1c/d28/d42/ffe [0,4194304] 0 2024-04-22T15:50:09.481 INFO:tasks.workunit.client.1.smithi027.stdout:7/836: dread d0/d11/f4d [0,4194304] 0 2024-04-22T15:50:09.486 INFO:tasks.workunit.client.1.smithi027.stdout:8/790: chown d1/d3/d1c/d32/d41/d52/d61/fa6 664517 1 2024-04-22T15:50:09.486 INFO:tasks.workunit.client.1.smithi027.stdout:7/837: unlink d0/d3f/c99 0 2024-04-22T15:50:09.491 INFO:tasks.workunit.client.1.smithi027.stdout:7/838: mkdir d0/d11/d1c/d2c/d93/d11c 0 2024-04-22T15:50:09.492 INFO:tasks.workunit.client.1.smithi027.stdout:2/868: dwrite dd/df/d1b/dc9/dca/ffa [0,4194304] 0 2024-04-22T15:50:09.493 INFO:tasks.workunit.client.1.smithi027.stdout:8/791: mkdir d1/d3/d1c/d32/d41/d52/d61/d111 0 2024-04-22T15:50:09.501 INFO:tasks.workunit.client.1.smithi027.stdout:8/792: creat d1/d3/d36/d57/df0/f112 x:0 0 0 2024-04-22T15:50:09.503 INFO:tasks.workunit.client.1.smithi027.stdout:8/793: dread d1/d3/d1c/f34 [0,4194304] 0 2024-04-22T15:50:09.504 INFO:tasks.workunit.client.1.smithi027.stdout:2/869: write dd/f7b [282369,118424] 0 2024-04-22T15:50:09.504 INFO:tasks.workunit.client.1.smithi027.stdout:7/839: write d0/d4c/d8b/d92/da3/da8/fef [3852163,122239] 0 2024-04-22T15:50:09.508 INFO:tasks.workunit.client.1.smithi027.stdout:7/840: readlink d0/d4c/d8b/d92/da3/da8/l117 0 2024-04-22T15:50:09.508 INFO:tasks.workunit.client.1.smithi027.stdout:7/841: chown d0/d4c/d8b/d72/db2/lea 30814 1 2024-04-22T15:50:09.509 INFO:tasks.workunit.client.1.smithi027.stdout:2/870: creat dd/dae/dd4/ddb/f10b x:0 0 0 2024-04-22T15:50:09.510 INFO:tasks.workunit.client.1.smithi027.stdout:8/794: creat d1/d3/d1c/d28/d42/def/f113 x:0 0 0 2024-04-22T15:50:09.518 INFO:tasks.workunit.client.1.smithi027.stdout:7/842: dwrite d0/d4c/d8b/d92/da3/da8/f3d [0,4194304] 0 2024-04-22T15:50:09.522 INFO:tasks.workunit.client.1.smithi027.stdout:8/795: dwrite d1/d3/d1c/d43/d5f/d7d/fa3 [0,4194304] 0 2024-04-22T15:50:09.530 INFO:tasks.workunit.client.1.smithi027.stdout:3/844: sync 2024-04-22T15:50:09.538 INFO:tasks.workunit.client.1.smithi027.stdout:2/871: dwrite dd/df/d1b/d96/fc3 [0,4194304] 0 2024-04-22T15:50:09.538 INFO:tasks.workunit.client.1.smithi027.stdout:3/845: dread d28/d5d/d79/fa1 [0,4194304] 0 2024-04-22T15:50:09.540 INFO:tasks.workunit.client.1.smithi027.stdout:7/843: dwrite d0/d11/fe [0,4194304] 0 2024-04-22T15:50:09.541 INFO:tasks.workunit.client.1.smithi027.stdout:8/796: rmdir d1/d3/d1c/d28/d42/dc4/d106 0 2024-04-22T15:50:09.542 INFO:tasks.workunit.client.1.smithi027.stdout:7/844: dread - d0/d4c/d8b/d72/db2/de7/f105 zero size 2024-04-22T15:50:09.547 INFO:tasks.workunit.client.1.smithi027.stdout:2/872: creat dd/df/d70/d4b/d52/d9a/d105/f10c x:0 0 0 2024-04-22T15:50:09.548 INFO:tasks.workunit.client.1.smithi027.stdout:2/873: truncate dd/df/d1b/ffc 433957 0 2024-04-22T15:50:09.550 INFO:tasks.workunit.client.1.smithi027.stdout:3/846: creat d28/d5d/db0/dce/dda/f10b x:0 0 0 2024-04-22T15:50:09.551 INFO:tasks.workunit.client.1.smithi027.stdout:3/847: readlink d28/d4b/d52/db2/deb/df3/l103 0 2024-04-22T15:50:09.556 INFO:tasks.workunit.client.1.smithi027.stdout:8/797: rename d1/d3/d1c/d32/d41/fed to d1/d3/d1c/d32/d41/d52/d61/d8f/ddd/f114 0 2024-04-22T15:50:09.560 INFO:tasks.workunit.client.1.smithi027.stdout:3/848: mkdir d28/d4b/dc6/d10c 0 2024-04-22T15:50:09.564 INFO:tasks.workunit.client.1.smithi027.stdout:2/874: creat dd/df/d1b/d96/f10d x:0 0 0 2024-04-22T15:50:09.565 INFO:tasks.workunit.client.1.smithi027.stdout:3/849: mknod d28/d5d/d62/dcf/c10d 0 2024-04-22T15:50:09.566 INFO:tasks.workunit.client.1.smithi027.stdout:7/845: dwrite d0/d11/d1c/d2c/f38 [0,4194304] 0 2024-04-22T15:50:09.573 INFO:tasks.workunit.client.1.smithi027.stdout:7/846: creat d0/d4c/d4e/df5/f11d x:0 0 0 2024-04-22T15:50:09.574 INFO:tasks.workunit.client.1.smithi027.stdout:2/875: fdatasync dd/df/d70/f61 0 2024-04-22T15:50:09.579 INFO:tasks.workunit.client.1.smithi027.stdout:3/850: rename d28/d5d/d79/lbc to d28/d4b/d52/db2/d101/l10e 0 2024-04-22T15:50:09.581 INFO:tasks.workunit.client.1.smithi027.stdout:7/847: dwrite d0/f29 [4194304,4194304] 0 2024-04-22T15:50:09.588 INFO:tasks.workunit.client.1.smithi027.stdout:3/851: unlink d28/f49 0 2024-04-22T15:50:09.590 INFO:tasks.workunit.client.1.smithi027.stdout:7/848: dwrite d0/d11/d3a/d46/d94/fba [0,4194304] 0 2024-04-22T15:50:09.592 INFO:tasks.workunit.client.1.smithi027.stdout:7/849: truncate d0/d4c/d8b/f116 105661 0 2024-04-22T15:50:09.593 INFO:tasks.workunit.client.1.smithi027.stdout:7/850: write d0/d3f/f48 [8912318,3954] 0 2024-04-22T15:50:09.593 INFO:tasks.workunit.client.1.smithi027.stdout:3/852: fsync d28/f3c 0 2024-04-22T15:50:09.595 INFO:tasks.workunit.client.1.smithi027.stdout:3/853: readlink d28/d4b/d52/d94/l9f 0 2024-04-22T15:50:09.596 INFO:tasks.workunit.client.1.smithi027.stdout:7/851: write d0/d4c/d8b/d92/da3/da8/f88 [4376290,7023] 0 2024-04-22T15:50:09.599 INFO:tasks.workunit.client.1.smithi027.stdout:3/854: creat d28/d5d/db0/dce/dda/f10f x:0 0 0 2024-04-22T15:50:09.602 INFO:tasks.workunit.client.1.smithi027.stdout:7/852: rename d0/d4c/d4e/f73 to d0/d4c/f11e 0 2024-04-22T15:50:09.607 INFO:tasks.workunit.client.1.smithi027.stdout:2/876: dread dd/df/d1b/f106 [0,4194304] 0 2024-04-22T15:50:09.607 INFO:tasks.workunit.client.1.smithi027.stdout:3/855: dwrite d28/f36 [0,4194304] 0 2024-04-22T15:50:09.609 INFO:tasks.workunit.client.1.smithi027.stdout:7/853: write d0/d11/d1c/d2c/d93/f59 [1945523,9590] 0 2024-04-22T15:50:09.617 INFO:tasks.workunit.client.1.smithi027.stdout:2/877: unlink dd/df/d1b/d38/d60/d75/ddf/c101 0 2024-04-22T15:50:09.618 INFO:tasks.workunit.client.1.smithi027.stdout:2/878: chown dd/df/d1b/d38/dd5/ff7 11 1 2024-04-22T15:50:09.619 INFO:tasks.workunit.client.1.smithi027.stdout:3/856: mkdir d28/d4b/d52/d94/d105/d110 0 2024-04-22T15:50:09.620 INFO:tasks.workunit.client.1.smithi027.stdout:7/854: mkdir d0/d11f 0 2024-04-22T15:50:09.621 INFO:tasks.workunit.client.1.smithi027.stdout:7/855: write d0/d11/d1c/ddc/ffd [2102145,34406] 0 2024-04-22T15:50:09.629 INFO:tasks.workunit.client.1.smithi027.stdout:7/856: dwrite d0/d11/d1c/d31/d8a/ff8 [0,4194304] 0 2024-04-22T15:50:09.630 INFO:tasks.workunit.client.1.smithi027.stdout:7/857: fdatasync d0/d4c/fd4 0 2024-04-22T15:50:09.639 INFO:tasks.workunit.client.1.smithi027.stdout:2/879: dread dd/df/d1b/d96/fbd [0,4194304] 0 2024-04-22T15:50:09.640 INFO:tasks.workunit.client.1.smithi027.stdout:3/857: getdents d28/d4b/d52/db2/d95/d9c/de9 0 2024-04-22T15:50:09.642 INFO:tasks.workunit.client.1.smithi027.stdout:7/858: link d0/d4c/d8b/la9 d0/d4c/d8b/d92/l120 0 2024-04-22T15:50:09.643 INFO:tasks.workunit.client.1.smithi027.stdout:7/859: stat d0/d11/d3a/fa7 0 2024-04-22T15:50:09.644 INFO:tasks.workunit.client.1.smithi027.stdout:3/858: mknod d28/d5d/db0/dce/dda/c111 0 2024-04-22T15:50:09.644 INFO:tasks.workunit.client.1.smithi027.stdout:7/860: readlink d0/d4c/d8b/d92/da3/da8/l2e 0 2024-04-22T15:50:09.645 INFO:tasks.workunit.client.1.smithi027.stdout:3/859: chown d28/d5d/d79/fb8 12 1 2024-04-22T15:50:09.646 INFO:tasks.workunit.client.1.smithi027.stdout:2/880: rename dd/d43/f95 to dd/df/d1b/d38/d60/f10e 0 2024-04-22T15:50:09.648 INFO:tasks.workunit.client.1.smithi027.stdout:7/861: unlink d0/d11/d1c/d2c/cb0 0 2024-04-22T15:50:09.650 INFO:tasks.workunit.client.1.smithi027.stdout:3/860: chown d28/d4b/d52/f61 26 1 2024-04-22T15:50:09.660 INFO:tasks.workunit.client.1.smithi027.stdout:3/861: link d28/d4b/d52/d94/dae/cd8 d28/d4b/d52/d94/d105/d110/c112 0 2024-04-22T15:50:09.660 INFO:tasks.workunit.client.1.smithi027.stdout:2/881: dread dd/df/d1b/d96/dc5/dd1/fde [0,4194304] 0 2024-04-22T15:50:09.670 INFO:tasks.workunit.client.1.smithi027.stdout:7/862: dwrite d0/f4b [0,4194304] 0 2024-04-22T15:50:09.672 INFO:tasks.workunit.client.1.smithi027.stdout:7/863: read d0/d11/d3a/d46/d94/fba [1436798,43154] 0 2024-04-22T15:50:09.687 INFO:tasks.workunit.client.1.smithi027.stdout:7/864: dread d0/d11/d1c/d31/d62/faf [0,4194304] 0 2024-04-22T15:50:09.693 INFO:tasks.workunit.client.1.smithi027.stdout:7/865: dwrite d0/fa0 [0,4194304] 0 2024-04-22T15:50:09.713 INFO:tasks.workunit.client.1.smithi027.stdout:7/866: dread d0/d11/d1c/d2c/fc2 [0,4194304] 0 2024-04-22T15:50:09.714 INFO:tasks.workunit.client.1.smithi027.stdout:7/867: fsync d0/d11/d1c/d31/d8a/ff8 0 2024-04-22T15:50:09.718 INFO:tasks.workunit.client.1.smithi027.stdout:7/868: read - d0/d4c/d8b/d72/db2/de7/f105 zero size 2024-04-22T15:50:09.724 INFO:tasks.workunit.client.1.smithi027.stdout:3/862: rename d28/d5d/d62/dcf/cdd to d28/d4b/d52/d94/c113 0 2024-04-22T15:50:09.725 INFO:tasks.workunit.client.1.smithi027.stdout:3/863: chown d28/d5d/d62 0 1 2024-04-22T15:50:09.735 INFO:tasks.workunit.client.1.smithi027.stdout:3/864: rename d28/d4b/d52/db2/d95/d9c/dc1/fe7 to d28/d5d/db0/dce/f114 0 2024-04-22T15:50:09.736 INFO:tasks.workunit.client.1.smithi027.stdout:7/869: link d0/d11/d1c/ddc/f113 d0/d11/d3a/d46/d94/d5a/f121 0 2024-04-22T15:50:09.739 INFO:tasks.workunit.client.1.smithi027.stdout:3/865: unlink d28/d5d/db0/dce/le2 0 2024-04-22T15:50:09.747 INFO:tasks.workunit.client.1.smithi027.stdout:7/870: creat d0/f122 x:0 0 0 2024-04-22T15:50:09.748 INFO:tasks.workunit.client.1.smithi027.stdout:7/871: chown d0/f122 10699008 1 2024-04-22T15:50:09.751 INFO:tasks.workunit.client.1.smithi027.stdout:7/872: read d0/d11/d1c/d31/f5f [27451,81876] 0 2024-04-22T15:50:09.763 INFO:tasks.workunit.client.1.smithi027.stdout:3/866: dwrite d28/f38 [0,4194304] 0 2024-04-22T15:50:09.775 INFO:tasks.workunit.client.1.smithi027.stdout:3/867: dwrite d28/d5d/fa8 [0,4194304] 0 2024-04-22T15:50:09.797 INFO:tasks.workunit.client.1.smithi027.stdout:3/868: write d28/d5d/d62/fb1 [1401452,34520] 0 2024-04-22T15:50:09.797 INFO:tasks.workunit.client.1.smithi027.stdout:4/772: sync 2024-04-22T15:50:09.801 INFO:tasks.workunit.client.1.smithi027.stdout:3/869: creat d28/d4b/d52/d94/dc4/f115 x:0 0 0 2024-04-22T15:50:09.805 INFO:tasks.workunit.client.1.smithi027.stdout:3/870: readlink d28/d5d/d62/lb7 0 2024-04-22T15:50:09.806 INFO:tasks.workunit.client.1.smithi027.stdout:4/773: dwrite d5/d45/f7d [0,4194304] 0 2024-04-22T15:50:09.810 INFO:tasks.workunit.client.1.smithi027.stdout:3/871: dread d28/d5d/f8b [0,4194304] 0 2024-04-22T15:50:09.810 INFO:tasks.workunit.client.1.smithi027.stdout:3/872: stat d28/d4b/d52/f7a 0 2024-04-22T15:50:09.811 INFO:tasks.workunit.client.1.smithi027.stdout:3/873: chown d28/d5d 57 1 2024-04-22T15:50:09.819 INFO:tasks.workunit.client.1.smithi027.stdout:4/774: creat d5/d11/d1d/f119 x:0 0 0 2024-04-22T15:50:09.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:09 smithi027 ceph-mon[34922]: pgmap v8: 65 pgs: 65 active+clean; 3.0 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 30 MiB/s rd, 62 MiB/s wr, 189 op/s 2024-04-22T15:50:09.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:09 smithi027 ceph-mon[34922]: Standby manager daemon smithi027.ukdgzs started 2024-04-22T15:50:09.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:09 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.27:0/1290047407' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.ukdgzs/crt"}]: dispatch 2024-04-22T15:50:09.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:09 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.27:0/1290047407' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:50:09.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:09 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.27:0/1290047407' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.ukdgzs/key"}]: dispatch 2024-04-22T15:50:09.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:09 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.27:0/1290047407' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:50:09.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:09 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:09.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:09 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:09.837 INFO:tasks.workunit.client.1.smithi027.stdout:4/775: truncate d5/d45/d47/fac 4191918 0 2024-04-22T15:50:09.865 INFO:tasks.workunit.client.1.smithi027.stdout:4/776: dwrite d5/d7/fb [8388608,4194304] 0 2024-04-22T15:50:09.868 INFO:tasks.workunit.client.1.smithi027.stdout:4/777: write d5/de/fbb [3534712,52850] 0 2024-04-22T15:50:09.879 INFO:tasks.workunit.client.1.smithi027.stdout:4/778: getdents d5/d45/ddc 0 2024-04-22T15:50:09.882 INFO:tasks.workunit.client.1.smithi027.stdout:4/779: dread d5/d11/d1d/d7b/f92 [0,4194304] 0 2024-04-22T15:50:09.883 INFO:tasks.workunit.client.1.smithi027.stdout:4/780: write d5/de/fbb [2545985,128862] 0 2024-04-22T15:50:09.900 INFO:tasks.workunit.client.1.smithi027.stdout:4/781: write d5/de/d2a/d58/f67 [902101,24927] 0 2024-04-22T15:50:09.905 INFO:tasks.workunit.client.1.smithi027.stdout:4/782: truncate d5/d11/d1d/d7b/f92 126936 0 2024-04-22T15:50:09.910 INFO:tasks.workunit.client.1.smithi027.stdout:4/783: chown d5/de/d2a/d58/f104 3 1 2024-04-22T15:50:09.911 INFO:tasks.workunit.client.1.smithi027.stdout:4/784: chown d5/d11/d6c/da3/d10b 24329 1 2024-04-22T15:50:09.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:09 smithi016 ceph-mon[26620]: pgmap v8: 65 pgs: 65 active+clean; 3.0 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 30 MiB/s rd, 62 MiB/s wr, 189 op/s 2024-04-22T15:50:09.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:09 smithi016 ceph-mon[26620]: Standby manager daemon smithi027.ukdgzs started 2024-04-22T15:50:09.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:09 smithi016 ceph-mon[26620]: from='mgr.? 172.21.15.27:0/1290047407' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.ukdgzs/crt"}]: dispatch 2024-04-22T15:50:09.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:09 smithi016 ceph-mon[26620]: from='mgr.? 172.21.15.27:0/1290047407' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:50:09.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:09 smithi016 ceph-mon[26620]: from='mgr.? 172.21.15.27:0/1290047407' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.ukdgzs/key"}]: dispatch 2024-04-22T15:50:09.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:09 smithi016 ceph-mon[26620]: from='mgr.? 172.21.15.27:0/1290047407' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:50:09.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:09 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:09.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:09 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:09.933 INFO:tasks.workunit.client.1.smithi027.stdout:4/785: dread d5/de/d2a/d58/f68 [0,4194304] 0 2024-04-22T15:50:09.936 INFO:tasks.workunit.client.1.smithi027.stdout:4/786: mknod d5/de/df2/de2/c11a 0 2024-04-22T15:50:09.938 INFO:tasks.workunit.client.1.smithi027.stdout:4/787: mknod d5/d7/dc6/c11b 0 2024-04-22T15:50:09.941 INFO:tasks.workunit.client.1.smithi027.stdout:4/788: creat d5/d11/d1d/d54/d8c/f11c x:0 0 0 2024-04-22T15:50:09.945 INFO:tasks.workunit.client.1.smithi027.stdout:4/789: rename d5/dea to d5/de/d11d 0 2024-04-22T15:50:09.952 INFO:tasks.workunit.client.1.smithi027.stdout:4/790: dwrite d5/d11/d1d/d24/f9c [0,4194304] 0 2024-04-22T15:50:09.998 INFO:tasks.workunit.client.1.smithi027.stdout:5/785: sync 2024-04-22T15:50:09.998 INFO:tasks.workunit.client.1.smithi027.stdout:9/732: sync 2024-04-22T15:50:09.998 INFO:tasks.workunit.client.1.smithi027.stdout:1/682: sync 2024-04-22T15:50:09.998 INFO:tasks.workunit.client.1.smithi027.stdout:6/860: sync 2024-04-22T15:50:10.007 INFO:tasks.workunit.client.1.smithi027.stdout:6/861: fsync d5/dc/dd/dd0/f97 0 2024-04-22T15:50:10.009 INFO:tasks.workunit.client.1.smithi027.stdout:4/791: dwrite d5/d11/dc1/fd2 [0,4194304] 0 2024-04-22T15:50:10.011 INFO:tasks.workunit.client.1.smithi027.stdout:9/733: rename d0/dc/d12/d30/d5b/d91/f9f to d0/dc/d12/dee/ff3 0 2024-04-22T15:50:10.014 INFO:tasks.workunit.client.1.smithi027.stdout:5/786: mknod d0/d97/dbf/cfd 0 2024-04-22T15:50:10.014 INFO:tasks.workunit.client.1.smithi027.stdout:1/683: rmdir d2/d36/d74 39 2024-04-22T15:50:10.018 INFO:tasks.workunit.client.1.smithi027.stdout:6/862: mknod d5/dc/dd/d79/d7b/d9a/c12c 0 2024-04-22T15:50:10.027 INFO:tasks.workunit.client.1.smithi027.stdout:6/863: link d5/dc/dd/d79/d7b/d5d/l6b d5/dc/d17/d10e/l12d 0 2024-04-22T15:50:10.029 INFO:tasks.workunit.client.1.smithi027.stdout:6/864: mkdir d5/db/d2e/d59/df0/d12e 0 2024-04-22T15:50:10.030 INFO:tasks.workunit.client.1.smithi027.stdout:5/787: dread d0/df/f12 [0,4194304] 0 2024-04-22T15:50:10.032 INFO:tasks.workunit.client.1.smithi027.stdout:6/865: mknod d5/dc/d17/d11c/c12f 0 2024-04-22T15:50:10.032 INFO:tasks.workunit.client.1.smithi027.stdout:6/866: readlink d5/laa 0 2024-04-22T15:50:10.038 INFO:tasks.workunit.client.1.smithi027.stdout:6/867: dwrite d5/dc/d26/d49/d86/ffb [0,4194304] 0 2024-04-22T15:50:10.046 INFO:tasks.workunit.client.1.smithi027.stdout:5/788: dread d0/df/de/f18 [0,4194304] 0 2024-04-22T15:50:10.047 INFO:tasks.workunit.client.1.smithi027.stdout:6/868: dread d5/f75 [0,4194304] 0 2024-04-22T15:50:10.048 INFO:tasks.workunit.client.1.smithi027.stdout:6/869: dread - d5/dc/fa6 zero size 2024-04-22T15:50:10.048 INFO:tasks.workunit.client.1.smithi027.stdout:6/870: chown d5/db/d2e/d59/db5 9 1 2024-04-22T15:50:10.049 INFO:tasks.workunit.client.1.smithi027.stdout:6/871: stat d5/dc/d17 0 2024-04-22T15:50:10.049 INFO:tasks.workunit.client.1.smithi027.stdout:5/789: creat d0/d1a/d22/d26/d32/d88/ffe x:0 0 0 2024-04-22T15:50:10.050 INFO:tasks.workunit.client.1.smithi027.stdout:5/790: readlink d0/df/d10/l39 0 2024-04-22T15:50:10.053 INFO:tasks.workunit.client.1.smithi027.stdout:5/791: unlink d0/db6/fe6 0 2024-04-22T15:50:10.054 INFO:tasks.workunit.client.1.smithi027.stdout:5/792: mkdir d0/d1a/d22/d72/dff 0 2024-04-22T15:50:10.058 INFO:tasks.workunit.client.1.smithi027.stdout:5/793: rename d0/d1a/d22/d26/d5d/de3/d3e/c7c to d0/df/de/d40/d89/dd0/c100 0 2024-04-22T15:50:10.058 INFO:tasks.workunit.client.1.smithi027.stdout:5/794: chown d0/d1a/d22/d26/d5d/de3/d3e/ca0 16 1 2024-04-22T15:50:10.113 INFO:tasks.workunit.client.1.smithi027.stdout:4/792: dwrite d5/d42/f61 [0,4194304] 0 2024-04-22T15:50:10.114 INFO:tasks.workunit.client.1.smithi027.stdout:9/734: dwrite d0/dc/d12/d58/d6b/d6e/f31 [0,4194304] 0 2024-04-22T15:50:10.115 INFO:tasks.workunit.client.1.smithi027.stdout:4/793: readlink d5/d11/d1d/d7b/ld8 0 2024-04-22T15:50:10.115 INFO:tasks.workunit.client.1.smithi027.stdout:4/794: readlink d5/lf1 0 2024-04-22T15:50:10.126 INFO:tasks.workunit.client.1.smithi027.stdout:6/872: write d5/dc/d17/d53/f72 [4041910,113510] 0 2024-04-22T15:50:10.126 INFO:tasks.workunit.client.1.smithi027.stdout:4/795: dwrite d5/de/f10d [0,4194304] 0 2024-04-22T15:50:10.131 INFO:tasks.workunit.client.1.smithi027.stdout:5/795: write d0/db6/fcf [1426406,36824] 0 2024-04-22T15:50:10.132 INFO:tasks.workunit.client.1.smithi027.stdout:6/873: mknod d5/dc/dd/dd0/c130 0 2024-04-22T15:50:10.133 INFO:tasks.workunit.client.1.smithi027.stdout:4/796: truncate d5/d7/f2c 3345620 0 2024-04-22T15:50:10.135 INFO:tasks.workunit.client.1.smithi027.stdout:4/797: write d5/d11/f80 [4526928,26757] 0 2024-04-22T15:50:10.137 INFO:tasks.workunit.client.1.smithi027.stdout:9/735: link d0/dc/d12/c55 d0/dc/d12/d30/d5b/d91/daf/cf4 0 2024-04-22T15:50:10.138 INFO:tasks.workunit.client.1.smithi027.stdout:6/874: mkdir d5/dc/d17/d8a/d131 0 2024-04-22T15:50:10.140 INFO:tasks.workunit.client.1.smithi027.stdout:5/796: dwrite d0/df/de/d24/d62/f7f [0,4194304] 0 2024-04-22T15:50:10.148 INFO:tasks.workunit.client.1.smithi027.stdout:9/736: unlink d0/dc/d12/d58/c7e 0 2024-04-22T15:50:10.151 INFO:tasks.workunit.client.1.smithi027.stdout:5/797: symlink d0/df/d3d/l101 0 2024-04-22T15:50:10.155 INFO:tasks.workunit.client.1.smithi027.stdout:6/875: dread d5/dc/dd/dd0/d34/fd5 [0,4194304] 0 2024-04-22T15:50:10.156 INFO:tasks.workunit.client.1.smithi027.stdout:9/737: dwrite d0/dc/d12/d58/d6b/f62 [0,4194304] 0 2024-04-22T15:50:10.157 INFO:tasks.workunit.client.1.smithi027.stdout:5/798: mkdir d0/d97/d102 0 2024-04-22T15:50:10.165 INFO:tasks.workunit.client.1.smithi027.stdout:5/799: dwrite d0/d1a/d22/d26/d32/d88/d90/fd4 [0,4194304] 0 2024-04-22T15:50:10.169 INFO:tasks.workunit.client.1.smithi027.stdout:4/798: write d5/d11/d6c/f112 [1020542,6763] 0 2024-04-22T15:50:10.170 INFO:tasks.workunit.client.1.smithi027.stdout:4/799: readlink d5/d11/d6c/l9d 0 2024-04-22T15:50:10.173 INFO:tasks.workunit.client.1.smithi027.stdout:9/738: mkdir d0/dc/d12/d30/d5b/dbc/df5 0 2024-04-22T15:50:10.175 INFO:tasks.workunit.client.1.smithi027.stdout:5/800: unlink d0/d1a/d22/d26/d5d/fda 0 2024-04-22T15:50:10.176 INFO:tasks.workunit.client.1.smithi027.stdout:6/876: write d5/dc/d17/d108/f10a [367149,50010] 0 2024-04-22T15:50:10.181 INFO:tasks.workunit.client.1.smithi027.stdout:9/739: rename d0/dc/d12/d30/d99/db4/fde to d0/dc/d12/d30/d5b/d91/da9/ff6 0 2024-04-22T15:50:10.183 INFO:tasks.workunit.client.1.smithi027.stdout:5/801: creat d0/d1a/d22/d26/d5d/de3/f103 x:0 0 0 2024-04-22T15:50:10.186 INFO:tasks.workunit.client.1.smithi027.stdout:6/877: rename d5/dc/d17/d53/fa1 to d5/dc/dae/f132 0 2024-04-22T15:50:10.188 INFO:tasks.workunit.client.1.smithi027.stdout:4/800: dread d5/d11/f80 [0,4194304] 0 2024-04-22T15:50:10.190 INFO:tasks.workunit.client.1.smithi027.stdout:9/740: mkdir d0/dc/de/df7 0 2024-04-22T15:50:10.193 INFO:tasks.workunit.client.1.smithi027.stdout:4/801: mkdir d5/de/da9/d11e 0 2024-04-22T15:50:10.197 INFO:tasks.workunit.client.1.smithi027.stdout:9/741: rmdir d0/dc/d12/d58/d6b 39 2024-04-22T15:50:10.207 INFO:tasks.workunit.client.1.smithi027.stdout:6/878: truncate d5/dc/dd/dd0/d34/f65 3683831 0 2024-04-22T15:50:10.209 INFO:tasks.workunit.client.1.smithi027.stdout:9/742: mknod d0/dc/d12/d58/cf8 0 2024-04-22T15:50:10.210 INFO:tasks.workunit.client.1.smithi027.stdout:5/802: dwrite d0/df/de/d24/fa8 [0,4194304] 0 2024-04-22T15:50:10.210 INFO:tasks.workunit.client.1.smithi027.stdout:6/879: creat d5/d30/d54/dd8/f133 x:0 0 0 2024-04-22T15:50:10.216 INFO:tasks.workunit.client.1.smithi027.stdout:4/802: dwrite d5/d11/da6/fae [0,4194304] 0 2024-04-22T15:50:10.217 INFO:tasks.workunit.client.1.smithi027.stdout:4/803: readlink d5/de/da9/lbc 0 2024-04-22T15:50:10.224 INFO:tasks.workunit.client.1.smithi027.stdout:6/880: fsync d5/dc/d26/d3d/f51 0 2024-04-22T15:50:10.228 INFO:tasks.workunit.client.1.smithi027.stdout:9/743: getdents d0/dc/d12/d30/d5b/d91/daf 0 2024-04-22T15:50:10.233 INFO:tasks.workunit.client.1.smithi027.stdout:9/744: mkdir d0/dc/df9 0 2024-04-22T15:50:10.237 INFO:tasks.workunit.client.1.smithi027.stdout:9/745: rename d0/d64/f87 to d0/dc/d12/d30/d99/db4/ffa 0 2024-04-22T15:50:10.238 INFO:tasks.workunit.client.1.smithi027.stdout:5/803: write d0/d1a/d22/d26/d32/f4f [5237864,6969] 0 2024-04-22T15:50:10.240 INFO:tasks.workunit.client.1.smithi027.stdout:5/804: write d0/d97/dd7/ff3 [519256,70545] 0 2024-04-22T15:50:10.242 INFO:tasks.workunit.client.1.smithi027.stdout:9/746: mknod d0/dc/d12/d30/d5b/d91/daf/cfb 0 2024-04-22T15:50:10.246 INFO:tasks.workunit.client.1.smithi027.stdout:5/805: mknod d0/df/d10/c104 0 2024-04-22T15:50:10.247 INFO:tasks.workunit.client.1.smithi027.stdout:4/804: dwrite d5/d7/f2c [0,4194304] 0 2024-04-22T15:50:10.248 INFO:tasks.workunit.client.1.smithi027.stdout:5/806: write d0/d1a/d22/d26/d5d/de3/d3e/fbe [1141182,36823] 0 2024-04-22T15:50:10.250 INFO:tasks.workunit.client.1.smithi027.stdout:9/747: mknod d0/cfc 0 2024-04-22T15:50:10.255 INFO:tasks.workunit.client.1.smithi027.stdout:5/807: creat d0/db6/f105 x:0 0 0 2024-04-22T15:50:10.255 INFO:tasks.workunit.client.1.smithi027.stdout:4/805: dread d5/d7/f8f [0,4194304] 0 2024-04-22T15:50:10.258 INFO:tasks.workunit.client.1.smithi027.stdout:6/881: dwrite d5/dc/d26/d3d/f83 [0,4194304] 0 2024-04-22T15:50:10.258 INFO:tasks.workunit.client.1.smithi027.stdout:4/806: dread - d5/d11/d1d/d7b/de9/ffd zero size 2024-04-22T15:50:10.262 INFO:tasks.workunit.client.1.smithi027.stdout:6/882: creat d5/dc/d17/d8a/f134 x:0 0 0 2024-04-22T15:50:10.263 INFO:tasks.workunit.client.1.smithi027.stdout:4/807: symlink d5/d45/d47/l11f 0 2024-04-22T15:50:10.263 INFO:tasks.workunit.client.1.smithi027.stdout:6/883: chown d5/dc/d20/da3/fcc 357 1 2024-04-22T15:50:10.265 INFO:tasks.workunit.client.1.smithi027.stdout:6/884: chown d5/dc/dd/dd0/d34/l68 4 1 2024-04-22T15:50:10.268 INFO:tasks.workunit.client.1.smithi027.stdout:4/808: creat d5/d11/d1d/d24/dfc/f120 x:0 0 0 2024-04-22T15:50:10.269 INFO:tasks.workunit.client.1.smithi027.stdout:6/885: fdatasync d5/dc/dd/d79/d7b/fc1 0 2024-04-22T15:50:10.273 INFO:tasks.workunit.client.1.smithi027.stdout:6/886: readlink d5/dc/dd/ld4 0 2024-04-22T15:50:10.277 INFO:tasks.workunit.client.1.smithi027.stdout:9/748: truncate d0/dc/d12/f23 1080947 0 2024-04-22T15:50:10.277 INFO:tasks.workunit.client.1.smithi027.stdout:4/809: getdents d5/de/d30 0 2024-04-22T15:50:10.279 INFO:tasks.workunit.client.1.smithi027.stdout:6/887: mkdir d5/dc/d17/d11c/d11d/d135 0 2024-04-22T15:50:10.283 INFO:tasks.workunit.client.1.smithi027.stdout:4/810: fsync d5/de/f21 0 2024-04-22T15:50:10.285 INFO:tasks.workunit.client.1.smithi027.stdout:6/888: dread d5/dc/dd/d79/d7b/d5d/f70 [4194304,4194304] 0 2024-04-22T15:50:10.285 INFO:tasks.workunit.client.1.smithi027.stdout:5/808: dread d0/d1a/f3a [0,4194304] 0 2024-04-22T15:50:10.295 INFO:tasks.workunit.client.1.smithi027.stdout:5/809: fdatasync d0/d1a/d22/d26/fc4 0 2024-04-22T15:50:10.300 INFO:tasks.workunit.client.1.smithi027.stdout:9/749: getdents d0/dc/d12/d58/da6 0 2024-04-22T15:50:10.302 INFO:tasks.workunit.client.1.smithi027.stdout:4/811: creat d5/d11/d1d/d24/f121 x:0 0 0 2024-04-22T15:50:10.308 INFO:tasks.workunit.client.1.smithi027.stdout:4/812: symlink d5/d45/l122 0 2024-04-22T15:50:10.310 INFO:tasks.workunit.client.1.smithi027.stdout:9/750: write d0/dc/de/fc5 [378475,72668] 0 2024-04-22T15:50:10.310 INFO:tasks.workunit.client.1.smithi027.stdout:5/810: link d0/d1a/l1e d0/df/d10/l106 0 2024-04-22T15:50:10.310 INFO:tasks.workunit.client.1.smithi027.stdout:6/889: link d5/db/d2e/d59/db5/l121 d5/dc/d17/l136 0 2024-04-22T15:50:10.313 INFO:tasks.workunit.client.1.smithi027.stdout:6/890: unlink d5/d30/d54/c11e 0 2024-04-22T15:50:10.314 INFO:tasks.workunit.client.1.smithi027.stdout:5/811: mkdir d0/d1a/d22/d26/d32/d88/d90/d107 0 2024-04-22T15:50:10.318 INFO:tasks.workunit.client.1.smithi027.stdout:5/812: mknod d0/d1a/d22/d26/d32/d88/c108 0 2024-04-22T15:50:10.321 INFO:tasks.workunit.client.1.smithi027.stdout:6/891: truncate d5/dc/dd/d79/d7b/d5d/fb9 2250032 0 2024-04-22T15:50:10.323 INFO:tasks.workunit.client.1.smithi027.stdout:6/892: write d5/db/d2e/f62 [4889036,96990] 0 2024-04-22T15:50:10.323 INFO:tasks.workunit.client.1.smithi027.stdout:9/751: getdents d0/dc/d12/d30/d5b/d91 0 2024-04-22T15:50:10.334 INFO:tasks.workunit.client.1.smithi027.stdout:9/752: mkdir d0/dc/d12/d58/ddb/dfd 0 2024-04-22T15:50:10.334 INFO:tasks.workunit.client.1.smithi027.stdout:5/813: write d0/d1a/d22/d26/f48 [1980356,52526] 0 2024-04-22T15:50:10.334 INFO:tasks.workunit.client.1.smithi027.stdout:6/893: dwrite d5/dc/d26/d3d/f7c [0,4194304] 0 2024-04-22T15:50:10.340 INFO:tasks.workunit.client.1.smithi027.stdout:9/753: mkdir d0/dc/d12/d30/d5b/d91/da9/dfe 0 2024-04-22T15:50:10.342 INFO:tasks.workunit.client.1.smithi027.stdout:6/894: link d5/dc/d17/d8a/fbb d5/dc/dd/dd0/d34/f137 0 2024-04-22T15:50:10.344 INFO:tasks.workunit.client.1.smithi027.stdout:9/754: dwrite d0/d64/f7b [0,4194304] 0 2024-04-22T15:50:10.380 INFO:tasks.workunit.client.1.smithi027.stdout:9/755: write d0/dc/d12/d58/da6/fae [107866,120826] 0 2024-04-22T15:50:10.383 INFO:tasks.workunit.client.1.smithi027.stdout:9/756: rename d0/dc/d12/d30/d99/fab to d0/dc/d12/fff 0 2024-04-22T15:50:10.392 INFO:tasks.workunit.client.1.smithi027.stdout:9/757: dwrite d0/dc/d12/d30/d5b/f65 [0,4194304] 0 2024-04-22T15:50:10.395 INFO:tasks.workunit.client.1.smithi027.stdout:9/758: mkdir d0/dc/d100 0 2024-04-22T15:50:10.502 INFO:tasks.workunit.client.1.smithi027.stdout:9/759: symlink d0/dc/de/df7/l101 0 2024-04-22T15:50:10.503 INFO:tasks.workunit.client.1.smithi027.stdout:9/760: readlink d0/d51/d9d/lcf 0 2024-04-22T15:50:10.503 INFO:tasks.workunit.client.1.smithi027.stdout:9/761: chown d0/dc/f8e 7104750 1 2024-04-22T15:50:10.507 INFO:tasks.workunit.client.1.smithi027.stdout:9/762: rename d0/dc/d12/d58/d6b/d6e/fbd to d0/dc/de/df7/f102 0 2024-04-22T15:50:10.509 INFO:tasks.workunit.client.1.smithi027.stdout:9/763: fsync d0/f47 0 2024-04-22T15:50:10.514 INFO:tasks.workunit.client.1.smithi027.stdout:9/764: rename d0/dc/l17 to d0/dc/de/db5/l103 0 2024-04-22T15:50:10.521 INFO:tasks.workunit.client.1.smithi027.stdout:9/765: dread d0/dc/d12/f23 [0,4194304] 0 2024-04-22T15:50:10.526 INFO:tasks.workunit.client.1.smithi027.stdout:9/766: rename d0/dc/d12/d58 to d0/dc/d12/d30/d5b/dbc/df5/d104 0 2024-04-22T15:50:10.529 INFO:tasks.workunit.client.1.smithi027.stdout:9/767: creat d0/dc/d12/d30/d5b/dbc/df5/d104/ddb/dfd/f105 x:0 0 0 2024-04-22T15:50:10.531 INFO:tasks.workunit.client.1.smithi027.stdout:9/768: unlink d0/dc/c5a 0 2024-04-22T15:50:10.533 INFO:tasks.workunit.client.1.smithi027.stdout:9/769: creat d0/dc/d12/d30/d5b/d91/f106 x:0 0 0 2024-04-22T15:50:10.534 INFO:tasks.workunit.client.1.smithi027.stdout:9/770: stat d0/dc/d12/d30/d5b/dbc/df5/d104/da6/feb 0 2024-04-22T15:50:10.538 INFO:tasks.workunit.client.1.smithi027.stdout:9/771: dread - d0/dc/d12/d30/d5b/d91/da9/fc8 zero size 2024-04-22T15:50:10.540 INFO:tasks.workunit.client.1.smithi027.stdout:9/772: unlink d0/l13 0 2024-04-22T15:50:10.544 INFO:tasks.workunit.client.1.smithi027.stdout:9/773: mkdir d0/dc/de/d107 0 2024-04-22T15:50:10.550 INFO:tasks.workunit.client.1.smithi027.stdout:8/798: sync 2024-04-22T15:50:10.552 INFO:tasks.workunit.client.1.smithi027.stdout:9/774: write d0/dc/de/df7/f102 [1080301,6100] 0 2024-04-22T15:50:10.559 INFO:tasks.workunit.client.1.smithi027.stdout:8/799: write d1/d3/d1c/d32/d41/d52/fd0 [763292,40846] 0 2024-04-22T15:50:10.562 INFO:tasks.workunit.client.1.smithi027.stdout:9/775: creat d0/dc/d12/d30/d5b/dbc/df5/d104/d6b/f108 x:0 0 0 2024-04-22T15:50:10.563 INFO:tasks.workunit.client.1.smithi027.stdout:9/776: dread - d0/dc/d12/d30/d5b/dbc/df5/d104/ddb/ff2 zero size 2024-04-22T15:50:10.567 INFO:tasks.workunit.client.1.smithi027.stdout:8/800: symlink d1/d3/d1c/d32/l115 0 2024-04-22T15:50:10.569 INFO:tasks.workunit.client.1.smithi027.stdout:8/801: write d1/d1e/d47/d53/d76/fdf [514381,17411] 0 2024-04-22T15:50:10.573 INFO:tasks.workunit.client.1.smithi027.stdout:8/802: mkdir d1/d3/d1c/d32/d41/d52/d61/d8f/ddd/dbe/d116 0 2024-04-22T15:50:10.573 INFO:tasks.workunit.client.1.smithi027.stdout:8/803: stat d1/d1e/l29 0 2024-04-22T15:50:10.578 INFO:tasks.workunit.client.1.smithi027.stdout:8/804: creat d1/d3/d1c/d32/d41/d52/dec/f117 x:0 0 0 2024-04-22T15:50:10.580 INFO:tasks.workunit.client.1.smithi027.stdout:9/777: dwrite d0/dc/d12/f23 [0,4194304] 0 2024-04-22T15:50:10.585 INFO:tasks.workunit.client.1.smithi027.stdout:9/778: read d0/dc/f72 [1328737,14895] 0 2024-04-22T15:50:10.587 INFO:tasks.workunit.client.1.smithi027.stdout:9/779: read - d0/dc/d12/d30/d5b/dbc/df5/d104/da6/fd2 zero size 2024-04-22T15:50:10.588 INFO:tasks.workunit.client.1.smithi027.stdout:9/780: chown d0/dc/c4b 124 1 2024-04-22T15:50:10.591 INFO:tasks.workunit.client.1.smithi027.stdout:9/781: mknod d0/dc/d12/d30/d5b/d91/c109 0 2024-04-22T15:50:10.593 INFO:tasks.workunit.client.1.smithi027.stdout:8/805: dwrite d1/d3/d1c/d32/fc5 [0,4194304] 0 2024-04-22T15:50:10.602 INFO:tasks.workunit.client.1.smithi027.stdout:9/782: write d0/f68 [1706211,40590] 0 2024-04-22T15:50:10.603 INFO:tasks.workunit.client.1.smithi027.stdout:8/806: creat d1/d3/d1c/d32/d41/d52/d61/f118 x:0 0 0 2024-04-22T15:50:10.604 INFO:tasks.workunit.client.1.smithi027.stdout:8/807: chown d1/d4f 1795602 1 2024-04-22T15:50:10.608 INFO:tasks.workunit.client.1.smithi027.stdout:8/808: fsync d1/d1e/f4c 0 2024-04-22T15:50:10.625 INFO:tasks.workunit.client.1.smithi027.stdout:2/882: sync 2024-04-22T15:50:10.687 INFO:tasks.workunit.client.1.smithi027.stdout:9/783: dwrite d0/dc/de/f40 [0,4194304] 0 2024-04-22T15:50:10.688 INFO:tasks.workunit.client.1.smithi027.stdout:9/784: write d0/d51/d9d/fad [2630728,126329] 0 2024-04-22T15:50:10.692 INFO:tasks.workunit.client.1.smithi027.stdout:4/813: sync 2024-04-22T15:50:10.692 INFO:tasks.workunit.client.1.smithi027.stdout:5/814: sync 2024-04-22T15:50:10.693 INFO:tasks.workunit.client.1.smithi027.stdout:7/873: sync 2024-04-22T15:50:10.693 INFO:tasks.workunit.client.1.smithi027.stdout:6/895: sync 2024-04-22T15:50:10.693 INFO:tasks.workunit.client.1.smithi027.stdout:1/684: sync 2024-04-22T15:50:10.693 INFO:tasks.workunit.client.1.smithi027.stdout:3/874: sync 2024-04-22T15:50:10.695 INFO:tasks.workunit.client.1.smithi027.stdout:1/685: fdatasync d2/d1d/f5b 0 2024-04-22T15:50:10.736 INFO:tasks.workunit.client.1.smithi027.stdout:8/809: rename d1/d3/d36/f85 to d1/d3/d1c/d56/dc9/f119 0 2024-04-22T15:50:10.746 INFO:tasks.workunit.client.1.smithi027.stdout:2/883: link dd/df/d1b/d96/dc5/dd1/fe5 dd/df/d1b/d38/dd5/f10f 0 2024-04-22T15:50:10.750 INFO:tasks.workunit.client.1.smithi027.stdout:9/785: creat d0/dc/d12/d30/d5b/dbc/df5/d104/d6b/f10a x:0 0 0 2024-04-22T15:50:10.751 INFO:tasks.workunit.client.1.smithi027.stdout:9/786: write d0/dc/f80 [982462,5651] 0 2024-04-22T15:50:10.752 INFO:tasks.workunit.client.1.smithi027.stdout:9/787: chown d0/f36 2 1 2024-04-22T15:50:10.753 INFO:tasks.workunit.client.1.smithi027.stdout:7/874: mknod d0/d4c/d4e/df5/c123 0 2024-04-22T15:50:10.758 INFO:tasks.workunit.client.1.smithi027.stdout:3/875: symlink d28/d4b/d52/db2/d95/d9c/l116 0 2024-04-22T15:50:10.760 INFO:tasks.workunit.client.1.smithi027.stdout:3/876: chown d28/f44 5137 1 2024-04-22T15:50:10.760 INFO:tasks.workunit.client.1.smithi027.stdout:4/814: fsync d5/d45/d81/f98 0 2024-04-22T15:50:10.761 INFO:tasks.workunit.client.1.smithi027.stdout:7/875: dwrite d0/d4c/d8b/f116 [0,4194304] 0 2024-04-22T15:50:10.770 INFO:tasks.workunit.client.1.smithi027.stdout:2/884: truncate dd/d47/f7f 1426128 0 2024-04-22T15:50:10.772 INFO:tasks.workunit.client.1.smithi027.stdout:2/885: read dd/df/d70/d4b/f68 [143604,95320] 0 2024-04-22T15:50:10.787 INFO:tasks.workunit.client.1.smithi027.stdout:8/810: mkdir d1/d3/d1c/d11a 0 2024-04-22T15:50:10.792 INFO:tasks.workunit.client.1.smithi027.stdout:9/788: mknod d0/dc/df9/c10b 0 2024-04-22T15:50:10.794 INFO:tasks.workunit.client.1.smithi027.stdout:3/877: mknod d28/d4b/d52/c117 0 2024-04-22T15:50:10.794 INFO:tasks.workunit.client.1.smithi027.stdout:3/878: chown d28/d4b/d52/d94/fe0 36854 1 2024-04-22T15:50:10.795 INFO:tasks.workunit.client.1.smithi027.stdout:3/879: chown d28/d5d/c73 1614 1 2024-04-22T15:50:10.795 INFO:tasks.workunit.client.1.smithi027.stdout:4/815: mkdir d5/d11/d1d/d24/d123 0 2024-04-22T15:50:10.796 INFO:tasks.workunit.client.1.smithi027.stdout:5/815: link d0/d1a/d22/d26/c2c d0/d97/dd7/c109 0 2024-04-22T15:50:10.797 INFO:tasks.workunit.client.1.smithi027.stdout:5/816: fdatasync d0/d97/dd7/ff3 0 2024-04-22T15:50:10.799 INFO:tasks.workunit.client.1.smithi027.stdout:8/811: mknod d1/d3/d1c/d32/d41/d52/c11b 0 2024-04-22T15:50:10.810 INFO:tasks.workunit.client.1.smithi027.stdout:9/789: creat d0/d51/de6/f10c x:0 0 0 2024-04-22T15:50:10.810 INFO:tasks.workunit.client.1.smithi027.stdout:4/816: dwrite d5/d11/d1d/ffa [0,4194304] 0 2024-04-22T15:50:10.811 INFO:tasks.workunit.client.1.smithi027.stdout:3/880: creat d28/d4b/d52/db2/d95/d9c/de9/f118 x:0 0 0 2024-04-22T15:50:10.813 INFO:tasks.workunit.client.1.smithi027.stdout:4/817: write d5/d45/d5a/f96 [4155370,105343] 0 2024-04-22T15:50:10.814 INFO:tasks.workunit.client.1.smithi027.stdout:8/812: dread d1/d1e/f33 [0,4194304] 0 2024-04-22T15:50:10.814 INFO:tasks.workunit.client.1.smithi027.stdout:7/876: link d0/d11/d1c/d31/d56/lcb d0/d11/d1c/d2c/l124 0 2024-04-22T15:50:10.818 INFO:tasks.workunit.client.1.smithi027.stdout:8/813: fsync d1/d3/d1c/d32/d41/d52/d61/d8f/dd3/ff1 0 2024-04-22T15:50:10.820 INFO:tasks.workunit.client.1.smithi027.stdout:5/817: getdents d0/d97/d102 0 2024-04-22T15:50:10.822 INFO:tasks.workunit.client.1.smithi027.stdout:5/818: truncate d0/d1a/d22/d26/d5d/de3/f103 928770 0 2024-04-22T15:50:10.828 INFO:tasks.workunit.client.1.smithi027.stdout:3/881: mkdir d28/d4b/d52/db2/d95/d9c/de9/d119 0 2024-04-22T15:50:10.833 INFO:tasks.workunit.client.1.smithi027.stdout:8/814: truncate d1/d3/d1c/d32/d41/d52/d61/fb2 139089 0 2024-04-22T15:50:10.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:10 smithi027 ceph-mon[34922]: mgrmap e30: smithi016.rnsnnr(active, since 11s), standbys: smithi027.ukdgzs 2024-04-22T15:50:10.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:10 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr metadata", "who": "smithi027.ukdgzs", "id": "smithi027.ukdgzs"}]: dispatch 2024-04-22T15:50:10.838 INFO:tasks.workunit.client.1.smithi027.stdout:5/819: dread d0/f6 [0,4194304] 0 2024-04-22T15:50:10.842 INFO:tasks.workunit.client.1.smithi027.stdout:3/882: unlink d28/d4b/c58 0 2024-04-22T15:50:10.844 INFO:tasks.workunit.client.1.smithi027.stdout:5/820: rmdir d0/df/d3d 39 2024-04-22T15:50:10.845 INFO:tasks.workunit.client.1.smithi027.stdout:7/877: getdents d0/d11/d1c/d31/d56 0 2024-04-22T15:50:10.847 INFO:tasks.workunit.client.1.smithi027.stdout:3/883: rename d28/d4b/d52/d94/dde to d28/d4b/d52/d94/d105/d11a 0 2024-04-22T15:50:10.850 INFO:tasks.workunit.client.1.smithi027.stdout:7/878: rename d0/d4c/d4e/l8f to d0/d4c/d8b/d72/l125 0 2024-04-22T15:50:10.852 INFO:tasks.workunit.client.1.smithi027.stdout:3/884: write d28/f56 [3473514,43445] 0 2024-04-22T15:50:10.855 INFO:tasks.workunit.client.1.smithi027.stdout:7/879: truncate d0/d11/d3a/fa7 1024283 0 2024-04-22T15:50:10.860 INFO:tasks.workunit.client.1.smithi027.stdout:3/885: creat d28/d4b/dc6/f11b x:0 0 0 2024-04-22T15:50:10.861 INFO:tasks.workunit.client.1.smithi027.stdout:3/886: readlink d28/l75 0 2024-04-22T15:50:10.862 INFO:tasks.workunit.client.1.smithi027.stdout:7/880: dwrite d0/d4c/d8b/d92/da3/da8/fef [0,4194304] 0 2024-04-22T15:50:10.868 INFO:tasks.workunit.client.1.smithi027.stdout:3/887: truncate d28/d5d/db0/fbd 1684260 0 2024-04-22T15:50:10.869 INFO:tasks.workunit.client.1.smithi027.stdout:3/888: fsync d28/d4b/dc6/fff 0 2024-04-22T15:50:10.871 INFO:tasks.workunit.client.1.smithi027.stdout:7/881: dwrite d0/d4c/d8b/d92/da3/da8/f88 [4194304,4194304] 0 2024-04-22T15:50:10.873 INFO:tasks.workunit.client.1.smithi027.stdout:7/882: write d0/d49/f114 [321663,8963] 0 2024-04-22T15:50:10.881 INFO:tasks.workunit.client.1.smithi027.stdout:7/883: mkdir d0/d11/d1c/d2c/d96/d126 0 2024-04-22T15:50:10.886 INFO:tasks.workunit.client.1.smithi027.stdout:7/884: symlink d0/d11/d1c/d2c/d93/l127 0 2024-04-22T15:50:10.892 INFO:tasks.workunit.client.1.smithi027.stdout:7/885: symlink d0/d11/d1c/d31/d62/l128 0 2024-04-22T15:50:10.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:10 smithi016 ceph-mon[26620]: mgrmap e30: smithi016.rnsnnr(active, since 11s), standbys: smithi027.ukdgzs 2024-04-22T15:50:10.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:10 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr metadata", "who": "smithi027.ukdgzs", "id": "smithi027.ukdgzs"}]: dispatch 2024-04-22T15:50:10.934 INFO:tasks.workunit.client.1.smithi027.stdout:6/896: dwrite d5/f89 [0,4194304] 0 2024-04-22T15:50:10.935 INFO:tasks.workunit.client.1.smithi027.stdout:1/686: write d2/d5f/d64/d81/d3d/d82/d85/fa4 [5396,86825] 0 2024-04-22T15:50:10.939 INFO:tasks.workunit.client.1.smithi027.stdout:1/687: symlink d2/d5f/d64/d81/d7e/le5 0 2024-04-22T15:50:10.946 INFO:tasks.workunit.client.1.smithi027.stdout:7/886: creat d0/d4c/f129 x:0 0 0 2024-04-22T15:50:10.947 INFO:tasks.workunit.client.1.smithi027.stdout:1/688: creat d2/d5f/d64/d81/d3d/fe6 x:0 0 0 2024-04-22T15:50:10.947 INFO:tasks.workunit.client.1.smithi027.stdout:1/689: chown d2/f7f 424 1 2024-04-22T15:50:10.948 INFO:tasks.workunit.client.1.smithi027.stdout:1/690: fsync d2/d5f/d64/d81/d3d/d44/fd8 0 2024-04-22T15:50:10.950 INFO:tasks.workunit.client.1.smithi027.stdout:2/886: write dd/df/f23 [1996264,64444] 0 2024-04-22T15:50:10.950 INFO:tasks.workunit.client.1.smithi027.stdout:7/887: dwrite d0/d11/d1c/d31/faa [0,4194304] 0 2024-04-22T15:50:10.955 INFO:tasks.workunit.client.1.smithi027.stdout:2/887: unlink dd/f19 0 2024-04-22T15:50:10.962 INFO:tasks.workunit.client.1.smithi027.stdout:2/888: link dd/df/d1b/d38/d60/d75/ddf/ff3 dd/df/d1b/d38/d60/d75/d9f/f110 0 2024-04-22T15:50:10.962 INFO:tasks.workunit.client.1.smithi027.stdout:2/889: chown dd/df/d1b/d1f/f36 13607641 1 2024-04-22T15:50:10.968 INFO:tasks.workunit.client.1.smithi027.stdout:2/890: dread dd/df/d70/f6b [0,4194304] 0 2024-04-22T15:50:10.969 INFO:tasks.workunit.client.1.smithi027.stdout:9/790: dwrite d0/dc/d12/d30/d5b/dbc/df5/d104/f89 [0,4194304] 0 2024-04-22T15:50:10.970 INFO:tasks.workunit.client.1.smithi027.stdout:2/891: chown dd/df/d1b/f41 123 1 2024-04-22T15:50:10.974 INFO:tasks.workunit.client.1.smithi027.stdout:9/791: unlink d0/dc/d12/d30/d5b/dbc/df5/d104/d6b/f62 0 2024-04-22T15:50:10.976 INFO:tasks.workunit.client.1.smithi027.stdout:4/818: truncate d5/de/d2a/d58/faf 3990445 0 2024-04-22T15:50:10.980 INFO:tasks.workunit.client.1.smithi027.stdout:9/792: creat d0/dc/d12/d30/d5b/d91/daf/f10d x:0 0 0 2024-04-22T15:50:10.982 INFO:tasks.workunit.client.1.smithi027.stdout:2/892: creat dd/f111 x:0 0 0 2024-04-22T15:50:10.989 INFO:tasks.workunit.client.1.smithi027.stdout:9/793: link d0/dc/d12/d30/d5b/dbc/df5/d104/ddb/ff2 d0/dc/d12/d30/d5b/d91/daf/f10e 0 2024-04-22T15:50:10.991 INFO:tasks.workunit.client.1.smithi027.stdout:9/794: write d0/dc/d12/d30/d5b/dbc/df5/d104/fb7 [3006119,44992] 0 2024-04-22T15:50:10.991 INFO:tasks.workunit.client.1.smithi027.stdout:2/893: mkdir dd/df/d1b/d38/d112 0 2024-04-22T15:50:10.996 INFO:tasks.workunit.client.1.smithi027.stdout:9/795: creat d0/dc/d12/d30/d5b/d91/da9/dfe/f10f x:0 0 0 2024-04-22T15:50:10.999 INFO:tasks.workunit.client.1.smithi027.stdout:9/796: creat d0/dc/de/db5/f110 x:0 0 0 2024-04-22T15:50:11.000 INFO:tasks.workunit.client.1.smithi027.stdout:2/894: dwrite dd/df/d1b/d96/dc5/f100 [0,4194304] 0 2024-04-22T15:50:11.002 INFO:tasks.workunit.client.1.smithi027.stdout:9/797: chown d0/dc/d12/d30/d5b/d91/da9/dfe/f10f 0 1 2024-04-22T15:50:11.002 INFO:tasks.workunit.client.1.smithi027.stdout:5/821: truncate d0/df/de/d40/d89/fcc 1060957 0 2024-04-22T15:50:11.003 INFO:tasks.workunit.client.1.smithi027.stdout:5/822: truncate d0/df/de/feb 245250 0 2024-04-22T15:50:11.004 INFO:tasks.workunit.client.1.smithi027.stdout:5/823: dread - d0/db6/f105 zero size 2024-04-22T15:50:11.009 INFO:tasks.workunit.client.1.smithi027.stdout:5/824: creat d0/d1a/d22/d26/d5d/de3/d8f/dce/f10a x:0 0 0 2024-04-22T15:50:11.011 INFO:tasks.workunit.client.1.smithi027.stdout:5/825: write d0/d1a/d22/d26/d32/d88/d90/fd4 [703856,6458] 0 2024-04-22T15:50:11.013 INFO:tasks.workunit.client.1.smithi027.stdout:5/826: chown d0/df/d10/l1d 27 1 2024-04-22T15:50:11.015 INFO:tasks.workunit.client.1.smithi027.stdout:5/827: fdatasync d0/df/de/d24/f2e 0 2024-04-22T15:50:11.015 INFO:tasks.workunit.client.1.smithi027.stdout:2/895: unlink dd/df/d1b/d96/fbd 0 2024-04-22T15:50:11.021 INFO:tasks.workunit.client.1.smithi027.stdout:5/828: creat d0/d1a/d22/d26/d32/d88/d90/d107/f10b x:0 0 0 2024-04-22T15:50:11.042 INFO:tasks.workunit.client.1.smithi027.stdout:4/819: dwrite d5/de/f16 [4194304,4194304] 0 2024-04-22T15:50:11.046 INFO:tasks.workunit.client.1.smithi027.stdout:4/820: creat d5/d11/da6/f124 x:0 0 0 2024-04-22T15:50:11.055 INFO:tasks.workunit.client.1.smithi027.stdout:9/798: dwrite d0/dc/fa3 [0,4194304] 0 2024-04-22T15:50:11.058 INFO:tasks.workunit.client.1.smithi027.stdout:4/821: link d5/d11/f8d d5/d7/dc6/f125 0 2024-04-22T15:50:11.060 INFO:tasks.workunit.client.1.smithi027.stdout:4/822: write d5/d45/d5a/f96 [3846573,25329] 0 2024-04-22T15:50:11.060 INFO:tasks.workunit.client.1.smithi027.stdout:2/896: write dd/df/d70/d4b/d4e/f56 [2201952,43680] 0 2024-04-22T15:50:11.062 INFO:tasks.workunit.client.1.smithi027.stdout:4/823: chown d5/d11/dc1/fd2 16274607 1 2024-04-22T15:50:11.062 INFO:tasks.workunit.client.1.smithi027.stdout:2/897: stat dd/df/d1b/d38/d60/d75/d9f 0 2024-04-22T15:50:11.067 INFO:tasks.workunit.client.1.smithi027.stdout:9/799: dread d0/dc/d12/d30/f67 [0,4194304] 0 2024-04-22T15:50:11.069 INFO:tasks.workunit.client.1.smithi027.stdout:5/829: dwrite d0/d1a/d22/fc6 [0,4194304] 0 2024-04-22T15:50:11.077 INFO:tasks.workunit.client.1.smithi027.stdout:4/824: mknod d5/de/da9/d11e/c126 0 2024-04-22T15:50:11.078 INFO:tasks.workunit.client.1.smithi027.stdout:9/800: creat d0/dc/d12/d30/d5b/dbc/df5/d104/d6b/d6e/f111 x:0 0 0 2024-04-22T15:50:11.091 INFO:tasks.workunit.client.1.smithi027.stdout:2/898: dread dd/df/d70/d4b/d52/fb6 [0,4194304] 0 2024-04-22T15:50:11.093 INFO:tasks.workunit.client.1.smithi027.stdout:5/830: mkdir d0/df/de/d24/d10c 0 2024-04-22T15:50:11.094 INFO:tasks.workunit.client.1.smithi027.stdout:9/801: dread d0/dc/de/f6c [0,4194304] 0 2024-04-22T15:50:11.096 INFO:tasks.workunit.client.1.smithi027.stdout:2/899: dread dd/df/d1b/d38/d60/fa2 [0,4194304] 0 2024-04-22T15:50:11.099 INFO:tasks.workunit.client.1.smithi027.stdout:4/825: write d5/d7/f1f [352460,103503] 0 2024-04-22T15:50:11.103 INFO:tasks.workunit.client.1.smithi027.stdout:2/900: mkdir dd/dae/dd4/ddb/d113 0 2024-04-22T15:50:11.108 INFO:tasks.workunit.client.1.smithi027.stdout:9/802: write d0/dc/d12/f94 [5124458,59629] 0 2024-04-22T15:50:11.109 INFO:tasks.workunit.client.1.smithi027.stdout:9/803: stat d0/dc/d12/d30/d5b/dbc/df5/d104/f7d 0 2024-04-22T15:50:11.110 INFO:tasks.workunit.client.1.smithi027.stdout:4/826: write d5/d7/dc6/fdb [136179,62586] 0 2024-04-22T15:50:11.112 INFO:tasks.workunit.client.1.smithi027.stdout:2/901: symlink dd/df/d70/d4b/d52/d9a/da4/dad/d8d/l114 0 2024-04-22T15:50:11.118 INFO:tasks.workunit.client.1.smithi027.stdout:5/831: dread d0/df/f45 [0,4194304] 0 2024-04-22T15:50:11.121 INFO:tasks.workunit.client.1.smithi027.stdout:5/832: mkdir d0/d1a/d22/d26/d5d/de3/d8f/dce/d10d 0 2024-04-22T15:50:11.124 INFO:tasks.workunit.client.1.smithi027.stdout:2/902: write dd/d43/f55 [2241324,107503] 0 2024-04-22T15:50:11.129 INFO:tasks.workunit.client.1.smithi027.stdout:9/804: dwrite d0/dc/de/f41 [0,4194304] 0 2024-04-22T15:50:11.130 INFO:tasks.workunit.client.1.smithi027.stdout:5/833: getdents d0/db6 0 2024-04-22T15:50:11.133 INFO:tasks.workunit.client.1.smithi027.stdout:4/827: dread d5/de/d2a/d58/faf [0,4194304] 0 2024-04-22T15:50:11.134 INFO:tasks.workunit.client.1.smithi027.stdout:2/903: mkdir dd/df/d1b/d38/de8/d115 0 2024-04-22T15:50:11.139 INFO:tasks.workunit.client.1.smithi027.stdout:2/904: rmdir dd/df/d70/d4b 39 2024-04-22T15:50:11.143 INFO:tasks.workunit.client.1.smithi027.stdout:4/828: dread d5/d11/d1d/d24/f79 [0,4194304] 0 2024-04-22T15:50:11.149 INFO:tasks.workunit.client.1.smithi027.stdout:2/905: dread - dd/df/d70/d4b/d52/d9a/d105/f10c zero size 2024-04-22T15:50:11.150 INFO:tasks.workunit.client.1.smithi027.stdout:5/834: dwrite d0/d1a/d22/d72/f7b [0,4194304] 0 2024-04-22T15:50:11.150 INFO:tasks.workunit.client.1.smithi027.stdout:4/829: rmdir d5/de 39 2024-04-22T15:50:11.150 INFO:tasks.workunit.client.1.smithi027.stdout:9/805: link d0/dc/d12/d30/la5 d0/dc/d12/d30/d5b/l112 0 2024-04-22T15:50:11.153 INFO:tasks.workunit.client.1.smithi027.stdout:5/835: stat d0/df/d3d/c4b 0 2024-04-22T15:50:11.153 INFO:tasks.workunit.client.1.smithi027.stdout:5/836: chown d0/d97/dd7 1145 1 2024-04-22T15:50:11.153 INFO:tasks.workunit.client.1.smithi027.stdout:2/906: fsync dd/df/d1b/d38/d60/f80 0 2024-04-22T15:50:11.154 INFO:tasks.workunit.client.1.smithi027.stdout:5/837: readlink d0/df/d10/l6e 0 2024-04-22T15:50:11.154 INFO:tasks.workunit.client.1.smithi027.stdout:4/830: creat d5/d45/ddc/f127 x:0 0 0 2024-04-22T15:50:11.155 INFO:tasks.workunit.client.1.smithi027.stdout:5/838: fsync d0/d1a/d22/d26/d32/d79/dd6/fe9 0 2024-04-22T15:50:11.155 INFO:tasks.workunit.client.1.smithi027.stdout:9/806: rename d0/d8/ce4 to d0/dc/d12/d30/dd4/c113 0 2024-04-22T15:50:11.158 INFO:tasks.workunit.client.1.smithi027.stdout:2/907: unlink dd/df/d1b/d38/d60/f83 0 2024-04-22T15:50:11.160 INFO:tasks.workunit.client.1.smithi027.stdout:9/807: symlink d0/dc/d12/d30/d5b/dbc/df5/d104/d6b/d6e/l114 0 2024-04-22T15:50:11.168 INFO:tasks.workunit.client.1.smithi027.stdout:9/808: creat d0/dc/d12/d30/d5b/d91/da9/f115 x:0 0 0 2024-04-22T15:50:11.169 INFO:tasks.workunit.client.1.smithi027.stdout:4/831: link d5/d11/l22 d5/d11/d1d/d7b/l128 0 2024-04-22T15:50:11.171 INFO:tasks.workunit.client.1.smithi027.stdout:5/839: dwrite d0/df/de/d24/d62/f68 [0,4194304] 0 2024-04-22T15:50:11.174 INFO:tasks.workunit.client.1.smithi027.stdout:9/809: creat d0/dc/d12/d30/d99/f116 x:0 0 0 2024-04-22T15:50:11.179 INFO:tasks.workunit.client.1.smithi027.stdout:2/908: dwrite dd/f1e [0,4194304] 0 2024-04-22T15:50:11.180 INFO:tasks.workunit.client.1.smithi027.stdout:5/840: truncate d0/df/de/d24/d62/f7f 4943846 0 2024-04-22T15:50:11.181 INFO:tasks.workunit.client.1.smithi027.stdout:9/810: fdatasync d0/dc/d12/d30/d5b/dbc/df5/d104/ddb/dfd/f105 0 2024-04-22T15:50:11.182 INFO:tasks.workunit.client.1.smithi027.stdout:4/832: dwrite d5/da8/fef [0,4194304] 0 2024-04-22T15:50:11.187 INFO:tasks.workunit.client.1.smithi027.stdout:4/833: dread - d5/d11/d1d/d54/d8c/f106 zero size 2024-04-22T15:50:11.188 INFO:tasks.workunit.client.1.smithi027.stdout:2/909: write dd/df/d70/f45 [956654,13861] 0 2024-04-22T15:50:11.193 INFO:tasks.workunit.client.1.smithi027.stdout:5/841: fdatasync d0/df/de/d24/fb8 0 2024-04-22T15:50:11.196 INFO:tasks.workunit.client.1.smithi027.stdout:4/834: dwrite d5/d45/ddc/f127 [0,4194304] 0 2024-04-22T15:50:11.202 INFO:tasks.workunit.client.1.smithi027.stdout:5/842: rename d0/d1a/d22/d26/d5d/de3/cba to d0/d1a/d22/d72/dac/c10e 0 2024-04-22T15:50:11.206 INFO:tasks.workunit.client.1.smithi027.stdout:9/811: write d0/dc/d12/d30/d5b/dbc/df5/d104/ddb/ff2 [715049,120776] 0 2024-04-22T15:50:11.206 INFO:tasks.workunit.client.1.smithi027.stdout:4/835: dwrite d5/d11/d1d/d7b/de9/ffd [0,4194304] 0 2024-04-22T15:50:11.208 INFO:tasks.workunit.client.1.smithi027.stdout:9/812: stat d0/dc/d12/d30/d99/da7/fda 0 2024-04-22T15:50:11.209 INFO:tasks.workunit.client.1.smithi027.stdout:5/843: creat d0/df/de/f10f x:0 0 0 2024-04-22T15:50:11.215 INFO:tasks.workunit.client.1.smithi027.stdout:5/844: dread d0/df/de/d24/d62/f68 [0,4194304] 0 2024-04-22T15:50:11.229 INFO:tasks.workunit.client.1.smithi027.stdout:5/845: dwrite d0/df/d10/f33 [0,4194304] 0 2024-04-22T15:50:11.229 INFO:tasks.workunit.client.1.smithi027.stdout:9/813: truncate d0/f36 702779 0 2024-04-22T15:50:11.229 INFO:tasks.workunit.client.1.smithi027.stdout:2/910: dread dd/f35 [0,4194304] 0 2024-04-22T15:50:11.229 INFO:tasks.workunit.client.1.smithi027.stdout:4/836: dwrite d5/d45/d5a/f82 [0,4194304] 0 2024-04-22T15:50:11.231 INFO:tasks.workunit.client.1.smithi027.stdout:4/837: fdatasync d5/d45/f7d 0 2024-04-22T15:50:11.233 INFO:tasks.workunit.client.1.smithi027.stdout:2/911: dread - dd/df/d70/d4b/d4e/daf/fb9 zero size 2024-04-22T15:50:11.235 INFO:tasks.workunit.client.1.smithi027.stdout:5/846: read d0/df/de/d24/f35 [3616290,118857] 0 2024-04-22T15:50:11.238 INFO:tasks.workunit.client.1.smithi027.stdout:5/847: chown d0/d1a/d22/d26/d5d/de3/d3e/da2 1806024 1 2024-04-22T15:50:11.241 INFO:tasks.workunit.client.1.smithi027.stdout:2/912: mkdir dd/df/d1b/d96/d116 0 2024-04-22T15:50:11.244 INFO:tasks.workunit.client.1.smithi027.stdout:5/848: read d0/d1a/d22/d72/dac/dde/dfb/f75 [704140,108716] 0 2024-04-22T15:50:11.247 INFO:tasks.workunit.client.1.smithi027.stdout:5/849: fdatasync d0/d1a/d22/d26/d32/f70 0 2024-04-22T15:50:11.248 INFO:tasks.workunit.client.1.smithi027.stdout:2/913: truncate dd/df/d1b/d96/dc5/dd1/fe5 2476773 0 2024-04-22T15:50:11.250 INFO:tasks.workunit.client.1.smithi027.stdout:5/850: creat d0/d1a/d22/d72/de8/f110 x:0 0 0 2024-04-22T15:50:11.253 INFO:tasks.workunit.client.1.smithi027.stdout:5/851: truncate d0/d1a/d22/d26/d32/d79/dd6/fdf 4919756 0 2024-04-22T15:50:11.253 INFO:tasks.workunit.client.1.smithi027.stdout:2/914: symlink dd/df/d1b/d38/d60/d75/d9f/l117 0 2024-04-22T15:50:11.257 INFO:tasks.workunit.client.1.smithi027.stdout:5/852: unlink d0/df/de/l31 0 2024-04-22T15:50:11.257 INFO:tasks.workunit.client.1.smithi027.stdout:2/915: creat dd/df/f118 x:0 0 0 2024-04-22T15:50:11.258 INFO:tasks.workunit.client.1.smithi027.stdout:5/853: fdatasync d0/df/d3d/d81/fbc 0 2024-04-22T15:50:11.262 INFO:tasks.workunit.client.1.smithi027.stdout:4/838: dwrite d5/de/df2/f84 [0,4194304] 0 2024-04-22T15:50:11.264 INFO:tasks.workunit.client.1.smithi027.stdout:5/854: mkdir d0/db6/d111 0 2024-04-22T15:50:11.270 INFO:tasks.workunit.client.1.smithi027.stdout:2/916: creat dd/df/d70/d4b/dec/f119 x:0 0 0 2024-04-22T15:50:11.270 INFO:tasks.workunit.client.1.smithi027.stdout:2/917: chown dd/f6f 2 1 2024-04-22T15:50:11.273 INFO:tasks.workunit.client.1.smithi027.stdout:4/839: mknod d5/d42/d10f/c129 0 2024-04-22T15:50:11.276 INFO:tasks.workunit.client.1.smithi027.stdout:4/840: truncate d5/d11/d6c/da3/d10b/f110 333933 0 2024-04-22T15:50:11.278 INFO:tasks.workunit.client.1.smithi027.stdout:2/918: dread - dd/df/d1b/d1f/f78 zero size 2024-04-22T15:50:11.279 INFO:tasks.workunit.client.1.smithi027.stdout:5/855: creat d0/db6/d111/f112 x:0 0 0 2024-04-22T15:50:11.280 INFO:tasks.workunit.client.1.smithi027.stdout:5/856: fsync d0/df/f45 0 2024-04-22T15:50:11.291 INFO:tasks.workunit.client.1.smithi027.stdout:4/841: getdents d5/d11/d1d/d24/d123 0 2024-04-22T15:50:11.295 INFO:tasks.workunit.client.1.smithi027.stdout:4/842: symlink d5/d11/d6c/dca/l12a 0 2024-04-22T15:50:11.298 INFO:tasks.workunit.client.1.smithi027.stdout:5/857: dwrite d0/df/de/d24/f35 [0,4194304] 0 2024-04-22T15:50:11.298 INFO:tasks.workunit.client.1.smithi027.stdout:4/843: chown d5/d45/d5a/dcd/l117 30203679 1 2024-04-22T15:50:11.299 INFO:tasks.workunit.client.1.smithi027.stdout:2/919: dread dd/df/d1b/f5e [0,4194304] 0 2024-04-22T15:50:11.302 INFO:tasks.workunit.client.1.smithi027.stdout:4/844: mknod d5/de/c12b 0 2024-04-22T15:50:11.303 INFO:tasks.workunit.client.1.smithi027.stdout:2/920: mkdir dd/dae/dd4/ddb/d11a 0 2024-04-22T15:50:11.305 INFO:tasks.workunit.client.1.smithi027.stdout:5/858: dread - d0/d1a/d22/d26/d32/d88/d90/d107/f10b zero size 2024-04-22T15:50:11.306 INFO:tasks.workunit.client.1.smithi027.stdout:4/845: mkdir d5/d11/d6c/dca/d12c 0 2024-04-22T15:50:11.306 INFO:tasks.workunit.client.1.smithi027.stdout:2/921: creat dd/df/d1b/d38/dd5/f11b x:0 0 0 2024-04-22T15:50:11.308 INFO:tasks.workunit.client.1.smithi027.stdout:4/846: symlink d5/d45/db3/l12d 0 2024-04-22T15:50:11.315 INFO:tasks.workunit.client.1.smithi027.stdout:2/922: rename dd/dae/dd4/ddb/d113 to dd/df/d1b/d11c 0 2024-04-22T15:50:11.316 INFO:tasks.workunit.client.1.smithi027.stdout:4/847: dwrite d5/de/d2a/d58/f68 [0,4194304] 0 2024-04-22T15:50:11.321 INFO:tasks.workunit.client.1.smithi027.stdout:2/923: unlink dd/df/d1b/d38/f5c 0 2024-04-22T15:50:11.323 INFO:tasks.workunit.client.1.smithi027.stdout:4/848: rename d5/de/d2a/d58/f68 to d5/de/d30/f12e 0 2024-04-22T15:50:11.326 INFO:tasks.workunit.client.1.smithi027.stdout:2/924: creat dd/df/d70/d4b/d52/d9a/da4/dad/d8d/f11d x:0 0 0 2024-04-22T15:50:11.329 INFO:tasks.workunit.client.1.smithi027.stdout:5/859: dwrite d0/df/de/dbb/fe1 [0,4194304] 0 2024-04-22T15:50:11.335 INFO:tasks.workunit.client.1.smithi027.stdout:5/860: mknod d0/d1a/d22/d26/d32/d88/d90/c113 0 2024-04-22T15:50:11.335 INFO:tasks.workunit.client.1.smithi027.stdout:5/861: chown d0/c30 110 1 2024-04-22T15:50:11.337 INFO:tasks.workunit.client.1.smithi027.stdout:4/849: dread d5/d45/db3/fe8 [0,4194304] 0 2024-04-22T15:50:11.340 INFO:tasks.workunit.client.1.smithi027.stdout:5/862: mknod d0/d97/dbf/c114 0 2024-04-22T15:50:11.344 INFO:tasks.workunit.client.1.smithi027.stdout:2/925: rename dd/df/d70/d4b/d52/d9a/da4/dad/d8d/l114 to dd/df/d1b/d38/l11e 0 2024-04-22T15:50:11.344 INFO:tasks.workunit.client.1.smithi027.stdout:4/850: dwrite d5/d11/da6/dd0/f118 [0,4194304] 0 2024-04-22T15:50:11.345 INFO:tasks.workunit.client.1.smithi027.stdout:2/926: fsync dd/df/d70/fe1 0 2024-04-22T15:50:11.348 INFO:tasks.workunit.client.1.smithi027.stdout:5/863: creat d0/d1a/d22/d72/de8/f115 x:0 0 0 2024-04-22T15:50:11.354 INFO:tasks.workunit.client.1.smithi027.stdout:2/927: mknod dd/df/d1b/c11f 0 2024-04-22T15:50:11.355 INFO:tasks.workunit.client.1.smithi027.stdout:5/864: dwrite d0/df/de/f10f [0,4194304] 0 2024-04-22T15:50:11.363 INFO:tasks.workunit.client.1.smithi027.stdout:5/865: fsync d0/f8 0 2024-04-22T15:50:11.364 INFO:tasks.workunit.client.1.smithi027.stdout:5/866: chown d0/f6 5 1 2024-04-22T15:50:11.367 INFO:tasks.workunit.client.1.smithi027.stdout:5/867: mkdir d0/df/d3d/d81/d116 0 2024-04-22T15:50:11.368 INFO:tasks.workunit.client.1.smithi027.stdout:2/928: symlink dd/df/d70/d4b/d52/d9a/da4/dad/l120 0 2024-04-22T15:50:11.368 INFO:tasks.workunit.client.1.smithi027.stdout:5/868: rmdir d0/d97 39 2024-04-22T15:50:11.373 INFO:tasks.workunit.client.1.smithi027.stdout:2/929: dread dd/df/d70/d4b/d4e/f51 [0,4194304] 0 2024-04-22T15:50:11.375 INFO:tasks.workunit.client.1.smithi027.stdout:5/869: dwrite d0/db6/f105 [0,4194304] 0 2024-04-22T15:50:11.378 INFO:tasks.workunit.client.1.smithi027.stdout:4/851: truncate d5/d45/db3/fe8 461789 0 2024-04-22T15:50:11.381 INFO:tasks.workunit.client.1.smithi027.stdout:5/870: rename d0/df/de/l3f to d0/df/de/d40/l117 0 2024-04-22T15:50:11.384 INFO:tasks.workunit.client.1.smithi027.stdout:4/852: dread d5/de/d30/f12e [0,4194304] 0 2024-04-22T15:50:11.386 INFO:tasks.workunit.client.1.smithi027.stdout:5/871: fdatasync d0/d1a/d22/d26/d5d/de3/f103 0 2024-04-22T15:50:11.392 INFO:tasks.workunit.client.1.smithi027.stdout:2/930: dread dd/df/d1b/d38/d60/d75/fb3 [0,4194304] 0 2024-04-22T15:50:11.393 INFO:tasks.workunit.client.1.smithi027.stdout:5/872: truncate d0/df/d3d/fb5 52711 0 2024-04-22T15:50:11.395 INFO:tasks.workunit.client.1.smithi027.stdout:4/853: link d5/d11/d1d/d24/f79 d5/d45/d5a/dcd/f12f 0 2024-04-22T15:50:11.395 INFO:tasks.workunit.client.1.smithi027.stdout:4/854: fsync d5/d11/d6c/f112 0 2024-04-22T15:50:11.396 INFO:tasks.workunit.client.1.smithi027.stdout:4/855: chown d5/d11/d1d/d7b/de9 723 1 2024-04-22T15:50:11.397 INFO:tasks.workunit.client.1.smithi027.stdout:4/856: fsync d5/d11/d1d/d54/d8c/f11c 0 2024-04-22T15:50:11.401 INFO:tasks.workunit.client.1.smithi027.stdout:4/857: creat d5/d45/d5a/dcd/f130 x:0 0 0 2024-04-22T15:50:11.403 INFO:tasks.workunit.client.1.smithi027.stdout:2/931: link dd/df/d70/d5a/f87 dd/df/d1b/d38/f121 0 2024-04-22T15:50:11.408 INFO:tasks.workunit.client.1.smithi027.stdout:4/858: link d5/d11/d1d/d24/f79 d5/d11/d1d/d24/d123/f131 0 2024-04-22T15:50:11.414 INFO:tasks.workunit.client.1.smithi027.stdout:5/873: truncate d0/df/de/feb 98992 0 2024-04-22T15:50:11.419 INFO:tasks.workunit.client.1.smithi027.stdout:5/874: rename d0/d1a/d22/d26/d32/d88/d90/c113 to d0/d1a/d22/d26/d32/d79/c118 0 2024-04-22T15:50:11.423 INFO:tasks.workunit.client.1.smithi027.stdout:5/875: symlink d0/d1a/d22/d72/dac/l119 0 2024-04-22T15:50:11.428 INFO:tasks.workunit.client.1.smithi027.stdout:2/932: dwrite dd/df/d70/f6e [0,4194304] 0 2024-04-22T15:50:11.429 INFO:tasks.workunit.client.1.smithi027.stdout:2/933: readlink dd/df/l79 0 2024-04-22T15:50:11.429 INFO:tasks.workunit.client.1.smithi027.stdout:5/876: creat d0/d1a/d22/d26/d32/f11a x:0 0 0 2024-04-22T15:50:11.432 INFO:tasks.workunit.client.1.smithi027.stdout:4/859: dwrite d5/fd5 [0,4194304] 0 2024-04-22T15:50:11.432 INFO:tasks.workunit.client.1.smithi027.stdout:5/877: rmdir d0/df/d3d/d81 39 2024-04-22T15:50:11.434 INFO:tasks.workunit.client.1.smithi027.stdout:4/860: write d5/da8/f10e [161157,107133] 0 2024-04-22T15:50:11.443 INFO:tasks.workunit.client.1.smithi027.stdout:4/861: read d5/f23 [830034,29134] 0 2024-04-22T15:50:11.448 INFO:tasks.workunit.client.1.smithi027.stdout:5/878: write d0/d1a/d22/d26/d5d/de3/d3e/d55/f9c [636066,88054] 0 2024-04-22T15:50:11.469 INFO:tasks.workunit.client.1.smithi027.stdout:5/879: dwrite d0/df/de/d40/d89/fcc [0,4194304] 0 2024-04-22T15:50:11.469 INFO:tasks.workunit.client.1.smithi027.stdout:4/862: dread d5/d11/d1d/d7b/fdd [0,4194304] 0 2024-04-22T15:50:11.471 INFO:tasks.workunit.client.1.smithi027.stdout:4/863: fdatasync d5/d45/d5a/dcd/f111 0 2024-04-22T15:50:11.472 INFO:tasks.workunit.client.1.smithi027.stdout:5/880: write d0/d1a/d22/d26/d5d/de3/d3e/d55/f9c [2018905,130850] 0 2024-04-22T15:50:11.477 INFO:tasks.workunit.client.1.smithi027.stdout:4/864: read d5/de/f37 [4034139,32549] 0 2024-04-22T15:50:11.481 INFO:tasks.workunit.client.1.smithi027.stdout:5/881: creat d0/df/de/d40/d89/f11b x:0 0 0 2024-04-22T15:50:11.495 INFO:tasks.workunit.client.1.smithi027.stdout:4/865: dwrite d5/de/f37 [0,4194304] 0 2024-04-22T15:50:11.498 INFO:tasks.workunit.client.1.smithi027.stdout:5/882: dwrite d0/df/de/f18 [4194304,4194304] 0 2024-04-22T15:50:11.508 INFO:tasks.workunit.client.1.smithi027.stdout:4/866: getdents d5/d11/da6 0 2024-04-22T15:50:11.513 INFO:tasks.workunit.client.1.smithi027.stdout:4/867: creat d5/d45/ddc/f132 x:0 0 0 2024-04-22T15:50:11.515 INFO:tasks.workunit.client.1.smithi027.stdout:5/883: write d0/df/de/d40/fb2 [356059,27066] 0 2024-04-22T15:50:11.518 INFO:tasks.workunit.client.1.smithi027.stdout:5/884: creat d0/d1a/d22/d26/d5d/de3/d8f/f11c x:0 0 0 2024-04-22T15:50:11.524 INFO:tasks.workunit.client.1.smithi027.stdout:5/885: link d0/d1a/l1e d0/df/de/dbb/l11d 0 2024-04-22T15:50:11.529 INFO:tasks.workunit.client.1.smithi027.stdout:4/868: dread d5/d11/f80 [0,4194304] 0 2024-04-22T15:50:11.532 INFO:tasks.workunit.client.1.smithi027.stdout:4/869: dread d5/da8/f10e [0,4194304] 0 2024-04-22T15:50:11.535 INFO:tasks.workunit.client.1.smithi027.stdout:4/870: write d5/d11/f80 [2046113,26900] 0 2024-04-22T15:50:11.536 INFO:tasks.workunit.client.1.smithi027.stdout:4/871: stat d5/de/d2a/d58/f67 0 2024-04-22T15:50:11.539 INFO:tasks.workunit.client.1.smithi027.stdout:4/872: creat d5/d45/d5a/f133 x:0 0 0 2024-04-22T15:50:11.547 INFO:tasks.workunit.client.1.smithi027.stdout:4/873: dwrite d5/d45/ddc/f132 [0,4194304] 0 2024-04-22T15:50:11.547 INFO:tasks.workunit.client.1.smithi027.stdout:5/886: dwrite d0/d1a/f3a [4194304,4194304] 0 2024-04-22T15:50:11.555 INFO:tasks.workunit.client.1.smithi027.stdout:4/874: mkdir d5/d11/d1d/d24/d134 0 2024-04-22T15:50:11.556 INFO:tasks.workunit.client.1.smithi027.stdout:4/875: chown d5/d45/f7d 9049549 1 2024-04-22T15:50:11.563 INFO:tasks.workunit.client.1.smithi027.stdout:4/876: dread d5/de/d30/f12e [0,4194304] 0 2024-04-22T15:50:11.567 INFO:tasks.workunit.client.1.smithi027.stdout:4/877: read d5/d7/f1f [117354,106500] 0 2024-04-22T15:50:11.569 INFO:tasks.workunit.client.1.smithi027.stdout:4/878: mkdir d5/d11/d1d/d7b/d135 0 2024-04-22T15:50:11.574 INFO:tasks.workunit.client.1.smithi027.stdout:5/887: write d0/d1a/d22/d26/d32/d79/dd6/fdc [1492868,52852] 0 2024-04-22T15:50:11.580 INFO:tasks.workunit.client.1.smithi027.stdout:5/888: rmdir d0/d1a/d22 39 2024-04-22T15:50:11.585 INFO:tasks.workunit.client.1.smithi027.stdout:5/889: creat d0/d97/dd7/f11e x:0 0 0 2024-04-22T15:50:11.589 INFO:tasks.workunit.client.1.smithi027.stdout:4/879: truncate d5/d11/f7a 2128558 0 2024-04-22T15:50:11.590 INFO:tasks.workunit.client.1.smithi027.stdout:5/890: mknod d0/d1a/d22/d26/d32/c11f 0 2024-04-22T15:50:11.594 INFO:tasks.workunit.client.1.smithi027.stdout:3/889: sync 2024-04-22T15:50:11.594 INFO:tasks.workunit.client.1.smithi027.stdout:8/815: sync 2024-04-22T15:50:11.597 INFO:tasks.workunit.client.1.smithi027.stdout:8/816: chown d1/d3/l10c 3921 1 2024-04-22T15:50:11.598 INFO:tasks.workunit.client.1.smithi027.stdout:5/891: dwrite d0/df/d10/f64 [0,4194304] 0 2024-04-22T15:50:11.600 INFO:tasks.workunit.client.1.smithi027.stdout:3/890: symlink d28/d4b/d52/db2/d101/l11c 0 2024-04-22T15:50:11.604 INFO:tasks.workunit.client.1.smithi027.stdout:4/880: dread d5/d45/d5a/fe6 [0,4194304] 0 2024-04-22T15:50:11.606 INFO:tasks.workunit.client.1.smithi027.stdout:5/892: mkdir d0/df/d10/d120 0 2024-04-22T15:50:11.650 INFO:tasks.workunit.client.1.smithi027.stdout:3/891: rmdir d28/d4b/d52/d94/d105/d11a 39 2024-04-22T15:50:11.650 INFO:tasks.workunit.client.1.smithi027.stdout:8/817: mknod d1/d3/d1c/d32/c11c 0 2024-04-22T15:50:11.653 INFO:tasks.workunit.client.1.smithi027.stdout:5/893: creat d0/d1a/d22/d26/d5d/de3/d3e/dd5/f121 x:0 0 0 2024-04-22T15:50:11.656 INFO:tasks.workunit.client.1.smithi027.stdout:5/894: stat d0/df/d3d/fb5 0 2024-04-22T15:50:11.657 INFO:tasks.workunit.client.1.smithi027.stdout:8/818: creat d1/d3/d1c/d43/d5f/f11d x:0 0 0 2024-04-22T15:50:11.661 INFO:tasks.workunit.client.1.smithi027.stdout:3/892: link d28/d4b/d52/db2/deb/c108 d28/d4b/d52/d94/d105/d11a/c11d 0 2024-04-22T15:50:11.663 INFO:tasks.workunit.client.1.smithi027.stdout:5/895: dwrite d0/d1a/d22/d26/d32/f11a [0,4194304] 0 2024-04-22T15:50:11.667 INFO:tasks.workunit.client.1.smithi027.stdout:8/819: mknod d1/d3/d1c/d28/d42/dc4/df3/c11e 0 2024-04-22T15:50:11.671 INFO:tasks.workunit.client.1.smithi027.stdout:3/893: rmdir d28/d4b/d52/d9a 39 2024-04-22T15:50:11.746 INFO:tasks.workunit.client.1.smithi027.stdout:5/896: mknod d0/d1a/d22/c122 0 2024-04-22T15:50:11.747 INFO:tasks.workunit.client.1.smithi027.stdout:4/881: dread d5/f9 [0,4194304] 0 2024-04-22T15:50:11.749 INFO:tasks.workunit.client.1.smithi027.stdout:8/820: unlink d1/d4f/fcc 0 2024-04-22T15:50:11.750 INFO:tasks.workunit.client.1.smithi027.stdout:4/882: truncate d5/d11/f28 4227794 0 2024-04-22T15:50:11.752 INFO:tasks.workunit.client.1.smithi027.stdout:5/897: mkdir d0/d1a/d22/d26/d5d/de3/d3e/d123 0 2024-04-22T15:50:11.758 INFO:tasks.workunit.client.1.smithi027.stdout:5/898: mknod d0/df/d3d/c124 0 2024-04-22T15:50:11.759 INFO:tasks.workunit.client.1.smithi027.stdout:4/883: getdents d5/d11/dc1 0 2024-04-22T15:50:11.766 INFO:tasks.workunit.client.1.smithi027.stdout:5/899: dwrite d0/d97/dd7/f11e [0,4194304] 0 2024-04-22T15:50:11.768 INFO:tasks.workunit.client.1.smithi027.stdout:4/884: dwrite d5/d11/d1d/d54/d8c/f106 [0,4194304] 0 2024-04-22T15:50:11.770 INFO:tasks.workunit.client.1.smithi027.stdout:5/900: mknod d0/d1a/d22/d26/d32/d79/c125 0 2024-04-22T15:50:11.776 INFO:tasks.workunit.client.1.smithi027.stdout:3/894: dwrite d28/d4b/d52/f7a [0,4194304] 0 2024-04-22T15:50:11.778 INFO:tasks.workunit.client.1.smithi027.stdout:3/895: write d28/d4b/d52/d94/dc4/f115 [224561,128276] 0 2024-04-22T15:50:11.779 INFO:tasks.workunit.client.1.smithi027.stdout:8/821: truncate d1/d3/d1c/d32/fc5 1827769 0 2024-04-22T15:50:11.785 INFO:tasks.workunit.client.1.smithi027.stdout:3/896: mkdir d28/d4b/d52/d94/dc4/dfc/d11e 0 2024-04-22T15:50:11.788 INFO:tasks.workunit.client.1.smithi027.stdout:8/822: mkdir d1/d3/d1c/d49/d94/d11f 0 2024-04-22T15:50:11.788 INFO:tasks.workunit.client.1.smithi027.stdout:5/901: link d0/d1a/d22/d26/d32/d88/c108 d0/db6/c126 0 2024-04-22T15:50:11.791 INFO:tasks.workunit.client.1.smithi027.stdout:5/902: write d0/df/de/d24/d62/f7f [3875115,71542] 0 2024-04-22T15:50:11.791 INFO:tasks.workunit.client.1.smithi027.stdout:3/897: rename d28/l2c to d28/d69/l11f 0 2024-04-22T15:50:11.798 INFO:tasks.workunit.client.1.smithi027.stdout:8/823: creat d1/d3/d1c/d32/d41/d52/d61/d8f/ddd/dbe/d116/f120 x:0 0 0 2024-04-22T15:50:11.801 INFO:tasks.workunit.client.1.smithi027.stdout:8/824: write d1/d3/d1c/d32/d41/d52/d61/fa6 [1930543,35360] 0 2024-04-22T15:50:11.808 INFO:tasks.workunit.client.1.smithi027.stdout:8/825: dread - d1/d3/d1c/d32/d41/d52/d61/d8f/fc8 zero size 2024-04-22T15:50:11.812 INFO:tasks.workunit.client.1.smithi027.stdout:8/826: mkdir d1/d3/d1c/d43/d5f/d7d/d121 0 2024-04-22T15:50:11.813 INFO:tasks.workunit.client.1.smithi027.stdout:8/827: chown d1/d1e/d47/d53/fdc 11796 1 2024-04-22T15:50:11.816 INFO:tasks.workunit.client.1.smithi027.stdout:3/898: write d28/d4b/d52/d94/fe0 [520985,122526] 0 2024-04-22T15:50:11.817 INFO:tasks.workunit.client.1.smithi027.stdout:8/828: unlink d1/d3/f46 0 2024-04-22T15:50:11.826 INFO:tasks.workunit.client.1.smithi027.stdout:3/899: rename d28/d4b/ce3 to d28/d4b/dc6/d10c/c120 0 2024-04-22T15:50:11.829 INFO:tasks.workunit.client.1.smithi027.stdout:8/829: creat d1/d1e/d47/f122 x:0 0 0 2024-04-22T15:50:11.830 INFO:tasks.workunit.client.1.smithi027.stdout:3/900: chown d28/d69/f82 8 1 2024-04-22T15:50:11.834 INFO:tasks.workunit.client.1.smithi027.stdout:8/830: dread d1/d1e/d47/d53/d76/fdf [0,4194304] 0 2024-04-22T15:50:11.834 INFO:tasks.workunit.client.1.smithi027.stdout:3/901: symlink d28/d4b/d52/db2/d95/d9c/dc1/l121 0 2024-04-22T15:50:11.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:11 smithi027 ceph-mon[34922]: pgmap v9: 65 pgs: 65 active+clean; 2.9 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 51 MiB/s rd, 107 MiB/s wr, 302 op/s 2024-04-22T15:50:11.840 INFO:tasks.workunit.client.1.smithi027.stdout:3/902: rename d28/f34 to d28/d4b/d52/db2/d95/d9c/de9/d119/f122 0 2024-04-22T15:50:11.842 INFO:tasks.workunit.client.1.smithi027.stdout:8/831: fdatasync d1/d3/d1c/d49/f63 0 2024-04-22T15:50:11.845 INFO:tasks.workunit.client.1.smithi027.stdout:3/903: mknod d28/d4b/d52/db2/deb/c123 0 2024-04-22T15:50:11.846 INFO:tasks.workunit.client.1.smithi027.stdout:8/832: truncate d1/d3/d1c/d28/d42/def/f113 968401 0 2024-04-22T15:50:11.848 INFO:tasks.workunit.client.1.smithi027.stdout:8/833: mknod d1/d3/d1c/d32/d41/c123 0 2024-04-22T15:50:11.851 INFO:tasks.workunit.client.1.smithi027.stdout:3/904: creat d28/d4b/d52/d94/d105/f124 x:0 0 0 2024-04-22T15:50:11.852 INFO:tasks.workunit.client.1.smithi027.stdout:3/905: chown d28/f4e 752 1 2024-04-22T15:50:11.854 INFO:tasks.workunit.client.1.smithi027.stdout:8/834: rename d1/d1e/d47/f70 to d1/d3/d1c/d32/d41/d52/d61/d8f/ddd/dbe/d116/f124 0 2024-04-22T15:50:11.854 INFO:tasks.workunit.client.1.smithi027.stdout:8/835: chown d1/d3/d1c/d56/l64 16303 1 2024-04-22T15:50:11.855 INFO:tasks.workunit.client.1.smithi027.stdout:3/906: creat d28/d4b/d52/d94/dae/f125 x:0 0 0 2024-04-22T15:50:11.861 INFO:tasks.workunit.client.1.smithi027.stdout:8/836: truncate d1/d3/d1c/d32/d41/f83 967702 0 2024-04-22T15:50:11.889 INFO:tasks.workunit.client.1.smithi027.stdout:3/907: write d28/f3b [1025036,129891] 0 2024-04-22T15:50:11.890 INFO:tasks.workunit.client.1.smithi027.stdout:3/908: stat d28/d5d/fcb 0 2024-04-22T15:50:11.896 INFO:tasks.workunit.client.1.smithi027.stdout:3/909: creat d28/d4b/d52/db2/d95/d9c/dc1/f126 x:0 0 0 2024-04-22T15:50:11.913 INFO:tasks.workunit.client.1.smithi027.stdout:8/837: write d1/d3/d1c/d32/d41/d52/d61/d8f/ddd/dbe/d116/f124 [8136508,64532] 0 2024-04-22T15:50:11.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:11 smithi016 ceph-mon[26620]: pgmap v9: 65 pgs: 65 active+clean; 2.9 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 51 MiB/s rd, 107 MiB/s wr, 302 op/s 2024-04-22T15:50:11.920 INFO:tasks.workunit.client.1.smithi027.stdout:8/838: dread d1/d3/d36/d88/da9/ff8 [0,4194304] 0 2024-04-22T15:50:11.922 INFO:tasks.workunit.client.1.smithi027.stdout:3/910: dread d28/d4b/d52/db2/d95/fe5 [0,4194304] 0 2024-04-22T15:50:11.926 INFO:tasks.workunit.client.1.smithi027.stdout:8/839: mknod d1/d3/d1c/d49/d94/d11f/c125 0 2024-04-22T15:50:11.936 INFO:tasks.workunit.client.1.smithi027.stdout:3/911: dwrite d28/d4b/d52/d94/dc4/dfc/f102 [0,4194304] 0 2024-04-22T15:50:11.984 INFO:tasks.workunit.client.1.smithi027.stdout:6/897: sync 2024-04-22T15:50:11.985 INFO:tasks.workunit.client.1.smithi027.stdout:7/888: sync 2024-04-22T15:50:11.985 INFO:tasks.workunit.client.1.smithi027.stdout:1/691: sync 2024-04-22T15:50:11.985 INFO:tasks.workunit.client.1.smithi027.stdout:7/889: chown d0/d49/f114 20558 1 2024-04-22T15:50:11.998 INFO:tasks.workunit.client.1.smithi027.stdout:1/692: dread d2/d5f/d64/d81/d3d/d44/f92 [0,4194304] 0 2024-04-22T15:50:12.000 INFO:tasks.workunit.client.1.smithi027.stdout:8/840: write d1/d3/d1c/d32/d41/fe4 [633705,14142] 0 2024-04-22T15:50:12.005 INFO:tasks.workunit.client.1.smithi027.stdout:3/912: unlink d28/d5d/db0/fba 0 2024-04-22T15:50:12.008 INFO:tasks.workunit.client.1.smithi027.stdout:6/898: creat d5/dc/dd/d79/d7b/f138 x:0 0 0 2024-04-22T15:50:12.009 INFO:tasks.workunit.client.1.smithi027.stdout:6/899: chown d5/dc/d26/d49/f10c 31716 1 2024-04-22T15:50:12.014 INFO:tasks.workunit.client.1.smithi027.stdout:7/890: truncate d0/d4c/d8b/d92/da3/fe1 4601784 0 2024-04-22T15:50:12.018 INFO:tasks.workunit.client.1.smithi027.stdout:1/693: creat d2/d5f/d64/fe7 x:0 0 0 2024-04-22T15:50:12.029 INFO:tasks.workunit.client.1.smithi027.stdout:7/891: mknod d0/d4c/d4e/df5/c12a 0 2024-04-22T15:50:12.034 INFO:tasks.workunit.client.1.smithi027.stdout:6/900: dwrite d5/d30/d40/fad [0,4194304] 0 2024-04-22T15:50:12.036 INFO:tasks.workunit.client.1.smithi027.stdout:8/841: symlink d1/l126 0 2024-04-22T15:50:12.043 INFO:tasks.workunit.client.1.smithi027.stdout:6/901: fsync d5/f75 0 2024-04-22T15:50:12.043 INFO:tasks.workunit.client.1.smithi027.stdout:1/694: chown d2/d5f/d64/d81/da7/l6a 30275 1 2024-04-22T15:50:12.045 INFO:tasks.workunit.client.1.smithi027.stdout:8/842: creat d1/d3/d36/d57/df0/f127 x:0 0 0 2024-04-22T15:50:12.045 INFO:tasks.workunit.client.1.smithi027.stdout:3/913: link d28/l75 d28/d4b/d52/d94/dc4/d10a/l127 0 2024-04-22T15:50:12.050 INFO:tasks.workunit.client.1.smithi027.stdout:8/843: mknod d1/d4f/c128 0 2024-04-22T15:50:12.053 INFO:tasks.workunit.client.1.smithi027.stdout:6/902: dwrite d5/d30/dbe/f10f [0,4194304] 0 2024-04-22T15:50:12.055 INFO:tasks.workunit.client.1.smithi027.stdout:7/892: getdents d0/d4c/d8b/d92/da3 0 2024-04-22T15:50:12.056 INFO:tasks.workunit.client.1.smithi027.stdout:7/893: chown d0/d11/d1c/d31/d62/de9 16147518 1 2024-04-22T15:50:12.059 INFO:tasks.workunit.client.1.smithi027.stdout:1/695: dwrite d2/d5f/d64/d81/d3d/d44/d9c/d9f/fac [0,4194304] 0 2024-04-22T15:50:12.059 INFO:tasks.workunit.client.1.smithi027.stdout:1/696: chown d2/f35 2317 1 2024-04-22T15:50:12.060 INFO:tasks.workunit.client.1.smithi027.stdout:8/844: mknod d1/d3/d1c/d28/d42/dc4/df3/c129 0 2024-04-22T15:50:12.064 INFO:tasks.workunit.client.1.smithi027.stdout:6/903: dread - d5/dc/dae/f118 zero size 2024-04-22T15:50:12.065 INFO:tasks.workunit.client.1.smithi027.stdout:7/894: creat d0/d4c/f12b x:0 0 0 2024-04-22T15:50:12.067 INFO:tasks.workunit.client.1.smithi027.stdout:3/914: link d28/d4b/d52/d94/dae/f125 d28/d5d/d62/dcf/dec/f128 0 2024-04-22T15:50:12.067 INFO:tasks.workunit.client.1.smithi027.stdout:6/904: read d5/dc/dd/dd0/d34/f65 [3634828,127501] 0 2024-04-22T15:50:12.069 INFO:tasks.workunit.client.1.smithi027.stdout:1/697: chown d2/d30/l5c 356407 1 2024-04-22T15:50:12.073 INFO:tasks.workunit.client.1.smithi027.stdout:3/915: rename d28/d5d/f100 to d28/d5d/d62/dcf/dec/f129 0 2024-04-22T15:50:12.074 INFO:tasks.workunit.client.1.smithi027.stdout:8/845: creat d1/d3/d1c/d43/d82/f12a x:0 0 0 2024-04-22T15:50:12.075 INFO:tasks.workunit.client.1.smithi027.stdout:8/846: read - d1/d3/d1c/d49/d94/f10d zero size 2024-04-22T15:50:12.076 INFO:tasks.workunit.client.1.smithi027.stdout:6/905: truncate d5/dc/dd/dd0/d34/f65 393358 0 2024-04-22T15:50:12.082 INFO:tasks.workunit.client.1.smithi027.stdout:1/698: link d2/f3 d2/d5f/d64/d81/d7e/fe8 0 2024-04-22T15:50:12.086 INFO:tasks.workunit.client.1.smithi027.stdout:8/847: rmdir d1/d3/d1c/d32/d41/d52/d61/d8f 39 2024-04-22T15:50:12.087 INFO:tasks.workunit.client.1.smithi027.stdout:2/934: sync 2024-04-22T15:50:12.087 INFO:tasks.workunit.client.1.smithi027.stdout:8/848: stat d1/d1e/d47/f122 0 2024-04-22T15:50:12.087 INFO:tasks.workunit.client.1.smithi027.stdout:9/814: sync 2024-04-22T15:50:12.088 INFO:tasks.workunit.client.1.smithi027.stdout:3/916: dread d28/f63 [0,4194304] 0 2024-04-22T15:50:12.094 INFO:tasks.workunit.client.1.smithi027.stdout:1/699: read d2/d5f/d64/d81/da7/f9e [3385779,9202] 0 2024-04-22T15:50:12.101 INFO:tasks.workunit.client.1.smithi027.stdout:7/895: dwrite d0/d11/d3a/d46/f9c [0,4194304] 0 2024-04-22T15:50:12.103 INFO:tasks.workunit.client.1.smithi027.stdout:9/815: dwrite d0/dc/d12/d30/d5b/d91/da9/fdc [0,4194304] 0 2024-04-22T15:50:12.103 INFO:tasks.workunit.client.1.smithi027.stdout:2/935: stat dd/df/d1b/d1f/c24 0 2024-04-22T15:50:12.109 INFO:tasks.workunit.client.1.smithi027.stdout:8/849: symlink d1/l12b 0 2024-04-22T15:50:12.111 INFO:tasks.workunit.client.1.smithi027.stdout:2/936: fsync dd/df/d1b/d1f/f78 0 2024-04-22T15:50:12.111 INFO:tasks.workunit.client.1.smithi027.stdout:6/906: dread d5/dc/dd/d79/d7b/d9a/fe5 [0,4194304] 0 2024-04-22T15:50:12.118 INFO:tasks.workunit.client.1.smithi027.stdout:7/896: mkdir d0/d11/d1c/d2c/d96/d126/d12c 0 2024-04-22T15:50:12.121 INFO:tasks.workunit.client.1.smithi027.stdout:9/816: dread d0/dc/d12/d30/d5b/dbc/df5/d104/d6b/d6e/f31 [0,4194304] 0 2024-04-22T15:50:12.124 INFO:tasks.workunit.client.1.smithi027.stdout:1/700: write d2/d1d/f91 [3471155,17634] 0 2024-04-22T15:50:12.126 INFO:tasks.workunit.client.1.smithi027.stdout:6/907: symlink d5/db/d2e/l139 0 2024-04-22T15:50:12.128 INFO:tasks.workunit.client.1.smithi027.stdout:2/937: symlink dd/df/d1b/dc9/dca/l122 0 2024-04-22T15:50:12.130 INFO:tasks.workunit.client.1.smithi027.stdout:2/938: chown dd/d43/fb4 288741 1 2024-04-22T15:50:12.131 INFO:tasks.workunit.client.1.smithi027.stdout:7/897: unlink d0/l1 0 2024-04-22T15:50:12.132 INFO:tasks.workunit.client.1.smithi027.stdout:4/885: sync 2024-04-22T15:50:12.132 INFO:tasks.workunit.client.1.smithi027.stdout:5/903: sync 2024-04-22T15:50:12.133 INFO:tasks.workunit.client.1.smithi027.stdout:9/817: dwrite d0/dc/d12/d30/d5b/dbc/df5/d104/d6b/f10a [0,4194304] 0 2024-04-22T15:50:12.136 INFO:tasks.workunit.client.1.smithi027.stdout:8/850: write d1/d3/d1c/d28/d42/fb1 [830279,91653] 0 2024-04-22T15:50:12.138 INFO:tasks.workunit.client.1.smithi027.stdout:5/904: read d0/df/d3d/fb5 [35037,76560] 0 2024-04-22T15:50:12.141 INFO:tasks.workunit.client.1.smithi027.stdout:3/917: link d28/d4b/d52/d9a/fd1 d28/d5d/db0/f12a 0 2024-04-22T15:50:12.146 INFO:tasks.workunit.client.1.smithi027.stdout:7/898: creat d0/d11/d1c/ddc/f12d x:0 0 0 2024-04-22T15:50:12.147 INFO:tasks.workunit.client.1.smithi027.stdout:4/886: fsync d5/de/f86 0 2024-04-22T15:50:12.149 INFO:tasks.workunit.client.1.smithi027.stdout:7/899: write d0/d11/d1c/d31/d8a/ff8 [5803149,56255] 0 2024-04-22T15:50:12.150 INFO:tasks.workunit.client.1.smithi027.stdout:7/900: chown d0/d3f/la6 0 1 2024-04-22T15:50:12.150 INFO:tasks.workunit.client.1.smithi027.stdout:7/901: readlink d0/l10 0 2024-04-22T15:50:12.151 INFO:tasks.workunit.client.1.smithi027.stdout:7/902: chown d0/d11/d1c/d2c/d93/c85 16 1 2024-04-22T15:50:12.159 INFO:tasks.workunit.client.1.smithi027.stdout:1/701: dwrite d2/d5f/d64/d81/d3d/d44/d9c/d9f/fb5 [0,4194304] 0 2024-04-22T15:50:12.166 INFO:tasks.workunit.client.1.smithi027.stdout:6/908: truncate d5/d30/dbe/fc6 3891173 0 2024-04-22T15:50:12.166 INFO:tasks.workunit.client.1.smithi027.stdout:9/818: readlink d0/d64/l81 0 2024-04-22T15:50:12.166 INFO:tasks.workunit.client.1.smithi027.stdout:4/887: truncate d5/de/df2/f107 338678 0 2024-04-22T15:50:12.170 INFO:tasks.workunit.client.1.smithi027.stdout:3/918: dwrite d28/d5d/d79/fcd [0,4194304] 0 2024-04-22T15:50:12.170 INFO:tasks.workunit.client.1.smithi027.stdout:5/905: link d0/d1a/d22/d72/de8/ff9 d0/df/de/d24/d62/f127 0 2024-04-22T15:50:12.179 INFO:tasks.workunit.client.1.smithi027.stdout:4/888: dwrite d5/d11/d6c/f112 [0,4194304] 0 2024-04-22T15:50:12.179 INFO:tasks.workunit.client.1.smithi027.stdout:8/851: creat d1/d3/d1c/d32/d41/d52/d61/f12c x:0 0 0 2024-04-22T15:50:12.181 INFO:tasks.workunit.client.1.smithi027.stdout:6/909: creat d5/dc/d26/d3d/f13a x:0 0 0 2024-04-22T15:50:12.187 INFO:tasks.workunit.client.1.smithi027.stdout:2/939: link dd/df/d1b/f2e dd/df/d1b/d11c/f123 0 2024-04-22T15:50:12.190 INFO:tasks.workunit.client.1.smithi027.stdout:7/903: dwrite d0/d4c/d4e/ff0 [0,4194304] 0 2024-04-22T15:50:12.193 INFO:tasks.workunit.client.1.smithi027.stdout:4/889: creat d5/d11/d1d/d7b/f136 x:0 0 0 2024-04-22T15:50:12.196 INFO:tasks.workunit.client.1.smithi027.stdout:4/890: chown d5/de/da9/d11e/c126 0 1 2024-04-22T15:50:12.199 INFO:tasks.workunit.client.1.smithi027.stdout:4/891: write d5/d11/d6c/f9b [729912,98728] 0 2024-04-22T15:50:12.200 INFO:tasks.workunit.client.1.smithi027.stdout:1/702: dwrite d2/d5f/d64/d81/d7e/fe8 [0,4194304] 0 2024-04-22T15:50:12.202 INFO:tasks.workunit.client.1.smithi027.stdout:4/892: write d5/fd5 [1755057,102632] 0 2024-04-22T15:50:12.202 INFO:tasks.workunit.client.1.smithi027.stdout:6/910: truncate d5/dc/d26/f88 776082 0 2024-04-22T15:50:12.207 INFO:tasks.workunit.client.1.smithi027.stdout:5/906: symlink d0/d1a/d22/d26/d5d/de3/d8f/dce/d10d/l128 0 2024-04-22T15:50:12.208 INFO:tasks.workunit.client.1.smithi027.stdout:2/940: rmdir dd/df/d70/d4b/d4e 39 2024-04-22T15:50:12.209 INFO:tasks.workunit.client.1.smithi027.stdout:2/941: dread - dd/f111 zero size 2024-04-22T15:50:12.212 INFO:tasks.workunit.client.1.smithi027.stdout:8/852: unlink d1/d1e/dbc/c110 0 2024-04-22T15:50:12.213 INFO:tasks.workunit.client.1.smithi027.stdout:7/904: truncate d0/d4c/d8b/d92/da3/da8/f23 3814828 0 2024-04-22T15:50:12.216 INFO:tasks.workunit.client.1.smithi027.stdout:3/919: creat d28/d4b/d52/f12b x:0 0 0 2024-04-22T15:50:12.223 INFO:tasks.workunit.client.1.smithi027.stdout:5/907: unlink d0/d1a/l47 0 2024-04-22T15:50:12.224 INFO:tasks.workunit.client.1.smithi027.stdout:2/942: readlink dd/df/l21 0 2024-04-22T15:50:12.227 INFO:tasks.workunit.client.1.smithi027.stdout:9/819: getdents d0/dc/de 0 2024-04-22T15:50:12.229 INFO:tasks.workunit.client.1.smithi027.stdout:3/920: unlink d28/d5d/d79/fb8 0 2024-04-22T15:50:12.231 INFO:tasks.workunit.client.1.smithi027.stdout:6/911: mknod d5/dc/dd/d79/d7b/d5d/dba/df2/c13b 0 2024-04-22T15:50:12.233 INFO:tasks.workunit.client.1.smithi027.stdout:1/703: dread d2/d5f/d64/d81/d3d/d82/f93 [0,4194304] 0 2024-04-22T15:50:12.238 INFO:tasks.workunit.client.1.smithi027.stdout:2/943: dread dd/df/d1b/d1f/f3e [0,4194304] 0 2024-04-22T15:50:12.238 INFO:tasks.workunit.client.1.smithi027.stdout:2/944: fdatasync dd/df/f23 0 2024-04-22T15:50:12.246 INFO:tasks.workunit.client.1.smithi027.stdout:1/704: truncate d2/f75 2922154 0 2024-04-22T15:50:12.247 INFO:tasks.workunit.client.1.smithi027.stdout:1/705: dread - d2/d1d/d56/dc4/fce zero size 2024-04-22T15:50:12.258 INFO:tasks.workunit.client.1.smithi027.stdout:3/921: rename d28/d4b/f59 to d28/d4b/d52/db2/d95/d9c/f12c 0 2024-04-22T15:50:12.259 INFO:tasks.workunit.client.1.smithi027.stdout:3/922: dread - d28/d4b/d52/d9a/fab zero size 2024-04-22T15:50:12.259 INFO:tasks.workunit.client.1.smithi027.stdout:3/923: fsync f1c 0 2024-04-22T15:50:12.260 INFO:tasks.workunit.client.1.smithi027.stdout:6/912: mkdir d5/dc/d13c 0 2024-04-22T15:50:12.262 INFO:tasks.workunit.client.1.smithi027.stdout:4/893: write d5/d11/d6c/f8b [2399608,50270] 0 2024-04-22T15:50:12.262 INFO:tasks.workunit.client.1.smithi027.stdout:1/706: read - d2/d5f/d64/d81/d7e/fd2 zero size 2024-04-22T15:50:12.263 INFO:tasks.workunit.client.1.smithi027.stdout:8/853: link d1/d3/d1c/d32/d41/d52/c11b d1/d1e/d47/c12d 0 2024-04-22T15:50:12.268 INFO:tasks.workunit.client.1.smithi027.stdout:7/905: getdents d0/d11 0 2024-04-22T15:50:12.271 INFO:tasks.workunit.client.1.smithi027.stdout:5/908: truncate d0/d1a/d22/d26/d5d/de3/fcb 539546 0 2024-04-22T15:50:12.272 INFO:tasks.workunit.client.1.smithi027.stdout:9/820: dwrite d0/dc/d12/d30/d5b/dbc/df5/d104/da6/fc9 [0,4194304] 0 2024-04-22T15:50:12.274 INFO:tasks.workunit.client.1.smithi027.stdout:5/909: readlink d0/d1a/d22/d72/la9 0 2024-04-22T15:50:12.276 INFO:tasks.workunit.client.1.smithi027.stdout:8/854: dwrite d1/d3/d1c/d32/d41/d52/fd0 [0,4194304] 0 2024-04-22T15:50:12.280 INFO:tasks.workunit.client.1.smithi027.stdout:6/913: dread d5/dc/dd/f107 [0,4194304] 0 2024-04-22T15:50:12.286 INFO:tasks.workunit.client.1.smithi027.stdout:2/945: dwrite dd/df/d70/f6b [0,4194304] 0 2024-04-22T15:50:12.287 INFO:tasks.workunit.client.1.smithi027.stdout:2/946: chown dd/df/d1b/f5e 2946030 1 2024-04-22T15:50:12.289 INFO:tasks.workunit.client.1.smithi027.stdout:3/924: rename d28/d4b/d52/db2/d95 to d28/d5d/d62/dcf/d12d 0 2024-04-22T15:50:12.292 INFO:tasks.workunit.client.1.smithi027.stdout:3/925: chown d28/d4b/d52/la9 1 1 2024-04-22T15:50:12.298 INFO:tasks.workunit.client.1.smithi027.stdout:5/910: read d0/df/de/d40/f6a [2166444,24411] 0 2024-04-22T15:50:12.299 INFO:tasks.workunit.client.1.smithi027.stdout:8/855: dread d1/d1e/f33 [0,4194304] 0 2024-04-22T15:50:12.301 INFO:tasks.workunit.client.1.smithi027.stdout:6/914: fsync d5/dc/dd/d79/f8b 0 2024-04-22T15:50:12.304 INFO:tasks.workunit.client.1.smithi027.stdout:2/947: creat dd/df/d70/d5a/f124 x:0 0 0 2024-04-22T15:50:12.306 INFO:tasks.workunit.client.1.smithi027.stdout:9/821: rename d0/dc/d12/d30/d5b/dbc/df5/d104/d6b/d6e/l3c to d0/dc/de/l117 0 2024-04-22T15:50:12.307 INFO:tasks.workunit.client.1.smithi027.stdout:5/911: dwrite d0/db6/d111/f112 [0,4194304] 0 2024-04-22T15:50:12.310 INFO:tasks.workunit.client.1.smithi027.stdout:6/915: dread d5/dc/dd/dd0/d34/fd5 [0,4194304] 0 2024-04-22T15:50:12.315 INFO:tasks.workunit.client.1.smithi027.stdout:3/926: creat d28/d5d/d62/dcf/f12e x:0 0 0 2024-04-22T15:50:12.317 INFO:tasks.workunit.client.1.smithi027.stdout:1/707: dwrite d2/d5f/d64/d81/f50 [0,4194304] 0 2024-04-22T15:50:12.318 INFO:tasks.workunit.client.1.smithi027.stdout:8/856: fsync d1/d3/d1c/d49/d94/f105 0 2024-04-22T15:50:12.320 INFO:tasks.workunit.client.1.smithi027.stdout:1/708: dread - d2/d5f/d64/d81/d3d/d44/d9c/d9f/fd0 zero size 2024-04-22T15:50:12.320 INFO:tasks.workunit.client.1.smithi027.stdout:8/857: fsync d1/d3/d1c/d32/d41/d52/dec/f117 0 2024-04-22T15:50:12.321 INFO:tasks.workunit.client.1.smithi027.stdout:4/894: write d5/d7/f5b [493085,573] 0 2024-04-22T15:50:12.327 INFO:tasks.workunit.client.1.smithi027.stdout:2/948: mknod dd/df/d1b/d38/d60/d75/ddf/c125 0 2024-04-22T15:50:12.331 INFO:tasks.workunit.client.1.smithi027.stdout:9/822: fsync d0/dc/d12/d30/d99/da7/fda 0 2024-04-22T15:50:12.334 INFO:tasks.workunit.client.1.smithi027.stdout:7/906: truncate d0/d4c/d8b/d92/f112 27465 0 2024-04-22T15:50:12.336 INFO:tasks.workunit.client.1.smithi027.stdout:6/916: truncate d5/dc/d26/dee/f10d 1181152 0 2024-04-22T15:50:12.339 INFO:tasks.workunit.client.1.smithi027.stdout:3/927: rmdir d28/d4b/d52/d94/dc4/d10a 39 2024-04-22T15:50:12.354 INFO:tasks.workunit.client.1.smithi027.stdout:9/823: dread d0/d8b/fc0 [0,4194304] 0 2024-04-22T15:50:12.356 INFO:tasks.workunit.client.1.smithi027.stdout:5/912: creat d0/df/de/d24/d62/def/f129 x:0 0 0 2024-04-22T15:50:12.357 INFO:tasks.workunit.client.1.smithi027.stdout:2/949: dwrite dd/df/d70/d4b/d52/fb6 [0,4194304] 0 2024-04-22T15:50:12.357 INFO:tasks.workunit.client.1.smithi027.stdout:7/907: rmdir d0/d11 39 2024-04-22T15:50:12.359 INFO:tasks.workunit.client.1.smithi027.stdout:6/917: mknod d5/dc/dd/d79/c13d 0 2024-04-22T15:50:12.364 INFO:tasks.workunit.client.1.smithi027.stdout:4/895: dwrite d5/d11/fe5 [0,4194304] 0 2024-04-22T15:50:12.369 INFO:tasks.workunit.client.1.smithi027.stdout:8/858: dwrite d1/d3/d1c/d32/d41/d52/d61/fa2 [0,4194304] 0 2024-04-22T15:50:12.370 INFO:tasks.workunit.client.1.smithi027.stdout:9/824: dread d0/dc/d12/d30/d5b/d91/da9/fdc [0,4194304] 0 2024-04-22T15:50:12.372 INFO:tasks.workunit.client.1.smithi027.stdout:7/908: readlink d0/d11/d1c/d31/l43 0 2024-04-22T15:50:12.373 INFO:tasks.workunit.client.1.smithi027.stdout:6/918: symlink d5/d30/dbe/l13e 0 2024-04-22T15:50:12.375 INFO:tasks.workunit.client.1.smithi027.stdout:5/913: dread - d0/d1a/d22/d26/d5d/de3/d3e/dd5/ff2 zero size 2024-04-22T15:50:12.376 INFO:tasks.workunit.client.1.smithi027.stdout:2/950: dread - dd/df/d70/d4b/d52/d9a/da4/dad/fdd zero size 2024-04-22T15:50:12.382 INFO:tasks.workunit.client.1.smithi027.stdout:4/896: fsync d5/d45/d47/f8a 0 2024-04-22T15:50:12.388 INFO:tasks.workunit.client.1.smithi027.stdout:3/928: dwrite d28/fb5 [0,4194304] 0 2024-04-22T15:50:12.391 INFO:tasks.workunit.client.1.smithi027.stdout:1/709: getdents d2/d5f/d64/d81/d7e 0 2024-04-22T15:50:12.393 INFO:tasks.workunit.client.1.smithi027.stdout:2/951: mknod dd/df/d1b/dc9/dca/c126 0 2024-04-22T15:50:12.394 INFO:tasks.workunit.client.1.smithi027.stdout:6/919: write d5/dc/dd/dd0/d34/fbf [1585617,120686] 0 2024-04-22T15:50:12.407 INFO:tasks.workunit.client.1.smithi027.stdout:3/929: write f1d [1710375,131046] 0 2024-04-22T15:50:12.409 INFO:tasks.workunit.client.1.smithi027.stdout:8/859: creat d1/d3/d1c/f12e x:0 0 0 2024-04-22T15:50:12.410 INFO:tasks.workunit.client.1.smithi027.stdout:4/897: truncate d5/d11/d1d/d24/d123/f131 187786 0 2024-04-22T15:50:12.414 INFO:tasks.workunit.client.1.smithi027.stdout:6/920: rename d5/dc/l21 to d5/dc/d17/d108/l13f 0 2024-04-22T15:50:12.414 INFO:tasks.workunit.client.1.smithi027.stdout:9/825: fsync d0/dc/de/fc2 0 2024-04-22T15:50:12.448 INFO:tasks.workunit.client.1.smithi027.stdout:7/909: link d0/d11/d1c/d31/d56/fc6 d0/d11/d1c/d31/f12e 0 2024-04-22T15:50:12.449 INFO:tasks.workunit.client.1.smithi027.stdout:5/914: link d0/d1a/d22/d72/dac/cad d0/df/de/d40/c12a 0 2024-04-22T15:50:12.451 INFO:tasks.workunit.client.1.smithi027.stdout:6/921: write d5/dc/dd/dd0/f12 [12972240,124988] 0 2024-04-22T15:50:12.452 INFO:tasks.workunit.client.1.smithi027.stdout:9/826: mknod d0/dc/d12/d30/d99/c118 0 2024-04-22T15:50:12.455 INFO:tasks.workunit.client.1.smithi027.stdout:4/898: mknod d5/d11/d1d/d24/d134/c137 0 2024-04-22T15:50:12.462 INFO:tasks.workunit.client.1.smithi027.stdout:6/922: read d5/d30/d40/fce [4013611,34644] 0 2024-04-22T15:50:12.465 INFO:tasks.workunit.client.1.smithi027.stdout:5/915: rename d0/d1a/d22/d26/d32/d79/lc7 to d0/d97/dbf/l12b 0 2024-04-22T15:50:12.467 INFO:tasks.workunit.client.1.smithi027.stdout:9/827: fdatasync d0/dc/de/f40 0 2024-04-22T15:50:12.468 INFO:tasks.workunit.client.1.smithi027.stdout:8/860: write d1/d3/d1c/d43/d5f/fa4 [201640,127250] 0 2024-04-22T15:50:12.477 INFO:tasks.workunit.client.1.smithi027.stdout:4/899: mkdir d5/d45/d113/d138 0 2024-04-22T15:50:12.481 INFO:tasks.workunit.client.1.smithi027.stdout:9/828: mkdir d0/dc/d12/d30/d5b/dbc/df5/d104/ddb/dfd/d119 0 2024-04-22T15:50:12.483 INFO:tasks.workunit.client.1.smithi027.stdout:7/910: link d0/d11/d1c/d31/d62/f8d d0/d4c/d8b/d72/db2/de7/d110/f12f 0 2024-04-22T15:50:12.483 INFO:tasks.workunit.client.1.smithi027.stdout:6/923: dwrite d5/d30/d54/dd8/f10b [0,4194304] 0 2024-04-22T15:50:12.485 INFO:tasks.workunit.client.1.smithi027.stdout:4/900: creat d5/d45/d47/f139 x:0 0 0 2024-04-22T15:50:12.488 INFO:tasks.workunit.client.1.smithi027.stdout:8/861: mkdir d1/d3/d1c/d12f 0 2024-04-22T15:50:12.491 INFO:tasks.workunit.client.1.smithi027.stdout:5/916: creat d0/df/de/d24/f12c x:0 0 0 2024-04-22T15:50:12.492 INFO:tasks.workunit.client.1.smithi027.stdout:6/924: unlink d5/dc/d26/d3d/f83 0 2024-04-22T15:50:12.493 INFO:tasks.workunit.client.1.smithi027.stdout:4/901: rmdir d5/d11/dc1 39 2024-04-22T15:50:12.497 INFO:tasks.workunit.client.1.smithi027.stdout:8/862: creat d1/d3/d1c/d56/f130 x:0 0 0 2024-04-22T15:50:12.502 INFO:tasks.workunit.client.1.smithi027.stdout:7/911: creat d0/d4c/d8b/d72/db2/de7/d110/f130 x:0 0 0 2024-04-22T15:50:12.503 INFO:tasks.workunit.client.1.smithi027.stdout:7/912: fsync d0/d11/f4d 0 2024-04-22T15:50:12.506 INFO:tasks.workunit.client.1.smithi027.stdout:9/829: dread d0/dc/de/f3a [0,4194304] 0 2024-04-22T15:50:12.507 INFO:tasks.workunit.client.1.smithi027.stdout:9/830: readlink d0/dc/d12/d30/d5b/dbc/df5/d104/d6b/d6e/l114 0 2024-04-22T15:50:12.510 INFO:tasks.workunit.client.1.smithi027.stdout:6/925: dread d5/dc/dd/d79/d7b/d5d/f70 [0,4194304] 0 2024-04-22T15:50:12.581 INFO:tasks.workunit.client.1.smithi027.stdout:4/902: symlink d5/de/d30/de3/l13a 0 2024-04-22T15:50:12.587 INFO:tasks.workunit.client.1.smithi027.stdout:5/917: dwrite d0/df/de/d24/fb8 [4194304,4194304] 0 2024-04-22T15:50:12.587 INFO:tasks.workunit.client.1.smithi027.stdout:9/831: fsync d0/d64/f9c 0 2024-04-22T15:50:12.587 INFO:tasks.workunit.client.1.smithi027.stdout:6/926: creat d5/dc/d99/f140 x:0 0 0 2024-04-22T15:50:12.589 INFO:tasks.workunit.client.1.smithi027.stdout:7/913: dread d0/d11/d3a/d46/f82 [0,4194304] 0 2024-04-22T15:50:12.590 INFO:tasks.workunit.client.1.smithi027.stdout:8/863: symlink d1/d1e/d47/l131 0 2024-04-22T15:50:12.600 INFO:tasks.workunit.client.1.smithi027.stdout:7/914: creat d0/d11/d1c/ddc/f131 x:0 0 0 2024-04-22T15:50:12.606 INFO:tasks.workunit.client.1.smithi027.stdout:5/918: mknod d0/df/c12d 0 2024-04-22T15:50:12.608 INFO:tasks.workunit.client.1.smithi027.stdout:9/832: fsync d0/dc/de/f24 0 2024-04-22T15:50:12.615 INFO:tasks.workunit.client.1.smithi027.stdout:8/864: write d1/d3/d1c/d32/d41/fab [133432,26414] 0 2024-04-22T15:50:12.619 INFO:tasks.workunit.client.1.smithi027.stdout:4/903: link d5/d11/d6c/f112 d5/d11/d1d/f13b 0 2024-04-22T15:50:12.621 INFO:tasks.workunit.client.1.smithi027.stdout:4/904: read d5/d11/d6c/f9b [623414,119257] 0 2024-04-22T15:50:12.625 INFO:tasks.workunit.client.1.smithi027.stdout:5/919: mkdir d0/d97/dbf/d12e 0 2024-04-22T15:50:12.626 INFO:tasks.workunit.client.1.smithi027.stdout:7/915: write d0/d11/d3a/d46/fd6 [952720,33799] 0 2024-04-22T15:50:12.629 INFO:tasks.workunit.client.1.smithi027.stdout:9/833: fsync d0/d51/f8d 0 2024-04-22T15:50:12.632 INFO:tasks.workunit.client.1.smithi027.stdout:6/927: creat d5/dc/d17/f141 x:0 0 0 2024-04-22T15:50:12.641 INFO:tasks.workunit.client.1.smithi027.stdout:7/916: dread d0/d11/d1c/d31/d8a/ff8 [0,4194304] 0 2024-04-22T15:50:12.641 INFO:tasks.workunit.client.1.smithi027.stdout:8/865: write d1/d3/d1c/d32/d41/d52/fa1 [540670,82689] 0 2024-04-22T15:50:12.646 INFO:tasks.workunit.client.1.smithi027.stdout:9/834: symlink d0/dc/d12/d30/l11a 0 2024-04-22T15:50:12.647 INFO:tasks.workunit.client.1.smithi027.stdout:6/928: symlink d5/db/da5/l142 0 2024-04-22T15:50:12.648 INFO:tasks.workunit.client.1.smithi027.stdout:4/905: dwrite d5/d11/d1d/d7b/f85 [0,4194304] 0 2024-04-22T15:50:12.651 INFO:tasks.workunit.client.1.smithi027.stdout:5/920: mknod d0/d1a/d22/d26/d5d/de3/d3e/c12f 0 2024-04-22T15:50:12.652 INFO:tasks.workunit.client.1.smithi027.stdout:7/917: rmdir d0/d11/d1c 39 2024-04-22T15:50:12.656 INFO:tasks.workunit.client.1.smithi027.stdout:5/921: read d0/d1a/d22/d26/d32/f46 [479899,69054] 0 2024-04-22T15:50:12.660 INFO:tasks.workunit.client.1.smithi027.stdout:6/929: mkdir d5/dc/d17/d53/d8c/d143 0 2024-04-22T15:50:12.662 INFO:tasks.workunit.client.1.smithi027.stdout:4/906: symlink d5/d11/d1d/d24/d134/l13c 0 2024-04-22T15:50:12.662 INFO:tasks.workunit.client.1.smithi027.stdout:5/922: dwrite d0/df/de/dbb/fe1 [0,4194304] 0 2024-04-22T15:50:12.667 INFO:tasks.workunit.client.1.smithi027.stdout:7/918: dread - d0/d4c/f11e zero size 2024-04-22T15:50:12.671 INFO:tasks.workunit.client.1.smithi027.stdout:4/907: dwrite d5/d45/ddc/f132 [0,4194304] 0 2024-04-22T15:50:12.672 INFO:tasks.workunit.client.1.smithi027.stdout:4/908: readlink d5/lf1 0 2024-04-22T15:50:12.673 INFO:tasks.workunit.client.1.smithi027.stdout:4/909: chown d5/de/df2/de2/c11a 123048 1 2024-04-22T15:50:12.674 INFO:tasks.workunit.client.1.smithi027.stdout:4/910: stat d5/d45/d81 0 2024-04-22T15:50:12.680 INFO:tasks.workunit.client.1.smithi027.stdout:6/930: creat d5/db/d2e/d59/df0/f144 x:0 0 0 2024-04-22T15:50:12.681 INFO:tasks.workunit.client.1.smithi027.stdout:6/931: write d5/dc/d26/d49/d86/ffb [28859,61374] 0 2024-04-22T15:50:12.689 INFO:tasks.workunit.client.1.smithi027.stdout:9/835: dwrite d0/dc/d12/d30/d99/da7/fda [0,4194304] 0 2024-04-22T15:50:12.693 INFO:tasks.workunit.client.1.smithi027.stdout:4/911: dread d5/d11/d6c/f9b [0,4194304] 0 2024-04-22T15:50:12.696 INFO:tasks.workunit.client.1.smithi027.stdout:8/866: getdents d1/d3/d36/d57 0 2024-04-22T15:50:12.700 INFO:tasks.workunit.client.1.smithi027.stdout:9/836: dwrite d0/dc/d12/d30/d5b/dbc/df5/d104/ddb/dfd/f105 [0,4194304] 0 2024-04-22T15:50:12.710 INFO:tasks.workunit.client.1.smithi027.stdout:4/912: dwrite d5/de/d30/f12e [0,4194304] 0 2024-04-22T15:50:12.710 INFO:tasks.workunit.client.1.smithi027.stdout:6/932: dread d5/dc/dd/fa2 [0,4194304] 0 2024-04-22T15:50:12.711 INFO:tasks.workunit.client.1.smithi027.stdout:5/923: getdents d0/d1a/d22/d72/de8 0 2024-04-22T15:50:12.715 INFO:tasks.workunit.client.1.smithi027.stdout:7/919: mknod d0/d11/d1c/d2c/d96/c132 0 2024-04-22T15:50:12.716 INFO:tasks.workunit.client.1.smithi027.stdout:8/867: write d1/d3/d1c/d32/d41/d52/d61/fce [945942,26028] 0 2024-04-22T15:50:12.718 INFO:tasks.workunit.client.1.smithi027.stdout:5/924: mkdir d0/d1a/d22/d26/d5d/de3/d3e/d55/d130 0 2024-04-22T15:50:12.721 INFO:tasks.workunit.client.1.smithi027.stdout:6/933: mkdir d5/db/d145 0 2024-04-22T15:50:12.724 INFO:tasks.workunit.client.1.smithi027.stdout:8/868: symlink d1/d3/d1c/d12f/l132 0 2024-04-22T15:50:12.724 INFO:tasks.workunit.client.1.smithi027.stdout:6/934: read - d5/dc/dd/f110 zero size 2024-04-22T15:50:12.734 INFO:tasks.workunit.client.1.smithi027.stdout:6/935: symlink d5/dc/dd/d79/d7b/d5d/l146 0 2024-04-22T15:50:12.734 INFO:tasks.workunit.client.1.smithi027.stdout:7/920: link d0/d4c/d4e/c101 d0/d4c/d8b/d72/db2/de7/d110/c133 0 2024-04-22T15:50:12.737 INFO:tasks.workunit.client.1.smithi027.stdout:8/869: mkdir d1/d3/d1c/d133 0 2024-04-22T15:50:12.740 INFO:tasks.workunit.client.1.smithi027.stdout:6/936: dwrite d5/d30/d54/f7f [0,4194304] 0 2024-04-22T15:50:12.743 INFO:tasks.workunit.client.1.smithi027.stdout:5/925: getdents d0/df/d3d/d81 0 2024-04-22T15:50:12.753 INFO:tasks.workunit.client.1.smithi027.stdout:5/926: mknod d0/d1a/d22/d72/dac/dde/c131 0 2024-04-22T15:50:12.761 INFO:tasks.workunit.client.1.smithi027.stdout:7/921: truncate d0/d11/d1c/d31/d8a/fce 271695 0 2024-04-22T15:50:12.792 INFO:tasks.workunit.client.1.smithi027.stdout:8/870: unlink d1/d3/d1c/d32/d41/d52/d61/d8f/c9f 0 2024-04-22T15:50:12.794 INFO:tasks.workunit.client.1.smithi027.stdout:6/937: link d5/cf4 d5/dc/dd/dd0/d34/c147 0 2024-04-22T15:50:12.805 INFO:tasks.workunit.client.1.smithi027.stdout:5/927: dread d0/d1a/d22/d26/d32/f46 [0,4194304] 0 2024-04-22T15:50:12.806 INFO:tasks.workunit.client.1.smithi027.stdout:5/928: chown d0/df/d3d/d81/f98 26074043 1 2024-04-22T15:50:12.825 INFO:tasks.workunit.client.1.smithi027.stdout:3/930: sync 2024-04-22T15:50:12.825 INFO:tasks.workunit.client.1.smithi027.stdout:1/710: sync 2024-04-22T15:50:12.825 INFO:tasks.workunit.client.1.smithi027.stdout:2/952: sync 2024-04-22T15:50:12.857 INFO:tasks.workunit.client.1.smithi027.stdout:6/938: unlink d5/dc/d17/d8a/c95 0 2024-04-22T15:50:12.858 INFO:tasks.workunit.client.1.smithi027.stdout:6/939: chown d5/dc/dd/d79/d7b/d5d/dba 32326712 1 2024-04-22T15:50:12.859 INFO:tasks.workunit.client.1.smithi027.stdout:2/953: mkdir dd/df/d1b/dc9/dca/d127 0 2024-04-22T15:50:12.862 INFO:tasks.workunit.client.1.smithi027.stdout:1/711: mkdir d2/d5f/d64/d81/d3d/d44/d9c/d9f/de9 0 2024-04-22T15:50:12.863 INFO:tasks.workunit.client.1.smithi027.stdout:5/929: dwrite d0/d1a/d22/d26/f2a [0,4194304] 0 2024-04-22T15:50:12.869 INFO:tasks.workunit.client.1.smithi027.stdout:6/940: fdatasync d5/dc/dae/f118 0 2024-04-22T15:50:12.873 INFO:tasks.workunit.client.1.smithi027.stdout:5/930: dwrite d0/d1a/d22/d26/d32/d79/fa7 [0,4194304] 0 2024-04-22T15:50:12.875 INFO:tasks.workunit.client.1.smithi027.stdout:1/712: fsync d2/d5f/fd1 0 2024-04-22T15:50:12.878 INFO:tasks.workunit.client.1.smithi027.stdout:7/922: rename d0/d11/d3a/d46/f9c to d0/d11/d1c/d2c/d96/f134 0 2024-04-22T15:50:12.884 INFO:tasks.workunit.client.1.smithi027.stdout:1/713: dwrite d2/d1d/d56/dc4/fce [0,4194304] 0 2024-04-22T15:50:12.887 INFO:tasks.workunit.client.1.smithi027.stdout:1/714: write d2/d1d/f91 [3353999,11631] 0 2024-04-22T15:50:12.890 INFO:tasks.workunit.client.1.smithi027.stdout:7/923: dwrite d0/d11/d1c/d2c/f38 [0,4194304] 0 2024-04-22T15:50:12.891 INFO:tasks.workunit.client.1.smithi027.stdout:1/715: chown d2/d5f/d64/d81/d3d/d44/c4a 86256 1 2024-04-22T15:50:12.892 INFO:tasks.workunit.client.1.smithi027.stdout:7/924: write d0/d4c/d8b/d72/db2/fac [2566175,95549] 0 2024-04-22T15:50:12.893 INFO:tasks.workunit.client.1.smithi027.stdout:7/925: chown d0/d4c/d8b/f116 296680797 1 2024-04-22T15:50:12.893 INFO:tasks.workunit.client.1.smithi027.stdout:1/716: chown d2/d5f/d64/fe7 453 1 2024-04-22T15:50:12.894 INFO:tasks.workunit.client.1.smithi027.stdout:7/926: chown d0/d4c/f11e 2572 1 2024-04-22T15:50:12.894 INFO:tasks.workunit.client.1.smithi027.stdout:5/931: unlink d0/d1a/d22/d26/d5d/de3/d3e/d55/f7d 0 2024-04-22T15:50:12.895 INFO:tasks.workunit.client.1.smithi027.stdout:5/932: chown d0/db6/f105 121125 1 2024-04-22T15:50:12.896 INFO:tasks.workunit.client.1.smithi027.stdout:7/927: write d0/d11/d1c/ddc/f131 [529981,87044] 0 2024-04-22T15:50:12.903 INFO:tasks.workunit.client.1.smithi027.stdout:8/871: truncate d1/d3/d1c/d43/f71 3575137 0 2024-04-22T15:50:12.913 INFO:tasks.workunit.client.1.smithi027.stdout:6/941: mkdir d5/dc/d17/d8a/d131/d148 0 2024-04-22T15:50:12.924 INFO:tasks.workunit.client.1.smithi027.stdout:3/931: truncate d28/d5d/d79/fcd 565652 0 2024-04-22T15:50:12.927 INFO:tasks.workunit.client.1.smithi027.stdout:6/942: creat d5/db/d2e/d59/df0/f149 x:0 0 0 2024-04-22T15:50:12.934 INFO:tasks.workunit.client.1.smithi027.stdout:2/954: rename dd/df/d1b/d38/dd5/ff7 to dd/df/d1b/f128 0 2024-04-22T15:50:12.934 INFO:tasks.workunit.client.1.smithi027.stdout:2/955: chown dd/df/d1b/d96/fc3 9 1 2024-04-22T15:50:12.935 INFO:tasks.workunit.client.1.smithi027.stdout:3/932: rmdir d28/d4b/d52/db2/d101 39 2024-04-22T15:50:12.940 INFO:tasks.workunit.client.1.smithi027.stdout:6/943: creat d5/dc/d17/d53/d8c/f14a x:0 0 0 2024-04-22T15:50:12.940 INFO:tasks.workunit.client.1.smithi027.stdout:4/913: sync 2024-04-22T15:50:12.940 INFO:tasks.workunit.client.1.smithi027.stdout:9/837: sync 2024-04-22T15:50:12.942 INFO:tasks.workunit.client.1.smithi027.stdout:4/914: write d5/d45/d47/f139 [288193,99198] 0 2024-04-22T15:50:12.943 INFO:tasks.workunit.client.1.smithi027.stdout:6/944: write d5/dc/d26/d3d/f13a [25286,37452] 0 2024-04-22T15:50:12.944 INFO:tasks.workunit.client.1.smithi027.stdout:6/945: chown d5/dc/dd/d79/c13d 10 1 2024-04-22T15:50:12.946 INFO:tasks.workunit.client.1.smithi027.stdout:2/956: rename dd/d43/fb4 to dd/df/d1b/d38/dd5/f129 0 2024-04-22T15:50:12.947 INFO:tasks.workunit.client.1.smithi027.stdout:5/933: getdents d0/df/de/d24/d62/def 0 2024-04-22T15:50:12.949 INFO:tasks.workunit.client.1.smithi027.stdout:9/838: creat d0/d51/de6/f11b x:0 0 0 2024-04-22T15:50:12.951 INFO:tasks.workunit.client.1.smithi027.stdout:3/933: dread d28/f56 [0,4194304] 0 2024-04-22T15:50:12.951 INFO:tasks.workunit.client.1.smithi027.stdout:4/915: unlink d5/d11/d1d/d54/d8c/f106 0 2024-04-22T15:50:12.953 INFO:tasks.workunit.client.1.smithi027.stdout:4/916: chown d5/de/d11d/l109 196165437 1 2024-04-22T15:50:12.953 INFO:tasks.workunit.client.1.smithi027.stdout:3/934: write d28/d4b/d52/d9a/ffe [770837,5122] 0 2024-04-22T15:50:12.954 INFO:tasks.workunit.client.1.smithi027.stdout:6/946: symlink d5/d30/d54/dd8/dfc/l14b 0 2024-04-22T15:50:12.957 INFO:tasks.workunit.client.1.smithi027.stdout:5/934: creat d0/d1a/d22/d72/de8/f132 x:0 0 0 2024-04-22T15:50:12.959 INFO:tasks.workunit.client.1.smithi027.stdout:9/839: mknod d0/dc/d12/d30/d5b/dbc/df5/d104/ddb/dfd/c11c 0 2024-04-22T15:50:12.960 INFO:tasks.workunit.client.1.smithi027.stdout:4/917: symlink d5/de/dc5/l13d 0 2024-04-22T15:50:12.962 INFO:tasks.workunit.client.1.smithi027.stdout:3/935: creat d28/d5d/d62/dcf/d12d/f12f x:0 0 0 2024-04-22T15:50:12.964 INFO:tasks.workunit.client.1.smithi027.stdout:6/947: mknod d5/dc/d26/d49/d86/c14c 0 2024-04-22T15:50:12.969 INFO:tasks.workunit.client.1.smithi027.stdout:2/957: link dd/dae/lc8 dd/df/d1b/dc9/dca/d127/l12a 0 2024-04-22T15:50:12.971 INFO:tasks.workunit.client.1.smithi027.stdout:3/936: rename d28/d4b/d52/d94/dc4/d10a to d28/d4b/dc6/d130 0 2024-04-22T15:50:12.972 INFO:tasks.workunit.client.1.smithi027.stdout:3/937: write d28/fb9 [2349549,47174] 0 2024-04-22T15:50:12.973 INFO:tasks.workunit.client.1.smithi027.stdout:6/948: mknod d5/dc/dd/d79/d7b/d5d/dba/df2/c14d 0 2024-04-22T15:50:12.973 INFO:tasks.workunit.client.1.smithi027.stdout:4/918: dread d5/de/d30/d5d/f9f [0,4194304] 0 2024-04-22T15:50:12.978 INFO:tasks.workunit.client.1.smithi027.stdout:3/938: chown d28/d4b/f7c 47146 1 2024-04-22T15:50:12.985 INFO:tasks.workunit.client.1.smithi027.stdout:9/840: rename d0/dc/de/db5/dbf/cd6 to d0/dc/de/d107/c11d 0 2024-04-22T15:50:12.986 INFO:tasks.workunit.client.1.smithi027.stdout:3/939: write d28/d5d/d62/dcf/f12e [988404,117171] 0 2024-04-22T15:50:12.995 INFO:tasks.workunit.client.1.smithi027.stdout:4/919: link d5/d11/d1d/f5e d5/d11/d6c/dca/d12c/f13e 0 2024-04-22T15:50:13.001 INFO:tasks.workunit.client.1.smithi027.stdout:1/717: dwrite d2/d5f/d64/d81/d7e/fb4 [0,4194304] 0 2024-04-22T15:50:13.005 INFO:tasks.workunit.client.1.smithi027.stdout:6/949: getdents d5/dc/dd 0 2024-04-22T15:50:13.006 INFO:tasks.workunit.client.1.smithi027.stdout:6/950: readlink d5/d30/d40/l4c 0 2024-04-22T15:50:13.020 INFO:tasks.workunit.client.1.smithi027.stdout:7/928: truncate d0/d4c/d4e/ff0 599628 0 2024-04-22T15:50:13.025 INFO:tasks.workunit.client.1.smithi027.stdout:4/920: unlink d5/d45/ddc/ff5 0 2024-04-22T15:50:13.026 INFO:tasks.workunit.client.1.smithi027.stdout:1/718: truncate d2/d5f/d64/d81/d69/fd4 4351312 0 2024-04-22T15:50:13.027 INFO:tasks.workunit.client.1.smithi027.stdout:8/872: write d1/d3/d1c/d32/d41/f83 [1317305,51007] 0 2024-04-22T15:50:13.028 INFO:tasks.workunit.client.1.smithi027.stdout:4/921: truncate d5/d11/d1d/d24/f121 575837 0 2024-04-22T15:50:13.036 INFO:tasks.workunit.client.1.smithi027.stdout:8/873: rename d1/d1e/d47/d53/d76/fdf to d1/d3/f134 0 2024-04-22T15:50:13.039 INFO:tasks.workunit.client.1.smithi027.stdout:4/922: rmdir d5/de/df2/de2 39 2024-04-22T15:50:13.040 INFO:tasks.workunit.client.1.smithi027.stdout:1/719: write d2/d36/d74/f79 [3817184,82465] 0 2024-04-22T15:50:13.042 INFO:tasks.workunit.client.1.smithi027.stdout:7/929: creat d0/d11/d1c/d2c/f135 x:0 0 0 2024-04-22T15:50:13.048 INFO:tasks.workunit.client.1.smithi027.stdout:1/720: dwrite d2/d5f/d64/d81/d7e/fb4 [0,4194304] 0 2024-04-22T15:50:13.049 INFO:tasks.workunit.client.1.smithi027.stdout:1/721: chown d2/c4c 19341121 1 2024-04-22T15:50:13.051 INFO:tasks.workunit.client.1.smithi027.stdout:7/930: creat d0/d11/d1c/ddc/f136 x:0 0 0 2024-04-22T15:50:13.055 INFO:tasks.workunit.client.1.smithi027.stdout:4/923: dread d5/f26 [0,4194304] 0 2024-04-22T15:50:13.060 INFO:tasks.workunit.client.1.smithi027.stdout:1/722: mknod d2/d5f/d64/d81/cea 0 2024-04-22T15:50:13.064 INFO:tasks.workunit.client.1.smithi027.stdout:7/931: link d0/d11/d1c/f7e d0/d4c/d8b/d72/db2/de7/d110/f137 0 2024-04-22T15:50:13.064 INFO:tasks.workunit.client.1.smithi027.stdout:3/940: rmdir d28/d5d/d62/dcf 39 2024-04-22T15:50:13.065 INFO:tasks.workunit.client.1.smithi027.stdout:4/924: getdents d5/d45/d113/d138 0 2024-04-22T15:50:13.066 INFO:tasks.workunit.client.1.smithi027.stdout:7/932: read d0/d11/d1c/d31/d56/f58 [107732,88969] 0 2024-04-22T15:50:13.071 INFO:tasks.workunit.client.1.smithi027.stdout:7/933: dread d0/d11/d3a/f45 [0,4194304] 0 2024-04-22T15:50:13.074 INFO:tasks.workunit.client.1.smithi027.stdout:3/941: unlink d28/d4b/d52/d94/dc4/dfc/lfd 0 2024-04-22T15:50:13.076 INFO:tasks.workunit.client.1.smithi027.stdout:5/935: write d0/d1a/d22/d26/d32/f67 [1729179,93845] 0 2024-04-22T15:50:13.079 INFO:tasks.workunit.client.1.smithi027.stdout:7/934: symlink d0/d4c/d8b/d72/db2/de7/d110/l138 0 2024-04-22T15:50:13.081 INFO:tasks.workunit.client.1.smithi027.stdout:4/925: dread d5/d7/fb [8388608,4194304] 0 2024-04-22T15:50:13.091 INFO:tasks.workunit.client.1.smithi027.stdout:4/926: unlink d5/d11/d1d/d7b/fdd 0 2024-04-22T15:50:13.091 INFO:tasks.workunit.client.1.smithi027.stdout:2/958: write dd/df/d1b/f40 [5048751,46418] 0 2024-04-22T15:50:13.092 INFO:tasks.workunit.client.1.smithi027.stdout:2/959: chown dd/df/d1b 87 1 2024-04-22T15:50:13.094 INFO:tasks.workunit.client.1.smithi027.stdout:5/936: dread d0/d1a/d22/d26/f2a [4194304,4194304] 0 2024-04-22T15:50:13.095 INFO:tasks.workunit.client.1.smithi027.stdout:2/960: truncate dd/df/d70/f45 4775843 0 2024-04-22T15:50:13.095 INFO:tasks.workunit.client.1.smithi027.stdout:3/942: symlink d28/d4b/d52/d94/d105/d110/l131 0 2024-04-22T15:50:13.096 INFO:tasks.workunit.client.1.smithi027.stdout:2/961: truncate dd/df/d70/d5a/f124 334851 0 2024-04-22T15:50:13.098 INFO:tasks.workunit.client.1.smithi027.stdout:7/935: fdatasync d0/d4c/d8b/d72/db2/f69 0 2024-04-22T15:50:13.099 INFO:tasks.workunit.client.1.smithi027.stdout:7/936: stat d0/d11/d1c/d31/d56 0 2024-04-22T15:50:13.103 INFO:tasks.workunit.client.1.smithi027.stdout:5/937: unlink d0/df/de/l1f 0 2024-04-22T15:50:13.105 INFO:tasks.workunit.client.1.smithi027.stdout:3/943: fsync d28/d5d/d79/fa1 0 2024-04-22T15:50:13.112 INFO:tasks.workunit.client.1.smithi027.stdout:5/938: creat d0/d97/dbf/f133 x:0 0 0 2024-04-22T15:50:13.114 INFO:tasks.workunit.client.1.smithi027.stdout:6/951: dwrite d5/dc/d99/fab [0,4194304] 0 2024-04-22T15:50:13.115 INFO:tasks.workunit.client.1.smithi027.stdout:9/841: sync 2024-04-22T15:50:13.127 INFO:tasks.workunit.client.1.smithi027.stdout:4/927: link d5/d45/d5a/dcd/f130 d5/de/f13f 0 2024-04-22T15:50:13.128 INFO:tasks.workunit.client.1.smithi027.stdout:8/874: dwrite d1/d1e/f2f [0,4194304] 0 2024-04-22T15:50:13.130 INFO:tasks.workunit.client.1.smithi027.stdout:7/937: mkdir d0/d11/d3a/d46/d94/d5a/d119/d10e/d139 0 2024-04-22T15:50:13.132 INFO:tasks.workunit.client.1.smithi027.stdout:5/939: mknod d0/d1a/d22/d26/d32/d88/d90/d107/c134 0 2024-04-22T15:50:13.135 INFO:tasks.workunit.client.1.smithi027.stdout:5/940: dread d0/d1a/d22/d26/f2a [4194304,4194304] 0 2024-04-22T15:50:13.136 INFO:tasks.workunit.client.1.smithi027.stdout:1/723: write d2/d5f/d64/d81/d7e/fd2 [960847,123890] 0 2024-04-22T15:50:13.138 INFO:tasks.workunit.client.1.smithi027.stdout:6/952: rename d5/dc/dd/dd0/d34/f106 to d5/db/d2e/f14e 0 2024-04-22T15:50:13.138 INFO:tasks.workunit.client.1.smithi027.stdout:5/941: write d0/d1a/d22/d26/d5d/de3/d3e/fbe [1757778,27844] 0 2024-04-22T15:50:13.150 INFO:tasks.workunit.client.1.smithi027.stdout:8/875: creat d1/d3/d36/d88/da9/f135 x:0 0 0 2024-04-22T15:50:13.152 INFO:tasks.workunit.client.1.smithi027.stdout:7/938: mkdir d0/d4c/d8b/d13a 0 2024-04-22T15:50:13.154 INFO:tasks.workunit.client.1.smithi027.stdout:9/842: creat d0/dc/de/df7/f11e x:0 0 0 2024-04-22T15:50:13.157 INFO:tasks.workunit.client.1.smithi027.stdout:6/953: creat d5/d30/dbe/f14f x:0 0 0 2024-04-22T15:50:13.158 INFO:tasks.workunit.client.1.smithi027.stdout:5/942: write d0/d1a/f3a [3326332,41960] 0 2024-04-22T15:50:13.161 INFO:tasks.workunit.client.1.smithi027.stdout:3/944: symlink d28/d5d/d62/dcf/d12d/l132 0 2024-04-22T15:50:13.162 INFO:tasks.workunit.client.1.smithi027.stdout:3/945: read - d28/d4b/d52/d9a/fab zero size 2024-04-22T15:50:13.162 INFO:tasks.workunit.client.1.smithi027.stdout:4/928: symlink d5/d11/d6c/dca/d12c/l140 0 2024-04-22T15:50:13.164 INFO:tasks.workunit.client.1.smithi027.stdout:3/946: fdatasync d28/d5d/db0/dce/dda/f10b 0 2024-04-22T15:50:13.165 INFO:tasks.workunit.client.1.smithi027.stdout:9/843: creat d0/dc/d12/d30/d99/da7/f11f x:0 0 0 2024-04-22T15:50:13.168 INFO:tasks.workunit.client.1.smithi027.stdout:5/943: creat d0/d1a/d22/d72/de8/f135 x:0 0 0 2024-04-22T15:50:13.169 INFO:tasks.workunit.client.1.smithi027.stdout:5/944: readlink d0/df/d10/l1d 0 2024-04-22T15:50:13.169 INFO:tasks.workunit.client.1.smithi027.stdout:8/876: unlink d1/d3/d1c/d28/c30 0 2024-04-22T15:50:13.170 INFO:tasks.workunit.client.1.smithi027.stdout:2/962: write dd/df/d1b/d96/dc5/dd1/fe5 [2618834,63965] 0 2024-04-22T15:50:13.170 INFO:tasks.workunit.client.1.smithi027.stdout:4/929: dread - d5/d11/d1d/d54/d8c/ff0 zero size 2024-04-22T15:50:13.172 INFO:tasks.workunit.client.1.smithi027.stdout:7/939: symlink d0/d4c/d8b/d13a/l13b 0 2024-04-22T15:50:13.172 INFO:tasks.workunit.client.1.smithi027.stdout:4/930: stat d5/d11/da6 0 2024-04-22T15:50:13.173 INFO:tasks.workunit.client.1.smithi027.stdout:7/940: readlink d0/d4c/d8b/d13a/l13b 0 2024-04-22T15:50:13.174 INFO:tasks.workunit.client.1.smithi027.stdout:7/941: stat d0/d11/d1c/ddc 0 2024-04-22T15:50:13.175 INFO:tasks.workunit.client.1.smithi027.stdout:1/724: link d2/d5f/d64/d81/d69/d7d/fa8 d2/d5f/d64/d81/d3d/d82/d85/feb 0 2024-04-22T15:50:13.178 INFO:tasks.workunit.client.1.smithi027.stdout:2/963: readlink dd/df/d1b/d38/l11e 0 2024-04-22T15:50:13.180 INFO:tasks.workunit.client.1.smithi027.stdout:8/877: write d1/d3/d1c/d32/d41/d52/d61/d8f/ddd/ffb [1075049,130185] 0 2024-04-22T15:50:13.182 INFO:tasks.workunit.client.1.smithi027.stdout:4/931: chown d5/de/d30/d5d/c8e 6759337 1 2024-04-22T15:50:13.184 INFO:tasks.workunit.client.1.smithi027.stdout:9/844: link d0/dc/d12/d30/d5b/f65 d0/dc/d12/d30/d5b/d91/da9/f120 0 2024-04-22T15:50:13.187 INFO:tasks.workunit.client.1.smithi027.stdout:5/945: getdents d0/df/d3d/d81/d116 0 2024-04-22T15:50:13.187 INFO:tasks.workunit.client.1.smithi027.stdout:7/942: read d0/d3f/f5b [3335775,70347] 0 2024-04-22T15:50:13.188 INFO:tasks.workunit.client.1.smithi027.stdout:7/943: dread - d0/d4c/d4e/df5/f11d zero size 2024-04-22T15:50:13.189 INFO:tasks.workunit.client.1.smithi027.stdout:3/947: creat d28/d5d/d62/dcf/d12d/f133 x:0 0 0 2024-04-22T15:50:13.192 INFO:tasks.workunit.client.1.smithi027.stdout:4/932: rmdir d5/d11/d6c/da3/d10b 39 2024-04-22T15:50:13.193 INFO:tasks.workunit.client.1.smithi027.stdout:9/845: write d0/dc/d12/d30/d5b/dbc/df5/d104/d6b/d6e/f31 [483705,58132] 0 2024-04-22T15:50:13.194 INFO:tasks.workunit.client.1.smithi027.stdout:9/846: chown d0/d8b/ld8 8 1 2024-04-22T15:50:13.196 INFO:tasks.workunit.client.1.smithi027.stdout:1/725: creat d2/d5f/d64/d81/d3d/d44/d9c/d9f/de9/fec x:0 0 0 2024-04-22T15:50:13.199 INFO:tasks.workunit.client.1.smithi027.stdout:9/847: dread d0/d8b/fc0 [0,4194304] 0 2024-04-22T15:50:13.209 INFO:tasks.workunit.client.1.smithi027.stdout:4/933: symlink d5/d11/d1d/d54/l141 0 2024-04-22T15:50:13.213 INFO:tasks.workunit.client.1.smithi027.stdout:9/848: mknod d0/dc/d12/d30/d5b/dbc/c121 0 2024-04-22T15:50:13.216 INFO:tasks.workunit.client.1.smithi027.stdout:9/849: truncate d0/dc/d12/d30/d5b/d91/da9/dfe/f10f 887908 0 2024-04-22T15:50:13.217 INFO:tasks.workunit.client.1.smithi027.stdout:3/948: getdents d28/d4b/d52/d94/dae/d104 0 2024-04-22T15:50:13.218 INFO:tasks.workunit.client.1.smithi027.stdout:8/878: link d1/d3/c104 d1/d3/d1c/d133/c136 0 2024-04-22T15:50:13.219 INFO:tasks.workunit.client.1.smithi027.stdout:2/964: truncate dd/df/d1b/dc9/dca/ffa 1036157 0 2024-04-22T15:50:13.221 INFO:tasks.workunit.client.1.smithi027.stdout:1/726: creat d2/d5f/d64/d81/d3d/d82/dc2/fed x:0 0 0 2024-04-22T15:50:13.222 INFO:tasks.workunit.client.1.smithi027.stdout:4/934: mknod d5/de/d30/d5d/c142 0 2024-04-22T15:50:13.224 INFO:tasks.workunit.client.1.smithi027.stdout:1/727: write d2/d1d/fd9 [1368878,32651] 0 2024-04-22T15:50:13.225 INFO:tasks.workunit.client.1.smithi027.stdout:4/935: chown d5/de/da9/fc7 366 1 2024-04-22T15:50:13.227 INFO:tasks.workunit.client.1.smithi027.stdout:4/936: readlink d5/d45/d47/l11f 0 2024-04-22T15:50:13.228 INFO:tasks.workunit.client.1.smithi027.stdout:8/879: dwrite d1/d3/d1c/d43/d82/f12a [0,4194304] 0 2024-04-22T15:50:13.228 INFO:tasks.workunit.client.1.smithi027.stdout:7/944: link d0/d4c/d8b/d92/da3/da8/f25 d0/d11/d1c/d31/f13c 0 2024-04-22T15:50:13.230 INFO:tasks.workunit.client.1.smithi027.stdout:1/728: dread - d2/d5f/d64/d81/d3d/d82/dc2/fed zero size 2024-04-22T15:50:13.231 INFO:tasks.workunit.client.1.smithi027.stdout:3/949: read d28/d5d/d62/f8f [3575181,14485] 0 2024-04-22T15:50:13.233 INFO:tasks.workunit.client.1.smithi027.stdout:9/850: rename d0/dc/d12/d30/d5b/dbc/df5/d104/cf8 to d0/dc/d12/d30/d5b/dbc/c122 0 2024-04-22T15:50:13.235 INFO:tasks.workunit.client.1.smithi027.stdout:2/965: dread dd/f8e [0,4194304] 0 2024-04-22T15:50:13.238 INFO:tasks.workunit.client.1.smithi027.stdout:5/946: link d0/df/l5f d0/d1a/d22/d26/d5d/de3/l136 0 2024-04-22T15:50:13.246 INFO:tasks.workunit.client.1.smithi027.stdout:8/880: mkdir d1/d1e/d47/d53/d76/d137 0 2024-04-22T15:50:13.248 INFO:tasks.workunit.client.1.smithi027.stdout:7/945: creat d0/d11/d1c/d31/d8a/f13d x:0 0 0 2024-04-22T15:50:13.252 INFO:tasks.workunit.client.1.smithi027.stdout:9/851: mknod d0/dc/de/db5/c123 0 2024-04-22T15:50:13.257 INFO:tasks.workunit.client.1.smithi027.stdout:4/937: write d5/d42/f9e [1270770,102213] 0 2024-04-22T15:50:13.259 INFO:tasks.workunit.client.1.smithi027.stdout:1/729: symlink d2/db9/lee 0 2024-04-22T15:50:13.260 INFO:tasks.workunit.client.1.smithi027.stdout:8/881: symlink d1/d1e/d47/d53/d76/l138 0 2024-04-22T15:50:13.261 INFO:tasks.workunit.client.1.smithi027.stdout:7/946: fsync d0/d11/d1c/ddc/ffd 0 2024-04-22T15:50:13.265 INFO:tasks.workunit.client.1.smithi027.stdout:5/947: mknod d0/df/d3d/d81/d116/c137 0 2024-04-22T15:50:13.266 INFO:tasks.workunit.client.1.smithi027.stdout:5/948: fdatasync d0/d1a/d22/d72/f7b 0 2024-04-22T15:50:13.268 INFO:tasks.workunit.client.1.smithi027.stdout:1/730: rmdir d2/d36/d74 39 2024-04-22T15:50:13.269 INFO:tasks.workunit.client.1.smithi027.stdout:8/882: symlink d1/d3/d36/d88/l139 0 2024-04-22T15:50:13.275 INFO:tasks.workunit.client.1.smithi027.stdout:7/947: dread d0/d11/d3a/d46/fd6 [0,4194304] 0 2024-04-22T15:50:13.275 INFO:tasks.workunit.client.1.smithi027.stdout:8/883: dwrite d1/d4f/f10a [0,4194304] 0 2024-04-22T15:50:13.278 INFO:tasks.workunit.client.1.smithi027.stdout:2/966: rename dd/df/d70/f6b to dd/df/d70/d4b/d52/d9a/da4/dad/f12b 0 2024-04-22T15:50:13.280 INFO:tasks.workunit.client.1.smithi027.stdout:2/967: fsync dd/df/fe6 0 2024-04-22T15:50:13.281 INFO:tasks.workunit.client.1.smithi027.stdout:4/938: mkdir d5/de/d143 0 2024-04-22T15:50:13.284 INFO:tasks.workunit.client.1.smithi027.stdout:1/731: symlink d2/db9/lef 0 2024-04-22T15:50:13.285 INFO:tasks.workunit.client.1.smithi027.stdout:3/950: getdents d28/d4b/dc6/d130 0 2024-04-22T15:50:13.288 INFO:tasks.workunit.client.1.smithi027.stdout:8/884: read d1/f92 [656525,130109] 0 2024-04-22T15:50:13.289 INFO:tasks.workunit.client.1.smithi027.stdout:7/948: mknod d0/d11/d3a/d46/c13e 0 2024-04-22T15:50:13.290 INFO:tasks.workunit.client.1.smithi027.stdout:7/949: write d0/d11/d1c/ddc/f131 [450624,14648] 0 2024-04-22T15:50:13.292 INFO:tasks.workunit.client.1.smithi027.stdout:7/950: chown d0/d11/d1c/d2c/d96/f10a 1259718 1 2024-04-22T15:50:13.292 INFO:tasks.workunit.client.1.smithi027.stdout:9/852: rename d0/dc/d12/d30/d5b/dbc/df5/d104/d6b/d6e/f31 to d0/dc/d12/d30/d99/da7/f124 0 2024-04-22T15:50:13.300 INFO:tasks.workunit.client.1.smithi027.stdout:7/951: dwrite d0/d11/d1c/ddc/f136 [0,4194304] 0 2024-04-22T15:50:13.301 INFO:tasks.workunit.client.1.smithi027.stdout:1/732: mkdir d2/df0 0 2024-04-22T15:50:13.306 INFO:tasks.workunit.client.1.smithi027.stdout:6/954: sync 2024-04-22T15:50:13.311 INFO:tasks.workunit.client.1.smithi027.stdout:9/853: unlink d0/c27 0 2024-04-22T15:50:13.317 INFO:tasks.workunit.client.1.smithi027.stdout:2/968: unlink dd/dae/lc8 0 2024-04-22T15:50:13.319 INFO:tasks.workunit.client.1.smithi027.stdout:2/969: stat dd/df/d1b/d38/d60/d75/fb3 0 2024-04-22T15:50:13.319 INFO:tasks.workunit.client.1.smithi027.stdout:8/885: dread d1/d3/d1c/d32/f8a [4194304,4194304] 0 2024-04-22T15:50:13.321 INFO:tasks.workunit.client.1.smithi027.stdout:1/733: dread d2/d5f/d64/d81/d3d/d44/f86 [0,4194304] 0 2024-04-22T15:50:13.322 INFO:tasks.workunit.client.1.smithi027.stdout:8/886: readlink d1/d3/d1c/d32/d41/d52/d61/d8f/ddd/lf2 0 2024-04-22T15:50:13.322 INFO:tasks.workunit.client.1.smithi027.stdout:4/939: mkdir d5/d11/d6c/da3/d144 0 2024-04-22T15:50:13.325 INFO:tasks.workunit.client.1.smithi027.stdout:7/952: unlink d0/d11/d1c/d2c/fc2 0 2024-04-22T15:50:13.326 INFO:tasks.workunit.client.1.smithi027.stdout:3/951: write d28/f63 [207460,62174] 0 2024-04-22T15:50:13.328 INFO:tasks.workunit.client.1.smithi027.stdout:5/949: dwrite d0/d1a/d22/d26/fc4 [0,4194304] 0 2024-04-22T15:50:13.330 INFO:tasks.workunit.client.1.smithi027.stdout:6/955: fdatasync d5/d30/dbe/d112/ff3 0 2024-04-22T15:50:13.334 INFO:tasks.workunit.client.1.smithi027.stdout:2/970: unlink dd/df/d70/d4b/d52/d9a/da4/cbf 0 2024-04-22T15:50:13.335 INFO:tasks.workunit.client.1.smithi027.stdout:1/734: creat d2/d5f/ff1 x:0 0 0 2024-04-22T15:50:13.336 INFO:tasks.workunit.client.1.smithi027.stdout:1/735: readlink d2/lb1 0 2024-04-22T15:50:13.339 INFO:tasks.workunit.client.1.smithi027.stdout:9/854: dwrite d0/d64/f7b [4194304,4194304] 0 2024-04-22T15:50:13.349 INFO:tasks.workunit.client.1.smithi027.stdout:3/952: creat d28/d69/f134 x:0 0 0 2024-04-22T15:50:13.350 INFO:tasks.workunit.client.1.smithi027.stdout:5/950: mknod d0/d1a/d22/d72/dac/dde/dfb/c138 0 2024-04-22T15:50:13.351 INFO:tasks.workunit.client.1.smithi027.stdout:7/953: read d0/d11/fe [2817887,67445] 0 2024-04-22T15:50:13.353 INFO:tasks.workunit.client.1.smithi027.stdout:7/954: write d0/d11/d1c/d31/d56/fa1 [2803940,12286] 0 2024-04-22T15:50:13.354 INFO:tasks.workunit.client.1.smithi027.stdout:7/955: chown d0/d4c/cd2 14 1 2024-04-22T15:50:13.354 INFO:tasks.workunit.client.1.smithi027.stdout:1/736: mkdir d2/d5f/d64/d81/dcf/df2 0 2024-04-22T15:50:13.355 INFO:tasks.workunit.client.1.smithi027.stdout:7/956: stat d0/d4c/de5/lfb 0 2024-04-22T15:50:13.355 INFO:tasks.workunit.client.1.smithi027.stdout:1/737: readlink d2/l19 0 2024-04-22T15:50:13.355 INFO:tasks.workunit.client.1.smithi027.stdout:4/940: mkdir d5/d11/d145 0 2024-04-22T15:50:13.357 INFO:tasks.workunit.client.1.smithi027.stdout:9/855: rename d0/dc/l69 to d0/d8b/l125 0 2024-04-22T15:50:13.361 INFO:tasks.workunit.client.1.smithi027.stdout:8/887: creat d1/d1e/d47/d53/d76/d137/f13a x:0 0 0 2024-04-22T15:50:13.365 INFO:tasks.workunit.client.1.smithi027.stdout:6/956: link d5/dc/dd/f107 d5/dc/d17/d53/d8c/f150 0 2024-04-22T15:50:13.369 INFO:tasks.workunit.client.1.smithi027.stdout:4/941: creat d5/d45/db3/f146 x:0 0 0 2024-04-22T15:50:13.371 INFO:tasks.workunit.client.1.smithi027.stdout:7/957: creat d0/d11/d1c/d2c/d93/f13f x:0 0 0 2024-04-22T15:50:13.373 INFO:tasks.workunit.client.1.smithi027.stdout:2/971: dwrite dd/d47/f48 [0,4194304] 0 2024-04-22T15:50:13.375 INFO:tasks.workunit.client.1.smithi027.stdout:2/972: write dd/df/d1b/f40 [610367,76491] 0 2024-04-22T15:50:13.379 INFO:tasks.workunit.client.1.smithi027.stdout:9/856: dread - d0/dc/d12/d30/d99/db4/ffa zero size 2024-04-22T15:50:13.381 INFO:tasks.workunit.client.1.smithi027.stdout:3/953: symlink d28/d5d/l135 0 2024-04-22T15:50:13.382 INFO:tasks.workunit.client.1.smithi027.stdout:8/888: rmdir d1/d3/d1c/d32 39 2024-04-22T15:50:13.386 INFO:tasks.workunit.client.1.smithi027.stdout:4/942: mknod d5/d11/d1d/d54/d8c/c147 0 2024-04-22T15:50:13.392 INFO:tasks.workunit.client.1.smithi027.stdout:6/957: write d5/db/f11 [424925,17801] 0 2024-04-22T15:50:13.399 INFO:tasks.workunit.client.1.smithi027.stdout:1/738: creat d2/d5f/d64/d81/d69/ff3 x:0 0 0 2024-04-22T15:50:13.401 INFO:tasks.workunit.client.1.smithi027.stdout:5/951: rename d0/df/de/d24/d62/f68 to d0/df/d3d/f139 0 2024-04-22T15:50:13.403 INFO:tasks.workunit.client.1.smithi027.stdout:9/857: dwrite d0/dc/d12/d30/d5b/d91/da9/fdc [0,4194304] 0 2024-04-22T15:50:13.403 INFO:tasks.workunit.client.1.smithi027.stdout:4/943: mkdir d5/d45/d47/d148 0 2024-04-22T15:50:13.404 INFO:tasks.workunit.client.1.smithi027.stdout:4/944: chown d5/d45/db3/l12d 7798422 1 2024-04-22T15:50:13.404 INFO:tasks.workunit.client.1.smithi027.stdout:2/973: unlink dd/df/d70/d4b/lbc 0 2024-04-22T15:50:13.410 INFO:tasks.workunit.client.1.smithi027.stdout:3/954: mknod d28/d5d/d62/c136 0 2024-04-22T15:50:13.410 INFO:tasks.workunit.client.1.smithi027.stdout:9/858: dread d0/dc/d12/d30/d5b/dbc/df5/d104/da6/fae [0,4194304] 0 2024-04-22T15:50:13.410 INFO:tasks.workunit.client.1.smithi027.stdout:6/958: fsync d5/d30/d54/fea 0 2024-04-22T15:50:13.412 INFO:tasks.workunit.client.1.smithi027.stdout:6/959: write d5/dc/d17/d8a/fdc [1850915,74026] 0 2024-04-22T15:50:13.413 INFO:tasks.workunit.client.1.smithi027.stdout:6/960: stat d5/dc/d20/da3/dc9 0 2024-04-22T15:50:13.414 INFO:tasks.workunit.client.1.smithi027.stdout:8/889: creat d1/d3/d1c/d32/d41/d52/d61/d8f/ddd/f13b x:0 0 0 2024-04-22T15:50:13.416 INFO:tasks.workunit.client.1.smithi027.stdout:1/739: stat d2/d5f/d64/l6e 0 2024-04-22T15:50:13.419 INFO:tasks.workunit.client.1.smithi027.stdout:7/958: rename d0/l10c to d0/d11/d1c/d2c/d96/d126/l140 0 2024-04-22T15:50:13.426 INFO:tasks.workunit.client.1.smithi027.stdout:3/955: rmdir d28/d4b/d52/d94/dc4/dfc 39 2024-04-22T15:50:13.429 INFO:tasks.workunit.client.1.smithi027.stdout:2/974: dwrite dd/df/d70/d4b/d4e/f107 [0,4194304] 0 2024-04-22T15:50:13.434 INFO:tasks.workunit.client.1.smithi027.stdout:6/961: unlink d5/d30/dbe/f14f 0 2024-04-22T15:50:13.447 INFO:tasks.workunit.client.1.smithi027.stdout:8/890: write d1/d3/d1c/d32/f69 [631438,35436] 0 2024-04-22T15:50:13.449 INFO:tasks.workunit.client.1.smithi027.stdout:1/740: write d2/d5f/d64/d81/d3d/d82/d85/f95 [309477,53845] 0 2024-04-22T15:50:13.451 INFO:tasks.workunit.client.1.smithi027.stdout:3/956: chown d28/d5d/d79/f87 891975354 1 2024-04-22T15:50:13.455 INFO:tasks.workunit.client.1.smithi027.stdout:4/945: truncate d5/d11/fe5 3007670 0 2024-04-22T15:50:13.457 INFO:tasks.workunit.client.1.smithi027.stdout:9/859: write d0/dc/f80 [538747,13658] 0 2024-04-22T15:50:13.459 INFO:tasks.workunit.client.1.smithi027.stdout:4/946: dread d5/d45/d47/f139 [0,4194304] 0 2024-04-22T15:50:13.459 INFO:tasks.workunit.client.1.smithi027.stdout:2/975: write dd/df/d1b/d1f/f30 [307452,93617] 0 2024-04-22T15:50:13.461 INFO:tasks.workunit.client.1.smithi027.stdout:4/947: chown d5/d11/d1d/l6f 245549 1 2024-04-22T15:50:13.463 INFO:tasks.workunit.client.1.smithi027.stdout:5/952: rename d0/d97/dbf/l12b to d0/db6/l13a 0 2024-04-22T15:50:13.464 INFO:tasks.workunit.client.1.smithi027.stdout:8/891: fsync d1/d3/d1c/d56/f7f 0 2024-04-22T15:50:13.467 INFO:tasks.workunit.client.1.smithi027.stdout:1/741: dread d2/d1d/fd9 [0,4194304] 0 2024-04-22T15:50:13.468 INFO:tasks.workunit.client.1.smithi027.stdout:8/892: chown d1/d3/d1c/d32/d41/cf5 14954872 1 2024-04-22T15:50:13.470 INFO:tasks.workunit.client.1.smithi027.stdout:1/742: stat d2/d5f/d64/d81/da7 0 2024-04-22T15:50:13.471 INFO:tasks.workunit.client.1.smithi027.stdout:8/893: dread - d1/d3/d36/d57/fbd zero size 2024-04-22T15:50:13.471 INFO:tasks.workunit.client.1.smithi027.stdout:9/860: creat d0/dc/d12/d30/d5b/d91/da9/f126 x:0 0 0 2024-04-22T15:50:13.474 INFO:tasks.workunit.client.1.smithi027.stdout:7/959: rename d0/d3f/fca to d0/d11/d1c/d31/d56/f141 0 2024-04-22T15:50:13.475 INFO:tasks.workunit.client.1.smithi027.stdout:6/962: dread d5/dc/dd/d79/d7b/d5d/fb9 [0,4194304] 0 2024-04-22T15:50:13.478 INFO:tasks.workunit.client.1.smithi027.stdout:5/953: dwrite d0/d1a/d22/d26/d32/d88/ffe [0,4194304] 0 2024-04-22T15:50:13.481 INFO:tasks.workunit.client.1.smithi027.stdout:9/861: rmdir d0/dc/d12/d30/d5b/dbc 39 2024-04-22T15:50:13.484 INFO:tasks.workunit.client.1.smithi027.stdout:2/976: dread dd/df/d70/f45 [0,4194304] 0 2024-04-22T15:50:13.484 INFO:tasks.workunit.client.1.smithi027.stdout:2/977: write dd/df/d1b/f40 [2850553,51023] 0 2024-04-22T15:50:13.486 INFO:tasks.workunit.client.1.smithi027.stdout:2/978: read dd/df/d70/d4b/d4e/daf/ffb [264955,8279] 0 2024-04-22T15:50:13.487 INFO:tasks.workunit.client.1.smithi027.stdout:4/948: rename d5/d11/d1d/f119 to d5/d11/da6/f149 0 2024-04-22T15:50:13.490 INFO:tasks.workunit.client.1.smithi027.stdout:8/894: write d1/d1e/d47/d53/fdc [1035923,10324] 0 2024-04-22T15:50:13.491 INFO:tasks.workunit.client.1.smithi027.stdout:8/895: chown d1/f18 59082 1 2024-04-22T15:50:13.493 INFO:tasks.workunit.client.1.smithi027.stdout:1/743: symlink d2/d5f/d64/d81/dcf/df2/lf4 0 2024-04-22T15:50:13.493 INFO:tasks.workunit.client.1.smithi027.stdout:2/979: dread dd/df/d70/d4b/d52/fb6 [0,4194304] 0 2024-04-22T15:50:13.494 INFO:tasks.workunit.client.1.smithi027.stdout:1/744: stat d2/d36/f4b 0 2024-04-22T15:50:13.498 INFO:tasks.workunit.client.1.smithi027.stdout:7/960: write d0/d4c/de5/f100 [5243,112247] 0 2024-04-22T15:50:13.500 INFO:tasks.workunit.client.1.smithi027.stdout:7/961: chown d0/d4c/d8b/d92/da3/da8/f81 168849 1 2024-04-22T15:50:13.509 INFO:tasks.workunit.client.1.smithi027.stdout:9/862: write d0/dc/d12/d30/d5b/dbc/df5/d104/d6b/d6e/f61 [7410022,35252] 0 2024-04-22T15:50:13.511 INFO:tasks.workunit.client.1.smithi027.stdout:4/949: dread d5/d45/d5a/f82 [0,4194304] 0 2024-04-22T15:50:13.514 INFO:tasks.workunit.client.1.smithi027.stdout:4/950: write d5/d7/f10 [1658156,913] 0 2024-04-22T15:50:13.515 INFO:tasks.workunit.client.1.smithi027.stdout:1/745: rename d2/d5f/d64/d81/da7/lda to d2/d5f/d64/lf5 0 2024-04-22T15:50:13.515 INFO:tasks.workunit.client.1.smithi027.stdout:2/980: write dd/d47/f7f [66390,37493] 0 2024-04-22T15:50:13.517 INFO:tasks.workunit.client.1.smithi027.stdout:8/896: dwrite d1/d3/d1c/d32/d41/f89 [0,4194304] 0 2024-04-22T15:50:13.520 INFO:tasks.workunit.client.1.smithi027.stdout:9/863: mknod d0/d64/c127 0 2024-04-22T15:50:13.522 INFO:tasks.workunit.client.1.smithi027.stdout:4/951: stat d5/d11/da6/fd6 0 2024-04-22T15:50:13.522 INFO:tasks.workunit.client.1.smithi027.stdout:8/897: readlink d1/d3/d1c/l3e 0 2024-04-22T15:50:13.524 INFO:tasks.workunit.client.1.smithi027.stdout:1/746: symlink d2/d36/lf6 0 2024-04-22T15:50:13.524 INFO:tasks.workunit.client.1.smithi027.stdout:7/962: dwrite d0/d11/f4d [0,4194304] 0 2024-04-22T15:50:13.526 INFO:tasks.workunit.client.1.smithi027.stdout:4/952: dread - d5/d11/d1d/d24/dfc/f120 zero size 2024-04-22T15:50:13.530 INFO:tasks.workunit.client.1.smithi027.stdout:9/864: creat d0/dc/de/df7/f128 x:0 0 0 2024-04-22T15:50:13.534 INFO:tasks.workunit.client.1.smithi027.stdout:2/981: dread dd/df/d70/d4b/d52/d9a/da4/fcc [0,4194304] 0 2024-04-22T15:50:13.537 INFO:tasks.workunit.client.1.smithi027.stdout:8/898: dwrite d1/d3/d1c/d49/d94/f105 [0,4194304] 0 2024-04-22T15:50:13.545 INFO:tasks.workunit.client.1.smithi027.stdout:7/963: dwrite d0/d11/d1c/d2c/d96/f134 [0,4194304] 0 2024-04-22T15:50:13.547 INFO:tasks.workunit.client.1.smithi027.stdout:1/747: dread d2/d5f/d64/d81/d3d/d44/f86 [0,4194304] 0 2024-04-22T15:50:13.549 INFO:tasks.workunit.client.1.smithi027.stdout:4/953: truncate d5/d11/f80 169976 0 2024-04-22T15:50:13.550 INFO:tasks.workunit.client.1.smithi027.stdout:4/954: chown d5/de/d2a/fc3 894 1 2024-04-22T15:50:13.551 INFO:tasks.workunit.client.1.smithi027.stdout:8/899: rename d1/d3/d36/d57/c108 to d1/d3/d1c/d133/c13c 0 2024-04-22T15:50:13.558 INFO:tasks.workunit.client.1.smithi027.stdout:1/748: dread - d2/d36/d74/fbc zero size 2024-04-22T15:50:13.562 INFO:tasks.workunit.client.1.smithi027.stdout:8/900: unlink d1/d3/d1c/d43/l6f 0 2024-04-22T15:50:13.564 INFO:tasks.workunit.client.1.smithi027.stdout:8/901: truncate d1/d3/d1c/d56/f101 753115 0 2024-04-22T15:50:13.565 INFO:tasks.workunit.client.1.smithi027.stdout:9/865: getdents d0/dc/d12/d30/d5b/dbc/df5/d104/ddb/dfd 0 2024-04-22T15:50:13.568 INFO:tasks.workunit.client.1.smithi027.stdout:2/982: getdents dd/df/d1b 0 2024-04-22T15:50:13.568 INFO:tasks.workunit.client.1.smithi027.stdout:7/964: write d0/d49/d9f/f9b [630005,118891] 0 2024-04-22T15:50:13.569 INFO:tasks.workunit.client.1.smithi027.stdout:1/749: mknod d2/d5f/d64/d81/d7e/cf7 0 2024-04-22T15:50:13.570 INFO:tasks.workunit.client.1.smithi027.stdout:2/983: chown dd/l3a 128 1 2024-04-22T15:50:13.574 INFO:tasks.workunit.client.1.smithi027.stdout:9/866: symlink d0/dc/de/df7/l129 0 2024-04-22T15:50:13.576 INFO:tasks.workunit.client.1.smithi027.stdout:7/965: symlink d0/d11/d3a/d46/l142 0 2024-04-22T15:50:13.577 INFO:tasks.workunit.client.1.smithi027.stdout:4/955: getdents d5/d42/d10f 0 2024-04-22T15:50:13.583 INFO:tasks.workunit.client.1.smithi027.stdout:1/750: truncate d2/f57 2810643 0 2024-04-22T15:50:13.585 INFO:tasks.workunit.client.1.smithi027.stdout:4/956: dwrite d5/de/d30/f12e [4194304,4194304] 0 2024-04-22T15:50:13.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:13 smithi027 ceph-mon[34922]: pgmap v10: 65 pgs: 65 active+clean; 2.9 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 46 MiB/s rd, 98 MiB/s wr, 276 op/s 2024-04-22T15:50:13.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:13 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:13.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:13 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:13.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:13 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:50:13.587 INFO:tasks.workunit.client.1.smithi027.stdout:2/984: fsync dd/df/d70/d4b/d52/f8f 0 2024-04-22T15:50:13.589 INFO:tasks.workunit.client.1.smithi027.stdout:2/985: truncate dd/df/d1b/d38/dd5/f10f 3326969 0 2024-04-22T15:50:13.595 INFO:tasks.workunit.client.1.smithi027.stdout:8/902: dwrite d1/d3/d1c/d56/dc9/f119 [0,4194304] 0 2024-04-22T15:50:13.598 INFO:tasks.workunit.client.1.smithi027.stdout:7/966: creat d0/d11/d1c/d2c/d96/d126/d12c/f143 x:0 0 0 2024-04-22T15:50:13.601 INFO:tasks.workunit.client.1.smithi027.stdout:9/867: link d0/la2 d0/dc/de/df7/l12a 0 2024-04-22T15:50:13.610 INFO:tasks.workunit.client.1.smithi027.stdout:8/903: dread d1/d3/f9b [0,4194304] 0 2024-04-22T15:50:13.610 INFO:tasks.workunit.client.1.smithi027.stdout:4/957: creat d5/d11/d145/f14a x:0 0 0 2024-04-22T15:50:13.610 INFO:tasks.workunit.client.1.smithi027.stdout:2/986: getdents dd/df/d1b/d38/d112 0 2024-04-22T15:50:13.615 INFO:tasks.workunit.client.1.smithi027.stdout:8/904: rmdir d1/d3/d1c/d32/d41/d52/d61/d8f/dd3 39 2024-04-22T15:50:13.617 INFO:tasks.workunit.client.1.smithi027.stdout:4/958: rename d5/d11/d1d/d54/d8c/fde to d5/da8/f14b 0 2024-04-22T15:50:13.618 INFO:tasks.workunit.client.1.smithi027.stdout:2/987: symlink dd/df/d1b/d38/d60/d75/ddf/l12c 0 2024-04-22T15:50:13.620 INFO:tasks.workunit.client.1.smithi027.stdout:1/751: getdents d2/d5f/d64/d81/dcf 0 2024-04-22T15:50:13.620 INFO:tasks.workunit.client.1.smithi027.stdout:1/752: fsync d2/d1d/f5b 0 2024-04-22T15:50:13.622 INFO:tasks.workunit.client.1.smithi027.stdout:4/959: creat d5/d45/db3/f14c x:0 0 0 2024-04-22T15:50:13.625 INFO:tasks.workunit.client.1.smithi027.stdout:2/988: rename dd/f6f to dd/df/d1b/d38/dd5/f12d 0 2024-04-22T15:50:13.625 INFO:tasks.workunit.client.1.smithi027.stdout:7/967: getdents d0/d4c/d4e 0 2024-04-22T15:50:13.628 INFO:tasks.workunit.client.1.smithi027.stdout:7/968: write d0/d11/d1c/f103 [467745,111520] 0 2024-04-22T15:50:13.628 INFO:tasks.workunit.client.1.smithi027.stdout:7/969: chown d0/d49 287065905 1 2024-04-22T15:50:13.629 INFO:tasks.workunit.client.1.smithi027.stdout:1/753: rmdir d2/d5f/d64/d81/da7 39 2024-04-22T15:50:13.633 INFO:tasks.workunit.client.1.smithi027.stdout:9/868: getdents d0/dc/d12 0 2024-04-22T15:50:13.634 INFO:tasks.workunit.client.1.smithi027.stdout:4/960: mknod d5/de/d30/de3/c14d 0 2024-04-22T15:50:13.635 INFO:tasks.workunit.client.1.smithi027.stdout:9/869: read d0/f47 [3599274,13194] 0 2024-04-22T15:50:13.635 INFO:tasks.workunit.client.1.smithi027.stdout:4/961: write d5/d45/d5a/f96 [844949,20216] 0 2024-04-22T15:50:13.638 INFO:tasks.workunit.client.1.smithi027.stdout:8/905: creat d1/d3/d1c/d32/d41/d52/d61/d8f/f13d x:0 0 0 2024-04-22T15:50:13.639 INFO:tasks.workunit.client.1.smithi027.stdout:8/906: chown d1/d1e/d47/d9d/faa 42 1 2024-04-22T15:50:13.640 INFO:tasks.workunit.client.1.smithi027.stdout:9/870: fsync d0/dc/de/fc5 0 2024-04-22T15:50:13.643 INFO:tasks.workunit.client.1.smithi027.stdout:7/970: symlink d0/d11/d1c/d2c/d93/d11c/l144 0 2024-04-22T15:50:13.645 INFO:tasks.workunit.client.1.smithi027.stdout:7/971: stat d0/d11/d1c/d2c/d93/f59 0 2024-04-22T15:50:13.645 INFO:tasks.workunit.client.1.smithi027.stdout:8/907: rmdir d1/d3/d1c/d28/d42/dc4/df3 39 2024-04-22T15:50:13.647 INFO:tasks.workunit.client.1.smithi027.stdout:9/871: dwrite d0/dc/fa3 [0,4194304] 0 2024-04-22T15:50:13.649 INFO:tasks.workunit.client.1.smithi027.stdout:8/908: read d1/d3/d1c/d49/f63 [8818,19188] 0 2024-04-22T15:50:13.653 INFO:tasks.workunit.client.1.smithi027.stdout:1/754: rename d2/d5f/d64/d81/d69 to d2/d5f/d64/d81/d3d/d44/d9c/df8 0 2024-04-22T15:50:13.655 INFO:tasks.workunit.client.1.smithi027.stdout:7/972: readlink d0/d11/d1c/l34 0 2024-04-22T15:50:13.656 INFO:tasks.workunit.client.1.smithi027.stdout:7/973: dread - d0/d11/d1c/d31/d56/fc6 zero size 2024-04-22T15:50:13.657 INFO:tasks.workunit.client.1.smithi027.stdout:9/872: symlink d0/d8b/l12b 0 2024-04-22T15:50:13.659 INFO:tasks.workunit.client.1.smithi027.stdout:4/962: rmdir d5/de/d143 0 2024-04-22T15:50:13.663 INFO:tasks.workunit.client.1.smithi027.stdout:1/755: symlink d2/d5f/d64/d81/d3d/d44/d9c/d9f/de9/lf9 0 2024-04-22T15:50:13.664 INFO:tasks.workunit.client.1.smithi027.stdout:9/873: dwrite d0/dc/f80 [0,4194304] 0 2024-04-22T15:50:13.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:13 smithi016 ceph-mon[26620]: pgmap v10: 65 pgs: 65 active+clean; 2.9 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 46 MiB/s rd, 98 MiB/s wr, 276 op/s 2024-04-22T15:50:13.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:13 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:13.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:13 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:13.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:13 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:50:13.667 INFO:tasks.workunit.client.1.smithi027.stdout:9/874: chown d0/d51/l6d 27 1 2024-04-22T15:50:13.669 INFO:tasks.workunit.client.1.smithi027.stdout:7/974: symlink d0/d11/d1c/d2c/d96/d126/l145 0 2024-04-22T15:50:13.675 INFO:tasks.workunit.client.1.smithi027.stdout:9/875: dwrite d0/dc/d12/d30/d5b/d91/da9/f115 [0,4194304] 0 2024-04-22T15:50:13.677 INFO:tasks.workunit.client.1.smithi027.stdout:8/909: mkdir d1/d3/d1c/d43/d5f/d13e 0 2024-04-22T15:50:13.680 INFO:tasks.workunit.client.1.smithi027.stdout:1/756: creat d2/db9/ffa x:0 0 0 2024-04-22T15:50:13.688 INFO:tasks.workunit.client.1.smithi027.stdout:7/975: rename d0/d11/d1c/d2c/d96/f10a to d0/d4c/f146 0 2024-04-22T15:50:13.692 INFO:tasks.workunit.client.1.smithi027.stdout:1/757: dwrite d2/d5f/d64/d81/d3d/d82/dc2/fed [0,4194304] 0 2024-04-22T15:50:13.694 INFO:tasks.workunit.client.1.smithi027.stdout:1/758: dread - d2/d5f/ff1 zero size 2024-04-22T15:50:13.694 INFO:tasks.workunit.client.1.smithi027.stdout:7/976: dread d0/d4c/fd1 [4194304,4194304] 0 2024-04-22T15:50:13.694 INFO:tasks.workunit.client.1.smithi027.stdout:8/910: mkdir d1/d3/d1c/d43/d82/d13f 0 2024-04-22T15:50:13.698 INFO:tasks.workunit.client.1.smithi027.stdout:4/963: dwrite d5/de/d30/d5d/f6a [0,4194304] 0 2024-04-22T15:50:13.702 INFO:tasks.workunit.client.1.smithi027.stdout:9/876: dread d0/dc/d12/d30/d5b/d91/da9/f120 [0,4194304] 0 2024-04-22T15:50:13.707 INFO:tasks.workunit.client.1.smithi027.stdout:1/759: creat d2/d1d/ffb x:0 0 0 2024-04-22T15:50:13.713 INFO:tasks.workunit.client.1.smithi027.stdout:9/877: mkdir d0/dc/d12/d30/d5b/dbc/df5/d104/d6b/d12c 0 2024-04-22T15:50:13.714 INFO:tasks.workunit.client.1.smithi027.stdout:1/760: fsync d2/d5f/d64/d81/d3d/d82/f93 0 2024-04-22T15:50:13.715 INFO:tasks.workunit.client.1.smithi027.stdout:9/878: chown d0/dc/de/df7/l101 27 1 2024-04-22T15:50:13.716 INFO:tasks.workunit.client.1.smithi027.stdout:8/911: dread d1/d1e/d47/d53/f77 [0,4194304] 0 2024-04-22T15:50:13.716 INFO:tasks.workunit.client.1.smithi027.stdout:7/977: write d0/d3f/f5b [389556,27459] 0 2024-04-22T15:50:13.723 INFO:tasks.workunit.client.1.smithi027.stdout:4/964: dread d5/d11/d1d/ffa [0,4194304] 0 2024-04-22T15:50:13.725 INFO:tasks.workunit.client.1.smithi027.stdout:8/912: dwrite d1/d3/d1c/d32/f103 [0,4194304] 0 2024-04-22T15:50:13.726 INFO:tasks.workunit.client.1.smithi027.stdout:7/978: truncate d0/d49/f71 3126316 0 2024-04-22T15:50:13.733 INFO:tasks.workunit.client.1.smithi027.stdout:8/913: creat d1/d3/d1c/d49/f140 x:0 0 0 2024-04-22T15:50:13.735 INFO:tasks.workunit.client.1.smithi027.stdout:1/761: dwrite d2/d5f/d64/d81/d7e/fa1 [0,4194304] 0 2024-04-22T15:50:13.738 INFO:tasks.workunit.client.1.smithi027.stdout:4/965: rmdir d5/de/d114 0 2024-04-22T15:50:13.740 INFO:tasks.workunit.client.1.smithi027.stdout:8/914: fsync d1/d3/d1c/d56/f7f 0 2024-04-22T15:50:13.743 INFO:tasks.workunit.client.1.smithi027.stdout:4/966: read d5/de/d2a/d58/f67 [934827,20387] 0 2024-04-22T15:50:13.744 INFO:tasks.workunit.client.1.smithi027.stdout:8/915: dread - d1/d3/d1c/d32/d41/d52/d61/f12c zero size 2024-04-22T15:50:13.744 INFO:tasks.workunit.client.1.smithi027.stdout:9/879: dwrite d0/dc/d12/d30/d5b/dbc/df5/d104/da6/feb [0,4194304] 0 2024-04-22T15:50:13.745 INFO:tasks.workunit.client.1.smithi027.stdout:1/762: truncate d2/f18 1708227 0 2024-04-22T15:50:13.750 INFO:tasks.workunit.client.1.smithi027.stdout:4/967: write d5/d11/d6c/da3/d10b/f110 [355781,73969] 0 2024-04-22T15:50:13.772 INFO:tasks.workunit.client.1.smithi027.stdout:4/968: dread d5/d42/f61 [0,4194304] 0 2024-04-22T15:50:13.798 INFO:tasks.workunit.client.1.smithi027.stdout:1/763: mkdir d2/d1d/d56/dc0/dfc 0 2024-04-22T15:50:13.799 INFO:tasks.workunit.client.1.smithi027.stdout:4/969: creat d5/da8/f14e x:0 0 0 2024-04-22T15:50:13.800 INFO:tasks.workunit.client.1.smithi027.stdout:8/916: link d1/l126 d1/d1e/d47/d9d/l141 0 2024-04-22T15:50:13.802 INFO:tasks.workunit.client.1.smithi027.stdout:4/970: unlink d5/d45/d5a/f82 0 2024-04-22T15:50:13.802 INFO:tasks.workunit.client.1.smithi027.stdout:8/917: mknod d1/d3/c142 0 2024-04-22T15:50:13.804 INFO:tasks.workunit.client.1.smithi027.stdout:8/918: write d1/d3/d1c/d28/d42/ffe [598836,25124] 0 2024-04-22T15:50:13.806 INFO:tasks.workunit.client.1.smithi027.stdout:4/971: unlink d5/de/f21 0 2024-04-22T15:50:13.808 INFO:tasks.workunit.client.1.smithi027.stdout:8/919: creat d1/d3/d1c/d49/d94/f143 x:0 0 0 2024-04-22T15:50:13.814 INFO:tasks.workunit.client.1.smithi027.stdout:8/920: dread d1/d3/d1c/d32/f103 [0,4194304] 0 2024-04-22T15:50:13.820 INFO:tasks.workunit.client.1.smithi027.stdout:8/921: creat d1/d3/d1c/d43/d5f/f144 x:0 0 0 2024-04-22T15:50:13.831 INFO:tasks.workunit.client.1.smithi027.stdout:3/957: sync 2024-04-22T15:50:13.833 INFO:tasks.workunit.client.1.smithi027.stdout:3/958: readlink d28/d4b/d52/db2/d101/l11c 0 2024-04-22T15:50:13.835 INFO:tasks.workunit.client.1.smithi027.stdout:3/959: creat d28/d4b/dc6/d130/f137 x:0 0 0 2024-04-22T15:50:13.846 INFO:tasks.workunit.client.1.smithi027.stdout:3/960: getdents d28/d69 0 2024-04-22T15:50:13.850 INFO:tasks.workunit.client.1.smithi027.stdout:3/961: symlink d28/d5d/db0/dce/dd4/l138 0 2024-04-22T15:50:13.850 INFO:tasks.workunit.client.1.smithi027.stdout:3/962: dread - d28/d69/f134 zero size 2024-04-22T15:50:13.854 INFO:tasks.workunit.client.1.smithi027.stdout:3/963: read d28/d5d/db0/f12a [227004,104053] 0 2024-04-22T15:50:13.864 INFO:tasks.workunit.client.1.smithi027.stdout:3/964: creat d28/d4b/dc6/d10c/f139 x:0 0 0 2024-04-22T15:50:13.866 INFO:tasks.workunit.client.1.smithi027.stdout:3/965: read d28/d4b/f7c [938370,11802] 0 2024-04-22T15:50:13.869 INFO:tasks.workunit.client.1.smithi027.stdout:3/966: fdatasync d28/d5d/d62/dcf/d12d/d9c/dc1/f126 0 2024-04-22T15:50:13.877 INFO:tasks.workunit.client.1.smithi027.stdout:3/967: dwrite d28/d4b/dc6/d130/f137 [0,4194304] 0 2024-04-22T15:50:13.885 INFO:tasks.workunit.client.1.smithi027.stdout:4/972: dwrite d5/d11/f8d [0,4194304] 0 2024-04-22T15:50:13.887 INFO:tasks.workunit.client.1.smithi027.stdout:3/968: unlink d28/d4b/d52/db2/cca 0 2024-04-22T15:50:13.889 INFO:tasks.workunit.client.1.smithi027.stdout:4/973: mknod d5/de/da9/c14f 0 2024-04-22T15:50:13.890 INFO:tasks.workunit.client.1.smithi027.stdout:4/974: chown d5/d45/c101 0 1 2024-04-22T15:50:13.891 INFO:tasks.workunit.client.1.smithi027.stdout:4/975: fsync d5/d11/d1d/d54/d8c/f11c 0 2024-04-22T15:50:13.896 INFO:tasks.workunit.client.1.smithi027.stdout:4/976: mkdir d5/de/d30/d5d/d150 0 2024-04-22T15:50:13.900 INFO:tasks.workunit.client.1.smithi027.stdout:3/969: getdents d28/d5d/d62/dcf/d12d/d9c/dc1 0 2024-04-22T15:50:13.904 INFO:tasks.workunit.client.1.smithi027.stdout:4/977: dwrite d5/d11/d1d/d24/f121 [0,4194304] 0 2024-04-22T15:50:13.906 INFO:tasks.workunit.client.1.smithi027.stdout:4/978: dread - d5/da8/f14e zero size 2024-04-22T15:50:13.907 INFO:tasks.workunit.client.1.smithi027.stdout:3/970: mkdir d28/d5d/d62/dcf/d12d/d9c/de9/d119/d13a 0 2024-04-22T15:50:13.914 INFO:tasks.workunit.client.1.smithi027.stdout:4/979: dread d5/d45/d5a/dcd/f12f [0,4194304] 0 2024-04-22T15:50:13.929 INFO:tasks.workunit.client.1.smithi027.stdout:4/980: dread d5/de/f37 [0,4194304] 0 2024-04-22T15:50:13.930 INFO:tasks.workunit.client.1.smithi027.stdout:4/981: rmdir d5/de/da9/d11e 39 2024-04-22T15:50:13.939 INFO:tasks.workunit.client.1.smithi027.stdout:3/971: dwrite d28/d5d/d62/faa [0,4194304] 0 2024-04-22T15:50:13.942 INFO:tasks.workunit.client.1.smithi027.stdout:3/972: unlink l25 0 2024-04-22T15:50:13.962 INFO:tasks.workunit.client.1.smithi027.stdout:3/973: dread d28/d5d/d62/dcf/dec/f129 [0,4194304] 0 2024-04-22T15:50:13.998 INFO:tasks.workunit.client.1.smithi027.stdout:3/974: dread d28/d5d/f96 [0,4194304] 0 2024-04-22T15:50:13.998 INFO:tasks.workunit.client.1.smithi027.stdout:3/975: chown d28/d4b/d52/d9a/l106 87077707 1 2024-04-22T15:50:14.002 INFO:tasks.workunit.client.1.smithi027.stdout:6/963: sync 2024-04-22T15:50:14.171 INFO:tasks.workunit.client.1.smithi027.stdout:2/989: sync 2024-04-22T15:50:14.172 INFO:tasks.workunit.client.1.smithi027.stdout:5/954: sync 2024-04-22T15:50:14.175 INFO:tasks.workunit.client.1.smithi027.stdout:5/955: creat d0/d1a/d22/d26/d5d/de3/d8f/f13b x:0 0 0 2024-04-22T15:50:14.177 INFO:tasks.workunit.client.1.smithi027.stdout:2/990: rename dd/df/d70/d4b/d52/d9a/da4/dad/fdd to dd/df/d70/d4b/d52/d9a/f12e 0 2024-04-22T15:50:14.284 INFO:tasks.workunit.client.1.smithi027.stdout:5/956: mkdir d0/df/de/dbb/d13c 0 2024-04-22T15:50:14.285 INFO:tasks.workunit.client.1.smithi027.stdout:2/991: stat dd/df/d70/d4b/lff 0 2024-04-22T15:50:14.286 INFO:tasks.workunit.client.1.smithi027.stdout:2/992: readlink dd/df/d70/d4b/d52/d9a/da4/dad/l120 0 2024-04-22T15:50:14.350 INFO:tasks.workunit.client.1.smithi027.stdout:7/979: sync 2024-04-22T15:50:14.351 INFO:tasks.workunit.client.1.smithi027.stdout:5/957: fsync d0/df/de/d24/d62/f127 0 2024-04-22T15:50:14.354 INFO:tasks.workunit.client.1.smithi027.stdout:7/980: truncate d0/d4c/d8b/d92/fe6 357182 0 2024-04-22T15:50:14.355 INFO:tasks.workunit.client.1.smithi027.stdout:7/981: chown d0/d11/d1c/d31/d8a/fa4 0 1 2024-04-22T15:50:14.358 INFO:tasks.workunit.client.1.smithi027.stdout:2/993: dread dd/df/d1b/f2e [0,4194304] 0 2024-04-22T15:50:14.362 INFO:tasks.workunit.client.1.smithi027.stdout:7/982: rename d0/d11/d1c/d2c/d93/l127 to d0/d4c/d8b/d72/l147 0 2024-04-22T15:50:14.366 INFO:tasks.workunit.client.1.smithi027.stdout:7/983: creat d0/d4c/d4e/f148 x:0 0 0 2024-04-22T15:50:14.367 INFO:tasks.workunit.client.1.smithi027.stdout:5/958: write d0/d1a/d22/d26/d32/f46 [5223346,30119] 0 2024-04-22T15:50:14.369 INFO:tasks.workunit.client.1.smithi027.stdout:7/984: readlink d0/d11/d3a/d46/d94/led 0 2024-04-22T15:50:14.369 INFO:tasks.workunit.client.1.smithi027.stdout:2/994: fsync dd/df/d70/d4b/f58 0 2024-04-22T15:50:14.378 INFO:tasks.workunit.client.1.smithi027.stdout:2/995: readlink dd/dae/dd4/ddb/l102 0 2024-04-22T15:50:14.380 INFO:tasks.workunit.client.1.smithi027.stdout:7/985: unlink d0/d11/d1c/d31/d8a/ff8 0 2024-04-22T15:50:14.381 INFO:tasks.workunit.client.1.smithi027.stdout:7/986: dread - d0/d11/d1c/d2c/f135 zero size 2024-04-22T15:50:14.383 INFO:tasks.workunit.client.1.smithi027.stdout:5/959: mknod d0/df/de/d24/d10c/c13d 0 2024-04-22T15:50:14.391 INFO:tasks.workunit.client.1.smithi027.stdout:5/960: rename d0/d1a/d22/d72/de8/f115 to d0/d1a/d22/d26/d5d/de3/d8f/f13e 0 2024-04-22T15:50:14.393 INFO:tasks.workunit.client.1.smithi027.stdout:7/987: dread d0/d11/d1c/d31/d8a/fa4 [0,4194304] 0 2024-04-22T15:50:14.396 INFO:tasks.workunit.client.1.smithi027.stdout:2/996: write dd/df/f88 [1486449,88471] 0 2024-04-22T15:50:14.402 INFO:tasks.workunit.client.1.smithi027.stdout:7/988: dwrite d0/d4c/fc9 [4194304,4194304] 0 2024-04-22T15:50:14.403 INFO:tasks.workunit.client.1.smithi027.stdout:7/989: readlink d0/d4c/d8b/d13a/l13b 0 2024-04-22T15:50:14.403 INFO:tasks.workunit.client.1.smithi027.stdout:2/997: rename dd/df/d70/l92 to dd/df/d70/d4b/dec/l12f 0 2024-04-22T15:50:14.407 INFO:tasks.workunit.client.1.smithi027.stdout:5/961: write d0/d1a/d22/d26/d32/d88/d90/f91 [628695,2721] 0 2024-04-22T15:50:14.421 INFO:tasks.workunit.client.1.smithi027.stdout:2/998: dwrite dd/f1c [0,4194304] 0 2024-04-22T15:50:14.424 INFO:tasks.workunit.client.1.smithi027.stdout:7/990: dwrite d0/d4c/d8b/d92/da3/da8/fd8 [0,4194304] 0 2024-04-22T15:50:14.434 INFO:tasks.workunit.client.1.smithi027.stdout:5/962: dread d0/d1a/d22/d26/d32/f46 [0,4194304] 0 2024-04-22T15:50:14.436 INFO:tasks.workunit.client.1.smithi027.stdout:2/999: dwrite dd/d47/f7f [0,4194304] 0 2024-04-22T15:50:14.446 INFO:tasks.workunit.client.1.smithi027.stdout:5/963: dread d0/df/f12 [0,4194304] 0 2024-04-22T15:50:14.448 INFO:tasks.workunit.client.1.smithi027.stdout:5/964: write d0/df/de/f10f [1545432,100356] 0 2024-04-22T15:50:14.451 INFO:tasks.workunit.client.1.smithi027.stdout:5/965: truncate d0/df/d3d/f51 267519 0 2024-04-22T15:50:14.453 INFO:tasks.workunit.client.1.smithi027.stdout:5/966: symlink d0/d1a/d22/d26/d32/d88/d90/d107/l13f 0 2024-04-22T15:50:14.455 INFO:tasks.workunit.client.1.smithi027.stdout:5/967: mkdir d0/db6/d140 0 2024-04-22T15:50:14.457 INFO:tasks.workunit.client.1.smithi027.stdout:5/968: rmdir d0/d1a/d22/d26/d5d/de3/d3e/da2 39 2024-04-22T15:50:14.458 INFO:tasks.workunit.client.1.smithi027.stdout:5/969: read d0/df/d3d/d81/fbc [3234945,2851] 0 2024-04-22T15:50:14.461 INFO:tasks.workunit.client.1.smithi027.stdout:5/970: creat d0/d1a/d22/d26/d32/f141 x:0 0 0 2024-04-22T15:50:14.468 INFO:tasks.workunit.client.1.smithi027.stdout:7/991: dread d0/f4b [0,4194304] 0 2024-04-22T15:50:14.468 INFO:tasks.workunit.client.1.smithi027.stdout:9/880: sync 2024-04-22T15:50:14.473 INFO:tasks.workunit.client.1.smithi027.stdout:9/881: creat d0/d51/de6/f12d x:0 0 0 2024-04-22T15:50:14.477 INFO:tasks.workunit.client.1.smithi027.stdout:9/882: rmdir d0/dc/de/d107 39 2024-04-22T15:50:14.481 INFO:tasks.workunit.client.1.smithi027.stdout:9/883: fsync d0/dc/f72 0 2024-04-22T15:50:14.482 INFO:tasks.workunit.client.1.smithi027.stdout:5/971: dwrite d0/d1a/f82 [0,4194304] 0 2024-04-22T15:50:14.486 INFO:tasks.workunit.client.1.smithi027.stdout:5/972: mknod d0/df/de/d40/d89/c142 0 2024-04-22T15:50:14.488 INFO:tasks.workunit.client.1.smithi027.stdout:7/992: dwrite d0/d11/d1c/d2c/d93/f59 [0,4194304] 0 2024-04-22T15:50:14.489 INFO:tasks.workunit.client.1.smithi027.stdout:5/973: chown d0/d1a/d22/d26/d5d/dae 52825 1 2024-04-22T15:50:14.493 INFO:tasks.workunit.client.1.smithi027.stdout:9/884: creat d0/dc/d12/d30/d5b/dbc/df5/d104/d6b/d12c/f12e x:0 0 0 2024-04-22T15:50:14.493 INFO:tasks.workunit.client.1.smithi027.stdout:7/993: symlink d0/d11/d1c/d31/d8a/l149 0 2024-04-22T15:50:14.495 INFO:tasks.workunit.client.1.smithi027.stdout:5/974: mknod d0/d1a/d22/d72/de8/c143 0 2024-04-22T15:50:14.499 INFO:tasks.workunit.client.1.smithi027.stdout:9/885: mkdir d0/d64/d12f 0 2024-04-22T15:50:14.502 INFO:tasks.workunit.client.1.smithi027.stdout:9/886: stat d0/d64/f7c 0 2024-04-22T15:50:14.503 INFO:tasks.workunit.client.1.smithi027.stdout:5/975: creat d0/db6/d140/f144 x:0 0 0 2024-04-22T15:50:14.504 INFO:tasks.workunit.client.1.smithi027.stdout:9/887: chown d0/dc/d12/d30/d5b/d91/da9/ff6 0 1 2024-04-22T15:50:14.504 INFO:tasks.workunit.client.1.smithi027.stdout:5/976: fdatasync d0/d1a/d22/d72/de8/f110 0 2024-04-22T15:50:14.506 INFO:tasks.workunit.client.1.smithi027.stdout:9/888: stat d0/dc/d12/d30/d5b/dbc/df5/d104/lcd 0 2024-04-22T15:50:14.513 INFO:tasks.workunit.client.1.smithi027.stdout:5/977: rename d0/db6/l13a to d0/d97/dbf/l145 0 2024-04-22T15:50:14.513 INFO:tasks.workunit.client.1.smithi027.stdout:7/994: dwrite d0/d4c/f11e [0,4194304] 0 2024-04-22T15:50:14.514 INFO:tasks.workunit.client.1.smithi027.stdout:7/995: stat d0/f14 0 2024-04-22T15:50:14.520 INFO:tasks.workunit.client.1.smithi027.stdout:9/889: getdents d0/dc/d12/d30/d99/da7 0 2024-04-22T15:50:14.530 INFO:tasks.workunit.client.1.smithi027.stdout:5/978: write d0/d97/dbf/fe4 [910046,73295] 0 2024-04-22T15:50:14.532 INFO:tasks.workunit.client.1.smithi027.stdout:5/979: fdatasync d0/d1a/d22/d26/d5d/de3/d8f/f13b 0 2024-04-22T15:50:14.533 INFO:tasks.workunit.client.1.smithi027.stdout:7/996: truncate d0/d11/d1c/ddc/f136 1782761 0 2024-04-22T15:50:14.533 INFO:tasks.workunit.client.1.smithi027.stdout:1/764: sync 2024-04-22T15:50:14.533 INFO:tasks.workunit.client.1.smithi027.stdout:8/922: sync 2024-04-22T15:50:14.533 INFO:tasks.workunit.client.1.smithi027.stdout:4/982: sync 2024-04-22T15:50:14.533 INFO:tasks.workunit.client.1.smithi027.stdout:6/964: sync 2024-04-22T15:50:14.534 INFO:tasks.workunit.client.1.smithi027.stdout:3/976: sync 2024-04-22T15:50:14.534 INFO:tasks.workunit.client.1.smithi027.stdout:9/890: creat d0/dc/d12/d30/d5b/dbc/df5/d104/ddb/dfd/d119/f130 x:0 0 0 2024-04-22T15:50:14.538 INFO:tasks.workunit.client.1.smithi027.stdout:9/891: dread - d0/dc/d12/d30/d5b/d91/daf/f10d zero size 2024-04-22T15:50:14.538 INFO:tasks.workunit.client.1.smithi027.stdout:5/980: dread d0/df/de/d40/fb2 [0,4194304] 0 2024-04-22T15:50:14.542 INFO:tasks.workunit.client.1.smithi027.stdout:6/965: dread d5/dc/dd/d79/d7b/d5d/f70 [4194304,4194304] 0 2024-04-22T15:50:14.544 INFO:tasks.workunit.client.1.smithi027.stdout:4/983: creat d5/d11/d1d/d24/dfc/f151 x:0 0 0 2024-04-22T15:50:14.546 INFO:tasks.workunit.client.1.smithi027.stdout:3/977: symlink d28/d5d/d62/dcf/d12d/d9c/de9/l13b 0 2024-04-22T15:50:14.548 INFO:tasks.workunit.client.1.smithi027.stdout:1/765: write d2/f75 [227747,45038] 0 2024-04-22T15:50:14.549 INFO:tasks.workunit.client.1.smithi027.stdout:1/766: stat d2/d5f/d64/d81/d3d/da9/fe3 0 2024-04-22T15:50:14.550 INFO:tasks.workunit.client.1.smithi027.stdout:7/997: unlink d0/d4c/d4e/c101 0 2024-04-22T15:50:14.551 INFO:tasks.workunit.client.1.smithi027.stdout:8/923: write d1/d3/d1c/d43/d82/fee [487210,113956] 0 2024-04-22T15:50:14.555 INFO:tasks.workunit.client.1.smithi027.stdout:5/981: symlink d0/d1a/d22/d26/d32/d88/d90/d107/l146 0 2024-04-22T15:50:14.556 INFO:tasks.workunit.client.1.smithi027.stdout:9/892: dread d0/dc/d12/d30/d99/db4/fcc [0,4194304] 0 2024-04-22T15:50:14.557 INFO:tasks.workunit.client.1.smithi027.stdout:9/893: chown d0/f90 985721 1 2024-04-22T15:50:14.557 INFO:tasks.workunit.client.1.smithi027.stdout:6/966: symlink d5/db/d2e/d59/df0/l151 0 2024-04-22T15:50:14.558 INFO:tasks.workunit.client.1.smithi027.stdout:4/984: fsync d5/d42/f61 0 2024-04-22T15:50:14.559 INFO:tasks.workunit.client.1.smithi027.stdout:3/978: truncate d28/d69/f85 4190732 0 2024-04-22T15:50:14.562 INFO:tasks.workunit.client.1.smithi027.stdout:7/998: rename d0/d11/d1c/d31/d8a to d0/d4c/de5/d14a 0 2024-04-22T15:50:14.571 INFO:tasks.workunit.client.1.smithi027.stdout:6/967: creat d5/dc/dd/d79/f152 x:0 0 0 2024-04-22T15:50:14.573 INFO:tasks.workunit.client.1.smithi027.stdout:9/894: symlink d0/dc/d12/d30/d5b/dbc/df5/l131 0 2024-04-22T15:50:14.575 INFO:tasks.workunit.client.1.smithi027.stdout:9/895: dread - d0/dc/d12/d30/d99/f116 zero size 2024-04-22T15:50:14.577 INFO:tasks.workunit.client.1.smithi027.stdout:7/999: dread d0/d11/d1c/d31/faa [0,4194304] 0 2024-04-22T15:50:14.579 INFO:tasks.workunit.client.1.smithi027.stdout:4/985: symlink d5/d11/d6c/da3/d10b/l152 0 2024-04-22T15:50:14.585 INFO:tasks.workunit.client.1.smithi027.stdout:5/982: symlink d0/d1a/l147 0 2024-04-22T15:50:14.589 INFO:tasks.workunit.client.1.smithi027.stdout:6/968: mkdir d5/dc/dd/d79/d153 0 2024-04-22T15:50:14.591 INFO:tasks.workunit.client.1.smithi027.stdout:9/896: mknod d0/dc/d12/d30/d5b/d91/daf/c132 0 2024-04-22T15:50:14.593 INFO:tasks.workunit.client.1.smithi027.stdout:4/986: fsync d5/de/d2a/d58/f67 0 2024-04-22T15:50:14.600 INFO:tasks.workunit.client.1.smithi027.stdout:8/924: link d1/d3/c23 d1/d3/d1c/d28/d42/c145 0 2024-04-22T15:50:14.601 INFO:tasks.workunit.client.1.smithi027.stdout:5/983: symlink d0/d1a/d22/d26/d5d/l148 0 2024-04-22T15:50:14.604 INFO:tasks.workunit.client.1.smithi027.stdout:4/987: mkdir d5/d11/d1d/d7b/de9/d153 0 2024-04-22T15:50:14.604 INFO:tasks.workunit.client.1.smithi027.stdout:3/979: link d28/d4b/d52/d9a/la2 d28/d5d/d79/l13c 0 2024-04-22T15:50:14.606 INFO:tasks.workunit.client.1.smithi027.stdout:1/767: link d2/d5f/d64/d81/d3d/d82/cbe d2/d5f/cfd 0 2024-04-22T15:50:14.607 INFO:tasks.workunit.client.1.smithi027.stdout:8/925: rmdir d1/d3/d1c/d28/d42 39 2024-04-22T15:50:14.610 INFO:tasks.workunit.client.1.smithi027.stdout:9/897: symlink d0/d64/l133 0 2024-04-22T15:50:14.614 INFO:tasks.workunit.client.1.smithi027.stdout:1/768: fsync d2/d36/d74/fb8 0 2024-04-22T15:50:14.615 INFO:tasks.workunit.client.1.smithi027.stdout:8/926: dread - d1/d1e/d47/d53/fe8 zero size 2024-04-22T15:50:14.618 INFO:tasks.workunit.client.1.smithi027.stdout:6/969: rename d5/db/fa9 to d5/dc/d20/f154 0 2024-04-22T15:50:14.619 INFO:tasks.workunit.client.1.smithi027.stdout:5/984: creat d0/d97/dbf/df0/f149 x:0 0 0 2024-04-22T15:50:14.622 INFO:tasks.workunit.client.1.smithi027.stdout:4/988: fsync d5/d11/d1d/f5e 0 2024-04-22T15:50:14.629 INFO:tasks.workunit.client.1.smithi027.stdout:5/985: symlink d0/d97/dbf/df0/l14a 0 2024-04-22T15:50:14.638 INFO:tasks.workunit.client.1.smithi027.stdout:1/769: creat d2/d5f/d64/d81/da7/ffe x:0 0 0 2024-04-22T15:50:14.638 INFO:tasks.workunit.client.1.smithi027.stdout:3/980: dwrite f26 [0,4194304] 0 2024-04-22T15:50:14.639 INFO:tasks.workunit.client.1.smithi027.stdout:1/770: write d2/d5f/d64/d81/f50 [2780769,31113] 0 2024-04-22T15:50:14.639 INFO:tasks.workunit.client.1.smithi027.stdout:1/771: readlink d2/l28 0 2024-04-22T15:50:14.650 INFO:tasks.workunit.client.1.smithi027.stdout:5/986: unlink d0/d1a/d22/d26/d32/d88/c108 0 2024-04-22T15:50:14.653 INFO:tasks.workunit.client.1.smithi027.stdout:1/772: stat d2/d1d/l48 0 2024-04-22T15:50:14.658 INFO:tasks.workunit.client.1.smithi027.stdout:6/970: link d5/dc/d20/c43 d5/db/d2e/d59/c155 0 2024-04-22T15:50:14.661 INFO:tasks.workunit.client.1.smithi027.stdout:1/773: dwrite d2/d5f/d64/d81/da7/ffe [0,4194304] 0 2024-04-22T15:50:14.662 INFO:tasks.workunit.client.1.smithi027.stdout:4/989: dread d5/d11/dc1/fd2 [0,4194304] 0 2024-04-22T15:50:14.663 INFO:tasks.workunit.client.1.smithi027.stdout:5/987: mknod d0/d97/dbf/df0/c14b 0 2024-04-22T15:50:14.670 INFO:tasks.workunit.client.1.smithi027.stdout:1/774: rmdir d2/d5f/d64/d81/d3d/d82/d85 39 2024-04-22T15:50:14.671 INFO:tasks.workunit.client.1.smithi027.stdout:3/981: link d28/d5d/d79/fcd d28/d5d/db0/dce/f13d 0 2024-04-22T15:50:14.673 INFO:tasks.workunit.client.1.smithi027.stdout:4/990: dwrite d5/d11/d1d/d54/d8c/f11c [0,4194304] 0 2024-04-22T15:50:14.678 INFO:tasks.workunit.client.1.smithi027.stdout:3/982: dread d28/d4b/d52/d94/fc2 [0,4194304] 0 2024-04-22T15:50:14.682 INFO:tasks.workunit.client.1.smithi027.stdout:5/988: dread d0/df/de/d40/d89/fcc [0,4194304] 0 2024-04-22T15:50:14.701 INFO:tasks.workunit.client.1.smithi027.stdout:6/971: write d5/dc/dd/dd0/f78 [106686,49746] 0 2024-04-22T15:50:14.701 INFO:tasks.workunit.client.1.smithi027.stdout:1/775: chown d2/d5f/d64/d81/c5e 42697 1 2024-04-22T15:50:14.702 INFO:tasks.workunit.client.1.smithi027.stdout:1/776: write d2/d1d/d56/dc4/fce [660451,80307] 0 2024-04-22T15:50:14.726 INFO:tasks.workunit.client.1.smithi027.stdout:4/991: unlink d5/d7/dc6/fdb 0 2024-04-22T15:50:14.727 INFO:tasks.workunit.client.1.smithi027.stdout:3/983: chown d28/d4b/f4c 136695 1 2024-04-22T15:50:14.727 INFO:tasks.workunit.client.1.smithi027.stdout:4/992: read d5/d11/d1d/ffa [633312,126137] 0 2024-04-22T15:50:14.728 INFO:tasks.workunit.client.1.smithi027.stdout:5/989: mkdir d0/df/de/d40/d89/d14c 0 2024-04-22T15:50:14.729 INFO:tasks.workunit.client.1.smithi027.stdout:5/990: chown d0/d1a/d22/d26/d32/d79/cf8 360120 1 2024-04-22T15:50:14.732 INFO:tasks.workunit.client.1.smithi027.stdout:6/972: truncate d5/dc/dd/dd0/f27 3987808 0 2024-04-22T15:50:14.733 INFO:tasks.workunit.client.1.smithi027.stdout:1/777: creat d2/d5f/d64/d81/d3d/d44/fff x:0 0 0 2024-04-22T15:50:14.738 INFO:tasks.workunit.client.1.smithi027.stdout:5/991: write d0/d1a/d22/d26/d32/f4f [4183641,57707] 0 2024-04-22T15:50:14.741 INFO:tasks.workunit.client.1.smithi027.stdout:1/778: creat d2/f100 x:0 0 0 2024-04-22T15:50:14.743 INFO:tasks.workunit.client.1.smithi027.stdout:6/973: dread - d5/dc/dd/d79/d7b/d9a/f119 zero size 2024-04-22T15:50:14.746 INFO:tasks.workunit.client.1.smithi027.stdout:5/992: stat d0/d1a/d22/d26/d5d/de3/fcb 0 2024-04-22T15:50:14.748 INFO:tasks.workunit.client.1.smithi027.stdout:6/974: symlink d5/d30/d40/l156 0 2024-04-22T15:50:14.749 INFO:tasks.workunit.client.1.smithi027.stdout:4/993: link d5/d11/d1d/d7b/fce d5/d11/d145/f154 0 2024-04-22T15:50:14.753 INFO:tasks.workunit.client.1.smithi027.stdout:4/994: unlink d5/d45/fda 0 2024-04-22T15:50:14.756 INFO:tasks.workunit.client.1.smithi027.stdout:6/975: rename d5/dc/d26 to d5/db/d2e/d59/df0/d157 0 2024-04-22T15:50:14.756 INFO:tasks.workunit.client.1.smithi027.stdout:6/976: chown d5/dc/d99/f11f 22343 1 2024-04-22T15:50:14.758 INFO:tasks.workunit.client.1.smithi027.stdout:5/993: mknod d0/df/c14d 0 2024-04-22T15:50:14.759 INFO:tasks.workunit.client.1.smithi027.stdout:1/779: write d2/d5f/d64/d81/d3d/d44/f73 [1240522,38673] 0 2024-04-22T15:50:14.767 INFO:tasks.workunit.client.1.smithi027.stdout:6/977: rename d5/d30/d54/dd8/cfd to d5/dc/dd/d79/d7b/c158 0 2024-04-22T15:50:14.767 INFO:tasks.workunit.client.1.smithi027.stdout:5/994: dread - d0/d1a/d22/d26/d5d/de3/ffa zero size 2024-04-22T15:50:14.769 INFO:tasks.workunit.client.1.smithi027.stdout:1/780: write d2/d1d/f91 [5080758,112643] 0 2024-04-22T15:50:14.770 INFO:tasks.workunit.client.1.smithi027.stdout:6/978: read d5/dc/dd/dd0/d34/fd5 [348030,45363] 0 2024-04-22T15:50:14.830 INFO:tasks.workunit.client.1.smithi027.stdout:5/995: mknod d0/df/d3d/d81/c14e 0 2024-04-22T15:50:14.833 INFO:tasks.workunit.client.1.smithi027.stdout:6/979: truncate d5/dc/dd/d79/d7b/d9a/f119 20586 0 2024-04-22T15:50:14.834 INFO:tasks.workunit.client.1.smithi027.stdout:6/980: readlink d5/d30/d54/dd8/dfc/l12a 0 2024-04-22T15:50:14.835 INFO:tasks.workunit.client.1.smithi027.stdout:4/995: creat d5/de/f155 x:0 0 0 2024-04-22T15:50:14.835 INFO:tasks.workunit.client.1.smithi027.stdout:5/996: mknod d0/d97/dbf/c14f 0 2024-04-22T15:50:14.835 INFO:tasks.workunit.client.1.smithi027.stdout:4/996: fsync d5/fd5 0 2024-04-22T15:50:14.843 INFO:tasks.workunit.client.1.smithi027.stdout:6/981: dread d5/dc/dd/dd0/d34/fd5 [0,4194304] 0 2024-04-22T15:50:14.843 INFO:tasks.workunit.client.1.smithi027.stdout:4/997: creat d5/d45/db3/f156 x:0 0 0 2024-04-22T15:50:14.843 INFO:tasks.workunit.client.1.smithi027.stdout:5/997: fdatasync d0/df/de/d24/d62/f127 0 2024-04-22T15:50:14.845 INFO:tasks.workunit.client.1.smithi027.stdout:6/982: rmdir d5/db/d2e/d59/df0/d157/d3d 39 2024-04-22T15:50:14.854 INFO:tasks.workunit.client.1.smithi027.stdout:5/998: rename d0/l17 to d0/df/de/d40/d89/d14c/l150 0 2024-04-22T15:50:14.855 INFO:tasks.workunit.client.1.smithi027.stdout:5/999: write d0/d1a/d22/d26/d32/f70 [987940,108077] 0 2024-04-22T15:50:14.857 INFO:tasks.workunit.client.1.smithi027.stdout:4/998: rmdir d5/de/d30/d5d/d150 0 2024-04-22T15:50:14.858 INFO:tasks.workunit.client.1.smithi027.stdout:4/999: fsync d5/d11/d145/f14a 0 2024-04-22T15:50:14.860 INFO:tasks.workunit.client.1.smithi027.stdout:6/983: rename d5/d30/dbe to d5/dc/dd/d79/d7b/d159 0 2024-04-22T15:50:14.879 INFO:tasks.workunit.client.1.smithi027.stdout:1/781: dwrite d2/d5f/f76 [0,4194304] 0 2024-04-22T15:50:14.880 INFO:tasks.workunit.client.1.smithi027.stdout:1/782: stat d2/d5f/d64/d81/d3d/d44/d9c/d9f/de9/lf9 0 2024-04-22T15:50:14.898 INFO:tasks.workunit.client.1.smithi027.stdout:9/898: sync 2024-04-22T15:50:14.899 INFO:tasks.workunit.client.1.smithi027.stdout:8/927: sync 2024-04-22T15:50:14.900 INFO:tasks.workunit.client.1.smithi027.stdout:9/899: chown d0/d8/l34 860631704 1 2024-04-22T15:50:14.902 INFO:tasks.workunit.client.1.smithi027.stdout:6/984: dwrite d5/dc/dd/d79/d7b/d9a/faf [0,4194304] 0 2024-04-22T15:50:14.908 INFO:tasks.workunit.client.1.smithi027.stdout:9/900: dread d0/d64/f9c [0,4194304] 0 2024-04-22T15:50:14.911 INFO:tasks.workunit.client.1.smithi027.stdout:9/901: symlink d0/dc/de/df7/l134 0 2024-04-22T15:50:14.912 INFO:tasks.workunit.client.1.smithi027.stdout:9/902: chown d0/dc/d12/d30/d5b 60880895 1 2024-04-22T15:50:14.913 INFO:tasks.workunit.client.1.smithi027.stdout:1/783: truncate d2/d1d/d56/dc4/fce 1582352 0 2024-04-22T15:50:14.917 INFO:tasks.workunit.client.1.smithi027.stdout:9/903: creat d0/dc/d12/d30/d5b/dbc/f135 x:0 0 0 2024-04-22T15:50:14.918 INFO:tasks.workunit.client.1.smithi027.stdout:1/784: truncate d2/d1d/d56/f9a 2900032 0 2024-04-22T15:50:14.920 INFO:tasks.workunit.client.1.smithi027.stdout:9/904: mkdir d0/dc/d12/d30/d5b/dbc/d136 0 2024-04-22T15:50:14.927 INFO:tasks.workunit.client.1.smithi027.stdout:1/785: creat d2/d5f/d64/d81/d3d/d82/f101 x:0 0 0 2024-04-22T15:50:14.928 INFO:tasks.workunit.client.1.smithi027.stdout:9/905: dwrite d0/dc/d12/d30/fea [0,4194304] 0 2024-04-22T15:50:14.930 INFO:tasks.workunit.client.1.smithi027.stdout:9/906: chown d0/dc/d12/d30/d5b/c78 250 1 2024-04-22T15:50:14.931 INFO:tasks.workunit.client.1.smithi027.stdout:1/786: symlink d2/d5f/d64/d81/da7/l102 0 2024-04-22T15:50:14.931 INFO:tasks.workunit.client.1.smithi027.stdout:9/907: dread d0/d64/f7c [0,4194304] 0 2024-04-22T15:50:14.937 INFO:tasks.workunit.client.1.smithi027.stdout:1/787: rmdir d2/d1d/d56/dc0/dfc 0 2024-04-22T15:50:14.938 INFO:tasks.workunit.client.1.smithi027.stdout:3/984: sync 2024-04-22T15:50:14.938 INFO:tasks.workunit.client.1.smithi027.stdout:8/928: write d1/d3/d1c/d32/fc5 [1990684,25719] 0 2024-04-22T15:50:14.939 INFO:tasks.workunit.client.1.smithi027.stdout:1/788: chown d2/d5f/f7a 4 1 2024-04-22T15:50:14.940 INFO:tasks.workunit.client.1.smithi027.stdout:8/929: chown d1/d3/d1c/d32/d41/d52/d61/l10f 2103372 1 2024-04-22T15:50:14.942 INFO:tasks.workunit.client.1.smithi027.stdout:1/789: mknod d2/c103 0 2024-04-22T15:50:14.944 INFO:tasks.workunit.client.1.smithi027.stdout:3/985: truncate d28/d4b/d52/d9a/fd1 1251698 0 2024-04-22T15:50:14.947 INFO:tasks.workunit.client.1.smithi027.stdout:9/908: getdents d0/dc/de/df7 0 2024-04-22T15:50:14.951 INFO:tasks.workunit.client.1.smithi027.stdout:6/985: dwrite d5/dc/d20/da3/fcc [0,4194304] 0 2024-04-22T15:50:14.954 INFO:tasks.workunit.client.1.smithi027.stdout:8/930: dwrite d1/d3/d1c/d32/d41/f83 [0,4194304] 0 2024-04-22T15:50:14.956 INFO:tasks.workunit.client.1.smithi027.stdout:1/790: mknod d2/d5f/d64/d81/c104 0 2024-04-22T15:50:14.956 INFO:tasks.workunit.client.1.smithi027.stdout:8/931: chown d1/d3/d1c/d32/d41/cf5 120429 1 2024-04-22T15:50:14.958 INFO:tasks.workunit.client.1.smithi027.stdout:3/986: creat d28/d4b/dc6/d130/f13e x:0 0 0 2024-04-22T15:50:14.959 INFO:tasks.workunit.client.1.smithi027.stdout:1/791: dread - d2/d5f/d64/d81/d3d/d44/fff zero size 2024-04-22T15:50:15.007 INFO:tasks.workunit.client.1.smithi027.stdout:1/792: dread - d2/d5f/d64/d81/d3d/d44/fff zero size 2024-04-22T15:50:15.011 INFO:tasks.workunit.client.1.smithi027.stdout:8/932: rmdir d1/d3/d1c/d28/d42/dc4/df3 39 2024-04-22T15:50:15.013 INFO:tasks.workunit.client.1.smithi027.stdout:3/987: truncate d28/d5d/d62/dcf/d12d/d9c/fe6 76744 0 2024-04-22T15:50:15.015 INFO:tasks.workunit.client.1.smithi027.stdout:8/933: stat d1/d3/d1c/d28/d42/ca5 0 2024-04-22T15:50:15.019 INFO:tasks.workunit.client.1.smithi027.stdout:8/934: write d1/d3/f134 [995503,69014] 0 2024-04-22T15:50:15.020 INFO:tasks.workunit.client.1.smithi027.stdout:8/935: write d1/d3/d1c/d43/d5f/fa4 [1354189,34558] 0 2024-04-22T15:50:15.021 INFO:tasks.workunit.client.1.smithi027.stdout:1/793: write d2/d30/f47 [5027624,82411] 0 2024-04-22T15:50:15.034 INFO:tasks.workunit.client.1.smithi027.stdout:3/988: getdents d28/d4b/dc6/d10c 0 2024-04-22T15:50:15.038 INFO:tasks.workunit.client.1.smithi027.stdout:1/794: write d2/d5f/d64/d81/da7/f67 [4822922,96428] 0 2024-04-22T15:50:15.041 INFO:tasks.workunit.client.1.smithi027.stdout:8/936: dwrite d1/d1e/d47/d53/f77 [0,4194304] 0 2024-04-22T15:50:15.048 INFO:tasks.workunit.client.1.smithi027.stdout:8/937: write d1/d3/d1c/d32/d41/d52/d61/fce [1101326,2182] 0 2024-04-22T15:50:15.048 INFO:tasks.workunit.client.1.smithi027.stdout:1/795: mkdir d2/d30/d105 0 2024-04-22T15:50:15.048 INFO:tasks.workunit.client.1.smithi027.stdout:3/989: dwrite d28/d4b/dc6/f11b [0,4194304] 0 2024-04-22T15:50:15.049 INFO:tasks.workunit.client.1.smithi027.stdout:1/796: truncate d2/f100 785355 0 2024-04-22T15:50:15.058 INFO:tasks.workunit.client.1.smithi027.stdout:8/938: link d1/d3/d1c/d32/fb0 d1/d3/d1c/d43/d5f/d7d/d121/f146 0 2024-04-22T15:50:15.059 INFO:tasks.workunit.client.1.smithi027.stdout:3/990: truncate d28/d4b/f57 789737 0 2024-04-22T15:50:15.066 INFO:tasks.workunit.client.1.smithi027.stdout:8/939: mkdir d1/d3/d36/d88/d147 0 2024-04-22T15:50:15.068 INFO:tasks.workunit.client.1.smithi027.stdout:3/991: chown d28/d5d/lac 2 1 2024-04-22T15:50:15.068 INFO:tasks.workunit.client.1.smithi027.stdout:1/797: getdents d2/d5f/d64/d81/d7e 0 2024-04-22T15:50:15.072 INFO:tasks.workunit.client.1.smithi027.stdout:3/992: unlink d28/d4b/f53 0 2024-04-22T15:50:15.073 INFO:tasks.workunit.client.1.smithi027.stdout:3/993: write d28/f63 [926192,120022] 0 2024-04-22T15:50:15.074 INFO:tasks.workunit.client.1.smithi027.stdout:3/994: dread - d28/d4b/dc6/d10c/f139 zero size 2024-04-22T15:50:15.077 INFO:tasks.workunit.client.1.smithi027.stdout:8/940: write d1/d3/d1c/d32/d41/f7b [495619,48768] 0 2024-04-22T15:50:15.078 INFO:tasks.workunit.client.1.smithi027.stdout:8/941: dread - d1/d3/d1c/d56/f130 zero size 2024-04-22T15:50:15.079 INFO:tasks.workunit.client.1.smithi027.stdout:3/995: rename d28/d4b/dc6/d10c to d28/d4b/d52/db2/deb/d13f 0 2024-04-22T15:50:15.083 INFO:tasks.workunit.client.1.smithi027.stdout:1/798: dread d2/d36/daa/fbf [0,4194304] 0 2024-04-22T15:50:15.084 INFO:tasks.workunit.client.1.smithi027.stdout:3/996: mkdir d28/d4b/d52/d9a/d140 0 2024-04-22T15:50:15.085 INFO:tasks.workunit.client.1.smithi027.stdout:3/997: readlink d28/d4b/d52/la9 0 2024-04-22T15:50:15.085 INFO:tasks.workunit.client.1.smithi027.stdout:8/942: dread d1/d1e/d47/dc1/fd8 [0,4194304] 0 2024-04-22T15:50:15.087 INFO:tasks.workunit.client.1.smithi027.stdout:1/799: creat d2/d5f/d64/d81/d3d/da9/f106 x:0 0 0 2024-04-22T15:50:15.087 INFO:tasks.workunit.client.1.smithi027.stdout:3/998: readlink d28/d5d/d79/l13c 0 2024-04-22T15:50:15.093 INFO:tasks.workunit.client.1.smithi027.stdout:8/943: link d1/d3/d36/d57/df0/f127 d1/d3/d36/f148 0 2024-04-22T15:50:15.095 INFO:tasks.workunit.client.1.smithi027.stdout:1/800: mkdir d2/d5f/ddb/d107 0 2024-04-22T15:50:15.095 INFO:tasks.workunit.client.1.smithi027.stdout:8/944: chown d1/d3/d1c/d32/d41/d52/d61/d8f/ddd/dbe 92184 1 2024-04-22T15:50:15.096 INFO:tasks.workunit.client.1.smithi027.stdout:3/999: write d28/f6c [1120418,24163] 0 2024-04-22T15:50:15.097 INFO:tasks.workunit.client.1.smithi027.stdout:8/945: read d1/d3/d1c/d32/d41/fab [109651,17375] 0 2024-04-22T15:50:15.101 INFO:tasks.workunit.client.1.smithi027.stdout:8/946: dread d1/d3/d1c/d56/dc9/f119 [0,4194304] 0 2024-04-22T15:50:15.110 INFO:tasks.workunit.client.1.smithi027.stdout:1/801: dwrite d2/d5f/d64/d81/d3d/d44/d9c/fe1 [0,4194304] 0 2024-04-22T15:50:15.111 INFO:tasks.workunit.client.1.smithi027.stdout:1/802: write d2/db9/ffa [727171,107833] 0 2024-04-22T15:50:15.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:14 smithi027 ceph-mon[34922]: pgmap v11: 65 pgs: 65 active+clean; 2.9 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 46 MiB/s rd, 98 MiB/s wr, 276 op/s 2024-04-22T15:50:15.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:14 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:15.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:14 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:15.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:14 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:50:15.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:14 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:50:15.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:14 smithi027 ceph-mon[34922]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-22T15:50:15.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:14 smithi027 ceph-mon[34922]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-22T15:50:15.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:14 smithi027 ceph-mon[34922]: Updating smithi016:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:50:15.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:14 smithi027 ceph-mon[34922]: Updating smithi027:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:50:15.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:14 smithi027 ceph-mon[34922]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:50:15.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:14 smithi027 ceph-mon[34922]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:50:15.407 INFO:tasks.workunit.client.1.smithi027.stdout:9/909: sync 2024-04-22T15:50:15.413 INFO:tasks.workunit.client.1.smithi027.stdout:9/910: link d0/d64/le5 d0/dc/d12/d30/d5b/d91/l137 0 2024-04-22T15:50:15.415 INFO:tasks.workunit.client.1.smithi027.stdout:9/911: write d0/dc/d12/d30/d5b/dbc/df5/d104/ddb/dfd/f105 [218862,108661] 0 2024-04-22T15:50:15.416 INFO:tasks.workunit.client.1.smithi027.stdout:9/912: fdatasync d0/d51/de6/f12d 0 2024-04-22T15:50:15.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:14 smithi016 ceph-mon[26620]: pgmap v11: 65 pgs: 65 active+clean; 2.9 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 46 MiB/s rd, 98 MiB/s wr, 276 op/s 2024-04-22T15:50:15.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:14 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:15.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:14 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:15.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:14 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:50:15.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:14 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:50:15.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:14 smithi016 ceph-mon[26620]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-22T15:50:15.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:14 smithi016 ceph-mon[26620]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-22T15:50:15.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:14 smithi016 ceph-mon[26620]: Updating smithi016:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:50:15.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:14 smithi016 ceph-mon[26620]: Updating smithi027:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:50:15.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:14 smithi016 ceph-mon[26620]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:50:15.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:14 smithi016 ceph-mon[26620]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:50:15.418 INFO:tasks.workunit.client.1.smithi027.stdout:9/913: chown d0/dc/d12/d30/d5b/dbc/df5/d104/d6b/d6e/f92 6 1 2024-04-22T15:50:15.420 INFO:tasks.workunit.client.1.smithi027.stdout:9/914: rmdir d0/dc/d12/d30/d5b/dbc/df5/d104 39 2024-04-22T15:50:15.421 INFO:tasks.workunit.client.1.smithi027.stdout:9/915: stat d0/d64/f7c 0 2024-04-22T15:50:15.421 INFO:tasks.workunit.client.1.smithi027.stdout:9/916: stat d0/dc/d12/d30/f67 0 2024-04-22T15:50:15.422 INFO:tasks.workunit.client.1.smithi027.stdout:9/917: write d0/f44 [1103038,17183] 0 2024-04-22T15:50:15.426 INFO:tasks.workunit.client.1.smithi027.stdout:9/918: symlink d0/dc/d12/d30/d5b/dbc/df5/d104/d6b/l138 0 2024-04-22T15:50:15.430 INFO:tasks.workunit.client.1.smithi027.stdout:9/919: fdatasync d0/dc/d12/d30/d5b/d91/da9/fca 0 2024-04-22T15:50:15.432 INFO:tasks.workunit.client.1.smithi027.stdout:9/920: creat d0/dc/d12/d30/d5b/dbc/df5/d104/ddb/f139 x:0 0 0 2024-04-22T15:50:15.434 INFO:tasks.workunit.client.1.smithi027.stdout:9/921: unlink d0/d8b/ld5 0 2024-04-22T15:50:15.436 INFO:tasks.workunit.client.1.smithi027.stdout:9/922: mkdir d0/dc/d12/d30/d99/da7/d13a 0 2024-04-22T15:50:15.437 INFO:tasks.workunit.client.1.smithi027.stdout:9/923: stat d0/dc/d12/d30/d5b/d91/daf/cfb 0 2024-04-22T15:50:15.438 INFO:tasks.workunit.client.1.smithi027.stdout:9/924: readlink d0/dc/d12/d30/d5b/dbc/df5/d104/la0 0 2024-04-22T15:50:15.439 INFO:tasks.workunit.client.1.smithi027.stdout:9/925: symlink d0/dc/d12/d30/d99/l13b 0 2024-04-22T15:50:15.442 INFO:tasks.workunit.client.1.smithi027.stdout:9/926: unlink d0/d8/f11 0 2024-04-22T15:50:15.457 INFO:tasks.workunit.client.1.smithi027.stdout:9/927: truncate d0/dc/d12/d30/f4f 3387475 0 2024-04-22T15:50:15.457 INFO:tasks.workunit.client.1.smithi027.stdout:9/928: chown d0/d8b/c9a 3 1 2024-04-22T15:50:15.462 INFO:tasks.workunit.client.1.smithi027.stdout:8/947: sync 2024-04-22T15:50:15.462 INFO:tasks.workunit.client.1.smithi027.stdout:1/803: sync 2024-04-22T15:50:15.462 INFO:tasks.workunit.client.1.smithi027.stdout:6/986: sync 2024-04-22T15:50:15.464 INFO:tasks.workunit.client.1.smithi027.stdout:1/804: fdatasync d2/d5f/d64/d81/d3d/d44/f73 0 2024-04-22T15:50:15.466 INFO:tasks.workunit.client.1.smithi027.stdout:1/805: write d2/d5f/d64/d81/d7e/fa1 [2030200,13696] 0 2024-04-22T15:50:15.469 INFO:tasks.workunit.client.1.smithi027.stdout:6/987: creat d5/db/d2e/f15a x:0 0 0 2024-04-22T15:50:15.473 INFO:tasks.workunit.client.1.smithi027.stdout:8/948: truncate d1/d3/d1c/f34 438887 0 2024-04-22T15:50:15.473 INFO:tasks.workunit.client.1.smithi027.stdout:1/806: dread d2/d5f/fd6 [0,4194304] 0 2024-04-22T15:50:15.480 INFO:tasks.workunit.client.1.smithi027.stdout:1/807: dwrite d2/d5f/d64/d81/d3d/d44/d9c/d9f/fac [0,4194304] 0 2024-04-22T15:50:15.482 INFO:tasks.workunit.client.1.smithi027.stdout:8/949: creat d1/d1e/dbc/f149 x:0 0 0 2024-04-22T15:50:15.487 INFO:tasks.workunit.client.1.smithi027.stdout:9/929: fdatasync d0/dc/d12/d30/d5b/d91/da9/fca 0 2024-04-22T15:50:15.490 INFO:tasks.workunit.client.1.smithi027.stdout:6/988: truncate d5/db/d2e/d59/df0/f11a 238632 0 2024-04-22T15:50:15.492 INFO:tasks.workunit.client.1.smithi027.stdout:1/808: rename d2/d1d/ffb to d2/d1d/f108 0 2024-04-22T15:50:15.492 INFO:tasks.workunit.client.1.smithi027.stdout:8/950: creat d1/d3/d1c/d49/d94/f14a x:0 0 0 2024-04-22T15:50:15.500 INFO:tasks.workunit.client.1.smithi027.stdout:1/809: mkdir d2/d1d/d56/d109 0 2024-04-22T15:50:15.502 INFO:tasks.workunit.client.1.smithi027.stdout:8/951: mkdir d1/d3/d1c/d133/d14b 0 2024-04-22T15:50:15.506 INFO:tasks.workunit.client.1.smithi027.stdout:6/989: write d5/dc/dd/d79/d7b/d159/d112/ff3 [604658,98469] 0 2024-04-22T15:50:15.506 INFO:tasks.workunit.client.1.smithi027.stdout:9/930: dread d0/dc/d12/d30/d5b/d91/fe0 [0,4194304] 0 2024-04-22T15:50:15.507 INFO:tasks.workunit.client.1.smithi027.stdout:1/810: rmdir d2/d36 39 2024-04-22T15:50:15.507 INFO:tasks.workunit.client.1.smithi027.stdout:1/811: readlink d2/l19 0 2024-04-22T15:50:15.509 INFO:tasks.workunit.client.1.smithi027.stdout:9/931: chown d0/dc/d12/d30/d5b/dbc/df5/d104/d6b/f108 109 1 2024-04-22T15:50:15.509 INFO:tasks.workunit.client.1.smithi027.stdout:8/952: fdatasync d1/d3/d1c/d49/f5d 0 2024-04-22T15:50:15.511 INFO:tasks.workunit.client.1.smithi027.stdout:8/953: read - d1/d3/d1c/d49/d94/f143 zero size 2024-04-22T15:50:15.516 INFO:tasks.workunit.client.1.smithi027.stdout:9/932: creat d0/dc/d12/d30/d5b/dbc/df5/d104/ddb/dfd/f13c x:0 0 0 2024-04-22T15:50:15.518 INFO:tasks.workunit.client.1.smithi027.stdout:6/990: link d5/db/d2e/d59/df0/d157/d49/d86/ffb d5/dc/d20/da3/dde/f15b 0 2024-04-22T15:50:15.522 INFO:tasks.workunit.client.1.smithi027.stdout:9/933: unlink d0/dc/d12/d30/d99/da7/f11f 0 2024-04-22T15:50:15.524 INFO:tasks.workunit.client.1.smithi027.stdout:6/991: unlink d5/dc/dd/d79/d7b/d9a/faf 0 2024-04-22T15:50:15.526 INFO:tasks.workunit.client.1.smithi027.stdout:8/954: symlink d1/d3/d1c/d43/d82/d13f/l14c 0 2024-04-22T15:50:15.526 INFO:tasks.workunit.client.1.smithi027.stdout:1/812: dwrite d2/d5f/d64/d81/fc9 [0,4194304] 0 2024-04-22T15:50:15.531 INFO:tasks.workunit.client.1.smithi027.stdout:1/813: chown d2/d36/daa 31 1 2024-04-22T15:50:15.532 INFO:tasks.workunit.client.1.smithi027.stdout:1/814: write d2/d30/f47 [1126439,19779] 0 2024-04-22T15:50:15.533 INFO:tasks.workunit.client.1.smithi027.stdout:1/815: read d2/d5f/d64/d81/d3d/d82/dc2/fed [3185759,12596] 0 2024-04-22T15:50:15.535 INFO:tasks.workunit.client.1.smithi027.stdout:9/934: readlink d0/dc/de/l43 0 2024-04-22T15:50:15.543 INFO:tasks.workunit.client.1.smithi027.stdout:9/935: fdatasync d0/dc/d12/d30/d5b/d91/da9/f120 0 2024-04-22T15:50:15.544 INFO:tasks.workunit.client.1.smithi027.stdout:6/992: link d5/dc/d17/d53/d8c/cb3 d5/dc/dd/c15c 0 2024-04-22T15:50:15.547 INFO:tasks.workunit.client.1.smithi027.stdout:9/936: mknod d0/dc/d12/d30/d99/db4/c13d 0 2024-04-22T15:50:15.550 INFO:tasks.workunit.client.1.smithi027.stdout:1/816: creat d2/d5f/d64/f10a x:0 0 0 2024-04-22T15:50:15.552 INFO:tasks.workunit.client.1.smithi027.stdout:6/993: mkdir d5/db/d145/d15d 0 2024-04-22T15:50:15.555 INFO:tasks.workunit.client.1.smithi027.stdout:9/937: rename d0/dc/l3e to d0/d51/l13e 0 2024-04-22T15:50:15.556 INFO:tasks.workunit.client.1.smithi027.stdout:9/938: write d0/dc/de/f40 [2796054,17332] 0 2024-04-22T15:50:15.558 INFO:tasks.workunit.client.1.smithi027.stdout:9/939: chown d0/dc/d12/d30/d5b/dbc/cba 21466 1 2024-04-22T15:50:15.561 INFO:tasks.workunit.client.1.smithi027.stdout:1/817: getdents d2/d30 0 2024-04-22T15:50:15.562 INFO:tasks.workunit.client.1.smithi027.stdout:1/818: write d2/d5f/fd6 [2285928,76379] 0 2024-04-22T15:50:15.569 INFO:tasks.workunit.client.1.smithi027.stdout:9/940: mkdir d0/dc/d12/d30/d99/da7/d13a/d13f 0 2024-04-22T15:50:15.570 INFO:tasks.workunit.client.1.smithi027.stdout:9/941: stat d0/dc/d12/d30/d5b/dbc/df5/d104/ddb/ff2 0 2024-04-22T15:50:15.570 INFO:tasks.workunit.client.1.smithi027.stdout:1/819: mkdir d2/d5f/d64/d10b 0 2024-04-22T15:50:15.574 INFO:tasks.workunit.client.1.smithi027.stdout:1/820: creat d2/df0/f10c x:0 0 0 2024-04-22T15:50:15.575 INFO:tasks.workunit.client.1.smithi027.stdout:9/942: truncate d0/dc/d12/d30/f7a 5077046 0 2024-04-22T15:50:15.576 INFO:tasks.workunit.client.1.smithi027.stdout:6/994: dwrite d5/db/d2e/d59/df0/d157/d3d/feb [0,4194304] 0 2024-04-22T15:50:15.579 INFO:tasks.workunit.client.1.smithi027.stdout:6/995: dread - d5/dc/d17/d53/d8c/f14a zero size 2024-04-22T15:50:15.582 INFO:tasks.workunit.client.1.smithi027.stdout:1/821: unlink d2/d5f/d64/d81/d3d/d82/d85/cad 0 2024-04-22T15:50:15.587 INFO:tasks.workunit.client.1.smithi027.stdout:1/822: fsync d2/d1d/fd9 0 2024-04-22T15:50:15.590 INFO:tasks.workunit.client.1.smithi027.stdout:9/943: getdents d0/dc/d12/d30/d5b/dbc/df5/d104/da6/de8 0 2024-04-22T15:50:15.592 INFO:tasks.workunit.client.1.smithi027.stdout:6/996: creat d5/dc/dd/dd0/d34/f15e x:0 0 0 2024-04-22T15:50:15.593 INFO:tasks.workunit.client.1.smithi027.stdout:1/823: symlink d2/d36/l10d 0 2024-04-22T15:50:15.594 INFO:tasks.workunit.client.1.smithi027.stdout:1/824: chown d2/d30/f3c 89913 1 2024-04-22T15:50:15.598 INFO:tasks.workunit.client.1.smithi027.stdout:1/825: fsync d2/f68 0 2024-04-22T15:50:15.601 INFO:tasks.workunit.client.1.smithi027.stdout:1/826: symlink d2/d36/l10e 0 2024-04-22T15:50:15.602 INFO:tasks.workunit.client.1.smithi027.stdout:1/827: fdatasync d2/d1d/f5b 0 2024-04-22T15:50:15.606 INFO:tasks.workunit.client.1.smithi027.stdout:1/828: mkdir d2/d5f/d64/d81/d3d/d44/d9c/d9f/de9/d10f 0 2024-04-22T15:50:15.607 INFO:tasks.workunit.client.1.smithi027.stdout:6/997: dwrite d5/dc/dd/d79/d7b/d9a/fa0 [0,4194304] 0 2024-04-22T15:50:15.610 INFO:tasks.workunit.client.1.smithi027.stdout:1/829: truncate d2/d36/d74/fb8 550101 0 2024-04-22T15:50:15.735 INFO:tasks.workunit.client.1.smithi027.stdout:1/830: mkdir d2/d5f/d64/d81/dcf/d110 0 2024-04-22T15:50:15.741 INFO:tasks.workunit.client.1.smithi027.stdout:1/831: dwrite d2/f3 [0,4194304] 0 2024-04-22T15:50:15.745 INFO:tasks.workunit.client.1.smithi027.stdout:1/832: creat d2/d5f/ddb/d107/f111 x:0 0 0 2024-04-22T15:50:15.747 INFO:tasks.workunit.client.1.smithi027.stdout:1/833: fsync d2/d36/f7c 0 2024-04-22T15:50:15.751 INFO:tasks.workunit.client.1.smithi027.stdout:1/834: mkdir d2/d5f/d64/d81/dcf/df2/d112 0 2024-04-22T15:50:15.761 INFO:tasks.workunit.client.1.smithi027.stdout:1/835: dwrite d2/d5f/d64/d81/d3d/d44/d9c/d9f/fd0 [0,4194304] 0 2024-04-22T15:50:15.764 INFO:tasks.workunit.client.1.smithi027.stdout:1/836: read d2/d5f/d64/d81/d3d/d82/d85/f95 [242675,59361] 0 2024-04-22T15:50:15.772 INFO:tasks.workunit.client.1.smithi027.stdout:8/955: sync 2024-04-22T15:50:15.772 INFO:tasks.workunit.client.1.smithi027.stdout:9/944: sync 2024-04-22T15:50:15.772 INFO:tasks.workunit.client.1.smithi027.stdout:1/837: dread d2/d36/daa/fbf [0,4194304] 0 2024-04-22T15:50:15.775 INFO:tasks.workunit.client.1.smithi027.stdout:8/956: write d1/d3/d1c/d32/d41/d52/fd0 [1600222,63397] 0 2024-04-22T15:50:15.777 INFO:tasks.workunit.client.1.smithi027.stdout:9/945: dread d0/fb6 [0,4194304] 0 2024-04-22T15:50:15.811 INFO:tasks.workunit.client.1.smithi027.stdout:1/838: getdents d2/d36/d74 0 2024-04-22T15:50:15.811 INFO:tasks.workunit.client.1.smithi027.stdout:1/839: dread - d2/d5f/d64/f10a zero size 2024-04-22T15:50:15.818 INFO:tasks.workunit.client.1.smithi027.stdout:1/840: link d2/d5f/d64/d81/da7/ffe d2/d5f/d64/d81/dcf/df2/d112/f113 0 2024-04-22T15:50:15.831 INFO:tasks.workunit.client.1.smithi027.stdout:1/841: getdents d2/d5f/d64/d81/d3d/d44 0 2024-04-22T15:50:15.832 INFO:tasks.workunit.client.1.smithi027.stdout:1/842: fsync d2/d5f/d64/d81/d3d/d44/d9c/fe1 0 2024-04-22T15:50:15.840 INFO:tasks.workunit.client.1.smithi027.stdout:1/843: dwrite d2/d5f/d64/d81/d3d/d44/d9c/df8/ff3 [0,4194304] 0 2024-04-22T15:50:15.844 INFO:tasks.workunit.client.1.smithi027.stdout:6/998: rename d5/dc/d17/d11c/d11d/c102 to d5/db/d2e/d59/c15f 0 2024-04-22T15:50:15.846 INFO:tasks.workunit.client.1.smithi027.stdout:1/844: fdatasync d2/d5f/fd1 0 2024-04-22T15:50:15.862 INFO:tasks.workunit.client.1.smithi027.stdout:1/845: getdents d2/d5f/d64/d81/d3d 0 2024-04-22T15:50:15.863 INFO:tasks.workunit.client.1.smithi027.stdout:1/846: readlink d2/l31 0 2024-04-22T15:50:15.863 INFO:tasks.workunit.client.1.smithi027.stdout:1/847: creat d2/d30/f114 x:0 0 0 2024-04-22T15:50:15.863 INFO:tasks.workunit.client.1.smithi027.stdout:1/848: truncate d2/df0/f10c 393904 0 2024-04-22T15:50:15.883 INFO:tasks.workunit.client.1.smithi027.stdout:6/999: unlink d5/l32 0 2024-04-22T15:50:15.927 INFO:tasks.workunit.client.1.smithi027.stdout:8/957: mkdir d1/d3/d1c/d32/d41/d52/d14d 0 2024-04-22T15:50:15.929 INFO:tasks.workunit.client.1.smithi027.stdout:9/946: sync 2024-04-22T15:50:15.930 INFO:tasks.workunit.client.1.smithi027.stdout:9/947: readlink d0/d8b/lf1 0 2024-04-22T15:50:15.934 INFO:tasks.workunit.client.1.smithi027.stdout:8/958: rename d1/d3/d1c/d32/d41/d52/d61/d8f/ddd/l66 to d1/d3/d1c/d133/l14e 0 2024-04-22T15:50:15.935 INFO:tasks.workunit.client.1.smithi027.stdout:9/948: symlink d0/dc/d12/d30/d5b/dbc/df5/d104/d6b/d6e/l140 0 2024-04-22T15:50:15.955 INFO:tasks.workunit.client.1.smithi027.stdout:8/959: link d1/d1e/d47/c12d d1/d3/d1c/d49/d94/d11f/c14f 0 2024-04-22T15:50:15.961 INFO:tasks.workunit.client.1.smithi027.stdout:1/849: rename d2/d5f/d64/d81/d3d/d82/d85 to d2/d30/d115 0 2024-04-22T15:50:15.962 INFO:tasks.workunit.client.1.smithi027.stdout:8/960: mknod d1/d1e/d47/c150 0 2024-04-22T15:50:15.969 INFO:tasks.workunit.client.1.smithi027.stdout:1/850: creat d2/d30/d105/f116 x:0 0 0 2024-04-22T15:50:15.971 INFO:tasks.workunit.client.1.smithi027.stdout:1/851: creat d2/d1d/f117 x:0 0 0 2024-04-22T15:50:15.975 INFO:tasks.workunit.client.1.smithi027.stdout:1/852: getdents d2/d5f 0 2024-04-22T15:50:15.977 INFO:tasks.workunit.client.1.smithi027.stdout:1/853: write d2/d5f/d64/d81/d3d/fe6 [941980,13008] 0 2024-04-22T15:50:15.979 INFO:tasks.workunit.client.1.smithi027.stdout:1/854: write d2/d5f/d64/d81/d7e/fd2 [331753,61436] 0 2024-04-22T15:50:15.984 INFO:tasks.workunit.client.1.smithi027.stdout:1/855: dwrite d2/d5f/d64/d81/fc9 [0,4194304] 0 2024-04-22T15:50:15.987 INFO:tasks.workunit.client.1.smithi027.stdout:1/856: fsync d2/d36/daa/fbf 0 2024-04-22T15:50:15.988 INFO:tasks.workunit.client.1.smithi027.stdout:1/857: dread - d2/d5f/ff1 zero size 2024-04-22T15:50:15.989 INFO:tasks.workunit.client.1.smithi027.stdout:1/858: write d2/d5f/d64/d81/d3d/d44/d9c/d9f/de9/fec [827724,40211] 0 2024-04-22T15:50:15.993 INFO:tasks.workunit.client.1.smithi027.stdout:1/859: creat d2/d5f/d64/d81/dcf/df2/f118 x:0 0 0 2024-04-22T15:50:16.003 INFO:tasks.workunit.client.1.smithi027.stdout:9/949: dwrite d0/d51/d9d/fad [4194304,4194304] 0 2024-04-22T15:50:16.004 INFO:tasks.workunit.client.1.smithi027.stdout:9/950: stat d0/d8/cc4 0 2024-04-22T15:50:16.011 INFO:tasks.workunit.client.1.smithi027.stdout:9/951: creat d0/dc/d12/d30/d5b/dbc/df5/d104/d6b/f141 x:0 0 0 2024-04-22T15:50:16.012 INFO:tasks.workunit.client.1.smithi027.stdout:8/961: write d1/f92 [121589,32236] 0 2024-04-22T15:50:16.014 INFO:tasks.workunit.client.1.smithi027.stdout:8/962: read d1/d1e/f2f [2432976,28591] 0 2024-04-22T15:50:16.025 INFO:tasks.workunit.client.1.smithi027.stdout:8/963: mknod d1/d3/d1c/d133/d14b/c151 0 2024-04-22T15:50:16.028 INFO:tasks.workunit.client.1.smithi027.stdout:1/860: write d2/f7f [1859755,83334] 0 2024-04-22T15:50:16.029 INFO:tasks.workunit.client.1.smithi027.stdout:1/861: fdatasync d2/d30/f114 0 2024-04-22T15:50:16.030 INFO:tasks.workunit.client.1.smithi027.stdout:1/862: stat d2/d5f/d64/d81/d3d/d44/f6d 0 2024-04-22T15:50:16.033 INFO:tasks.workunit.client.1.smithi027.stdout:1/863: mkdir d2/d1d/d56/dc4/d119 0 2024-04-22T15:50:16.037 INFO:tasks.workunit.client.1.smithi027.stdout:1/864: creat d2/d5f/d64/d81/dcf/df2/d112/f11a x:0 0 0 2024-04-22T15:50:16.041 INFO:tasks.workunit.client.1.smithi027.stdout:8/964: dwrite d1/d3/d1c/d56/f7f [0,4194304] 0 2024-04-22T15:50:16.043 INFO:tasks.workunit.client.1.smithi027.stdout:1/865: mknod d2/d5f/d64/d81/c11b 0 2024-04-22T15:50:16.049 INFO:tasks.workunit.client.1.smithi027.stdout:8/965: dwrite d1/d4f/f10a [4194304,4194304] 0 2024-04-22T15:50:16.052 INFO:tasks.workunit.client.1.smithi027.stdout:1/866: mkdir d2/d5f/d64/d81/d3d/d44/d9c/d9f/de9/d10f/d11c 0 2024-04-22T15:50:16.055 INFO:tasks.workunit.client.1.smithi027.stdout:1/867: stat d2/lb 0 2024-04-22T15:50:16.059 INFO:tasks.workunit.client.1.smithi027.stdout:8/966: getdents d1/d3/d1c/d32 0 2024-04-22T15:50:16.066 INFO:tasks.workunit.client.1.smithi027.stdout:1/868: dwrite d2/d5f/d64/d81/d7e/fe8 [0,4194304] 0 2024-04-22T15:50:16.069 INFO:tasks.workunit.client.1.smithi027.stdout:8/967: dwrite d1/d3/d1c/d49/f140 [0,4194304] 0 2024-04-22T15:50:16.072 INFO:tasks.workunit.client.1.smithi027.stdout:1/869: mkdir d2/d5f/d64/d81/d3d/d44/d9c/df8/d11d 0 2024-04-22T15:50:16.076 INFO:tasks.workunit.client.1.smithi027.stdout:8/968: unlink d1/d3/d1c/d32/d41/d52/d61/d8f/ddd/dbe/fde 0 2024-04-22T15:50:16.076 INFO:tasks.workunit.client.1.smithi027.stdout:8/969: chown d1/d3 2755177 1 2024-04-22T15:50:16.077 INFO:tasks.workunit.client.1.smithi027.stdout:8/970: chown d1/d3/d36/d57 434375961 1 2024-04-22T15:50:16.082 INFO:tasks.workunit.client.1.smithi027.stdout:8/971: creat d1/d3/d1c/d32/d41/d52/d61/d8f/ddd/f152 x:0 0 0 2024-04-22T15:50:16.084 INFO:tasks.workunit.client.1.smithi027.stdout:1/870: creat d2/d5f/d64/d81/d3d/d44/d9c/f11e x:0 0 0 2024-04-22T15:50:16.086 INFO:tasks.workunit.client.1.smithi027.stdout:8/972: symlink d1/d3/d1c/d32/d41/d52/d61/d8f/ddd/dbe/l153 0 2024-04-22T15:50:16.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:16 smithi027 ceph-mon[34922]: Updating smithi016:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:50:16.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:16 smithi027 ceph-mon[34922]: Updating smithi027:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:50:16.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:16 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:16.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:16 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:16.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:16 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:16.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:16 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:16.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:16 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:16.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:16 smithi027 ceph-mon[34922]: Reconfiguring prometheus.smithi016 (dependencies changed)... 2024-04-22T15:50:16.087 INFO:tasks.workunit.client.1.smithi027.stdout:8/973: readlink d1/d3/d36/d57/l58 0 2024-04-22T15:50:16.090 INFO:tasks.workunit.client.1.smithi027.stdout:1/871: rename d2/d5f/d64/d81/d3d/d44/d9c/f11e to d2/db9/f11f 0 2024-04-22T15:50:16.096 INFO:tasks.workunit.client.1.smithi027.stdout:1/872: mknod d2/d5f/c120 0 2024-04-22T15:50:16.096 INFO:tasks.workunit.client.1.smithi027.stdout:1/873: stat d2/d1d/c1b 0 2024-04-22T15:50:16.096 INFO:tasks.workunit.client.1.smithi027.stdout:1/874: chown d2/lb 395321351 1 2024-04-22T15:50:16.098 INFO:tasks.workunit.client.1.smithi027.stdout:1/875: read d2/d1d/d56/dc4/fc8 [3149035,66077] 0 2024-04-22T15:50:16.098 INFO:tasks.workunit.client.1.smithi027.stdout:1/876: fsync d2/d5f/d64/d81/d3d/d44/d9c/df8/ff3 0 2024-04-22T15:50:16.099 INFO:tasks.workunit.client.1.smithi027.stdout:1/877: stat d2/d5f/d64/d81 0 2024-04-22T15:50:16.105 INFO:tasks.workunit.client.1.smithi027.stdout:8/974: dwrite d1/d3/fa [0,4194304] 0 2024-04-22T15:50:16.109 INFO:tasks.workunit.client.1.smithi027.stdout:1/878: write d2/d5f/d64/d81/d3d/d82/dc2/fed [1926792,4869] 0 2024-04-22T15:50:16.111 INFO:tasks.workunit.client.1.smithi027.stdout:1/879: write d2/db9/ffa [1155970,57281] 0 2024-04-22T15:50:16.114 INFO:tasks.workunit.client.1.smithi027.stdout:1/880: unlink d2/d1d/fd9 0 2024-04-22T15:50:16.121 INFO:tasks.workunit.client.1.smithi027.stdout:8/975: getdents d1/d3/d1c/d49 0 2024-04-22T15:50:16.122 INFO:tasks.workunit.client.1.smithi027.stdout:8/976: chown d1/d3/d1c/d56/l64 463574210 1 2024-04-22T15:50:16.128 INFO:tasks.workunit.client.1.smithi027.stdout:8/977: write d1/d1e/f33 [28464,8227] 0 2024-04-22T15:50:16.133 INFO:tasks.workunit.client.1.smithi027.stdout:1/881: dread d2/d1d/f91 [0,4194304] 0 2024-04-22T15:50:16.136 INFO:tasks.workunit.client.1.smithi027.stdout:1/882: mknod d2/d5f/d64/d81/d3d/d44/d9c/d9f/c121 0 2024-04-22T15:50:16.138 INFO:tasks.workunit.client.1.smithi027.stdout:8/978: dwrite d1/d3/d1c/d28/f51 [0,4194304] 0 2024-04-22T15:50:16.145 INFO:tasks.workunit.client.1.smithi027.stdout:8/979: symlink d1/d3/d1c/l154 0 2024-04-22T15:50:16.147 INFO:tasks.workunit.client.1.smithi027.stdout:1/883: getdents d2/d5f/d64/d81/dcf/df2 0 2024-04-22T15:50:16.150 INFO:tasks.workunit.client.1.smithi027.stdout:8/980: read d1/d1e/d47/d53/d76/fad [1535939,119483] 0 2024-04-22T15:50:16.152 INFO:tasks.workunit.client.1.smithi027.stdout:8/981: dread - d1/d3/d1c/d49/fe6 zero size 2024-04-22T15:50:16.153 INFO:tasks.workunit.client.1.smithi027.stdout:1/884: dwrite d2/f7f [0,4194304] 0 2024-04-22T15:50:16.160 INFO:tasks.workunit.client.1.smithi027.stdout:8/982: dwrite d1/d3/f134 [0,4194304] 0 2024-04-22T15:50:16.160 INFO:tasks.workunit.client.1.smithi027.stdout:1/885: truncate d2/d1d/f32 43670 0 2024-04-22T15:50:16.168 INFO:tasks.workunit.client.1.smithi027.stdout:1/886: dwrite d2/d1d/f117 [0,4194304] 0 2024-04-22T15:50:16.169 INFO:tasks.workunit.client.1.smithi027.stdout:1/887: dread - d2/d5f/d64/f10a zero size 2024-04-22T15:50:16.174 INFO:tasks.workunit.client.1.smithi027.stdout:1/888: creat d2/d5f/ddb/d107/f122 x:0 0 0 2024-04-22T15:50:16.182 INFO:tasks.workunit.client.1.smithi027.stdout:1/889: getdents d2/d5f/d64 0 2024-04-22T15:50:16.183 INFO:tasks.workunit.client.1.smithi027.stdout:8/983: dwrite d1/d1e/d47/dc1/fd8 [0,4194304] 0 2024-04-22T15:50:16.185 INFO:tasks.workunit.client.1.smithi027.stdout:1/890: creat d2/d1d/f123 x:0 0 0 2024-04-22T15:50:16.188 INFO:tasks.workunit.client.1.smithi027.stdout:8/984: readlink d1/d3/d1c/d32/d41/d52/d61/d8f/lbb 0 2024-04-22T15:50:16.192 INFO:tasks.workunit.client.1.smithi027.stdout:1/891: fdatasync d2/f18 0 2024-04-22T15:50:16.194 INFO:tasks.workunit.client.1.smithi027.stdout:8/985: creat d1/d3/d36/d88/f155 x:0 0 0 2024-04-22T15:50:16.197 INFO:tasks.workunit.client.1.smithi027.stdout:1/892: mkdir d2/d5f/d64/d81/da7/d124 0 2024-04-22T15:50:16.200 INFO:tasks.workunit.client.1.smithi027.stdout:1/893: creat d2/d5f/d64/d81/d3d/d82/dc2/f125 x:0 0 0 2024-04-22T15:50:16.218 INFO:tasks.workunit.client.1.smithi027.stdout:1/894: write d2/d5f/d64/d81/d3d/d44/d9c/df8/fd4 [864460,27143] 0 2024-04-22T15:50:16.223 INFO:tasks.workunit.client.1.smithi027.stdout:1/895: dread d2/d5f/d64/d81/dcf/df2/d112/f113 [0,4194304] 0 2024-04-22T15:50:16.225 INFO:tasks.workunit.client.1.smithi027.stdout:1/896: creat d2/d5f/d64/d81/dcf/f126 x:0 0 0 2024-04-22T15:50:16.227 INFO:tasks.workunit.client.1.smithi027.stdout:1/897: truncate d2/d1d/f96 296645 0 2024-04-22T15:50:16.233 INFO:tasks.workunit.client.1.smithi027.stdout:1/898: dwrite d2/fa [0,4194304] 0 2024-04-22T15:50:16.246 INFO:tasks.workunit.client.1.smithi027.stdout:1/899: getdents d2 0 2024-04-22T15:50:16.250 INFO:tasks.workunit.client.1.smithi027.stdout:1/900: dwrite d2/d5f/d64/d81/d7e/fb4 [0,4194304] 0 2024-04-22T15:50:16.251 INFO:tasks.workunit.client.1.smithi027.stdout:1/901: stat d2/d5f/d64/d81/d3d 0 2024-04-22T15:50:16.252 INFO:tasks.workunit.client.1.smithi027.stdout:1/902: chown d2/d5f/d64/lf5 387 1 2024-04-22T15:50:16.255 INFO:tasks.workunit.client.1.smithi027.stdout:1/903: rmdir d2/d1d/d56/dc4 39 2024-04-22T15:50:16.266 INFO:tasks.workunit.client.1.smithi027.stdout:1/904: dwrite d2/f60 [0,4194304] 0 2024-04-22T15:50:16.270 INFO:tasks.workunit.client.1.smithi027.stdout:1/905: unlink d2/d5f/d64/d81/d7e/fdd 0 2024-04-22T15:50:16.277 INFO:tasks.workunit.client.1.smithi027.stdout:1/906: dwrite d2/d1d/f123 [0,4194304] 0 2024-04-22T15:50:16.283 INFO:tasks.workunit.client.1.smithi027.stdout:1/907: mknod d2/d5f/d64/d81/d3d/d44/d9c/d9f/de9/d10f/d11c/c127 0 2024-04-22T15:50:16.301 INFO:tasks.workunit.client.1.smithi027.stdout:1/908: dwrite d2/d1d/d56/dc4/fce [0,4194304] 0 2024-04-22T15:50:16.304 INFO:tasks.workunit.client.1.smithi027.stdout:1/909: symlink d2/d5f/d64/l128 0 2024-04-22T15:50:16.306 INFO:tasks.workunit.client.1.smithi027.stdout:1/910: stat d2/d1d/d56/f9a 0 2024-04-22T15:50:16.310 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:16 smithi016 ceph-mon[26620]: Updating smithi016:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:50:16.310 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:16 smithi016 ceph-mon[26620]: Updating smithi027:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:50:16.310 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:16 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:16.310 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:16 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:16.311 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:16 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:16.311 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:16 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:16.311 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:16 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:16.311 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:16 smithi016 ceph-mon[26620]: Reconfiguring prometheus.smithi016 (dependencies changed)... 2024-04-22T15:50:16.312 INFO:tasks.workunit.client.1.smithi027.stdout:1/911: truncate d2/db9/ffa 888621 0 2024-04-22T15:50:16.314 INFO:tasks.workunit.client.1.smithi027.stdout:1/912: truncate d2/d36/f62 278761 0 2024-04-22T15:50:16.320 INFO:tasks.workunit.client.1.smithi027.stdout:1/913: dwrite d2/df0/f10c [0,4194304] 0 2024-04-22T15:50:16.333 INFO:tasks.workunit.client.1.smithi027.stdout:1/914: rename d2/d5f/d64 to d2/d5f/ddb/d107/d129 0 2024-04-22T15:50:16.333 INFO:tasks.workunit.client.1.smithi027.stdout:1/915: stat d2/lb 0 2024-04-22T15:50:16.344 INFO:tasks.workunit.client.1.smithi027.stdout:9/952: sync 2024-04-22T15:50:16.345 INFO:tasks.workunit.client.1.smithi027.stdout:9/953: truncate d0/dc/de/df7/f11e 95087 0 2024-04-22T15:50:16.348 INFO:tasks.workunit.client.1.smithi027.stdout:9/954: symlink d0/d64/l142 0 2024-04-22T15:50:16.350 INFO:tasks.workunit.client.1.smithi027.stdout:9/955: chown d0/d8/l4e 60 1 2024-04-22T15:50:16.351 INFO:tasks.workunit.client.1.smithi027.stdout:1/916: truncate d2/d1d/f117 2623766 0 2024-04-22T15:50:16.353 INFO:tasks.workunit.client.1.smithi027.stdout:1/917: rmdir d2/d36/d74 39 2024-04-22T15:50:16.365 INFO:tasks.workunit.client.1.smithi027.stdout:1/918: write d2/d30/d115/feb [3782794,122821] 0 2024-04-22T15:50:16.371 INFO:tasks.workunit.client.1.smithi027.stdout:1/919: creat d2/d5f/ddb/d107/d129/d81/d3d/da9/f12a x:0 0 0 2024-04-22T15:50:16.381 INFO:tasks.workunit.client.1.smithi027.stdout:1/920: creat d2/d36/f12b x:0 0 0 2024-04-22T15:50:16.382 INFO:tasks.workunit.client.1.smithi027.stdout:1/921: getdents d2/d1d/d56/d109 0 2024-04-22T15:50:16.386 INFO:tasks.workunit.client.1.smithi027.stdout:1/922: dread d2/d5f/ddb/d107/d129/d81/da7/ffe [0,4194304] 0 2024-04-22T15:50:16.512 INFO:tasks.workunit.client.1.smithi027.stdout:8/986: sync 2024-04-22T15:50:16.515 INFO:tasks.workunit.client.1.smithi027.stdout:8/987: creat d1/d3/d36/d57/df0/f156 x:0 0 0 2024-04-22T15:50:16.533 INFO:tasks.workunit.client.1.smithi027.stdout:8/988: write d1/d3/d1c/d32/d41/d52/f60 [288479,70590] 0 2024-04-22T15:50:16.533 INFO:tasks.workunit.client.1.smithi027.stdout:8/989: rename d1 to d1/d4f/d157 22 2024-04-22T15:50:16.537 INFO:tasks.workunit.client.1.smithi027.stdout:8/990: dread d1/d3/f134 [0,4194304] 0 2024-04-22T15:50:16.541 INFO:tasks.workunit.client.1.smithi027.stdout:8/991: mknod d1/d3/d1c/d32/d41/c158 0 2024-04-22T15:50:16.547 INFO:tasks.workunit.client.1.smithi027.stdout:8/992: dwrite d1/d3/d1c/d32/f69 [0,4194304] 0 2024-04-22T15:50:16.548 INFO:tasks.workunit.client.1.smithi027.stdout:8/993: chown d1/d3/d1c/d56/c72 2083 1 2024-04-22T15:50:16.548 INFO:tasks.workunit.client.1.smithi027.stdout:8/994: readlink d1/d1e/l24 0 2024-04-22T15:50:16.551 INFO:tasks.workunit.client.1.smithi027.stdout:8/995: mkdir d1/d1e/dbc/d159 0 2024-04-22T15:50:16.555 INFO:tasks.workunit.client.1.smithi027.stdout:8/996: mknod d1/d3/d1c/d28/c15a 0 2024-04-22T15:50:16.561 INFO:tasks.workunit.client.1.smithi027.stdout:8/997: creat d1/d3/d1c/d28/f15b x:0 0 0 2024-04-22T15:50:16.567 INFO:tasks.workunit.client.1.smithi027.stdout:8/998: dwrite d1/d3/d1c/d32/d41/d52/fa1 [0,4194304] 0 2024-04-22T15:50:16.568 INFO:tasks.workunit.client.1.smithi027.stdout:8/999: dread - d1/d3/d1c/d49/d94/f10d zero size 2024-04-22T15:50:16.639 INFO:tasks.workunit.client.1.smithi027.stdout:1/923: sync 2024-04-22T15:50:16.639 INFO:tasks.workunit.client.1.smithi027.stdout:9/956: sync 2024-04-22T15:50:16.640 INFO:tasks.workunit.client.1.smithi027.stdout:1/924: write d2/db9/f11f [883830,56509] 0 2024-04-22T15:50:16.642 INFO:tasks.workunit.client.1.smithi027.stdout:1/925: write d2/f3 [4462946,126941] 0 2024-04-22T15:50:16.642 INFO:tasks.workunit.client.1.smithi027.stdout:9/957: creat d0/dc/d12/d30/d99/f143 x:0 0 0 2024-04-22T15:50:16.645 INFO:tasks.workunit.client.1.smithi027.stdout:9/958: creat d0/d51/d9d/f144 x:0 0 0 2024-04-22T15:50:16.647 INFO:tasks.workunit.client.1.smithi027.stdout:1/926: rename d2/d5f/la5 to d2/d5f/ddb/d107/d129/d81/d3d/d44/d9c/d9f/de9/d10f/d11c/l12c 0 2024-04-22T15:50:16.648 INFO:tasks.workunit.client.1.smithi027.stdout:9/959: rmdir d0/dc/d12/d30/d5b/dbc/df5/d104 39 2024-04-22T15:50:16.650 INFO:tasks.workunit.client.1.smithi027.stdout:1/927: mknod d2/d5f/ddb/d107/d129/d81/dcf/df2/d112/c12d 0 2024-04-22T15:50:16.650 INFO:tasks.workunit.client.1.smithi027.stdout:1/928: write d2/db9/f11f [1022725,91052] 0 2024-04-22T15:50:16.652 INFO:tasks.workunit.client.1.smithi027.stdout:9/960: creat d0/dc/d12/d30/d99/da7/d13a/f145 x:0 0 0 2024-04-22T15:50:16.671 INFO:tasks.workunit.client.1.smithi027.stdout:9/961: dwrite d0/dc/d12/d30/d5b/dbc/df5/d104/f59 [0,4194304] 0 2024-04-22T15:50:16.673 INFO:tasks.workunit.client.1.smithi027.stdout:9/962: read d0/dc/d12/d30/fea [3302371,18381] 0 2024-04-22T15:50:16.676 INFO:tasks.workunit.client.1.smithi027.stdout:9/963: fdatasync d0/dc/d12/fff 0 2024-04-22T15:50:16.682 INFO:tasks.workunit.client.1.smithi027.stdout:9/964: symlink d0/d51/d9d/l146 0 2024-04-22T15:50:16.840 INFO:tasks.workunit.client.1.smithi027.stdout:1/929: sync 2024-04-22T15:50:16.843 INFO:tasks.workunit.client.1.smithi027.stdout:1/930: rename d2/d30/f114 to d2/d5f/ddb/d107/d129/d81/d3d/d44/d9c/df8/d7d/f12e 0 2024-04-22T15:50:16.845 INFO:tasks.workunit.client.1.smithi027.stdout:1/931: mkdir d2/d5f/ddb/d107/d12f 0 2024-04-22T15:50:16.848 INFO:tasks.workunit.client.1.smithi027.stdout:1/932: creat d2/d5f/ddb/d107/d129/f130 x:0 0 0 2024-04-22T15:50:16.850 INFO:tasks.workunit.client.1.smithi027.stdout:1/933: truncate d2/d5f/fd1 804664 0 2024-04-22T15:50:16.855 INFO:tasks.workunit.client.1.smithi027.stdout:1/934: rename d2/d30/d115/fa4 to d2/d1d/d56/dc4/f131 0 2024-04-22T15:50:16.869 INFO:tasks.workunit.client.1.smithi027.stdout:1/935: dwrite d2/d1d/d56/f9a [0,4194304] 0 2024-04-22T15:50:16.874 INFO:tasks.workunit.client.1.smithi027.stdout:1/936: dread d2/d36/f62 [0,4194304] 0 2024-04-22T15:50:16.881 INFO:tasks.workunit.client.1.smithi027.stdout:1/937: link d2/d30/d105/f116 d2/d30/d105/f132 0 2024-04-22T15:50:16.888 INFO:tasks.workunit.client.1.smithi027.stdout:9/965: sync 2024-04-22T15:50:16.888 INFO:tasks.workunit.client.1.smithi027.stdout:1/938: dwrite d2/d5f/ddb/d107/f122 [0,4194304] 0 2024-04-22T15:50:16.890 INFO:tasks.workunit.client.1.smithi027.stdout:9/966: dread - d0/d51/de6/f12d zero size 2024-04-22T15:50:16.892 INFO:tasks.workunit.client.1.smithi027.stdout:9/967: write d0/dc/d12/d30/d5b/dbc/df5/d104/d6b/d6e/f61 [8546202,64] 0 2024-04-22T15:50:16.901 INFO:tasks.workunit.client.1.smithi027.stdout:1/939: fdatasync d2/d5f/ddb/d107/d129/d81/d3d/d44/d9c/d9f/fac 0 2024-04-22T15:50:16.913 INFO:tasks.workunit.client.1.smithi027.stdout:1/940: rmdir d2/d5f/ddb/d107/d129/d81/dcf/d110 0 2024-04-22T15:50:16.914 INFO:tasks.workunit.client.1.smithi027.stdout:9/968: getdents d0/dc/d12/d30/d5b/d91/da9 0 2024-04-22T15:50:16.915 INFO:tasks.workunit.client.1.smithi027.stdout:1/941: fsync d2/d36/daa/fd5 0 2024-04-22T15:50:16.920 INFO:tasks.workunit.client.1.smithi027.stdout:9/969: fdatasync d0/dc/d12/d30/f7a 0 2024-04-22T15:50:16.923 INFO:tasks.workunit.client.1.smithi027.stdout:1/942: dwrite d2/d5f/ddb/d107/d129/d81/da7/f67 [0,4194304] 0 2024-04-22T15:50:16.926 INFO:tasks.workunit.client.1.smithi027.stdout:9/970: chown d0/dc/d12/d30/d5b/l112 501 1 2024-04-22T15:50:16.929 INFO:tasks.workunit.client.1.smithi027.stdout:1/943: symlink d2/d5f/ddb/d107/d129/d81/d3d/d44/d9c/df8/l133 0 2024-04-22T15:50:16.930 INFO:tasks.workunit.client.1.smithi027.stdout:9/971: chown d0/dc/d12/d30/dd4/c113 605 1 2024-04-22T15:50:16.934 INFO:tasks.workunit.client.1.smithi027.stdout:9/972: creat d0/dc/d12/d30/d99/da7/f147 x:0 0 0 2024-04-22T15:50:16.935 INFO:tasks.workunit.client.1.smithi027.stdout:9/973: dread - d0/f90 zero size 2024-04-22T15:50:16.936 INFO:tasks.workunit.client.1.smithi027.stdout:1/944: unlink d2/d36/d74/f83 0 2024-04-22T15:50:16.940 INFO:tasks.workunit.client.1.smithi027.stdout:9/974: rename d0/dc/d12/d30/d5b/d91/da9/fc8 to d0/dc/d12/d30/d5b/dbc/df5/d104/d6b/d6e/f148 0 2024-04-22T15:50:16.942 INFO:tasks.workunit.client.1.smithi027.stdout:1/945: getdents d2/d5f/ddb/d107/d129/d10b 0 2024-04-22T15:50:16.944 INFO:tasks.workunit.client.1.smithi027.stdout:1/946: symlink d2/l134 0 2024-04-22T15:50:16.949 INFO:tasks.workunit.client.1.smithi027.stdout:9/975: getdents d0/dc/d12/d30/d5b/dbc/df5/d104/d6b 0 2024-04-22T15:50:16.953 INFO:tasks.workunit.client.1.smithi027.stdout:9/976: creat d0/dc/d12/d30/d99/db4/f149 x:0 0 0 2024-04-22T15:50:16.976 INFO:tasks.workunit.client.1.smithi027.stdout:9/977: dread d0/dc/d12/d30/d99/da7/fda [0,4194304] 0 2024-04-22T15:50:16.976 INFO:tasks.workunit.client.1.smithi027.stdout:9/978: dread - d0/f90 zero size 2024-04-22T15:50:16.990 INFO:tasks.workunit.client.1.smithi027.stdout:9/979: creat d0/d8/f14a x:0 0 0 2024-04-22T15:50:16.993 INFO:tasks.workunit.client.1.smithi027.stdout:1/947: chown d2/d30/l89 1 1 2024-04-22T15:50:16.998 INFO:tasks.workunit.client.1.smithi027.stdout:1/948: symlink d2/l135 0 2024-04-22T15:50:17.006 INFO:tasks.workunit.client.1.smithi027.stdout:9/980: dwrite d0/dc/d12/d30/f4f [0,4194304] 0 2024-04-22T15:50:17.007 INFO:tasks.workunit.client.1.smithi027.stdout:1/949: dwrite d2/d5f/ddb/d107/f111 [0,4194304] 0 2024-04-22T15:50:17.008 INFO:tasks.workunit.client.1.smithi027.stdout:9/981: read d0/dc/d12/fff [9110,55025] 0 2024-04-22T15:50:17.014 INFO:tasks.workunit.client.1.smithi027.stdout:1/950: dread - d2/d30/d105/f132 zero size 2024-04-22T15:50:17.016 INFO:tasks.workunit.client.1.smithi027.stdout:1/951: stat d2/d5f/ddb/d107/d129/d81/d3d/d44/f73 0 2024-04-22T15:50:17.016 INFO:tasks.workunit.client.1.smithi027.stdout:9/982: truncate d0/dc/d12/d30/d5b/dbc/df5/d104/ddb/ff2 643240 0 2024-04-22T15:50:17.020 INFO:tasks.workunit.client.1.smithi027.stdout:9/983: truncate d0/dc/d12/d30/f7a 3984439 0 2024-04-22T15:50:17.022 INFO:tasks.workunit.client.1.smithi027.stdout:1/952: creat d2/d5f/ddb/d107/d129/d81/d3d/d44/d9c/df8/f136 x:0 0 0 2024-04-22T15:50:17.024 INFO:tasks.workunit.client.1.smithi027.stdout:1/953: dread - d2/d5f/ddb/d107/d129/d81/d3d/d44/d9c/df8/f136 zero size 2024-04-22T15:50:17.026 INFO:tasks.workunit.client.1.smithi027.stdout:9/984: creat d0/dc/d100/f14b x:0 0 0 2024-04-22T15:50:17.027 INFO:tasks.workunit.client.1.smithi027.stdout:9/985: fsync d0/dc/d12/d30/d5b/d91/daf/f10d 0 2024-04-22T15:50:17.027 INFO:tasks.workunit.client.1.smithi027.stdout:1/954: mkdir d2/d5f/ddb/d137 0 2024-04-22T15:50:17.030 INFO:tasks.workunit.client.1.smithi027.stdout:9/986: mkdir d0/dc/d12/d30/d5b/d91/da9/dfe/d14c 0 2024-04-22T15:50:17.032 INFO:tasks.workunit.client.1.smithi027.stdout:1/955: dread d2/d5f/ddb/d107/d129/d81/d3d/d44/d9c/d9f/de9/fec [0,4194304] 0 2024-04-22T15:50:17.034 INFO:tasks.workunit.client.1.smithi027.stdout:9/987: mkdir d0/dc/d12/d30/d5b/dbc/df5/d104/d6b/d14d 0 2024-04-22T15:50:17.036 INFO:tasks.workunit.client.1.smithi027.stdout:1/956: truncate d2/d5f/ddb/d107/d129/d81/d3d/d82/dc2/f125 547025 0 2024-04-22T15:50:17.044 INFO:tasks.workunit.client.1.smithi027.stdout:1/957: dwrite d2/d5f/ddb/d107/d129/d81/d3d/fe6 [0,4194304] 0 2024-04-22T15:50:17.050 INFO:tasks.workunit.client.1.smithi027.stdout:1/958: mkdir d2/d5f/ddb/d107/d129/d138 0 2024-04-22T15:50:17.055 INFO:tasks.workunit.client.1.smithi027.stdout:1/959: rename d2/d5f/ddb/d107/d129/d81/d3d/d44/d9c/d9f/de9/fec to d2/d5f/ddb/d107/d129/d81/da7/f139 0 2024-04-22T15:50:17.058 INFO:tasks.workunit.client.1.smithi027.stdout:1/960: chown d2/d5f/ddb/d107/d129/d81/d3d/d44/d9c/df8/lba 218466 1 2024-04-22T15:50:17.061 INFO:tasks.workunit.client.1.smithi027.stdout:9/988: dwrite d0/d51/de6/f10c [0,4194304] 0 2024-04-22T15:50:17.064 INFO:tasks.workunit.client.1.smithi027.stdout:1/961: creat d2/d5f/f13a x:0 0 0 2024-04-22T15:50:17.067 INFO:tasks.workunit.client.1.smithi027.stdout:9/989: creat d0/dc/d12/d30/d5b/dbc/df5/d104/ddb/f14e x:0 0 0 2024-04-22T15:50:17.069 INFO:tasks.workunit.client.1.smithi027.stdout:9/990: dread - d0/dc/d12/d30/d5b/d91/daf/f10d zero size 2024-04-22T15:50:17.069 INFO:tasks.workunit.client.1.smithi027.stdout:1/962: rmdir d2/d5f/ddb/d107/d129/d81/d7e 39 2024-04-22T15:50:17.073 INFO:tasks.workunit.client.1.smithi027.stdout:9/991: creat d0/dc/d12/d30/d5b/dbc/f14f x:0 0 0 2024-04-22T15:50:17.074 INFO:tasks.workunit.client.1.smithi027.stdout:1/963: creat d2/d36/d74/f13b x:0 0 0 2024-04-22T15:50:17.075 INFO:tasks.workunit.client.1.smithi027.stdout:9/992: fsync d0/dc/d12/f94 0 2024-04-22T15:50:17.079 INFO:tasks.workunit.client.1.smithi027.stdout:1/964: read d2/d1d/f96 [78118,87326] 0 2024-04-22T15:50:17.085 INFO:tasks.workunit.client.1.smithi027.stdout:1/965: symlink d2/d5f/ddb/d107/d129/d81/l13c 0 2024-04-22T15:50:17.085 INFO:tasks.workunit.client.1.smithi027.stdout:9/993: write d0/d8b/fd9 [686541,14135] 0 2024-04-22T15:50:17.086 INFO:tasks.workunit.client.1.smithi027.stdout:1/966: chown d2/d5f/ddb/d107/d129/d81/da7/ffe 220959221 1 2024-04-22T15:50:17.088 INFO:tasks.workunit.client.1.smithi027.stdout:1/967: dread - d2/d5f/ddb/d107/d129/d81/d3d/fcc zero size 2024-04-22T15:50:17.094 INFO:tasks.workunit.client.1.smithi027.stdout:1/968: truncate d2/d5f/ddb/d107/d129/d81/d3d/d44/d9c/df8/d7d/f12e 934839 0 2024-04-22T15:50:17.095 INFO:tasks.workunit.client.1.smithi027.stdout:1/969: mkdir d2/d5f/ddb/d107/d13d 0 2024-04-22T15:50:17.096 INFO:tasks.workunit.client.1.smithi027.stdout:9/994: dread d0/dc/d12/d30/d99/da7/f124 [0,4194304] 0 2024-04-22T15:50:17.098 INFO:tasks.workunit.client.1.smithi027.stdout:9/995: truncate d0/dc/d12/d30/d99/da7/f147 449770 0 2024-04-22T15:50:17.111 INFO:tasks.workunit.client.1.smithi027.stdout:9/996: dread d0/dc/de/f40 [0,4194304] 0 2024-04-22T15:50:17.115 INFO:tasks.workunit.client.1.smithi027.stdout:1/970: truncate d2/d5f/ddb/d107/d129/d81/dcf/df2/d112/f113 1344802 0 2024-04-22T15:50:17.116 INFO:tasks.workunit.client.1.smithi027.stdout:9/997: write d0/d64/f7c [853797,8035] 0 2024-04-22T15:50:17.119 INFO:tasks.workunit.client.1.smithi027.stdout:1/971: creat d2/d5f/ddb/d107/d129/d81/d3d/d44/d9c/f13e x:0 0 0 2024-04-22T15:50:17.120 INFO:tasks.workunit.client.1.smithi027.stdout:1/972: chown d2/d5f/ddb/d107/d129/d81/da7/l6a 408 1 2024-04-22T15:50:17.121 INFO:tasks.workunit.client.1.smithi027.stdout:9/998: creat d0/dc/d12/d30/d5b/dbc/df5/f150 x:0 0 0 2024-04-22T15:50:17.126 INFO:tasks.workunit.client.1.smithi027.stdout:1/973: dread d2/d5f/ddb/d107/d129/d81/d3d/d44/d9c/d9f/fac [0,4194304] 0 2024-04-22T15:50:17.127 INFO:tasks.workunit.client.1.smithi027.stdout:1/974: readlink d2/d5f/ddb/d107/d129/d81/da7/l6a 0 2024-04-22T15:50:17.133 INFO:tasks.workunit.client.1.smithi027.stdout:1/975: dwrite d2/d5f/ddb/d107/d129/d81/d3d/d44/d9c/fe1 [4194304,4194304] 0 2024-04-22T15:50:17.138 INFO:tasks.workunit.client.1.smithi027.stdout:9/999: write d0/dc/d12/d30/d5b/dbc/df5/d104/da6/fae [1175050,8174] 0 2024-04-22T15:50:17.144 INFO:tasks.workunit.client.1.smithi027.stdout:1/976: rmdir d2/d5f/ddb/d107/d13d 0 2024-04-22T15:50:17.148 INFO:tasks.workunit.client.1.smithi027.stdout:1/977: rename d2/d1d/d56/dc4/fc8 to d2/d5f/ddb/d107/d129/d81/d3d/d44/d9c/d9f/de9/f13f 0 2024-04-22T15:50:17.154 INFO:tasks.workunit.client.1.smithi027.stdout:1/978: dwrite d2/f7f [0,4194304] 0 2024-04-22T15:50:17.159 INFO:tasks.workunit.client.1.smithi027.stdout:1/979: dread d2/d5f/ddb/d107/d129/d81/d7e/fcd [0,4194304] 0 2024-04-22T15:50:17.164 INFO:tasks.workunit.client.1.smithi027.stdout:1/980: mknod d2/d5f/ddb/d107/d12f/c140 0 2024-04-22T15:50:17.176 INFO:tasks.workunit.client.1.smithi027.stdout:1/981: truncate d2/d1d/f123 2107227 0 2024-04-22T15:50:17.177 INFO:tasks.workunit.client.1.smithi027.stdout:1/982: chown d2/d5f/ddb/d107/d129/d81/d3d/d44/fb2 5554 1 2024-04-22T15:50:17.183 INFO:tasks.workunit.client.1.smithi027.stdout:1/983: link d2/f75 d2/d5f/ddb/d107/d129/d81/d3d/d82/f141 0 2024-04-22T15:50:17.185 INFO:tasks.workunit.client.1.smithi027.stdout:1/984: unlink d2/d5f/ddb/d107/d129/d81/dcf/df2/d112/f11a 0 2024-04-22T15:50:17.188 INFO:tasks.workunit.client.1.smithi027.stdout:1/985: symlink d2/d1d/d56/d109/l142 0 2024-04-22T15:50:17.190 INFO:tasks.workunit.client.1.smithi027.stdout:1/986: creat d2/d5f/ddb/d107/d129/d81/d3d/d44/d9c/d9f/f143 x:0 0 0 2024-04-22T15:50:17.193 INFO:tasks.workunit.client.1.smithi027.stdout:1/987: rename d2/d5f/ddb/d107/d129/d10b to d2/d36/d74/d144 0 2024-04-22T15:50:17.221 INFO:tasks.workunit.client.1.smithi027.stdout:1/988: dwrite d2/d1d/f117 [0,4194304] 0 2024-04-22T15:50:17.225 INFO:tasks.workunit.client.1.smithi027.stdout:1/989: mknod d2/d5f/ddb/d137/c145 0 2024-04-22T15:50:17.228 INFO:tasks.workunit.client.1.smithi027.stdout:1/990: mknod d2/d5f/ddb/d107/d129/d81/dcf/c146 0 2024-04-22T15:50:17.240 INFO:tasks.workunit.client.1.smithi027.stdout:1/991: truncate d2/d5f/ddb/d107/d129/d81/d3d/d44/d9c/df8/fd4 384578 0 2024-04-22T15:50:17.243 INFO:tasks.workunit.client.1.smithi027.stdout:1/992: mkdir d2/d5f/ddb/d107/d129/d81/d3d/d44/d147 0 2024-04-22T15:50:17.245 INFO:tasks.workunit.client.1.smithi027.stdout:1/993: mknod d2/d1d/d56/dc0/c148 0 2024-04-22T15:50:17.252 INFO:tasks.workunit.client.1.smithi027.stdout:1/994: dwrite d2/d5f/ddb/d107/d129/d81/d3d/d44/f73 [0,4194304] 0 2024-04-22T15:50:17.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:17 smithi027 ceph-mon[34922]: Reconfiguring daemon prometheus.smithi016 on smithi016 2024-04-22T15:50:17.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:17 smithi027 ceph-mon[34922]: pgmap v12: 65 pgs: 65 active+clean; 2.8 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 70 MiB/s rd, 152 MiB/s wr, 418 op/s 2024-04-22T15:50:17.407 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:17 smithi016 ceph-mon[26620]: Reconfiguring daemon prometheus.smithi016 on smithi016 2024-04-22T15:50:17.407 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:17 smithi016 ceph-mon[26620]: pgmap v12: 65 pgs: 65 active+clean; 2.8 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 70 MiB/s rd, 152 MiB/s wr, 418 op/s 2024-04-22T15:50:17.666 INFO:tasks.workunit.client.1.smithi027.stdout:1/995: sync 2024-04-22T15:50:17.669 INFO:tasks.workunit.client.1.smithi027.stdout:1/996: dread d2/d5f/ddb/d107/f111 [0,4194304] 0 2024-04-22T15:50:17.682 INFO:tasks.workunit.client.1.smithi027.stdout:1/997: dwrite d2/f68 [4194304,4194304] 0 2024-04-22T15:50:17.687 INFO:tasks.workunit.client.1.smithi027.stdout:1/998: truncate d2/d5f/ddb/d107/d129/d81/d3d/d82/f141 152665 0 2024-04-22T15:50:17.899 INFO:tasks.workunit.client.1.smithi027.stdout:1/999: sync 2024-04-22T15:50:17.904 INFO:tasks.workunit.client.1.smithi027.stderr:+ rm -rf -- ./tmp.RI33Yxg0i3 2024-04-22T15:50:18.677 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-22T15:50:18.677 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-22T15:50:19.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:19 smithi027 ceph-mon[34922]: pgmap v13: 65 pgs: 65 active+clean; 2.8 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 48 MiB/s rd, 106 MiB/s wr, 277 op/s 2024-04-22T15:50:19.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:19 smithi016 ceph-mon[26620]: pgmap v13: 65 pgs: 65 active+clean; 2.8 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 48 MiB/s rd, 106 MiB/s wr, 277 op/s 2024-04-22T15:50:21.330 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:21 smithi016 ceph-mon[26620]: pgmap v14: 65 pgs: 65 active+clean; 2.5 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 54 MiB/s rd, 127 MiB/s wr, 365 op/s 2024-04-22T15:50:21.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:21 smithi027 ceph-mon[34922]: pgmap v14: 65 pgs: 65 active+clean; 2.5 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 54 MiB/s rd, 127 MiB/s wr, 365 op/s 2024-04-22T15:50:23.507 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:23 smithi016 ceph-mon[26620]: pgmap v15: 65 pgs: 65 active+clean; 2.5 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 30 MiB/s rd, 76 MiB/s wr, 230 op/s 2024-04-22T15:50:23.507 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:23 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:23.507 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:23 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:23.507 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:23 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:50:23.507 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:23 smithi016 ceph-mon[26620]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:50:23.507 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:23 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi016:9095"}]: dispatch 2024-04-22T15:50:23.507 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:23 smithi016 ceph-mon[26620]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi016:9095"}]: dispatch 2024-04-22T15:50:23.507 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:23 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:23.507 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:23 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:50:23.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:23 smithi027 ceph-mon[34922]: pgmap v15: 65 pgs: 65 active+clean; 2.5 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 30 MiB/s rd, 76 MiB/s wr, 230 op/s 2024-04-22T15:50:23.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:23 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:23.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:23 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:23.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:23 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:50:23.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:23 smithi027 ceph-mon[34922]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:50:23.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:23 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi016:9095"}]: dispatch 2024-04-22T15:50:23.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:23 smithi027 ceph-mon[34922]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi016:9095"}]: dispatch 2024-04-22T15:50:23.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:23 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:23.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:23 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:50:25.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:25 smithi027 ceph-mon[34922]: pgmap v16: 65 pgs: 65 active+clean; 2.5 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 30 MiB/s rd, 76 MiB/s wr, 230 op/s 2024-04-22T15:50:25.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:25 smithi027 ceph-mon[34922]: Upgrade: Updating mgr.smithi027.ukdgzs 2024-04-22T15:50:25.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:25 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi027.ukdgzs", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:50:25.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:25 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:50:25.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:25 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:50:25.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:25 smithi027 ceph-mon[34922]: Deploying daemon mgr.smithi027.ukdgzs on smithi027 2024-04-22T15:50:25.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:25 smithi016 ceph-mon[26620]: pgmap v16: 65 pgs: 65 active+clean; 2.5 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 30 MiB/s rd, 76 MiB/s wr, 230 op/s 2024-04-22T15:50:25.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:25 smithi016 ceph-mon[26620]: Upgrade: Updating mgr.smithi027.ukdgzs 2024-04-22T15:50:25.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:25 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi027.ukdgzs", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:50:25.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:25 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:50:25.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:25 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:50:25.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:25 smithi016 ceph-mon[26620]: Deploying daemon mgr.smithi027.ukdgzs on smithi027 2024-04-22T15:50:26.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:26 smithi027 ceph-mon[34922]: pgmap v17: 65 pgs: 65 active+clean; 1.8 GiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 31 MiB/s rd, 76 MiB/s wr, 307 op/s 2024-04-22T15:50:26.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:26 smithi016 ceph-mon[26620]: pgmap v17: 65 pgs: 65 active+clean; 1.8 GiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 31 MiB/s rd, 76 MiB/s wr, 307 op/s 2024-04-22T15:50:28.997 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:28 smithi027 ceph-mon[34922]: pgmap v18: 65 pgs: 65 active+clean; 1.8 GiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 6.8 MiB/s rd, 22 MiB/s wr, 164 op/s 2024-04-22T15:50:28.998 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:28 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:50:28.998 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:28 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:28.998 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:28 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:28.998 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:28 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:50:29.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:28 smithi016 ceph-mon[26620]: pgmap v18: 65 pgs: 65 active+clean; 1.8 GiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 6.8 MiB/s rd, 22 MiB/s wr, 164 op/s 2024-04-22T15:50:29.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:28 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:50:29.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:28 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:29.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:28 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:29.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:28 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:50:30.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:30 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:30.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:30 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:30.799 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:30 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:30.799 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:30 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:31.650 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:31 smithi027 ceph-mon[34922]: pgmap v19: 65 pgs: 65 active+clean; 1.3 GiB data, 4.3 GiB used, 532 GiB / 536 GiB avail; 7.4 MiB/s rd, 23 MiB/s wr, 227 op/s 2024-04-22T15:50:31.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:31 smithi016 ceph-mon[26620]: pgmap v19: 65 pgs: 65 active+clean; 1.3 GiB data, 4.3 GiB used, 532 GiB / 536 GiB avail; 7.4 MiB/s rd, 23 MiB/s wr, 227 op/s 2024-04-22T15:50:33.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:33 smithi016 ceph-mon[26620]: pgmap v20: 65 pgs: 65 active+clean; 1.3 GiB data, 4.3 GiB used, 532 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 139 op/s 2024-04-22T15:50:33.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:33 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:33.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:33 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:33.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:33 smithi027 ceph-mon[34922]: pgmap v20: 65 pgs: 65 active+clean; 1.3 GiB data, 4.3 GiB used, 532 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 139 op/s 2024-04-22T15:50:33.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:33 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:33.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:33 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:34.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.321+0000 7f7efda16700 1 -- 172.21.15.16:0/1042959625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef80fed20 msgr2=0x7f7ef80ff1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:34.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.321+0000 7f7efda16700 1 --2- 172.21.15.16:0/1042959625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef80fed20 0x7f7ef80ff1a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7eec009910 tx=0x7f7eec009c20 comp rx=0 tx=0).stop 2024-04-22T15:50:34.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.321+0000 7f7efda16700 1 -- 172.21.15.16:0/1042959625 shutdown_connections 2024-04-22T15:50:34.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.321+0000 7f7efda16700 1 --2- 172.21.15.16:0/1042959625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef80fed20 0x7f7ef80ff1a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:34.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.321+0000 7f7efda16700 1 --2- 172.21.15.16:0/1042959625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ef81022a0 0x7f7ef81026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:34.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.321+0000 7f7efda16700 1 -- 172.21.15.16:0/1042959625 >> 172.21.15.16:0/1042959625 conn(0x7f7ef80fa8c0 msgr2=0x7f7ef80fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:34.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.322+0000 7f7efda16700 1 -- 172.21.15.16:0/1042959625 shutdown_connections 2024-04-22T15:50:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.322+0000 7f7efda16700 1 -- 172.21.15.16:0/1042959625 wait complete. 2024-04-22T15:50:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.322+0000 7f7efda16700 1 Processor -- start 2024-04-22T15:50:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.322+0000 7f7efda16700 1 -- start start 2024-04-22T15:50:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.323+0000 7f7efda16700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ef80fed20 0x7f7ef81964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.323+0000 7f7efda16700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef81022a0 0x7f7ef8196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.323+0000 7f7efda16700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef8197030 con 0x7f7ef81022a0 2024-04-22T15:50:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.323+0000 7f7efda16700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef8197170 con 0x7f7ef80fed20 2024-04-22T15:50:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.323+0000 7f7ef67fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef81022a0 0x7f7ef8196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.323+0000 7f7ef67fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef81022a0 0x7f7ef8196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48164/0 (socket says 172.21.15.16:48164) 2024-04-22T15:50:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.323+0000 7f7ef67fc700 1 -- 172.21.15.16:0/4189534659 learned_addr learned my addr 172.21.15.16:0/4189534659 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:50:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.323+0000 7f7ef6ffd700 1 --2- 172.21.15.16:0/4189534659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ef80fed20 0x7f7ef81964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.323+0000 7f7ef67fc700 1 -- 172.21.15.16:0/4189534659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ef80fed20 msgr2=0x7f7ef81964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.323+0000 7f7ef67fc700 1 --2- 172.21.15.16:0/4189534659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ef80fed20 0x7f7ef81964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.323+0000 7f7ef67fc700 1 -- 172.21.15.16:0/4189534659 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eec0095f0 con 0x7f7ef81022a0 2024-04-22T15:50:34.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.323+0000 7f7ef67fc700 1 --2- 172.21.15.16:0/4189534659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef81022a0 0x7f7ef8196a10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7eec000c00 tx=0x7f7eec00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:34.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.323+0000 7f7efca14700 1 -- 172.21.15.16:0/4189534659 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7eec01d070 con 0x7f7ef81022a0 2024-04-22T15:50:34.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.323+0000 7f7efda16700 1 -- 172.21.15.16:0/4189534659 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef819bbc0 con 0x7f7ef81022a0 2024-04-22T15:50:34.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.323+0000 7f7efca14700 1 -- 172.21.15.16:0/4189534659 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7eec005dc0 con 0x7f7ef81022a0 2024-04-22T15:50:34.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.323+0000 7f7efca14700 1 -- 172.21.15.16:0/4189534659 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7eec022470 con 0x7f7ef81022a0 2024-04-22T15:50:34.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.323+0000 7f7efda16700 1 -- 172.21.15.16:0/4189534659 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef819c0b0 con 0x7f7ef81022a0 2024-04-22T15:50:34.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.324+0000 7f7efda16700 1 -- 172.21.15.16:0/4189534659 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ef8109820 con 0x7f7ef81022a0 2024-04-22T15:50:34.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.325+0000 7f7efca14700 1 -- 172.21.15.16:0/4189534659 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 30) v1 ==== 92700+0+0 (secure 0 0 0) 0x7f7eec003810 con 0x7f7ef81022a0 2024-04-22T15:50:34.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.326+0000 7f7efca14700 1 --2- 172.21.15.16:0/4189534659 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f7ee006ea80 0x7f7ee0070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:34.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.326+0000 7f7efca14700 1 -- 172.21.15.16:0/4189534659 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f7eec090050 con 0x7f7ef81022a0 2024-04-22T15:50:34.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.326+0000 7f7ef6ffd700 1 --2- 172.21.15.16:0/4189534659 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f7ee006ea80 0x7f7ee0070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:34.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.326+0000 7f7ef6ffd700 1 --2- 172.21.15.16:0/4189534659 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f7ee006ea80 0x7f7ee0070f40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8068b10 tx=0x7f7ee8009260 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:34.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.332+0000 7f7efca14700 1 -- 172.21.15.16:0/4189534659 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f7eec05cc40 con 0x7f7ef81022a0 2024-04-22T15:50:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.646+0000 7f7efda16700 1 -- 172.21.15.16:0/4189534659 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ef8106990 con 0x7f7ee006ea80 2024-04-22T15:50:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.647+0000 7f7efca14700 1 -- 172.21.15.16:0/4189534659 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f7ef8106990 con 0x7f7ee006ea80 2024-04-22T15:50:34.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.649+0000 7f7efda16700 1 -- 172.21.15.16:0/4189534659 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f7ee006ea80 msgr2=0x7f7ee0070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:34.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.649+0000 7f7efda16700 1 --2- 172.21.15.16:0/4189534659 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f7ee006ea80 0x7f7ee0070f40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8068b10 tx=0x7f7ee8009260 comp rx=0 tx=0).stop 2024-04-22T15:50:34.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.649+0000 7f7efda16700 1 -- 172.21.15.16:0/4189534659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef81022a0 msgr2=0x7f7ef8196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:34.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.649+0000 7f7efda16700 1 --2- 172.21.15.16:0/4189534659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef81022a0 0x7f7ef8196a10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7eec000c00 tx=0x7f7eec00f690 comp rx=0 tx=0).stop 2024-04-22T15:50:34.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.649+0000 7f7efda16700 1 -- 172.21.15.16:0/4189534659 shutdown_connections 2024-04-22T15:50:34.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.649+0000 7f7efda16700 1 --2- 172.21.15.16:0/4189534659 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f7ee006ea80 0x7f7ee0070f40 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:34.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.649+0000 7f7efda16700 1 --2- 172.21.15.16:0/4189534659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ef80fed20 0x7f7ef81964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:34.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.649+0000 7f7efda16700 1 --2- 172.21.15.16:0/4189534659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef81022a0 0x7f7ef8196a10 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:34.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.649+0000 7f7efda16700 1 -- 172.21.15.16:0/4189534659 >> 172.21.15.16:0/4189534659 conn(0x7f7ef80fa8c0 msgr2=0x7f7ef8105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:34.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.649+0000 7f7efda16700 1 -- 172.21.15.16:0/4189534659 shutdown_connections 2024-04-22T15:50:34.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.649+0000 7f7efda16700 1 -- 172.21.15.16:0/4189534659 wait complete. 2024-04-22T15:50:34.663 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:50:34.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.763+0000 7f56e99bc700 1 -- 172.21.15.16:0/1157092355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56e4073770 msgr2=0x7f56e4073bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:34.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.763+0000 7f56e99bc700 1 --2- 172.21.15.16:0/1157092355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56e4073770 0x7f56e4073bf0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f56d8009910 tx=0x7f56d8009c20 comp rx=0 tx=0).stop 2024-04-22T15:50:34.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.763+0000 7f56e99bc700 1 -- 172.21.15.16:0/1157092355 shutdown_connections 2024-04-22T15:50:34.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.763+0000 7f56e99bc700 1 --2- 172.21.15.16:0/1157092355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56e4073770 0x7f56e4073bf0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:34.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.763+0000 7f56e99bc700 1 --2- 172.21.15.16:0/1157092355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56e4074d40 0x7f56e40731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:34.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.763+0000 7f56e99bc700 1 -- 172.21.15.16:0/1157092355 >> 172.21.15.16:0/1157092355 conn(0x7f56e40fc260 msgr2=0x7f56e40fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:34.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.764+0000 7f56e99bc700 1 -- 172.21.15.16:0/1157092355 shutdown_connections 2024-04-22T15:50:34.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.764+0000 7f56e99bc700 1 -- 172.21.15.16:0/1157092355 wait complete. 2024-04-22T15:50:34.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.764+0000 7f56e99bc700 1 Processor -- start 2024-04-22T15:50:34.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.765+0000 7f56e99bc700 1 -- start start 2024-04-22T15:50:34.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.765+0000 7f56e99bc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56e4073770 0x7f56e4198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:34.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.765+0000 7f56e99bc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56e4074d40 0x7f56e4198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:34.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.765+0000 7f56e99bc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e41991f0 con 0x7f56e4073770 2024-04-22T15:50:34.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.765+0000 7f56e99bc700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e4199330 con 0x7f56e4074d40 2024-04-22T15:50:34.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.765+0000 7f56e2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56e4073770 0x7f56e4198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:34.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.765+0000 7f56e2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56e4073770 0x7f56e4198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48194/0 (socket says 172.21.15.16:48194) 2024-04-22T15:50:34.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.765+0000 7f56e2ffd700 1 -- 172.21.15.16:0/280698142 learned_addr learned my addr 172.21.15.16:0/280698142 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:50:34.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.765+0000 7f56e27fc700 1 --2- 172.21.15.16:0/280698142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56e4074d40 0x7f56e4198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:34.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.766+0000 7f56e2ffd700 1 -- 172.21.15.16:0/280698142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56e4074d40 msgr2=0x7f56e4198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:34.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.766+0000 7f56e2ffd700 1 --2- 172.21.15.16:0/280698142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56e4074d40 0x7f56e4198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:34.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.766+0000 7f56e2ffd700 1 -- 172.21.15.16:0/280698142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d80095f0 con 0x7f56e4073770 2024-04-22T15:50:34.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.766+0000 7f56e27fc700 1 --2- 172.21.15.16:0/280698142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56e4074d40 0x7f56e4198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:50:34.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.766+0000 7f56e2ffd700 1 --2- 172.21.15.16:0/280698142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56e4073770 0x7f56e4198690 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f56d400e920 tx=0x7f56d400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:34.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.766+0000 7f56e89ba700 1 -- 172.21.15.16:0/280698142 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56d400c9d0 con 0x7f56e4073770 2024-04-22T15:50:34.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.766+0000 7f56e99bc700 1 -- 172.21.15.16:0/280698142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e419dde0 con 0x7f56e4073770 2024-04-22T15:50:34.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.766+0000 7f56e89ba700 1 -- 172.21.15.16:0/280698142 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f56d4010430 con 0x7f56e4073770 2024-04-22T15:50:34.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.767+0000 7f56e89ba700 1 -- 172.21.15.16:0/280698142 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56d4018650 con 0x7f56e4073770 2024-04-22T15:50:34.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.767+0000 7f56e99bc700 1 -- 172.21.15.16:0/280698142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e419e260 con 0x7f56e4073770 2024-04-22T15:50:34.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.768+0000 7f56e89ba700 1 -- 172.21.15.16:0/280698142 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 30) v1 ==== 92700+0+0 (secure 0 0 0) 0x7f56d40105a0 con 0x7f56e4073770 2024-04-22T15:50:34.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.768+0000 7f56e99bc700 1 -- 172.21.15.16:0/280698142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e410b9f0 con 0x7f56e4073770 2024-04-22T15:50:34.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.769+0000 7f56e89ba700 1 --2- 172.21.15.16:0/280698142 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f56cc06ea80 0x7f56cc070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:34.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.769+0000 7f56e89ba700 1 -- 172.21.15.16:0/280698142 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f56d4014070 con 0x7f56e4073770 2024-04-22T15:50:34.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.769+0000 7f56e27fc700 1 --2- 172.21.15.16:0/280698142 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f56cc06ea80 0x7f56cc070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:34.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.770+0000 7f56e27fc700 1 --2- 172.21.15.16:0/280698142 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f56cc06ea80 0x7f56cc070f40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f56d8009fd0 tx=0x7f56d8005fd0 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:34.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:34.774+0000 7f56e89ba700 1 -- 172.21.15.16:0/280698142 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f56d4053e20 con 0x7f56e4073770 2024-04-22T15:50:35.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.032+0000 7f56e99bc700 1 -- 172.21.15.16:0/280698142 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f56e4108220 con 0x7f56cc06ea80 2024-04-22T15:50:35.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.034+0000 7f56e89ba700 1 -- 172.21.15.16:0/280698142 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f56e4108220 con 0x7f56cc06ea80 2024-04-22T15:50:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.037+0000 7f56e99bc700 1 -- 172.21.15.16:0/280698142 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f56cc06ea80 msgr2=0x7f56cc070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.037+0000 7f56e99bc700 1 --2- 172.21.15.16:0/280698142 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f56cc06ea80 0x7f56cc070f40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f56d8009fd0 tx=0x7f56d8005fd0 comp rx=0 tx=0).stop 2024-04-22T15:50:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.037+0000 7f56e99bc700 1 -- 172.21.15.16:0/280698142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56e4073770 msgr2=0x7f56e4198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.037+0000 7f56e99bc700 1 --2- 172.21.15.16:0/280698142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56e4073770 0x7f56e4198690 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f56d400e920 tx=0x7f56d400ec30 comp rx=0 tx=0).stop 2024-04-22T15:50:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.037+0000 7f56e99bc700 1 -- 172.21.15.16:0/280698142 shutdown_connections 2024-04-22T15:50:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.037+0000 7f56e99bc700 1 --2- 172.21.15.16:0/280698142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56e4073770 0x7f56e4198690 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.037+0000 7f56e99bc700 1 --2- 172.21.15.16:0/280698142 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f56cc06ea80 0x7f56cc070f40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.037+0000 7f56e99bc700 1 --2- 172.21.15.16:0/280698142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56e4074d40 0x7f56e4198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.037+0000 7f56e99bc700 1 -- 172.21.15.16:0/280698142 >> 172.21.15.16:0/280698142 conn(0x7f56e40fc260 msgr2=0x7f56e4106b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.037+0000 7f56e99bc700 1 -- 172.21.15.16:0/280698142 shutdown_connections 2024-04-22T15:50:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.037+0000 7f56e99bc700 1 -- 172.21.15.16:0/280698142 wait complete. 2024-04-22T15:50:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.159+0000 7f3a53479700 1 -- 172.21.15.16:0/2526904883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a4c104150 msgr2=0x7f3a4c106540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.159+0000 7f3a53479700 1 --2- 172.21.15.16:0/2526904883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a4c104150 0x7f3a4c106540 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c009910 tx=0x7f3a3c009c20 comp rx=0 tx=0).stop 2024-04-22T15:50:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.159+0000 7f3a53479700 1 -- 172.21.15.16:0/2526904883 shutdown_connections 2024-04-22T15:50:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.159+0000 7f3a53479700 1 --2- 172.21.15.16:0/2526904883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a4c104150 0x7f3a4c106540 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.159+0000 7f3a53479700 1 --2- 172.21.15.16:0/2526904883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a4c101820 0x7f3a4c103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.159+0000 7f3a53479700 1 -- 172.21.15.16:0/2526904883 >> 172.21.15.16:0/2526904883 conn(0x7f3a4c0fb190 msgr2=0x7f3a4c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.159+0000 7f3a53479700 1 -- 172.21.15.16:0/2526904883 shutdown_connections 2024-04-22T15:50:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.159+0000 7f3a53479700 1 -- 172.21.15.16:0/2526904883 wait complete. 2024-04-22T15:50:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.159+0000 7f3a53479700 1 Processor -- start 2024-04-22T15:50:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.160+0000 7f3a53479700 1 -- start start 2024-04-22T15:50:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.160+0000 7f3a53479700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a4c101820 0x7f3a4c196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.160+0000 7f3a53479700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a4c104150 0x7f3a4c196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.160+0000 7f3a53479700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c196fb0 con 0x7f3a4c101820 2024-04-22T15:50:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.160+0000 7f3a53479700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c1970f0 con 0x7f3a4c104150 2024-04-22T15:50:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.160+0000 7f3a51215700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a4c101820 0x7f3a4c196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.160+0000 7f3a51215700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a4c101820 0x7f3a4c196450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48220/0 (socket says 172.21.15.16:48220) 2024-04-22T15:50:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.160+0000 7f3a51215700 1 -- 172.21.15.16:0/2069657119 learned_addr learned my addr 172.21.15.16:0/2069657119 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:50:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.160+0000 7f3a50a14700 1 --2- 172.21.15.16:0/2069657119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a4c104150 0x7f3a4c196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.160+0000 7f3a51215700 1 -- 172.21.15.16:0/2069657119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a4c104150 msgr2=0x7f3a4c196990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.160+0000 7f3a51215700 1 --2- 172.21.15.16:0/2069657119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a4c104150 0x7f3a4c196990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.160+0000 7f3a51215700 1 -- 172.21.15.16:0/2069657119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a3c0095f0 con 0x7f3a4c101820 2024-04-22T15:50:35.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.160+0000 7f3a51215700 1 --2- 172.21.15.16:0/2069657119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a4c101820 0x7f3a4c196450 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800d8d0 tx=0x7f3a4800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:35.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.161+0000 7f3a427fc700 1 -- 172.21.15.16:0/2069657119 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a480098a0 con 0x7f3a4c101820 2024-04-22T15:50:35.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.161+0000 7f3a53479700 1 -- 172.21.15.16:0/2069657119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a4c19bba0 con 0x7f3a4c101820 2024-04-22T15:50:35.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.161+0000 7f3a427fc700 1 -- 172.21.15.16:0/2069657119 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3a48010460 con 0x7f3a4c101820 2024-04-22T15:50:35.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.161+0000 7f3a427fc700 1 -- 172.21.15.16:0/2069657119 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a4800f5f0 con 0x7f3a4c101820 2024-04-22T15:50:35.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.161+0000 7f3a53479700 1 -- 172.21.15.16:0/2069657119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a4c19c0f0 con 0x7f3a4c101820 2024-04-22T15:50:35.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.162+0000 7f3a53479700 1 -- 172.21.15.16:0/2069657119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a4c190610 con 0x7f3a4c101820 2024-04-22T15:50:35.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.162+0000 7f3a427fc700 1 -- 172.21.15.16:0/2069657119 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 30) v1 ==== 92700+0+0 (secure 0 0 0) 0x7f3a48009a00 con 0x7f3a4c101820 2024-04-22T15:50:35.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.163+0000 7f3a427fc700 1 --2- 172.21.15.16:0/2069657119 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f3a3806ea80 0x7f3a38070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:35.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.163+0000 7f3a427fc700 1 -- 172.21.15.16:0/2069657119 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f3a4808e9c0 con 0x7f3a4c101820 2024-04-22T15:50:35.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.163+0000 7f3a50a14700 1 --2- 172.21.15.16:0/2069657119 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f3a3806ea80 0x7f3a38070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:35.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.163+0000 7f3a50a14700 1 --2- 172.21.15.16:0/2069657119 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f3a3806ea80 0x7f3a38070f40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c006010 tx=0x7f3a3c00b540 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:35.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.167+0000 7f3a427fc700 1 -- 172.21.15.16:0/2069657119 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f3a4805b5b0 con 0x7f3a4c101820 2024-04-22T15:50:35.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:35 smithi016 ceph-mon[26620]: pgmap v21: 65 pgs: 65 active+clean; 1.3 GiB data, 4.3 GiB used, 532 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 139 op/s 2024-04-22T15:50:35.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:35 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:35.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:35 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:35.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:35 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:35.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:35 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.449+0000 7f3a53479700 1 -- 172.21.15.16:0/2069657119 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a4c0611d0 con 0x7f3a3806ea80 2024-04-22T15:50:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.461+0000 7f3a427fc700 1 -- 172.21.15.16:0/2069657119 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f3a4c0611d0 con 0x7f3a3806ea80 2024-04-22T15:50:35.464 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:50:35.464 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.smithi016 smithi016 *:9093,9094 running (8m) 0s ago 9m 24.9M - 0.25.0 c8568f914cd2 fab40987f1b0 2024-04-22T15:50:35.464 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi016 smithi016 running (9m) 0s ago 9m 7746k - 18.2.1 5be31c24972a f8b627d3f41e 2024-04-22T15:50:35.464 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi027 smithi027 running (8m) 2s ago 8m 7734k - 18.2.1 5be31c24972a f00fd40b4220 2024-04-22T15:50:35.464 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi016 smithi016 running (9m) 0s ago 9m 6656k - 18.2.1 5be31c24972a 6b93db1fa0d9 2024-04-22T15:50:35.464 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi027 smithi027 running (8m) 2s ago 8m 6665k - 18.2.1 5be31c24972a d3bd439f30a0 2024-04-22T15:50:35.464 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.smithi016 smithi016 *:3000 running (7m) 0s ago 9m 84.7M - 9.4.7 954c08fa6188 a85bdbf5073d 2024-04-22T15:50:35.465 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.fkkejm smithi016 running (4m) 0s ago 4m 13.9M - 18.2.1 5be31c24972a 4153ac94cea0 2024-04-22T15:50:35.465 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.pgvphu smithi016 running (3m) 0s ago 3m 294M - 18.2.1 5be31c24972a 93dc072068a2 2024-04-22T15:50:35.465 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.dgqyes smithi027 running (3m) 2s ago 3m 17.0M - 18.2.1 5be31c24972a 41a9220430fd 2024-04-22T15:50:35.465 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.iobjds smithi027 running (4m) 2s ago 3m 248M - 18.2.1 5be31c24972a c1c8d6770f02 2024-04-22T15:50:35.465 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi016.rnsnnr smithi016 *:8443,9283,8765 running (58s) 0s ago 11m 499M - 19.0.0-3223-g5883d494 3d983c26e278 09b4fe285bc4 2024-04-22T15:50:35.465 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi027.ukdgzs smithi027 *:8443,9283,8765 running (7s) 2s ago 8m 63.3M - 19.0.0-3223-g5883d494 3d983c26e278 7548da9b5eb9 2024-04-22T15:50:35.465 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi016 smithi016 running (11m) 0s ago 11m 55.4M 2048M 18.2.1 5be31c24972a 72af40873cb6 2024-04-22T15:50:35.465 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi027 smithi027 running (8m) 2s ago 8m 41.6M 2048M 18.2.1 5be31c24972a 9f21ce7fa7ef 2024-04-22T15:50:35.465 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi016 smithi016 *:9100 running (9m) 0s ago 9m 16.0M - 1.5.0 0da6a335fe13 1937a0bc1959 2024-04-22T15:50:35.465 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi027 smithi027 *:9100 running (8m) 2s ago 8m 15.8M - 1.5.0 0da6a335fe13 00440c991b3f 2024-04-22T15:50:35.465 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (7m) 0s ago 7m 180M 1217M 18.2.1 5be31c24972a 3c6ebc4ea041 2024-04-22T15:50:35.465 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (7m) 0s ago 7m 173M 1217M 18.2.1 5be31c24972a 6aa2e6796784 2024-04-22T15:50:35.465 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (6m) 0s ago 6m 149M 1217M 18.2.1 5be31c24972a 3d9d310aa9eb 2024-04-22T15:50:35.465 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi027 running (6m) 2s ago 6m 213M 2240M 18.2.1 5be31c24972a cdf9b7aaf865 2024-04-22T15:50:35.465 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (5m) 2s ago 5m 182M 2240M 18.2.1 5be31c24972a 629397d49c76 2024-04-22T15:50:35.466 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi027 running (5m) 2s ago 5m 184M 2240M 18.2.1 5be31c24972a 23adcf675b04 2024-04-22T15:50:35.466 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.smithi016 smithi016 *:9095 running (13s) 0s ago 9m 50.5M - 2.43.0 a07b618ecd1d 804a7d8595e4 2024-04-22T15:50:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.463+0000 7f3a53479700 1 -- 172.21.15.16:0/2069657119 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f3a3806ea80 msgr2=0x7f3a38070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.463+0000 7f3a53479700 1 --2- 172.21.15.16:0/2069657119 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f3a3806ea80 0x7f3a38070f40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c006010 tx=0x7f3a3c00b540 comp rx=0 tx=0).stop 2024-04-22T15:50:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.463+0000 7f3a53479700 1 -- 172.21.15.16:0/2069657119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a4c101820 msgr2=0x7f3a4c196450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.463+0000 7f3a53479700 1 --2- 172.21.15.16:0/2069657119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a4c101820 0x7f3a4c196450 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800d8d0 tx=0x7f3a4800dbe0 comp rx=0 tx=0).stop 2024-04-22T15:50:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.463+0000 7f3a53479700 1 -- 172.21.15.16:0/2069657119 shutdown_connections 2024-04-22T15:50:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.463+0000 7f3a53479700 1 --2- 172.21.15.16:0/2069657119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a4c101820 0x7f3a4c196450 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.463+0000 7f3a53479700 1 --2- 172.21.15.16:0/2069657119 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f3a3806ea80 0x7f3a38070f40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.463+0000 7f3a53479700 1 --2- 172.21.15.16:0/2069657119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a4c104150 0x7f3a4c196990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.463+0000 7f3a53479700 1 -- 172.21.15.16:0/2069657119 >> 172.21.15.16:0/2069657119 conn(0x7f3a4c0fb190 msgr2=0x7f3a4c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.463+0000 7f3a53479700 1 -- 172.21.15.16:0/2069657119 shutdown_connections 2024-04-22T15:50:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.463+0000 7f3a53479700 1 -- 172.21.15.16:0/2069657119 wait complete. 2024-04-22T15:50:35.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.574+0000 7f0a93f0d700 1 -- 172.21.15.16:0/1003405985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a8c104130 msgr2=0x7f0a8c106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:35.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.574+0000 7f0a93f0d700 1 --2- 172.21.15.16:0/1003405985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a8c104130 0x7f0a8c106520 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c009910 tx=0x7f0a7c009c20 comp rx=0 tx=0).stop 2024-04-22T15:50:35.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.574+0000 7f0a93f0d700 1 -- 172.21.15.16:0/1003405985 shutdown_connections 2024-04-22T15:50:35.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.574+0000 7f0a93f0d700 1 --2- 172.21.15.16:0/1003405985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a8c104130 0x7f0a8c106520 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:35.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.574+0000 7f0a93f0d700 1 --2- 172.21.15.16:0/1003405985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a8c101800 0x7f0a8c103bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:35.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.574+0000 7f0a93f0d700 1 -- 172.21.15.16:0/1003405985 >> 172.21.15.16:0/1003405985 conn(0x7f0a8c0fb190 msgr2=0x7f0a8c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:35.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.574+0000 7f0a93f0d700 1 -- 172.21.15.16:0/1003405985 shutdown_connections 2024-04-22T15:50:35.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.574+0000 7f0a93f0d700 1 -- 172.21.15.16:0/1003405985 wait complete. 2024-04-22T15:50:35.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.575+0000 7f0a93f0d700 1 Processor -- start 2024-04-22T15:50:35.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.575+0000 7f0a93f0d700 1 -- start start 2024-04-22T15:50:35.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.575+0000 7f0a93f0d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a8c101800 0x7f0a8c196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:35.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.575+0000 7f0a93f0d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a8c104130 0x7f0a8c196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:35.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.575+0000 7f0a93f0d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8c196fb0 con 0x7f0a8c101800 2024-04-22T15:50:35.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.575+0000 7f0a93f0d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8c1970f0 con 0x7f0a8c104130 2024-04-22T15:50:35.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.575+0000 7f0a91ca9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a8c101800 0x7f0a8c196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:35.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.575+0000 7f0a91ca9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a8c101800 0x7f0a8c196450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48252/0 (socket says 172.21.15.16:48252) 2024-04-22T15:50:35.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.575+0000 7f0a91ca9700 1 -- 172.21.15.16:0/117505470 learned_addr learned my addr 172.21.15.16:0/117505470 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:50:35.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.575+0000 7f0a914a8700 1 --2- 172.21.15.16:0/117505470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a8c104130 0x7f0a8c196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:35.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.575+0000 7f0a91ca9700 1 -- 172.21.15.16:0/117505470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a8c104130 msgr2=0x7f0a8c196990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:35.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.575+0000 7f0a91ca9700 1 --2- 172.21.15.16:0/117505470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a8c104130 0x7f0a8c196990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:35.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.575+0000 7f0a91ca9700 1 -- 172.21.15.16:0/117505470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a7c0095f0 con 0x7f0a8c101800 2024-04-22T15:50:35.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.575+0000 7f0a91ca9700 1 --2- 172.21.15.16:0/117505470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a8c101800 0x7f0a8c196450 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0a8800ef10 tx=0x7f0a8800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:35.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.576+0000 7f0a82ffd700 1 -- 172.21.15.16:0/117505470 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a8800c9f0 con 0x7f0a8c101800 2024-04-22T15:50:35.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.576+0000 7f0a93f0d700 1 -- 172.21.15.16:0/117505470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a8c19bb40 con 0x7f0a8c101800 2024-04-22T15:50:35.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.576+0000 7f0a82ffd700 1 -- 172.21.15.16:0/117505470 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0a8800f040 con 0x7f0a8c101800 2024-04-22T15:50:35.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.576+0000 7f0a82ffd700 1 -- 172.21.15.16:0/117505470 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a880146b0 con 0x7f0a8c101800 2024-04-22T15:50:35.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.576+0000 7f0a93f0d700 1 -- 172.21.15.16:0/117505470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a8c19c090 con 0x7f0a8c101800 2024-04-22T15:50:35.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.577+0000 7f0a93f0d700 1 -- 172.21.15.16:0/117505470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a8c190610 con 0x7f0a8c101800 2024-04-22T15:50:35.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.577+0000 7f0a82ffd700 1 -- 172.21.15.16:0/117505470 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 30) v1 ==== 92700+0+0 (secure 0 0 0) 0x7f0a880078b0 con 0x7f0a8c101800 2024-04-22T15:50:35.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.578+0000 7f0a82ffd700 1 --2- 172.21.15.16:0/117505470 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f0a7806ea70 0x7f0a78070f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:35.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.578+0000 7f0a82ffd700 1 -- 172.21.15.16:0/117505470 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f0a8808ee70 con 0x7f0a8c101800 2024-04-22T15:50:35.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.578+0000 7f0a914a8700 1 --2- 172.21.15.16:0/117505470 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f0a7806ea70 0x7f0a78070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:35.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.578+0000 7f0a914a8700 1 --2- 172.21.15.16:0/117505470 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f0a7806ea70 0x7f0a78070f30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c00b6f0 tx=0x7f0a7c005a90 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:35.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.583+0000 7f0a82ffd700 1 -- 172.21.15.16:0/117505470 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f0a8805ba60 con 0x7f0a8c101800 2024-04-22T15:50:35.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:35 smithi027 ceph-mon[34922]: pgmap v21: 65 pgs: 65 active+clean; 1.3 GiB data, 4.3 GiB used, 532 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 139 op/s 2024-04-22T15:50:35.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:35 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:35.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:35 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:35.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:35 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:35.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:35 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:35.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.901+0000 7f0a93f0d700 1 -- 172.21.15.16:0/117505470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0a8c02cee0 con 0x7f0a8c101800 2024-04-22T15:50:35.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.902+0000 7f0a82ffd700 1 -- 172.21.15.16:0/117505470 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f0a8805b5f0 con 0x7f0a8c101800 2024-04-22T15:50:35.905 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:50:35.905 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-22T15:50:35.905 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T15:50:35.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:50:35.905 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-22T15:50:35.905 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:50:35.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:50:35.905 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-22T15:50:35.905 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-22T15:50:35.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:50:35.905 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-22T15:50:35.905 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T15:50:35.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:50:35.906 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-22T15:50:35.906 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-04-22T15:50:35.906 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:50:35.906 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-22T15:50:35.906 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:50:35.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.905+0000 7f0a93f0d700 1 -- 172.21.15.16:0/117505470 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f0a7806ea70 msgr2=0x7f0a78070f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:35.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.905+0000 7f0a93f0d700 1 --2- 172.21.15.16:0/117505470 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f0a7806ea70 0x7f0a78070f30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c00b6f0 tx=0x7f0a7c005a90 comp rx=0 tx=0).stop 2024-04-22T15:50:35.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.905+0000 7f0a93f0d700 1 -- 172.21.15.16:0/117505470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a8c101800 msgr2=0x7f0a8c196450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:35.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.905+0000 7f0a93f0d700 1 --2- 172.21.15.16:0/117505470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a8c101800 0x7f0a8c196450 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0a8800ef10 tx=0x7f0a8800c3a0 comp rx=0 tx=0).stop 2024-04-22T15:50:35.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.905+0000 7f0a93f0d700 1 -- 172.21.15.16:0/117505470 shutdown_connections 2024-04-22T15:50:35.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.905+0000 7f0a93f0d700 1 --2- 172.21.15.16:0/117505470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a8c101800 0x7f0a8c196450 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:35.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.905+0000 7f0a93f0d700 1 --2- 172.21.15.16:0/117505470 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f0a7806ea70 0x7f0a78070f30 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:35.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.905+0000 7f0a93f0d700 1 --2- 172.21.15.16:0/117505470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a8c104130 0x7f0a8c196990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:35.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.905+0000 7f0a93f0d700 1 -- 172.21.15.16:0/117505470 >> 172.21.15.16:0/117505470 conn(0x7f0a8c0fb190 msgr2=0x7f0a8c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:35.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.905+0000 7f0a93f0d700 1 -- 172.21.15.16:0/117505470 shutdown_connections 2024-04-22T15:50:35.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:35.905+0000 7f0a93f0d700 1 -- 172.21.15.16:0/117505470 wait complete. 2024-04-22T15:50:36.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.023+0000 7faf57b85700 1 -- 172.21.15.16:0/51305118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf500ffea0 msgr2=0x7faf50100320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:36.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.023+0000 7faf57b85700 1 --2- 172.21.15.16:0/51305118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf500ffea0 0x7faf50100320 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7faf44009910 tx=0x7faf44009c20 comp rx=0 tx=0).stop 2024-04-22T15:50:36.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.023+0000 7faf57b85700 1 -- 172.21.15.16:0/51305118 shutdown_connections 2024-04-22T15:50:36.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.023+0000 7faf57b85700 1 --2- 172.21.15.16:0/51305118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf500ffea0 0x7faf50100320 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:36.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.023+0000 7faf57b85700 1 --2- 172.21.15.16:0/51305118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf500ff540 0x7faf500ff960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:36.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.023+0000 7faf57b85700 1 -- 172.21.15.16:0/51305118 >> 172.21.15.16:0/51305118 conn(0x7faf500fb160 msgr2=0x7faf500fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:36.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.023+0000 7faf57b85700 1 -- 172.21.15.16:0/51305118 shutdown_connections 2024-04-22T15:50:36.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.023+0000 7faf57b85700 1 -- 172.21.15.16:0/51305118 wait complete. 2024-04-22T15:50:36.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.024+0000 7faf57b85700 1 Processor -- start 2024-04-22T15:50:36.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.024+0000 7faf57b85700 1 -- start start 2024-04-22T15:50:36.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.024+0000 7faf57b85700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf500ff540 0x7faf50198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:36.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.024+0000 7faf57b85700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf500ffea0 0x7faf50198ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:36.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.024+0000 7faf57b85700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf501994f0 con 0x7faf500ff540 2024-04-22T15:50:36.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.024+0000 7faf57b85700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf50199630 con 0x7faf500ffea0 2024-04-22T15:50:36.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.024+0000 7faf55921700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf500ff540 0x7faf50198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:36.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.024+0000 7faf55921700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf500ff540 0x7faf50198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48272/0 (socket says 172.21.15.16:48272) 2024-04-22T15:50:36.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.024+0000 7faf55921700 1 -- 172.21.15.16:0/2549192420 learned_addr learned my addr 172.21.15.16:0/2549192420 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:50:36.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.024+0000 7faf55120700 1 --2- 172.21.15.16:0/2549192420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf500ffea0 0x7faf50198ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:36.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.025+0000 7faf55921700 1 -- 172.21.15.16:0/2549192420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf500ffea0 msgr2=0x7faf50198ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:36.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.025+0000 7faf55921700 1 --2- 172.21.15.16:0/2549192420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf500ffea0 0x7faf50198ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:36.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.025+0000 7faf55921700 1 -- 172.21.15.16:0/2549192420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf440095f0 con 0x7faf500ff540 2024-04-22T15:50:36.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.025+0000 7faf55921700 1 --2- 172.21.15.16:0/2549192420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf500ff540 0x7faf50198990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7faf480060b0 tx=0x7faf4800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:36.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.025+0000 7faf4effd700 1 -- 172.21.15.16:0/2549192420 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf4800dda0 con 0x7faf500ff540 2024-04-22T15:50:36.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.025+0000 7faf4effd700 1 -- 172.21.15.16:0/2549192420 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7faf4800f040 con 0x7faf500ff540 2024-04-22T15:50:36.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.025+0000 7faf57b85700 1 -- 172.21.15.16:0/2549192420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf5019e0e0 con 0x7faf500ff540 2024-04-22T15:50:36.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.025+0000 7faf4effd700 1 -- 172.21.15.16:0/2549192420 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf480146f0 con 0x7faf500ff540 2024-04-22T15:50:36.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.025+0000 7faf57b85700 1 -- 172.21.15.16:0/2549192420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf5019e630 con 0x7faf500ff540 2024-04-22T15:50:36.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.026+0000 7faf57b85700 1 -- 172.21.15.16:0/2549192420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf50068af0 con 0x7faf500ff540 2024-04-22T15:50:36.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.027+0000 7faf4effd700 1 -- 172.21.15.16:0/2549192420 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 30) v1 ==== 92700+0+0 (secure 0 0 0) 0x7faf48015400 con 0x7faf500ff540 2024-04-22T15:50:36.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.027+0000 7faf4effd700 1 --2- 172.21.15.16:0/2549192420 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7faf3806ea80 0x7faf38070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:50:36.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.027+0000 7faf4effd700 1 -- 172.21.15.16:0/2549192420 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7faf4808e450 con 0x7faf500ff540 2024-04-22T15:50:36.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.027+0000 7faf55120700 1 --2- 172.21.15.16:0/2549192420 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7faf3806ea80 0x7faf38070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:50:36.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.028+0000 7faf55120700 1 --2- 172.21.15.16:0/2549192420 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7faf3806ea80 0x7faf38070f40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7faf44006010 tx=0x7faf4400b540 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:50:36.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.032+0000 7faf4effd700 1 -- 172.21.15.16:0/2549192420 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7faf4805b170 con 0x7faf500ff540 2024-04-22T15:50:36.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:36 smithi016 ceph-mon[26620]: from='client.14726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:36.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:36 smithi016 ceph-mon[26620]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:36.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:36 smithi016 ceph-mon[26620]: from='client.14734 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:36.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:36 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/117505470' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:50:36.262 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:36 smithi027 ceph-mon[34922]: from='client.14726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:36.262 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:36 smithi027 ceph-mon[34922]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:36.262 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:36 smithi027 ceph-mon[34922]: from='client.14734 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:36.262 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:36 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/117505470' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:50:36.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.285+0000 7faf57b85700 1 -- 172.21.15.16:0/2549192420 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faf501093b0 con 0x7faf3806ea80 2024-04-22T15:50:36.290 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:50:36.290 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:50:36.290 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-22T15:50:36.290 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T15:50:36.290 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-22T15:50:36.290 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-04-22T15:50:36.290 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-22T15:50:36.290 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "2/2 daemons upgraded", 2024-04-22T15:50:36.290 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading mgr daemons", 2024-04-22T15:50:36.290 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-22T15:50:36.291 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:50:36.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.287+0000 7faf4effd700 1 -- 172.21.15.16:0/2549192420 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7faf501093b0 con 0x7faf3806ea80 2024-04-22T15:50:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.291+0000 7faf57b85700 1 -- 172.21.15.16:0/2549192420 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7faf3806ea80 msgr2=0x7faf38070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.291+0000 7faf57b85700 1 --2- 172.21.15.16:0/2549192420 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7faf3806ea80 0x7faf38070f40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7faf44006010 tx=0x7faf4400b540 comp rx=0 tx=0).stop 2024-04-22T15:50:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.291+0000 7faf57b85700 1 -- 172.21.15.16:0/2549192420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf500ff540 msgr2=0x7faf50198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:50:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.291+0000 7faf57b85700 1 --2- 172.21.15.16:0/2549192420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf500ff540 0x7faf50198990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7faf480060b0 tx=0x7faf4800d3b0 comp rx=0 tx=0).stop 2024-04-22T15:50:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.291+0000 7faf57b85700 1 -- 172.21.15.16:0/2549192420 shutdown_connections 2024-04-22T15:50:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.291+0000 7faf57b85700 1 --2- 172.21.15.16:0/2549192420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf500ff540 0x7faf50198990 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.291+0000 7faf57b85700 1 --2- 172.21.15.16:0/2549192420 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7faf3806ea80 0x7faf38070f40 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.291+0000 7faf57b85700 1 --2- 172.21.15.16:0/2549192420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf500ffea0 0x7faf50198ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:50:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.291+0000 7faf57b85700 1 -- 172.21.15.16:0/2549192420 >> 172.21.15.16:0/2549192420 conn(0x7faf500fb160 msgr2=0x7faf50107c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:50:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.291+0000 7faf57b85700 1 -- 172.21.15.16:0/2549192420 shutdown_connections 2024-04-22T15:50:36.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:50:36.291+0000 7faf57b85700 1 -- 172.21.15.16:0/2549192420 wait complete. 2024-04-22T15:50:37.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:37 smithi016 ceph-mon[26620]: pgmap v22: 65 pgs: 65 active+clean; 865 MiB data, 3.0 GiB used, 533 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 190 op/s 2024-04-22T15:50:37.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:37 smithi016 ceph-mon[26620]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:37.433 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:37 smithi027 ceph-mon[34922]: pgmap v22: 65 pgs: 65 active+clean; 865 MiB data, 3.0 GiB used, 533 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 190 op/s 2024-04-22T15:50:37.433 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:37 smithi027 ceph-mon[34922]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:50:38.065 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-22T15:50:38.065 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-04-22T15:50:39.453 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: pgmap v23: 65 pgs: 65 active+clean; 865 MiB data, 3.0 GiB used, 533 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 114 op/s 2024-04-22T15:50:39.453 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:39.453 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:39.453 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:50:39.453 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:50:39.453 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:39.453 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:50:39.453 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:50:39.453 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: Upgrade: Setting container_image for all mgr 2024-04-22T15:50:39.453 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:39.453 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi016.rnsnnr"}]: dispatch 2024-04-22T15:50:39.454 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi016.rnsnnr"}]': finished 2024-04-22T15:50:39.454 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi027.ukdgzs"}]: dispatch 2024-04-22T15:50:39.454 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi027.ukdgzs"}]': finished 2024-04-22T15:50:39.454 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:50:39.454 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: Upgrade: Setting container_image for all rgw 2024-04-22T15:50:39.454 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:39.454 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:50:39.454 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: Upgrade: Setting container_image for all rbd-mirror 2024-04-22T15:50:39.454 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:39.454 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:50:39.454 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-22T15:50:39.454 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:39.454 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:50:39.454 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: Upgrade: Setting container_image for all iscsi 2024-04-22T15:50:39.455 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:39.455 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:50:39.455 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: Upgrade: Setting container_image for all nfs 2024-04-22T15:50:39.455 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:39.455 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:50:39.455 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: Upgrade: Setting container_image for all nvmeof 2024-04-22T15:50:39.455 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:39.455 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: Standby manager daemon smithi027.ukdgzs restarted 2024-04-22T15:50:39.455 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: Standby manager daemon smithi027.ukdgzs started 2024-04-22T15:50:39.455 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.? 172.21.15.27:0/1172902450' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.ukdgzs/crt"}]: dispatch 2024-04-22T15:50:39.455 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.? 172.21.15.27:0/1172902450' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:50:39.455 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.? 172.21.15.27:0/1172902450' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.ukdgzs/key"}]: dispatch 2024-04-22T15:50:39.455 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:39 smithi016 ceph-mon[26620]: from='mgr.? 172.21.15.27:0/1172902450' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:50:39.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: pgmap v23: 65 pgs: 65 active+clean; 865 MiB data, 3.0 GiB used, 533 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 114 op/s 2024-04-22T15:50:39.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:39.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:39.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:50:39.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:50:39.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:39.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:50:39.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:50:39.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: Upgrade: Setting container_image for all mgr 2024-04-22T15:50:39.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:39.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi016.rnsnnr"}]: dispatch 2024-04-22T15:50:39.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi016.rnsnnr"}]': finished 2024-04-22T15:50:39.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi027.ukdgzs"}]: dispatch 2024-04-22T15:50:39.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi027.ukdgzs"}]': finished 2024-04-22T15:50:39.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:50:39.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: Upgrade: Setting container_image for all rgw 2024-04-22T15:50:39.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:39.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:50:39.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: Upgrade: Setting container_image for all rbd-mirror 2024-04-22T15:50:39.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:39.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:50:39.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-22T15:50:39.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:39.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:50:39.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: Upgrade: Setting container_image for all iscsi 2024-04-22T15:50:39.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:39.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:50:39.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: Upgrade: Setting container_image for all nfs 2024-04-22T15:50:39.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:39.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:50:39.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: Upgrade: Setting container_image for all nvmeof 2024-04-22T15:50:39.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:39.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: Standby manager daemon smithi027.ukdgzs restarted 2024-04-22T15:50:39.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: Standby manager daemon smithi027.ukdgzs started 2024-04-22T15:50:39.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.27:0/1172902450' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.ukdgzs/crt"}]: dispatch 2024-04-22T15:50:39.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.27:0/1172902450' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:50:39.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.27:0/1172902450' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.ukdgzs/key"}]: dispatch 2024-04-22T15:50:39.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:39 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.27:0/1172902450' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:50:40.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:40 smithi027 ceph-mon[34922]: mgrmap e31: smithi016.rnsnnr(active, since 41s), standbys: smithi027.ukdgzs 2024-04-22T15:50:40.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:40 smithi016 ceph-mon[26620]: mgrmap e31: smithi016.rnsnnr(active, since 41s), standbys: smithi027.ukdgzs 2024-04-22T15:50:41.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:41 smithi027 ceph-mon[34922]: pgmap v24: 65 pgs: 65 active+clean; 391 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 167 op/s 2024-04-22T15:50:41.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:41 smithi027 ceph-mon[34922]: Upgrade: Updating node-exporter.smithi016 (1/2) 2024-04-22T15:50:41.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:41 smithi027 ceph-mon[34922]: Deploying daemon node-exporter.smithi016 on smithi016 2024-04-22T15:50:41.855 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:41 smithi016 ceph-mon[26620]: pgmap v24: 65 pgs: 65 active+clean; 391 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 167 op/s 2024-04-22T15:50:41.856 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:41 smithi016 ceph-mon[26620]: Upgrade: Updating node-exporter.smithi016 (1/2) 2024-04-22T15:50:41.856 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:41 smithi016 ceph-mon[26620]: Deploying daemon node-exporter.smithi016 on smithi016 2024-04-22T15:50:42.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:42 smithi016 ceph-mon[26620]: pgmap v25: 65 pgs: 65 active+clean; 391 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 105 op/s 2024-04-22T15:50:43.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:42 smithi027 ceph-mon[34922]: pgmap v25: 65 pgs: 65 active+clean; 391 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 105 op/s 2024-04-22T15:50:43.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:43 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:50:44.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:43 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:50:45.520 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:45 smithi027 ceph-mon[34922]: pgmap v26: 65 pgs: 65 active+clean; 391 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 105 op/s 2024-04-22T15:50:45.763 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:45 smithi016 ceph-mon[26620]: pgmap v26: 65 pgs: 65 active+clean; 391 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 105 op/s 2024-04-22T15:50:45.763 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:45 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:45.764 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:45 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:45.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:45 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:45.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:45 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:46.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:46 smithi027 ceph-mon[34922]: pgmap v27: 65 pgs: 65 active+clean; 292 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 156 op/s 2024-04-22T15:50:47.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:46 smithi016 ceph-mon[26620]: pgmap v27: 65 pgs: 65 active+clean; 292 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 156 op/s 2024-04-22T15:50:49.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:48 smithi016 ceph-mon[26620]: pgmap v28: 65 pgs: 65 active+clean; 292 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 105 op/s 2024-04-22T15:50:49.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:48 smithi027 ceph-mon[34922]: pgmap v28: 65 pgs: 65 active+clean; 292 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 105 op/s 2024-04-22T15:50:51.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:50 smithi016 ceph-mon[26620]: pgmap v29: 65 pgs: 65 active+clean; 292 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 155 op/s 2024-04-22T15:50:51.261 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:50 smithi027 ceph-mon[34922]: pgmap v29: 65 pgs: 65 active+clean; 292 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 155 op/s 2024-04-22T15:50:53.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:52 smithi027 ceph-mon[34922]: pgmap v30: 65 pgs: 65 active+clean; 292 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 101 op/s 2024-04-22T15:50:53.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:52 smithi016 ceph-mon[26620]: pgmap v30: 65 pgs: 65 active+clean; 292 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 101 op/s 2024-04-22T15:50:55.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:54 smithi016 ceph-mon[26620]: pgmap v31: 65 pgs: 65 active+clean; 292 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 101 op/s 2024-04-22T15:50:55.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:54 smithi027 ceph-mon[34922]: pgmap v31: 65 pgs: 65 active+clean; 292 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 101 op/s 2024-04-22T15:50:56.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:55 smithi027 ceph-mon[34922]: Upgrade: Updating node-exporter.smithi027 (2/2) 2024-04-22T15:50:56.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:55 smithi027 ceph-mon[34922]: Deploying daemon node-exporter.smithi027 on smithi027 2024-04-22T15:50:56.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:55 smithi016 ceph-mon[26620]: Upgrade: Updating node-exporter.smithi027 (2/2) 2024-04-22T15:50:56.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:55 smithi016 ceph-mon[26620]: Deploying daemon node-exporter.smithi027 on smithi027 2024-04-22T15:50:57.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:56 smithi016 ceph-mon[26620]: pgmap v32: 65 pgs: 65 active+clean; 297 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 143 op/s 2024-04-22T15:50:57.264 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:56 smithi027 ceph-mon[34922]: pgmap v32: 65 pgs: 65 active+clean; 297 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 143 op/s 2024-04-22T15:50:59.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:58 smithi016 ceph-mon[26620]: pgmap v33: 65 pgs: 65 active+clean; 297 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 91 op/s 2024-04-22T15:50:59.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:58 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:50:59.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:58 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:59.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:58 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:59.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:50:58 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:50:59.217 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:58 smithi027 ceph-mon[34922]: pgmap v33: 65 pgs: 65 active+clean; 297 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 91 op/s 2024-04-22T15:50:59.217 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:58 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:50:59.217 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:58 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:59.218 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:58 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:50:59.218 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:50:58 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:51:01.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:00 smithi016 ceph-mon[26620]: pgmap v34: 65 pgs: 65 active+clean; 304 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 142 op/s 2024-04-22T15:51:01.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:00 smithi027 ceph-mon[34922]: pgmap v34: 65 pgs: 65 active+clean; 304 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 142 op/s 2024-04-22T15:51:03.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:02 smithi016 ceph-mon[26620]: pgmap v35: 65 pgs: 65 active+clean; 304 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 92 op/s 2024-04-22T15:51:03.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:02 smithi027 ceph-mon[34922]: pgmap v35: 65 pgs: 65 active+clean; 304 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 92 op/s 2024-04-22T15:51:05.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:05 smithi027 ceph-mon[34922]: pgmap v36: 65 pgs: 65 active+clean; 304 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 92 op/s 2024-04-22T15:51:05.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:05 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:05.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:05 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:05.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:05 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:05.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:05 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:05.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:05 smithi016 ceph-mon[26620]: pgmap v36: 65 pgs: 65 active+clean; 304 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 92 op/s 2024-04-22T15:51:05.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:05 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:05.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:05 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:05.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:05 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:05.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:05 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:06.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.420+0000 7f0664d80700 1 -- 172.21.15.16:0/4154265915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06600ffbc0 msgr2=0x7f06600fffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:06.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.420+0000 7f0664d80700 1 --2- 172.21.15.16:0/4154265915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06600ffbc0 0x7f06600fffe0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0650009910 tx=0x7f0650009c20 comp rx=0 tx=0).stop 2024-04-22T15:51:06.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.420+0000 7f0664d80700 1 -- 172.21.15.16:0/4154265915 shutdown_connections 2024-04-22T15:51:06.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.420+0000 7f0664d80700 1 --2- 172.21.15.16:0/4154265915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0660100dc0 0x7f0660101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.420+0000 7f0664d80700 1 --2- 172.21.15.16:0/4154265915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06600ffbc0 0x7f06600fffe0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.420+0000 7f0664d80700 1 -- 172.21.15.16:0/4154265915 >> 172.21.15.16:0/4154265915 conn(0x7f06600fb160 msgr2=0x7f06600fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:06.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.420+0000 7f0664d80700 1 -- 172.21.15.16:0/4154265915 shutdown_connections 2024-04-22T15:51:06.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.420+0000 7f0664d80700 1 -- 172.21.15.16:0/4154265915 wait complete. 2024-04-22T15:51:06.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.420+0000 7f0664d80700 1 Processor -- start 2024-04-22T15:51:06.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.421+0000 7f0664d80700 1 -- start start 2024-04-22T15:51:06.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.421+0000 7f0664d80700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06600ffbc0 0x7f06601967f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:06.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.421+0000 7f0664d80700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0660100dc0 0x7f0660196d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:06.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.421+0000 7f0664d80700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0660197350 con 0x7f06600ffbc0 2024-04-22T15:51:06.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.421+0000 7f0664d80700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0660197490 con 0x7f0660100dc0 2024-04-22T15:51:06.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.421+0000 7f065e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06600ffbc0 0x7f06601967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:06.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.421+0000 7f065e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06600ffbc0 0x7f06601967f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57618/0 (socket says 172.21.15.16:57618) 2024-04-22T15:51:06.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.421+0000 7f065e59c700 1 -- 172.21.15.16:0/3190756126 learned_addr learned my addr 172.21.15.16:0/3190756126 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:51:06.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.421+0000 7f0657fff700 1 --2- 172.21.15.16:0/3190756126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0660100dc0 0x7f0660196d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:06.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.421+0000 7f065e59c700 1 -- 172.21.15.16:0/3190756126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0660100dc0 msgr2=0x7f0660196d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:06.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.421+0000 7f065e59c700 1 --2- 172.21.15.16:0/3190756126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0660100dc0 0x7f0660196d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.421+0000 7f065e59c700 1 -- 172.21.15.16:0/3190756126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06500095f0 con 0x7f06600ffbc0 2024-04-22T15:51:06.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.421+0000 7f065e59c700 1 --2- 172.21.15.16:0/3190756126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06600ffbc0 0x7f06601967f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f0650000c00 tx=0x7f065000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:06.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.421+0000 7f06577fe700 1 -- 172.21.15.16:0/3190756126 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f065001d070 con 0x7f06600ffbc0 2024-04-22T15:51:06.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.422+0000 7f0664d80700 1 -- 172.21.15.16:0/3190756126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f066019bee0 con 0x7f06600ffbc0 2024-04-22T15:51:06.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.422+0000 7f0664d80700 1 -- 172.21.15.16:0/3190756126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f066019c350 con 0x7f06600ffbc0 2024-04-22T15:51:06.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.422+0000 7f06577fe700 1 -- 172.21.15.16:0/3190756126 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0650005dc0 con 0x7f06600ffbc0 2024-04-22T15:51:06.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.422+0000 7f06577fe700 1 -- 172.21.15.16:0/3190756126 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0650022470 con 0x7f06600ffbc0 2024-04-22T15:51:06.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.423+0000 7f0664d80700 1 -- 172.21.15.16:0/3190756126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0640005320 con 0x7f06600ffbc0 2024-04-22T15:51:06.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.423+0000 7f06577fe700 1 -- 172.21.15.16:0/3190756126 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 92700+0+0 (secure 0 0 0) 0x7f065000fa70 con 0x7f06600ffbc0 2024-04-22T15:51:06.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.424+0000 7f06577fe700 1 --2- 172.21.15.16:0/3190756126 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f064c06ea80 0x7f064c070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:06.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.424+0000 7f06577fe700 1 -- 172.21.15.16:0/3190756126 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f065008fec0 con 0x7f06600ffbc0 2024-04-22T15:51:06.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.424+0000 7f0657fff700 1 --2- 172.21.15.16:0/3190756126 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f064c06ea80 0x7f064c070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:06.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.424+0000 7f0657fff700 1 --2- 172.21.15.16:0/3190756126 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f064c06ea80 0x7f064c070f40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0648009c60 tx=0x7f0648009260 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:06.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.429+0000 7f06577fe700 1 -- 172.21.15.16:0/3190756126 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f065005cb30 con 0x7f06600ffbc0 2024-04-22T15:51:06.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.714+0000 7f0664d80700 1 -- 172.21.15.16:0/3190756126 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0640000bf0 con 0x7f064c06ea80 2024-04-22T15:51:06.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.717+0000 7f06577fe700 1 -- 172.21.15.16:0/3190756126 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7f0640000bf0 con 0x7f064c06ea80 2024-04-22T15:51:06.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.721+0000 7f0664d80700 1 -- 172.21.15.16:0/3190756126 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f064c06ea80 msgr2=0x7f064c070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:06.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.721+0000 7f0664d80700 1 --2- 172.21.15.16:0/3190756126 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f064c06ea80 0x7f064c070f40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0648009c60 tx=0x7f0648009260 comp rx=0 tx=0).stop 2024-04-22T15:51:06.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.721+0000 7f0664d80700 1 -- 172.21.15.16:0/3190756126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06600ffbc0 msgr2=0x7f06601967f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:06.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.721+0000 7f0664d80700 1 --2- 172.21.15.16:0/3190756126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06600ffbc0 0x7f06601967f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f0650000c00 tx=0x7f065000f690 comp rx=0 tx=0).stop 2024-04-22T15:51:06.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.721+0000 7f0664d80700 1 -- 172.21.15.16:0/3190756126 shutdown_connections 2024-04-22T15:51:06.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.721+0000 7f0664d80700 1 --2- 172.21.15.16:0/3190756126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06600ffbc0 0x7f06601967f0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.721+0000 7f0664d80700 1 --2- 172.21.15.16:0/3190756126 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f064c06ea80 0x7f064c070f40 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.721+0000 7f0664d80700 1 --2- 172.21.15.16:0/3190756126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0660100dc0 0x7f0660196d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.721+0000 7f0664d80700 1 -- 172.21.15.16:0/3190756126 >> 172.21.15.16:0/3190756126 conn(0x7f06600fb160 msgr2=0x7f0660068e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:06.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.721+0000 7f0664d80700 1 -- 172.21.15.16:0/3190756126 shutdown_connections 2024-04-22T15:51:06.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.721+0000 7f0664d80700 1 -- 172.21.15.16:0/3190756126 wait complete. 2024-04-22T15:51:06.730 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:06 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:06.731 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:06 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:06.731 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:06 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:06.731 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:06 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:06.745 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:51:06.746 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:06 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:06.746 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:06 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:06.746 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:06 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:06.746 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:06 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:06.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.872+0000 7f97fd1dd700 1 -- 172.21.15.16:0/4021368894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97f8101eb0 msgr2=0x7f97f8102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:06.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.872+0000 7f97fd1dd700 1 --2- 172.21.15.16:0/4021368894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97f8101eb0 0x7f97f8102310 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f97ec009910 tx=0x7f97ec009c20 comp rx=0 tx=0).stop 2024-04-22T15:51:06.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.872+0000 7f97fd1dd700 1 -- 172.21.15.16:0/4021368894 shutdown_connections 2024-04-22T15:51:06.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.872+0000 7f97fd1dd700 1 --2- 172.21.15.16:0/4021368894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97f8101eb0 0x7f97f8102310 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.872+0000 7f97fd1dd700 1 --2- 172.21.15.16:0/4021368894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f8100cb0 0x7f97f81010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.872+0000 7f97fd1dd700 1 -- 172.21.15.16:0/4021368894 >> 172.21.15.16:0/4021368894 conn(0x7f97f80fc230 msgr2=0x7f97f80fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:06.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.872+0000 7f97fd1dd700 1 -- 172.21.15.16:0/4021368894 shutdown_connections 2024-04-22T15:51:06.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.872+0000 7f97fd1dd700 1 -- 172.21.15.16:0/4021368894 wait complete. 2024-04-22T15:51:06.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.872+0000 7f97fd1dd700 1 Processor -- start 2024-04-22T15:51:06.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.873+0000 7f97fd1dd700 1 -- start start 2024-04-22T15:51:06.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.873+0000 7f97fd1dd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97f8100cb0 0x7f97f81964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:06.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.873+0000 7f97fd1dd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f8101eb0 0x7f97f81969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:06.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.873+0000 7f97fd1dd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f8197010 con 0x7f97f8100cb0 2024-04-22T15:51:06.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.873+0000 7f97fd1dd700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f8197150 con 0x7f97f8101eb0 2024-04-22T15:51:06.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.873+0000 7f97f6d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97f8100cb0 0x7f97f81964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:06.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.873+0000 7f97f6d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97f8100cb0 0x7f97f81964b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57636/0 (socket says 172.21.15.16:57636) 2024-04-22T15:51:06.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.873+0000 7f97f6d9d700 1 -- 172.21.15.16:0/3278600163 learned_addr learned my addr 172.21.15.16:0/3278600163 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:51:06.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.873+0000 7f97f659c700 1 --2- 172.21.15.16:0/3278600163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f8101eb0 0x7f97f81969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:06.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.873+0000 7f97f6d9d700 1 -- 172.21.15.16:0/3278600163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f8101eb0 msgr2=0x7f97f81969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:06.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.873+0000 7f97f6d9d700 1 --2- 172.21.15.16:0/3278600163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f8101eb0 0x7f97f81969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:06.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.873+0000 7f97f6d9d700 1 -- 172.21.15.16:0/3278600163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ec0095f0 con 0x7f97f8100cb0 2024-04-22T15:51:06.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.873+0000 7f97f659c700 1 --2- 172.21.15.16:0/3278600163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f8101eb0 0x7f97f81969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:51:06.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.873+0000 7f97f6d9d700 1 --2- 172.21.15.16:0/3278600163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97f8100cb0 0x7f97f81964b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f97e800e8f0 tx=0x7f97e800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:06.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.874+0000 7f97dffff700 1 -- 172.21.15.16:0/3278600163 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97e8009790 con 0x7f97f8100cb0 2024-04-22T15:51:06.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.874+0000 7f97fd1dd700 1 -- 172.21.15.16:0/3278600163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f819bc00 con 0x7f97f8100cb0 2024-04-22T15:51:06.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.874+0000 7f97dffff700 1 -- 172.21.15.16:0/3278600163 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f97e800c810 con 0x7f97f8100cb0 2024-04-22T15:51:06.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.874+0000 7f97dffff700 1 -- 172.21.15.16:0/3278600163 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97e8003d20 con 0x7f97f8100cb0 2024-04-22T15:51:06.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.874+0000 7f97fd1dd700 1 -- 172.21.15.16:0/3278600163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f819c150 con 0x7f97f8100cb0 2024-04-22T15:51:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.875+0000 7f97fd1dd700 1 -- 172.21.15.16:0/3278600163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f8109810 con 0x7f97f8100cb0 2024-04-22T15:51:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.875+0000 7f97dffff700 1 -- 172.21.15.16:0/3278600163 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 92700+0+0 (secure 0 0 0) 0x7f97e80041f0 con 0x7f97f8100cb0 2024-04-22T15:51:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.876+0000 7f97dffff700 1 --2- 172.21.15.16:0/3278600163 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f97e006ea70 0x7f97e0070f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.876+0000 7f97dffff700 1 -- 172.21.15.16:0/3278600163 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f97e8014070 con 0x7f97f8100cb0 2024-04-22T15:51:06.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.876+0000 7f97f659c700 1 --2- 172.21.15.16:0/3278600163 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f97e006ea70 0x7f97e0070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:06.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.876+0000 7f97f659c700 1 --2- 172.21.15.16:0/3278600163 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f97e006ea70 0x7f97e0070f30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f97ec00b6f0 tx=0x7f97ec005a90 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:06.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:06.880+0000 7f97dffff700 1 -- 172.21.15.16:0/3278600163 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f97e8093070 con 0x7f97f8100cb0 2024-04-22T15:51:07.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.128+0000 7f97fd1dd700 1 -- 172.21.15.16:0/3278600163 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f97f8106800 con 0x7f97e006ea70 2024-04-22T15:51:07.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.129+0000 7f97dffff700 1 -- 172.21.15.16:0/3278600163 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7f97f8106800 con 0x7f97e006ea70 2024-04-22T15:51:07.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.131+0000 7f97fd1dd700 1 -- 172.21.15.16:0/3278600163 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f97e006ea70 msgr2=0x7f97e0070f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:07.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.131+0000 7f97fd1dd700 1 --2- 172.21.15.16:0/3278600163 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f97e006ea70 0x7f97e0070f30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f97ec00b6f0 tx=0x7f97ec005a90 comp rx=0 tx=0).stop 2024-04-22T15:51:07.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.131+0000 7f97fd1dd700 1 -- 172.21.15.16:0/3278600163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97f8100cb0 msgr2=0x7f97f81964b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:07.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.131+0000 7f97fd1dd700 1 --2- 172.21.15.16:0/3278600163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97f8100cb0 0x7f97f81964b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f97e800e8f0 tx=0x7f97e800ec00 comp rx=0 tx=0).stop 2024-04-22T15:51:07.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.131+0000 7f97fd1dd700 1 -- 172.21.15.16:0/3278600163 shutdown_connections 2024-04-22T15:51:07.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.131+0000 7f97fd1dd700 1 --2- 172.21.15.16:0/3278600163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97f8100cb0 0x7f97f81964b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:07.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.131+0000 7f97fd1dd700 1 --2- 172.21.15.16:0/3278600163 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f97e006ea70 0x7f97e0070f30 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:07.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.131+0000 7f97fd1dd700 1 --2- 172.21.15.16:0/3278600163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f8101eb0 0x7f97f81969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:07.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.131+0000 7f97fd1dd700 1 -- 172.21.15.16:0/3278600163 >> 172.21.15.16:0/3278600163 conn(0x7f97f80fc230 msgr2=0x7f97f81050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:07.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.132+0000 7f97fd1dd700 1 -- 172.21.15.16:0/3278600163 shutdown_connections 2024-04-22T15:51:07.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.132+0000 7f97fd1dd700 1 -- 172.21.15.16:0/3278600163 wait complete. 2024-04-22T15:51:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.247+0000 7ff7ff984700 1 -- 172.21.15.16:0/1064946793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f8101eb0 msgr2=0x7ff7f8102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.247+0000 7ff7ff984700 1 --2- 172.21.15.16:0/1064946793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f8101eb0 0x7ff7f8102310 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8009910 tx=0x7ff7e8009c20 comp rx=0 tx=0).stop 2024-04-22T15:51:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.247+0000 7ff7ff984700 1 -- 172.21.15.16:0/1064946793 shutdown_connections 2024-04-22T15:51:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.247+0000 7ff7ff984700 1 --2- 172.21.15.16:0/1064946793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f8101eb0 0x7ff7f8102310 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.247+0000 7ff7ff984700 1 --2- 172.21.15.16:0/1064946793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8100cb0 0x7ff7f81010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.247+0000 7ff7ff984700 1 -- 172.21.15.16:0/1064946793 >> 172.21.15.16:0/1064946793 conn(0x7ff7f80fc230 msgr2=0x7ff7f80fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.247+0000 7ff7ff984700 1 -- 172.21.15.16:0/1064946793 shutdown_connections 2024-04-22T15:51:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.247+0000 7ff7ff984700 1 -- 172.21.15.16:0/1064946793 wait complete. 2024-04-22T15:51:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.248+0000 7ff7ff984700 1 Processor -- start 2024-04-22T15:51:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.248+0000 7ff7ff984700 1 -- start start 2024-04-22T15:51:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.248+0000 7ff7ff984700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f8100cb0 0x7ff7f8074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.248+0000 7ff7ff984700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8101eb0 0x7ff7f8073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.248+0000 7ff7ff984700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f80736c0 con 0x7ff7f8100cb0 2024-04-22T15:51:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.248+0000 7ff7ff984700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f8073800 con 0x7ff7f8101eb0 2024-04-22T15:51:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.248+0000 7ff7fd720700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f8100cb0 0x7ff7f8074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.248+0000 7ff7fd720700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f8100cb0 0x7ff7f8074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57654/0 (socket says 172.21.15.16:57654) 2024-04-22T15:51:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.248+0000 7ff7fd720700 1 -- 172.21.15.16:0/812857757 learned_addr learned my addr 172.21.15.16:0/812857757 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:51:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.248+0000 7ff7fcf1f700 1 --2- 172.21.15.16:0/812857757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8101eb0 0x7ff7f8073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.249+0000 7ff7fd720700 1 -- 172.21.15.16:0/812857757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8101eb0 msgr2=0x7ff7f8073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.249+0000 7ff7fd720700 1 --2- 172.21.15.16:0/812857757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8101eb0 0x7ff7f8073180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.249+0000 7ff7fd720700 1 -- 172.21.15.16:0/812857757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e80095f0 con 0x7ff7f8100cb0 2024-04-22T15:51:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.249+0000 7ff7fd720700 1 --2- 172.21.15.16:0/812857757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f8100cb0 0x7ff7f8074b30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff7f400e8f0 tx=0x7ff7f400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:07.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.249+0000 7ff7ee7fc700 1 -- 172.21.15.16:0/812857757 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7f4009790 con 0x7ff7f8100cb0 2024-04-22T15:51:07.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.249+0000 7ff7ff984700 1 -- 172.21.15.16:0/812857757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7f8073ae0 con 0x7ff7f8100cb0 2024-04-22T15:51:07.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.249+0000 7ff7ee7fc700 1 -- 172.21.15.16:0/812857757 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff7f400c810 con 0x7ff7f8100cb0 2024-04-22T15:51:07.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.249+0000 7ff7ee7fc700 1 -- 172.21.15.16:0/812857757 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7f4003d20 con 0x7ff7f8100cb0 2024-04-22T15:51:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.249+0000 7ff7ff984700 1 -- 172.21.15.16:0/812857757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7f819e540 con 0x7ff7f8100cb0 2024-04-22T15:51:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.250+0000 7ff7ff984700 1 -- 172.21.15.16:0/812857757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7f8109810 con 0x7ff7f8100cb0 2024-04-22T15:51:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.251+0000 7ff7ee7fc700 1 -- 172.21.15.16:0/812857757 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 92700+0+0 (secure 0 0 0) 0x7ff7f40041f0 con 0x7ff7f8100cb0 2024-04-22T15:51:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.251+0000 7ff7ee7fc700 1 --2- 172.21.15.16:0/812857757 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7ff7e406ea80 0x7ff7e4070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.251+0000 7ff7ee7fc700 1 -- 172.21.15.16:0/812857757 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7ff7f4014070 con 0x7ff7f8100cb0 2024-04-22T15:51:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.251+0000 7ff7fcf1f700 1 --2- 172.21.15.16:0/812857757 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7ff7e406ea80 0x7ff7e4070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.252+0000 7ff7fcf1f700 1 --2- 172.21.15.16:0/812857757 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7ff7e406ea80 0x7ff7e4070f40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8009fd0 tx=0x7ff7e8005940 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.256+0000 7ff7ee7fc700 1 -- 172.21.15.16:0/812857757 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7ff7f4093070 con 0x7ff7f8100cb0 2024-04-22T15:51:07.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.524+0000 7ff7ff984700 1 -- 172.21.15.16:0/812857757 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7f8106800 con 0x7ff7e406ea80 2024-04-22T15:51:07.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.535+0000 7ff7ee7fc700 1 -- 172.21.15.16:0/812857757 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ff7f8106800 con 0x7ff7e406ea80 2024-04-22T15:51:07.538 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:51:07.538 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.smithi016 smithi016 *:9093,9094 running (8m) 3s ago 10m 25.0M - 0.25.0 c8568f914cd2 fab40987f1b0 2024-04-22T15:51:07.538 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi016 smithi016 running (10m) 3s ago 10m 7788k - 18.2.1 5be31c24972a f8b627d3f41e 2024-04-22T15:51:07.538 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi027 smithi027 running (9m) 3s ago 9m 7776k - 18.2.1 5be31c24972a f00fd40b4220 2024-04-22T15:51:07.538 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi016 smithi016 running (10m) 3s ago 10m 6660k - 18.2.1 5be31c24972a 6b93db1fa0d9 2024-04-22T15:51:07.538 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi027 smithi027 running (9m) 3s ago 9m 6665k - 18.2.1 5be31c24972a d3bd439f30a0 2024-04-22T15:51:07.538 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.smithi016 smithi016 *:3000 running (8m) 3s ago 9m 84.7M - 9.4.7 954c08fa6188 a85bdbf5073d 2024-04-22T15:51:07.539 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.fkkejm smithi016 running (4m) 3s ago 4m 14.2M - 18.2.1 5be31c24972a 4153ac94cea0 2024-04-22T15:51:07.539 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.pgvphu smithi016 running (4m) 3s ago 4m 292M - 18.2.1 5be31c24972a 93dc072068a2 2024-04-22T15:51:07.539 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.dgqyes smithi027 running (4m) 3s ago 4m 17.3M - 18.2.1 5be31c24972a 41a9220430fd 2024-04-22T15:51:07.539 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.iobjds smithi027 running (4m) 3s ago 4m 241M - 18.2.1 5be31c24972a c1c8d6770f02 2024-04-22T15:51:07.539 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi016.rnsnnr smithi016 *:8443,9283,8765 running (90s) 3s ago 11m 501M - 19.0.0-3223-g5883d494 3d983c26e278 09b4fe285bc4 2024-04-22T15:51:07.539 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi027.ukdgzs smithi027 *:8443,9283,8765 running (39s) 3s ago 9m 450M - 19.0.0-3223-g5883d494 3d983c26e278 7548da9b5eb9 2024-04-22T15:51:07.539 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi016 smithi016 running (11m) 3s ago 11m 56.9M 2048M 18.2.1 5be31c24972a 72af40873cb6 2024-04-22T15:51:07.539 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi027 smithi027 running (8m) 3s ago 8m 43.7M 2048M 18.2.1 5be31c24972a 9f21ce7fa7ef 2024-04-22T15:51:07.539 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi016 smithi016 *:9100 running (23s) 3s ago 10m 8086k - 1.5.0 0da6a335fe13 35e46e4d2f3f 2024-04-22T15:51:07.539 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi027 smithi027 *:9100 running (10s) 3s ago 9m 3598k - 1.5.0 0da6a335fe13 e682a7672e9b 2024-04-22T15:51:07.539 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (8m) 3s ago 8m 180M 1217M 18.2.1 5be31c24972a 3c6ebc4ea041 2024-04-22T15:51:07.539 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (7m) 3s ago 7m 180M 1217M 18.2.1 5be31c24972a 6aa2e6796784 2024-04-22T15:51:07.539 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (7m) 3s ago 7m 156M 1217M 18.2.1 5be31c24972a 3d9d310aa9eb 2024-04-22T15:51:07.540 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi027 running (6m) 3s ago 6m 227M 2240M 18.2.1 5be31c24972a cdf9b7aaf865 2024-04-22T15:51:07.540 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (6m) 3s ago 6m 184M 2240M 18.2.1 5be31c24972a 629397d49c76 2024-04-22T15:51:07.540 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi027 running (6m) 3s ago 6m 173M 2240M 18.2.1 5be31c24972a 23adcf675b04 2024-04-22T15:51:07.540 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.smithi016 smithi016 *:9095 running (45s) 3s ago 9m 56.2M - 2.43.0 a07b618ecd1d 804a7d8595e4 2024-04-22T15:51:07.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.537+0000 7ff7ff984700 1 -- 172.21.15.16:0/812857757 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7ff7e406ea80 msgr2=0x7ff7e4070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:07.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.537+0000 7ff7ff984700 1 --2- 172.21.15.16:0/812857757 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7ff7e406ea80 0x7ff7e4070f40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8009fd0 tx=0x7ff7e8005940 comp rx=0 tx=0).stop 2024-04-22T15:51:07.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.537+0000 7ff7ff984700 1 -- 172.21.15.16:0/812857757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f8100cb0 msgr2=0x7ff7f8074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:07.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.537+0000 7ff7ff984700 1 --2- 172.21.15.16:0/812857757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f8100cb0 0x7ff7f8074b30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff7f400e8f0 tx=0x7ff7f400ec00 comp rx=0 tx=0).stop 2024-04-22T15:51:07.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.537+0000 7ff7ff984700 1 -- 172.21.15.16:0/812857757 shutdown_connections 2024-04-22T15:51:07.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.537+0000 7ff7ff984700 1 --2- 172.21.15.16:0/812857757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7f8100cb0 0x7ff7f8074b30 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:07.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.537+0000 7ff7ff984700 1 --2- 172.21.15.16:0/812857757 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7ff7e406ea80 0x7ff7e4070f40 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:07.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.537+0000 7ff7ff984700 1 --2- 172.21.15.16:0/812857757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8101eb0 0x7ff7f8073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:07.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.537+0000 7ff7ff984700 1 -- 172.21.15.16:0/812857757 >> 172.21.15.16:0/812857757 conn(0x7ff7f80fc230 msgr2=0x7ff7f81050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:07.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.537+0000 7ff7ff984700 1 -- 172.21.15.16:0/812857757 shutdown_connections 2024-04-22T15:51:07.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.537+0000 7ff7ff984700 1 -- 172.21.15.16:0/812857757 wait complete. 2024-04-22T15:51:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.648+0000 7f1adc66b700 1 -- 172.21.15.16:0/1897645208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad41040c0 msgr2=0x7f1ad41064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.648+0000 7f1adc66b700 1 --2- 172.21.15.16:0/1897645208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad41040c0 0x7f1ad41064b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4009910 tx=0x7f1ac4009c20 comp rx=0 tx=0).stop 2024-04-22T15:51:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.648+0000 7f1adc66b700 1 -- 172.21.15.16:0/1897645208 shutdown_connections 2024-04-22T15:51:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.648+0000 7f1adc66b700 1 --2- 172.21.15.16:0/1897645208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad41040c0 0x7f1ad41064b0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.648+0000 7f1adc66b700 1 --2- 172.21.15.16:0/1897645208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ad4101790 0x7f1ad4103b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.648+0000 7f1adc66b700 1 -- 172.21.15.16:0/1897645208 >> 172.21.15.16:0/1897645208 conn(0x7f1ad40fb120 msgr2=0x7f1ad40fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.648+0000 7f1adc66b700 1 -- 172.21.15.16:0/1897645208 shutdown_connections 2024-04-22T15:51:07.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.648+0000 7f1adc66b700 1 -- 172.21.15.16:0/1897645208 wait complete. 2024-04-22T15:51:07.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.649+0000 7f1adc66b700 1 Processor -- start 2024-04-22T15:51:07.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.649+0000 7f1adc66b700 1 -- start start 2024-04-22T15:51:07.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.649+0000 7f1adc66b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad4101790 0x7f1ad4196440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:07.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.649+0000 7f1adc66b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ad41040c0 0x7f1ad4196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:07.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.649+0000 7f1adc66b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad4196fa0 con 0x7f1ad4101790 2024-04-22T15:51:07.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.649+0000 7f1adc66b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad41970e0 con 0x7f1ad41040c0 2024-04-22T15:51:07.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.649+0000 7f1ada407700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad4101790 0x7f1ad4196440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:07.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.649+0000 7f1ada407700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad4101790 0x7f1ad4196440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57672/0 (socket says 172.21.15.16:57672) 2024-04-22T15:51:07.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.649+0000 7f1ada407700 1 -- 172.21.15.16:0/3427083089 learned_addr learned my addr 172.21.15.16:0/3427083089 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:51:07.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.649+0000 7f1ada407700 1 -- 172.21.15.16:0/3427083089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ad41040c0 msgr2=0x7f1ad4196980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:07.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.650+0000 7f1ad9c06700 1 --2- 172.21.15.16:0/3427083089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ad41040c0 0x7f1ad4196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:07.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.650+0000 7f1ada407700 1 --2- 172.21.15.16:0/3427083089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ad41040c0 0x7f1ad4196980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:07.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.650+0000 7f1ada407700 1 -- 172.21.15.16:0/3427083089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac40095f0 con 0x7f1ad4101790 2024-04-22T15:51:07.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.650+0000 7f1ada407700 1 --2- 172.21.15.16:0/3427083089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad4101790 0x7f1ad4196440 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1ad000e920 tx=0x7f1ad000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:07.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.650+0000 7f1acb7fe700 1 -- 172.21.15.16:0/3427083089 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ad0009790 con 0x7f1ad4101790 2024-04-22T15:51:07.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.650+0000 7f1acb7fe700 1 -- 172.21.15.16:0/3427083089 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1ad000cba0 con 0x7f1ad4101790 2024-04-22T15:51:07.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.650+0000 7f1acb7fe700 1 -- 172.21.15.16:0/3427083089 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ad0018630 con 0x7f1ad4101790 2024-04-22T15:51:07.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.650+0000 7f1adc66b700 1 -- 172.21.15.16:0/3427083089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad419bb90 con 0x7f1ad4101790 2024-04-22T15:51:07.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.650+0000 7f1adc66b700 1 -- 172.21.15.16:0/3427083089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad40ff860 con 0x7f1ad4101790 2024-04-22T15:51:07.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.652+0000 7f1adc66b700 1 -- 172.21.15.16:0/3427083089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad4190610 con 0x7f1ad4101790 2024-04-22T15:51:07.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.652+0000 7f1acb7fe700 1 -- 172.21.15.16:0/3427083089 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 92700+0+0 (secure 0 0 0) 0x7f1ad00188a0 con 0x7f1ad4101790 2024-04-22T15:51:07.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.653+0000 7f1acb7fe700 1 --2- 172.21.15.16:0/3427083089 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f1ac006ea70 0x7f1ac0070f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:07.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.653+0000 7f1acb7fe700 1 -- 172.21.15.16:0/3427083089 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f1ad0014070 con 0x7f1ad4101790 2024-04-22T15:51:07.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.653+0000 7f1ad9c06700 1 --2- 172.21.15.16:0/3427083089 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f1ac006ea70 0x7f1ac0070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:07.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.654+0000 7f1ad9c06700 1 --2- 172.21.15.16:0/3427083089 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f1ac006ea70 0x7f1ac0070f30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4000c00 tx=0x7f1ac4009f70 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:07.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:07.662+0000 7f1acb7fe700 1 -- 172.21.15.16:0/3427083089 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f1ad0053e20 con 0x7f1ad4101790 2024-04-22T15:51:07.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:07 smithi016 ceph-mon[26620]: pgmap v37: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 142 op/s 2024-04-22T15:51:07.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:07 smithi027 ceph-mon[34922]: pgmap v37: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 142 op/s 2024-04-22T15:51:08.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.072+0000 7f1adc66b700 1 -- 172.21.15.16:0/3427083089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1ad402ced0 con 0x7f1ad4101790 2024-04-22T15:51:08.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.072+0000 7f1acb7fe700 1 -- 172.21.15.16:0/3427083089 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f1ad0094020 con 0x7f1ad4101790 2024-04-22T15:51:08.076 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:51:08.076 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-22T15:51:08.076 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T15:51:08.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:51:08.076 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-22T15:51:08.076 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:51:08.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:51:08.076 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-22T15:51:08.076 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-22T15:51:08.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:51:08.076 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-22T15:51:08.076 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T15:51:08.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:51:08.077 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-22T15:51:08.077 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-04-22T15:51:08.077 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:51:08.077 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-22T15:51:08.077 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:51:08.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.075+0000 7f1adc66b700 1 -- 172.21.15.16:0/3427083089 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f1ac006ea70 msgr2=0x7f1ac0070f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:08.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.075+0000 7f1adc66b700 1 --2- 172.21.15.16:0/3427083089 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f1ac006ea70 0x7f1ac0070f30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4000c00 tx=0x7f1ac4009f70 comp rx=0 tx=0).stop 2024-04-22T15:51:08.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.075+0000 7f1adc66b700 1 -- 172.21.15.16:0/3427083089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad4101790 msgr2=0x7f1ad4196440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:08.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.075+0000 7f1adc66b700 1 --2- 172.21.15.16:0/3427083089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad4101790 0x7f1ad4196440 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1ad000e920 tx=0x7f1ad000ece0 comp rx=0 tx=0).stop 2024-04-22T15:51:08.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.075+0000 7f1adc66b700 1 -- 172.21.15.16:0/3427083089 shutdown_connections 2024-04-22T15:51:08.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.075+0000 7f1adc66b700 1 --2- 172.21.15.16:0/3427083089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad4101790 0x7f1ad4196440 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:08.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.075+0000 7f1adc66b700 1 --2- 172.21.15.16:0/3427083089 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f1ac006ea70 0x7f1ac0070f30 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:08.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.075+0000 7f1adc66b700 1 --2- 172.21.15.16:0/3427083089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ad41040c0 0x7f1ad4196980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:08.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.075+0000 7f1adc66b700 1 -- 172.21.15.16:0/3427083089 >> 172.21.15.16:0/3427083089 conn(0x7f1ad40fb120 msgr2=0x7f1ad40fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:08.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.075+0000 7f1adc66b700 1 -- 172.21.15.16:0/3427083089 shutdown_connections 2024-04-22T15:51:08.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.075+0000 7f1adc66b700 1 -- 172.21.15.16:0/3427083089 wait complete. 2024-04-22T15:51:08.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.190+0000 7f09f09b4700 1 -- 172.21.15.16:0/903094090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09ec101eb0 msgr2=0x7f09ec102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:08.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.190+0000 7f09f09b4700 1 --2- 172.21.15.16:0/903094090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09ec101eb0 0x7f09ec102310 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f09e0009910 tx=0x7f09e0009c20 comp rx=0 tx=0).stop 2024-04-22T15:51:08.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.191+0000 7f09f09b4700 1 -- 172.21.15.16:0/903094090 shutdown_connections 2024-04-22T15:51:08.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.191+0000 7f09f09b4700 1 --2- 172.21.15.16:0/903094090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09ec101eb0 0x7f09ec102310 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:08.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.191+0000 7f09f09b4700 1 --2- 172.21.15.16:0/903094090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09ec100cb0 0x7f09ec1010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:08.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.191+0000 7f09f09b4700 1 -- 172.21.15.16:0/903094090 >> 172.21.15.16:0/903094090 conn(0x7f09ec0fc230 msgr2=0x7f09ec0fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:08.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.191+0000 7f09f09b4700 1 -- 172.21.15.16:0/903094090 shutdown_connections 2024-04-22T15:51:08.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.191+0000 7f09f09b4700 1 -- 172.21.15.16:0/903094090 wait complete. 2024-04-22T15:51:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.191+0000 7f09f09b4700 1 Processor -- start 2024-04-22T15:51:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.191+0000 7f09f09b4700 1 -- start start 2024-04-22T15:51:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.192+0000 7f09f09b4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09ec100cb0 0x7f09ec074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.192+0000 7f09f09b4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09ec101eb0 0x7f09ec073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.192+0000 7f09f09b4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec0736c0 con 0x7f09ec100cb0 2024-04-22T15:51:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.192+0000 7f09f09b4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec073800 con 0x7f09ec101eb0 2024-04-22T15:51:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.192+0000 7f09ea59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09ec100cb0 0x7f09ec074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.192+0000 7f09ea59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09ec100cb0 0x7f09ec074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57696/0 (socket says 172.21.15.16:57696) 2024-04-22T15:51:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.192+0000 7f09ea59c700 1 -- 172.21.15.16:0/1149380048 learned_addr learned my addr 172.21.15.16:0/1149380048 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:51:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.192+0000 7f09e9d9b700 1 --2- 172.21.15.16:0/1149380048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09ec101eb0 0x7f09ec073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:08.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.192+0000 7f09ea59c700 1 -- 172.21.15.16:0/1149380048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09ec101eb0 msgr2=0x7f09ec073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:08.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.192+0000 7f09ea59c700 1 --2- 172.21.15.16:0/1149380048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09ec101eb0 0x7f09ec073180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:08.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.192+0000 7f09ea59c700 1 -- 172.21.15.16:0/1149380048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e00095f0 con 0x7f09ec100cb0 2024-04-22T15:51:08.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.192+0000 7f09e9d9b700 1 --2- 172.21.15.16:0/1149380048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09ec101eb0 0x7f09ec073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:51:08.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.192+0000 7f09ea59c700 1 --2- 172.21.15.16:0/1149380048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09ec100cb0 0x7f09ec074b30 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f09dc00e8f0 tx=0x7f09dc00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:08.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.193+0000 7f09db7fe700 1 -- 172.21.15.16:0/1149380048 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09dc009790 con 0x7f09ec100cb0 2024-04-22T15:51:08.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.193+0000 7f09db7fe700 1 -- 172.21.15.16:0/1149380048 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f09dc00c810 con 0x7f09ec100cb0 2024-04-22T15:51:08.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.193+0000 7f09f09b4700 1 -- 172.21.15.16:0/1149380048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09ec073ae0 con 0x7f09ec100cb0 2024-04-22T15:51:08.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.194+0000 7f09db7fe700 1 -- 172.21.15.16:0/1149380048 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09dc003d20 con 0x7f09ec100cb0 2024-04-22T15:51:08.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.194+0000 7f09f09b4700 1 -- 172.21.15.16:0/1149380048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09ec19e510 con 0x7f09ec100cb0 2024-04-22T15:51:08.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.195+0000 7f09f09b4700 1 -- 172.21.15.16:0/1149380048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09ec109810 con 0x7f09ec100cb0 2024-04-22T15:51:08.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.195+0000 7f09db7fe700 1 -- 172.21.15.16:0/1149380048 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 92700+0+0 (secure 0 0 0) 0x7f09dc018430 con 0x7f09ec100cb0 2024-04-22T15:51:08.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.195+0000 7f09db7fe700 1 --2- 172.21.15.16:0/1149380048 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f09d406eab0 0x7f09d4070f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:08.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.195+0000 7f09db7fe700 1 -- 172.21.15.16:0/1149380048 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f09dc014070 con 0x7f09ec100cb0 2024-04-22T15:51:08.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.195+0000 7f09e9d9b700 1 --2- 172.21.15.16:0/1149380048 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f09d406eab0 0x7f09d4070f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:08.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.196+0000 7f09e9d9b700 1 --2- 172.21.15.16:0/1149380048 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f09d406eab0 0x7f09d4070f70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f09e0009fd0 tx=0x7f09e0005940 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:08.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.200+0000 7f09db7fe700 1 -- 172.21.15.16:0/1149380048 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f09dc05bbe0 con 0x7f09ec100cb0 2024-04-22T15:51:08.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.487+0000 7f09f09b4700 1 -- 172.21.15.16:0/1149380048 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f09ec19eac0 con 0x7f09d406eab0 2024-04-22T15:51:08.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.489+0000 7f09db7fe700 1 -- 172.21.15.16:0/1149380048 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7f09ec19eac0 con 0x7f09d406eab0 2024-04-22T15:51:08.494 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:51:08.494 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:51:08.494 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-22T15:51:08.494 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T15:51:08.494 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-22T15:51:08.495 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-04-22T15:51:08.495 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-22T15:51:08.495 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "2/2 daemons upgraded", 2024-04-22T15:51:08.495 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading node-exporter daemons", 2024-04-22T15:51:08.495 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-22T15:51:08.495 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:51:08.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.491+0000 7f09f09b4700 1 -- 172.21.15.16:0/1149380048 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f09d406eab0 msgr2=0x7f09d4070f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:08.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.491+0000 7f09f09b4700 1 --2- 172.21.15.16:0/1149380048 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f09d406eab0 0x7f09d4070f70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f09e0009fd0 tx=0x7f09e0005940 comp rx=0 tx=0).stop 2024-04-22T15:51:08.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.491+0000 7f09f09b4700 1 -- 172.21.15.16:0/1149380048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09ec100cb0 msgr2=0x7f09ec074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:08.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.491+0000 7f09f09b4700 1 --2- 172.21.15.16:0/1149380048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09ec100cb0 0x7f09ec074b30 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f09dc00e8f0 tx=0x7f09dc00ec00 comp rx=0 tx=0).stop 2024-04-22T15:51:08.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.491+0000 7f09f09b4700 1 -- 172.21.15.16:0/1149380048 shutdown_connections 2024-04-22T15:51:08.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.491+0000 7f09f09b4700 1 --2- 172.21.15.16:0/1149380048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09ec100cb0 0x7f09ec074b30 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:08.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.491+0000 7f09f09b4700 1 --2- 172.21.15.16:0/1149380048 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f09d406eab0 0x7f09d4070f70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:08.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.491+0000 7f09f09b4700 1 --2- 172.21.15.16:0/1149380048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09ec101eb0 0x7f09ec073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:08.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.491+0000 7f09f09b4700 1 -- 172.21.15.16:0/1149380048 >> 172.21.15.16:0/1149380048 conn(0x7f09ec0fc230 msgr2=0x7f09ec1050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:08.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.491+0000 7f09f09b4700 1 -- 172.21.15.16:0/1149380048 shutdown_connections 2024-04-22T15:51:08.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:08.492+0000 7f09f09b4700 1 -- 172.21.15.16:0/1149380048 wait complete. 2024-04-22T15:51:08.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:08 smithi027 ceph-mon[34922]: from='client.14746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:08.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:08 smithi027 ceph-mon[34922]: from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:08.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:08 smithi027 ceph-mon[34922]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:08.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:08 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/3427083089' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:08.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:08 smithi016 ceph-mon[26620]: from='client.14746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:08.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:08 smithi016 ceph-mon[26620]: from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:08.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:08 smithi016 ceph-mon[26620]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:08.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:08 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/3427083089' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:09.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:09 smithi027 ceph-mon[34922]: pgmap v38: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 100 op/s 2024-04-22T15:51:09.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:09 smithi027 ceph-mon[34922]: from='client.14762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:09.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:09 smithi016 ceph-mon[26620]: pgmap v38: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 100 op/s 2024-04-22T15:51:09.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:09 smithi016 ceph-mon[26620]: from='client.14762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:11.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:11 smithi027 ceph-mon[34922]: pgmap v39: 65 pgs: 65 active+clean; 308 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 158 op/s 2024-04-22T15:51:11.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:11 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:11.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:11 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:11.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:11 smithi016 ceph-mon[26620]: pgmap v39: 65 pgs: 65 active+clean; 308 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 158 op/s 2024-04-22T15:51:11.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:11 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:11.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:11 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:13.225 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:13 smithi027 ceph-mon[34922]: pgmap v40: 65 pgs: 65 active+clean; 308 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 108 op/s 2024-04-22T15:51:13.225 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:13 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:51:13.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:13 smithi016 ceph-mon[26620]: pgmap v40: 65 pgs: 65 active+clean; 308 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 108 op/s 2024-04-22T15:51:13.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:13 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:51:15.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:15 smithi027 ceph-mon[34922]: pgmap v41: 65 pgs: 65 active+clean; 308 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 108 op/s 2024-04-22T15:51:15.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:15 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:15.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:15 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:15.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:15 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:51:15.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:15 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:51:15.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:15 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:15.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:15 smithi016 ceph-mon[26620]: pgmap v41: 65 pgs: 65 active+clean; 308 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 108 op/s 2024-04-22T15:51:15.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:15 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:15.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:15 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:15.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:15 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:51:15.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:15 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:51:15.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:15 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:16.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:16 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:51:16.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:16 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:16.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:16 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:16.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:16 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:16.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:16 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:16.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:16 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:16.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:16 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:16.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:16 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:16.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:16 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:16.340 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:16 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:51:16.340 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:16 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:16.340 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:16 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:16.340 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:16 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:16.341 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:16 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:16.341 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:16 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:16.341 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:16 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:16.341 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:16 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:16.341 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:16 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:17.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:17 smithi027 ceph-mon[34922]: pgmap v42: 65 pgs: 65 active+clean; 308 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 160 op/s 2024-04-22T15:51:17.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:17 smithi016 ceph-mon[26620]: pgmap v42: 65 pgs: 65 active+clean; 308 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 160 op/s 2024-04-22T15:51:18.294 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:18 smithi016 ceph-mon[26620]: Upgrade: Updating prometheus.smithi016 2024-04-22T15:51:18.295 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:18 smithi016 ceph-mon[26620]: Deploying daemon prometheus.smithi016 on smithi016 2024-04-22T15:51:18.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:18 smithi027 ceph-mon[34922]: Upgrade: Updating prometheus.smithi016 2024-04-22T15:51:18.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:18 smithi027 ceph-mon[34922]: Deploying daemon prometheus.smithi016 on smithi016 2024-04-22T15:51:19.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:19 smithi016 ceph-mon[26620]: pgmap v43: 65 pgs: 65 active+clean; 308 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 111 op/s 2024-04-22T15:51:19.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:19 smithi027 ceph-mon[34922]: pgmap v43: 65 pgs: 65 active+clean; 308 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 111 op/s 2024-04-22T15:51:21.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:21 smithi016 ceph-mon[26620]: pgmap v44: 65 pgs: 65 active+clean; 307 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 178 op/s 2024-04-22T15:51:21.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:21 smithi027 ceph-mon[34922]: pgmap v44: 65 pgs: 65 active+clean; 307 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 178 op/s 2024-04-22T15:51:23.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:23 smithi027 ceph-mon[34922]: pgmap v45: 65 pgs: 65 active+clean; 307 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 120 op/s 2024-04-22T15:51:23.377 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:23 smithi016 ceph-mon[26620]: pgmap v45: 65 pgs: 65 active+clean; 307 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 120 op/s 2024-04-22T15:51:24.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:24 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:24.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:24 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:24.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:24 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:51:24.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:24 smithi016 ceph-mon[26620]: pgmap v46: 65 pgs: 65 active+clean; 307 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 120 op/s 2024-04-22T15:51:25.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:24 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:25.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:24 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:25.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:24 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:51:25.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:24 smithi027 ceph-mon[34922]: pgmap v46: 65 pgs: 65 active+clean; 307 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 120 op/s 2024-04-22T15:51:27.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:26 smithi027 ceph-mon[34922]: pgmap v47: 65 pgs: 65 active+clean; 302 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 238 op/s 2024-04-22T15:51:27.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:26 smithi016 ceph-mon[26620]: pgmap v47: 65 pgs: 65 active+clean; 302 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 238 op/s 2024-04-22T15:51:29.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:28 smithi016 ceph-mon[26620]: pgmap v48: 65 pgs: 65 active+clean; 302 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 186 op/s 2024-04-22T15:51:29.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:28 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:51:29.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:28 smithi027 ceph-mon[34922]: pgmap v48: 65 pgs: 65 active+clean; 302 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 186 op/s 2024-04-22T15:51:29.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:28 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:51:30.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:29 smithi016 ceph-mon[26620]: mgrmap e32: smithi016.rnsnnr(active, since 91s), standbys: smithi027.ukdgzs 2024-04-22T15:51:30.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:29 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:30.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:29 smithi027 ceph-mon[34922]: mgrmap e32: smithi016.rnsnnr(active, since 91s), standbys: smithi027.ukdgzs 2024-04-22T15:51:30.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:29 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:31.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:30 smithi027 ceph-mon[34922]: pgmap v49: 65 pgs: 65 active+clean; 297 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 281 op/s 2024-04-22T15:51:31.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:30 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:31.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:30 smithi016 ceph-mon[26620]: pgmap v49: 65 pgs: 65 active+clean; 297 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 281 op/s 2024-04-22T15:51:31.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:30 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:31.506 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-04-22T15:51:31.506 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-22T15:51:31.832 DEBUG:teuthology.parallel:result is None 2024-04-22T15:51:32.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:32 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:32.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:32 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:32.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:32 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:32.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:32 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:33.507 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:33 smithi016 ceph-mon[26620]: pgmap v50: 65 pgs: 65 active+clean; 297 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 213 op/s 2024-04-22T15:51:33.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:33 smithi027 ceph-mon[34922]: pgmap v50: 65 pgs: 65 active+clean; 297 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 213 op/s 2024-04-22T15:51:35.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:35 smithi027 ceph-mon[34922]: pgmap v51: 65 pgs: 65 active+clean; 297 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 213 op/s 2024-04-22T15:51:35.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:35 smithi016 ceph-mon[26620]: pgmap v51: 65 pgs: 65 active+clean; 297 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 213 op/s 2024-04-22T15:51:37.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:36 smithi027 ceph-mon[34922]: pgmap v52: 65 pgs: 65 active+clean; 276 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 305 op/s 2024-04-22T15:51:37.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:36 smithi016 ceph-mon[26620]: pgmap v52: 65 pgs: 65 active+clean; 276 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 305 op/s 2024-04-22T15:51:37.986 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:37 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:37.986 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:37 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:37.987 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:37 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:51:37.987 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:37 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:51:37.987 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:37 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:37.987 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:37 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:51:37.987 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:37 smithi016 ceph-mon[26620]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:51:37.987 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:37 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:51:37.987 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:37 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:37.987 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:37 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:37.987 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:37 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:37.987 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:37 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:37.987 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:37 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:37.987 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:37 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:37.987 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:37 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:37.988 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:37 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:37.988 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:37 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:38.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:37 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:38.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:37 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:38.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:37 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:51:38.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:37 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:51:38.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:37 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:38.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:37 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:51:38.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:37 smithi027 ceph-mon[34922]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T15:51:38.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:37 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:51:38.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:37 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:38.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:37 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:38.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:37 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:38.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:37 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:38.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:37 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:38.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:37 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:38.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:37 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:38.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:37 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:38.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:37 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:38.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.603+0000 7f782eaa2700 1 -- 172.21.15.16:0/4110142412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7828101eb0 msgr2=0x7f7828102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:38.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.603+0000 7f782eaa2700 1 --2- 172.21.15.16:0/4110142412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7828101eb0 0x7f7828102310 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f781c009910 tx=0x7f781c009c20 comp rx=0 tx=0).stop 2024-04-22T15:51:38.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.603+0000 7f782eaa2700 1 -- 172.21.15.16:0/4110142412 shutdown_connections 2024-04-22T15:51:38.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.603+0000 7f782eaa2700 1 --2- 172.21.15.16:0/4110142412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7828101eb0 0x7f7828102310 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:38.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.603+0000 7f782eaa2700 1 --2- 172.21.15.16:0/4110142412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7828100cb0 0x7f78281010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:38.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.603+0000 7f782eaa2700 1 -- 172.21.15.16:0/4110142412 >> 172.21.15.16:0/4110142412 conn(0x7f78280fc230 msgr2=0x7f78280fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:38.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.603+0000 7f782eaa2700 1 -- 172.21.15.16:0/4110142412 shutdown_connections 2024-04-22T15:51:38.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.603+0000 7f782eaa2700 1 -- 172.21.15.16:0/4110142412 wait complete. 2024-04-22T15:51:38.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.604+0000 7f782eaa2700 1 Processor -- start 2024-04-22T15:51:38.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.604+0000 7f782eaa2700 1 -- start start 2024-04-22T15:51:38.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.604+0000 7f782eaa2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7828100cb0 0x7f78281964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:38.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.604+0000 7f782eaa2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7828101eb0 0x7f78281969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:38.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.604+0000 7f782eaa2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7828197010 con 0x7f7828101eb0 2024-04-22T15:51:38.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.604+0000 7f782eaa2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7828197150 con 0x7f7828100cb0 2024-04-22T15:51:38.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.604+0000 7f7827fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7828101eb0 0x7f78281969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:38.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.604+0000 7f7827fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7828101eb0 0x7f78281969f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40070/0 (socket says 172.21.15.16:40070) 2024-04-22T15:51:38.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.604+0000 7f782c83e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7828100cb0 0x7f78281964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:38.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.604+0000 7f7827fff700 1 -- 172.21.15.16:0/2039447915 learned_addr learned my addr 172.21.15.16:0/2039447915 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:51:38.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.604+0000 7f7827fff700 1 -- 172.21.15.16:0/2039447915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7828100cb0 msgr2=0x7f78281964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:38.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.604+0000 7f7827fff700 1 --2- 172.21.15.16:0/2039447915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7828100cb0 0x7f78281964b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:38.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.605+0000 7f7827fff700 1 -- 172.21.15.16:0/2039447915 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f781c0095f0 con 0x7f7828101eb0 2024-04-22T15:51:38.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.605+0000 7f782c83e700 1 --2- 172.21.15.16:0/2039447915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7828100cb0 0x7f78281964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:51:38.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.605+0000 7f7827fff700 1 --2- 172.21.15.16:0/2039447915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7828101eb0 0x7f78281969f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f781c000c00 tx=0x7f781c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:38.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.605+0000 7f7825ffb700 1 -- 172.21.15.16:0/2039447915 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f781c01d070 con 0x7f7828101eb0 2024-04-22T15:51:38.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.605+0000 7f782eaa2700 1 -- 172.21.15.16:0/2039447915 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f782819bba0 con 0x7f7828101eb0 2024-04-22T15:51:38.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.605+0000 7f7825ffb700 1 -- 172.21.15.16:0/2039447915 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f781c005dc0 con 0x7f7828101eb0 2024-04-22T15:51:38.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.605+0000 7f7825ffb700 1 -- 172.21.15.16:0/2039447915 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f781c0226a0 con 0x7f7828101eb0 2024-04-22T15:51:38.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.605+0000 7f782eaa2700 1 -- 172.21.15.16:0/2039447915 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f782819c090 con 0x7f7828101eb0 2024-04-22T15:51:38.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.606+0000 7f782eaa2700 1 -- 172.21.15.16:0/2039447915 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7828109810 con 0x7f7828101eb0 2024-04-22T15:51:38.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.607+0000 7f7825ffb700 1 -- 172.21.15.16:0/2039447915 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f781c0038f0 con 0x7f7828101eb0 2024-04-22T15:51:38.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.607+0000 7f7825ffb700 1 --2- 172.21.15.16:0/2039447915 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f781006e940 0x7f7810070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:38.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.607+0000 7f7825ffb700 1 -- 172.21.15.16:0/2039447915 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f781c08fd30 con 0x7f7828101eb0 2024-04-22T15:51:38.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.607+0000 7f782c83e700 1 --2- 172.21.15.16:0/2039447915 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f781006e940 0x7f7810070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:38.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.608+0000 7f782c83e700 1 --2- 172.21.15.16:0/2039447915 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f781006e940 0x7f7810070e00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7828101d10 tx=0x7f7818009260 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:38.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.612+0000 7f7825ffb700 1 -- 172.21.15.16:0/2039447915 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f781c05cb40 con 0x7f7828101eb0 2024-04-22T15:51:38.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.843+0000 7f782eaa2700 1 -- 172.21.15.16:0/2039447915 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7828106800 con 0x7f781006e940 2024-04-22T15:51:38.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.844+0000 7f7825ffb700 1 -- 172.21.15.16:0/2039447915 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7f7828106800 con 0x7f781006e940 2024-04-22T15:51:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.846+0000 7f782eaa2700 1 -- 172.21.15.16:0/2039447915 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f781006e940 msgr2=0x7f7810070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.846+0000 7f782eaa2700 1 --2- 172.21.15.16:0/2039447915 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f781006e940 0x7f7810070e00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7828101d10 tx=0x7f7818009260 comp rx=0 tx=0).stop 2024-04-22T15:51:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.846+0000 7f782eaa2700 1 -- 172.21.15.16:0/2039447915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7828101eb0 msgr2=0x7f78281969f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.846+0000 7f782eaa2700 1 --2- 172.21.15.16:0/2039447915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7828101eb0 0x7f78281969f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f781c000c00 tx=0x7f781c00f690 comp rx=0 tx=0).stop 2024-04-22T15:51:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.846+0000 7f782eaa2700 1 -- 172.21.15.16:0/2039447915 shutdown_connections 2024-04-22T15:51:38.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.846+0000 7f782eaa2700 1 --2- 172.21.15.16:0/2039447915 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f781006e940 0x7f7810070e00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:38.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.846+0000 7f782eaa2700 1 --2- 172.21.15.16:0/2039447915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7828100cb0 0x7f78281964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:38.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.846+0000 7f782eaa2700 1 --2- 172.21.15.16:0/2039447915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7828101eb0 0x7f78281969f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:38.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.846+0000 7f782eaa2700 1 -- 172.21.15.16:0/2039447915 >> 172.21.15.16:0/2039447915 conn(0x7f78280fc230 msgr2=0x7f78281050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:38.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.846+0000 7f782eaa2700 1 -- 172.21.15.16:0/2039447915 shutdown_connections 2024-04-22T15:51:38.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.846+0000 7f782eaa2700 1 -- 172.21.15.16:0/2039447915 wait complete. 2024-04-22T15:51:38.860 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:51:38.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.962+0000 7fef5b448700 1 -- 172.21.15.16:0/1245897330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef540ffbb0 msgr2=0x7fef540fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:38.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.962+0000 7fef5b448700 1 --2- 172.21.15.16:0/1245897330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef540ffbb0 0x7fef540fffd0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fef48009910 tx=0x7fef48009c20 comp rx=0 tx=0).stop 2024-04-22T15:51:38.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.962+0000 7fef5b448700 1 -- 172.21.15.16:0/1245897330 shutdown_connections 2024-04-22T15:51:38.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.962+0000 7fef5b448700 1 --2- 172.21.15.16:0/1245897330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef54100db0 0x7fef54101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:38.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.962+0000 7fef5b448700 1 --2- 172.21.15.16:0/1245897330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef540ffbb0 0x7fef540fffd0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:38.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.962+0000 7fef5b448700 1 -- 172.21.15.16:0/1245897330 >> 172.21.15.16:0/1245897330 conn(0x7fef540fb150 msgr2=0x7fef540fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:38.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.962+0000 7fef5b448700 1 -- 172.21.15.16:0/1245897330 shutdown_connections 2024-04-22T15:51:38.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.962+0000 7fef5b448700 1 -- 172.21.15.16:0/1245897330 wait complete. 2024-04-22T15:51:38.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.963+0000 7fef5b448700 1 Processor -- start 2024-04-22T15:51:38.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.963+0000 7fef5b448700 1 -- start start 2024-04-22T15:51:38.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.963+0000 7fef5b448700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef540ffbb0 0x7fef54196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:38.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.963+0000 7fef5b448700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef54100db0 0x7fef54196a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:38.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.963+0000 7fef5b448700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef54197060 con 0x7fef540ffbb0 2024-04-22T15:51:38.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.963+0000 7fef5b448700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef541971a0 con 0x7fef54100db0 2024-04-22T15:51:38.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.963+0000 7fef591e4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef540ffbb0 0x7fef54196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:38.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.963+0000 7fef591e4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef540ffbb0 0x7fef54196500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40098/0 (socket says 172.21.15.16:40098) 2024-04-22T15:51:38.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.963+0000 7fef591e4700 1 -- 172.21.15.16:0/2410554283 learned_addr learned my addr 172.21.15.16:0/2410554283 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:51:38.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.963+0000 7fef589e3700 1 --2- 172.21.15.16:0/2410554283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef54100db0 0x7fef54196a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:38.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.963+0000 7fef591e4700 1 -- 172.21.15.16:0/2410554283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef54100db0 msgr2=0x7fef54196a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:38.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.963+0000 7fef591e4700 1 --2- 172.21.15.16:0/2410554283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef54100db0 0x7fef54196a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:38.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.963+0000 7fef591e4700 1 -- 172.21.15.16:0/2410554283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef480095f0 con 0x7fef540ffbb0 2024-04-22T15:51:38.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.964+0000 7fef591e4700 1 --2- 172.21.15.16:0/2410554283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef540ffbb0 0x7fef54196500 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fef48000c00 tx=0x7fef4800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:38.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.964+0000 7fef467fc700 1 -- 172.21.15.16:0/2410554283 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef4801d070 con 0x7fef540ffbb0 2024-04-22T15:51:38.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.964+0000 7fef467fc700 1 -- 172.21.15.16:0/2410554283 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fef48005dc0 con 0x7fef540ffbb0 2024-04-22T15:51:38.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.964+0000 7fef5b448700 1 -- 172.21.15.16:0/2410554283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef5419bbf0 con 0x7fef540ffbb0 2024-04-22T15:51:38.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.965+0000 7fef467fc700 1 -- 172.21.15.16:0/2410554283 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef48022470 con 0x7fef540ffbb0 2024-04-22T15:51:38.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.965+0000 7fef5b448700 1 -- 172.21.15.16:0/2410554283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef5419c0e0 con 0x7fef540ffbb0 2024-04-22T15:51:38.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.965+0000 7fef467fc700 1 -- 172.21.15.16:0/2410554283 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7fef4800fa70 con 0x7fef540ffbb0 2024-04-22T15:51:38.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.965+0000 7fef5b448700 1 -- 172.21.15.16:0/2410554283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef54108730 con 0x7fef540ffbb0 2024-04-22T15:51:38.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.966+0000 7fef467fc700 1 --2- 172.21.15.16:0/2410554283 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fef4006e940 0x7fef40070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:38.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.966+0000 7fef467fc700 1 -- 172.21.15.16:0/2410554283 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fef4808fd90 con 0x7fef540ffbb0 2024-04-22T15:51:38.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.966+0000 7fef589e3700 1 --2- 172.21.15.16:0/2410554283 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fef4006e940 0x7fef40070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:38.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.967+0000 7fef589e3700 1 --2- 172.21.15.16:0/2410554283 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fef4006e940 0x7fef40070e00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fef50009c60 tx=0x7fef50009260 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:38.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:38.971+0000 7fef467fc700 1 -- 172.21.15.16:0/2410554283 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7fef4805cba0 con 0x7fef540ffbb0 2024-04-22T15:51:39.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:38 smithi016 ceph-mon[26620]: pgmap v53: 65 pgs: 65 active+clean; 276 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 186 op/s 2024-04-22T15:51:39.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.253+0000 7fef5b448700 1 -- 172.21.15.16:0/2410554283 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fef54069520 con 0x7fef4006e940 2024-04-22T15:51:39.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.254+0000 7fef467fc700 1 -- 172.21.15.16:0/2410554283 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7fef54069520 con 0x7fef4006e940 2024-04-22T15:51:39.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.256+0000 7fef5b448700 1 -- 172.21.15.16:0/2410554283 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fef4006e940 msgr2=0x7fef40070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:39.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.256+0000 7fef5b448700 1 --2- 172.21.15.16:0/2410554283 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fef4006e940 0x7fef40070e00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fef50009c60 tx=0x7fef50009260 comp rx=0 tx=0).stop 2024-04-22T15:51:39.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.257+0000 7fef5b448700 1 -- 172.21.15.16:0/2410554283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef540ffbb0 msgr2=0x7fef54196500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:39.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.257+0000 7fef5b448700 1 --2- 172.21.15.16:0/2410554283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef540ffbb0 0x7fef54196500 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fef48000c00 tx=0x7fef4800f690 comp rx=0 tx=0).stop 2024-04-22T15:51:39.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.257+0000 7fef5b448700 1 -- 172.21.15.16:0/2410554283 shutdown_connections 2024-04-22T15:51:39.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.257+0000 7fef5b448700 1 --2- 172.21.15.16:0/2410554283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef540ffbb0 0x7fef54196500 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:39.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.257+0000 7fef5b448700 1 --2- 172.21.15.16:0/2410554283 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fef4006e940 0x7fef40070e00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:39.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.257+0000 7fef5b448700 1 --2- 172.21.15.16:0/2410554283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef54100db0 0x7fef54196a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:39.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.257+0000 7fef5b448700 1 -- 172.21.15.16:0/2410554283 >> 172.21.15.16:0/2410554283 conn(0x7fef540fb150 msgr2=0x7fef54068e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:39.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.257+0000 7fef5b448700 1 -- 172.21.15.16:0/2410554283 shutdown_connections 2024-04-22T15:51:39.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.257+0000 7fef5b448700 1 -- 172.21.15.16:0/2410554283 wait complete. 2024-04-22T15:51:39.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:38 smithi027 ceph-mon[34922]: pgmap v53: 65 pgs: 65 active+clean; 276 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 186 op/s 2024-04-22T15:51:39.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.366+0000 7f44ff736700 1 -- 172.21.15.16:0/3495081898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f8104130 msgr2=0x7f44f8106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.366+0000 7f44ff736700 1 --2- 172.21.15.16:0/3495081898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f8104130 0x7f44f8106520 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f44e8009910 tx=0x7f44e8009c20 comp rx=0 tx=0).stop 2024-04-22T15:51:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.366+0000 7f44ff736700 1 -- 172.21.15.16:0/3495081898 shutdown_connections 2024-04-22T15:51:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.366+0000 7f44ff736700 1 --2- 172.21.15.16:0/3495081898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f8104130 0x7f44f8106520 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.366+0000 7f44ff736700 1 --2- 172.21.15.16:0/3495081898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44f8101800 0x7f44f8103bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.366+0000 7f44ff736700 1 -- 172.21.15.16:0/3495081898 >> 172.21.15.16:0/3495081898 conn(0x7f44f80fb190 msgr2=0x7f44f80fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.367+0000 7f44ff736700 1 -- 172.21.15.16:0/3495081898 shutdown_connections 2024-04-22T15:51:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.367+0000 7f44ff736700 1 -- 172.21.15.16:0/3495081898 wait complete. 2024-04-22T15:51:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.367+0000 7f44ff736700 1 Processor -- start 2024-04-22T15:51:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.367+0000 7f44ff736700 1 -- start start 2024-04-22T15:51:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.367+0000 7f44ff736700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44f8101800 0x7f44f8101160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.367+0000 7f44ff736700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f8104130 0x7f44f80ff7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.367+0000 7f44ff736700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f80ffcf0 con 0x7f44f8104130 2024-04-22T15:51:39.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.367+0000 7f44ff736700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f80ffe30 con 0x7f44f8101800 2024-04-22T15:51:39.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.368+0000 7f44fccd1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f8104130 0x7f44f80ff7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:39.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.368+0000 7f44fccd1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f8104130 0x7f44f80ff7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40124/0 (socket says 172.21.15.16:40124) 2024-04-22T15:51:39.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.368+0000 7f44fd4d2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44f8101800 0x7f44f8101160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:39.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.368+0000 7f44fccd1700 1 -- 172.21.15.16:0/1723280916 learned_addr learned my addr 172.21.15.16:0/1723280916 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:51:39.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.368+0000 7f44fccd1700 1 -- 172.21.15.16:0/1723280916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44f8101800 msgr2=0x7f44f8101160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:39.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.368+0000 7f44fccd1700 1 --2- 172.21.15.16:0/1723280916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44f8101800 0x7f44f8101160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:39.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.368+0000 7f44fccd1700 1 -- 172.21.15.16:0/1723280916 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e80095f0 con 0x7f44f8104130 2024-04-22T15:51:39.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.368+0000 7f44fd4d2700 1 --2- 172.21.15.16:0/1723280916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44f8101800 0x7f44f8101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:51:39.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.368+0000 7f44fccd1700 1 --2- 172.21.15.16:0/1723280916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f8104130 0x7f44f80ff7b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f44e8000c00 tx=0x7f44e800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:39.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.369+0000 7f44ee7fc700 1 -- 172.21.15.16:0/1723280916 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44e801d070 con 0x7f44f8104130 2024-04-22T15:51:39.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.369+0000 7f44ff736700 1 -- 172.21.15.16:0/1723280916 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f81000b0 con 0x7f44f8104130 2024-04-22T15:51:39.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.369+0000 7f44ee7fc700 1 -- 172.21.15.16:0/1723280916 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f44e8005dc0 con 0x7f44f8104130 2024-04-22T15:51:39.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.369+0000 7f44ee7fc700 1 -- 172.21.15.16:0/1723280916 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44e80226a0 con 0x7f44f8104130 2024-04-22T15:51:39.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.369+0000 7f44ff736700 1 -- 172.21.15.16:0/1723280916 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f81005a0 con 0x7f44f8104130 2024-04-22T15:51:39.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.370+0000 7f44ff736700 1 -- 172.21.15.16:0/1723280916 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f8190610 con 0x7f44f8104130 2024-04-22T15:51:39.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.371+0000 7f44ee7fc700 1 -- 172.21.15.16:0/1723280916 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f44e80038f0 con 0x7f44f8104130 2024-04-22T15:51:39.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.371+0000 7f44ee7fc700 1 --2- 172.21.15.16:0/1723280916 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f44e406e940 0x7f44e4070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:39.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.371+0000 7f44ee7fc700 1 -- 172.21.15.16:0/1723280916 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f44e808fdf0 con 0x7f44f8104130 2024-04-22T15:51:39.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.371+0000 7f44fd4d2700 1 --2- 172.21.15.16:0/1723280916 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f44e406e940 0x7f44e4070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:39.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.372+0000 7f44fd4d2700 1 --2- 172.21.15.16:0/1723280916 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f44e406e940 0x7f44e4070e00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f44f4011490 tx=0x7f44f4009260 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:39.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.376+0000 7f44ee7fc700 1 -- 172.21.15.16:0/1723280916 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f44e805cc00 con 0x7f44f8104130 2024-04-22T15:51:39.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.633+0000 7f44ff736700 1 -- 172.21.15.16:0/1723280916 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f44f80611d0 con 0x7f44e406e940 2024-04-22T15:51:39.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.643+0000 7f44ee7fc700 1 -- 172.21.15.16:0/1723280916 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f44f80611d0 con 0x7f44e406e940 2024-04-22T15:51:39.647 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:51:39.647 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.smithi016 smithi016 *:9093,9094 running (9m) 9s ago 10m 25.0M - 0.25.0 c8568f914cd2 fab40987f1b0 2024-04-22T15:51:39.647 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi016 smithi016 running (10m) 9s ago 10m 7826k - 18.2.1 5be31c24972a f8b627d3f41e 2024-04-22T15:51:39.647 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi027 smithi027 running (9m) 35s ago 9m 7776k - 18.2.1 5be31c24972a f00fd40b4220 2024-04-22T15:51:39.647 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi016 smithi016 running (10m) 9s ago 10m 6660k - 18.2.1 5be31c24972a 6b93db1fa0d9 2024-04-22T15:51:39.647 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi027 smithi027 running (9m) 35s ago 9m 6665k - 18.2.1 5be31c24972a d3bd439f30a0 2024-04-22T15:51:39.647 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.smithi016 smithi016 *:3000 running (8m) 9s ago 10m 85.2M - 9.4.7 954c08fa6188 a85bdbf5073d 2024-04-22T15:51:39.647 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.fkkejm smithi016 running (5m) 9s ago 5m 14.4M - 18.2.1 5be31c24972a 4153ac94cea0 2024-04-22T15:51:39.647 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.pgvphu smithi016 running (5m) 9s ago 5m 219M - 18.2.1 5be31c24972a 93dc072068a2 2024-04-22T15:51:39.647 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.dgqyes smithi027 running (4m) 35s ago 4m 17.3M - 18.2.1 5be31c24972a 41a9220430fd 2024-04-22T15:51:39.647 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.iobjds smithi027 running (5m) 35s ago 5m 241M - 18.2.1 5be31c24972a c1c8d6770f02 2024-04-22T15:51:39.647 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi016.rnsnnr smithi016 *:8443,9283,8765 running (2m) 9s ago 12m 504M - 19.0.0-3223-g5883d494 3d983c26e278 09b4fe285bc4 2024-04-22T15:51:39.647 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi027.ukdgzs smithi027 *:8443,9283,8765 running (71s) 35s ago 9m 450M - 19.0.0-3223-g5883d494 3d983c26e278 7548da9b5eb9 2024-04-22T15:51:39.648 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi016 smithi016 running (12m) 9s ago 12m 58.6M 2048M 18.2.1 5be31c24972a 72af40873cb6 2024-04-22T15:51:39.648 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi027 smithi027 running (9m) 35s ago 9m 43.7M 2048M 18.2.1 5be31c24972a 9f21ce7fa7ef 2024-04-22T15:51:39.648 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi016 smithi016 *:9100 running (55s) 9s ago 10m 8116k - 1.5.0 0da6a335fe13 35e46e4d2f3f 2024-04-22T15:51:39.648 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi027 smithi027 *:9100 running (42s) 35s ago 9m 3598k - 1.5.0 0da6a335fe13 e682a7672e9b 2024-04-22T15:51:39.648 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (8m) 9s ago 8m 190M 1217M 18.2.1 5be31c24972a 3c6ebc4ea041 2024-04-22T15:51:39.648 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (8m) 9s ago 8m 192M 1217M 18.2.1 5be31c24972a 6aa2e6796784 2024-04-22T15:51:39.648 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (7m) 9s ago 7m 167M 1217M 18.2.1 5be31c24972a 3d9d310aa9eb 2024-04-22T15:51:39.648 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi027 running (7m) 35s ago 7m 227M 2240M 18.2.1 5be31c24972a cdf9b7aaf865 2024-04-22T15:51:39.648 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (7m) 35s ago 7m 184M 2240M 18.2.1 5be31c24972a 629397d49c76 2024-04-22T15:51:39.648 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi027 running (6m) 35s ago 6m 173M 2240M 18.2.1 5be31c24972a 23adcf675b04 2024-04-22T15:51:39.648 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.smithi016 smithi016 *:9095 running (16s) 9s ago 10m 48.0M - 2.43.0 a07b618ecd1d 373eaadee82d 2024-04-22T15:51:39.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.646+0000 7f44ff736700 1 -- 172.21.15.16:0/1723280916 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f44e406e940 msgr2=0x7f44e4070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:39.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.646+0000 7f44ff736700 1 --2- 172.21.15.16:0/1723280916 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f44e406e940 0x7f44e4070e00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f44f4011490 tx=0x7f44f4009260 comp rx=0 tx=0).stop 2024-04-22T15:51:39.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.646+0000 7f44ff736700 1 -- 172.21.15.16:0/1723280916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f8104130 msgr2=0x7f44f80ff7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:39.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.646+0000 7f44ff736700 1 --2- 172.21.15.16:0/1723280916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f8104130 0x7f44f80ff7b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f44e8000c00 tx=0x7f44e800f690 comp rx=0 tx=0).stop 2024-04-22T15:51:39.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.646+0000 7f44ff736700 1 -- 172.21.15.16:0/1723280916 shutdown_connections 2024-04-22T15:51:39.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.646+0000 7f44ff736700 1 --2- 172.21.15.16:0/1723280916 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f44e406e940 0x7f44e4070e00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:39.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.646+0000 7f44ff736700 1 --2- 172.21.15.16:0/1723280916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44f8101800 0x7f44f8101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:39.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.646+0000 7f44ff736700 1 --2- 172.21.15.16:0/1723280916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f8104130 0x7f44f80ff7b0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:39.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.646+0000 7f44ff736700 1 -- 172.21.15.16:0/1723280916 >> 172.21.15.16:0/1723280916 conn(0x7f44f80fb190 msgr2=0x7f44f80fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:39.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.646+0000 7f44ff736700 1 -- 172.21.15.16:0/1723280916 shutdown_connections 2024-04-22T15:51:39.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.646+0000 7f44ff736700 1 -- 172.21.15.16:0/1723280916 wait complete. 2024-04-22T15:51:39.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.754+0000 7fc189a43700 1 -- 172.21.15.16:0/3663284435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1841040c0 msgr2=0x7fc1841064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:39.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.754+0000 7fc189a43700 1 --2- 172.21.15.16:0/3663284435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1841040c0 0x7fc1841064b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc16c009910 tx=0x7fc16c009c20 comp rx=0 tx=0).stop 2024-04-22T15:51:39.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.754+0000 7fc189a43700 1 -- 172.21.15.16:0/3663284435 shutdown_connections 2024-04-22T15:51:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.754+0000 7fc189a43700 1 --2- 172.21.15.16:0/3663284435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1841040c0 0x7fc1841064b0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.754+0000 7fc189a43700 1 --2- 172.21.15.16:0/3663284435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc184101790 0x7fc184103b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.754+0000 7fc189a43700 1 -- 172.21.15.16:0/3663284435 >> 172.21.15.16:0/3663284435 conn(0x7fc1840fb120 msgr2=0x7fc1840fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.754+0000 7fc189a43700 1 -- 172.21.15.16:0/3663284435 shutdown_connections 2024-04-22T15:51:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.754+0000 7fc189a43700 1 -- 172.21.15.16:0/3663284435 wait complete. 2024-04-22T15:51:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.755+0000 7fc189a43700 1 Processor -- start 2024-04-22T15:51:39.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.755+0000 7fc189a43700 1 -- start start 2024-04-22T15:51:39.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.755+0000 7fc189a43700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc184101790 0x7fc184196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:39.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.755+0000 7fc189a43700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1841040c0 0x7fc184196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:39.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.755+0000 7fc189a43700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc184196fb0 con 0x7fc1841040c0 2024-04-22T15:51:39.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.755+0000 7fc189a43700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1841970f0 con 0x7fc184101790 2024-04-22T15:51:39.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.755+0000 7fc1827fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1841040c0 0x7fc184196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:39.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.755+0000 7fc182ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc184101790 0x7fc184196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:39.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.755+0000 7fc1827fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1841040c0 0x7fc184196990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40154/0 (socket says 172.21.15.16:40154) 2024-04-22T15:51:39.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.755+0000 7fc1827fc700 1 -- 172.21.15.16:0/1574022607 learned_addr learned my addr 172.21.15.16:0/1574022607 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:51:39.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.755+0000 7fc182ffd700 1 -- 172.21.15.16:0/1574022607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1841040c0 msgr2=0x7fc184196990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:39.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.755+0000 7fc182ffd700 1 --2- 172.21.15.16:0/1574022607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1841040c0 0x7fc184196990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:39.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.755+0000 7fc182ffd700 1 -- 172.21.15.16:0/1574022607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16c0095f0 con 0x7fc184101790 2024-04-22T15:51:39.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.755+0000 7fc1827fc700 1 --2- 172.21.15.16:0/1574022607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1841040c0 0x7fc184196990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:51:39.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.756+0000 7fc182ffd700 1 --2- 172.21.15.16:0/1574022607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc184101790 0x7fc184196450 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc17400ef10 tx=0x7fc17400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:39.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.756+0000 7fc188a41700 1 -- 172.21.15.16:0/1574022607 <== mon.1 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc17400c9f0 con 0x7fc184101790 2024-04-22T15:51:39.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.756+0000 7fc189a43700 1 -- 172.21.15.16:0/1574022607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc18419bb40 con 0x7fc184101790 2024-04-22T15:51:39.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.756+0000 7fc188a41700 1 -- 172.21.15.16:0/1574022607 <== mon.1 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc17400f040 con 0x7fc184101790 2024-04-22T15:51:39.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.756+0000 7fc188a41700 1 -- 172.21.15.16:0/1574022607 <== mon.1 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1740146b0 con 0x7fc184101790 2024-04-22T15:51:39.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.756+0000 7fc189a43700 1 -- 172.21.15.16:0/1574022607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc18419c090 con 0x7fc184101790 2024-04-22T15:51:39.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.757+0000 7fc189a43700 1 -- 172.21.15.16:0/1574022607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc184190610 con 0x7fc184101790 2024-04-22T15:51:39.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.758+0000 7fc188a41700 1 -- 172.21.15.16:0/1574022607 <== mon.1 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7fc174007890 con 0x7fc184101790 2024-04-22T15:51:39.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.758+0000 7fc188a41700 1 --2- 172.21.15.16:0/1574022607 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fc17006e940 0x7fc170070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:39.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.758+0000 7fc188a41700 1 -- 172.21.15.16:0/1574022607 <== mon.1 v2:172.21.15.27:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fc17408ec70 con 0x7fc184101790 2024-04-22T15:51:39.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.758+0000 7fc1827fc700 1 --2- 172.21.15.16:0/1574022607 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fc17006e940 0x7fc170070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:39.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.759+0000 7fc1827fc700 1 --2- 172.21.15.16:0/1574022607 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fc17006e940 0x7fc170070e00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc16c009fd0 tx=0x7fc16c005940 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:39.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:39.764+0000 7fc188a41700 1 -- 172.21.15.16:0/1574022607 <== mon.1 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7fc174095080 con 0x7fc184101790 2024-04-22T15:51:39.787 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-04-22T15:51:39.787 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-04-22T15:51:40.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.121+0000 7fc189a43700 1 -- 172.21.15.16:0/1574022607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc184061a40 con 0x7fc184101790 2024-04-22T15:51:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.121+0000 7fc188a41700 1 -- 172.21.15.16:0/1574022607 <== mon.1 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7fc17405b9e0 con 0x7fc184101790 2024-04-22T15:51:40.168 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:51:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-22T15:51:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T15:51:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:51:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-22T15:51:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:51:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:51:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-22T15:51:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-22T15:51:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:51:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-22T15:51:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T15:51:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:51:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-22T15:51:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-04-22T15:51:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:51:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-22T15:51:40.170 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:51:40.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.124+0000 7fc189a43700 1 -- 172.21.15.16:0/1574022607 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fc17006e940 msgr2=0x7fc170070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:40.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.124+0000 7fc189a43700 1 --2- 172.21.15.16:0/1574022607 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fc17006e940 0x7fc170070e00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc16c009fd0 tx=0x7fc16c005940 comp rx=0 tx=0).stop 2024-04-22T15:51:40.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.124+0000 7fc189a43700 1 -- 172.21.15.16:0/1574022607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc184101790 msgr2=0x7fc184196450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:40.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.124+0000 7fc189a43700 1 --2- 172.21.15.16:0/1574022607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc184101790 0x7fc184196450 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc17400ef10 tx=0x7fc17400c3a0 comp rx=0 tx=0).stop 2024-04-22T15:51:40.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.124+0000 7fc189a43700 1 -- 172.21.15.16:0/1574022607 shutdown_connections 2024-04-22T15:51:40.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.124+0000 7fc189a43700 1 --2- 172.21.15.16:0/1574022607 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fc17006e940 0x7fc170070e00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:40.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.124+0000 7fc189a43700 1 --2- 172.21.15.16:0/1574022607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc184101790 0x7fc184196450 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:40.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.124+0000 7fc189a43700 1 --2- 172.21.15.16:0/1574022607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1841040c0 0x7fc184196990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:40.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.124+0000 7fc189a43700 1 -- 172.21.15.16:0/1574022607 >> 172.21.15.16:0/1574022607 conn(0x7fc1840fb120 msgr2=0x7fc1840fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:40.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.124+0000 7fc189a43700 1 -- 172.21.15.16:0/1574022607 shutdown_connections 2024-04-22T15:51:40.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.124+0000 7fc189a43700 1 -- 172.21.15.16:0/1574022607 wait complete. 2024-04-22T15:51:40.171 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:39 smithi016 ceph-mon[26620]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:40.171 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:39 smithi016 ceph-mon[26620]: Upgrade: Updating alertmanager.smithi016 2024-04-22T15:51:40.171 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:39 smithi016 ceph-mon[26620]: Deploying daemon alertmanager.smithi016 on smithi016 2024-04-22T15:51:40.172 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:39 smithi016 ceph-mon[26620]: from='client.14770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:40.172 DEBUG:teuthology.parallel:result is None 2024-04-22T15:51:40.173 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-22T15:51:40.208 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-22T15:51:40.208 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2024-04-22T15:51:40.230 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:39 smithi027 ceph-mon[34922]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:40.230 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:39 smithi027 ceph-mon[34922]: Upgrade: Updating alertmanager.smithi016 2024-04-22T15:51:40.230 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:39 smithi027 ceph-mon[34922]: Deploying daemon alertmanager.smithi016 on smithi016 2024-04-22T15:51:40.230 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:39 smithi027 ceph-mon[34922]: from='client.14770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:40.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.234+0000 7fc3158d9700 1 -- 172.21.15.16:0/1130926489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310100db0 msgr2=0x7fc310101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:40.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.234+0000 7fc3158d9700 1 --2- 172.21.15.16:0/1130926489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310100db0 0x7fc310101210 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc304009910 tx=0x7fc304009c20 comp rx=0 tx=0).stop 2024-04-22T15:51:40.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.234+0000 7fc3158d9700 1 -- 172.21.15.16:0/1130926489 shutdown_connections 2024-04-22T15:51:40.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.234+0000 7fc3158d9700 1 --2- 172.21.15.16:0/1130926489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310100db0 0x7fc310101210 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.234+0000 7fc3158d9700 1 --2- 172.21.15.16:0/1130926489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3100ffbb0 0x7fc3100fffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.234+0000 7fc3158d9700 1 -- 172.21.15.16:0/1130926489 >> 172.21.15.16:0/1130926489 conn(0x7fc3100fb150 msgr2=0x7fc3100fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.234+0000 7fc3158d9700 1 -- 172.21.15.16:0/1130926489 shutdown_connections 2024-04-22T15:51:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.234+0000 7fc3158d9700 1 -- 172.21.15.16:0/1130926489 wait complete. 2024-04-22T15:51:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.234+0000 7fc3158d9700 1 Processor -- start 2024-04-22T15:51:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.235+0000 7fc3158d9700 1 -- start start 2024-04-22T15:51:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.235+0000 7fc3158d9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3100ffbb0 0x7fc310196490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.235+0000 7fc3158d9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc310100db0 0x7fc3101969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.235+0000 7fc3158d9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc310196ff0 con 0x7fc3100ffbb0 2024-04-22T15:51:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.235+0000 7fc3158d9700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc310197130 con 0x7fc310100db0 2024-04-22T15:51:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.235+0000 7fc30effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3100ffbb0 0x7fc310196490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.235+0000 7fc30effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3100ffbb0 0x7fc310196490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40176/0 (socket says 172.21.15.16:40176) 2024-04-22T15:51:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.235+0000 7fc30effd700 1 -- 172.21.15.16:0/2886513047 learned_addr learned my addr 172.21.15.16:0/2886513047 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:51:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.235+0000 7fc30e7fc700 1 --2- 172.21.15.16:0/2886513047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc310100db0 0x7fc3101969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.235+0000 7fc30effd700 1 -- 172.21.15.16:0/2886513047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc310100db0 msgr2=0x7fc3101969d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.235+0000 7fc30effd700 1 --2- 172.21.15.16:0/2886513047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc310100db0 0x7fc3101969d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.235+0000 7fc30effd700 1 -- 172.21.15.16:0/2886513047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3040095f0 con 0x7fc3100ffbb0 2024-04-22T15:51:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.235+0000 7fc30effd700 1 --2- 172.21.15.16:0/2886513047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3100ffbb0 0x7fc310196490 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc30000e8f0 tx=0x7fc30000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.235+0000 7fc3148d7700 1 -- 172.21.15.16:0/2886513047 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc300009790 con 0x7fc3100ffbb0 2024-04-22T15:51:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.235+0000 7fc3158d9700 1 -- 172.21.15.16:0/2886513047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc31019bbe0 con 0x7fc3100ffbb0 2024-04-22T15:51:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.236+0000 7fc3148d7700 1 -- 172.21.15.16:0/2886513047 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc30000c810 con 0x7fc3100ffbb0 2024-04-22T15:51:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.236+0000 7fc3148d7700 1 -- 172.21.15.16:0/2886513047 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc300003d20 con 0x7fc3100ffbb0 2024-04-22T15:51:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.236+0000 7fc3158d9700 1 -- 172.21.15.16:0/2886513047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc31019c130 con 0x7fc3100ffbb0 2024-04-22T15:51:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.236+0000 7fc3158d9700 1 -- 172.21.15.16:0/2886513047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc310108710 con 0x7fc3100ffbb0 2024-04-22T15:51:40.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.238+0000 7fc3148d7700 1 -- 172.21.15.16:0/2886513047 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7fc3000041f0 con 0x7fc3100ffbb0 2024-04-22T15:51:40.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.238+0000 7fc3148d7700 1 --2- 172.21.15.16:0/2886513047 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fc2f806e940 0x7fc2f8070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:51:40.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.238+0000 7fc3148d7700 1 -- 172.21.15.16:0/2886513047 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fc300014070 con 0x7fc3100ffbb0 2024-04-22T15:51:40.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.238+0000 7fc30e7fc700 1 --2- 172.21.15.16:0/2886513047 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fc2f806e940 0x7fc2f8070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:51:40.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.239+0000 7fc30e7fc700 1 --2- 172.21.15.16:0/2886513047 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fc2f806e940 0x7fc2f8070e00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc304009fd0 tx=0x7fc304005940 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:51:40.244 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2024-04-22T15:51:40.245 DEBUG:teuthology.parallel:result is None 2024-04-22T15:51:40.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.242+0000 7fc3148d7700 1 -- 172.21.15.16:0/2886513047 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7fc30005bb70 con 0x7fc3100ffbb0 2024-04-22T15:51:40.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.478+0000 7fc3158d9700 1 -- 172.21.15.16:0/2886513047 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc310069520 con 0x7fc2f806e940 2024-04-22T15:51:40.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.480+0000 7fc3148d7700 1 -- 172.21.15.16:0/2886513047 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7fc310069520 con 0x7fc2f806e940 2024-04-22T15:51:40.483 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:51:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:51:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-22T15:51:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T15:51:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-22T15:51:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-04-22T15:51:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-22T15:51:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "2/2 daemons upgraded", 2024-04-22T15:51:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading alertmanager daemons", 2024-04-22T15:51:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-22T15:51:40.484 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:51:40.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.482+0000 7fc3158d9700 1 -- 172.21.15.16:0/2886513047 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fc2f806e940 msgr2=0x7fc2f8070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:40.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.482+0000 7fc3158d9700 1 --2- 172.21.15.16:0/2886513047 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fc2f806e940 0x7fc2f8070e00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc304009fd0 tx=0x7fc304005940 comp rx=0 tx=0).stop 2024-04-22T15:51:40.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.482+0000 7fc3158d9700 1 -- 172.21.15.16:0/2886513047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3100ffbb0 msgr2=0x7fc310196490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:51:40.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.482+0000 7fc3158d9700 1 --2- 172.21.15.16:0/2886513047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3100ffbb0 0x7fc310196490 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc30000e8f0 tx=0x7fc30000ec00 comp rx=0 tx=0).stop 2024-04-22T15:51:40.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.482+0000 7fc3158d9700 1 -- 172.21.15.16:0/2886513047 shutdown_connections 2024-04-22T15:51:40.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.482+0000 7fc3158d9700 1 --2- 172.21.15.16:0/2886513047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3100ffbb0 0x7fc310196490 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:40.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.482+0000 7fc3158d9700 1 --2- 172.21.15.16:0/2886513047 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fc2f806e940 0x7fc2f8070e00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:40.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.482+0000 7fc3158d9700 1 --2- 172.21.15.16:0/2886513047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc310100db0 0x7fc3101969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:51:40.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.482+0000 7fc3158d9700 1 -- 172.21.15.16:0/2886513047 >> 172.21.15.16:0/2886513047 conn(0x7fc3100fb150 msgr2=0x7fc310068e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:51:40.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.482+0000 7fc3158d9700 1 -- 172.21.15.16:0/2886513047 shutdown_connections 2024-04-22T15:51:40.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:51:40.483+0000 7fc3158d9700 1 -- 172.21.15.16:0/2886513047 wait complete. 2024-04-22T15:51:41.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:40 smithi016 ceph-mon[26620]: from='client.14774 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:41.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:40 smithi016 ceph-mon[26620]: pgmap v54: 65 pgs: 65 active+clean; 264 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 279 op/s 2024-04-22T15:51:41.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:40 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/1574022607' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:41.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:40 smithi016 ceph-mon[26620]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:41.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:40 smithi027 ceph-mon[34922]: from='client.14774 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:41.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:40 smithi027 ceph-mon[34922]: pgmap v54: 65 pgs: 65 active+clean; 264 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 279 op/s 2024-04-22T15:51:41.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:40 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/1574022607' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:41.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:40 smithi027 ceph-mon[34922]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:51:43.288 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:42 smithi016 ceph-mon[26620]: pgmap v55: 65 pgs: 65 active+clean; 264 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 184 op/s 2024-04-22T15:51:43.288 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:42 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:51:43.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:42 smithi027 ceph-mon[34922]: pgmap v55: 65 pgs: 65 active+clean; 264 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 184 op/s 2024-04-22T15:51:43.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:42 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:51:45.052 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:44 smithi016 ceph-mon[26620]: pgmap v56: 65 pgs: 65 active+clean; 264 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 184 op/s 2024-04-22T15:51:45.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:44 smithi027 ceph-mon[34922]: pgmap v56: 65 pgs: 65 active+clean; 264 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 184 op/s 2024-04-22T15:51:46.414 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:46 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:46.414 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:46 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:46.414 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:46 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:51:46.414 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:46 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:46.415 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:46 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:46.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:46 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:46.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:46 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:46.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:46 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:51:46.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:46 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:46.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:46 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:47.324 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:47 smithi027 ceph-mon[34922]: pgmap v57: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 231 op/s 2024-04-22T15:51:47.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:47 smithi016 ceph-mon[26620]: pgmap v57: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 231 op/s 2024-04-22T15:51:49.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:49 smithi027 ceph-mon[34922]: pgmap v58: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 696 KiB/s rd, 632 KiB/s wr, 139 op/s 2024-04-22T15:51:49.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:49 smithi016 ceph-mon[26620]: pgmap v58: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 696 KiB/s rd, 632 KiB/s wr, 139 op/s 2024-04-22T15:51:51.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:51 smithi027 ceph-mon[34922]: pgmap v59: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 720 KiB/s rd, 656 KiB/s wr, 140 op/s 2024-04-22T15:51:51.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:51 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:51.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:51 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:51.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:51 smithi016 ceph-mon[26620]: pgmap v59: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 720 KiB/s rd, 656 KiB/s wr, 140 op/s 2024-04-22T15:51:51.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:51 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:51.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:51 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:53.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:52 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:53.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:52 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:53.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:52 smithi027 ceph-mon[34922]: pgmap v60: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 218 KiB/s rd, 155 KiB/s wr, 47 op/s 2024-04-22T15:51:53.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:52 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:53.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:52 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:53.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:52 smithi016 ceph-mon[26620]: pgmap v60: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 218 KiB/s rd, 155 KiB/s wr, 47 op/s 2024-04-22T15:51:55.193 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:54 smithi016 ceph-mon[26620]: pgmap v61: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 217 KiB/s rd, 154 KiB/s wr, 47 op/s 2024-04-22T15:51:55.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:54 smithi027 ceph-mon[34922]: pgmap v61: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 217 KiB/s rd, 154 KiB/s wr, 47 op/s 2024-04-22T15:51:56.498 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:56 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:56.498 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:56 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:56.498 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:56 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:51:56.499 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:56 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:51:56.499 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:56 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:56.499 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:56 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T15:51:56.499 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:56 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:51:56.499 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:56 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:56.499 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:56 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:56.500 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:56 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:56.500 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:56 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:56.500 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:56 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:56.500 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:56 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:56.500 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:56 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:56.500 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:56 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:56.500 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:56 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:56.500 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:56 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:56.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:56 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:56.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:56 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:56.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:56 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:51:56.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:56 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:51:56.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:56 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:51:56.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:56 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T15:51:56.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:56 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:51:56.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:56 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:56.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:56 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:56.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:56 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:56.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:56 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:56.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:56 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:56.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:56 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:56.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:56 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:56.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:56 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:56.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:56 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:56.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:56 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:51:57.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:57 smithi016 ceph-mon[26620]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T15:51:57.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:57 smithi016 ceph-mon[26620]: pgmap v62: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 218 KiB/s rd, 154 KiB/s wr, 48 op/s 2024-04-22T15:51:57.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:57 smithi027 ceph-mon[34922]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T15:51:57.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:57 smithi027 ceph-mon[34922]: pgmap v62: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 218 KiB/s rd, 154 KiB/s wr, 48 op/s 2024-04-22T15:51:58.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:58 smithi027 ceph-mon[34922]: Upgrade: Updating grafana.smithi016 2024-04-22T15:51:58.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:58 smithi027 ceph-mon[34922]: Deploying daemon grafana.smithi016 on smithi016 2024-04-22T15:51:58.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:58 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:51:58.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:58 smithi016 ceph-mon[26620]: Upgrade: Updating grafana.smithi016 2024-04-22T15:51:58.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:58 smithi016 ceph-mon[26620]: Deploying daemon grafana.smithi016 on smithi016 2024-04-22T15:51:58.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:58 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:51:59.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:51:59 smithi027 ceph-mon[34922]: pgmap v63: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 25 KiB/s rd, 24 KiB/s wr, 1 op/s 2024-04-22T15:51:59.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:51:59 smithi016 ceph-mon[26620]: pgmap v63: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 25 KiB/s rd, 24 KiB/s wr, 1 op/s 2024-04-22T15:52:01.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:01 smithi027 ceph-mon[34922]: pgmap v64: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 25 KiB/s rd, 24 KiB/s wr, 2 op/s 2024-04-22T15:52:01.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:01 smithi016 ceph-mon[26620]: pgmap v64: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 25 KiB/s rd, 24 KiB/s wr, 2 op/s 2024-04-22T15:52:03.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:03 smithi027 ceph-mon[34922]: pgmap v65: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 851 B/s rd, 1 op/s 2024-04-22T15:52:03.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:03 smithi016 ceph-mon[26620]: pgmap v65: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 851 B/s rd, 1 op/s 2024-04-22T15:52:05.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:05 smithi016 ceph-mon[26620]: pgmap v66: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 851 B/s rd, 1 op/s 2024-04-22T15:52:05.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:05 smithi027 ceph-mon[34922]: pgmap v66: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 851 B/s rd, 1 op/s 2024-04-22T15:52:07.020 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:06 smithi016 ceph-mon[26620]: pgmap v67: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:52:07.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:06 smithi027 ceph-mon[34922]: pgmap v67: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:52:09.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:08 smithi016 ceph-mon[26620]: pgmap v68: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:09.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:08 smithi027 ceph-mon[34922]: pgmap v68: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.605+0000 7f64255d1700 1 -- 172.21.15.16:0/2632799711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64200ffc10 msgr2=0x7f6420100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.605+0000 7f64255d1700 1 --2- 172.21.15.16:0/2632799711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64200ffc10 0x7f6420100030 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6410009910 tx=0x7f6410009c20 comp rx=0 tx=0).stop 2024-04-22T15:52:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.605+0000 7f64255d1700 1 -- 172.21.15.16:0/2632799711 shutdown_connections 2024-04-22T15:52:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.605+0000 7f64255d1700 1 --2- 172.21.15.16:0/2632799711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6420100e10 0x7f6420101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.605+0000 7f64255d1700 1 --2- 172.21.15.16:0/2632799711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64200ffc10 0x7f6420100030 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.605+0000 7f64255d1700 1 -- 172.21.15.16:0/2632799711 >> 172.21.15.16:0/2632799711 conn(0x7f64200fb190 msgr2=0x7f64200fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.605+0000 7f64255d1700 1 -- 172.21.15.16:0/2632799711 shutdown_connections 2024-04-22T15:52:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.605+0000 7f64255d1700 1 -- 172.21.15.16:0/2632799711 wait complete. 2024-04-22T15:52:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.606+0000 7f64255d1700 1 Processor -- start 2024-04-22T15:52:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.606+0000 7f64255d1700 1 -- start start 2024-04-22T15:52:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.606+0000 7f64255d1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64200ffc10 0x7f6420069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.606+0000 7f64255d1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6420100e10 0x7f642019d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.606+0000 7f64255d1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642019db10 con 0x7f64200ffc10 2024-04-22T15:52:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.606+0000 7f64255d1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642019dc50 con 0x7f6420100e10 2024-04-22T15:52:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.606+0000 7f641effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64200ffc10 0x7f6420069140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.606+0000 7f641effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64200ffc10 0x7f6420069140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44864/0 (socket says 172.21.15.16:44864) 2024-04-22T15:52:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.606+0000 7f641effd700 1 -- 172.21.15.16:0/1779378972 learned_addr learned my addr 172.21.15.16:0/1779378972 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:52:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.607+0000 7f641e7fc700 1 --2- 172.21.15.16:0/1779378972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6420100e10 0x7f642019d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.607+0000 7f641effd700 1 -- 172.21.15.16:0/1779378972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6420100e10 msgr2=0x7f642019d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.607+0000 7f641effd700 1 --2- 172.21.15.16:0/1779378972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6420100e10 0x7f642019d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.607+0000 7f641effd700 1 -- 172.21.15.16:0/1779378972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64100095f0 con 0x7f64200ffc10 2024-04-22T15:52:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.607+0000 7f641effd700 1 --2- 172.21.15.16:0/1779378972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64200ffc10 0x7f6420069140 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f641000bda0 tx=0x7f641000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.607+0000 7f6407fff700 1 -- 172.21.15.16:0/1779378972 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f641001d070 con 0x7f64200ffc10 2024-04-22T15:52:10.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.607+0000 7f64255d1700 1 -- 172.21.15.16:0/1779378972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642019ddf0 con 0x7f64200ffc10 2024-04-22T15:52:10.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.607+0000 7f64255d1700 1 -- 172.21.15.16:0/1779378972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642019e250 con 0x7f64200ffc10 2024-04-22T15:52:10.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.607+0000 7f6407fff700 1 -- 172.21.15.16:0/1779378972 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f641000f460 con 0x7f64200ffc10 2024-04-22T15:52:10.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.607+0000 7f6407fff700 1 -- 172.21.15.16:0/1779378972 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6410021600 con 0x7f64200ffc10 2024-04-22T15:52:10.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.608+0000 7f64255d1700 1 -- 172.21.15.16:0/1779378972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f640c005320 con 0x7f64200ffc10 2024-04-22T15:52:10.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.609+0000 7f6407fff700 1 -- 172.21.15.16:0/1779378972 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f6410003a60 con 0x7f64200ffc10 2024-04-22T15:52:10.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.610+0000 7f6407fff700 1 --2- 172.21.15.16:0/1779378972 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f640806e940 0x7f6408070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:10.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.610+0000 7f6407fff700 1 -- 172.21.15.16:0/1779378972 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f641008faa0 con 0x7f64200ffc10 2024-04-22T15:52:10.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.613+0000 7f641e7fc700 1 --2- 172.21.15.16:0/1779378972 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f640806e940 0x7f6408070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:10.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.613+0000 7f641e7fc700 1 --2- 172.21.15.16:0/1779378972 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f640806e940 0x7f6408070e00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6414005950 tx=0x7f64140058e0 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:10.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.620+0000 7f6407fff700 1 -- 172.21.15.16:0/1779378972 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f641005c960 con 0x7f64200ffc10 2024-04-22T15:52:10.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.868+0000 7f64255d1700 1 -- 172.21.15.16:0/1779378972 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f640c000bf0 con 0x7f640806e940 2024-04-22T15:52:10.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.870+0000 7f6407fff700 1 -- 172.21.15.16:0/1779378972 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7f640c000bf0 con 0x7f640806e940 2024-04-22T15:52:10.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.872+0000 7f64255d1700 1 -- 172.21.15.16:0/1779378972 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f640806e940 msgr2=0x7f6408070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:10.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.872+0000 7f64255d1700 1 --2- 172.21.15.16:0/1779378972 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f640806e940 0x7f6408070e00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6414005950 tx=0x7f64140058e0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.872+0000 7f64255d1700 1 -- 172.21.15.16:0/1779378972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64200ffc10 msgr2=0x7f6420069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:10.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.872+0000 7f64255d1700 1 --2- 172.21.15.16:0/1779378972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64200ffc10 0x7f6420069140 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f641000bda0 tx=0x7f641000be80 comp rx=0 tx=0).stop 2024-04-22T15:52:10.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.872+0000 7f64255d1700 1 -- 172.21.15.16:0/1779378972 shutdown_connections 2024-04-22T15:52:10.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.872+0000 7f64255d1700 1 --2- 172.21.15.16:0/1779378972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64200ffc10 0x7f6420069140 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.872+0000 7f64255d1700 1 --2- 172.21.15.16:0/1779378972 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f640806e940 0x7f6408070e00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.872+0000 7f64255d1700 1 --2- 172.21.15.16:0/1779378972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6420100e10 0x7f642019d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:10.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.872+0000 7f64255d1700 1 -- 172.21.15.16:0/1779378972 >> 172.21.15.16:0/1779378972 conn(0x7f64200fb190 msgr2=0x7f6420104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:10.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.872+0000 7f64255d1700 1 -- 172.21.15.16:0/1779378972 shutdown_connections 2024-04-22T15:52:10.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:10.872+0000 7f64255d1700 1 -- 172.21.15.16:0/1779378972 wait complete. 2024-04-22T15:52:10.886 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:52:11.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.014+0000 7f5e5ce72700 1 -- 172.21.15.16:0/1546040982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e580fff70 msgr2=0x7f5e581003f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:11.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.014+0000 7f5e5ce72700 1 --2- 172.21.15.16:0/1546040982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e580fff70 0x7f5e581003f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009910 tx=0x7f5e4c009c20 comp rx=0 tx=0).stop 2024-04-22T15:52:11.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.014+0000 7f5e5ce72700 1 -- 172.21.15.16:0/1546040982 shutdown_connections 2024-04-22T15:52:11.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.014+0000 7f5e5ce72700 1 --2- 172.21.15.16:0/1546040982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e580fff70 0x7f5e581003f0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.014+0000 7f5e5ce72700 1 --2- 172.21.15.16:0/1546040982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e580ff610 0x7f5e580ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.014+0000 7f5e5ce72700 1 -- 172.21.15.16:0/1546040982 >> 172.21.15.16:0/1546040982 conn(0x7f5e580fb190 msgr2=0x7f5e580fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:11.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.014+0000 7f5e5ce72700 1 -- 172.21.15.16:0/1546040982 shutdown_connections 2024-04-22T15:52:11.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.014+0000 7f5e5ce72700 1 -- 172.21.15.16:0/1546040982 wait complete. 2024-04-22T15:52:11.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.015+0000 7f5e5ce72700 1 Processor -- start 2024-04-22T15:52:11.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.015+0000 7f5e5ce72700 1 -- start start 2024-04-22T15:52:11.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.015+0000 7f5e5ce72700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e580ff610 0x7f5e58198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:11.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.015+0000 7f5e5ce72700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e580fff70 0x7f5e58198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:11.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.015+0000 7f5e5ce72700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e581991f0 con 0x7f5e580fff70 2024-04-22T15:52:11.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.015+0000 7f5e5ce72700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e58199330 con 0x7f5e580ff610 2024-04-22T15:52:11.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.015+0000 7f5e55d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e580fff70 0x7f5e58198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:11.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.015+0000 7f5e55d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e580fff70 0x7f5e58198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44886/0 (socket says 172.21.15.16:44886) 2024-04-22T15:52:11.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.015+0000 7f5e55d9b700 1 -- 172.21.15.16:0/3859465420 learned_addr learned my addr 172.21.15.16:0/3859465420 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:52:11.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.015+0000 7f5e5659c700 1 --2- 172.21.15.16:0/3859465420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e580ff610 0x7f5e58198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:11.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.016+0000 7f5e5659c700 1 -- 172.21.15.16:0/3859465420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e580fff70 msgr2=0x7f5e58198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:11.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.016+0000 7f5e5659c700 1 --2- 172.21.15.16:0/3859465420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e580fff70 0x7f5e58198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.016+0000 7f5e5659c700 1 -- 172.21.15.16:0/3859465420 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e4c0095f0 con 0x7f5e580ff610 2024-04-22T15:52:11.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.016+0000 7f5e55d9b700 1 --2- 172.21.15.16:0/3859465420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e580fff70 0x7f5e58198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:52:11.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.016+0000 7f5e5659c700 1 --2- 172.21.15.16:0/3859465420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e580ff610 0x7f5e58198690 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800e970 tx=0x7f5e4800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:11.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.016+0000 7f5e477fe700 1 -- 172.21.15.16:0/3859465420 <== mon.1 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e4800c9d0 con 0x7f5e580ff610 2024-04-22T15:52:11.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.016+0000 7f5e477fe700 1 -- 172.21.15.16:0/3859465420 <== mon.1 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5e48010430 con 0x7f5e580ff610 2024-04-22T15:52:11.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.016+0000 7f5e5ce72700 1 -- 172.21.15.16:0/3859465420 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5819dde0 con 0x7f5e580ff610 2024-04-22T15:52:11.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.016+0000 7f5e477fe700 1 -- 172.21.15.16:0/3859465420 <== mon.1 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e48018650 con 0x7f5e580ff610 2024-04-22T15:52:11.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.016+0000 7f5e5ce72700 1 -- 172.21.15.16:0/3859465420 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5819e260 con 0x7f5e580ff610 2024-04-22T15:52:11.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.018+0000 7f5e5ce72700 1 -- 172.21.15.16:0/3859465420 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e5810b9f0 con 0x7f5e580ff610 2024-04-22T15:52:11.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.018+0000 7f5e477fe700 1 -- 172.21.15.16:0/3859465420 <== mon.1 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f5e4800cb30 con 0x7f5e580ff610 2024-04-22T15:52:11.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.018+0000 7f5e477fe700 1 --2- 172.21.15.16:0/3859465420 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f5e4006e940 0x7f5e40070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:11.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.018+0000 7f5e477fe700 1 -- 172.21.15.16:0/3859465420 <== mon.1 v2:172.21.15.27:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f5e48014070 con 0x7f5e580ff610 2024-04-22T15:52:11.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.018+0000 7f5e55d9b700 1 --2- 172.21.15.16:0/3859465420 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f5e4006e940 0x7f5e40070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:11.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.019+0000 7f5e55d9b700 1 --2- 172.21.15.16:0/3859465420 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f5e4006e940 0x7f5e40070e00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009fd0 tx=0x7f5e4c005a90 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:11.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.023+0000 7f5e477fe700 1 -- 172.21.15.16:0/3859465420 <== mon.1 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f5e48053e20 con 0x7f5e580ff610 2024-04-22T15:52:11.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:10 smithi016 ceph-mon[26620]: pgmap v69: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:52:11.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:10 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:11.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:10 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:11.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:10 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:52:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.326+0000 7f5e5ce72700 1 -- 172.21.15.16:0/3859465420 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e58109360 con 0x7f5e4006e940 2024-04-22T15:52:11.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:10 smithi027 ceph-mon[34922]: pgmap v69: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:52:11.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:10 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:11.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:10 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:11.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:10 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:52:11.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.334+0000 7f5e477fe700 1 -- 172.21.15.16:0/3859465420 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7f5e58109360 con 0x7f5e4006e940 2024-04-22T15:52:11.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.338+0000 7f5e5ce72700 1 -- 172.21.15.16:0/3859465420 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f5e4006e940 msgr2=0x7f5e40070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:11.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.338+0000 7f5e5ce72700 1 --2- 172.21.15.16:0/3859465420 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f5e4006e940 0x7f5e40070e00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009fd0 tx=0x7f5e4c005a90 comp rx=0 tx=0).stop 2024-04-22T15:52:11.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.338+0000 7f5e5ce72700 1 -- 172.21.15.16:0/3859465420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e580ff610 msgr2=0x7f5e58198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:11.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.338+0000 7f5e5ce72700 1 --2- 172.21.15.16:0/3859465420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e580ff610 0x7f5e58198690 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800e970 tx=0x7f5e4800ec80 comp rx=0 tx=0).stop 2024-04-22T15:52:11.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.338+0000 7f5e5ce72700 1 -- 172.21.15.16:0/3859465420 shutdown_connections 2024-04-22T15:52:11.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.338+0000 7f5e5ce72700 1 --2- 172.21.15.16:0/3859465420 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f5e4006e940 0x7f5e40070e00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.338+0000 7f5e5ce72700 1 --2- 172.21.15.16:0/3859465420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e580ff610 0x7f5e58198690 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800e970 tx=0x7f5e4800ec80 comp rx=0 tx=0).stop 2024-04-22T15:52:11.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.338+0000 7f5e5ce72700 1 --2- 172.21.15.16:0/3859465420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e580fff70 0x7f5e58198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.339+0000 7f5e5ce72700 1 -- 172.21.15.16:0/3859465420 >> 172.21.15.16:0/3859465420 conn(0x7f5e580fb190 msgr2=0x7f5e58107c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:11.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.339+0000 7f5e5ce72700 1 -- 172.21.15.16:0/3859465420 shutdown_connections 2024-04-22T15:52:11.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.340+0000 7f5e5ce72700 1 -- 172.21.15.16:0/3859465420 wait complete. 2024-04-22T15:52:11.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.473+0000 7f61e7985700 1 -- 172.21.15.16:0/1583988069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61e0100cb0 msgr2=0x7f61e01010d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:11.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.473+0000 7f61e7985700 1 --2- 172.21.15.16:0/1583988069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61e0100cb0 0x7f61e01010d0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f61d000b0a0 tx=0x7f61d000b3b0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.473+0000 7f61e7985700 1 -- 172.21.15.16:0/1583988069 shutdown_connections 2024-04-22T15:52:11.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.473+0000 7f61e7985700 1 --2- 172.21.15.16:0/1583988069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61e0101eb0 0x7f61e0102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.473+0000 7f61e7985700 1 --2- 172.21.15.16:0/1583988069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61e0100cb0 0x7f61e01010d0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.473+0000 7f61e7985700 1 -- 172.21.15.16:0/1583988069 >> 172.21.15.16:0/1583988069 conn(0x7f61e00fc230 msgr2=0x7f61e00fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:11.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.473+0000 7f61e7985700 1 -- 172.21.15.16:0/1583988069 shutdown_connections 2024-04-22T15:52:11.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.473+0000 7f61e7985700 1 -- 172.21.15.16:0/1583988069 wait complete. 2024-04-22T15:52:11.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.474+0000 7f61e7985700 1 Processor -- start 2024-04-22T15:52:11.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.474+0000 7f61e7985700 1 -- start start 2024-04-22T15:52:11.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.474+0000 7f61e7985700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61e0100cb0 0x7f61e01964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:11.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.474+0000 7f61e7985700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61e0101eb0 0x7f61e0196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:11.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.474+0000 7f61e7985700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e0197050 con 0x7f61e0101eb0 2024-04-22T15:52:11.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.474+0000 7f61e7985700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e0197190 con 0x7f61e0100cb0 2024-04-22T15:52:11.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.474+0000 7f61e4f20700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61e0101eb0 0x7f61e0196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:11.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.474+0000 7f61e4f20700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61e0101eb0 0x7f61e0196a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44908/0 (socket says 172.21.15.16:44908) 2024-04-22T15:52:11.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.474+0000 7f61e4f20700 1 -- 172.21.15.16:0/3327379910 learned_addr learned my addr 172.21.15.16:0/3327379910 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:52:11.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.474+0000 7f61e5721700 1 --2- 172.21.15.16:0/3327379910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61e0100cb0 0x7f61e01964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:11.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.474+0000 7f61e4f20700 1 -- 172.21.15.16:0/3327379910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61e0100cb0 msgr2=0x7f61e01964f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:11.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.474+0000 7f61e4f20700 1 --2- 172.21.15.16:0/3327379910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61e0100cb0 0x7f61e01964f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.474+0000 7f61e4f20700 1 -- 172.21.15.16:0/3327379910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d0009d00 con 0x7f61e0101eb0 2024-04-22T15:52:11.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.475+0000 7f61e4f20700 1 --2- 172.21.15.16:0/3327379910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61e0101eb0 0x7f61e0196a30 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f61d400e920 tx=0x7f61d400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:11.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.475+0000 7f61de7fc700 1 -- 172.21.15.16:0/3327379910 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61d400c9d0 con 0x7f61e0101eb0 2024-04-22T15:52:11.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.475+0000 7f61de7fc700 1 -- 172.21.15.16:0/3327379910 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f61d4010430 con 0x7f61e0101eb0 2024-04-22T15:52:11.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.475+0000 7f61de7fc700 1 -- 172.21.15.16:0/3327379910 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61d4018650 con 0x7f61e0101eb0 2024-04-22T15:52:11.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.475+0000 7f61e7985700 1 -- 172.21.15.16:0/3327379910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e019bc40 con 0x7f61e0101eb0 2024-04-22T15:52:11.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.475+0000 7f61e7985700 1 -- 172.21.15.16:0/3327379910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e019c140 con 0x7f61e0101eb0 2024-04-22T15:52:11.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.476+0000 7f61e7985700 1 -- 172.21.15.16:0/3327379910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e0109810 con 0x7f61e0101eb0 2024-04-22T15:52:11.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.477+0000 7f61de7fc700 1 -- 172.21.15.16:0/3327379910 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f61d40105a0 con 0x7f61e0101eb0 2024-04-22T15:52:11.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.477+0000 7f61de7fc700 1 --2- 172.21.15.16:0/3327379910 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f61c806e940 0x7f61c8070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:11.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.477+0000 7f61de7fc700 1 -- 172.21.15.16:0/3327379910 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f61d4014070 con 0x7f61e0101eb0 2024-04-22T15:52:11.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.477+0000 7f61e5721700 1 --2- 172.21.15.16:0/3327379910 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f61c806e940 0x7f61c8070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:11.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.478+0000 7f61e5721700 1 --2- 172.21.15.16:0/3327379910 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f61c806e940 0x7f61c8070e00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f61d000b780 tx=0x7f61d00060c0 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:11.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.482+0000 7f61de7fc700 1 -- 172.21.15.16:0/3327379910 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f61d4053e20 con 0x7f61e0101eb0 2024-04-22T15:52:11.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.773+0000 7f61e7985700 1 -- 172.21.15.16:0/3327379910 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f61e0106800 con 0x7f61c806e940 2024-04-22T15:52:11.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.784+0000 7f61de7fc700 1 -- 172.21.15.16:0/3327379910 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f61e0106800 con 0x7f61c806e940 2024-04-22T15:52:11.787 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:52:11.787 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.smithi016 smithi016 *:9093,9094 running (27s) 21s ago 11m 17.1M - 0.25.0 c8568f914cd2 ea0e0217d6f9 2024-04-22T15:52:11.788 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi016 smithi016 running (11m) 21s ago 11m 7852k - 18.2.1 5be31c24972a f8b627d3f41e 2024-04-22T15:52:11.788 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi027 smithi027 running (10m) 67s ago 10m 7776k - 18.2.1 5be31c24972a f00fd40b4220 2024-04-22T15:52:11.788 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi016 smithi016 running (11m) 21s ago 11m 6660k - 18.2.1 5be31c24972a 6b93db1fa0d9 2024-04-22T15:52:11.788 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi027 smithi027 running (10m) 67s ago 10m 6665k - 18.2.1 5be31c24972a d3bd439f30a0 2024-04-22T15:52:11.788 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.smithi016 smithi016 *:3000 starting - - - - 2024-04-22T15:52:11.788 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.fkkejm smithi016 running (5m) 21s ago 5m 14.5M - 18.2.1 5be31c24972a 4153ac94cea0 2024-04-22T15:52:11.788 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.pgvphu smithi016 running (5m) 21s ago 5m 96.4M - 18.2.1 5be31c24972a 93dc072068a2 2024-04-22T15:52:11.788 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.dgqyes smithi027 running (5m) 67s ago 5m 17.3M - 18.2.1 5be31c24972a 41a9220430fd 2024-04-22T15:52:11.788 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.iobjds smithi027 running (5m) 67s ago 5m 241M - 18.2.1 5be31c24972a c1c8d6770f02 2024-04-22T15:52:11.788 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi016.rnsnnr smithi016 *:8443,9283,8765 running (2m) 21s ago 12m 505M - 19.0.0-3223-g5883d494 3d983c26e278 09b4fe285bc4 2024-04-22T15:52:11.789 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi027.ukdgzs smithi027 *:8443,9283,8765 running (103s) 67s ago 10m 450M - 19.0.0-3223-g5883d494 3d983c26e278 7548da9b5eb9 2024-04-22T15:52:11.789 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi016 smithi016 running (12m) 21s ago 12m 59.7M 2048M 18.2.1 5be31c24972a 72af40873cb6 2024-04-22T15:52:11.789 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi027 smithi027 running (10m) 67s ago 10m 43.7M 2048M 18.2.1 5be31c24972a 9f21ce7fa7ef 2024-04-22T15:52:11.789 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi016 smithi016 *:9100 running (87s) 21s ago 11m 15.5M - 1.5.0 0da6a335fe13 35e46e4d2f3f 2024-04-22T15:52:11.789 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi027 smithi027 *:9100 running (74s) 67s ago 10m 3598k - 1.5.0 0da6a335fe13 e682a7672e9b 2024-04-22T15:52:11.789 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (9m) 21s ago 9m 200M 1217M 18.2.1 5be31c24972a 3c6ebc4ea041 2024-04-22T15:52:11.789 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (8m) 21s ago 8m 202M 1217M 18.2.1 5be31c24972a 6aa2e6796784 2024-04-22T15:52:11.789 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (8m) 21s ago 8m 174M 1217M 18.2.1 5be31c24972a 3d9d310aa9eb 2024-04-22T15:52:11.789 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi027 running (7m) 67s ago 7m 227M 2240M 18.2.1 5be31c24972a cdf9b7aaf865 2024-04-22T15:52:11.789 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (7m) 67s ago 7m 184M 2240M 18.2.1 5be31c24972a 629397d49c76 2024-04-22T15:52:11.790 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi027 running (7m) 67s ago 7m 173M 2240M 18.2.1 5be31c24972a 23adcf675b04 2024-04-22T15:52:11.790 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.smithi016 smithi016 *:9095 running (48s) 21s ago 10m 51.4M - 2.43.0 a07b618ecd1d 373eaadee82d 2024-04-22T15:52:11.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.786+0000 7f61e7985700 1 -- 172.21.15.16:0/3327379910 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f61c806e940 msgr2=0x7f61c8070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:11.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.786+0000 7f61e7985700 1 --2- 172.21.15.16:0/3327379910 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f61c806e940 0x7f61c8070e00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f61d000b780 tx=0x7f61d00060c0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.786+0000 7f61e7985700 1 -- 172.21.15.16:0/3327379910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61e0101eb0 msgr2=0x7f61e0196a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:11.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.786+0000 7f61e7985700 1 --2- 172.21.15.16:0/3327379910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61e0101eb0 0x7f61e0196a30 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f61d400e920 tx=0x7f61d400ec30 comp rx=0 tx=0).stop 2024-04-22T15:52:11.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.786+0000 7f61e7985700 1 -- 172.21.15.16:0/3327379910 shutdown_connections 2024-04-22T15:52:11.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.786+0000 7f61e7985700 1 --2- 172.21.15.16:0/3327379910 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f61c806e940 0x7f61c8070e00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.786+0000 7f61e7985700 1 --2- 172.21.15.16:0/3327379910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61e0100cb0 0x7f61e01964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.786+0000 7f61e7985700 1 --2- 172.21.15.16:0/3327379910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61e0101eb0 0x7f61e0196a30 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.786+0000 7f61e7985700 1 -- 172.21.15.16:0/3327379910 >> 172.21.15.16:0/3327379910 conn(0x7f61e00fc230 msgr2=0x7f61e01050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:11.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.787+0000 7f61e7985700 1 -- 172.21.15.16:0/3327379910 shutdown_connections 2024-04-22T15:52:11.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.787+0000 7f61e7985700 1 -- 172.21.15.16:0/3327379910 wait complete. 2024-04-22T15:52:11.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.896+0000 7f873dda5700 1 -- 172.21.15.16:0/3541687869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8738101eb0 msgr2=0x7f8738102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:11.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.896+0000 7f873dda5700 1 --2- 172.21.15.16:0/3541687869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8738101eb0 0x7f8738102310 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8728009910 tx=0x7f8728009c20 comp rx=0 tx=0).stop 2024-04-22T15:52:11.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.896+0000 7f873dda5700 1 -- 172.21.15.16:0/3541687869 shutdown_connections 2024-04-22T15:52:11.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.896+0000 7f873dda5700 1 --2- 172.21.15.16:0/3541687869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8738101eb0 0x7f8738102310 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.896+0000 7f873dda5700 1 --2- 172.21.15.16:0/3541687869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8738100cb0 0x7f87381010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.896+0000 7f873dda5700 1 -- 172.21.15.16:0/3541687869 >> 172.21.15.16:0/3541687869 conn(0x7f87380fc230 msgr2=0x7f87380fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:11.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.896+0000 7f873dda5700 1 -- 172.21.15.16:0/3541687869 shutdown_connections 2024-04-22T15:52:11.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.896+0000 7f873dda5700 1 -- 172.21.15.16:0/3541687869 wait complete. 2024-04-22T15:52:11.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.896+0000 7f873dda5700 1 Processor -- start 2024-04-22T15:52:11.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.897+0000 7f873dda5700 1 -- start start 2024-04-22T15:52:11.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.897+0000 7f873dda5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8738100cb0 0x7f8738074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:11.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.897+0000 7f873dda5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8738101eb0 0x7f8738073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:11.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.897+0000 7f873dda5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87380736c0 con 0x7f8738101eb0 2024-04-22T15:52:11.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.897+0000 7f873dda5700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8738073800 con 0x7f8738100cb0 2024-04-22T15:52:11.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.897+0000 7f8736ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8738101eb0 0x7f8738073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:11.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.897+0000 7f8736ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8738101eb0 0x7f8738073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58098/0 (socket says 172.21.15.16:58098) 2024-04-22T15:52:11.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.897+0000 7f8736ffd700 1 -- 172.21.15.16:0/2797573692 learned_addr learned my addr 172.21.15.16:0/2797573692 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:52:11.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.897+0000 7f87377fe700 1 --2- 172.21.15.16:0/2797573692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8738100cb0 0x7f8738074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:11.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.897+0000 7f8736ffd700 1 -- 172.21.15.16:0/2797573692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8738100cb0 msgr2=0x7f8738074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:11.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.897+0000 7f8736ffd700 1 --2- 172.21.15.16:0/2797573692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8738100cb0 0x7f8738074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:11.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.897+0000 7f8736ffd700 1 -- 172.21.15.16:0/2797573692 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87280095f0 con 0x7f8738101eb0 2024-04-22T15:52:11.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.897+0000 7f8736ffd700 1 --2- 172.21.15.16:0/2797573692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8738101eb0 0x7f8738073180 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8728005b40 tx=0x7f872800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:11.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.898+0000 7f8734ff9700 1 -- 172.21.15.16:0/2797573692 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f872801d070 con 0x7f8738101eb0 2024-04-22T15:52:11.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.898+0000 7f8734ff9700 1 -- 172.21.15.16:0/2797573692 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8728022470 con 0x7f8738101eb0 2024-04-22T15:52:11.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.898+0000 7f873dda5700 1 -- 172.21.15.16:0/2797573692 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8738073a80 con 0x7f8738101eb0 2024-04-22T15:52:11.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.898+0000 7f8734ff9700 1 -- 172.21.15.16:0/2797573692 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f872800f690 con 0x7f8738101eb0 2024-04-22T15:52:11.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.898+0000 7f873dda5700 1 -- 172.21.15.16:0/2797573692 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8738073f70 con 0x7f8738101eb0 2024-04-22T15:52:11.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.899+0000 7f873dda5700 1 -- 172.21.15.16:0/2797573692 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87381098b0 con 0x7f8738101eb0 2024-04-22T15:52:11.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.899+0000 7f8734ff9700 1 -- 172.21.15.16:0/2797573692 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f8728022a10 con 0x7f8738101eb0 2024-04-22T15:52:11.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.900+0000 7f8734ff9700 1 --2- 172.21.15.16:0/2797573692 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f872406e940 0x7f8724070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:11.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.900+0000 7f8734ff9700 1 -- 172.21.15.16:0/2797573692 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f872808fee0 con 0x7f8738101eb0 2024-04-22T15:52:11.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.900+0000 7f87377fe700 1 --2- 172.21.15.16:0/2797573692 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f872406e940 0x7f8724070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:11.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.900+0000 7f87377fe700 1 --2- 172.21.15.16:0/2797573692 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f872406e940 0x7f8724070e00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8738101d10 tx=0x7f8720009190 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:11.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:11.904+0000 7f8734ff9700 1 -- 172.21.15.16:0/2797573692 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f872805cec0 con 0x7f8738101eb0 2024-04-22T15:52:12.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:11 smithi016 ceph-mon[26620]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:12.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:11 smithi016 ceph-mon[26620]: from='client.24567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:12.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.209+0000 7f873dda5700 1 -- 172.21.15.16:0/2797573692 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8738109aa0 con 0x7f8738101eb0 2024-04-22T15:52:12.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.209+0000 7f8734ff9700 1 -- 172.21.15.16:0/2797573692 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f872805ca50 con 0x7f8738101eb0 2024-04-22T15:52:12.213 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:52:12.213 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-22T15:52:12.213 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T15:52:12.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:52:12.213 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-22T15:52:12.213 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:52:12.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:52:12.213 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-22T15:52:12.213 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-22T15:52:12.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:52:12.213 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-22T15:52:12.213 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T15:52:12.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:52:12.213 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-22T15:52:12.214 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-04-22T15:52:12.214 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:52:12.214 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-22T15:52:12.214 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:52:12.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.211+0000 7f873dda5700 1 -- 172.21.15.16:0/2797573692 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f872406e940 msgr2=0x7f8724070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:12.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.211+0000 7f873dda5700 1 --2- 172.21.15.16:0/2797573692 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f872406e940 0x7f8724070e00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8738101d10 tx=0x7f8720009190 comp rx=0 tx=0).stop 2024-04-22T15:52:12.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.211+0000 7f873dda5700 1 -- 172.21.15.16:0/2797573692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8738101eb0 msgr2=0x7f8738073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:12.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.211+0000 7f873dda5700 1 --2- 172.21.15.16:0/2797573692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8738101eb0 0x7f8738073180 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8728005b40 tx=0x7f872800bd70 comp rx=0 tx=0).stop 2024-04-22T15:52:12.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.212+0000 7f873dda5700 1 -- 172.21.15.16:0/2797573692 shutdown_connections 2024-04-22T15:52:12.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.212+0000 7f873dda5700 1 --2- 172.21.15.16:0/2797573692 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f872406e940 0x7f8724070e00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:12.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.212+0000 7f873dda5700 1 --2- 172.21.15.16:0/2797573692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8738100cb0 0x7f8738074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:12.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.212+0000 7f873dda5700 1 --2- 172.21.15.16:0/2797573692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8738101eb0 0x7f8738073180 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:12.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.212+0000 7f873dda5700 1 -- 172.21.15.16:0/2797573692 >> 172.21.15.16:0/2797573692 conn(0x7f87380fc230 msgr2=0x7f87381050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:12.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.212+0000 7f873dda5700 1 -- 172.21.15.16:0/2797573692 shutdown_connections 2024-04-22T15:52:12.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.212+0000 7f873dda5700 1 -- 172.21.15.16:0/2797573692 wait complete. 2024-04-22T15:52:12.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.322+0000 7fe9340ee700 1 -- 172.21.15.16:0/4061231958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe92c104170 msgr2=0x7fe92c106560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:12.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.322+0000 7fe9340ee700 1 --2- 172.21.15.16:0/4061231958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe92c104170 0x7fe92c106560 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe91c009910 tx=0x7fe91c009c20 comp rx=0 tx=0).stop 2024-04-22T15:52:12.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.322+0000 7fe9340ee700 1 -- 172.21.15.16:0/4061231958 shutdown_connections 2024-04-22T15:52:12.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.322+0000 7fe9340ee700 1 --2- 172.21.15.16:0/4061231958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe92c104170 0x7fe92c106560 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:12.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.322+0000 7fe9340ee700 1 --2- 172.21.15.16:0/4061231958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92c101840 0x7fe92c103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:12.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.322+0000 7fe9340ee700 1 -- 172.21.15.16:0/4061231958 >> 172.21.15.16:0/4061231958 conn(0x7fe92c0fb190 msgr2=0x7fe92c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:12.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.322+0000 7fe9340ee700 1 -- 172.21.15.16:0/4061231958 shutdown_connections 2024-04-22T15:52:12.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.322+0000 7fe9340ee700 1 -- 172.21.15.16:0/4061231958 wait complete. 2024-04-22T15:52:12.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.322+0000 7fe9340ee700 1 Processor -- start 2024-04-22T15:52:12.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.322+0000 7fe9340ee700 1 -- start start 2024-04-22T15:52:12.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.323+0000 7fe9340ee700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe92c101840 0x7fe92c196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:12.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.323+0000 7fe9340ee700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92c104170 0x7fe92c1969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:12.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.323+0000 7fe9340ee700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c196fd0 con 0x7fe92c101840 2024-04-22T15:52:12.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.323+0000 7fe9340ee700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c197110 con 0x7fe92c104170 2024-04-22T15:52:12.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.323+0000 7fe931e8a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe92c101840 0x7fe92c196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:12.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.323+0000 7fe931e8a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe92c101840 0x7fe92c196470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58126/0 (socket says 172.21.15.16:58126) 2024-04-22T15:52:12.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.323+0000 7fe931e8a700 1 -- 172.21.15.16:0/654177767 learned_addr learned my addr 172.21.15.16:0/654177767 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:52:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.323+0000 7fe931689700 1 --2- 172.21.15.16:0/654177767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92c104170 0x7fe92c1969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.323+0000 7fe931e8a700 1 -- 172.21.15.16:0/654177767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92c104170 msgr2=0x7fe92c1969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.323+0000 7fe931e8a700 1 --2- 172.21.15.16:0/654177767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92c104170 0x7fe92c1969b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.323+0000 7fe931e8a700 1 -- 172.21.15.16:0/654177767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91c0095f0 con 0x7fe92c101840 2024-04-22T15:52:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.323+0000 7fe931689700 1 --2- 172.21.15.16:0/654177767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92c104170 0x7fe92c1969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:52:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.323+0000 7fe931e8a700 1 --2- 172.21.15.16:0/654177767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe92c101840 0x7fe92c196470 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe92800c8a0 tx=0x7fe92800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.323+0000 7fe922ffd700 1 -- 172.21.15.16:0/654177767 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe92800cea0 con 0x7fe92c101840 2024-04-22T15:52:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.323+0000 7fe922ffd700 1 -- 172.21.15.16:0/654177767 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe928004830 con 0x7fe92c101840 2024-04-22T15:52:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.323+0000 7fe9340ee700 1 -- 172.21.15.16:0/654177767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe92c19bbc0 con 0x7fe92c101840 2024-04-22T15:52:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.323+0000 7fe922ffd700 1 -- 172.21.15.16:0/654177767 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9280055b0 con 0x7fe92c101840 2024-04-22T15:52:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.324+0000 7fe9340ee700 1 -- 172.21.15.16:0/654177767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe92c19c110 con 0x7fe92c101840 2024-04-22T15:52:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.324+0000 7fe9340ee700 1 -- 172.21.15.16:0/654177767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe92c109810 con 0x7fe92c101840 2024-04-22T15:52:12.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.326+0000 7fe922ffd700 1 -- 172.21.15.16:0/654177767 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7fe928007ad0 con 0x7fe92c101840 2024-04-22T15:52:12.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.327+0000 7fe922ffd700 1 --2- 172.21.15.16:0/654177767 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fe91806e940 0x7fe918070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:12.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.327+0000 7fe922ffd700 1 -- 172.21.15.16:0/654177767 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fe92808ea90 con 0x7fe92c101840 2024-04-22T15:52:12.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.327+0000 7fe931689700 1 --2- 172.21.15.16:0/654177767 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fe91806e940 0x7fe918070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:12.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.327+0000 7fe931689700 1 --2- 172.21.15.16:0/654177767 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fe91806e940 0x7fe918070e00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe91c00b5c0 tx=0x7fe91c005e20 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:12.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.330+0000 7fe922ffd700 1 -- 172.21.15.16:0/654177767 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7fe92805b900 con 0x7fe92c101840 2024-04-22T15:52:12.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:11 smithi027 ceph-mon[34922]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:12.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:11 smithi027 ceph-mon[34922]: from='client.24567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:12.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.562+0000 7fe9340ee700 1 -- 172.21.15.16:0/654177767 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe92c104de0 con 0x7fe91806e940 2024-04-22T15:52:12.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.564+0000 7fe922ffd700 1 -- 172.21.15.16:0/654177767 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7fe92c104de0 con 0x7fe91806e940 2024-04-22T15:52:12.566 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:52:12.567 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:52:12.567 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-22T15:52:12.567 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T15:52:12.567 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-22T15:52:12.567 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-04-22T15:52:12.567 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-22T15:52:12.567 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "2/2 daemons upgraded", 2024-04-22T15:52:12.567 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading grafana daemons", 2024-04-22T15:52:12.567 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-22T15:52:12.567 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:52:12.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.566+0000 7fe9340ee700 1 -- 172.21.15.16:0/654177767 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fe91806e940 msgr2=0x7fe918070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:12.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.566+0000 7fe9340ee700 1 --2- 172.21.15.16:0/654177767 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fe91806e940 0x7fe918070e00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe91c00b5c0 tx=0x7fe91c005e20 comp rx=0 tx=0).stop 2024-04-22T15:52:12.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.566+0000 7fe9340ee700 1 -- 172.21.15.16:0/654177767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe92c101840 msgr2=0x7fe92c196470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:12.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.566+0000 7fe9340ee700 1 --2- 172.21.15.16:0/654177767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe92c101840 0x7fe92c196470 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe92800c8a0 tx=0x7fe92800cc60 comp rx=0 tx=0).stop 2024-04-22T15:52:12.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.566+0000 7fe9340ee700 1 -- 172.21.15.16:0/654177767 shutdown_connections 2024-04-22T15:52:12.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.566+0000 7fe9340ee700 1 --2- 172.21.15.16:0/654177767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe92c101840 0x7fe92c196470 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:12.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.566+0000 7fe9340ee700 1 --2- 172.21.15.16:0/654177767 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fe91806e940 0x7fe918070e00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:12.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.566+0000 7fe9340ee700 1 --2- 172.21.15.16:0/654177767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92c104170 0x7fe92c1969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:12.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.566+0000 7fe9340ee700 1 -- 172.21.15.16:0/654177767 >> 172.21.15.16:0/654177767 conn(0x7fe92c0fb190 msgr2=0x7fe92c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:12.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.566+0000 7fe9340ee700 1 -- 172.21.15.16:0/654177767 shutdown_connections 2024-04-22T15:52:12.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:12.566+0000 7fe9340ee700 1 -- 172.21.15.16:0/654177767 wait complete. 2024-04-22T15:52:13.292 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:12 smithi016 ceph-mon[26620]: from='client.14794 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:13.293 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:12 smithi016 ceph-mon[26620]: pgmap v70: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:13.293 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:12 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/2797573692' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:13.293 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:12 smithi016 ceph-mon[26620]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:13.293 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:12 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:52:13.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:12 smithi027 ceph-mon[34922]: from='client.14794 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:13.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:12 smithi027 ceph-mon[34922]: pgmap v70: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:13.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:12 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/2797573692' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:13.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:12 smithi027 ceph-mon[34922]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:13.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:12 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:52:15.299 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:14 smithi016 ceph-mon[26620]: pgmap v71: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:15.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:14 smithi027 ceph-mon[34922]: pgmap v71: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:17.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:17 smithi016 ceph-mon[26620]: pgmap v72: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:52:17.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:17 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:17.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:17 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:17.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:17 smithi027 ceph-mon[34922]: pgmap v72: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:52:17.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:17 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:17.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:17 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:18.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:18 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:18.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:18 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:18.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:18 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:18.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:18 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:19.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:19 smithi027 ceph-mon[34922]: pgmap v73: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:19.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:19 smithi016 ceph-mon[26620]: pgmap v73: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:21.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: pgmap v74: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:52:21.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:21.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:21.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:52:21.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:52:21.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:21.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T15:52:21.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:52:21.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:21.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-22T15:52:21.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-22T15:52:21.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:52:21.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-22T15:52:21.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-22T15:52:21.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-22T15:52:21.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-22T15:52:21.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-22T15:52:21.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-22T15:52:21.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-22T15:52:21.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-22T15:52:21.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:52:21.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-22T15:52:21.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-22T15:52:21.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-22T15:52:21.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-22T15:52:21.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-22T15:52:21.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-22T15:52:21.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-22T15:52:21.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:52:21.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:52:21.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:52:21.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:52:21.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:52:21.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:52:21.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-22T15:52:21.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-22T15:52:21.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:52:21.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:52:21.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:52:21.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:21.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:52:21.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:52:21.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:52:21.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:21 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:21.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: pgmap v74: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:52:21.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:21.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:21.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:52:21.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:52:21.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:21.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T15:52:21.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:52:21.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.669 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.669 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:21.669 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:21.669 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-22T15:52:21.669 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-22T15:52:21.669 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:52:21.669 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-22T15:52:21.669 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-22T15:52:21.669 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-22T15:52:21.669 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-22T15:52:21.669 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-22T15:52:21.669 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-22T15:52:21.669 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-22T15:52:21.669 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-22T15:52:21.670 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:52:21.670 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-22T15:52:21.670 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-22T15:52:21.670 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-22T15:52:21.670 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-22T15:52:21.670 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-22T15:52:21.670 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-22T15:52:21.670 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-22T15:52:21.670 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:52:21.670 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:52:21.670 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:52:21.670 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:52:21.670 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:52:21.671 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:52:21.671 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-22T15:52:21.671 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-22T15:52:21.671 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:52:21.671 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:52:21.671 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:52:21.671 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:21.671 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:52:21.671 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:52:21.671 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:52:21.671 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:21 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:22.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:22 smithi027 ceph-mon[34922]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T15:52:22.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:22 smithi027 ceph-mon[34922]: Upgrade: Finalizing container_image settings 2024-04-22T15:52:22.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:22 smithi027 ceph-mon[34922]: Upgrade: Complete! 2024-04-22T15:52:22.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:22 smithi016 ceph-mon[26620]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T15:52:22.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:22 smithi016 ceph-mon[26620]: Upgrade: Finalizing container_image settings 2024-04-22T15:52:22.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:22 smithi016 ceph-mon[26620]: Upgrade: Complete! 2024-04-22T15:52:23.508 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:23 smithi016 ceph-mon[26620]: pgmap v75: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:23.509 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:23 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:23.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:23 smithi027 ceph-mon[34922]: pgmap v75: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:23.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:23 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:52:25.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:25 smithi027 ceph-mon[34922]: pgmap v76: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:25.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:25 smithi016 ceph-mon[26620]: pgmap v76: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:27.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:26 smithi027 ceph-mon[34922]: pgmap v77: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:52:27.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:26 smithi016 ceph-mon[26620]: pgmap v77: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:52:28.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:27 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:52:28.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:27 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:52:29.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:28 smithi016 ceph-mon[26620]: pgmap v78: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:29.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:28 smithi027 ceph-mon[34922]: pgmap v78: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:31.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:30 smithi027 ceph-mon[34922]: pgmap v79: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:52:31.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:30 smithi016 ceph-mon[26620]: pgmap v79: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:52:33.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:32 smithi027 ceph-mon[34922]: pgmap v80: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:33.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:32 smithi016 ceph-mon[26620]: pgmap v80: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:35.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:34 smithi027 ceph-mon[34922]: pgmap v81: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:35.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:34 smithi016 ceph-mon[26620]: pgmap v81: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:37.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:36 smithi027 ceph-mon[34922]: pgmap v82: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:52:37.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:36 smithi016 ceph-mon[26620]: pgmap v82: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:52:39.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:38 smithi016 ceph-mon[26620]: pgmap v83: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:39.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:38 smithi027 ceph-mon[34922]: pgmap v83: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:41.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:40 smithi027 ceph-mon[34922]: pgmap v84: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:52:41.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:40 smithi016 ceph-mon[26620]: pgmap v84: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:52:42.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.688+0000 7f695fa60700 1 -- 172.21.15.16:0/575880885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6958101eb0 msgr2=0x7f6958102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:42.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.688+0000 7f695fa60700 1 --2- 172.21.15.16:0/575880885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6958101eb0 0x7f6958102310 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6948009910 tx=0x7f6948009c20 comp rx=0 tx=0).stop 2024-04-22T15:52:42.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.688+0000 7f695fa60700 1 -- 172.21.15.16:0/575880885 shutdown_connections 2024-04-22T15:52:42.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.688+0000 7f695fa60700 1 --2- 172.21.15.16:0/575880885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6958101eb0 0x7f6958102310 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:42.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.688+0000 7f695fa60700 1 --2- 172.21.15.16:0/575880885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6958100cb0 0x7f69581010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:42.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.688+0000 7f695fa60700 1 -- 172.21.15.16:0/575880885 >> 172.21.15.16:0/575880885 conn(0x7f69580fc230 msgr2=0x7f69580fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:42.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.688+0000 7f695fa60700 1 -- 172.21.15.16:0/575880885 shutdown_connections 2024-04-22T15:52:42.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.688+0000 7f695fa60700 1 -- 172.21.15.16:0/575880885 wait complete. 2024-04-22T15:52:42.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.689+0000 7f695fa60700 1 Processor -- start 2024-04-22T15:52:42.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.689+0000 7f695fa60700 1 -- start start 2024-04-22T15:52:42.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.689+0000 7f695fa60700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6958100cb0 0x7f69581986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:42.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.689+0000 7f695fa60700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6958101eb0 0x7f6958198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:42.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.689+0000 7f695fa60700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6958199210 con 0x7f6958101eb0 2024-04-22T15:52:42.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.689+0000 7f695fa60700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6958199350 con 0x7f6958100cb0 2024-04-22T15:52:42.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.689+0000 7f695cffb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6958101eb0 0x7f6958198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:42.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.689+0000 7f695cffb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6958101eb0 0x7f6958198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41542/0 (socket says 172.21.15.16:41542) 2024-04-22T15:52:42.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.689+0000 7f695cffb700 1 -- 172.21.15.16:0/215462747 learned_addr learned my addr 172.21.15.16:0/215462747 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:52:42.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.690+0000 7f695d7fc700 1 --2- 172.21.15.16:0/215462747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6958100cb0 0x7f69581986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:42.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.690+0000 7f695cffb700 1 -- 172.21.15.16:0/215462747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6958100cb0 msgr2=0x7f69581986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:42.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.690+0000 7f695cffb700 1 --2- 172.21.15.16:0/215462747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6958100cb0 0x7f69581986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:42.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.690+0000 7f695cffb700 1 -- 172.21.15.16:0/215462747 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69480095f0 con 0x7f6958101eb0 2024-04-22T15:52:42.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.690+0000 7f695d7fc700 1 --2- 172.21.15.16:0/215462747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6958100cb0 0x7f69581986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:52:42.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.690+0000 7f695cffb700 1 --2- 172.21.15.16:0/215462747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6958101eb0 0x7f6958198bf0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6948006010 tx=0x7f694800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:42.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.690+0000 7f694e7fc700 1 -- 172.21.15.16:0/215462747 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f694801d070 con 0x7f6958101eb0 2024-04-22T15:52:42.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.690+0000 7f695fa60700 1 -- 172.21.15.16:0/215462747 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f695819dda0 con 0x7f6958101eb0 2024-04-22T15:52:42.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.690+0000 7f694e7fc700 1 -- 172.21.15.16:0/215462747 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f694800f460 con 0x7f6958101eb0 2024-04-22T15:52:42.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.690+0000 7f694e7fc700 1 -- 172.21.15.16:0/215462747 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6948021640 con 0x7f6958101eb0 2024-04-22T15:52:42.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.690+0000 7f695fa60700 1 -- 172.21.15.16:0/215462747 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f695819e290 con 0x7f6958101eb0 2024-04-22T15:52:42.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.691+0000 7f695fa60700 1 -- 172.21.15.16:0/215462747 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6958105f90 con 0x7f6958101eb0 2024-04-22T15:52:42.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.692+0000 7f694e7fc700 1 -- 172.21.15.16:0/215462747 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f694800fab0 con 0x7f6958101eb0 2024-04-22T15:52:42.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.692+0000 7f694e7fc700 1 --2- 172.21.15.16:0/215462747 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f694406e940 0x7f6944070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:42.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.692+0000 7f694e7fc700 1 -- 172.21.15.16:0/215462747 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f694808fcd0 con 0x7f6958101eb0 2024-04-22T15:52:42.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.693+0000 7f695d7fc700 1 --2- 172.21.15.16:0/215462747 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f694406e940 0x7f6944070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:42.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.693+0000 7f695d7fc700 1 --2- 172.21.15.16:0/215462747 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f694406e940 0x7f6944070e00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6958101d10 tx=0x7f6954005f90 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:42.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.698+0000 7f694e7fc700 1 -- 172.21.15.16:0/215462747 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f694805cae0 con 0x7f6958101eb0 2024-04-22T15:52:42.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.927+0000 7f695fa60700 1 -- 172.21.15.16:0/215462747 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f69580611d0 con 0x7f694406e940 2024-04-22T15:52:42.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.930+0000 7f694e7fc700 1 -- 172.21.15.16:0/215462747 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f69580611d0 con 0x7f694406e940 2024-04-22T15:52:42.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.932+0000 7f695fa60700 1 -- 172.21.15.16:0/215462747 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f694406e940 msgr2=0x7f6944070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:42.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.932+0000 7f695fa60700 1 --2- 172.21.15.16:0/215462747 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f694406e940 0x7f6944070e00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6958101d10 tx=0x7f6954005f90 comp rx=0 tx=0).stop 2024-04-22T15:52:42.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.932+0000 7f695fa60700 1 -- 172.21.15.16:0/215462747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6958101eb0 msgr2=0x7f6958198bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:42.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.932+0000 7f695fa60700 1 --2- 172.21.15.16:0/215462747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6958101eb0 0x7f6958198bf0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6948006010 tx=0x7f694800be20 comp rx=0 tx=0).stop 2024-04-22T15:52:42.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.932+0000 7f695fa60700 1 -- 172.21.15.16:0/215462747 shutdown_connections 2024-04-22T15:52:42.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.932+0000 7f695fa60700 1 --2- 172.21.15.16:0/215462747 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f694406e940 0x7f6944070e00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:42.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.932+0000 7f695fa60700 1 --2- 172.21.15.16:0/215462747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6958100cb0 0x7f69581986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:42.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.932+0000 7f695fa60700 1 --2- 172.21.15.16:0/215462747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6958101eb0 0x7f6958198bf0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:42.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.932+0000 7f695fa60700 1 -- 172.21.15.16:0/215462747 >> 172.21.15.16:0/215462747 conn(0x7f69580fc230 msgr2=0x7f69581030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:42.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.933+0000 7f695fa60700 1 -- 172.21.15.16:0/215462747 shutdown_connections 2024-04-22T15:52:42.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:42.933+0000 7f695fa60700 1 -- 172.21.15.16:0/215462747 wait complete. 2024-04-22T15:52:43.233 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:42 smithi016 ceph-mon[26620]: pgmap v85: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:43.233 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:42 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:52:43.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:42 smithi027 ceph-mon[34922]: pgmap v85: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:43.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:42 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:52:43.548 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-04-22T15:52:43.803 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:52:44.151 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:43 smithi016 ceph-mon[26620]: from='client.14806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:44.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:43 smithi027 ceph-mon[34922]: from='client.14806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:45.223 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:44 smithi016 ceph-mon[26620]: pgmap v86: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:45.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:44 smithi027 ceph-mon[34922]: pgmap v86: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:45.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.430+0000 7ff7e9574700 1 -- 172.21.15.16:0/112834185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7e4100ea0 msgr2=0x7ff7e4101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:45.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.430+0000 7ff7e9574700 1 --2- 172.21.15.16:0/112834185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7e4100ea0 0x7ff7e4101280 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ff7d4009910 tx=0x7ff7d4009c20 comp rx=0 tx=0).stop 2024-04-22T15:52:45.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.430+0000 7ff7e9574700 1 -- 172.21.15.16:0/112834185 shutdown_connections 2024-04-22T15:52:45.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.430+0000 7ff7e9574700 1 --2- 172.21.15.16:0/112834185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7e4101850 0x7ff7e41058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:45.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.430+0000 7ff7e9574700 1 --2- 172.21.15.16:0/112834185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7e4100ea0 0x7ff7e4101280 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:45.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.430+0000 7ff7e9574700 1 -- 172.21.15.16:0/112834185 >> 172.21.15.16:0/112834185 conn(0x7ff7e40fc750 msgr2=0x7ff7e40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:45.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.430+0000 7ff7e9574700 1 -- 172.21.15.16:0/112834185 shutdown_connections 2024-04-22T15:52:45.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.430+0000 7ff7e9574700 1 -- 172.21.15.16:0/112834185 wait complete. 2024-04-22T15:52:45.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.431+0000 7ff7e9574700 1 Processor -- start 2024-04-22T15:52:45.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.431+0000 7ff7e9574700 1 -- start start 2024-04-22T15:52:45.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.431+0000 7ff7e9574700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7e4100ea0 0x7ff7e419c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:45.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.431+0000 7ff7e9574700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7e4101850 0x7ff7e419cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:45.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.431+0000 7ff7e9574700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e419d2c0 con 0x7ff7e4101850 2024-04-22T15:52:45.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.431+0000 7ff7e9574700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e4196770 con 0x7ff7e4100ea0 2024-04-22T15:52:45.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.432+0000 7ff7e27fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7e4101850 0x7ff7e419cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:45.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.432+0000 7ff7e27fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7e4101850 0x7ff7e419cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41558/0 (socket says 172.21.15.16:41558) 2024-04-22T15:52:45.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.432+0000 7ff7e27fc700 1 -- 172.21.15.16:0/2052851920 learned_addr learned my addr 172.21.15.16:0/2052851920 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:52:45.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.432+0000 7ff7e2ffd700 1 --2- 172.21.15.16:0/2052851920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7e4100ea0 0x7ff7e419c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:45.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.432+0000 7ff7e27fc700 1 -- 172.21.15.16:0/2052851920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7e4100ea0 msgr2=0x7ff7e419c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:45.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.432+0000 7ff7e27fc700 1 --2- 172.21.15.16:0/2052851920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7e4100ea0 0x7ff7e419c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:45.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.432+0000 7ff7e27fc700 1 -- 172.21.15.16:0/2052851920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d40095f0 con 0x7ff7e4101850 2024-04-22T15:52:45.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.432+0000 7ff7e27fc700 1 --2- 172.21.15.16:0/2052851920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7e4101850 0x7ff7e419cc30 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff7d800d900 tx=0x7ff7d800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:45.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.432+0000 7ff7cbfff700 1 -- 172.21.15.16:0/2052851920 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7d8009900 con 0x7ff7e4101850 2024-04-22T15:52:45.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.432+0000 7ff7e9574700 1 -- 172.21.15.16:0/2052851920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7e4196a50 con 0x7ff7e4101850 2024-04-22T15:52:45.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.432+0000 7ff7cbfff700 1 -- 172.21.15.16:0/2052851920 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff7d8010460 con 0x7ff7e4101850 2024-04-22T15:52:45.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.432+0000 7ff7cbfff700 1 -- 172.21.15.16:0/2052851920 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7d800f5f0 con 0x7ff7e4101850 2024-04-22T15:52:45.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.432+0000 7ff7e9574700 1 -- 172.21.15.16:0/2052851920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7e4196fa0 con 0x7ff7e4101850 2024-04-22T15:52:45.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.433+0000 7ff7e9574700 1 -- 172.21.15.16:0/2052851920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7e4109240 con 0x7ff7e4101850 2024-04-22T15:52:45.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.434+0000 7ff7cbfff700 1 -- 172.21.15.16:0/2052851920 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7ff7d80105d0 con 0x7ff7e4101850 2024-04-22T15:52:45.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.434+0000 7ff7cbfff700 1 --2- 172.21.15.16:0/2052851920 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7ff7cc06e940 0x7ff7cc070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:45.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.434+0000 7ff7cbfff700 1 -- 172.21.15.16:0/2052851920 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7ff7d808e470 con 0x7ff7e4101850 2024-04-22T15:52:45.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.434+0000 7ff7e2ffd700 1 --2- 172.21.15.16:0/2052851920 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7ff7cc06e940 0x7ff7cc070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:45.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.435+0000 7ff7e2ffd700 1 --2- 172.21.15.16:0/2052851920 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7ff7cc06e940 0x7ff7cc070e00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff7d400b5c0 tx=0x7ff7d401a040 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:45.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.439+0000 7ff7cbfff700 1 -- 172.21.15.16:0/2052851920 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7ff7d805b280 con 0x7ff7e4101850 2024-04-22T15:52:45.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.737+0000 7ff7e9574700 1 -- 172.21.15.16:0/2052851920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff7e404f2e0 con 0x7ff7e4101850 2024-04-22T15:52:45.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.737+0000 7ff7cbfff700 1 -- 172.21.15.16:0/2052851920 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7ff7d805ae10 con 0x7ff7e4101850 2024-04-22T15:52:45.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.739+0000 7ff7e9574700 1 -- 172.21.15.16:0/2052851920 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7ff7cc06e940 msgr2=0x7ff7cc070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:45.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.739+0000 7ff7e9574700 1 --2- 172.21.15.16:0/2052851920 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7ff7cc06e940 0x7ff7cc070e00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff7d400b5c0 tx=0x7ff7d401a040 comp rx=0 tx=0).stop 2024-04-22T15:52:45.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.739+0000 7ff7e9574700 1 -- 172.21.15.16:0/2052851920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7e4101850 msgr2=0x7ff7e419cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:45.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.739+0000 7ff7e9574700 1 --2- 172.21.15.16:0/2052851920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7e4101850 0x7ff7e419cc30 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff7d800d900 tx=0x7ff7d800dc10 comp rx=0 tx=0).stop 2024-04-22T15:52:45.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.739+0000 7ff7e9574700 1 -- 172.21.15.16:0/2052851920 shutdown_connections 2024-04-22T15:52:45.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.739+0000 7ff7e9574700 1 --2- 172.21.15.16:0/2052851920 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7ff7cc06e940 0x7ff7cc070e00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:45.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.739+0000 7ff7e9574700 1 --2- 172.21.15.16:0/2052851920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7e4100ea0 0x7ff7e419c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:45.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.739+0000 7ff7e9574700 1 --2- 172.21.15.16:0/2052851920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7e4101850 0x7ff7e419cc30 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:45.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.739+0000 7ff7e9574700 1 -- 172.21.15.16:0/2052851920 >> 172.21.15.16:0/2052851920 conn(0x7ff7e40fc750 msgr2=0x7ff7e40fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:45.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.740+0000 7ff7e9574700 1 -- 172.21.15.16:0/2052851920 shutdown_connections 2024-04-22T15:52:45.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:45.740+0000 7ff7e9574700 1 -- 172.21.15.16:0/2052851920 wait complete. 2024-04-22T15:52:45.753 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:52:46.367 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-04-22T15:52:46.591 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:52:46.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:46 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/2052851920' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:46.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:46 smithi016 ceph-mon[26620]: pgmap v87: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:52:47.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:46 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/2052851920' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:47.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:46 smithi027 ceph-mon[34922]: pgmap v87: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:52:48.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.310+0000 7fde8172f700 1 -- 172.21.15.16:0/3033651274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c0ffda0 msgr2=0x7fde7c100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:48.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.310+0000 7fde8172f700 1 --2- 172.21.15.16:0/3033651274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c0ffda0 0x7fde7c100180 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fde6c009910 tx=0x7fde6c009c20 comp rx=0 tx=0).stop 2024-04-22T15:52:48.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.310+0000 7fde8172f700 1 -- 172.21.15.16:0/3033651274 shutdown_connections 2024-04-22T15:52:48.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.310+0000 7fde8172f700 1 --2- 172.21.15.16:0/3033651274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde7c100750 0x7fde7c1047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:48.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.310+0000 7fde8172f700 1 --2- 172.21.15.16:0/3033651274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c0ffda0 0x7fde7c100180 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:48.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.310+0000 7fde8172f700 1 -- 172.21.15.16:0/3033651274 >> 172.21.15.16:0/3033651274 conn(0x7fde7c0fb650 msgr2=0x7fde7c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:48.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.310+0000 7fde8172f700 1 -- 172.21.15.16:0/3033651274 shutdown_connections 2024-04-22T15:52:48.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.310+0000 7fde8172f700 1 -- 172.21.15.16:0/3033651274 wait complete. 2024-04-22T15:52:48.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.310+0000 7fde8172f700 1 Processor -- start 2024-04-22T15:52:48.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.311+0000 7fde8172f700 1 -- start start 2024-04-22T15:52:48.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.311+0000 7fde8172f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde7c0ffda0 0x7fde7c19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:48.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.311+0000 7fde8172f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c100750 0x7fde7c19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:48.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.311+0000 7fde8172f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c19d3b0 con 0x7fde7c100750 2024-04-22T15:52:48.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.311+0000 7fde8172f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c196860 con 0x7fde7c0ffda0 2024-04-22T15:52:48.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.311+0000 7fde7a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c100750 0x7fde7c19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:48.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.311+0000 7fde7a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c100750 0x7fde7c19cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41574/0 (socket says 172.21.15.16:41574) 2024-04-22T15:52:48.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.311+0000 7fde7a7fc700 1 -- 172.21.15.16:0/13925325 learned_addr learned my addr 172.21.15.16:0/13925325 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:52:48.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.311+0000 7fde7affd700 1 --2- 172.21.15.16:0/13925325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde7c0ffda0 0x7fde7c19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:48.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.311+0000 7fde7a7fc700 1 -- 172.21.15.16:0/13925325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde7c0ffda0 msgr2=0x7fde7c19c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:48.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.311+0000 7fde7a7fc700 1 --2- 172.21.15.16:0/13925325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde7c0ffda0 0x7fde7c19c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:48.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.311+0000 7fde7a7fc700 1 -- 172.21.15.16:0/13925325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c0095f0 con 0x7fde7c100750 2024-04-22T15:52:48.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.312+0000 7fde7a7fc700 1 --2- 172.21.15.16:0/13925325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c100750 0x7fde7c19cd20 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fde7000b480 tx=0x7fde7000b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:48.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.312+0000 7fde63fff700 1 -- 172.21.15.16:0/13925325 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde70011800 con 0x7fde7c100750 2024-04-22T15:52:48.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.312+0000 7fde63fff700 1 -- 172.21.15.16:0/13925325 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fde70011e20 con 0x7fde7c100750 2024-04-22T15:52:48.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.312+0000 7fde8172f700 1 -- 172.21.15.16:0/13925325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde7c196b40 con 0x7fde7c100750 2024-04-22T15:52:48.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.312+0000 7fde63fff700 1 -- 172.21.15.16:0/13925325 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde70010ea0 con 0x7fde7c100750 2024-04-22T15:52:48.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.312+0000 7fde8172f700 1 -- 172.21.15.16:0/13925325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde7c197090 con 0x7fde7c100750 2024-04-22T15:52:48.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.313+0000 7fde8172f700 1 -- 172.21.15.16:0/13925325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde7c04f2e0 con 0x7fde7c100750 2024-04-22T15:52:48.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.313+0000 7fde63fff700 1 -- 172.21.15.16:0/13925325 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7fde70010380 con 0x7fde7c100750 2024-04-22T15:52:48.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.314+0000 7fde63fff700 1 --2- 172.21.15.16:0/13925325 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fde6406e940 0x7fde64070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:48.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.314+0000 7fde63fff700 1 -- 172.21.15.16:0/13925325 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fde7008e360 con 0x7fde7c100750 2024-04-22T15:52:48.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.314+0000 7fde7affd700 1 --2- 172.21.15.16:0/13925325 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fde6406e940 0x7fde64070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:48.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.314+0000 7fde7affd700 1 --2- 172.21.15.16:0/13925325 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fde6406e940 0x7fde64070e00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fde6c005f50 tx=0x7fde6c01a040 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:48.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.319+0000 7fde63fff700 1 -- 172.21.15.16:0/13925325 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7fde7005b170 con 0x7fde7c100750 2024-04-22T15:52:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.613+0000 7fde8172f700 1 -- 172.21.15.16:0/13925325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fde7c197310 con 0x7fde7c100750 2024-04-22T15:52:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.613+0000 7fde63fff700 1 -- 172.21.15.16:0/13925325 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7fde7005ad00 con 0x7fde7c100750 2024-04-22T15:52:48.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.615+0000 7fde8172f700 1 -- 172.21.15.16:0/13925325 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fde6406e940 msgr2=0x7fde64070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:48.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.615+0000 7fde8172f700 1 --2- 172.21.15.16:0/13925325 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fde6406e940 0x7fde64070e00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fde6c005f50 tx=0x7fde6c01a040 comp rx=0 tx=0).stop 2024-04-22T15:52:48.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.615+0000 7fde8172f700 1 -- 172.21.15.16:0/13925325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c100750 msgr2=0x7fde7c19cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:48.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.615+0000 7fde8172f700 1 --2- 172.21.15.16:0/13925325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c100750 0x7fde7c19cd20 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fde7000b480 tx=0x7fde7000b790 comp rx=0 tx=0).stop 2024-04-22T15:52:48.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.615+0000 7fde8172f700 1 -- 172.21.15.16:0/13925325 shutdown_connections 2024-04-22T15:52:48.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.615+0000 7fde8172f700 1 --2- 172.21.15.16:0/13925325 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fde6406e940 0x7fde64070e00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:48.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.615+0000 7fde8172f700 1 --2- 172.21.15.16:0/13925325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde7c0ffda0 0x7fde7c19c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:48.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.615+0000 7fde8172f700 1 --2- 172.21.15.16:0/13925325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c100750 0x7fde7c19cd20 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:48.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.615+0000 7fde8172f700 1 -- 172.21.15.16:0/13925325 >> 172.21.15.16:0/13925325 conn(0x7fde7c0fb650 msgr2=0x7fde7c103f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:48.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.615+0000 7fde8172f700 1 -- 172.21.15.16:0/13925325 shutdown_connections 2024-04-22T15:52:48.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:48.615+0000 7fde8172f700 1 -- 172.21.15.16:0/13925325 wait complete. 2024-04-22T15:52:48.629 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)" 2024-04-22T15:52:49.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:48 smithi027 ceph-mon[34922]: pgmap v88: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:49.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:48 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/13925325' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:49.373 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:48 smithi016 ceph-mon[26620]: pgmap v88: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:49.374 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:48 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/13925325' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:49.396 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-04-22T15:52:49.611 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:52:50.995 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:50 smithi016 ceph-mon[26620]: pgmap v89: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:52:51.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.220+0000 7f00199bf700 1 -- 172.21.15.16:0/4045563627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00140ffda0 msgr2=0x7f0014100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:51.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.220+0000 7f00199bf700 1 --2- 172.21.15.16:0/4045563627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00140ffda0 0x7f0014100180 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0004009910 tx=0x7f0004009c20 comp rx=0 tx=0).stop 2024-04-22T15:52:51.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.220+0000 7f00199bf700 1 -- 172.21.15.16:0/4045563627 shutdown_connections 2024-04-22T15:52:51.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.220+0000 7f00199bf700 1 --2- 172.21.15.16:0/4045563627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0014100750 0x7f00141047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:51.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.220+0000 7f00199bf700 1 --2- 172.21.15.16:0/4045563627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00140ffda0 0x7f0014100180 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:51.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.220+0000 7f00199bf700 1 -- 172.21.15.16:0/4045563627 >> 172.21.15.16:0/4045563627 conn(0x7f00140fb650 msgr2=0x7f00140fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:51.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.220+0000 7f00199bf700 1 -- 172.21.15.16:0/4045563627 shutdown_connections 2024-04-22T15:52:51.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.220+0000 7f00199bf700 1 -- 172.21.15.16:0/4045563627 wait complete. 2024-04-22T15:52:51.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.221+0000 7f00199bf700 1 Processor -- start 2024-04-22T15:52:51.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.221+0000 7f00199bf700 1 -- start start 2024-04-22T15:52:51.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.221+0000 7f00199bf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00140ffda0 0x7f001419c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:51.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.221+0000 7f00199bf700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0014100750 0x7f001419cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:51.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.221+0000 7f00199bf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001419d2c0 con 0x7f00140ffda0 2024-04-22T15:52:51.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.221+0000 7f00199bf700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0014196770 con 0x7f0014100750 2024-04-22T15:52:51.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.221+0000 7f0012ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00140ffda0 0x7f001419c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:51.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.222+0000 7f0012ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00140ffda0 0x7f001419c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41590/0 (socket says 172.21.15.16:41590) 2024-04-22T15:52:51.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.222+0000 7f0012ffd700 1 -- 172.21.15.16:0/3281749477 learned_addr learned my addr 172.21.15.16:0/3281749477 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:52:51.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.222+0000 7f00127fc700 1 --2- 172.21.15.16:0/3281749477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0014100750 0x7f001419cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:51.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.222+0000 7f0012ffd700 1 -- 172.21.15.16:0/3281749477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0014100750 msgr2=0x7f001419cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:51.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.222+0000 7f0012ffd700 1 --2- 172.21.15.16:0/3281749477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0014100750 0x7f001419cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:51.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.222+0000 7f0012ffd700 1 -- 172.21.15.16:0/3281749477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00040095f0 con 0x7f00140ffda0 2024-04-22T15:52:51.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.222+0000 7f0012ffd700 1 --2- 172.21.15.16:0/3281749477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00140ffda0 0x7f001419c6f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0004006010 tx=0x7f000400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:51.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.222+0000 7f00189bd700 1 -- 172.21.15.16:0/3281749477 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f000401d070 con 0x7f00140ffda0 2024-04-22T15:52:51.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.222+0000 7f00189bd700 1 -- 172.21.15.16:0/3281749477 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f000400f460 con 0x7f00140ffda0 2024-04-22T15:52:51.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.222+0000 7f00199bf700 1 -- 172.21.15.16:0/3281749477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00141969f0 con 0x7f00140ffda0 2024-04-22T15:52:51.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.222+0000 7f00189bd700 1 -- 172.21.15.16:0/3281749477 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0004017630 con 0x7f00140ffda0 2024-04-22T15:52:51.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.222+0000 7f00199bf700 1 -- 172.21.15.16:0/3281749477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0014196ee0 con 0x7f00140ffda0 2024-04-22T15:52:51.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.223+0000 7f00199bf700 1 -- 172.21.15.16:0/3281749477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f001404f2e0 con 0x7f00140ffda0 2024-04-22T15:52:51.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.224+0000 7f00189bd700 1 -- 172.21.15.16:0/3281749477 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f000400f5d0 con 0x7f00140ffda0 2024-04-22T15:52:51.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.224+0000 7f00189bd700 1 --2- 172.21.15.16:0/3281749477 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7efffc06e940 0x7efffc070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:51.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.224+0000 7f00189bd700 1 -- 172.21.15.16:0/3281749477 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f000408fd30 con 0x7f00140ffda0 2024-04-22T15:52:51.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.224+0000 7f00127fc700 1 --2- 172.21.15.16:0/3281749477 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7efffc06e940 0x7efffc070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:51.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.225+0000 7f00127fc700 1 --2- 172.21.15.16:0/3281749477 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7efffc06e940 0x7efffc070e00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0008005950 tx=0x7f00080058e0 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:51.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.229+0000 7f00189bd700 1 -- 172.21.15.16:0/3281749477 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f000405cb40 con 0x7f00140ffda0 2024-04-22T15:52:51.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:50 smithi027 ceph-mon[34922]: pgmap v89: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:52:51.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.528+0000 7f00199bf700 1 -- 172.21.15.16:0/3281749477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0014197100 con 0x7f00140ffda0 2024-04-22T15:52:51.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.529+0000 7f00189bd700 1 -- 172.21.15.16:0/3281749477 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f000405c6d0 con 0x7f00140ffda0 2024-04-22T15:52:51.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.530+0000 7f00199bf700 1 -- 172.21.15.16:0/3281749477 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7efffc06e940 msgr2=0x7efffc070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:51.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.531+0000 7f00199bf700 1 --2- 172.21.15.16:0/3281749477 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7efffc06e940 0x7efffc070e00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0008005950 tx=0x7f00080058e0 comp rx=0 tx=0).stop 2024-04-22T15:52:51.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.531+0000 7f00199bf700 1 -- 172.21.15.16:0/3281749477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00140ffda0 msgr2=0x7f001419c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:51.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.531+0000 7f00199bf700 1 --2- 172.21.15.16:0/3281749477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00140ffda0 0x7f001419c6f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0004006010 tx=0x7f000400be20 comp rx=0 tx=0).stop 2024-04-22T15:52:51.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.531+0000 7f00199bf700 1 -- 172.21.15.16:0/3281749477 shutdown_connections 2024-04-22T15:52:51.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.531+0000 7f00199bf700 1 --2- 172.21.15.16:0/3281749477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00140ffda0 0x7f001419c6f0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:51.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.531+0000 7f00199bf700 1 --2- 172.21.15.16:0/3281749477 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7efffc06e940 0x7efffc070e00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:51.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.531+0000 7f00199bf700 1 --2- 172.21.15.16:0/3281749477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0014100750 0x7f001419cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:51.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.531+0000 7f00199bf700 1 -- 172.21.15.16:0/3281749477 >> 172.21.15.16:0/3281749477 conn(0x7f00140fb650 msgr2=0x7f0014103fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:51.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.531+0000 7f00199bf700 1 -- 172.21.15.16:0/3281749477 shutdown_connections 2024-04-22T15:52:51.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:51.531+0000 7f00199bf700 1 -- 172.21.15.16:0/3281749477 wait complete. 2024-04-22T15:52:51.544 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:52:52.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:51 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/3281749477' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:52.223 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-04-22T15:52:52.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:51 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/3281749477' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:52:52.443 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:52:53.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:52 smithi016 ceph-mon[26620]: pgmap v90: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:53.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:52 smithi027 ceph-mon[34922]: pgmap v90: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:54.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.022+0000 7f484db4b700 1 -- 172.21.15.16:0/1847079738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48480ffc20 msgr2=0x7f4848100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:54.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.022+0000 7f484db4b700 1 --2- 172.21.15.16:0/1847079738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48480ffc20 0x7f4848100080 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4830009910 tx=0x7f4830009c20 comp rx=0 tx=0).stop 2024-04-22T15:52:54.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.022+0000 7f484db4b700 1 -- 172.21.15.16:0/1847079738 shutdown_connections 2024-04-22T15:52:54.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.022+0000 7f484db4b700 1 --2- 172.21.15.16:0/1847079738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48480ffc20 0x7f4848100080 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:54.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.022+0000 7f484db4b700 1 --2- 172.21.15.16:0/1847079738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4848105bd0 0x7f4848105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:54.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.022+0000 7f484db4b700 1 -- 172.21.15.16:0/1847079738 >> 172.21.15.16:0/1847079738 conn(0x7f4848075670 msgr2=0x7f4848075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:54.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.022+0000 7f484db4b700 1 -- 172.21.15.16:0/1847079738 shutdown_connections 2024-04-22T15:52:54.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.022+0000 7f484db4b700 1 -- 172.21.15.16:0/1847079738 wait complete. 2024-04-22T15:52:54.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.023+0000 7f484db4b700 1 Processor -- start 2024-04-22T15:52:54.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.023+0000 7f484db4b700 1 -- start start 2024-04-22T15:52:54.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.023+0000 7f484db4b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48480ffc20 0x7f484819c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:54.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.023+0000 7f484db4b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4848105bd0 0x7f484819cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:54.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.024+0000 7f484db4b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484819d160 con 0x7f48480ffc20 2024-04-22T15:52:54.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.024+0000 7f484db4b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48481966a0 con 0x7f4848105bd0 2024-04-22T15:52:54.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.024+0000 7f48477fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48480ffc20 0x7f484819c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:54.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.024+0000 7f48477fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48480ffc20 0x7f484819c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46112/0 (socket says 172.21.15.16:46112) 2024-04-22T15:52:54.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.024+0000 7f48477fe700 1 -- 172.21.15.16:0/2662340748 learned_addr learned my addr 172.21.15.16:0/2662340748 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:52:54.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.024+0000 7f4846ffd700 1 --2- 172.21.15.16:0/2662340748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4848105bd0 0x7f484819cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:54.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.024+0000 7f48477fe700 1 -- 172.21.15.16:0/2662340748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4848105bd0 msgr2=0x7f484819cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:54.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.024+0000 7f48477fe700 1 --2- 172.21.15.16:0/2662340748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4848105bd0 0x7f484819cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:54.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.024+0000 7f48477fe700 1 -- 172.21.15.16:0/2662340748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48300095f0 con 0x7f48480ffc20 2024-04-22T15:52:54.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.024+0000 7f48477fe700 1 --2- 172.21.15.16:0/2662340748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48480ffc20 0x7f484819c620 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f483800c930 tx=0x7f483800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:54.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.024+0000 7f4844ff9700 1 -- 172.21.15.16:0/2662340748 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4838007ad0 con 0x7f48480ffc20 2024-04-22T15:52:54.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.025+0000 7f484db4b700 1 -- 172.21.15.16:0/2662340748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4848196980 con 0x7f48480ffc20 2024-04-22T15:52:54.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.025+0000 7f4844ff9700 1 -- 172.21.15.16:0/2662340748 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f483800ce80 con 0x7f48480ffc20 2024-04-22T15:52:54.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.025+0000 7f4844ff9700 1 -- 172.21.15.16:0/2662340748 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4838018630 con 0x7f48480ffc20 2024-04-22T15:52:54.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.025+0000 7f484db4b700 1 -- 172.21.15.16:0/2662340748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4848196ed0 con 0x7f48480ffc20 2024-04-22T15:52:54.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.026+0000 7f484db4b700 1 -- 172.21.15.16:0/2662340748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f484804f2e0 con 0x7f48480ffc20 2024-04-22T15:52:54.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.026+0000 7f4844ff9700 1 -- 172.21.15.16:0/2662340748 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f4838007c30 con 0x7f48480ffc20 2024-04-22T15:52:54.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.027+0000 7f4844ff9700 1 --2- 172.21.15.16:0/2662340748 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f483406e940 0x7f4834070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:54.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.027+0000 7f4844ff9700 1 -- 172.21.15.16:0/2662340748 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f483808e820 con 0x7f48480ffc20 2024-04-22T15:52:54.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.027+0000 7f4846ffd700 1 --2- 172.21.15.16:0/2662340748 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f483406e940 0x7f4834070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:54.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.027+0000 7f4846ffd700 1 --2- 172.21.15.16:0/2662340748 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f483406e940 0x7f4834070e00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f48480ffa40 tx=0x7f483001a040 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:54.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.032+0000 7f4844ff9700 1 -- 172.21.15.16:0/2662340748 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f483805b610 con 0x7f48480ffc20 2024-04-22T15:52:54.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:54.260+0000 7f484db4b700 1 -- 172.21.15.16:0/2662340748 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa", "target": ["mon-mgr", ""]}) v1 -- 0x7f4848197200 con 0x7f483406e940 2024-04-22T15:52:55.221 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:54 smithi016 ceph-mon[26620]: pgmap v91: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:55.221 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:54 smithi016 ceph-mon[26620]: from='client.14822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:55.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:54 smithi027 ceph-mon[34922]: pgmap v91: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:55.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:54 smithi027 ceph-mon[34922]: from='client.14822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:52:56.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:56.381+0000 7f4844ff9700 1 -- 172.21.15.16:0/2662340748 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+5391 (secure 0 0 0) 0x7f4848197200 con 0x7f483406e940 2024-04-22T15:52:56.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:56.383+0000 7f484db4b700 1 -- 172.21.15.16:0/2662340748 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f483406e940 msgr2=0x7f4834070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:56.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:56.383+0000 7f484db4b700 1 --2- 172.21.15.16:0/2662340748 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f483406e940 0x7f4834070e00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f48480ffa40 tx=0x7f483001a040 comp rx=0 tx=0).stop 2024-04-22T15:52:56.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:56.383+0000 7f484db4b700 1 -- 172.21.15.16:0/2662340748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48480ffc20 msgr2=0x7f484819c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:56.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:56.383+0000 7f484db4b700 1 --2- 172.21.15.16:0/2662340748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48480ffc20 0x7f484819c620 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f483800c930 tx=0x7f483800cc40 comp rx=0 tx=0).stop 2024-04-22T15:52:56.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:56.383+0000 7f484db4b700 1 -- 172.21.15.16:0/2662340748 shutdown_connections 2024-04-22T15:52:56.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:56.383+0000 7f484db4b700 1 --2- 172.21.15.16:0/2662340748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48480ffc20 0x7f484819c620 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:56.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:56.383+0000 7f484db4b700 1 --2- 172.21.15.16:0/2662340748 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f483406e940 0x7f4834070e00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:56.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:56.383+0000 7f484db4b700 1 --2- 172.21.15.16:0/2662340748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4848105bd0 0x7f484819cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:56.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:56.384+0000 7f484db4b700 1 -- 172.21.15.16:0/2662340748 >> 172.21.15.16:0/2662340748 conn(0x7f4848075670 msgr2=0x7f4848068680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:56.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:56.384+0000 7f484db4b700 1 -- 172.21.15.16:0/2662340748 shutdown_connections 2024-04-22T15:52:56.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:56.384+0000 7f484db4b700 1 -- 172.21.15.16:0/2662340748 wait complete. 2024-04-22T15:52:56.400 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:52:56.972 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:56 smithi016 ceph-mon[26620]: pgmap v92: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:52:57.000 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph orch ps' 2024-04-22T15:52:57.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:56 smithi027 ceph-mon[34922]: pgmap v92: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:52:57.224 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:52:58.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:57 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:52:58.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:57 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:52:58.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.652+0000 7effe0a33700 1 -- 172.21.15.16:0/703983979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effdc102720 msgr2=0x7effdc10ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:58.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.652+0000 7effe0a33700 1 --2- 172.21.15.16:0/703983979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effdc102720 0x7effdc10ac10 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7effd0009910 tx=0x7effd0009c20 comp rx=0 tx=0).stop 2024-04-22T15:52:58.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.652+0000 7effe0a33700 1 -- 172.21.15.16:0/703983979 shutdown_connections 2024-04-22T15:52:58.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.652+0000 7effe0a33700 1 --2- 172.21.15.16:0/703983979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effdc102720 0x7effdc10ac10 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:58.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.652+0000 7effe0a33700 1 --2- 172.21.15.16:0/703983979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc101e00 0x7effdc1021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:58.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.652+0000 7effe0a33700 1 -- 172.21.15.16:0/703983979 >> 172.21.15.16:0/703983979 conn(0x7effdc0fb650 msgr2=0x7effdc0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:58.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.652+0000 7effe0a33700 1 -- 172.21.15.16:0/703983979 shutdown_connections 2024-04-22T15:52:58.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.652+0000 7effe0a33700 1 -- 172.21.15.16:0/703983979 wait complete. 2024-04-22T15:52:58.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.653+0000 7effe0a33700 1 Processor -- start 2024-04-22T15:52:58.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.653+0000 7effe0a33700 1 -- start start 2024-04-22T15:52:58.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.653+0000 7effe0a33700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc101e00 0x7effdc19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:58.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.653+0000 7effe0a33700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effdc102720 0x7effdc19cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:58.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.653+0000 7effe0a33700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc19d400 con 0x7effdc102720 2024-04-22T15:52:58.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.653+0000 7effe0a33700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc1968b0 con 0x7effdc101e00 2024-04-22T15:52:58.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.653+0000 7effd9d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effdc102720 0x7effdc19cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:58.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.653+0000 7effd9d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effdc102720 0x7effdc19cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46126/0 (socket says 172.21.15.16:46126) 2024-04-22T15:52:58.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.653+0000 7effd9d9b700 1 -- 172.21.15.16:0/3782203750 learned_addr learned my addr 172.21.15.16:0/3782203750 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:52:58.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.653+0000 7effda59c700 1 --2- 172.21.15.16:0/3782203750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc101e00 0x7effdc19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:58.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.654+0000 7effd9d9b700 1 -- 172.21.15.16:0/3782203750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc101e00 msgr2=0x7effdc19c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:58.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.654+0000 7effd9d9b700 1 --2- 172.21.15.16:0/3782203750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc101e00 0x7effdc19c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:58.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.654+0000 7effd9d9b700 1 -- 172.21.15.16:0/3782203750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd00095f0 con 0x7effdc102720 2024-04-22T15:52:58.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.654+0000 7effd9d9b700 1 --2- 172.21.15.16:0/3782203750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effdc102720 0x7effdc19cd70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7effd000bd70 tx=0x7effd000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:58.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.654+0000 7effcb7fe700 1 -- 172.21.15.16:0/3782203750 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effd001d070 con 0x7effdc102720 2024-04-22T15:52:58.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.654+0000 7effcb7fe700 1 -- 172.21.15.16:0/3782203750 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effd000f460 con 0x7effdc102720 2024-04-22T15:52:58.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.654+0000 7effcb7fe700 1 -- 172.21.15.16:0/3782203750 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effd0017630 con 0x7effdc102720 2024-04-22T15:52:58.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.654+0000 7effe0a33700 1 -- 172.21.15.16:0/3782203750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effdc196b30 con 0x7effdc102720 2024-04-22T15:52:58.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.654+0000 7effe0a33700 1 -- 172.21.15.16:0/3782203750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effdc196fd0 con 0x7effdc102720 2024-04-22T15:52:58.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.655+0000 7effe0a33700 1 -- 172.21.15.16:0/3782203750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effdc108310 con 0x7effdc102720 2024-04-22T15:52:58.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.656+0000 7effcb7fe700 1 -- 172.21.15.16:0/3782203750 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7effd000fab0 con 0x7effdc102720 2024-04-22T15:52:58.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.656+0000 7effcb7fe700 1 --2- 172.21.15.16:0/3782203750 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7effc406e940 0x7effc4070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:52:58.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.656+0000 7effcb7fe700 1 -- 172.21.15.16:0/3782203750 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7effd008fcd0 con 0x7effdc102720 2024-04-22T15:52:58.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.656+0000 7effda59c700 1 --2- 172.21.15.16:0/3782203750 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7effc406e940 0x7effc4070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:52:58.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.657+0000 7effda59c700 1 --2- 172.21.15.16:0/3782203750 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7effc406e940 0x7effc4070e00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7effcc009a10 tx=0x7effcc009190 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:52:58.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.661+0000 7effcb7fe700 1 -- 172.21.15.16:0/3782203750 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7effd005cae0 con 0x7effdc102720 2024-04-22T15:52:58.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.878+0000 7effe0a33700 1 -- 172.21.15.16:0/3782203750 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7effdc197dc0 con 0x7effc406e940 2024-04-22T15:52:58.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.889+0000 7effcb7fe700 1 -- 172.21.15.16:0/3782203750 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7effdc197dc0 con 0x7effc406e940 2024-04-22T15:52:58.893 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:52:58.893 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.smithi016 smithi016 *:9093,9094 running (74s) 42s ago 11m 17.1M - 0.25.0 c8568f914cd2 ea0e0217d6f9 2024-04-22T15:52:58.893 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi016 smithi016 running (12m) 42s ago 12m 7884k - 18.2.1 5be31c24972a f8b627d3f41e 2024-04-22T15:52:58.893 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi027 smithi027 running (11m) 114s ago 11m 7776k - 18.2.1 5be31c24972a f00fd40b4220 2024-04-22T15:52:58.893 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi016 smithi016 running (12m) 42s ago 12m 6660k - 18.2.1 5be31c24972a 6b93db1fa0d9 2024-04-22T15:52:58.893 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi027 smithi027 running (11m) 114s ago 11m 6665k - 18.2.1 5be31c24972a d3bd439f30a0 2024-04-22T15:52:58.893 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.smithi016 smithi016 *:3000 running (49s) 42s ago 11m 52.7M - 9.4.12 2bacad6d85d8 388a3c9c1b3f 2024-04-22T15:52:58.893 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.fkkejm smithi016 running (6m) 42s ago 6m 14.7M - 18.2.1 5be31c24972a 4153ac94cea0 2024-04-22T15:52:58.893 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.pgvphu smithi016 running (6m) 42s ago 6m 91.7M - 18.2.1 5be31c24972a 93dc072068a2 2024-04-22T15:52:58.893 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.dgqyes smithi027 running (6m) 114s ago 6m 17.3M - 18.2.1 5be31c24972a 41a9220430fd 2024-04-22T15:52:58.893 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.iobjds smithi027 running (6m) 114s ago 6m 241M - 18.2.1 5be31c24972a c1c8d6770f02 2024-04-22T15:52:58.893 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi016.rnsnnr smithi016 *:8443,9283,8765 running (3m) 42s ago 13m 513M - 19.0.0-3223-g5883d494 3d983c26e278 09b4fe285bc4 2024-04-22T15:52:58.894 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi027.ukdgzs smithi027 *:8443,9283,8765 running (2m) 114s ago 10m 450M - 19.0.0-3223-g5883d494 3d983c26e278 7548da9b5eb9 2024-04-22T15:52:58.894 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi016 smithi016 running (13m) 42s ago 13m 60.3M 2048M 18.2.1 5be31c24972a 72af40873cb6 2024-04-22T15:52:58.894 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi027 smithi027 running (10m) 114s ago 10m 43.7M 2048M 18.2.1 5be31c24972a 9f21ce7fa7ef 2024-04-22T15:52:58.894 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi016 smithi016 *:9100 running (2m) 42s ago 12m 15.6M - 1.5.0 0da6a335fe13 35e46e4d2f3f 2024-04-22T15:52:58.894 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi027 smithi027 *:9100 running (2m) 114s ago 10m 3598k - 1.5.0 0da6a335fe13 e682a7672e9b 2024-04-22T15:52:58.894 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (10m) 42s ago 10m 200M 1217M 18.2.1 5be31c24972a 3c6ebc4ea041 2024-04-22T15:52:58.894 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (9m) 42s ago 9m 203M 1217M 18.2.1 5be31c24972a 6aa2e6796784 2024-04-22T15:52:58.894 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (9m) 42s ago 9m 174M 1217M 18.2.1 5be31c24972a 3d9d310aa9eb 2024-04-22T15:52:58.894 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi027 running (8m) 114s ago 8m 227M 2240M 18.2.1 5be31c24972a cdf9b7aaf865 2024-04-22T15:52:58.894 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (8m) 114s ago 8m 184M 2240M 18.2.1 5be31c24972a 629397d49c76 2024-04-22T15:52:58.894 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi027 running (7m) 114s ago 7m 173M 2240M 18.2.1 5be31c24972a 23adcf675b04 2024-04-22T15:52:58.894 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.smithi016 smithi016 *:9095 running (95s) 42s ago 11m 51.9M - 2.43.0 a07b618ecd1d 373eaadee82d 2024-04-22T15:52:58.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.892+0000 7effe0a33700 1 -- 172.21.15.16:0/3782203750 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7effc406e940 msgr2=0x7effc4070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:58.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.892+0000 7effe0a33700 1 --2- 172.21.15.16:0/3782203750 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7effc406e940 0x7effc4070e00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7effcc009a10 tx=0x7effcc009190 comp rx=0 tx=0).stop 2024-04-22T15:52:58.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.892+0000 7effe0a33700 1 -- 172.21.15.16:0/3782203750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effdc102720 msgr2=0x7effdc19cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:52:58.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.892+0000 7effe0a33700 1 --2- 172.21.15.16:0/3782203750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effdc102720 0x7effdc19cd70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7effd000bd70 tx=0x7effd000bda0 comp rx=0 tx=0).stop 2024-04-22T15:52:58.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.892+0000 7effe0a33700 1 -- 172.21.15.16:0/3782203750 shutdown_connections 2024-04-22T15:52:58.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.892+0000 7effe0a33700 1 --2- 172.21.15.16:0/3782203750 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7effc406e940 0x7effc4070e00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:58.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.892+0000 7effe0a33700 1 --2- 172.21.15.16:0/3782203750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc101e00 0x7effdc19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:58.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.892+0000 7effe0a33700 1 --2- 172.21.15.16:0/3782203750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effdc102720 0x7effdc19cd70 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:52:58.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.892+0000 7effe0a33700 1 -- 172.21.15.16:0/3782203750 >> 172.21.15.16:0/3782203750 conn(0x7effdc0fb650 msgr2=0x7effdc105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:52:58.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.892+0000 7effe0a33700 1 -- 172.21.15.16:0/3782203750 shutdown_connections 2024-04-22T15:52:58.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:52:58.892+0000 7effe0a33700 1 -- 172.21.15.16:0/3782203750 wait complete. 2024-04-22T15:52:59.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:58 smithi016 ceph-mon[26620]: pgmap v93: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:59.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:58 smithi027 ceph-mon[34922]: pgmap v93: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:52:59.502 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-22T15:52:59.503 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-22T15:52:59.503 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs true' 2024-04-22T15:52:59.722 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:53:00.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:52:59 smithi016 ceph-mon[26620]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:00.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:52:59 smithi027 ceph-mon[34922]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:01.102 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:00 smithi016 ceph-mon[26620]: pgmap v94: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:53:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.331+0000 7f031664a700 1 -- 172.21.15.16:0/3347569258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03100739d0 msgr2=0x7f031010cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.331+0000 7f031664a700 1 --2- 172.21.15.16:0/3347569258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03100739d0 0x7f031010cfc0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0304009910 tx=0x7f0304009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.331+0000 7f031664a700 1 -- 172.21.15.16:0/3347569258 shutdown_connections 2024-04-22T15:53:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.331+0000 7f031664a700 1 --2- 172.21.15.16:0/3347569258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03100739d0 0x7f031010cfc0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.331+0000 7f031664a700 1 --2- 172.21.15.16:0/3347569258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03100730b0 0x7f0310073490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.331+0000 7f031664a700 1 -- 172.21.15.16:0/3347569258 >> 172.21.15.16:0/3347569258 conn(0x7f03100fc750 msgr2=0x7f03100feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.331+0000 7f031664a700 1 -- 172.21.15.16:0/3347569258 shutdown_connections 2024-04-22T15:53:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.331+0000 7f031664a700 1 -- 172.21.15.16:0/3347569258 wait complete. 2024-04-22T15:53:01.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.332+0000 7f031664a700 1 Processor -- start 2024-04-22T15:53:01.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.332+0000 7f031664a700 1 -- start start 2024-04-22T15:53:01.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.332+0000 7f031664a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03100730b0 0x7f031019e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:01.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.332+0000 7f031664a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03100739d0 0x7f031019ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:01.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.332+0000 7f031664a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031019f3f0 con 0x7f03100739d0 2024-04-22T15:53:01.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.332+0000 7f031664a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03101988a0 con 0x7f03100730b0 2024-04-22T15:53:01.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.332+0000 7f030f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03100739d0 0x7f031019ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:01.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.332+0000 7f030f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03100739d0 0x7f031019ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46138/0 (socket says 172.21.15.16:46138) 2024-04-22T15:53:01.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.332+0000 7f030f7fe700 1 -- 172.21.15.16:0/3161184111 learned_addr learned my addr 172.21.15.16:0/3161184111 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:53:01.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.332+0000 7f030ffff700 1 --2- 172.21.15.16:0/3161184111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03100730b0 0x7f031019e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:01.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.332+0000 7f030f7fe700 1 -- 172.21.15.16:0/3161184111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03100730b0 msgr2=0x7f031019e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:01.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.332+0000 7f030f7fe700 1 --2- 172.21.15.16:0/3161184111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03100730b0 0x7f031019e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:01.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.333+0000 7f030f7fe700 1 -- 172.21.15.16:0/3161184111 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03040095f0 con 0x7f03100739d0 2024-04-22T15:53:01.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.333+0000 7f030ffff700 1 --2- 172.21.15.16:0/3161184111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03100730b0 0x7f031019e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:53:01.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.333+0000 7f030f7fe700 1 --2- 172.21.15.16:0/3161184111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03100739d0 0x7f031019ed60 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f030400b5c0 tx=0x7f030400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:01.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.333+0000 7f030d7fa700 1 -- 172.21.15.16:0/3161184111 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f030401d070 con 0x7f03100739d0 2024-04-22T15:53:01.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.333+0000 7f031664a700 1 -- 172.21.15.16:0/3161184111 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0310198b20 con 0x7f03100739d0 2024-04-22T15:53:01.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.333+0000 7f030d7fa700 1 -- 172.21.15.16:0/3161184111 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0304005e00 con 0x7f03100739d0 2024-04-22T15:53:01.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.333+0000 7f030d7fa700 1 -- 172.21.15.16:0/3161184111 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0304017400 con 0x7f03100739d0 2024-04-22T15:53:01.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.333+0000 7f031664a700 1 -- 172.21.15.16:0/3161184111 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0310199010 con 0x7f03100739d0 2024-04-22T15:53:01.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:00 smithi027 ceph-mon[34922]: pgmap v94: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:53:01.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.334+0000 7f031664a700 1 -- 172.21.15.16:0/3161184111 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f031010a6c0 con 0x7f03100739d0 2024-04-22T15:53:01.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.334+0000 7f030d7fa700 1 -- 172.21.15.16:0/3161184111 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f030400f460 con 0x7f03100739d0 2024-04-22T15:53:01.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.335+0000 7f030d7fa700 1 --2- 172.21.15.16:0/3161184111 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f02f806e940 0x7f02f8070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:01.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.335+0000 7f030d7fa700 1 -- 172.21.15.16:0/3161184111 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f030408fdb0 con 0x7f03100739d0 2024-04-22T15:53:01.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.335+0000 7f030ffff700 1 --2- 172.21.15.16:0/3161184111 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f02f806e940 0x7f02f8070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:01.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.336+0000 7f030ffff700 1 --2- 172.21.15.16:0/3161184111 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f02f806e940 0x7f02f8070e00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0300005d90 tx=0x7f0300005d00 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:01.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.340+0000 7f030d7fa700 1 -- 172.21.15.16:0/3161184111 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f030405cbc0 con 0x7f03100739d0 2024-04-22T15:53:01.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.551+0000 7f031664a700 1 -- 172.21.15.16:0/3161184111 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7f0310199e20 con 0x7f03100739d0 2024-04-22T15:53:01.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.555+0000 7f030d7fa700 1 -- 172.21.15.16:0/3161184111 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=0 v46)=0 v46) v1 ==== 125+0+0 (secure 0 0 0) 0x7f0310199e20 con 0x7f03100739d0 2024-04-22T15:53:01.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.558+0000 7f031664a700 1 -- 172.21.15.16:0/3161184111 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f02f806e940 msgr2=0x7f02f8070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:01.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.558+0000 7f031664a700 1 --2- 172.21.15.16:0/3161184111 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f02f806e940 0x7f02f8070e00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0300005d90 tx=0x7f0300005d00 comp rx=0 tx=0).stop 2024-04-22T15:53:01.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.558+0000 7f031664a700 1 -- 172.21.15.16:0/3161184111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03100739d0 msgr2=0x7f031019ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:01.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.558+0000 7f031664a700 1 --2- 172.21.15.16:0/3161184111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03100739d0 0x7f031019ed60 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f030400b5c0 tx=0x7f030400be20 comp rx=0 tx=0).stop 2024-04-22T15:53:01.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.558+0000 7f031664a700 1 -- 172.21.15.16:0/3161184111 shutdown_connections 2024-04-22T15:53:01.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.558+0000 7f031664a700 1 --2- 172.21.15.16:0/3161184111 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f02f806e940 0x7f02f8070e00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:01.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.558+0000 7f031664a700 1 --2- 172.21.15.16:0/3161184111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03100730b0 0x7f031019e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:01.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.558+0000 7f031664a700 1 --2- 172.21.15.16:0/3161184111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03100739d0 0x7f031019ed60 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:01.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.558+0000 7f031664a700 1 -- 172.21.15.16:0/3161184111 >> 172.21.15.16:0/3161184111 conn(0x7f03100fc750 msgr2=0x7f0310107800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:01.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.558+0000 7f031664a700 1 -- 172.21.15.16:0/3161184111 shutdown_connections 2024-04-22T15:53:01.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:01.558+0000 7f031664a700 1 -- 172.21.15.16:0/3161184111 wait complete. 2024-04-22T15:53:02.142 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-22T15:53:02.142 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-22T15:53:02.143 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-22T15:53:02.357 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:53:02.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:02 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/3161184111' entity='client.admin' 2024-04-22T15:53:02.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:02 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:53:02.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:02 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:53:02.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:02 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:53:02.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:02 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:02.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:02 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/3161184111' entity='client.admin' 2024-04-22T15:53:02.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:02 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:53:02.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:02 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:53:02.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:02 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:53:02.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:02 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:03.778 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:03 smithi016 ceph-mon[26620]: pgmap v95: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:53:03.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:03 smithi027 ceph-mon[34922]: pgmap v95: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:53:04.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.145+0000 7f0042d5f700 1 -- 172.21.15.16:0/3751600834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f003c105bd0 msgr2=0x7f003c105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:04.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.145+0000 7f0042d5f700 1 --2- 172.21.15.16:0/3751600834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f003c105bd0 0x7f003c105fb0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f002c009910 tx=0x7f002c009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:04.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.145+0000 7f0042d5f700 1 -- 172.21.15.16:0/3751600834 shutdown_connections 2024-04-22T15:53:04.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.145+0000 7f0042d5f700 1 --2- 172.21.15.16:0/3751600834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f003c0ffc20 0x7f003c100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:04.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.145+0000 7f0042d5f700 1 --2- 172.21.15.16:0/3751600834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f003c105bd0 0x7f003c105fb0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:04.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.145+0000 7f0042d5f700 1 -- 172.21.15.16:0/3751600834 >> 172.21.15.16:0/3751600834 conn(0x7f003c075670 msgr2=0x7f003c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:04.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.145+0000 7f0042d5f700 1 -- 172.21.15.16:0/3751600834 shutdown_connections 2024-04-22T15:53:04.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.145+0000 7f0042d5f700 1 -- 172.21.15.16:0/3751600834 wait complete. 2024-04-22T15:53:04.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.145+0000 7f0042d5f700 1 Processor -- start 2024-04-22T15:53:04.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.146+0000 7f0042d5f700 1 -- start start 2024-04-22T15:53:04.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.146+0000 7f0042d5f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f003c0ffc20 0x7f003c1976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:04.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.146+0000 7f0042d5f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f003c105bd0 0x7f003c197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:04.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.146+0000 7f0042d5f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003c1982f0 con 0x7f003c0ffc20 2024-04-22T15:53:04.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.146+0000 7f0042d5f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003c198460 con 0x7f003c105bd0 2024-04-22T15:53:04.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.146+0000 7f0040afb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f003c0ffc20 0x7f003c1976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:04.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.146+0000 7f0040afb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f003c0ffc20 0x7f003c1976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46416/0 (socket says 172.21.15.16:46416) 2024-04-22T15:53:04.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.146+0000 7f0040afb700 1 -- 172.21.15.16:0/1310108530 learned_addr learned my addr 172.21.15.16:0/1310108530 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:53:04.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.146+0000 7f003bfff700 1 --2- 172.21.15.16:0/1310108530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f003c105bd0 0x7f003c197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:04.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.146+0000 7f0040afb700 1 -- 172.21.15.16:0/1310108530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f003c105bd0 msgr2=0x7f003c197c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:04.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.146+0000 7f0040afb700 1 --2- 172.21.15.16:0/1310108530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f003c105bd0 0x7f003c197c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:04.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.146+0000 7f0040afb700 1 -- 172.21.15.16:0/1310108530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002c0095f0 con 0x7f003c0ffc20 2024-04-22T15:53:04.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.146+0000 7f0040afb700 1 --2- 172.21.15.16:0/1310108530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f003c0ffc20 0x7f003c1976d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f002c006010 tx=0x7f002c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:04.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.147+0000 7f0039ffb700 1 -- 172.21.15.16:0/1310108530 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f002c01d070 con 0x7f003c0ffc20 2024-04-22T15:53:04.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.147+0000 7f0042d5f700 1 -- 172.21.15.16:0/1310108530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f003c1a0740 con 0x7f003c0ffc20 2024-04-22T15:53:04.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.147+0000 7f0039ffb700 1 -- 172.21.15.16:0/1310108530 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f002c00f460 con 0x7f003c0ffc20 2024-04-22T15:53:04.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.147+0000 7f0039ffb700 1 -- 172.21.15.16:0/1310108530 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f002c017610 con 0x7f003c0ffc20 2024-04-22T15:53:04.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.147+0000 7f0042d5f700 1 -- 172.21.15.16:0/1310108530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f003c1a0bb0 con 0x7f003c0ffc20 2024-04-22T15:53:04.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.148+0000 7f0042d5f700 1 -- 172.21.15.16:0/1310108530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f003c1080f0 con 0x7f003c0ffc20 2024-04-22T15:53:04.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.149+0000 7f0039ffb700 1 -- 172.21.15.16:0/1310108530 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f002c00f5d0 con 0x7f003c0ffc20 2024-04-22T15:53:04.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.149+0000 7f0039ffb700 1 --2- 172.21.15.16:0/1310108530 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f002806e940 0x7f0028070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:04.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.149+0000 7f0039ffb700 1 -- 172.21.15.16:0/1310108530 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f002c08fe10 con 0x7f003c0ffc20 2024-04-22T15:53:04.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.149+0000 7f003bfff700 1 --2- 172.21.15.16:0/1310108530 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f002806e940 0x7f0028070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:04.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.150+0000 7f003bfff700 1 --2- 172.21.15.16:0/1310108530 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f002806e940 0x7f0028070e00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f003c068e30 tx=0x7f002400a300 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:04.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.154+0000 7f0039ffb700 1 -- 172.21.15.16:0/1310108530 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f002c0590f0 con 0x7f003c0ffc20 2024-04-22T15:53:04.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.362+0000 7f0042d5f700 1 -- 172.21.15.16:0/1310108530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f003c068c10 con 0x7f003c0ffc20 2024-04-22T15:53:04.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.362+0000 7f0039ffb700 1 -- 172.21.15.16:0/1310108530 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v46)=0 v46) v1 ==== 155+0+0 (secure 0 0 0) 0x7f002c05c7b0 con 0x7f003c0ffc20 2024-04-22T15:53:04.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.364+0000 7f0042d5f700 1 -- 172.21.15.16:0/1310108530 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f002806e940 msgr2=0x7f0028070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:04.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.364+0000 7f0042d5f700 1 --2- 172.21.15.16:0/1310108530 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f002806e940 0x7f0028070e00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f003c068e30 tx=0x7f002400a300 comp rx=0 tx=0).stop 2024-04-22T15:53:04.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.364+0000 7f0042d5f700 1 -- 172.21.15.16:0/1310108530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f003c0ffc20 msgr2=0x7f003c1976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:04.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.364+0000 7f0042d5f700 1 --2- 172.21.15.16:0/1310108530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f003c0ffc20 0x7f003c1976d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f002c006010 tx=0x7f002c00bd70 comp rx=0 tx=0).stop 2024-04-22T15:53:04.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.364+0000 7f0042d5f700 1 -- 172.21.15.16:0/1310108530 shutdown_connections 2024-04-22T15:53:04.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.364+0000 7f0042d5f700 1 --2- 172.21.15.16:0/1310108530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f003c0ffc20 0x7f003c1976d0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:04.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.364+0000 7f0042d5f700 1 --2- 172.21.15.16:0/1310108530 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f002806e940 0x7f0028070e00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:04.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.364+0000 7f0042d5f700 1 --2- 172.21.15.16:0/1310108530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f003c105bd0 0x7f003c197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:04.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.364+0000 7f0042d5f700 1 -- 172.21.15.16:0/1310108530 >> 172.21.15.16:0/1310108530 conn(0x7f003c075670 msgr2=0x7f003c0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:04.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.364+0000 7f0042d5f700 1 -- 172.21.15.16:0/1310108530 shutdown_connections 2024-04-22T15:53:04.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:04.364+0000 7f0042d5f700 1 -- 172.21.15.16:0/1310108530 wait complete. 2024-04-22T15:53:04.910 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-22T15:53:05.123 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:53:05.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:05 smithi027 ceph-mon[34922]: pgmap v96: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:53:05.884 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:05 smithi016 ceph-mon[26620]: pgmap v96: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:53:06.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.679+0000 7ff8469be700 1 -- 172.21.15.16:0/1432912814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8400fffb0 msgr2=0x7ff8401058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:06.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.679+0000 7ff8469be700 1 --2- 172.21.15.16:0/1432912814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8400fffb0 0x7ff8401058e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff834009910 tx=0x7ff834009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:06.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.679+0000 7ff8469be700 1 -- 172.21.15.16:0/1432912814 shutdown_connections 2024-04-22T15:53:06.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.679+0000 7ff8469be700 1 --2- 172.21.15.16:0/1432912814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8400fffb0 0x7ff8401058e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:06.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.679+0000 7ff8469be700 1 --2- 172.21.15.16:0/1432912814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8400ff600 0x7ff8400ff9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:06.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.679+0000 7ff8469be700 1 -- 172.21.15.16:0/1432912814 >> 172.21.15.16:0/1432912814 conn(0x7ff840074df0 msgr2=0x7ff840075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:06.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.679+0000 7ff8469be700 1 -- 172.21.15.16:0/1432912814 shutdown_connections 2024-04-22T15:53:06.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.679+0000 7ff8469be700 1 -- 172.21.15.16:0/1432912814 wait complete. 2024-04-22T15:53:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.680+0000 7ff8469be700 1 Processor -- start 2024-04-22T15:53:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.680+0000 7ff8469be700 1 -- start start 2024-04-22T15:53:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.680+0000 7ff8469be700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8400ff600 0x7ff84019e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.680+0000 7ff8469be700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8400fffb0 0x7ff84019edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.681+0000 7ff8469be700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84019f460 con 0x7ff8400fffb0 2024-04-22T15:53:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.681+0000 7ff8469be700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff840198910 con 0x7ff8400ff600 2024-04-22T15:53:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.681+0000 7ff83f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8400fffb0 0x7ff84019edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.681+0000 7ff83f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8400fffb0 0x7ff84019edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46430/0 (socket says 172.21.15.16:46430) 2024-04-22T15:53:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.681+0000 7ff83f7fe700 1 -- 172.21.15.16:0/404163249 learned_addr learned my addr 172.21.15.16:0/404163249 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:53:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.681+0000 7ff83ffff700 1 --2- 172.21.15.16:0/404163249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8400ff600 0x7ff84019e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.681+0000 7ff83f7fe700 1 -- 172.21.15.16:0/404163249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8400ff600 msgr2=0x7ff84019e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.681+0000 7ff83f7fe700 1 --2- 172.21.15.16:0/404163249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8400ff600 0x7ff84019e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.681+0000 7ff83f7fe700 1 -- 172.21.15.16:0/404163249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8340095f0 con 0x7ff8400fffb0 2024-04-22T15:53:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.681+0000 7ff83f7fe700 1 --2- 172.21.15.16:0/404163249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8400fffb0 0x7ff84019edd0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff83400bda0 tx=0x7ff83400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:06.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.681+0000 7ff83d7fa700 1 -- 172.21.15.16:0/404163249 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff83401d070 con 0x7ff8400fffb0 2024-04-22T15:53:06.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.681+0000 7ff83d7fa700 1 -- 172.21.15.16:0/404163249 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff83400f460 con 0x7ff8400fffb0 2024-04-22T15:53:06.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.681+0000 7ff83d7fa700 1 -- 172.21.15.16:0/404163249 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff834017630 con 0x7ff8400fffb0 2024-04-22T15:53:06.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.681+0000 7ff8469be700 1 -- 172.21.15.16:0/404163249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff840198b90 con 0x7ff8400fffb0 2024-04-22T15:53:06.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.682+0000 7ff8469be700 1 -- 172.21.15.16:0/404163249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff840199080 con 0x7ff8400fffb0 2024-04-22T15:53:06.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.682+0000 7ff8469be700 1 -- 172.21.15.16:0/404163249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff840105060 con 0x7ff8400fffb0 2024-04-22T15:53:06.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.683+0000 7ff83d7fa700 1 -- 172.21.15.16:0/404163249 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7ff83400fa40 con 0x7ff8400fffb0 2024-04-22T15:53:06.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.684+0000 7ff83d7fa700 1 --2- 172.21.15.16:0/404163249 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7ff82806e940 0x7ff828070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:06.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.684+0000 7ff83d7fa700 1 -- 172.21.15.16:0/404163249 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7ff83408fce0 con 0x7ff8400fffb0 2024-04-22T15:53:06.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.684+0000 7ff83ffff700 1 --2- 172.21.15.16:0/404163249 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7ff82806e940 0x7ff828070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:06.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.684+0000 7ff83ffff700 1 --2- 172.21.15.16:0/404163249 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7ff82806e940 0x7ff828070e00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff830005950 tx=0x7ff8300058e0 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:06.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.689+0000 7ff83d7fa700 1 -- 172.21.15.16:0/404163249 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7ff83405caf0 con 0x7ff8400fffb0 2024-04-22T15:53:06.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.895+0000 7ff8469be700 1 -- 172.21.15.16:0/404163249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7ff84004f2e0 con 0x7ff8400fffb0 2024-04-22T15:53:06.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.895+0000 7ff83d7fa700 1 -- 172.21.15.16:0/404163249 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v46)=0 v46) v1 ==== 163+0+0 (secure 0 0 0) 0x7ff83405c680 con 0x7ff8400fffb0 2024-04-22T15:53:06.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.898+0000 7ff8469be700 1 -- 172.21.15.16:0/404163249 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7ff82806e940 msgr2=0x7ff828070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:06.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.898+0000 7ff8469be700 1 --2- 172.21.15.16:0/404163249 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7ff82806e940 0x7ff828070e00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff830005950 tx=0x7ff8300058e0 comp rx=0 tx=0).stop 2024-04-22T15:53:06.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.898+0000 7ff8469be700 1 -- 172.21.15.16:0/404163249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8400fffb0 msgr2=0x7ff84019edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:06.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.898+0000 7ff8469be700 1 --2- 172.21.15.16:0/404163249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8400fffb0 0x7ff84019edd0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff83400bda0 tx=0x7ff83400be80 comp rx=0 tx=0).stop 2024-04-22T15:53:06.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.898+0000 7ff8469be700 1 -- 172.21.15.16:0/404163249 shutdown_connections 2024-04-22T15:53:06.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.898+0000 7ff8469be700 1 --2- 172.21.15.16:0/404163249 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7ff82806e940 0x7ff828070e00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:06.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.898+0000 7ff8469be700 1 --2- 172.21.15.16:0/404163249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8400ff600 0x7ff84019e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:06.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.898+0000 7ff8469be700 1 --2- 172.21.15.16:0/404163249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8400fffb0 0x7ff84019edd0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:06.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.898+0000 7ff8469be700 1 -- 172.21.15.16:0/404163249 >> 172.21.15.16:0/404163249 conn(0x7ff840074df0 msgr2=0x7ff8400fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:06.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.898+0000 7ff8469be700 1 -- 172.21.15.16:0/404163249 shutdown_connections 2024-04-22T15:53:06.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:06.898+0000 7ff8469be700 1 -- 172.21.15.16:0/404163249 wait complete. 2024-04-22T15:53:07.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:06 smithi027 ceph-mon[34922]: pgmap v97: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:53:07.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:06 smithi016 ceph-mon[26620]: pgmap v97: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:53:07.448 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-22T15:53:07.662 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:53:09.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:08 smithi016 ceph-mon[26620]: pgmap v98: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:53:09.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:08 smithi027 ceph-mon[34922]: pgmap v98: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:53:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.344+0000 7f041a244700 1 -- 172.21.15.16:0/719985131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04140ff410 msgr2=0x7f04140ff890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.344+0000 7f041a244700 1 --2- 172.21.15.16:0/719985131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04140ff410 0x7f04140ff890 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f0404009910 tx=0x7f0404009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.344+0000 7f041a244700 1 -- 172.21.15.16:0/719985131 shutdown_connections 2024-04-22T15:53:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.344+0000 7f041a244700 1 --2- 172.21.15.16:0/719985131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04140ff410 0x7f04140ff890 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.344+0000 7f041a244700 1 --2- 172.21.15.16:0/719985131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0414106db0 0x7f0414107190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.344+0000 7f041a244700 1 -- 172.21.15.16:0/719985131 >> 172.21.15.16:0/719985131 conn(0x7f0414074e30 msgr2=0x7f0414075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.344+0000 7f041a244700 1 -- 172.21.15.16:0/719985131 shutdown_connections 2024-04-22T15:53:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.344+0000 7f041a244700 1 -- 172.21.15.16:0/719985131 wait complete. 2024-04-22T15:53:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.344+0000 7f041a244700 1 Processor -- start 2024-04-22T15:53:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.345+0000 7f041a244700 1 -- start start 2024-04-22T15:53:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.345+0000 7f041a244700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04140ff410 0x7f04141005f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.345+0000 7f041a244700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0414106db0 0x7f0414100b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.345+0000 7f041a244700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0414101070 con 0x7f04140ff410 2024-04-22T15:53:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.345+0000 7f041a244700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04141011e0 con 0x7f0414106db0 2024-04-22T15:53:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.345+0000 7f04137fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04140ff410 0x7f04141005f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.345+0000 7f04137fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04140ff410 0x7f04141005f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46448/0 (socket says 172.21.15.16:46448) 2024-04-22T15:53:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.345+0000 7f04137fe700 1 -- 172.21.15.16:0/2853651413 learned_addr learned my addr 172.21.15.16:0/2853651413 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:53:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.345+0000 7f0412ffd700 1 --2- 172.21.15.16:0/2853651413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0414106db0 0x7f0414100b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.345+0000 7f04137fe700 1 -- 172.21.15.16:0/2853651413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0414106db0 msgr2=0x7f0414100b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.345+0000 7f04137fe700 1 --2- 172.21.15.16:0/2853651413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0414106db0 0x7f0414100b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.345+0000 7f04137fe700 1 -- 172.21.15.16:0/2853651413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04040095f0 con 0x7f04140ff410 2024-04-22T15:53:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.346+0000 7f04137fe700 1 --2- 172.21.15.16:0/2853651413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04140ff410 0x7f04141005f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f040800c930 tx=0x7f040800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.346+0000 7f0410ff9700 1 -- 172.21.15.16:0/2853651413 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0408007ad0 con 0x7f04140ff410 2024-04-22T15:53:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.346+0000 7f0410ff9700 1 -- 172.21.15.16:0/2853651413 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f040800ce80 con 0x7f04140ff410 2024-04-22T15:53:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.346+0000 7f041a244700 1 -- 172.21.15.16:0/2853651413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0414101460 con 0x7f04140ff410 2024-04-22T15:53:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.346+0000 7f0410ff9700 1 -- 172.21.15.16:0/2853651413 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0408018630 con 0x7f04140ff410 2024-04-22T15:53:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.346+0000 7f041a244700 1 -- 172.21.15.16:0/2853651413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04141a2ed0 con 0x7f04140ff410 2024-04-22T15:53:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.347+0000 7f041a244700 1 -- 172.21.15.16:0/2853651413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04141092b0 con 0x7f04140ff410 2024-04-22T15:53:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.347+0000 7f0410ff9700 1 -- 172.21.15.16:0/2853651413 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f0408007c30 con 0x7f04140ff410 2024-04-22T15:53:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.348+0000 7f0410ff9700 1 --2- 172.21.15.16:0/2853651413 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f03fc06e940 0x7f03fc070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.348+0000 7f0410ff9700 1 -- 172.21.15.16:0/2853651413 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f040808e630 con 0x7f04140ff410 2024-04-22T15:53:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.348+0000 7f0412ffd700 1 --2- 172.21.15.16:0/2853651413 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f03fc06e940 0x7f03fc070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.349+0000 7f0412ffd700 1 --2- 172.21.15.16:0/2853651413 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f03fc06e940 0x7f03fc070e00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0414101de0 tx=0x7f040401a040 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:09.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.354+0000 7f0410ff9700 1 -- 172.21.15.16:0/2853651413 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f0408057990 con 0x7f04140ff410 2024-04-22T15:53:09.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.560+0000 7f041a244700 1 -- 172.21.15.16:0/2853651413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f0414101c30 con 0x7f04140ff410 2024-04-22T15:53:09.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.561+0000 7f0410ff9700 1 -- 172.21.15.16:0/2853651413 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v46)=0 v46) v1 ==== 135+0+0 (secure 0 0 0) 0x7f040805b050 con 0x7f04140ff410 2024-04-22T15:53:09.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.563+0000 7f041a244700 1 -- 172.21.15.16:0/2853651413 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f03fc06e940 msgr2=0x7f03fc070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:09.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.563+0000 7f041a244700 1 --2- 172.21.15.16:0/2853651413 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f03fc06e940 0x7f03fc070e00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0414101de0 tx=0x7f040401a040 comp rx=0 tx=0).stop 2024-04-22T15:53:09.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.563+0000 7f041a244700 1 -- 172.21.15.16:0/2853651413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04140ff410 msgr2=0x7f04141005f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:09.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.563+0000 7f041a244700 1 --2- 172.21.15.16:0/2853651413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04140ff410 0x7f04141005f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f040800c930 tx=0x7f040800cc40 comp rx=0 tx=0).stop 2024-04-22T15:53:09.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.563+0000 7f041a244700 1 -- 172.21.15.16:0/2853651413 shutdown_connections 2024-04-22T15:53:09.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.563+0000 7f041a244700 1 --2- 172.21.15.16:0/2853651413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04140ff410 0x7f04141005f0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:09.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.563+0000 7f041a244700 1 --2- 172.21.15.16:0/2853651413 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f03fc06e940 0x7f03fc070e00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:09.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.563+0000 7f041a244700 1 --2- 172.21.15.16:0/2853651413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0414106db0 0x7f0414100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:09.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.563+0000 7f041a244700 1 -- 172.21.15.16:0/2853651413 >> 172.21.15.16:0/2853651413 conn(0x7f0414074e30 msgr2=0x7f04140fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:09.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.564+0000 7f041a244700 1 -- 172.21.15.16:0/2853651413 shutdown_connections 2024-04-22T15:53:09.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:09.564+0000 7f041a244700 1 -- 172.21.15.16:0/2853651413 wait complete. 2024-04-22T15:53:10.169 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-22T15:53:10.391 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:53:11.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:10 smithi016 ceph-mon[26620]: pgmap v99: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:53:11.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:10 smithi027 ceph-mon[34922]: pgmap v99: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:53:12.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.991+0000 7fcc067ab700 1 -- 172.21.15.16:0/3229447625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc00101e60 msgr2=0x7fcc00102240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:12.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.991+0000 7fcc067ab700 1 --2- 172.21.15.16:0/3229447625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc00101e60 0x7fcc00102240 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0009910 tx=0x7fcbf0009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:12.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.991+0000 7fcc067ab700 1 -- 172.21.15.16:0/3229447625 shutdown_connections 2024-04-22T15:53:12.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.991+0000 7fcc067ab700 1 --2- 172.21.15.16:0/3229447625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc00102780 0x7fcc0010ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:12.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.991+0000 7fcc067ab700 1 --2- 172.21.15.16:0/3229447625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc00101e60 0x7fcc00102240 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:12.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.991+0000 7fcc067ab700 1 -- 172.21.15.16:0/3229447625 >> 172.21.15.16:0/3229447625 conn(0x7fcc000fb650 msgr2=0x7fcc000fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:12.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.991+0000 7fcc067ab700 1 -- 172.21.15.16:0/3229447625 shutdown_connections 2024-04-22T15:53:12.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.991+0000 7fcc067ab700 1 -- 172.21.15.16:0/3229447625 wait complete. 2024-04-22T15:53:12.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.991+0000 7fcc067ab700 1 Processor -- start 2024-04-22T15:53:12.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.991+0000 7fcc067ab700 1 -- start start 2024-04-22T15:53:12.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.992+0000 7fcc067ab700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc00101e60 0x7fcc0019c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:12.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.992+0000 7fcc067ab700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc00102780 0x7fcc0019cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:12.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.992+0000 7fcc067ab700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0019d220 con 0x7fcc00101e60 2024-04-22T15:53:12.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.992+0000 7fcc067ab700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc00196720 con 0x7fcc00102780 2024-04-22T15:53:12.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.992+0000 7fcbfffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc00101e60 0x7fcc0019c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:12.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.992+0000 7fcbfffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc00101e60 0x7fcc0019c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39242/0 (socket says 172.21.15.16:39242) 2024-04-22T15:53:12.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.992+0000 7fcbfffff700 1 -- 172.21.15.16:0/2432768218 learned_addr learned my addr 172.21.15.16:0/2432768218 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:53:12.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.992+0000 7fcbff7fe700 1 --2- 172.21.15.16:0/2432768218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc00102780 0x7fcc0019cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:12.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.992+0000 7fcbfffff700 1 -- 172.21.15.16:0/2432768218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc00102780 msgr2=0x7fcc0019cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:12.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.992+0000 7fcbfffff700 1 --2- 172.21.15.16:0/2432768218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc00102780 0x7fcc0019cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:12.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.992+0000 7fcbfffff700 1 -- 172.21.15.16:0/2432768218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf00095f0 con 0x7fcc00101e60 2024-04-22T15:53:12.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.992+0000 7fcbfffff700 1 --2- 172.21.15.16:0/2432768218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc00101e60 0x7fcc0019c650 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0006010 tx=0x7fcbf000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:12.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.993+0000 7fcbfd7fa700 1 -- 172.21.15.16:0/2432768218 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbf001d070 con 0x7fcc00101e60 2024-04-22T15:53:12.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.993+0000 7fcc067ab700 1 -- 172.21.15.16:0/2432768218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc001969a0 con 0x7fcc00101e60 2024-04-22T15:53:12.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.993+0000 7fcbfd7fa700 1 -- 172.21.15.16:0/2432768218 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcbf0005dc0 con 0x7fcc00101e60 2024-04-22T15:53:12.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.993+0000 7fcbfd7fa700 1 -- 172.21.15.16:0/2432768218 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbf0017400 con 0x7fcc00101e60 2024-04-22T15:53:12.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.993+0000 7fcc067ab700 1 -- 172.21.15.16:0/2432768218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc00196e90 con 0x7fcc00101e60 2024-04-22T15:53:12.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.994+0000 7fcc067ab700 1 -- 172.21.15.16:0/2432768218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc00108310 con 0x7fcc00101e60 2024-04-22T15:53:12.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.994+0000 7fcbfd7fa700 1 -- 172.21.15.16:0/2432768218 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7fcbf0003a60 con 0x7fcc00101e60 2024-04-22T15:53:12.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.995+0000 7fcbfd7fa700 1 --2- 172.21.15.16:0/2432768218 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fcbe806e940 0x7fcbe8070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:12.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.995+0000 7fcbff7fe700 1 --2- 172.21.15.16:0/2432768218 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fcbe806e940 0x7fcbe8070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:12.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.995+0000 7fcbfd7fa700 1 -- 172.21.15.16:0/2432768218 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fcbf008fd30 con 0x7fcc00101e60 2024-04-22T15:53:12.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:11.995+0000 7fcbff7fe700 1 --2- 172.21.15.16:0/2432768218 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fcbe806e940 0x7fcbe8070e00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcc00197b40 tx=0x7fcbf400b410 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:12.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:12.000+0000 7fcbfd7fa700 1 -- 172.21.15.16:0/2432768218 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7fcbf005cb40 con 0x7fcc00101e60 2024-04-22T15:53:12.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:12.229+0000 7fcc067ab700 1 -- 172.21.15.16:0/2432768218 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc001970b0 con 0x7fcbe806e940 2024-04-22T15:53:12.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:12.235+0000 7fcbfd7fa700 1 -- 172.21.15.16:0/2432768218 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7fcc001970b0 con 0x7fcbe806e940 2024-04-22T15:53:12.238 INFO:teuthology.orchestra.run.smithi016.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:53:12.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:12.237+0000 7fcc067ab700 1 -- 172.21.15.16:0/2432768218 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fcbe806e940 msgr2=0x7fcbe8070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:12.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:12.237+0000 7fcc067ab700 1 --2- 172.21.15.16:0/2432768218 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fcbe806e940 0x7fcbe8070e00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcc00197b40 tx=0x7fcbf400b410 comp rx=0 tx=0).stop 2024-04-22T15:53:12.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:12.237+0000 7fcc067ab700 1 -- 172.21.15.16:0/2432768218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc00101e60 msgr2=0x7fcc0019c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:12.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:12.237+0000 7fcc067ab700 1 --2- 172.21.15.16:0/2432768218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc00101e60 0x7fcc0019c650 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0006010 tx=0x7fcbf000bd70 comp rx=0 tx=0).stop 2024-04-22T15:53:12.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:12.237+0000 7fcc067ab700 1 -- 172.21.15.16:0/2432768218 shutdown_connections 2024-04-22T15:53:12.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:12.237+0000 7fcc067ab700 1 --2- 172.21.15.16:0/2432768218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc00101e60 0x7fcc0019c650 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:12.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:12.237+0000 7fcc067ab700 1 --2- 172.21.15.16:0/2432768218 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fcbe806e940 0x7fcbe8070e00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:12.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:12.237+0000 7fcc067ab700 1 --2- 172.21.15.16:0/2432768218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc00102780 0x7fcc0019cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:12.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:12.237+0000 7fcc067ab700 1 -- 172.21.15.16:0/2432768218 >> 172.21.15.16:0/2432768218 conn(0x7fcc000fb650 msgr2=0x7fcc00105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:12.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:12.237+0000 7fcc067ab700 1 -- 172.21.15.16:0/2432768218 shutdown_connections 2024-04-22T15:53:12.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:12.237+0000 7fcc067ab700 1 -- 172.21.15.16:0/2432768218 wait complete. 2024-04-22T15:53:12.900 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-22T15:53:12.900 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-22T15:53:12.901 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-22T15:53:13.124 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:53:13.508 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:13 smithi016 ceph-mon[26620]: pgmap v100: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:53:13.508 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:13 smithi016 ceph-mon[26620]: from='client.14846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:13.508 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:13 smithi016 ceph-mon[26620]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:53:13.508 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:13 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:13.508 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:13 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:53:13.508 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:13 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:53:13.508 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:13 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:53:13.508 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:13 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:53:13.508 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:13 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:13.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:13 smithi027 ceph-mon[34922]: pgmap v100: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:53:13.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:13 smithi027 ceph-mon[34922]: from='client.14846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:13.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:13 smithi027 ceph-mon[34922]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:53:13.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:13 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:13.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:13 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:53:13.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:13 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:53:13.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:13 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:53:13.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:13 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:53:13.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:13 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:14.531 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:14 smithi016 ceph-mon[26620]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:53:14.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:14 smithi027 ceph-mon[34922]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5883d494d3973b30b3f03e4a3f0e1e06e3c131aa 2024-04-22T15:53:14.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.814+0000 7fe9f209f700 1 -- 172.21.15.16:0/319096930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9ec101850 msgr2=0x7fe9ec1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:14.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.814+0000 7fe9f209f700 1 --2- 172.21.15.16:0/319096930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9ec101850 0x7fe9ec1058a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009910 tx=0x7fe9dc009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:14.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.814+0000 7fe9f209f700 1 -- 172.21.15.16:0/319096930 shutdown_connections 2024-04-22T15:53:14.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.814+0000 7fe9f209f700 1 --2- 172.21.15.16:0/319096930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9ec101850 0x7fe9ec1058a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:14.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.814+0000 7fe9f209f700 1 --2- 172.21.15.16:0/319096930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9ec100ea0 0x7fe9ec101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:14.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.814+0000 7fe9f209f700 1 -- 172.21.15.16:0/319096930 >> 172.21.15.16:0/319096930 conn(0x7fe9ec0fc750 msgr2=0x7fe9ec0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:14.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.814+0000 7fe9f209f700 1 -- 172.21.15.16:0/319096930 shutdown_connections 2024-04-22T15:53:14.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.814+0000 7fe9f209f700 1 -- 172.21.15.16:0/319096930 wait complete. 2024-04-22T15:53:14.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.815+0000 7fe9f209f700 1 Processor -- start 2024-04-22T15:53:14.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.815+0000 7fe9f209f700 1 -- start start 2024-04-22T15:53:14.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.816+0000 7fe9f209f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9ec100ea0 0x7fe9ec19e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:14.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.816+0000 7fe9f209f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9ec101850 0x7fe9ec19ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:14.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.816+0000 7fe9f209f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec19f4b0 con 0x7fe9ec100ea0 2024-04-22T15:53:14.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.816+0000 7fe9f209f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec198960 con 0x7fe9ec101850 2024-04-22T15:53:14.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.816+0000 7fe9eb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9ec100ea0 0x7fe9ec19e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:14.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.816+0000 7fe9eb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9ec100ea0 0x7fe9ec19e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39254/0 (socket says 172.21.15.16:39254) 2024-04-22T15:53:14.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.816+0000 7fe9eb7fe700 1 -- 172.21.15.16:0/3350130614 learned_addr learned my addr 172.21.15.16:0/3350130614 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:53:14.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.816+0000 7fe9eaffd700 1 --2- 172.21.15.16:0/3350130614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9ec101850 0x7fe9ec19ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:14.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.816+0000 7fe9eb7fe700 1 -- 172.21.15.16:0/3350130614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9ec101850 msgr2=0x7fe9ec19ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:14.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.816+0000 7fe9eb7fe700 1 --2- 172.21.15.16:0/3350130614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9ec101850 0x7fe9ec19ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:14.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.816+0000 7fe9eb7fe700 1 -- 172.21.15.16:0/3350130614 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9dc0095f0 con 0x7fe9ec100ea0 2024-04-22T15:53:14.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.816+0000 7fe9eb7fe700 1 --2- 172.21.15.16:0/3350130614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9ec100ea0 0x7fe9ec19e8e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe9e000d8d0 tx=0x7fe9e000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:14.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.816+0000 7fe9e8ff9700 1 -- 172.21.15.16:0/3350130614 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9e0009940 con 0x7fe9ec100ea0 2024-04-22T15:53:14.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.816+0000 7fe9f209f700 1 -- 172.21.15.16:0/3350130614 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9ec198c40 con 0x7fe9ec100ea0 2024-04-22T15:53:14.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.816+0000 7fe9e8ff9700 1 -- 172.21.15.16:0/3350130614 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9e0010460 con 0x7fe9ec100ea0 2024-04-22T15:53:14.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.816+0000 7fe9e8ff9700 1 -- 172.21.15.16:0/3350130614 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9e000f5f0 con 0x7fe9ec100ea0 2024-04-22T15:53:14.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.816+0000 7fe9f209f700 1 -- 172.21.15.16:0/3350130614 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9ec199190 con 0x7fe9ec100ea0 2024-04-22T15:53:14.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.817+0000 7fe9f209f700 1 -- 172.21.15.16:0/3350130614 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9ec105020 con 0x7fe9ec100ea0 2024-04-22T15:53:14.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.818+0000 7fe9e8ff9700 1 -- 172.21.15.16:0/3350130614 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7fe9e00105d0 con 0x7fe9ec100ea0 2024-04-22T15:53:14.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.818+0000 7fe9e8ff9700 1 --2- 172.21.15.16:0/3350130614 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fe9d406e940 0x7fe9d4070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:14.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.818+0000 7fe9e8ff9700 1 -- 172.21.15.16:0/3350130614 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fe9e008e8c0 con 0x7fe9ec100ea0 2024-04-22T15:53:14.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.818+0000 7fe9eaffd700 1 --2- 172.21.15.16:0/3350130614 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fe9d406e940 0x7fe9d4070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:14.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.819+0000 7fe9eaffd700 1 --2- 172.21.15.16:0/3350130614 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fe9d406e940 0x7fe9d4070e00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec0fcfc0 tx=0x7fe9dc01a040 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:14.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:14.823+0000 7fe9e8ff9700 1 -- 172.21.15.16:0/3350130614 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7fe9e005b6d0 con 0x7fe9ec100ea0 2024-04-22T15:53:15.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.065+0000 7fe9f209f700 1 -- 172.21.15.16:0/3350130614 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9ec199ec0 con 0x7fe9d406e940 2024-04-22T15:53:15.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.066+0000 7fe9e8ff9700 1 -- 172.21.15.16:0/3350130614 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7fe9ec199ec0 con 0x7fe9d406e940 2024-04-22T15:53:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.069+0000 7fe9f209f700 1 -- 172.21.15.16:0/3350130614 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fe9d406e940 msgr2=0x7fe9d4070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.069+0000 7fe9f209f700 1 --2- 172.21.15.16:0/3350130614 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fe9d406e940 0x7fe9d4070e00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec0fcfc0 tx=0x7fe9dc01a040 comp rx=0 tx=0).stop 2024-04-22T15:53:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.069+0000 7fe9f209f700 1 -- 172.21.15.16:0/3350130614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9ec100ea0 msgr2=0x7fe9ec19e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.069+0000 7fe9f209f700 1 --2- 172.21.15.16:0/3350130614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9ec100ea0 0x7fe9ec19e8e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe9e000d8d0 tx=0x7fe9e000dc90 comp rx=0 tx=0).stop 2024-04-22T15:53:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.069+0000 7fe9f209f700 1 -- 172.21.15.16:0/3350130614 shutdown_connections 2024-04-22T15:53:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.069+0000 7fe9f209f700 1 --2- 172.21.15.16:0/3350130614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9ec100ea0 0x7fe9ec19e8e0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.069+0000 7fe9f209f700 1 --2- 172.21.15.16:0/3350130614 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fe9d406e940 0x7fe9d4070e00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.069+0000 7fe9f209f700 1 --2- 172.21.15.16:0/3350130614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9ec101850 0x7fe9ec19ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.069+0000 7fe9f209f700 1 -- 172.21.15.16:0/3350130614 >> 172.21.15.16:0/3350130614 conn(0x7fe9ec0fc750 msgr2=0x7fe9ec0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.069+0000 7fe9f209f700 1 -- 172.21.15.16:0/3350130614 shutdown_connections 2024-04-22T15:53:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.069+0000 7fe9f209f700 1 -- 172.21.15.16:0/3350130614 wait complete. 2024-04-22T15:53:15.082 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:53:15.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.180+0000 7f5624ff3700 1 -- 172.21.15.16:0/26333944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5620106cd0 msgr2=0x7f56201070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:15.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.180+0000 7f5624ff3700 1 --2- 172.21.15.16:0/26333944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5620106cd0 0x7f56201070b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5610009910 tx=0x7f5610009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:15.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.180+0000 7f5624ff3700 1 -- 172.21.15.16:0/26333944 shutdown_connections 2024-04-22T15:53:15.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.180+0000 7f5624ff3700 1 --2- 172.21.15.16:0/26333944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5620068580 0x7f56200689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.180+0000 7f5624ff3700 1 --2- 172.21.15.16:0/26333944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5620106cd0 0x7f56201070b0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.180+0000 7f5624ff3700 1 -- 172.21.15.16:0/26333944 >> 172.21.15.16:0/26333944 conn(0x7f5620075670 msgr2=0x7f5620075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:15.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.180+0000 7f5624ff3700 1 -- 172.21.15.16:0/26333944 shutdown_connections 2024-04-22T15:53:15.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.180+0000 7f5624ff3700 1 -- 172.21.15.16:0/26333944 wait complete. 2024-04-22T15:53:15.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.181+0000 7f5624ff3700 1 Processor -- start 2024-04-22T15:53:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.181+0000 7f5624ff3700 1 -- start start 2024-04-22T15:53:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.181+0000 7f5624ff3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5620068580 0x7f56200ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.181+0000 7f5624ff3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5620106cd0 0x7f5620100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.181+0000 7f5624ff3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5620100770 con 0x7f5620068580 2024-04-22T15:53:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.181+0000 7f5624ff3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56201008b0 con 0x7f5620106cd0 2024-04-22T15:53:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.181+0000 7f561e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5620068580 0x7f56200ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.181+0000 7f561e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5620068580 0x7f56200ffcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39270/0 (socket says 172.21.15.16:39270) 2024-04-22T15:53:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.181+0000 7f561e59c700 1 -- 172.21.15.16:0/3044953086 learned_addr learned my addr 172.21.15.16:0/3044953086 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:53:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.182+0000 7f561dd9b700 1 --2- 172.21.15.16:0/3044953086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5620106cd0 0x7f5620100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.182+0000 7f561e59c700 1 -- 172.21.15.16:0/3044953086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5620106cd0 msgr2=0x7f5620100230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.182+0000 7f561e59c700 1 --2- 172.21.15.16:0/3044953086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5620106cd0 0x7f5620100230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.182+0000 7f561e59c700 1 -- 172.21.15.16:0/3044953086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56100095f0 con 0x7f5620068580 2024-04-22T15:53:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.182+0000 7f561e59c700 1 --2- 172.21.15.16:0/3044953086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5620068580 0x7f56200ffcf0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5610006010 tx=0x7f561000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:15.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.182+0000 7f56177fe700 1 -- 172.21.15.16:0/3044953086 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f561001d070 con 0x7f5620068580 2024-04-22T15:53:15.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.182+0000 7f56177fe700 1 -- 172.21.15.16:0/3044953086 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f561000f460 con 0x7f5620068580 2024-04-22T15:53:15.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.182+0000 7f5624ff3700 1 -- 172.21.15.16:0/3044953086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5620100b30 con 0x7f5620068580 2024-04-22T15:53:15.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.182+0000 7f56177fe700 1 -- 172.21.15.16:0/3044953086 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5610017630 con 0x7f5620068580 2024-04-22T15:53:15.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.182+0000 7f5624ff3700 1 -- 172.21.15.16:0/3044953086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56201a2d90 con 0x7f5620068580 2024-04-22T15:53:15.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.184+0000 7f5624ff3700 1 -- 172.21.15.16:0/3044953086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56201091f0 con 0x7f5620068580 2024-04-22T15:53:15.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.184+0000 7f56177fe700 1 -- 172.21.15.16:0/3044953086 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f561000f5d0 con 0x7f5620068580 2024-04-22T15:53:15.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.184+0000 7f56177fe700 1 --2- 172.21.15.16:0/3044953086 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f560c06e940 0x7f560c070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:15.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.184+0000 7f56177fe700 1 -- 172.21.15.16:0/3044953086 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f561008fca0 con 0x7f5620068580 2024-04-22T15:53:15.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.184+0000 7f561dd9b700 1 --2- 172.21.15.16:0/3044953086 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f560c06e940 0x7f560c070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:15.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.185+0000 7f561dd9b700 1 --2- 172.21.15.16:0/3044953086 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f560c06e940 0x7f560c070e00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5608005950 tx=0x7f56080058e0 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:15.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.189+0000 7f56177fe700 1 -- 172.21.15.16:0/3044953086 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f5610094070 con 0x7f5620068580 2024-04-22T15:53:15.342 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:15 smithi016 ceph-mon[26620]: pgmap v101: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:53:15.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.447+0000 7f5624ff3700 1 -- 172.21.15.16:0/3044953086 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f56200611d0 con 0x7f560c06e940 2024-04-22T15:53:15.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.448+0000 7f56177fe700 1 -- 172.21.15.16:0/3044953086 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f56200611d0 con 0x7f560c06e940 2024-04-22T15:53:15.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.452+0000 7f5624ff3700 1 -- 172.21.15.16:0/3044953086 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f560c06e940 msgr2=0x7f560c070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:15.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.452+0000 7f5624ff3700 1 --2- 172.21.15.16:0/3044953086 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f560c06e940 0x7f560c070e00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5608005950 tx=0x7f56080058e0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.452+0000 7f5624ff3700 1 -- 172.21.15.16:0/3044953086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5620068580 msgr2=0x7f56200ffcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:15.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.452+0000 7f5624ff3700 1 --2- 172.21.15.16:0/3044953086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5620068580 0x7f56200ffcf0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5610006010 tx=0x7f561000be20 comp rx=0 tx=0).stop 2024-04-22T15:53:15.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.452+0000 7f5624ff3700 1 -- 172.21.15.16:0/3044953086 shutdown_connections 2024-04-22T15:53:15.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.452+0000 7f5624ff3700 1 --2- 172.21.15.16:0/3044953086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5620068580 0x7f56200ffcf0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.452+0000 7f5624ff3700 1 --2- 172.21.15.16:0/3044953086 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f560c06e940 0x7f560c070e00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.452+0000 7f5624ff3700 1 --2- 172.21.15.16:0/3044953086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5620106cd0 0x7f5620100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.452+0000 7f5624ff3700 1 -- 172.21.15.16:0/3044953086 >> 172.21.15.16:0/3044953086 conn(0x7f5620075670 msgr2=0x7f56200fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:15.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.452+0000 7f5624ff3700 1 -- 172.21.15.16:0/3044953086 shutdown_connections 2024-04-22T15:53:15.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.452+0000 7f5624ff3700 1 -- 172.21.15.16:0/3044953086 wait complete. 2024-04-22T15:53:15.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.570+0000 7fd6567f2700 1 -- 172.21.15.16:0/2703152567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd650106cd0 msgr2=0x7fd6501070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.570+0000 7fd6567f2700 1 --2- 172.21.15.16:0/2703152567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd650106cd0 0x7fd6501070b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd640009910 tx=0x7fd640009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.570+0000 7fd6567f2700 1 -- 172.21.15.16:0/2703152567 shutdown_connections 2024-04-22T15:53:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.570+0000 7fd6567f2700 1 --2- 172.21.15.16:0/2703152567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd650068580 0x7fd6500689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.570+0000 7fd6567f2700 1 --2- 172.21.15.16:0/2703152567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd650106cd0 0x7fd6501070b0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.570+0000 7fd6567f2700 1 -- 172.21.15.16:0/2703152567 >> 172.21.15.16:0/2703152567 conn(0x7fd650075670 msgr2=0x7fd650075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.570+0000 7fd6567f2700 1 -- 172.21.15.16:0/2703152567 shutdown_connections 2024-04-22T15:53:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.570+0000 7fd6567f2700 1 -- 172.21.15.16:0/2703152567 wait complete. 2024-04-22T15:53:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.571+0000 7fd6567f2700 1 Processor -- start 2024-04-22T15:53:15.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.571+0000 7fd6567f2700 1 -- start start 2024-04-22T15:53:15.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.571+0000 7fd6567f2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd650068580 0x7fd65019c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:15.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.571+0000 7fd6567f2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd650106cd0 0x7fd65019cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:15.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.571+0000 7fd6567f2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65019d230 con 0x7fd650068580 2024-04-22T15:53:15.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.571+0000 7fd6567f2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6501966e0 con 0x7fd650106cd0 2024-04-22T15:53:15.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.571+0000 7fd64ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd650068580 0x7fd65019c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:15.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.571+0000 7fd64ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd650068580 0x7fd65019c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39288/0 (socket says 172.21.15.16:39288) 2024-04-22T15:53:15.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.571+0000 7fd64ffff700 1 -- 172.21.15.16:0/88323614 learned_addr learned my addr 172.21.15.16:0/88323614 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:53:15.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.571+0000 7fd64f7fe700 1 --2- 172.21.15.16:0/88323614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd650106cd0 0x7fd65019cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:15.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.572+0000 7fd64ffff700 1 -- 172.21.15.16:0/88323614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd650106cd0 msgr2=0x7fd65019cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:15.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.572+0000 7fd64ffff700 1 --2- 172.21.15.16:0/88323614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd650106cd0 0x7fd65019cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.572+0000 7fd64ffff700 1 -- 172.21.15.16:0/88323614 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6400095f0 con 0x7fd650068580 2024-04-22T15:53:15.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.572+0000 7fd64ffff700 1 --2- 172.21.15.16:0/88323614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd650068580 0x7fd65019c660 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd640006010 tx=0x7fd64000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:15.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.572+0000 7fd64d7fa700 1 -- 172.21.15.16:0/88323614 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd64001d070 con 0x7fd650068580 2024-04-22T15:53:15.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.572+0000 7fd64d7fa700 1 -- 172.21.15.16:0/88323614 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd640005dc0 con 0x7fd650068580 2024-04-22T15:53:15.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.572+0000 7fd64d7fa700 1 -- 172.21.15.16:0/88323614 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd640017400 con 0x7fd650068580 2024-04-22T15:53:15.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.573+0000 7fd6567f2700 1 -- 172.21.15.16:0/88323614 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd650196960 con 0x7fd650068580 2024-04-22T15:53:15.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.573+0000 7fd6567f2700 1 -- 172.21.15.16:0/88323614 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd650196dd0 con 0x7fd650068580 2024-04-22T15:53:15.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.574+0000 7fd6567f2700 1 -- 172.21.15.16:0/88323614 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6501091f0 con 0x7fd650068580 2024-04-22T15:53:15.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.574+0000 7fd64d7fa700 1 -- 172.21.15.16:0/88323614 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7fd640003a60 con 0x7fd650068580 2024-04-22T15:53:15.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.575+0000 7fd64d7fa700 1 --2- 172.21.15.16:0/88323614 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fd63806e940 0x7fd638070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:15.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.575+0000 7fd64d7fa700 1 -- 172.21.15.16:0/88323614 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fd64008fd60 con 0x7fd650068580 2024-04-22T15:53:15.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.575+0000 7fd64f7fe700 1 --2- 172.21.15.16:0/88323614 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fd63806e940 0x7fd638070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:15.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.576+0000 7fd64f7fe700 1 --2- 172.21.15.16:0/88323614 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fd63806e940 0x7fd638070e00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd650197b50 tx=0x7fd64400b410 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:15.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.580+0000 7fd64d7fa700 1 -- 172.21.15.16:0/88323614 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7fd64005cb70 con 0x7fd650068580 2024-04-22T15:53:15.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:15 smithi027 ceph-mon[34922]: pgmap v101: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:53:15.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.803+0000 7fd6567f2700 1 -- 172.21.15.16:0/88323614 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd650197d20 con 0x7fd63806e940 2024-04-22T15:53:15.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.815+0000 7fd64d7fa700 1 -- 172.21.15.16:0/88323614 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fd650197d20 con 0x7fd63806e940 2024-04-22T15:53:15.818 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:53:15.818 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.smithi016 smithi016 *:9093,9094 running (91s) 59s ago 12m 17.1M - 0.25.0 c8568f914cd2 ea0e0217d6f9 2024-04-22T15:53:15.818 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi016 smithi016 running (12m) 59s ago 12m 7884k - 18.2.1 5be31c24972a f8b627d3f41e 2024-04-22T15:53:15.818 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi027 smithi027 running (11m) 2m ago 11m 7776k - 18.2.1 5be31c24972a f00fd40b4220 2024-04-22T15:53:15.818 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi016 smithi016 running (12m) 59s ago 12m 6660k - 18.2.1 5be31c24972a 6b93db1fa0d9 2024-04-22T15:53:15.818 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi027 smithi027 running (11m) 2m ago 11m 6665k - 18.2.1 5be31c24972a d3bd439f30a0 2024-04-22T15:53:15.818 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.smithi016 smithi016 *:3000 running (66s) 59s ago 11m 52.7M - 9.4.12 2bacad6d85d8 388a3c9c1b3f 2024-04-22T15:53:15.819 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.fkkejm smithi016 running (6m) 59s ago 6m 14.7M - 18.2.1 5be31c24972a 4153ac94cea0 2024-04-22T15:53:15.819 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.pgvphu smithi016 running (6m) 59s ago 6m 91.7M - 18.2.1 5be31c24972a 93dc072068a2 2024-04-22T15:53:15.819 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.dgqyes smithi027 running (6m) 2m ago 6m 17.3M - 18.2.1 5be31c24972a 41a9220430fd 2024-04-22T15:53:15.819 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.iobjds smithi027 running (6m) 2m ago 6m 241M - 18.2.1 5be31c24972a c1c8d6770f02 2024-04-22T15:53:15.819 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi016.rnsnnr smithi016 *:8443,9283,8765 running (3m) 59s ago 13m 513M - 19.0.0-3223-g5883d494 3d983c26e278 09b4fe285bc4 2024-04-22T15:53:15.819 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi027.ukdgzs smithi027 *:8443,9283,8765 running (2m) 2m ago 11m 450M - 19.0.0-3223-g5883d494 3d983c26e278 7548da9b5eb9 2024-04-22T15:53:15.819 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi016 smithi016 running (13m) 59s ago 13m 60.3M 2048M 18.2.1 5be31c24972a 72af40873cb6 2024-04-22T15:53:15.819 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi027 smithi027 running (11m) 2m ago 11m 43.7M 2048M 18.2.1 5be31c24972a 9f21ce7fa7ef 2024-04-22T15:53:15.819 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi016 smithi016 *:9100 running (2m) 59s ago 12m 15.6M - 1.5.0 0da6a335fe13 35e46e4d2f3f 2024-04-22T15:53:15.819 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi027 smithi027 *:9100 running (2m) 2m ago 11m 3598k - 1.5.0 0da6a335fe13 e682a7672e9b 2024-04-22T15:53:15.819 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (10m) 59s ago 10m 200M 1217M 18.2.1 5be31c24972a 3c6ebc4ea041 2024-04-22T15:53:15.819 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (9m) 59s ago 9m 203M 1217M 18.2.1 5be31c24972a 6aa2e6796784 2024-04-22T15:53:15.819 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (9m) 59s ago 9m 174M 1217M 18.2.1 5be31c24972a 3d9d310aa9eb 2024-04-22T15:53:15.819 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi027 running (9m) 2m ago 9m 227M 2240M 18.2.1 5be31c24972a cdf9b7aaf865 2024-04-22T15:53:15.819 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (8m) 2m ago 8m 184M 2240M 18.2.1 5be31c24972a 629397d49c76 2024-04-22T15:53:15.819 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi027 running (8m) 2m ago 8m 173M 2240M 18.2.1 5be31c24972a 23adcf675b04 2024-04-22T15:53:15.820 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.smithi016 smithi016 *:9095 running (112s) 59s ago 11m 51.9M - 2.43.0 a07b618ecd1d 373eaadee82d 2024-04-22T15:53:15.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.818+0000 7fd6567f2700 1 -- 172.21.15.16:0/88323614 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fd63806e940 msgr2=0x7fd638070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:15.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.818+0000 7fd6567f2700 1 --2- 172.21.15.16:0/88323614 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fd63806e940 0x7fd638070e00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd650197b50 tx=0x7fd64400b410 comp rx=0 tx=0).stop 2024-04-22T15:53:15.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.819+0000 7fd6567f2700 1 -- 172.21.15.16:0/88323614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd650068580 msgr2=0x7fd65019c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:15.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.819+0000 7fd6567f2700 1 --2- 172.21.15.16:0/88323614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd650068580 0x7fd65019c660 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd640006010 tx=0x7fd64000bd70 comp rx=0 tx=0).stop 2024-04-22T15:53:15.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.819+0000 7fd6567f2700 1 -- 172.21.15.16:0/88323614 shutdown_connections 2024-04-22T15:53:15.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.819+0000 7fd6567f2700 1 --2- 172.21.15.16:0/88323614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd650068580 0x7fd65019c660 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.819+0000 7fd6567f2700 1 --2- 172.21.15.16:0/88323614 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7fd63806e940 0x7fd638070e00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.819+0000 7fd6567f2700 1 --2- 172.21.15.16:0/88323614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd650106cd0 0x7fd65019cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.819+0000 7fd6567f2700 1 -- 172.21.15.16:0/88323614 >> 172.21.15.16:0/88323614 conn(0x7fd650075670 msgr2=0x7fd6500fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:15.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.819+0000 7fd6567f2700 1 -- 172.21.15.16:0/88323614 shutdown_connections 2024-04-22T15:53:15.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.819+0000 7fd6567f2700 1 -- 172.21.15.16:0/88323614 wait complete. 2024-04-22T15:53:15.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.929+0000 7f6008679700 1 -- 172.21.15.16:0/3893650879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6000105bf0 msgr2=0x7f6000105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:15.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.929+0000 7f6008679700 1 --2- 172.21.15.16:0/3893650879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6000105bf0 0x7f6000105fd0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc009910 tx=0x7f5ffc009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:15.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.929+0000 7f6008679700 1 -- 172.21.15.16:0/3893650879 shutdown_connections 2024-04-22T15:53:15.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.929+0000 7f6008679700 1 --2- 172.21.15.16:0/3893650879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60000ffb40 0x7f60000fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.929+0000 7f6008679700 1 --2- 172.21.15.16:0/3893650879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6000105bf0 0x7f6000105fd0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.929+0000 7f6008679700 1 -- 172.21.15.16:0/3893650879 >> 172.21.15.16:0/3893650879 conn(0x7f60000fb610 msgr2=0x7f60000fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:15.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.930+0000 7f6008679700 1 -- 172.21.15.16:0/3893650879 shutdown_connections 2024-04-22T15:53:15.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.930+0000 7f6008679700 1 -- 172.21.15.16:0/3893650879 wait complete. 2024-04-22T15:53:15.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.930+0000 7f6008679700 1 Processor -- start 2024-04-22T15:53:15.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.930+0000 7f6008679700 1 -- start start 2024-04-22T15:53:15.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.930+0000 7f6008679700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60000ffb40 0x7f600019c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:15.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.930+0000 7f6008679700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6000105bf0 0x7f600019cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:15.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.930+0000 7f6008679700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600019d0d0 con 0x7f60000ffb40 2024-04-22T15:53:15.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.930+0000 7f6008679700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6000196610 con 0x7f6000105bf0 2024-04-22T15:53:15.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.931+0000 7f6006415700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60000ffb40 0x7f600019c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:15.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.931+0000 7f6006415700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60000ffb40 0x7f600019c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39296/0 (socket says 172.21.15.16:39296) 2024-04-22T15:53:15.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.931+0000 7f6006415700 1 -- 172.21.15.16:0/4121768545 learned_addr learned my addr 172.21.15.16:0/4121768545 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:53:15.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.931+0000 7f6005c14700 1 --2- 172.21.15.16:0/4121768545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6000105bf0 0x7f600019cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:15.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.931+0000 7f6006415700 1 -- 172.21.15.16:0/4121768545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6000105bf0 msgr2=0x7f600019cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:15.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.931+0000 7f6006415700 1 --2- 172.21.15.16:0/4121768545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6000105bf0 0x7f600019cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:15.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.931+0000 7f6006415700 1 -- 172.21.15.16:0/4121768545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ffc0095f0 con 0x7f60000ffb40 2024-04-22T15:53:15.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.931+0000 7f6006415700 1 --2- 172.21.15.16:0/4121768545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60000ffb40 0x7f600019c590 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc006010 tx=0x7f5ffc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:15.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.931+0000 7f5ff37fe700 1 -- 172.21.15.16:0/4121768545 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ffc01d070 con 0x7f60000ffb40 2024-04-22T15:53:15.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.931+0000 7f5ff37fe700 1 -- 172.21.15.16:0/4121768545 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ffc00f460 con 0x7f60000ffb40 2024-04-22T15:53:15.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.931+0000 7f6008679700 1 -- 172.21.15.16:0/4121768545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6000196890 con 0x7f60000ffb40 2024-04-22T15:53:15.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.931+0000 7f5ff37fe700 1 -- 172.21.15.16:0/4121768545 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ffc017630 con 0x7f60000ffb40 2024-04-22T15:53:15.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.931+0000 7f6008679700 1 -- 172.21.15.16:0/4121768545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6000196d80 con 0x7f60000ffb40 2024-04-22T15:53:15.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.933+0000 7f6008679700 1 -- 172.21.15.16:0/4121768545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f600004f2e0 con 0x7f60000ffb40 2024-04-22T15:53:15.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.934+0000 7f5ff37fe700 1 -- 172.21.15.16:0/4121768545 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f5ffc00f5d0 con 0x7f60000ffb40 2024-04-22T15:53:15.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.934+0000 7f5ff37fe700 1 --2- 172.21.15.16:0/4121768545 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f5fec072d50 0x7f5fec075210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:15.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.934+0000 7f5ff37fe700 1 -- 172.21.15.16:0/4121768545 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f5ffc08fdc0 con 0x7f60000ffb40 2024-04-22T15:53:15.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.934+0000 7f6005c14700 1 --2- 172.21.15.16:0/4121768545 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f5fec072d50 0x7f5fec075210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:15.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.935+0000 7f6005c14700 1 --2- 172.21.15.16:0/4121768545 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f5fec072d50 0x7f5fec075210 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f60000feb50 tx=0x7f5ff400a380 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:15.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:15.939+0000 7f5ff37fe700 1 -- 172.21.15.16:0/4121768545 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f5ffc05cbd0 con 0x7f60000ffb40 2024-04-22T15:53:16.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.246+0000 7f6008679700 1 -- 172.21.15.16:0/4121768545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6000197b20 con 0x7f60000ffb40 2024-04-22T15:53:16.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.247+0000 7f5ff37fe700 1 -- 172.21.15.16:0/4121768545 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f5ffc05c760 con 0x7f60000ffb40 2024-04-22T15:53:16.250 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:53:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-22T15:53:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T15:53:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:53:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-22T15:53:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:53:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:53:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-22T15:53:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-22T15:53:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:53:16.251 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-22T15:53:16.251 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T15:53:16.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:53:16.251 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-22T15:53:16.251 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-04-22T15:53:16.251 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:53:16.251 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-22T15:53:16.251 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:53:16.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.250+0000 7f6008679700 1 -- 172.21.15.16:0/4121768545 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f5fec072d50 msgr2=0x7f5fec075210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:16.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.250+0000 7f6008679700 1 --2- 172.21.15.16:0/4121768545 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f5fec072d50 0x7f5fec075210 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f60000feb50 tx=0x7f5ff400a380 comp rx=0 tx=0).stop 2024-04-22T15:53:16.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.250+0000 7f6008679700 1 -- 172.21.15.16:0/4121768545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60000ffb40 msgr2=0x7f600019c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:16.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.250+0000 7f6008679700 1 --2- 172.21.15.16:0/4121768545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60000ffb40 0x7f600019c590 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc006010 tx=0x7f5ffc00bd70 comp rx=0 tx=0).stop 2024-04-22T15:53:16.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.250+0000 7f6008679700 1 -- 172.21.15.16:0/4121768545 shutdown_connections 2024-04-22T15:53:16.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.250+0000 7f6008679700 1 --2- 172.21.15.16:0/4121768545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60000ffb40 0x7f600019c590 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.250+0000 7f6008679700 1 --2- 172.21.15.16:0/4121768545 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f5fec072d50 0x7f5fec075210 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.250+0000 7f6008679700 1 --2- 172.21.15.16:0/4121768545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6000105bf0 0x7f600019cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.250+0000 7f6008679700 1 -- 172.21.15.16:0/4121768545 >> 172.21.15.16:0/4121768545 conn(0x7f60000fb610 msgr2=0x7f6000068ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:16.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.250+0000 7f6008679700 1 -- 172.21.15.16:0/4121768545 shutdown_connections 2024-04-22T15:53:16.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.250+0000 7f6008679700 1 -- 172.21.15.16:0/4121768545 wait complete. 2024-04-22T15:53:16.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.360+0000 7f7372677700 1 -- 172.21.15.16:0/696865787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f736c0ffda0 msgr2=0x7f736c100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:16.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.360+0000 7f7372677700 1 --2- 172.21.15.16:0/696865787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f736c0ffda0 0x7f736c100180 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f735c009910 tx=0x7f735c009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:16.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.360+0000 7f7372677700 1 -- 172.21.15.16:0/696865787 shutdown_connections 2024-04-22T15:53:16.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.360+0000 7f7372677700 1 --2- 172.21.15.16:0/696865787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f736c100750 0x7f736c1047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.360+0000 7f7372677700 1 --2- 172.21.15.16:0/696865787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f736c0ffda0 0x7f736c100180 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.360+0000 7f7372677700 1 -- 172.21.15.16:0/696865787 >> 172.21.15.16:0/696865787 conn(0x7f736c0fb650 msgr2=0x7f736c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:16.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.360+0000 7f7372677700 1 -- 172.21.15.16:0/696865787 shutdown_connections 2024-04-22T15:53:16.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.360+0000 7f7372677700 1 -- 172.21.15.16:0/696865787 wait complete. 2024-04-22T15:53:16.365 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:16 smithi016 ceph-mon[26620]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:16.365 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:16 smithi016 ceph-mon[26620]: from='client.14854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:16.365 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:16 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:16.365 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:16 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:53:16.365 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:16 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:53:16.365 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:16 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:16.365 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:16 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-22T15:53:16.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.361+0000 7f7372677700 1 Processor -- start 2024-04-22T15:53:16.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.361+0000 7f7372677700 1 -- start start 2024-04-22T15:53:16.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.362+0000 7f7372677700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f736c0ffda0 0x7f736c101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:16.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.362+0000 7f7372677700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f736c100750 0x7f736c101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:16.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.362+0000 7f7372677700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736c102350 con 0x7f736c100750 2024-04-22T15:53:16.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.362+0000 7f7372677700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736c1024c0 con 0x7f736c0ffda0 2024-04-22T15:53:16.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.362+0000 7f736b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f736c100750 0x7f736c101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:16.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.362+0000 7f736bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f736c0ffda0 0x7f736c101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:16.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.362+0000 7f736b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f736c100750 0x7f736c101d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39318/0 (socket says 172.21.15.16:39318) 2024-04-22T15:53:16.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.362+0000 7f736b7fe700 1 -- 172.21.15.16:0/3629164071 learned_addr learned my addr 172.21.15.16:0/3629164071 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:53:16.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.362+0000 7f736b7fe700 1 -- 172.21.15.16:0/3629164071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f736c0ffda0 msgr2=0x7f736c101840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:16.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.362+0000 7f736b7fe700 1 --2- 172.21.15.16:0/3629164071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f736c0ffda0 0x7f736c101840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.362+0000 7f736b7fe700 1 -- 172.21.15.16:0/3629164071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735c0095f0 con 0x7f736c100750 2024-04-22T15:53:16.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.363+0000 7f736b7fe700 1 --2- 172.21.15.16:0/3629164071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f736c100750 0x7f736c101d80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f736000b480 tx=0x7f736000b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:16.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.363+0000 7f73697fa700 1 -- 172.21.15.16:0/3629164071 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7360011800 con 0x7f736c100750 2024-04-22T15:53:16.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.363+0000 7f73697fa700 1 -- 172.21.15.16:0/3629164071 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7360011e20 con 0x7f736c100750 2024-04-22T15:53:16.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.363+0000 7f7372677700 1 -- 172.21.15.16:0/3629164071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f736c1a2b70 con 0x7f736c100750 2024-04-22T15:53:16.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.364+0000 7f73697fa700 1 -- 172.21.15.16:0/3629164071 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7360010ea0 con 0x7f736c100750 2024-04-22T15:53:16.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.364+0000 7f7372677700 1 -- 172.21.15.16:0/3629164071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f736c1a2fe0 con 0x7f736c100750 2024-04-22T15:53:16.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.365+0000 7f7372677700 1 -- 172.21.15.16:0/3629164071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f736c04f2e0 con 0x7f736c100750 2024-04-22T15:53:16.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.365+0000 7f73697fa700 1 -- 172.21.15.16:0/3629164071 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f7360010380 con 0x7f736c100750 2024-04-22T15:53:16.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.366+0000 7f73697fa700 1 --2- 172.21.15.16:0/3629164071 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f735406e940 0x7f7354070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:16.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.366+0000 7f73697fa700 1 -- 172.21.15.16:0/3629164071 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f736008e2d0 con 0x7f736c100750 2024-04-22T15:53:16.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.366+0000 7f736bfff700 1 --2- 172.21.15.16:0/3629164071 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f735406e940 0x7f7354070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:16.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.367+0000 7f736bfff700 1 --2- 172.21.15.16:0/3629164071 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f735406e940 0x7f7354070e00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f735c006010 tx=0x7f735c01a040 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:16.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.371+0000 7f73697fa700 1 -- 172.21.15.16:0/3629164071 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f7360057630 con 0x7f736c100750 2024-04-22T15:53:16.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:16 smithi027 ceph-mon[34922]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:16.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:16 smithi027 ceph-mon[34922]: from='client.14854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:16.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:16 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:16.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:16 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:53:16.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:16 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:53:16.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:16 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:16.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:16 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-22T15:53:16.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.629+0000 7f7372677700 1 -- 172.21.15.16:0/3629164071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f736c102680 con 0x7f736c100750 2024-04-22T15:53:16.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.630+0000 7f73697fa700 1 -- 172.21.15.16:0/3629164071 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1849 (secure 0 0 0) 0x7f736005acf0 con 0x7f736c100750 2024-04-22T15:53:16.633 INFO:teuthology.orchestra.run.smithi016.stdout:e16 2024-04-22T15:53:16.633 INFO:teuthology.orchestra.run.smithi016.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:53:16.633 INFO:teuthology.orchestra.run.smithi016.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:53:16.633 INFO:teuthology.orchestra.run.smithi016.stdout:legacy client fscid: 1 2024-04-22T15:53:16.633 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:53:16.633 INFO:teuthology.orchestra.run.smithi016.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:53:16.633 INFO:teuthology.orchestra.run.smithi016.stdout:fs_name cephfs 2024-04-22T15:53:16.634 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 16 2024-04-22T15:53:16.634 INFO:teuthology.orchestra.run.smithi016.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T15:53:16.634 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-04-22T15:46:27.774571+0000 2024-04-22T15:53:16.634 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-04-22T15:46:47.381709+0000 2024-04-22T15:53:16.634 INFO:teuthology.orchestra.run.smithi016.stdout:tableserver 0 2024-04-22T15:53:16.634 INFO:teuthology.orchestra.run.smithi016.stdout:root 0 2024-04-22T15:53:16.634 INFO:teuthology.orchestra.run.smithi016.stdout:session_timeout 60 2024-04-22T15:53:16.634 INFO:teuthology.orchestra.run.smithi016.stdout:session_autoclose 300 2024-04-22T15:53:16.634 INFO:teuthology.orchestra.run.smithi016.stdout:max_file_size 1099511627776 2024-04-22T15:53:16.634 INFO:teuthology.orchestra.run.smithi016.stdout:required_client_features {} 2024-04-22T15:53:16.634 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure 0 2024-04-22T15:53:16.634 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure_osd_epoch 46 2024-04-22T15:53:16.634 INFO:teuthology.orchestra.run.smithi016.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:53:16.635 INFO:teuthology.orchestra.run.smithi016.stdout:max_mds 1 2024-04-22T15:53:16.635 INFO:teuthology.orchestra.run.smithi016.stdout:in 0 2024-04-22T15:53:16.635 INFO:teuthology.orchestra.run.smithi016.stdout:up {0=24279} 2024-04-22T15:53:16.635 INFO:teuthology.orchestra.run.smithi016.stdout:failed 2024-04-22T15:53:16.635 INFO:teuthology.orchestra.run.smithi016.stdout:damaged 2024-04-22T15:53:16.635 INFO:teuthology.orchestra.run.smithi016.stdout:stopped 2024-04-22T15:53:16.635 INFO:teuthology.orchestra.run.smithi016.stdout:data_pools [3] 2024-04-22T15:53:16.635 INFO:teuthology.orchestra.run.smithi016.stdout:metadata_pool 2 2024-04-22T15:53:16.635 INFO:teuthology.orchestra.run.smithi016.stdout:inline_data enabled 2024-04-22T15:53:16.635 INFO:teuthology.orchestra.run.smithi016.stdout:balancer 2024-04-22T15:53:16.635 INFO:teuthology.orchestra.run.smithi016.stdout:bal_rank_mask -1 2024-04-22T15:53:16.635 INFO:teuthology.orchestra.run.smithi016.stdout:standby_count_wanted 1 2024-04-22T15:53:16.635 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.iobjds{0:24279} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.27:6824/4216277609,v1:172.21.15.27:6825/4216277609] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:53:16.635 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.pgvphu{0:14494} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.16:6828/1006073414,v1:172.21.15.16:6829/1006073414] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:53:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:53:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:53:16.636 INFO:teuthology.orchestra.run.smithi016.stdout:Standby daemons: 2024-04-22T15:53:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:53:16.636 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.fkkejm{-1:14504} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.16:6826/3484907755,v1:172.21.15.16:6827/3484907755] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:53:16.636 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.dgqyes{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.27:6826/476026355,v1:172.21.15.27:6827/476026355] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:53:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.632+0000 7f7372677700 1 -- 172.21.15.16:0/3629164071 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f735406e940 msgr2=0x7f7354070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.632+0000 7f7372677700 1 --2- 172.21.15.16:0/3629164071 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f735406e940 0x7f7354070e00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f735c006010 tx=0x7f735c01a040 comp rx=0 tx=0).stop 2024-04-22T15:53:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.632+0000 7f7372677700 1 -- 172.21.15.16:0/3629164071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f736c100750 msgr2=0x7f736c101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.632+0000 7f7372677700 1 --2- 172.21.15.16:0/3629164071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f736c100750 0x7f736c101d80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f736000b480 tx=0x7f736000b790 comp rx=0 tx=0).stop 2024-04-22T15:53:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.632+0000 7f7372677700 1 -- 172.21.15.16:0/3629164071 shutdown_connections 2024-04-22T15:53:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.632+0000 7f7372677700 1 --2- 172.21.15.16:0/3629164071 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f735406e940 0x7f7354070e00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.632+0000 7f7372677700 1 --2- 172.21.15.16:0/3629164071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f736c0ffda0 0x7f736c101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.632+0000 7f7372677700 1 --2- 172.21.15.16:0/3629164071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f736c100750 0x7f736c101d80 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.632+0000 7f7372677700 1 -- 172.21.15.16:0/3629164071 >> 172.21.15.16:0/3629164071 conn(0x7f736c0fb650 msgr2=0x7f736c104030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.632+0000 7f7372677700 1 -- 172.21.15.16:0/3629164071 shutdown_connections 2024-04-22T15:53:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.632+0000 7f7372677700 1 -- 172.21.15.16:0/3629164071 wait complete. 2024-04-22T15:53:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 16 2024-04-22T15:53:16.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.747+0000 7f3c93654700 1 -- 172.21.15.16:0/3800674842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c0ff5a0 msgr2=0x7f3c8c0ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:16.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.747+0000 7f3c93654700 1 --2- 172.21.15.16:0/3800674842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c0ff5a0 0x7f3c8c0ff980 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3c8800b0a0 tx=0x7f3c8800b3b0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.747+0000 7f3c93654700 1 -- 172.21.15.16:0/3800674842 shutdown_connections 2024-04-22T15:53:16.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.747+0000 7f3c93654700 1 --2- 172.21.15.16:0/3800674842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c8c0fff50 0x7f3c8c105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.747+0000 7f3c93654700 1 --2- 172.21.15.16:0/3800674842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c0ff5a0 0x7f3c8c0ff980 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.747+0000 7f3c93654700 1 -- 172.21.15.16:0/3800674842 >> 172.21.15.16:0/3800674842 conn(0x7f3c8c074e30 msgr2=0x7f3c8c075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:16.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.747+0000 7f3c93654700 1 -- 172.21.15.16:0/3800674842 shutdown_connections 2024-04-22T15:53:16.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.748+0000 7f3c93654700 1 -- 172.21.15.16:0/3800674842 wait complete. 2024-04-22T15:53:16.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.748+0000 7f3c93654700 1 Processor -- start 2024-04-22T15:53:16.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.748+0000 7f3c93654700 1 -- start start 2024-04-22T15:53:16.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.749+0000 7f3c93654700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c8c0ff5a0 0x7f3c8c100520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:16.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.749+0000 7f3c93654700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c0fff50 0x7f3c8c100a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:16.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.749+0000 7f3c93654700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c100fa0 con 0x7f3c8c0fff50 2024-04-22T15:53:16.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.749+0000 7f3c93654700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c1010e0 con 0x7f3c8c0ff5a0 2024-04-22T15:53:16.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.749+0000 7f3c90bef700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c0fff50 0x7f3c8c100a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:16.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.749+0000 7f3c90bef700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c0fff50 0x7f3c8c100a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39324/0 (socket says 172.21.15.16:39324) 2024-04-22T15:53:16.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.749+0000 7f3c90bef700 1 -- 172.21.15.16:0/1032302132 learned_addr learned my addr 172.21.15.16:0/1032302132 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:53:16.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.749+0000 7f3c913f0700 1 --2- 172.21.15.16:0/1032302132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c8c0ff5a0 0x7f3c8c100520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:16.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.749+0000 7f3c90bef700 1 -- 172.21.15.16:0/1032302132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c8c0ff5a0 msgr2=0x7f3c8c100520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:16.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.749+0000 7f3c90bef700 1 --2- 172.21.15.16:0/1032302132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c8c0ff5a0 0x7f3c8c100520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.749+0000 7f3c90bef700 1 -- 172.21.15.16:0/1032302132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c88009d00 con 0x7f3c8c0fff50 2024-04-22T15:53:16.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.749+0000 7f3c913f0700 1 --2- 172.21.15.16:0/1032302132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c8c0ff5a0 0x7f3c8c100520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:53:16.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.749+0000 7f3c90bef700 1 --2- 172.21.15.16:0/1032302132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c0fff50 0x7f3c8c100a60 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c00b480 tx=0x7f3c7c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:16.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.749+0000 7f3c827fc700 1 -- 172.21.15.16:0/1032302132 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c7c0118c0 con 0x7f3c8c0fff50 2024-04-22T15:53:16.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.749+0000 7f3c827fc700 1 -- 172.21.15.16:0/1032302132 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c7c010380 con 0x7f3c8c0fff50 2024-04-22T15:53:16.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.749+0000 7f3c93654700 1 -- 172.21.15.16:0/1032302132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c8c1a2be0 con 0x7f3c8c0fff50 2024-04-22T15:53:16.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.749+0000 7f3c827fc700 1 -- 172.21.15.16:0/1032302132 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c7c00f4f0 con 0x7f3c8c0fff50 2024-04-22T15:53:16.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.750+0000 7f3c93654700 1 -- 172.21.15.16:0/1032302132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c8c1a2ff0 con 0x7f3c8c0fff50 2024-04-22T15:53:16.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.751+0000 7f3c93654700 1 -- 172.21.15.16:0/1032302132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c8c109270 con 0x7f3c8c0fff50 2024-04-22T15:53:16.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.751+0000 7f3c827fc700 1 -- 172.21.15.16:0/1032302132 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f3c7c0104f0 con 0x7f3c8c0fff50 2024-04-22T15:53:16.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.752+0000 7f3c827fc700 1 --2- 172.21.15.16:0/1032302132 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f3c7806e940 0x7f3c78070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:16.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.752+0000 7f3c827fc700 1 -- 172.21.15.16:0/1032302132 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f3c7c08e4a0 con 0x7f3c8c0fff50 2024-04-22T15:53:16.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.752+0000 7f3c913f0700 1 --2- 172.21.15.16:0/1032302132 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f3c7806e940 0x7f3c78070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:16.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.752+0000 7f3c913f0700 1 --2- 172.21.15.16:0/1032302132 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f3c7806e940 0x7f3c78070e00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3c880062a0 tx=0x7f3c88006210 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:16.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.756+0000 7f3c827fc700 1 -- 172.21.15.16:0/1032302132 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f3c7c057760 con 0x7f3c8c0fff50 2024-04-22T15:53:16.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.990+0000 7f3c93654700 1 -- 172.21.15.16:0/1032302132 --> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c8c101b60 con 0x7f3c7806e940 2024-04-22T15:53:16.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.991+0000 7f3c827fc700 1 -- 172.21.15.16:0/1032302132 <== mgr.24461 v2:172.21.15.16:6800/3148690064 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f3c8c101b60 con 0x7f3c7806e940 2024-04-22T15:53:16.995 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:53:16.995 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:53:16.995 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-22T15:53:16.995 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:53:16.995 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-22T15:53:16.995 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-04-22T15:53:16.995 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-22T15:53:16.995 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "2/23 daemons upgraded", 2024-04-22T15:53:16.995 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading mon daemons", 2024-04-22T15:53:16.995 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-22T15:53:16.995 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:53:16.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.993+0000 7f3c93654700 1 -- 172.21.15.16:0/1032302132 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f3c7806e940 msgr2=0x7f3c78070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:16.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.993+0000 7f3c93654700 1 --2- 172.21.15.16:0/1032302132 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f3c7806e940 0x7f3c78070e00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3c880062a0 tx=0x7f3c88006210 comp rx=0 tx=0).stop 2024-04-22T15:53:16.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.993+0000 7f3c93654700 1 -- 172.21.15.16:0/1032302132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c0fff50 msgr2=0x7f3c8c100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:16.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.993+0000 7f3c93654700 1 --2- 172.21.15.16:0/1032302132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c0fff50 0x7f3c8c100a60 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c00b480 tx=0x7f3c7c00b840 comp rx=0 tx=0).stop 2024-04-22T15:53:16.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.994+0000 7f3c93654700 1 -- 172.21.15.16:0/1032302132 shutdown_connections 2024-04-22T15:53:16.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.994+0000 7f3c93654700 1 --2- 172.21.15.16:0/1032302132 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f3c7806e940 0x7f3c78070e00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.994+0000 7f3c93654700 1 --2- 172.21.15.16:0/1032302132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c8c0ff5a0 0x7f3c8c100520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.994+0000 7f3c93654700 1 --2- 172.21.15.16:0/1032302132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c0fff50 0x7f3c8c100a60 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:16.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.994+0000 7f3c93654700 1 -- 172.21.15.16:0/1032302132 >> 172.21.15.16:0/1032302132 conn(0x7f3c8c074e30 msgr2=0x7f3c8c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:16.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.994+0000 7f3c93654700 1 -- 172.21.15.16:0/1032302132 shutdown_connections 2024-04-22T15:53:16.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:16.994+0000 7f3c93654700 1 -- 172.21.15.16:0/1032302132 wait complete. 2024-04-22T15:53:17.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.103+0000 7f1db9194700 1 -- 172.21.15.16:0/1522029730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1db4105bd0 msgr2=0x7f1db4105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:17.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.103+0000 7f1db9194700 1 --2- 172.21.15.16:0/1522029730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1db4105bd0 0x7f1db4105fb0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1da4009910 tx=0x7f1da4009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:17.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.103+0000 7f1db9194700 1 -- 172.21.15.16:0/1522029730 shutdown_connections 2024-04-22T15:53:17.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.103+0000 7f1db9194700 1 --2- 172.21.15.16:0/1522029730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1db40ffc20 0x7f1db4100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:17.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.103+0000 7f1db9194700 1 --2- 172.21.15.16:0/1522029730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1db4105bd0 0x7f1db4105fb0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:17.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.103+0000 7f1db9194700 1 -- 172.21.15.16:0/1522029730 >> 172.21.15.16:0/1522029730 conn(0x7f1db4075670 msgr2=0x7f1db4075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:17.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.103+0000 7f1db9194700 1 -- 172.21.15.16:0/1522029730 shutdown_connections 2024-04-22T15:53:17.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.103+0000 7f1db9194700 1 -- 172.21.15.16:0/1522029730 wait complete. 2024-04-22T15:53:17.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.104+0000 7f1db9194700 1 Processor -- start 2024-04-22T15:53:17.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.104+0000 7f1db9194700 1 -- start start 2024-04-22T15:53:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.104+0000 7f1db9194700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1db40ffc20 0x7f1db419c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.104+0000 7f1db9194700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1db4105bd0 0x7f1db419cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.104+0000 7f1db9194700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db419d1f0 con 0x7f1db40ffc20 2024-04-22T15:53:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.104+0000 7f1db9194700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db41966a0 con 0x7f1db4105bd0 2024-04-22T15:53:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.104+0000 7f1db2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1db40ffc20 0x7f1db419c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.104+0000 7f1db2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1db40ffc20 0x7f1db419c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39354/0 (socket says 172.21.15.16:39354) 2024-04-22T15:53:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.104+0000 7f1db2d9d700 1 -- 172.21.15.16:0/1656706570 learned_addr learned my addr 172.21.15.16:0/1656706570 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:53:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.104+0000 7f1db259c700 1 --2- 172.21.15.16:0/1656706570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1db4105bd0 0x7f1db419cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.105+0000 7f1db2d9d700 1 -- 172.21.15.16:0/1656706570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1db4105bd0 msgr2=0x7f1db419cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.105+0000 7f1db2d9d700 1 --2- 172.21.15.16:0/1656706570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1db4105bd0 0x7f1db419cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.105+0000 7f1db2d9d700 1 -- 172.21.15.16:0/1656706570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da40095f0 con 0x7f1db40ffc20 2024-04-22T15:53:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.105+0000 7f1db2d9d700 1 --2- 172.21.15.16:0/1656706570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1db40ffc20 0x7f1db419c620 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1da4005f50 tx=0x7f1da400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:17.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.105+0000 7f1d9bfff700 1 -- 172.21.15.16:0/1656706570 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1da401d070 con 0x7f1db40ffc20 2024-04-22T15:53:17.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.105+0000 7f1d9bfff700 1 -- 172.21.15.16:0/1656706570 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1da400b610 con 0x7f1db40ffc20 2024-04-22T15:53:17.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.105+0000 7f1d9bfff700 1 -- 172.21.15.16:0/1656706570 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1da4022470 con 0x7f1db40ffc20 2024-04-22T15:53:17.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.105+0000 7f1db9194700 1 -- 172.21.15.16:0/1656706570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1db4196920 con 0x7f1db40ffc20 2024-04-22T15:53:17.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.105+0000 7f1db9194700 1 -- 172.21.15.16:0/1656706570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1db4196e10 con 0x7f1db40ffc20 2024-04-22T15:53:17.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.106+0000 7f1db9194700 1 -- 172.21.15.16:0/1656706570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1db404f2e0 con 0x7f1db40ffc20 2024-04-22T15:53:17.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.107+0000 7f1d9bfff700 1 -- 172.21.15.16:0/1656706570 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f1da4003780 con 0x7f1db40ffc20 2024-04-22T15:53:17.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.108+0000 7f1d9bfff700 1 --2- 172.21.15.16:0/1656706570 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f1d9c06e940 0x7f1d9c070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:17.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.108+0000 7f1d9bfff700 1 -- 172.21.15.16:0/1656706570 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f1da408fe80 con 0x7f1db40ffc20 2024-04-22T15:53:17.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.108+0000 7f1db259c700 1 --2- 172.21.15.16:0/1656706570 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f1d9c06e940 0x7f1d9c070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:17.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.108+0000 7f1db259c700 1 --2- 172.21.15.16:0/1656706570 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f1d9c06e940 0x7f1d9c070e00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1db40ffa40 tx=0x7f1da8009260 comp rx=0 tx=0).ready entity=mgr.24461 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:17.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.112+0000 7f1d9bfff700 1 -- 172.21.15.16:0/1656706570 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f1da405cdd0 con 0x7f1db40ffc20 2024-04-22T15:53:17.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.414+0000 7f1db9194700 1 -- 172.21.15.16:0/1656706570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1db4197bb0 con 0x7f1db40ffc20 2024-04-22T15:53:17.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.414+0000 7f1d9bfff700 1 -- 172.21.15.16:0/1656706570 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f1da405c960 con 0x7f1db40ffc20 2024-04-22T15:53:17.417 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:53:17.417 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:53:17.417 INFO:teuthology.orchestra.run.smithi016.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:53:17.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.416+0000 7f1db9194700 1 -- 172.21.15.16:0/1656706570 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f1d9c06e940 msgr2=0x7f1d9c070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:17.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.416+0000 7f1db9194700 1 --2- 172.21.15.16:0/1656706570 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f1d9c06e940 0x7f1d9c070e00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1db40ffa40 tx=0x7f1da8009260 comp rx=0 tx=0).stop 2024-04-22T15:53:17.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.416+0000 7f1db9194700 1 -- 172.21.15.16:0/1656706570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1db40ffc20 msgr2=0x7f1db419c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:17.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.416+0000 7f1db9194700 1 --2- 172.21.15.16:0/1656706570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1db40ffc20 0x7f1db419c620 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1da4005f50 tx=0x7f1da400f690 comp rx=0 tx=0).stop 2024-04-22T15:53:17.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.416+0000 7f1db9194700 1 -- 172.21.15.16:0/1656706570 shutdown_connections 2024-04-22T15:53:17.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.416+0000 7f1db9194700 1 --2- 172.21.15.16:0/1656706570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1db40ffc20 0x7f1db419c620 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:17.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.416+0000 7f1db9194700 1 --2- 172.21.15.16:0/1656706570 >> [v2:172.21.15.16:6800/3148690064,v1:172.21.15.16:6801/3148690064] conn(0x7f1d9c06e940 0x7f1d9c070e00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:17.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.416+0000 7f1db9194700 1 --2- 172.21.15.16:0/1656706570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1db4105bd0 0x7f1db419cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:17.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.416+0000 7f1db9194700 1 -- 172.21.15.16:0/1656706570 >> 172.21.15.16:0/1656706570 conn(0x7f1db4075670 msgr2=0x7f1db40686c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:17.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.416+0000 7f1db9194700 1 -- 172.21.15.16:0/1656706570 shutdown_connections 2024-04-22T15:53:17.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:17.416+0000 7f1db9194700 1 -- 172.21.15.16:0/1656706570 wait complete. 2024-04-22T15:53:17.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:17 smithi027 ceph-mon[34922]: from='client.14858 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:17.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:17 smithi027 ceph-mon[34922]: pgmap v102: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:53:17.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:17 smithi027 ceph-mon[34922]: Upgrade: Target is version 19.0.0-3223-g5883d494 (squid) 2024-04-22T15:53:17.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:17 smithi027 ceph-mon[34922]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, digests ['quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb'] 2024-04-22T15:53:17.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:17 smithi027 ceph-mon[34922]: Upgrade: Setting container_image for all mgr 2024-04-22T15:53:17.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:17 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/4121768545' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:53:17.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:17 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/3629164071' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:53:17.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:17 smithi016 ceph-mon[26620]: from='client.14858 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:17.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:17 smithi016 ceph-mon[26620]: pgmap v102: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:53:17.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:17 smithi016 ceph-mon[26620]: Upgrade: Target is version 19.0.0-3223-g5883d494 (squid) 2024-04-22T15:53:17.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:17 smithi016 ceph-mon[26620]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, digests ['quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb'] 2024-04-22T15:53:17.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:17 smithi016 ceph-mon[26620]: Upgrade: Setting container_image for all mgr 2024-04-22T15:53:17.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:17 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/4121768545' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:53:17.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:17 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/3629164071' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:53:18.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:18 smithi027 ceph-mon[34922]: from='client.14870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:18.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:18 smithi027 ceph-mon[34922]: from='client.? 172.21.15.16:0/1656706570' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:53:18.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:18 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:18.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:18 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:53:18.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:18 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T15:53:18.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:18 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:53:18.591 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:18 smithi016 ceph-mon[26620]: from='client.14870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:18.591 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:18 smithi016 ceph-mon[26620]: from='client.? 172.21.15.16:0/1656706570' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:53:18.591 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:18 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:18.591 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:18 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:53:18.591 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:18 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T15:53:18.591 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:18 smithi016 ceph-mon[26620]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:53:19.387 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:19 smithi016 ceph-mon[26620]: pgmap v103: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:53:19.387 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:19 smithi016 ceph-mon[26620]: Upgrade: Updating mon.smithi016 2024-04-22T15:53:19.387 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:19 smithi016 ceph-mon[26620]: Deploying daemon mon.smithi016 on smithi016 2024-04-22T15:53:19.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:19 smithi027 ceph-mon[34922]: pgmap v103: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:53:19.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:19 smithi027 ceph-mon[34922]: Upgrade: Updating mon.smithi016 2024-04-22T15:53:19.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:19 smithi027 ceph-mon[34922]: Deploying daemon mon.smithi016 on smithi016 2024-04-22T15:53:19.934 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:19 smithi016 systemd[1]: Stopping Ceph mon.smithi016 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T15:53:20.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:19 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016[26616]: 2024-04-22T15:53:19.931+0000 7f3307b1c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi016 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T15:53:20.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:19 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016[26616]: 2024-04-22T15:53:19.931+0000 7f3307b1c700 -1 mon.smithi016@0(leader) e2 *** Got Signal Terminated *** 2024-04-22T15:53:20.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:20 smithi016 podman[110562]: 2024-04-22 15:53:20.146508563 +0000 UTC m=+0.323833308 container died 72af40873cb62540258a2e042b4224fc5ef49c8669e8586975f9575603a75acf (image=quay.io/ceph/ceph:v18.2.1, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-22T15:53:20.785 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:20 smithi016 podman[110562]: 2024-04-22 15:53:20.514775044 +0000 UTC m=+0.692099742 container cleanup 72af40873cb62540258a2e042b4224fc5ef49c8669e8586975f9575603a75acf (image=quay.io/ceph/ceph:v18.2.1, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-22T15:53:20.785 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:20 smithi016 bash[110562]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016 2024-04-22T15:53:21.112 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:20 smithi016 podman[110587]: 2024-04-22 15:53:20.782995361 +0000 UTC m=+0.629521001 container remove 72af40873cb62540258a2e042b4224fc5ef49c8669e8586975f9575603a75acf (image=quay.io/ceph/ceph:v18.2.1, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, io.buildah.version=1.29.1) 2024-04-22T15:53:21.112 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:20 smithi016 systemd[1]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968@mon.smithi016.service: Deactivated successfully. 2024-04-22T15:53:21.112 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:20 smithi016 systemd[1]: Stopped Ceph mon.smithi016 for 67ab3a80-00be-11ef-bc93-c7b262605968. 2024-04-22T15:53:21.112 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:20 smithi016 systemd[1]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968@mon.smithi016.service: Consumed 15.107s CPU time. 2024-04-22T15:53:21.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 systemd[1]: Starting Ceph mon.smithi016 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T15:53:21.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 podman[110721]: 2024-04-22 15:53:21.229989797 +0000 UTC m=+0.018670280 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:53:21.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 podman[110721]: 2024-04-22 15:53:21.33432491 +0000 UTC m=+0.123005371 container create 4ae529a67806e75f9205eb4a9422ee24002b45a3de36f4e17bff538a0d94e9d0 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-22T15:53:21.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 podman[110721]: 2024-04-22 15:53:21.52857824 +0000 UTC m=+0.317258702 container init 4ae529a67806e75f9205eb4a9422ee24002b45a3de36f4e17bff538a0d94e9d0 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016, org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-04-22T15:53:21.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 podman[110721]: 2024-04-22 15:53:21.531803736 +0000 UTC m=+0.320484200 container start 4ae529a67806e75f9205eb4a9422ee24002b45a3de36f4e17bff538a0d94e9d0 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T15:53:21.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T15:53:21.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev), process ceph-mon, pid 2 2024-04-22T15:53:21.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: pidfile_write: ignore empty --pid-file 2024-04-22T15:53:21.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: load: jerasure load: lrc 2024-04-22T15:53:21.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: RocksDB version: 7.9.2 2024-04-22T15:53:21.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Git sha 0 2024-04-22T15:53:21.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Compile date 2024-04-22 12:55:29 2024-04-22T15:53:21.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: DB SUMMARY 2024-04-22T15:53:21.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: DB Session ID: ZA3WCAHNJ6862A2MPRH5 2024-04-22T15:53:21.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: CURRENT file: CURRENT 2024-04-22T15:53:21.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T15:53:21.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1501 Bytes 2024-04-22T15:53:21.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi016/store.db dir, Total Num: 1, files: 000032.sst 2024-04-22T15:53:21.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi016/store.db: 000030.log size: 3762346 ; 2024-04-22T15:53:21.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.error_if_exists: 0 2024-04-22T15:53:21.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.create_if_missing: 0 2024-04-22T15:53:21.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.paranoid_checks: 1 2024-04-22T15:53:21.920 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-22T15:53:21.920 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T15:53:21.920 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-22T15:53:21.920 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.env: 0x562185ba9cc0 2024-04-22T15:53:21.920 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.fs: PosixFileSystem 2024-04-22T15:53:21.920 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.info_log: 0x5621865c5aa0 2024-04-22T15:53:21.920 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T15:53:21.920 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.statistics: (nil) 2024-04-22T15:53:21.920 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.use_fsync: 0 2024-04-22T15:53:21.920 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_log_file_size: 0 2024-04-22T15:53:21.920 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T15:53:21.920 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T15:53:21.920 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T15:53:21.920 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T15:53:21.920 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.allow_fallocate: 1 2024-04-22T15:53:21.921 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T15:53:21.921 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T15:53:21.921 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.use_direct_reads: 0 2024-04-22T15:53:21.921 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T15:53:21.921 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T15:53:21.921 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.db_log_dir: 2024-04-22T15:53:21.921 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.wal_dir: 2024-04-22T15:53:21.921 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T15:53:21.921 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T15:53:21.921 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T15:53:21.921 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T15:53:21.922 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T15:53:21.922 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T15:53:21.922 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T15:53:21.922 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T15:53:21.922 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.write_buffer_manager: 0x5621865590e0 2024-04-22T15:53:21.922 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T15:53:21.922 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T15:53:21.922 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T15:53:21.922 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T15:53:21.922 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T15:53:21.922 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T15:53:21.923 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T15:53:21.923 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T15:53:21.923 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.unordered_write: 0 2024-04-22T15:53:21.923 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T15:53:21.923 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T15:53:21.923 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T15:53:21.923 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T15:53:21.924 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.row_cache: None 2024-04-22T15:53:21.925 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.wal_filter: None 2024-04-22T15:53:21.925 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T15:53:21.925 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T15:53:21.925 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.two_write_queues: 0 2024-04-22T15:53:21.925 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T15:53:21.925 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.wal_compression: 0 2024-04-22T15:53:21.925 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.atomic_flush: 0 2024-04-22T15:53:21.925 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T15:53:21.926 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T15:53:21.926 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T15:53:21.926 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.log_readahead_size: 0 2024-04-22T15:53:21.926 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T15:53:21.926 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T15:53:21.926 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T15:53:21.926 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T15:53:21.926 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T15:53:21.926 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T15:53:21.926 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-22T15:53:21.926 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_background_jobs: 2 2024-04-22T15:53:21.926 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_background_compactions: -1 2024-04-22T15:53:21.926 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_subcompactions: 1 2024-04-22T15:53:21.927 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T15:53:21.927 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T15:53:21.927 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T15:53:21.927 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T15:53:21.927 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T15:53:21.927 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T15:53:21.927 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T15:53:21.927 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T15:53:21.927 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_open_files: -1 2024-04-22T15:53:21.927 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T15:53:21.927 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T15:53:21.928 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T15:53:21.928 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T15:53:21.928 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_background_flushes: -1 2024-04-22T15:53:21.928 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Compression algorithms supported: 2024-04-22T15:53:21.928 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T15:53:21.928 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: kZSTD supported: 0 2024-04-22T15:53:21.928 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: kXpressCompression supported: 0 2024-04-22T15:53:21.928 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T15:53:21.928 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: kZlibCompression supported: 1 2024-04-22T15:53:21.928 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: kSnappyCompression supported: 1 2024-04-22T15:53:21.928 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: kLZ4Compression supported: 1 2024-04-22T15:53:21.928 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: kBZip2Compression supported: 0 2024-04-22T15:53:21.928 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T15:53:21.928 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-22T15:53:21.928 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi016/store.db/MANIFEST-000015 2024-04-22T15:53:21.929 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-22T15:53:21.929 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T15:53:21.929 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.merge_operator: 2024-04-22T15:53:21.929 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.compaction_filter: None 2024-04-22T15:53:21.929 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T15:53:21.929 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T15:53:21.929 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T15:53:21.930 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T15:53:21.930 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5621865c5c80) 2024-04-22T15:53:21.930 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: cache_index_and_filter_blocks: 1 2024-04-22T15:53:21.930 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T15:53:21.930 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T15:53:21.930 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: pin_top_level_index_and_filter: 1 2024-04-22T15:53:21.930 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: index_type: 0 2024-04-22T15:53:21.930 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: data_block_index_type: 0 2024-04-22T15:53:21.930 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: index_shortening: 1 2024-04-22T15:53:21.930 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T15:53:21.930 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: checksum: 4 2024-04-22T15:53:21.930 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: no_block_cache: 0 2024-04-22T15:53:21.930 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: block_cache: 0x56218658b1f0 2024-04-22T15:53:21.931 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: block_cache_name: BinnedLRUCache 2024-04-22T15:53:21.931 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: block_cache_options: 2024-04-22T15:53:21.931 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: capacity : 536870912 2024-04-22T15:53:21.931 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: num_shard_bits : 4 2024-04-22T15:53:21.931 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: strict_capacity_limit : 0 2024-04-22T15:53:21.931 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: high_pri_pool_ratio: 0.000 2024-04-22T15:53:21.931 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: block_cache_compressed: (nil) 2024-04-22T15:53:21.931 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: persistent_cache: (nil) 2024-04-22T15:53:21.931 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: block_size: 4096 2024-04-22T15:53:21.931 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: block_size_deviation: 10 2024-04-22T15:53:21.931 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: block_restart_interval: 16 2024-04-22T15:53:21.931 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: index_block_restart_interval: 1 2024-04-22T15:53:21.931 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: metadata_block_size: 4096 2024-04-22T15:53:21.931 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: partition_filters: 0 2024-04-22T15:53:21.931 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: use_delta_encoding: 1 2024-04-22T15:53:21.932 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: filter_policy: bloomfilter 2024-04-22T15:53:21.932 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: whole_key_filtering: 1 2024-04-22T15:53:21.932 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: verify_compression: 0 2024-04-22T15:53:21.932 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: read_amp_bytes_per_bit: 0 2024-04-22T15:53:21.932 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: format_version: 5 2024-04-22T15:53:21.932 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: enable_index_compression: 1 2024-04-22T15:53:21.932 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: block_align: 0 2024-04-22T15:53:21.932 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: max_auto_readahead_size: 262144 2024-04-22T15:53:21.932 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: prepopulate_block_cache: 0 2024-04-22T15:53:21.932 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: initial_auto_readahead_size: 8192 2024-04-22T15:53:21.932 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: num_file_reads_for_auto_readahead: 2 2024-04-22T15:53:21.932 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T15:53:21.932 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T15:53:21.932 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.compression: NoCompression 2024-04-22T15:53:21.932 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T15:53:21.933 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T15:53:21.933 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T15:53:21.933 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.num_levels: 7 2024-04-22T15:53:21.933 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T15:53:21.933 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T15:53:21.933 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T15:53:21.933 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T15:53:21.933 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T15:53:21.933 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T15:53:21.933 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T15:53:21.933 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T15:53:21.933 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T15:53:21.933 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T15:53:21.933 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-22T15:53:21.934 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-22T15:53:21.934 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T15:53:21.934 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T15:53:21.934 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T15:53:21.934 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T15:53:21.934 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T15:53:21.934 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-22T15:53:21.934 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T15:53:21.934 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T15:53:21.934 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-22T15:53:21.934 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T15:53:21.934 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T15:53:21.934 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T15:53:21.934 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T15:53:21.935 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T15:53:21.935 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T15:53:21.935 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T15:53:21.935 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T15:53:21.935 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T15:53:21.935 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T15:53:21.935 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T15:53:21.935 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T15:53:21.935 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T15:53:21.935 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T15:53:21.935 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T15:53:21.935 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T15:53:21.935 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T15:53:21.935 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-22T15:53:21.935 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.arena_block_size: 1048576 2024-04-22T15:53:21.936 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T15:53:21.936 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T15:53:21.936 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T15:53:21.936 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T15:53:21.936 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T15:53:21.936 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T15:53:21.936 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T15:53:21.936 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T15:53:21.936 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T15:53:21.936 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T15:53:21.936 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T15:53:21.936 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T15:53:21.936 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T15:53:21.936 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-22T15:53:21.936 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.inplace_update_support: 0 2024-04-22T15:53:21.937 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T15:53:21.937 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T15:53:21.937 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T15:53:21.937 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T15:53:21.937 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.bloom_locality: 0 2024-04-22T15:53:21.937 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.max_successive_merges: 0 2024-04-22T15:53:21.937 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T15:53:21.937 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T15:53:21.937 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T15:53:21.937 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T15:53:21.937 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.ttl: 2592000 2024-04-22T15:53:21.937 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T15:53:21.937 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-22T15:53:21.937 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-22T15:53:21.937 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.enable_blob_files: false 2024-04-22T15:53:21.938 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.min_blob_size: 0 2024-04-22T15:53:21.938 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T15:53:21.938 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T15:53:21.938 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T15:53:21.938 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T15:53:21.938 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-22T15:53:21.938 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-22T15:53:21.938 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.blob_file_starting_level: 0 2024-04-22T15:53:21.938 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-22T15:53:21.938 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-22T15:53:21.938 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi016/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 34, last_sequence is 12095, log_number is 30,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 30 2024-04-22T15:53:21.938 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 30 2024-04-22T15:53:21.938 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b478f51e-83dc-4f3f-a0a5-72396151bec8 2024-04-22T15:53:21.939 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713801201562323, "job": 1, "event": "recovery_started", "wal_files": [30]} 2024-04-22T15:53:21.939 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #30 mode 2 2024-04-22T15:53:21.939 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713801201577126, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 35, "file_size": 3261300, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12096, "largest_seqno": 15253, "table_properties": {"data_size": 3249186, "index_size": 7483, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3589, "raw_key_size": 31101, "raw_average_key_size": 22, "raw_value_size": 3221587, "raw_average_value_size": 2284, "num_data_blocks": 339, "num_entries": 1410, "num_filter_entries": 1410, "num_deletions": 259, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713801201, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b478f51e-83dc-4f3f-a0a5-72396151bec8", "db_session_id": "ZA3WCAHNJ6862A2MPRH5", "orig_file_number": 35, "seqno_to_time_mapping": "N/A"}} 2024-04-22T15:53:21.939 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713801201577243, "job": 1, "event": "recovery_finished"} 2024-04-22T15:53:21.939 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: [db/version_set.cc:5047] Creating manifest 37 2024-04-22T15:53:21.939 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-22T15:53:21.939 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi016/store.db/000030.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T15:53:21.939 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5621866a0000 2024-04-22T15:53:21.939 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: DB pointer 0x56218668a000 2024-04-22T15:53:21.939 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-22T15:53:21.939 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-22T15:53:21.939 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: ** DB Stats ** 2024-04-22T15:53:21.939 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T15:53:21.939 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T15:53:21.939 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T15:53:21.940 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T15:53:21.940 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T15:53:21.940 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T15:53:21.940 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T15:53:21.940 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: 2024-04-22T15:53:21.940 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: ** Compaction Stats [default] ** 2024-04-22T15:53:21.940 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T15:53:21.940 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-22T15:53:21.940 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: L0 1/0 3.11 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 506.7 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-22T15:53:21.940 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: L6 1/0 5.87 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-22T15:53:21.941 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Sum 2/0 8.98 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 506.7 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-22T15:53:21.941 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 506.7 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-22T15:53:21.941 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: 2024-04-22T15:53:21.941 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: ** Compaction Stats [default] ** 2024-04-22T15:53:21.941 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T15:53:21.941 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T15:53:21.941 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 506.7 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-22T15:53:21.941 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: 2024-04-22T15:53:21.941 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-22T15:53:21.941 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: 2024-04-22T15:53:21.941 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T15:53:21.941 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-04-22T15:53:21.941 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T15:53:21.942 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T15:53:21.942 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T15:53:21.942 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T15:53:21.942 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Cumulative compaction: 0.00 GB write, 175.31 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T15:53:21.942 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Interval compaction: 0.00 GB write, 175.31 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T15:53:21.942 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T15:53:21.942 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Block cache BinnedLRUCache@0x56218658b1f0#2 capacity: 512.00 MB usage: 12.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-22T15:53:21.942 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,4.03 KB,0.0007689%) IndexBlock(1,8.08 KB,0.00154078%) Misc(1,0.00 KB,0%) 2024-04-22T15:53:21.942 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: 2024-04-22T15:53:21.942 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T15:53:21.942 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: starting mon.smithi016 rank 0 at public addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] at bind addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi016 fsid 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T15:53:21.943 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: mon.smithi016@-1(???) e2 preinit fsid 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T15:53:21.943 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: mon.smithi016@-1(???).mds e16 new map 2024-04-22T15:53:21.943 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: mon.smithi016@-1(???).mds e16 print_map 2024-04-22T15:53:21.943 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: e16 2024-04-22T15:53:21.943 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:53:21.943 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:53:21.943 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: legacy client fscid: 1 2024-04-22T15:53:21.943 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: 2024-04-22T15:53:21.944 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Filesystem 'cephfs' (1) 2024-04-22T15:53:21.944 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: fs_name cephfs 2024-04-22T15:53:21.944 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: epoch 16 2024-04-22T15:53:21.944 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T15:53:21.944 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: created 2024-04-22T15:46:27.774571+0000 2024-04-22T15:53:21.944 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: modified 2024-04-22T15:46:47.381709+0000 2024-04-22T15:53:21.944 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: tableserver 0 2024-04-22T15:53:21.945 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: root 0 2024-04-22T15:53:21.945 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: session_timeout 60 2024-04-22T15:53:21.945 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: session_autoclose 300 2024-04-22T15:53:21.945 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: max_file_size 1099511627776 2024-04-22T15:53:21.945 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: max_xattr_size 65536 2024-04-22T15:53:21.946 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: required_client_features {} 2024-04-22T15:53:21.946 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: last_failure 0 2024-04-22T15:53:21.946 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: last_failure_osd_epoch 46 2024-04-22T15:53:21.946 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:53:21.946 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: max_mds 1 2024-04-22T15:53:21.946 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: in 0 2024-04-22T15:53:21.947 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: up {0=24279} 2024-04-22T15:53:21.947 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: failed 2024-04-22T15:53:21.947 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: damaged 2024-04-22T15:53:21.947 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: stopped 2024-04-22T15:53:21.947 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: data_pools [3] 2024-04-22T15:53:21.947 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: metadata_pool 2 2024-04-22T15:53:21.947 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: inline_data enabled 2024-04-22T15:53:21.948 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: balancer 2024-04-22T15:53:21.948 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: bal_rank_mask -1 2024-04-22T15:53:21.948 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: standby_count_wanted 1 2024-04-22T15:53:21.948 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: qdb_cluster leader: 0 members: 2024-04-22T15:53:21.948 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: [mds.cephfs.smithi027.iobjds{0:24279} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.27:6824/4216277609,v1:172.21.15.27:6825/4216277609] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:53:21.948 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: [mds.cephfs.smithi016.pgvphu{0:14494} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.16:6828/1006073414,v1:172.21.15.16:6829/1006073414] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:53:21.948 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: 2024-04-22T15:53:21.949 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: 2024-04-22T15:53:21.949 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Standby daemons: 2024-04-22T15:53:21.949 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: 2024-04-22T15:53:21.949 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: [mds.cephfs.smithi016.fkkejm{-1:14504} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.16:6826/3484907755,v1:172.21.15.16:6827/3484907755] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:53:21.949 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: [mds.cephfs.smithi027.dgqyes{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.27:6826/476026355,v1:172.21.15.27:6827/476026355] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:53:21.949 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: mon.smithi016@-1(???).osd e48 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-22T15:53:21.950 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: mon.smithi016@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-22T15:53:21.950 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: mon.smithi016@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-22T15:53:21.950 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: mon.smithi016@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-22T15:53:21.950 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 ceph-mon[110740]: mon.smithi016@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-04-22T15:53:21.950 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 bash[110721]: 4ae529a67806e75f9205eb4a9422ee24002b45a3de36f4e17bff538a0d94e9d0 2024-04-22T15:53:21.950 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:21 smithi016 systemd[1]: Started Ceph mon.smithi016 for 67ab3a80-00be-11ef-bc93-c7b262605968. 2024-04-22T15:53:23.194 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:23 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-04-22T15:53:23.194 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:23 smithi027 ceph-mon[34922]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:53:23.194 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:23 smithi027 ceph-mon[34922]: pgmap v105: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:53:23.194 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:23 smithi027 ceph-mon[34922]: mon.smithi016 calling monitor election 2024-04-22T15:53:23.194 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:23 smithi027 ceph-mon[34922]: mon.smithi016 is new leader, mons smithi016,smithi027 in quorum (ranks 0,1) 2024-04-22T15:53:23.194 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:23 smithi027 ceph-mon[34922]: monmap epoch 2 2024-04-22T15:53:23.194 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:23 smithi027 ceph-mon[34922]: fsid 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T15:53:23.194 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:23 smithi027 ceph-mon[34922]: last_changed 2024-04-22T15:42:09.358969+0000 2024-04-22T15:53:23.195 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:23 smithi027 ceph-mon[34922]: created 2024-04-22T15:39:17.702487+0000 2024-04-22T15:53:23.195 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:23 smithi027 ceph-mon[34922]: min_mon_release 18 (reef) 2024-04-22T15:53:23.195 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:23 smithi027 ceph-mon[34922]: election_strategy: 1 2024-04-22T15:53:23.195 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:23 smithi027 ceph-mon[34922]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.smithi016 2024-04-22T15:53:23.195 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:23 smithi027 ceph-mon[34922]: 1: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.smithi027 2024-04-22T15:53:23.195 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:23 smithi027 ceph-mon[34922]: fsmap cephfs:1 {0=cephfs.smithi027.iobjds=up:active} 1 up:standby-replay 2 up:standby 2024-04-22T15:53:23.195 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:23 smithi027 ceph-mon[34922]: osdmap e48: 6 total, 6 up, 6 in 2024-04-22T15:53:23.195 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:23 smithi027 ceph-mon[34922]: mgrmap e32: smithi016.rnsnnr(active, since 3m), standbys: smithi027.ukdgzs 2024-04-22T15:53:23.195 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:23 smithi027 ceph-mon[34922]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:53:23.195 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:23 smithi027 ceph-mon[34922]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:53:23.195 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:23 smithi027 ceph-mon[34922]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:53:23.195 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:23 smithi027 ceph-mon[34922]: from='mgr.24461 ' entity='' 2024-04-22T15:53:23.195 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:23 smithi027 ceph-mon[34922]: mgrmap e33: smithi016.rnsnnr(active, since 3m), standbys: smithi027.ukdgzs 2024-04-22T15:53:23.399 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:23 smithi016 ceph-mon[110740]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-04-22T15:53:23.399 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:23 smithi016 ceph-mon[110740]: from='mgr.24461 172.21.15.16:0/2416349917' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:53:23.400 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:23 smithi016 ceph-mon[110740]: pgmap v105: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:53:23.400 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:23 smithi016 ceph-mon[110740]: mon.smithi016 calling monitor election 2024-04-22T15:53:23.400 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:23 smithi016 ceph-mon[110740]: mon.smithi016 is new leader, mons smithi016,smithi027 in quorum (ranks 0,1) 2024-04-22T15:53:23.400 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:23 smithi016 ceph-mon[110740]: monmap epoch 2 2024-04-22T15:53:23.400 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:23 smithi016 ceph-mon[110740]: fsid 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T15:53:23.400 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:23 smithi016 ceph-mon[110740]: last_changed 2024-04-22T15:42:09.358969+0000 2024-04-22T15:53:23.400 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:23 smithi016 ceph-mon[110740]: created 2024-04-22T15:39:17.702487+0000 2024-04-22T15:53:23.400 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:23 smithi016 ceph-mon[110740]: min_mon_release 18 (reef) 2024-04-22T15:53:23.400 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:23 smithi016 ceph-mon[110740]: election_strategy: 1 2024-04-22T15:53:23.400 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:23 smithi016 ceph-mon[110740]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.smithi016 2024-04-22T15:53:23.400 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:23 smithi016 ceph-mon[110740]: 1: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.smithi027 2024-04-22T15:53:23.400 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:23 smithi016 ceph-mon[110740]: fsmap cephfs:1 {0=cephfs.smithi027.iobjds=up:active} 1 up:standby-replay 2 up:standby 2024-04-22T15:53:23.400 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:23 smithi016 ceph-mon[110740]: osdmap e48: 6 total, 6 up, 6 in 2024-04-22T15:53:23.401 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:23 smithi016 ceph-mon[110740]: mgrmap e32: smithi016.rnsnnr(active, since 3m), standbys: smithi027.ukdgzs 2024-04-22T15:53:23.401 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:23 smithi016 ceph-mon[110740]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:53:23.401 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:23 smithi016 ceph-mon[110740]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:53:23.401 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:23 smithi016 ceph-mon[110740]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:53:23.401 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:23 smithi016 ceph-mon[110740]: from='mgr.24461 ' entity='' 2024-04-22T15:53:23.401 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:23 smithi016 ceph-mon[110740]: mgrmap e33: smithi016.rnsnnr(active, since 3m), standbys: smithi027.ukdgzs 2024-04-22T15:53:30.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: Standby manager daemon smithi027.ukdgzs restarted 2024-04-22T15:53:30.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: Standby manager daemon smithi027.ukdgzs started 2024-04-22T15:53:30.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.27:0/4190886874' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.ukdgzs/crt"}]: dispatch 2024-04-22T15:53:30.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.27:0/4190886874' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:53:30.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.27:0/4190886874' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.ukdgzs/key"}]: dispatch 2024-04-22T15:53:30.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: from='mgr.? 172.21.15.27:0/4190886874' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:53:30.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: Active manager daemon smithi016.rnsnnr restarted 2024-04-22T15:53:30.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: Activating manager daemon smithi016.rnsnnr 2024-04-22T15:53:30.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: osdmap e49: 6 total, 6 up, 6 in 2024-04-22T15:53:30.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: mgrmap e34: smithi016.rnsnnr(active, starting, since 0.00589389s), standbys: smithi027.ukdgzs 2024-04-22T15:53:30.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-04-22T15:53:30.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:53:30.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.pgvphu"}]: dispatch 2024-04-22T15:53:30.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.fkkejm"}]: dispatch 2024-04-22T15:53:30.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.iobjds"}]: dispatch 2024-04-22T15:53:30.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.dgqyes"}]: dispatch 2024-04-22T15:53:30.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr metadata", "who": "smithi016.rnsnnr", "id": "smithi016.rnsnnr"}]: dispatch 2024-04-22T15:53:30.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr metadata", "who": "smithi027.ukdgzs", "id": "smithi027.ukdgzs"}]: dispatch 2024-04-22T15:53:30.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:53:30.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:53:30.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:53:30.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:53:30.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:53:30.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:53:30.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T15:53:30.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T15:53:30.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T15:53:30.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:30 smithi027 ceph-mon[34922]: Manager daemon smithi016.rnsnnr is now available 2024-04-22T15:53:30.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: Standby manager daemon smithi027.ukdgzs restarted 2024-04-22T15:53:30.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: Standby manager daemon smithi027.ukdgzs started 2024-04-22T15:53:30.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: from='mgr.? 172.21.15.27:0/4190886874' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.ukdgzs/crt"}]: dispatch 2024-04-22T15:53:30.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: from='mgr.? 172.21.15.27:0/4190886874' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T15:53:30.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: from='mgr.? 172.21.15.27:0/4190886874' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.ukdgzs/key"}]: dispatch 2024-04-22T15:53:30.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: from='mgr.? 172.21.15.27:0/4190886874' entity='mgr.smithi027.ukdgzs' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T15:53:30.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: Active manager daemon smithi016.rnsnnr restarted 2024-04-22T15:53:30.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: Activating manager daemon smithi016.rnsnnr 2024-04-22T15:53:30.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: osdmap e49: 6 total, 6 up, 6 in 2024-04-22T15:53:30.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: mgrmap e34: smithi016.rnsnnr(active, starting, since 0.00589389s), standbys: smithi027.ukdgzs 2024-04-22T15:53:30.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-04-22T15:53:30.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:53:30.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.pgvphu"}]: dispatch 2024-04-22T15:53:30.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.fkkejm"}]: dispatch 2024-04-22T15:53:30.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.iobjds"}]: dispatch 2024-04-22T15:53:30.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.dgqyes"}]: dispatch 2024-04-22T15:53:30.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr metadata", "who": "smithi016.rnsnnr", "id": "smithi016.rnsnnr"}]: dispatch 2024-04-22T15:53:30.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr metadata", "who": "smithi027.ukdgzs", "id": "smithi027.ukdgzs"}]: dispatch 2024-04-22T15:53:30.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:53:30.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:53:30.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:53:30.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:53:30.669 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:53:30.669 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:53:30.669 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T15:53:30.669 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T15:53:30.669 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T15:53:30.669 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:30 smithi016 ceph-mon[110740]: Manager daemon smithi016.rnsnnr is now available 2024-04-22T15:53:31.545 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:31 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:53:31.545 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:31 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:53:31.545 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:31 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi016.rnsnnr/mirror_snapshot_schedule"}]: dispatch 2024-04-22T15:53:31.546 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:31 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi016.rnsnnr/trash_purge_schedule"}]: dispatch 2024-04-22T15:53:31.546 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:31 smithi016 ceph-mon[110740]: mgrmap e35: smithi016.rnsnnr(active, since 1.00997s), standbys: smithi027.ukdgzs 2024-04-22T15:53:31.568 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:31 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:53:31.568 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:31 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:53:31.568 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:31 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi016.rnsnnr/mirror_snapshot_schedule"}]: dispatch 2024-04-22T15:53:31.568 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:31 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi016.rnsnnr/trash_purge_schedule"}]: dispatch 2024-04-22T15:53:31.568 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:31 smithi027 ceph-mon[34922]: mgrmap e35: smithi016.rnsnnr(active, since 1.00997s), standbys: smithi027.ukdgzs 2024-04-22T15:53:32.314 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:32 smithi027 ceph-mon[34922]: [22/Apr/2024:15:53:31] ENGINE Bus STARTING 2024-04-22T15:53:32.314 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:32 smithi027 ceph-mon[34922]: pgmap v3: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:32.314 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:32 smithi027 ceph-mon[34922]: [22/Apr/2024:15:53:31] ENGINE Serving on https://172.21.15.16:7150 2024-04-22T15:53:32.314 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:32 smithi027 ceph-mon[34922]: [22/Apr/2024:15:53:31] ENGINE Serving on http://172.21.15.16:8765 2024-04-22T15:53:32.314 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:32 smithi027 ceph-mon[34922]: [22/Apr/2024:15:53:31] ENGINE Bus STARTED 2024-04-22T15:53:32.521 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:32 smithi016 ceph-mon[110740]: [22/Apr/2024:15:53:31] ENGINE Bus STARTING 2024-04-22T15:53:32.521 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:32 smithi016 ceph-mon[110740]: pgmap v3: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:32.521 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:32 smithi016 ceph-mon[110740]: [22/Apr/2024:15:53:31] ENGINE Serving on https://172.21.15.16:7150 2024-04-22T15:53:32.521 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:32 smithi016 ceph-mon[110740]: [22/Apr/2024:15:53:31] ENGINE Serving on http://172.21.15.16:8765 2024-04-22T15:53:32.521 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:32 smithi016 ceph-mon[110740]: [22/Apr/2024:15:53:31] ENGINE Bus STARTED 2024-04-22T15:53:33.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:33 smithi016 ceph-mon[110740]: pgmap v4: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:33.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:33 smithi016 ceph-mon[110740]: mgrmap e36: smithi016.rnsnnr(active, since 2s), standbys: smithi027.ukdgzs 2024-04-22T15:53:33.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:33 smithi027 ceph-mon[34922]: pgmap v4: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:33.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:33 smithi027 ceph-mon[34922]: mgrmap e36: smithi016.rnsnnr(active, since 2s), standbys: smithi027.ukdgzs 2024-04-22T15:53:34.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:34 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:34.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:34 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:35.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:34 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:35.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:34 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:35.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:35 smithi016 ceph-mon[110740]: pgmap v5: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:35.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:35 smithi016 ceph-mon[110740]: mgrmap e37: smithi016.rnsnnr(active, since 4s), standbys: smithi027.ukdgzs 2024-04-22T15:53:35.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:35 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:35.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:35 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:36.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:35 smithi027 ceph-mon[34922]: pgmap v5: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:36.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:35 smithi027 ceph-mon[34922]: mgrmap e37: smithi016.rnsnnr(active, since 4s), standbys: smithi027.ukdgzs 2024-04-22T15:53:36.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:35 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:36.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:35 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:37.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:37 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:37.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:37 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:37.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:37 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:37.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:37 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:38.063 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:38 smithi016 ceph-mon[110740]: pgmap v6: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:38.063 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:38 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:38.063 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:38 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:38.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:38 smithi027 ceph-mon[34922]: pgmap v6: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-22T15:53:38.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:38 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:38.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:38 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:40.318 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:40 smithi016 ceph-mon[110740]: pgmap v7: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 18 op/s 2024-04-22T15:53:40.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:40 smithi027 ceph-mon[34922]: pgmap v7: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 18 op/s 2024-04-22T15:53:42.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:41 smithi027 ceph-mon[34922]: pgmap v8: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-22T15:53:42.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:41 smithi027 ceph-mon[34922]: Detected new or changed devices on smithi016 2024-04-22T15:53:42.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:41 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:42.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:41 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:42.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:41 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:53:42.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:41 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:53:42.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:41 smithi027 ceph-mon[34922]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-22T15:53:42.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:41 smithi027 ceph-mon[34922]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-22T15:53:42.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:41 smithi027 ceph-mon[34922]: Updating smithi027:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:53:42.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:41 smithi027 ceph-mon[34922]: Updating smithi016:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:53:42.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:41 smithi016 ceph-mon[110740]: pgmap v8: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-22T15:53:42.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:41 smithi016 ceph-mon[110740]: Detected new or changed devices on smithi016 2024-04-22T15:53:42.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:41 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:42.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:41 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:42.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:41 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:53:42.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:41 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:53:42.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:41 smithi016 ceph-mon[110740]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-22T15:53:42.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:41 smithi016 ceph-mon[110740]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-22T15:53:42.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:41 smithi016 ceph-mon[110740]: Updating smithi027:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:53:42.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:41 smithi016 ceph-mon[110740]: Updating smithi016:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T15:53:43.066 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:42 smithi027 ceph-mon[34922]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:53:43.066 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:42 smithi027 ceph-mon[34922]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:53:43.066 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:42 smithi027 ceph-mon[34922]: Updating smithi027:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:53:43.066 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:42 smithi027 ceph-mon[34922]: Updating smithi016:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:53:43.069 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:42 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:43.070 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:42 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:43.070 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:42 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:43.070 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:42 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:43.070 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:42 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:43.070 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:42 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:53:43.070 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:42 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:53:43.070 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:42 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-22T15:53:43.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:42 smithi016 ceph-mon[110740]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:53:43.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:42 smithi016 ceph-mon[110740]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-22T15:53:43.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:42 smithi016 ceph-mon[110740]: Updating smithi027:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:53:43.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:42 smithi016 ceph-mon[110740]: Updating smithi016:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T15:53:43.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:42 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:43.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:42 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:43.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:42 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:43.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:42 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:43.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:42 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:43.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:42 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:53:43.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:42 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:53:43.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:42 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-22T15:53:44.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:43 smithi027 ceph-mon[34922]: pgmap v9: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-22T15:53:44.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:43 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:44.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:43 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:53:44.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:43 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T15:53:44.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:43 smithi027 ceph-mon[34922]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:53:44.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:43 smithi016 ceph-mon[110740]: pgmap v9: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-22T15:53:44.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:43 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:44.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:43 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:53:44.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:43 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T15:53:44.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:43 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:53:44.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:44 smithi016 ceph-mon[110740]: Upgrade: Updating mon.smithi027 2024-04-22T15:53:44.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:44 smithi016 ceph-mon[110740]: Deploying daemon mon.smithi027 on smithi027 2024-04-22T15:53:45.048 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:44 smithi027 ceph-mon[34922]: Upgrade: Updating mon.smithi027 2024-04-22T15:53:45.048 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:44 smithi027 ceph-mon[34922]: Deploying daemon mon.smithi027 on smithi027 2024-04-22T15:53:45.620 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:45 smithi027 systemd[1]: Stopping Ceph mon.smithi027 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T15:53:45.874 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:45 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi027[34904]: 2024-04-22T15:53:45.617+0000 7f407e4f8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi027 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T15:53:45.874 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:45 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi027[34904]: 2024-04-22T15:53:45.617+0000 7f407e4f8700 -1 mon.smithi027@1(peon) e2 *** Got Signal Terminated *** 2024-04-22T15:53:46.153 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:45 smithi027 podman[82888]: 2024-04-22 15:53:45.872669793 +0000 UTC m=+0.373945612 container died 9f21ce7fa7efbcb8069a76aea9e8eb14b09826943ef4eb05c8f00ff39f6f02e8 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi027, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-22T15:53:46.520 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:46 smithi027 podman[82888]: 2024-04-22 15:53:46.249980024 +0000 UTC m=+0.751255842 container cleanup 9f21ce7fa7efbcb8069a76aea9e8eb14b09826943ef4eb05c8f00ff39f6f02e8 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi027, RELEASE=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-22T15:53:46.520 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:46 smithi027 bash[82888]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi027 2024-04-22T15:53:46.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:46 smithi027 podman[82902]: 2024-04-22 15:53:46.51805434 +0000 UTC m=+0.649361189 container remove 9f21ce7fa7efbcb8069a76aea9e8eb14b09826943ef4eb05c8f00ff39f6f02e8 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi027, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-04-22T15:53:46.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:46 smithi027 systemd[1]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968@mon.smithi027.service: Deactivated successfully. 2024-04-22T15:53:46.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:46 smithi027 systemd[1]: Stopped Ceph mon.smithi027 for 67ab3a80-00be-11ef-bc93-c7b262605968. 2024-04-22T15:53:46.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:46 smithi027 systemd[1]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968@mon.smithi027.service: Consumed 8.410s CPU time. 2024-04-22T15:53:47.252 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:46 smithi027 systemd[1]: Starting Ceph mon.smithi027 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T15:53:47.252 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 podman[83036]: 2024-04-22 15:53:47.013743203 +0000 UTC m=+0.018746870 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:53:47.252 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 podman[83036]: 2024-04-22 15:53:47.138411971 +0000 UTC m=+0.143415627 container create f29fcb888801b2457b3c08120cd69643eb68a4b51ad1349541c63382ba1a1ee1 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi027, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, io.buildah.version=1.33.5) 2024-04-22T15:53:47.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.530+0000 7f352aed8700 1 -- 172.21.15.16:0/3151218775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35241027e0 msgr2=0x7f352410acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:47.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.530+0000 7f352aed8700 1 --2- 172.21.15.16:0/3151218775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35241027e0 0x7f352410acd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3510009940 tx=0x7f3510009c50 comp rx=0 tx=0).stop 2024-04-22T15:53:47.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.530+0000 7f352aed8700 1 -- 172.21.15.16:0/3151218775 shutdown_connections 2024-04-22T15:53:47.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.530+0000 7f352aed8700 1 --2- 172.21.15.16:0/3151218775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35241027e0 0x7f352410acd0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.530+0000 7f352aed8700 1 --2- 172.21.15.16:0/3151218775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3524101ec0 0x7f35241022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.530+0000 7f352aed8700 1 -- 172.21.15.16:0/3151218775 >> 172.21.15.16:0/3151218775 conn(0x7f35240fb650 msgr2=0x7f35240fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:47.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.530+0000 7f352aed8700 1 -- 172.21.15.16:0/3151218775 shutdown_connections 2024-04-22T15:53:47.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.530+0000 7f352aed8700 1 -- 172.21.15.16:0/3151218775 wait complete. 2024-04-22T15:53:47.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.531+0000 7f352aed8700 1 Processor -- start 2024-04-22T15:53:47.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.531+0000 7f352aed8700 1 -- start start 2024-04-22T15:53:47.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.531+0000 7f352aed8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3524101ec0 0x7f352419c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:47.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.531+0000 7f352aed8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35241027e0 0x7f352419cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:47.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.531+0000 7f352aed8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352419d1a0 con 0x7f3524101ec0 2024-04-22T15:53:47.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.531+0000 7f352aed8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35241966e0 con 0x7f35241027e0 2024-04-22T15:53:47.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.531+0000 7f3528c74700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3524101ec0 0x7f352419c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:47.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.531+0000 7f3528c74700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3524101ec0 0x7f352419c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33304/0 (socket says 172.21.15.16:33304) 2024-04-22T15:53:47.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.531+0000 7f3528c74700 1 -- 172.21.15.16:0/2748139719 learned_addr learned my addr 172.21.15.16:0/2748139719 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:53:47.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.532+0000 7f3523fff700 1 --2- 172.21.15.16:0/2748139719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35241027e0 0x7f352419cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:47.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.532+0000 7f3523fff700 1 -- 172.21.15.16:0/2748139719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35241027e0 msgr2=0x7f352419cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-22T15:53:47.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.532+0000 7f3523fff700 1 -- 172.21.15.16:0/2748139719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35241027e0 msgr2=0x7f352419cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T15:53:47.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.532+0000 7f3523fff700 1 --2- 172.21.15.16:0/2748139719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35241027e0 0x7f352419cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T15:53:47.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.532+0000 7f3528c74700 1 -- 172.21.15.16:0/2748139719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35241027e0 msgr2=0x7f352419cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:47.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.532+0000 7f3523fff700 1 --2- 172.21.15.16:0/2748139719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35241027e0 0x7f352419cba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:53:47.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.532+0000 7f3528c74700 1 --2- 172.21.15.16:0/2748139719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35241027e0 0x7f352419cba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.532+0000 7f3528c74700 1 -- 172.21.15.16:0/2748139719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35100095f0 con 0x7f3524101ec0 2024-04-22T15:53:47.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.532+0000 7f3528c74700 1 --2- 172.21.15.16:0/2748139719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3524101ec0 0x7f352419c660 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f351800c930 tx=0x7f351800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:47.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.532+0000 7f3521ffb700 1 -- 172.21.15.16:0/2748139719 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3518007ad0 con 0x7f3524101ec0 2024-04-22T15:53:47.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.532+0000 7f3521ffb700 1 -- 172.21.15.16:0/2748139719 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3518007c30 con 0x7f3524101ec0 2024-04-22T15:53:47.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.532+0000 7f3521ffb700 1 -- 172.21.15.16:0/2748139719 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3518018610 con 0x7f3524101ec0 2024-04-22T15:53:47.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.532+0000 7f352aed8700 1 -- 172.21.15.16:0/2748139719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35241969c0 con 0x7f3524101ec0 2024-04-22T15:53:47.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.533+0000 7f352aed8700 1 -- 172.21.15.16:0/2748139719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3524196e90 con 0x7f3524101ec0 2024-04-22T15:53:47.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.533+0000 7f352aed8700 1 -- 172.21.15.16:0/2748139719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3524108360 con 0x7f3524101ec0 2024-04-22T15:53:47.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.534+0000 7f3521ffb700 1 -- 172.21.15.16:0/2748139719 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f351800f450 con 0x7f3524101ec0 2024-04-22T15:53:47.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.534+0000 7f3521ffb700 1 --2- 172.21.15.16:0/2748139719 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f351406ee00 0x7f35140712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:47.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.534+0000 7f3521ffb700 1 -- 172.21.15.16:0/2748139719 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f351808f180 con 0x7f3524101ec0 2024-04-22T15:53:47.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.535+0000 7f3523fff700 1 --2- 172.21.15.16:0/2748139719 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f351406ee00 0x7f35140712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:47.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.535+0000 7f3523fff700 1 --2- 172.21.15.16:0/2748139719 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f351406ee00 0x7f35140712c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3510009910 tx=0x7f3510005ea0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:47.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.541+0000 7f3521ffb700 1 -- 172.21.15.16:0/2748139719 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f351805b970 con 0x7f3524101ec0 2024-04-22T15:53:47.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 podman[83036]: 2024-04-22 15:53:47.380608711 +0000 UTC m=+0.385612377 container init f29fcb888801b2457b3c08120cd69643eb68a4b51ad1349541c63382ba1a1ee1 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi027, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, CEPH_POINT_RELEASE=) 2024-04-22T15:53:47.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 podman[83036]: 2024-04-22 15:53:47.384231036 +0000 UTC m=+0.389234702 container start f29fcb888801b2457b3c08120cd69643eb68a4b51ad1349541c63382ba1a1ee1 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi027, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, io.buildah.version=1.33.5) 2024-04-22T15:53:47.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T15:53:47.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev), process ceph-mon, pid 2 2024-04-22T15:53:47.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: pidfile_write: ignore empty --pid-file 2024-04-22T15:53:47.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: load: jerasure load: lrc 2024-04-22T15:53:47.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: RocksDB version: 7.9.2 2024-04-22T15:53:47.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Git sha 0 2024-04-22T15:53:47.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Compile date 2024-04-22 12:55:29 2024-04-22T15:53:47.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: DB SUMMARY 2024-04-22T15:53:47.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: DB Session ID: JDXYY8V22WZ2L5NCIKIK 2024-04-22T15:53:47.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: CURRENT file: CURRENT 2024-04-22T15:53:47.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T15:53:47.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: MANIFEST file: MANIFEST-000010 size: 1395 Bytes 2024-04-22T15:53:47.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi027/store.db dir, Total Num: 1, files: 000027.sst 2024-04-22T15:53:47.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi027/store.db: 000025.log size: 8325263 ; 2024-04-22T15:53:47.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.error_if_exists: 0 2024-04-22T15:53:47.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.create_if_missing: 0 2024-04-22T15:53:47.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.paranoid_checks: 1 2024-04-22T15:53:47.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-22T15:53:47.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T15:53:47.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-22T15:53:47.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.env: 0x5617ccad2cc0 2024-04-22T15:53:47.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.fs: PosixFileSystem 2024-04-22T15:53:47.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.info_log: 0x5617ce8f1aa0 2024-04-22T15:53:47.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T15:53:47.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.statistics: (nil) 2024-04-22T15:53:47.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.use_fsync: 0 2024-04-22T15:53:47.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_log_file_size: 0 2024-04-22T15:53:47.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T15:53:47.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T15:53:47.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T15:53:47.589 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T15:53:47.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.allow_fallocate: 1 2024-04-22T15:53:47.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T15:53:47.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T15:53:47.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.use_direct_reads: 0 2024-04-22T15:53:47.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T15:53:47.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T15:53:47.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.db_log_dir: 2024-04-22T15:53:47.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.wal_dir: 2024-04-22T15:53:47.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T15:53:47.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T15:53:47.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T15:53:47.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T15:53:47.590 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T15:53:47.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T15:53:47.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T15:53:47.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T15:53:47.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.write_buffer_manager: 0x5617ce8850e0 2024-04-22T15:53:47.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T15:53:47.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T15:53:47.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T15:53:47.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T15:53:47.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T15:53:47.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T15:53:47.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T15:53:47.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T15:53:47.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.unordered_write: 0 2024-04-22T15:53:47.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T15:53:47.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T15:53:47.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T15:53:47.591 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T15:53:47.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.row_cache: None 2024-04-22T15:53:47.592 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.wal_filter: None 2024-04-22T15:53:47.593 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T15:53:47.593 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T15:53:47.593 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.two_write_queues: 0 2024-04-22T15:53:47.593 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T15:53:47.593 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.wal_compression: 0 2024-04-22T15:53:47.593 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.atomic_flush: 0 2024-04-22T15:53:47.593 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T15:53:47.593 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T15:53:47.593 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T15:53:47.593 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.log_readahead_size: 0 2024-04-22T15:53:47.594 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T15:53:47.594 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T15:53:47.594 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T15:53:47.594 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T15:53:47.594 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T15:53:47.594 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T15:53:47.594 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-22T15:53:47.594 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_background_jobs: 2 2024-04-22T15:53:47.594 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_background_compactions: -1 2024-04-22T15:53:47.594 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_subcompactions: 1 2024-04-22T15:53:47.594 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T15:53:47.594 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T15:53:47.594 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T15:53:47.594 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T15:53:47.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T15:53:47.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T15:53:47.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T15:53:47.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T15:53:47.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_open_files: -1 2024-04-22T15:53:47.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T15:53:47.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T15:53:47.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T15:53:47.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T15:53:47.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_background_flushes: -1 2024-04-22T15:53:47.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Compression algorithms supported: 2024-04-22T15:53:47.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T15:53:47.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: kZSTD supported: 0 2024-04-22T15:53:47.595 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: kXpressCompression supported: 0 2024-04-22T15:53:47.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T15:53:47.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: kZlibCompression supported: 1 2024-04-22T15:53:47.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: kSnappyCompression supported: 1 2024-04-22T15:53:47.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: kLZ4Compression supported: 1 2024-04-22T15:53:47.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: kBZip2Compression supported: 0 2024-04-22T15:53:47.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T15:53:47.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-22T15:53:47.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi027/store.db/MANIFEST-000010 2024-04-22T15:53:47.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-22T15:53:47.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T15:53:47.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.merge_operator: 2024-04-22T15:53:47.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.compaction_filter: None 2024-04-22T15:53:47.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T15:53:47.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T15:53:47.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T15:53:47.596 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T15:53:47.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5617ce8f1c80) 2024-04-22T15:53:47.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: cache_index_and_filter_blocks: 1 2024-04-22T15:53:47.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T15:53:47.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T15:53:47.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: pin_top_level_index_and_filter: 1 2024-04-22T15:53:47.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: index_type: 0 2024-04-22T15:53:47.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: data_block_index_type: 0 2024-04-22T15:53:47.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: index_shortening: 1 2024-04-22T15:53:47.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T15:53:47.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: checksum: 4 2024-04-22T15:53:47.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: no_block_cache: 0 2024-04-22T15:53:47.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_cache: 0x5617ce8b71f0 2024-04-22T15:53:47.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_cache_name: BinnedLRUCache 2024-04-22T15:53:47.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_cache_options: 2024-04-22T15:53:47.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: capacity : 536870912 2024-04-22T15:53:47.597 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: num_shard_bits : 4 2024-04-22T15:53:47.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: strict_capacity_limit : 0 2024-04-22T15:53:47.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: high_pri_pool_ratio: 0.000 2024-04-22T15:53:47.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_cache_compressed: (nil) 2024-04-22T15:53:47.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: persistent_cache: (nil) 2024-04-22T15:53:47.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_size: 4096 2024-04-22T15:53:47.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_size_deviation: 10 2024-04-22T15:53:47.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_restart_interval: 16 2024-04-22T15:53:47.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: index_block_restart_interval: 1 2024-04-22T15:53:47.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: metadata_block_size: 4096 2024-04-22T15:53:47.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: partition_filters: 0 2024-04-22T15:53:47.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: use_delta_encoding: 1 2024-04-22T15:53:47.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: filter_policy: bloomfilter 2024-04-22T15:53:47.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: whole_key_filtering: 1 2024-04-22T15:53:47.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: verify_compression: 0 2024-04-22T15:53:47.598 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: read_amp_bytes_per_bit: 0 2024-04-22T15:53:47.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: format_version: 5 2024-04-22T15:53:47.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: enable_index_compression: 1 2024-04-22T15:53:47.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_align: 0 2024-04-22T15:53:47.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: max_auto_readahead_size: 262144 2024-04-22T15:53:47.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: prepopulate_block_cache: 0 2024-04-22T15:53:47.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: initial_auto_readahead_size: 8192 2024-04-22T15:53:47.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: num_file_reads_for_auto_readahead: 2 2024-04-22T15:53:47.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T15:53:47.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T15:53:47.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.compression: NoCompression 2024-04-22T15:53:47.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T15:53:47.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T15:53:47.599 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T15:53:47.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.num_levels: 7 2024-04-22T15:53:47.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T15:53:47.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T15:53:47.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T15:53:47.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T15:53:47.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T15:53:47.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T15:53:47.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T15:53:47.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T15:53:47.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T15:53:47.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T15:53:47.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-22T15:53:47.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-22T15:53:47.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T15:53:47.600 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T15:53:47.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T15:53:47.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T15:53:47.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T15:53:47.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-22T15:53:47.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T15:53:47.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T15:53:47.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-22T15:53:47.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T15:53:47.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T15:53:47.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T15:53:47.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T15:53:47.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T15:53:47.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T15:53:47.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T15:53:47.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T15:53:47.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T15:53:47.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T15:53:47.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T15:53:47.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T15:53:47.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T15:53:47.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T15:53:47.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T15:53:47.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T15:53:47.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T15:53:47.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-22T15:53:47.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.arena_block_size: 1048576 2024-04-22T15:53:47.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T15:53:47.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T15:53:47.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T15:53:47.602 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T15:53:47.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T15:53:47.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T15:53:47.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T15:53:47.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T15:53:47.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T15:53:47.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T15:53:47.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T15:53:47.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T15:53:47.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T15:53:47.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-22T15:53:47.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.inplace_update_support: 0 2024-04-22T15:53:47.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T15:53:47.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T15:53:47.603 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T15:53:47.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T15:53:47.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.bloom_locality: 0 2024-04-22T15:53:47.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.max_successive_merges: 0 2024-04-22T15:53:47.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T15:53:47.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T15:53:47.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T15:53:47.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T15:53:47.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.ttl: 2592000 2024-04-22T15:53:47.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T15:53:47.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-22T15:53:47.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-22T15:53:47.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.enable_blob_files: false 2024-04-22T15:53:47.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.min_blob_size: 0 2024-04-22T15:53:47.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T15:53:47.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T15:53:47.604 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T15:53:47.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T15:53:47.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-22T15:53:47.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-22T15:53:47.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.blob_file_starting_level: 0 2024-04-22T15:53:47.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-22T15:53:47.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-22T15:53:47.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi027/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 29, last_sequence is 12085, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 25 2024-04-22T15:53:47.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-04-22T15:53:47.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7cea9a7e-595e-4cf5-874d-5063d0cedd5d 2024-04-22T15:53:47.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713801227413952, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-04-22T15:53:47.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-04-22T15:53:47.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713801227435035, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 4931897, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12090, "largest_seqno": 16071, "table_properties": {"data_size": 4917815, "index_size": 8939, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 4101, "raw_key_size": 35954, "raw_average_key_size": 22, "raw_value_size": 4885824, "raw_average_value_size": 3021, "num_data_blocks": 404, "num_entries": 1617, "num_filter_entries": 1617, "num_deletions": 259, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713801227, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7cea9a7e-595e-4cf5-874d-5063d0cedd5d", "db_session_id": "JDXYY8V22WZ2L5NCIKIK", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-04-22T15:53:47.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713801227435145, "job": 1, "event": "recovery_finished"} 2024-04-22T15:53:47.605 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-04-22T15:53:47.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-22T15:53:47.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi027/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T15:53:47.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5617ce9cc000 2024-04-22T15:53:47.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: DB pointer 0x5617ce9b6000 2024-04-22T15:53:47.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-22T15:53:47.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-22T15:53:47.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ** DB Stats ** 2024-04-22T15:53:47.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T15:53:47.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T15:53:47.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T15:53:47.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T15:53:47.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T15:53:47.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T15:53:47.606 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T15:53:47.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-22T15:53:47.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ** Compaction Stats [default] ** 2024-04-22T15:53:47.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T15:53:47.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-22T15:53:47.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: L0 1/0 4.70 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 547.9 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-04-22T15:53:47.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: L6 1/0 5.87 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-22T15:53:47.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Sum 2/0 10.57 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 547.9 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-04-22T15:53:47.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 547.9 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-04-22T15:53:47.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-22T15:53:47.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ** Compaction Stats [default] ** 2024-04-22T15:53:47.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T15:53:47.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T15:53:47.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 547.9 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-04-22T15:53:47.607 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-22T15:53:47.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-22T15:53:47.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-22T15:53:47.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T15:53:47.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Flush(GB): cumulative 0.005, interval 0.005 2024-04-22T15:53:47.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T15:53:47.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T15:53:47.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T15:53:47.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T15:53:47.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Cumulative compaction: 0.00 GB write, 197.23 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T15:53:47.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Interval compaction: 0.00 GB write, 197.23 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T15:53:47.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T15:53:47.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Block cache BinnedLRUCache@0x5617ce8b71f0#2 capacity: 512.00 MB usage: 13.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-22T15:53:47.608 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,4.53 KB,0.000864267%) IndexBlock(1,9.08 KB,0.00173151%) Misc(1,0.00 KB,0%) 2024-04-22T15:53:47.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-22T15:53:47.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T15:53:47.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: starting mon.smithi027 rank 1 at public addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] at bind addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi027 fsid 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T15:53:47.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: mon.smithi027@-1(???) e2 preinit fsid 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T15:53:47.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: mon.smithi027@-1(???).mds e16 new map 2024-04-22T15:53:47.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: mon.smithi027@-1(???).mds e16 print_map 2024-04-22T15:53:47.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: e16 2024-04-22T15:53:47.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:53:47.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:53:47.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: legacy client fscid: 1 2024-04-22T15:53:47.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-22T15:53:47.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Filesystem 'cephfs' (1) 2024-04-22T15:53:47.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: fs_name cephfs 2024-04-22T15:53:47.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: epoch 16 2024-04-22T15:53:47.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T15:53:47.609 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: created 2024-04-22T15:46:27.774571+0000 2024-04-22T15:53:47.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: modified 2024-04-22T15:46:47.381709+0000 2024-04-22T15:53:47.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: tableserver 0 2024-04-22T15:53:47.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: root 0 2024-04-22T15:53:47.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: session_timeout 60 2024-04-22T15:53:47.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: session_autoclose 300 2024-04-22T15:53:47.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: max_file_size 1099511627776 2024-04-22T15:53:47.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: max_xattr_size 65536 2024-04-22T15:53:47.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: required_client_features {} 2024-04-22T15:53:47.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: last_failure 0 2024-04-22T15:53:47.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: last_failure_osd_epoch 46 2024-04-22T15:53:47.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:53:47.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: max_mds 1 2024-04-22T15:53:47.610 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: in 0 2024-04-22T15:53:47.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: up {0=24279} 2024-04-22T15:53:47.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: failed 2024-04-22T15:53:47.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: damaged 2024-04-22T15:53:47.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: stopped 2024-04-22T15:53:47.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: data_pools [3] 2024-04-22T15:53:47.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: metadata_pool 2 2024-04-22T15:53:47.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: inline_data enabled 2024-04-22T15:53:47.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: balancer 2024-04-22T15:53:47.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: bal_rank_mask -1 2024-04-22T15:53:47.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: standby_count_wanted 1 2024-04-22T15:53:47.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: qdb_cluster leader: 0 members: 2024-04-22T15:53:47.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: [mds.cephfs.smithi027.iobjds{0:24279} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.27:6824/4216277609,v1:172.21.15.27:6825/4216277609] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:53:47.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: [mds.cephfs.smithi016.pgvphu{0:14494} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.16:6828/1006073414,v1:172.21.15.16:6829/1006073414] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:53:47.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-22T15:53:47.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-22T15:53:47.611 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Standby daemons: 2024-04-22T15:53:47.612 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-22T15:53:47.612 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: [mds.cephfs.smithi016.fkkejm{-1:14504} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.16:6826/3484907755,v1:172.21.15.16:6827/3484907755] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:53:47.612 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: [mds.cephfs.smithi027.dgqyes{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.27:6826/476026355,v1:172.21.15.27:6827/476026355] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:53:47.612 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: mon.smithi027@-1(???).osd e49 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-22T15:53:47.612 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: mon.smithi027@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-04-22T15:53:47.612 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: mon.smithi027@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-04-22T15:53:47.612 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: mon.smithi027@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-04-22T15:53:47.612 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 ceph-mon[83054]: mon.smithi027@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-04-22T15:53:47.612 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 bash[83036]: f29fcb888801b2457b3c08120cd69643eb68a4b51ad1349541c63382ba1a1ee1 2024-04-22T15:53:47.612 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:47 smithi027 systemd[1]: Started Ceph mon.smithi027 for 67ab3a80-00be-11ef-bc93-c7b262605968. 2024-04-22T15:53:47.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.781+0000 7f352aed8700 1 -- 172.21.15.16:0/2748139719 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3524197d00 con 0x7f351406ee00 2024-04-22T15:53:47.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.783+0000 7f3521ffb700 1 -- 172.21.15.16:0/2748139719 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f3524197d00 con 0x7f351406ee00 2024-04-22T15:53:47.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.785+0000 7f352aed8700 1 -- 172.21.15.16:0/2748139719 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f351406ee00 msgr2=0x7f35140712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:47.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.785+0000 7f352aed8700 1 --2- 172.21.15.16:0/2748139719 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f351406ee00 0x7f35140712c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3510009910 tx=0x7f3510005ea0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.785+0000 7f352aed8700 1 -- 172.21.15.16:0/2748139719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3524101ec0 msgr2=0x7f352419c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:47.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.785+0000 7f352aed8700 1 --2- 172.21.15.16:0/2748139719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3524101ec0 0x7f352419c660 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f351800c930 tx=0x7f351800ccf0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.785+0000 7f352aed8700 1 -- 172.21.15.16:0/2748139719 shutdown_connections 2024-04-22T15:53:47.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.785+0000 7f352aed8700 1 --2- 172.21.15.16:0/2748139719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3524101ec0 0x7f352419c660 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.785+0000 7f352aed8700 1 --2- 172.21.15.16:0/2748139719 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f351406ee00 0x7f35140712c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.785+0000 7f352aed8700 1 --2- 172.21.15.16:0/2748139719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35241027e0 0x7f352419cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.785+0000 7f352aed8700 1 -- 172.21.15.16:0/2748139719 >> 172.21.15.16:0/2748139719 conn(0x7f35240fb650 msgr2=0x7f35241054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:47.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.785+0000 7f352aed8700 1 -- 172.21.15.16:0/2748139719 shutdown_connections 2024-04-22T15:53:47.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.785+0000 7f352aed8700 1 -- 172.21.15.16:0/2748139719 wait complete. 2024-04-22T15:53:47.799 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:53:47.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.897+0000 7f30521c1700 1 -- 172.21.15.16:0/3508535049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f304c101840 msgr2=0x7f304c105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:47.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.897+0000 7f30521c1700 1 --2- 172.21.15.16:0/3508535049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f304c101840 0x7f304c105890 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3034009910 tx=0x7f3034009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:47.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.897+0000 7f30521c1700 1 -- 172.21.15.16:0/3508535049 shutdown_connections 2024-04-22T15:53:47.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.897+0000 7f30521c1700 1 --2- 172.21.15.16:0/3508535049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f304c101840 0x7f304c105890 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.897+0000 7f30521c1700 1 --2- 172.21.15.16:0/3508535049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f304c100e90 0x7f304c101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.897+0000 7f30521c1700 1 -- 172.21.15.16:0/3508535049 >> 172.21.15.16:0/3508535049 conn(0x7f304c0fc740 msgr2=0x7f304c0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:47.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.897+0000 7f30521c1700 1 -- 172.21.15.16:0/3508535049 shutdown_connections 2024-04-22T15:53:47.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.897+0000 7f30521c1700 1 -- 172.21.15.16:0/3508535049 wait complete. 2024-04-22T15:53:47.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.897+0000 7f30521c1700 1 Processor -- start 2024-04-22T15:53:47.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.898+0000 7f30521c1700 1 -- start start 2024-04-22T15:53:47.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.898+0000 7f30521c1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f304c100e90 0x7f304c19c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:47.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.898+0000 7f30521c1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f304c101840 0x7f304c19cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:47.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.898+0000 7f30521c1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f304c19d1d0 con 0x7f304c101840 2024-04-22T15:53:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.898+0000 7f30521c1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f304c196680 con 0x7f304c100e90 2024-04-22T15:53:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.898+0000 7f304affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f304c101840 0x7f304c19cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.898+0000 7f304affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f304c101840 0x7f304c19cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33332/0 (socket says 172.21.15.16:33332) 2024-04-22T15:53:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.898+0000 7f304affd700 1 -- 172.21.15.16:0/2383256752 learned_addr learned my addr 172.21.15.16:0/2383256752 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:53:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.898+0000 7f304b7fe700 1 --2- 172.21.15.16:0/2383256752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f304c100e90 0x7f304c19c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.898+0000 7f304b7fe700 1 -- 172.21.15.16:0/2383256752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f304c100e90 msgr2=0x7f304c19c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-22T15:53:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.898+0000 7f304b7fe700 1 -- 172.21.15.16:0/2383256752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f304c100e90 msgr2=0x7f304c19c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T15:53:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.898+0000 7f304b7fe700 1 --2- 172.21.15.16:0/2383256752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f304c100e90 0x7f304c19c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T15:53:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.898+0000 7f304b7fe700 1 --2- 172.21.15.16:0/2383256752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f304c100e90 0x7f304c19c600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:53:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.898+0000 7f304affd700 1 -- 172.21.15.16:0/2383256752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f304c100e90 msgr2=0x7f304c19c600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T15:53:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.898+0000 7f304affd700 1 --2- 172.21.15.16:0/2383256752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f304c100e90 0x7f304c19c600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.898+0000 7f304affd700 1 -- 172.21.15.16:0/2383256752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30340095f0 con 0x7f304c101840 2024-04-22T15:53:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.899+0000 7f304affd700 1 --2- 172.21.15.16:0/2383256752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f304c101840 0x7f304c19cb40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f30340092e0 tx=0x7f303400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.899+0000 7f3048ff9700 1 -- 172.21.15.16:0/2383256752 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f303401d070 con 0x7f304c101840 2024-04-22T15:53:47.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.899+0000 7f3048ff9700 1 -- 172.21.15.16:0/2383256752 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3034005dc0 con 0x7f304c101840 2024-04-22T15:53:47.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.899+0000 7f30521c1700 1 -- 172.21.15.16:0/2383256752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f304c196900 con 0x7f304c101840 2024-04-22T15:53:47.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.899+0000 7f3048ff9700 1 -- 172.21.15.16:0/2383256752 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3034017400 con 0x7f304c101840 2024-04-22T15:53:47.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.899+0000 7f30521c1700 1 -- 172.21.15.16:0/2383256752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f304c196df0 con 0x7f304c101840 2024-04-22T15:53:47.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.900+0000 7f30521c1700 1 -- 172.21.15.16:0/2383256752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f304c1091a0 con 0x7f304c101840 2024-04-22T15:53:47.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.901+0000 7f3048ff9700 1 -- 172.21.15.16:0/2383256752 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f303400f460 con 0x7f304c101840 2024-04-22T15:53:47.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.901+0000 7f3048ff9700 1 --2- 172.21.15.16:0/2383256752 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f303806ee00 0x7f30380712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:47.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.901+0000 7f3048ff9700 1 -- 172.21.15.16:0/2383256752 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f3034090490 con 0x7f304c101840 2024-04-22T15:53:47.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.901+0000 7f304b7fe700 1 --2- 172.21.15.16:0/2383256752 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f303806ee00 0x7f30380712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:47.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.902+0000 7f304b7fe700 1 --2- 172.21.15.16:0/2383256752 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f303806ee00 0x7f30380712c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3044005950 tx=0x7f30440058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:47.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:47.906+0000 7f3048ff9700 1 -- 172.21.15.16:0/2383256752 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f303405cc80 con 0x7f304c101840 2024-04-22T15:53:48.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.136+0000 7f30521c1700 1 -- 172.21.15.16:0/2383256752 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f304c197060 con 0x7f303806ee00 2024-04-22T15:53:48.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.137+0000 7f3048ff9700 1 -- 172.21.15.16:0/2383256752 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f304c197060 con 0x7f303806ee00 2024-04-22T15:53:48.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.140+0000 7f30521c1700 1 -- 172.21.15.16:0/2383256752 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f303806ee00 msgr2=0x7f30380712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:48.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.140+0000 7f30521c1700 1 --2- 172.21.15.16:0/2383256752 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f303806ee00 0x7f30380712c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3044005950 tx=0x7f30440058e0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.140+0000 7f30521c1700 1 -- 172.21.15.16:0/2383256752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f304c101840 msgr2=0x7f304c19cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:48.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.140+0000 7f30521c1700 1 --2- 172.21.15.16:0/2383256752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f304c101840 0x7f304c19cb40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f30340092e0 tx=0x7f303400be20 comp rx=0 tx=0).stop 2024-04-22T15:53:48.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.140+0000 7f30521c1700 1 -- 172.21.15.16:0/2383256752 shutdown_connections 2024-04-22T15:53:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.140+0000 7f30521c1700 1 --2- 172.21.15.16:0/2383256752 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f303806ee00 0x7f30380712c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.140+0000 7f30521c1700 1 --2- 172.21.15.16:0/2383256752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f304c100e90 0x7f304c19c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.140+0000 7f30521c1700 1 --2- 172.21.15.16:0/2383256752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f304c101840 0x7f304c19cb40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.140+0000 7f30521c1700 1 -- 172.21.15.16:0/2383256752 >> 172.21.15.16:0/2383256752 conn(0x7f304c0fc740 msgr2=0x7f304c105100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.140+0000 7f30521c1700 1 -- 172.21.15.16:0/2383256752 shutdown_connections 2024-04-22T15:53:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.140+0000 7f30521c1700 1 -- 172.21.15.16:0/2383256752 wait complete. 2024-04-22T15:53:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.249+0000 7fe6000ba700 1 -- 172.21.15.16:0/1961722513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5f8102700 msgr2=0x7fe5f810abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.249+0000 7fe6000ba700 1 --2- 172.21.15.16:0/1961722513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5f8102700 0x7fe5f810abf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8009910 tx=0x7fe5e8009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.249+0000 7fe6000ba700 1 -- 172.21.15.16:0/1961722513 shutdown_connections 2024-04-22T15:53:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.249+0000 7fe6000ba700 1 --2- 172.21.15.16:0/1961722513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5f8102700 0x7fe5f810abf0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.249+0000 7fe6000ba700 1 --2- 172.21.15.16:0/1961722513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5f8101de0 0x7fe5f81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.249+0000 7fe6000ba700 1 -- 172.21.15.16:0/1961722513 >> 172.21.15.16:0/1961722513 conn(0x7fe5f80fb610 msgr2=0x7fe5f80fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.249+0000 7fe6000ba700 1 -- 172.21.15.16:0/1961722513 shutdown_connections 2024-04-22T15:53:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.249+0000 7fe6000ba700 1 -- 172.21.15.16:0/1961722513 wait complete. 2024-04-22T15:53:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.250+0000 7fe6000ba700 1 Processor -- start 2024-04-22T15:53:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.250+0000 7fe6000ba700 1 -- start start 2024-04-22T15:53:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.250+0000 7fe6000ba700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5f8101de0 0x7fe5f819c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.250+0000 7fe6000ba700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5f8102700 0x7fe5f819cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.250+0000 7fe6000ba700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f819d220 con 0x7fe5f8102700 2024-04-22T15:53:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.250+0000 7fe6000ba700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f8196720 con 0x7fe5f8101de0 2024-04-22T15:53:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.250+0000 7fe5fd655700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5f8102700 0x7fe5f819cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.250+0000 7fe5fde56700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5f8101de0 0x7fe5f819c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.250+0000 7fe5fd655700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5f8102700 0x7fe5f819cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33356/0 (socket says 172.21.15.16:33356) 2024-04-22T15:53:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.250+0000 7fe5fd655700 1 -- 172.21.15.16:0/4206884663 learned_addr learned my addr 172.21.15.16:0/4206884663 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:53:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.250+0000 7fe5fde56700 1 -- 172.21.15.16:0/4206884663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5f8101de0 msgr2=0x7fe5f819c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-22T15:53:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.250+0000 7fe5fde56700 1 -- 172.21.15.16:0/4206884663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5f8101de0 msgr2=0x7fe5f819c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T15:53:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.250+0000 7fe5fde56700 1 --2- 172.21.15.16:0/4206884663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5f8101de0 0x7fe5f819c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T15:53:48.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.250+0000 7fe5fde56700 1 --2- 172.21.15.16:0/4206884663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5f8101de0 0x7fe5f819c650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:53:48.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.250+0000 7fe5fd655700 1 -- 172.21.15.16:0/4206884663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5f8101de0 msgr2=0x7fe5f819c650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T15:53:48.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.250+0000 7fe5fd655700 1 --2- 172.21.15.16:0/4206884663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5f8101de0 0x7fe5f819c650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.250+0000 7fe5fd655700 1 -- 172.21.15.16:0/4206884663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e80095f0 con 0x7fe5f8102700 2024-04-22T15:53:48.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.251+0000 7fe5fd655700 1 --2- 172.21.15.16:0/4206884663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5f8102700 0x7fe5f819cb90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800bd70 tx=0x7fe5e800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:48.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.251+0000 7fe5eeffd700 1 -- 172.21.15.16:0/4206884663 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5e801d070 con 0x7fe5f8102700 2024-04-22T15:53:48.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.251+0000 7fe5eeffd700 1 -- 172.21.15.16:0/4206884663 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe5e800f460 con 0x7fe5f8102700 2024-04-22T15:53:48.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.251+0000 7fe6000ba700 1 -- 172.21.15.16:0/4206884663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5f81969a0 con 0x7fe5f8102700 2024-04-22T15:53:48.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.251+0000 7fe5eeffd700 1 -- 172.21.15.16:0/4206884663 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5e8017630 con 0x7fe5f8102700 2024-04-22T15:53:48.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.251+0000 7fe6000ba700 1 -- 172.21.15.16:0/4206884663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f8196e90 con 0x7fe5f8102700 2024-04-22T15:53:48.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.252+0000 7fe6000ba700 1 -- 172.21.15.16:0/4206884663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5f81082f0 con 0x7fe5f8102700 2024-04-22T15:53:48.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.252+0000 7fe5eeffd700 1 -- 172.21.15.16:0/4206884663 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fe5e800f5d0 con 0x7fe5f8102700 2024-04-22T15:53:48.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.253+0000 7fe5eeffd700 1 --2- 172.21.15.16:0/4206884663 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe5e406edf0 0x7fe5e40712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:48.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.253+0000 7fe5eeffd700 1 -- 172.21.15.16:0/4206884663 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fe5e8090470 con 0x7fe5f8102700 2024-04-22T15:53:48.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.253+0000 7fe5fde56700 1 --2- 172.21.15.16:0/4206884663 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe5e406edf0 0x7fe5e40712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:48.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.253+0000 7fe5fde56700 1 --2- 172.21.15.16:0/4206884663 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe5e406edf0 0x7fe5e40712b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4005920 tx=0x7fe5f400b2c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:48.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.262+0000 7fe5eeffd700 1 -- 172.21.15.16:0/4206884663 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe5e805cca0 con 0x7fe5f8102700 2024-04-22T15:53:48.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.483+0000 7fe6000ba700 1 -- 172.21.15.16:0/4206884663 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5f81970b0 con 0x7fe5e406edf0 2024-04-22T15:53:48.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.494+0000 7fe5eeffd700 1 -- 172.21.15.16:0/4206884663 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fe5f81970b0 con 0x7fe5e406edf0 2024-04-22T15:53:48.497 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:53:48.497 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.smithi016 smithi016 *:9093,9094 running (2m) 12s ago 12m 26.4M - 0.25.0 c8568f914cd2 ea0e0217d6f9 2024-04-22T15:53:48.497 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi016 smithi016 running (13m) 12s ago 13m 7994k - 18.2.1 5be31c24972a f8b627d3f41e 2024-04-22T15:53:48.497 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi027 smithi027 running (11m) 14s ago 11m 7977k - 18.2.1 5be31c24972a f00fd40b4220 2024-04-22T15:53:48.497 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi016 smithi016 running (13m) 12s ago 13m 6660k - 18.2.1 5be31c24972a 6b93db1fa0d9 2024-04-22T15:53:48.497 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi027 smithi027 running (11m) 14s ago 11m 6669k - 18.2.1 5be31c24972a d3bd439f30a0 2024-04-22T15:53:48.497 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.smithi016 smithi016 *:3000 running (99s) 12s ago 12m 75.9M - 9.4.12 2bacad6d85d8 388a3c9c1b3f 2024-04-22T15:53:48.497 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.fkkejm smithi016 running (7m) 12s ago 7m 15.5M - 18.2.1 5be31c24972a 4153ac94cea0 2024-04-22T15:53:48.497 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.pgvphu smithi016 running (7m) 12s ago 7m 92.0M - 18.2.1 5be31c24972a 93dc072068a2 2024-04-22T15:53:48.498 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.dgqyes smithi027 running (7m) 14s ago 7m 18.5M - 18.2.1 5be31c24972a 41a9220430fd 2024-04-22T15:53:48.498 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.iobjds smithi027 running (7m) 14s ago 7m 150M - 18.2.1 5be31c24972a c1c8d6770f02 2024-04-22T15:53:48.498 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi016.rnsnnr smithi016 *:8443,9283,8765 running (4m) 12s ago 14m 472M - 19.0.0-3223-g5883d494 3d983c26e278 09b4fe285bc4 2024-04-22T15:53:48.498 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi027.ukdgzs smithi027 *:8443,9283,8765 running (3m) 14s ago 11m 447M - 19.0.0-3223-g5883d494 3d983c26e278 7548da9b5eb9 2024-04-22T15:53:48.498 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi016 smithi016 running (27s) 12s ago 14m 29.0M 2048M 19.0.0-3223-g5883d494 3d983c26e278 4ae529a67806 2024-04-22T15:53:48.498 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi027 smithi027 starting - - - 2048M 2024-04-22T15:53:48.498 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi016 smithi016 *:9100 running (3m) 12s ago 12m 15.9M - 1.5.0 0da6a335fe13 35e46e4d2f3f 2024-04-22T15:53:48.498 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi027 smithi027 *:9100 running (2m) 14s ago 11m 15.6M - 1.5.0 0da6a335fe13 e682a7672e9b 2024-04-22T15:53:48.498 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (10m) 12s ago 10m 201M 1217M 18.2.1 5be31c24972a 3c6ebc4ea041 2024-04-22T15:53:48.499 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (10m) 12s ago 10m 205M 1217M 18.2.1 5be31c24972a 6aa2e6796784 2024-04-22T15:53:48.499 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (9m) 12s ago 9m 176M 1217M 18.2.1 5be31c24972a 3d9d310aa9eb 2024-04-22T15:53:48.499 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi027 running (9m) 14s ago 9m 250M 2240M 18.2.1 5be31c24972a cdf9b7aaf865 2024-04-22T15:53:48.499 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (9m) 14s ago 9m 207M 2240M 18.2.1 5be31c24972a 629397d49c76 2024-04-22T15:53:48.499 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi027 running (8m) 14s ago 8m 192M 2240M 18.2.1 5be31c24972a 23adcf675b04 2024-04-22T15:53:48.499 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.smithi016 smithi016 *:9095 running (2m) 12s ago 12m 67.3M - 2.43.0 a07b618ecd1d 373eaadee82d 2024-04-22T15:53:48.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.496+0000 7fe6000ba700 1 -- 172.21.15.16:0/4206884663 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe5e406edf0 msgr2=0x7fe5e40712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:48.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.496+0000 7fe6000ba700 1 --2- 172.21.15.16:0/4206884663 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe5e406edf0 0x7fe5e40712b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4005920 tx=0x7fe5f400b2c0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.496+0000 7fe6000ba700 1 -- 172.21.15.16:0/4206884663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5f8102700 msgr2=0x7fe5f819cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:48.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.496+0000 7fe6000ba700 1 --2- 172.21.15.16:0/4206884663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5f8102700 0x7fe5f819cb90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800bd70 tx=0x7fe5e800be50 comp rx=0 tx=0).stop 2024-04-22T15:53:48.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.496+0000 7fe6000ba700 1 -- 172.21.15.16:0/4206884663 shutdown_connections 2024-04-22T15:53:48.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.496+0000 7fe6000ba700 1 --2- 172.21.15.16:0/4206884663 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe5e406edf0 0x7fe5e40712b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.496+0000 7fe6000ba700 1 --2- 172.21.15.16:0/4206884663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5f8101de0 0x7fe5f819c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.496+0000 7fe6000ba700 1 --2- 172.21.15.16:0/4206884663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5f8102700 0x7fe5f819cb90 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.496+0000 7fe6000ba700 1 -- 172.21.15.16:0/4206884663 >> 172.21.15.16:0/4206884663 conn(0x7fe5f80fb610 msgr2=0x7fe5f8105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:48.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.496+0000 7fe6000ba700 1 -- 172.21.15.16:0/4206884663 shutdown_connections 2024-04-22T15:53:48.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.496+0000 7fe6000ba700 1 -- 172.21.15.16:0/4206884663 wait complete. 2024-04-22T15:53:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.602+0000 7f9910fec700 1 -- 172.21.15.16:0/1627838456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f990c101850 msgr2=0x7f990c1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.602+0000 7f9910fec700 1 --2- 172.21.15.16:0/1627838456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f990c101850 0x7f990c1058a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f98fc009910 tx=0x7f98fc009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.602+0000 7f9910fec700 1 -- 172.21.15.16:0/1627838456 shutdown_connections 2024-04-22T15:53:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.602+0000 7f9910fec700 1 --2- 172.21.15.16:0/1627838456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f990c101850 0x7f990c1058a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.602+0000 7f9910fec700 1 --2- 172.21.15.16:0/1627838456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f990c100ea0 0x7f990c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.602+0000 7f9910fec700 1 -- 172.21.15.16:0/1627838456 >> 172.21.15.16:0/1627838456 conn(0x7f990c0fc750 msgr2=0x7f990c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.602+0000 7f9910fec700 1 -- 172.21.15.16:0/1627838456 shutdown_connections 2024-04-22T15:53:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.602+0000 7f9910fec700 1 -- 172.21.15.16:0/1627838456 wait complete. 2024-04-22T15:53:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.603+0000 7f9910fec700 1 Processor -- start 2024-04-22T15:53:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.603+0000 7f9910fec700 1 -- start start 2024-04-22T15:53:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.603+0000 7f9910fec700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f990c100ea0 0x7f990c073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.603+0000 7f9910fec700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f990c101850 0x7f990c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.603+0000 7f9910fec700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990c073b10 con 0x7f990c101850 2024-04-22T15:53:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.603+0000 7f9910fec700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990c073c50 con 0x7f990c100ea0 2024-04-22T15:53:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.603+0000 7f9909d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f990c101850 0x7f990c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.603+0000 7f9909d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f990c101850 0x7f990c0735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33368/0 (socket says 172.21.15.16:33368) 2024-04-22T15:53:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.603+0000 7f9909d9b700 1 -- 172.21.15.16:0/2342639461 learned_addr learned my addr 172.21.15.16:0/2342639461 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:53:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.603+0000 7f990a59c700 1 --2- 172.21.15.16:0/2342639461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f990c100ea0 0x7f990c073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.603+0000 7f990a59c700 1 -- 172.21.15.16:0/2342639461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f990c100ea0 msgr2=0x7f990c073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-22T15:53:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.603+0000 7f990a59c700 1 -- 172.21.15.16:0/2342639461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f990c100ea0 msgr2=0x7f990c073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T15:53:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.603+0000 7f990a59c700 1 --2- 172.21.15.16:0/2342639461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f990c100ea0 0x7f990c073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T15:53:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.603+0000 7f990a59c700 1 --2- 172.21.15.16:0/2342639461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f990c100ea0 0x7f990c073090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:53:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.603+0000 7f9909d9b700 1 -- 172.21.15.16:0/2342639461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f990c100ea0 msgr2=0x7f990c073090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T15:53:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.603+0000 7f9909d9b700 1 --2- 172.21.15.16:0/2342639461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f990c100ea0 0x7f990c073090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.603+0000 7f9909d9b700 1 -- 172.21.15.16:0/2342639461 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc0095f0 con 0x7f990c101850 2024-04-22T15:53:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.603+0000 7f9909d9b700 1 --2- 172.21.15.16:0/2342639461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f990c101850 0x7f990c0735d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0092e0 tx=0x7f98fc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.604+0000 7f98fb7fe700 1 -- 172.21.15.16:0/2342639461 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98fc01d070 con 0x7f990c101850 2024-04-22T15:53:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.604+0000 7f98fb7fe700 1 -- 172.21.15.16:0/2342639461 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f98fc00f460 con 0x7f990c101850 2024-04-22T15:53:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.604+0000 7f9910fec700 1 -- 172.21.15.16:0/2342639461 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f990c073ed0 con 0x7f990c101850 2024-04-22T15:53:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.604+0000 7f98fb7fe700 1 -- 172.21.15.16:0/2342639461 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98fc0175f0 con 0x7f990c101850 2024-04-22T15:53:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.604+0000 7f9910fec700 1 -- 172.21.15.16:0/2342639461 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f990c1a2ec0 con 0x7f990c101850 2024-04-22T15:53:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.605+0000 7f9910fec700 1 -- 172.21.15.16:0/2342639461 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f990c1091d0 con 0x7f990c101850 2024-04-22T15:53:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.605+0000 7f98fb7fe700 1 -- 172.21.15.16:0/2342639461 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f98fc00f5d0 con 0x7f990c101850 2024-04-22T15:53:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.605+0000 7f98fb7fe700 1 --2- 172.21.15.16:0/2342639461 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f98f406ee00 0x7f98f40712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.606+0000 7f98fb7fe700 1 -- 172.21.15.16:0/2342639461 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f98fc090580 con 0x7f990c101850 2024-04-22T15:53:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.606+0000 7f990a59c700 1 --2- 172.21.15.16:0/2342639461 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f98f406ee00 0x7f98f40712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.606+0000 7f990a59c700 1 --2- 172.21.15.16:0/2342639461 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f98f406ee00 0x7f98f40712c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9900006f50 tx=0x7f9900008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.610+0000 7f98fb7fe700 1 -- 172.21.15.16:0/2342639461 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f98fc0591c0 con 0x7f990c101850 2024-04-22T15:53:48.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.788+0000 7f98fb7fe700 1 -- 172.21.15.16:0/2342639461 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98fc0591c0 con 0x7f990c101850 2024-04-22T15:53:48.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.914+0000 7f9910fec700 1 -- 172.21.15.16:0/2342639461 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f990c04f2e0 con 0x7f990c101850 2024-04-22T15:53:48.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.915+0000 7f98fb7fe700 1 -- 172.21.15.16:0/2342639461 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+688 (secure 0 0 0) 0x7f98fc05c880 con 0x7f990c101850 2024-04-22T15:53:48.918 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:53:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-22T15:53:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:53:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:53:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-22T15:53:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:53:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:53:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-22T15:53:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-22T15:53:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:53:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-22T15:53:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T15:53:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:53:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-22T15:53:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 10, 2024-04-22T15:53:48.920 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:53:48.920 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-22T15:53:48.920 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:53:48.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.917+0000 7f9910fec700 1 -- 172.21.15.16:0/2342639461 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f98f406ee00 msgr2=0x7f98f40712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:48.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.917+0000 7f9910fec700 1 --2- 172.21.15.16:0/2342639461 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f98f406ee00 0x7f98f40712c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9900006f50 tx=0x7f9900008040 comp rx=0 tx=0).stop 2024-04-22T15:53:48.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.917+0000 7f9910fec700 1 -- 172.21.15.16:0/2342639461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f990c101850 msgr2=0x7f990c0735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:48.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.917+0000 7f9910fec700 1 --2- 172.21.15.16:0/2342639461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f990c101850 0x7f990c0735d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0092e0 tx=0x7f98fc00be20 comp rx=0 tx=0).stop 2024-04-22T15:53:48.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.917+0000 7f9910fec700 1 -- 172.21.15.16:0/2342639461 shutdown_connections 2024-04-22T15:53:48.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.917+0000 7f9910fec700 1 --2- 172.21.15.16:0/2342639461 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f98f406ee00 0x7f98f40712c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.917+0000 7f9910fec700 1 --2- 172.21.15.16:0/2342639461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f990c100ea0 0x7f990c073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.917+0000 7f9910fec700 1 --2- 172.21.15.16:0/2342639461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f990c101850 0x7f990c0735d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:48.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.917+0000 7f9910fec700 1 -- 172.21.15.16:0/2342639461 >> 172.21.15.16:0/2342639461 conn(0x7f990c0fc750 msgr2=0x7f990c0fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:48.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.918+0000 7f9910fec700 1 -- 172.21.15.16:0/2342639461 shutdown_connections 2024-04-22T15:53:48.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:48.918+0000 7f9910fec700 1 -- 172.21.15.16:0/2342639461 wait complete. 2024-04-22T15:53:49.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.024+0000 7fa34ca05700 1 -- 172.21.15.16:0/4121705372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa348068580 msgr2=0x7fa3480689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:49.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.024+0000 7fa34ca05700 1 --2- 172.21.15.16:0/4121705372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa348068580 0x7fa3480689e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa33c009910 tx=0x7fa33c009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:49.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.024+0000 7fa34ca05700 1 -- 172.21.15.16:0/4121705372 shutdown_connections 2024-04-22T15:53:49.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.024+0000 7fa34ca05700 1 --2- 172.21.15.16:0/4121705372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa348068580 0x7fa3480689e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.024+0000 7fa34ca05700 1 --2- 172.21.15.16:0/4121705372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa348106cd0 0x7fa3481070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.024+0000 7fa34ca05700 1 -- 172.21.15.16:0/4121705372 >> 172.21.15.16:0/4121705372 conn(0x7fa348075670 msgr2=0x7fa348075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:49.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.024+0000 7fa34ca05700 1 -- 172.21.15.16:0/4121705372 shutdown_connections 2024-04-22T15:53:49.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.024+0000 7fa34ca05700 1 -- 172.21.15.16:0/4121705372 wait complete. 2024-04-22T15:53:49.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.025+0000 7fa34ca05700 1 Processor -- start 2024-04-22T15:53:49.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.025+0000 7fa34ca05700 1 -- start start 2024-04-22T15:53:49.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.025+0000 7fa34ca05700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa348068580 0x7fa34819c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:49.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.025+0000 7fa34ca05700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa348106cd0 0x7fa34819cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:49.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.025+0000 7fa34ca05700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34819d1f0 con 0x7fa348106cd0 2024-04-22T15:53:49.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.025+0000 7fa34ca05700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3481966a0 con 0x7fa348068580 2024-04-22T15:53:49.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.025+0000 7fa345d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa348106cd0 0x7fa34819cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:49.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.025+0000 7fa345d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa348106cd0 0x7fa34819cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33388/0 (socket says 172.21.15.16:33388) 2024-04-22T15:53:49.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.025+0000 7fa345d9b700 1 -- 172.21.15.16:0/1198761260 learned_addr learned my addr 172.21.15.16:0/1198761260 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:53:49.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.025+0000 7fa34659c700 1 --2- 172.21.15.16:0/1198761260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa348068580 0x7fa34819c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:49.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.025+0000 7fa34659c700 1 -- 172.21.15.16:0/1198761260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa348068580 msgr2=0x7fa34819c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-22T15:53:49.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.025+0000 7fa34659c700 1 -- 172.21.15.16:0/1198761260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa348068580 msgr2=0x7fa34819c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T15:53:49.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.025+0000 7fa34659c700 1 --2- 172.21.15.16:0/1198761260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa348068580 0x7fa34819c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T15:53:49.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.025+0000 7fa34659c700 1 --2- 172.21.15.16:0/1198761260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa348068580 0x7fa34819c620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:53:49.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.026+0000 7fa345d9b700 1 -- 172.21.15.16:0/1198761260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa348068580 msgr2=0x7fa34819c620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T15:53:49.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.026+0000 7fa345d9b700 1 --2- 172.21.15.16:0/1198761260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa348068580 0x7fa34819c620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.026+0000 7fa345d9b700 1 -- 172.21.15.16:0/1198761260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33c0095f0 con 0x7fa348106cd0 2024-04-22T15:53:49.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.026+0000 7fa345d9b700 1 --2- 172.21.15.16:0/1198761260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa348106cd0 0x7fa34819cb60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa33c0092e0 tx=0x7fa33c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:49.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.026+0000 7fa3377fe700 1 -- 172.21.15.16:0/1198761260 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa33c01d070 con 0x7fa348106cd0 2024-04-22T15:53:49.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.026+0000 7fa3377fe700 1 -- 172.21.15.16:0/1198761260 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa33c005e00 con 0x7fa348106cd0 2024-04-22T15:53:49.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.026+0000 7fa34ca05700 1 -- 172.21.15.16:0/1198761260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa348196920 con 0x7fa348106cd0 2024-04-22T15:53:49.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.026+0000 7fa3377fe700 1 -- 172.21.15.16:0/1198761260 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa33c017400 con 0x7fa348106cd0 2024-04-22T15:53:49.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.026+0000 7fa34ca05700 1 -- 172.21.15.16:0/1198761260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa348196e10 con 0x7fa348106cd0 2024-04-22T15:53:49.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.027+0000 7fa34ca05700 1 -- 172.21.15.16:0/1198761260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3481091d0 con 0x7fa348106cd0 2024-04-22T15:53:49.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.027+0000 7fa3377fe700 1 -- 172.21.15.16:0/1198761260 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fa33c00f460 con 0x7fa348106cd0 2024-04-22T15:53:49.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.028+0000 7fa3377fe700 1 --2- 172.21.15.16:0/1198761260 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fa33006edf0 0x7fa3300712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:49.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.028+0000 7fa3377fe700 1 -- 172.21.15.16:0/1198761260 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa33c090550 con 0x7fa348106cd0 2024-04-22T15:53:49.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.028+0000 7fa34659c700 1 --2- 172.21.15.16:0/1198761260 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fa33006edf0 0x7fa3300712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:49.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.028+0000 7fa34659c700 1 --2- 172.21.15.16:0/1198761260 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fa33006edf0 0x7fa3300712b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa338005950 tx=0x7fa3380058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:49.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.033+0000 7fa3377fe700 1 -- 172.21.15.16:0/1198761260 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa33c05ccc0 con 0x7fa348106cd0 2024-04-22T15:53:49.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:48 smithi027 ceph-mon[83054]: pgmap v10: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 18 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-22T15:53:49.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:48 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:53:49.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:48 smithi016 ceph-mon[110740]: pgmap v10: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 18 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-22T15:53:49.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:48 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:53:49.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.287+0000 7fa34ca05700 1 -- 172.21.15.16:0/1198761260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa348197be0 con 0x7fa348106cd0 2024-04-22T15:53:49.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.287+0000 7fa3377fe700 1 -- 172.21.15.16:0/1198761260 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1902 (secure 0 0 0) 0x7fa33c05c850 con 0x7fa348106cd0 2024-04-22T15:53:49.291 INFO:teuthology.orchestra.run.smithi016.stdout:e16 2024-04-22T15:53:49.291 INFO:teuthology.orchestra.run.smithi016.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:53:49.291 INFO:teuthology.orchestra.run.smithi016.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:53:49.291 INFO:teuthology.orchestra.run.smithi016.stdout:legacy client fscid: 1 2024-04-22T15:53:49.291 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:53:49.291 INFO:teuthology.orchestra.run.smithi016.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:53:49.291 INFO:teuthology.orchestra.run.smithi016.stdout:fs_name cephfs 2024-04-22T15:53:49.291 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 16 2024-04-22T15:53:49.291 INFO:teuthology.orchestra.run.smithi016.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T15:53:49.291 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-04-22T15:46:27.774571+0000 2024-04-22T15:53:49.291 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-04-22T15:46:47.381709+0000 2024-04-22T15:53:49.291 INFO:teuthology.orchestra.run.smithi016.stdout:tableserver 0 2024-04-22T15:53:49.291 INFO:teuthology.orchestra.run.smithi016.stdout:root 0 2024-04-22T15:53:49.291 INFO:teuthology.orchestra.run.smithi016.stdout:session_timeout 60 2024-04-22T15:53:49.292 INFO:teuthology.orchestra.run.smithi016.stdout:session_autoclose 300 2024-04-22T15:53:49.292 INFO:teuthology.orchestra.run.smithi016.stdout:max_file_size 1099511627776 2024-04-22T15:53:49.292 INFO:teuthology.orchestra.run.smithi016.stdout:max_xattr_size 65536 2024-04-22T15:53:49.292 INFO:teuthology.orchestra.run.smithi016.stdout:required_client_features {} 2024-04-22T15:53:49.292 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure 0 2024-04-22T15:53:49.292 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure_osd_epoch 46 2024-04-22T15:53:49.292 INFO:teuthology.orchestra.run.smithi016.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:53:49.292 INFO:teuthology.orchestra.run.smithi016.stdout:max_mds 1 2024-04-22T15:53:49.292 INFO:teuthology.orchestra.run.smithi016.stdout:in 0 2024-04-22T15:53:49.292 INFO:teuthology.orchestra.run.smithi016.stdout:up {0=24279} 2024-04-22T15:53:49.292 INFO:teuthology.orchestra.run.smithi016.stdout:failed 2024-04-22T15:53:49.292 INFO:teuthology.orchestra.run.smithi016.stdout:damaged 2024-04-22T15:53:49.292 INFO:teuthology.orchestra.run.smithi016.stdout:stopped 2024-04-22T15:53:49.292 INFO:teuthology.orchestra.run.smithi016.stdout:data_pools [3] 2024-04-22T15:53:49.293 INFO:teuthology.orchestra.run.smithi016.stdout:metadata_pool 2 2024-04-22T15:53:49.293 INFO:teuthology.orchestra.run.smithi016.stdout:inline_data enabled 2024-04-22T15:53:49.293 INFO:teuthology.orchestra.run.smithi016.stdout:balancer 2024-04-22T15:53:49.293 INFO:teuthology.orchestra.run.smithi016.stdout:bal_rank_mask -1 2024-04-22T15:53:49.293 INFO:teuthology.orchestra.run.smithi016.stdout:standby_count_wanted 1 2024-04-22T15:53:49.293 INFO:teuthology.orchestra.run.smithi016.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:53:49.294 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.iobjds{0:24279} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.27:6824/4216277609,v1:172.21.15.27:6825/4216277609] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:53:49.294 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.pgvphu{0:14494} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.16:6828/1006073414,v1:172.21.15.16:6829/1006073414] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:53:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:53:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:53:49.294 INFO:teuthology.orchestra.run.smithi016.stdout:Standby daemons: 2024-04-22T15:53:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:53:49.294 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.fkkejm{-1:14504} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.16:6826/3484907755,v1:172.21.15.16:6827/3484907755] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:53:49.295 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.dgqyes{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.27:6826/476026355,v1:172.21.15.27:6827/476026355] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:53:49.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.290+0000 7fa34ca05700 1 -- 172.21.15.16:0/1198761260 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fa33006edf0 msgr2=0x7fa3300712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:49.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.290+0000 7fa34ca05700 1 --2- 172.21.15.16:0/1198761260 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fa33006edf0 0x7fa3300712b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa338005950 tx=0x7fa3380058e0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.290+0000 7fa34ca05700 1 -- 172.21.15.16:0/1198761260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa348106cd0 msgr2=0x7fa34819cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:49.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.290+0000 7fa34ca05700 1 --2- 172.21.15.16:0/1198761260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa348106cd0 0x7fa34819cb60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa33c0092e0 tx=0x7fa33c00be20 comp rx=0 tx=0).stop 2024-04-22T15:53:49.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.290+0000 7fa34ca05700 1 -- 172.21.15.16:0/1198761260 shutdown_connections 2024-04-22T15:53:49.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.290+0000 7fa34ca05700 1 --2- 172.21.15.16:0/1198761260 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fa33006edf0 0x7fa3300712b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.290+0000 7fa34ca05700 1 --2- 172.21.15.16:0/1198761260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa348068580 0x7fa34819c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.290+0000 7fa34ca05700 1 --2- 172.21.15.16:0/1198761260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa348106cd0 0x7fa34819cb60 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.290+0000 7fa34ca05700 1 -- 172.21.15.16:0/1198761260 >> 172.21.15.16:0/1198761260 conn(0x7fa348075670 msgr2=0x7fa3480feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:49.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.290+0000 7fa34ca05700 1 -- 172.21.15.16:0/1198761260 shutdown_connections 2024-04-22T15:53:49.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.290+0000 7fa34ca05700 1 -- 172.21.15.16:0/1198761260 wait complete. 2024-04-22T15:53:49.297 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 16 2024-04-22T15:53:49.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.396+0000 7f9806227700 1 -- 172.21.15.16:0/3477679936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9800100ea0 msgr2=0x7f9800101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:49.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.396+0000 7f9806227700 1 --2- 172.21.15.16:0/3477679936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9800100ea0 0x7f9800101280 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f97e8009910 tx=0x7f97e8009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:49.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.397+0000 7f9806227700 1 -- 172.21.15.16:0/3477679936 shutdown_connections 2024-04-22T15:53:49.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.397+0000 7f9806227700 1 --2- 172.21.15.16:0/3477679936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9800101850 0x7f98001058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.397+0000 7f9806227700 1 --2- 172.21.15.16:0/3477679936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9800100ea0 0x7f9800101280 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.397+0000 7f9806227700 1 -- 172.21.15.16:0/3477679936 >> 172.21.15.16:0/3477679936 conn(0x7f98000fc750 msgr2=0x7f98000feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:49.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.397+0000 7f9806227700 1 -- 172.21.15.16:0/3477679936 shutdown_connections 2024-04-22T15:53:49.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.397+0000 7f9806227700 1 -- 172.21.15.16:0/3477679936 wait complete. 2024-04-22T15:53:49.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.397+0000 7f9806227700 1 Processor -- start 2024-04-22T15:53:49.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.397+0000 7f9806227700 1 -- start start 2024-04-22T15:53:49.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.398+0000 7f9806227700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9800100ea0 0x7f980019c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:49.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.398+0000 7f9806227700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9800101850 0x7f980019cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:49.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.398+0000 7f9806227700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980019d2c0 con 0x7f9800100ea0 2024-04-22T15:53:49.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.398+0000 7f9806227700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9800196770 con 0x7f9800101850 2024-04-22T15:53:49.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.398+0000 7f97ff7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9800100ea0 0x7f980019c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:49.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.398+0000 7f97ff7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9800100ea0 0x7f980019c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33406/0 (socket says 172.21.15.16:33406) 2024-04-22T15:53:49.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.398+0000 7f97ff7fe700 1 -- 172.21.15.16:0/2145015335 learned_addr learned my addr 172.21.15.16:0/2145015335 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:53:49.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.398+0000 7f97feffd700 1 --2- 172.21.15.16:0/2145015335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9800101850 0x7f980019cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:49.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.398+0000 7f97feffd700 1 -- 172.21.15.16:0/2145015335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9800101850 msgr2=0x7f980019cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-22T15:53:49.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.398+0000 7f97feffd700 1 -- 172.21.15.16:0/2145015335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9800101850 msgr2=0x7f980019cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T15:53:49.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.398+0000 7f97feffd700 1 --2- 172.21.15.16:0/2145015335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9800101850 0x7f980019cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T15:53:49.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.398+0000 7f97ff7fe700 1 -- 172.21.15.16:0/2145015335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9800101850 msgr2=0x7f980019cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:49.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.398+0000 7f97feffd700 1 --2- 172.21.15.16:0/2145015335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9800101850 0x7f980019cc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:53:49.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.398+0000 7f97ff7fe700 1 --2- 172.21.15.16:0/2145015335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9800101850 0x7f980019cc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.398+0000 7f97ff7fe700 1 -- 172.21.15.16:0/2145015335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e80095f0 con 0x7f9800100ea0 2024-04-22T15:53:49.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.398+0000 7f97ff7fe700 1 --2- 172.21.15.16:0/2145015335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9800100ea0 0x7f980019c6f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f97e8006010 tx=0x7f97e800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:49.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.398+0000 7f97fcff9700 1 -- 172.21.15.16:0/2145015335 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97e801d070 con 0x7f9800100ea0 2024-04-22T15:53:49.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.398+0000 7f9806227700 1 -- 172.21.15.16:0/2145015335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98001969f0 con 0x7f9800100ea0 2024-04-22T15:53:49.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.399+0000 7f97fcff9700 1 -- 172.21.15.16:0/2145015335 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f97e800f460 con 0x7f9800100ea0 2024-04-22T15:53:49.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.399+0000 7f97fcff9700 1 -- 172.21.15.16:0/2145015335 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97e8017630 con 0x7f9800100ea0 2024-04-22T15:53:49.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.399+0000 7f9806227700 1 -- 172.21.15.16:0/2145015335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9800196ee0 con 0x7f9800100ea0 2024-04-22T15:53:49.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.399+0000 7f9806227700 1 -- 172.21.15.16:0/2145015335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9800109240 con 0x7f9800100ea0 2024-04-22T15:53:49.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.400+0000 7f97fcff9700 1 -- 172.21.15.16:0/2145015335 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f97e800f5d0 con 0x7f9800100ea0 2024-04-22T15:53:49.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.400+0000 7f97fcff9700 1 --2- 172.21.15.16:0/2145015335 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f97ec06ee00 0x7f97ec0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:49.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.401+0000 7f97fcff9700 1 -- 172.21.15.16:0/2145015335 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f97e80904d0 con 0x7f9800100ea0 2024-04-22T15:53:49.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.401+0000 7f97feffd700 1 --2- 172.21.15.16:0/2145015335 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f97ec06ee00 0x7f97ec0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:49.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.401+0000 7f97feffd700 1 --2- 172.21.15.16:0/2145015335 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f97ec06ee00 0x7f97ec0712c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f97f0005950 tx=0x7f97f00058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:49.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.405+0000 7f97fcff9700 1 -- 172.21.15.16:0/2145015335 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f97e805cc40 con 0x7f9800100ea0 2024-04-22T15:53:49.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.636+0000 7f9806227700 1 -- 172.21.15.16:0/2145015335 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9800197db0 con 0x7f97ec06ee00 2024-04-22T15:53:49.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.637+0000 7f97fcff9700 1 -- 172.21.15.16:0/2145015335 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f9800197db0 con 0x7f97ec06ee00 2024-04-22T15:53:49.640 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:53:49.640 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:53:49.640 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-22T15:53:49.640 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:53:49.640 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-22T15:53:49.640 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-04-22T15:53:49.640 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-22T15:53:49.641 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "3/23 daemons upgraded", 2024-04-22T15:53:49.641 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading mon daemons", 2024-04-22T15:53:49.641 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-22T15:53:49.641 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:53:49.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.639+0000 7f9806227700 1 -- 172.21.15.16:0/2145015335 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f97ec06ee00 msgr2=0x7f97ec0712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:49.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.639+0000 7f9806227700 1 --2- 172.21.15.16:0/2145015335 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f97ec06ee00 0x7f97ec0712c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f97f0005950 tx=0x7f97f00058e0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.639+0000 7f9806227700 1 -- 172.21.15.16:0/2145015335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9800100ea0 msgr2=0x7f980019c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:49.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.639+0000 7f9806227700 1 --2- 172.21.15.16:0/2145015335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9800100ea0 0x7f980019c6f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f97e8006010 tx=0x7f97e800be20 comp rx=0 tx=0).stop 2024-04-22T15:53:49.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.639+0000 7f9806227700 1 -- 172.21.15.16:0/2145015335 shutdown_connections 2024-04-22T15:53:49.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.639+0000 7f9806227700 1 --2- 172.21.15.16:0/2145015335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9800100ea0 0x7f980019c6f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.639+0000 7f9806227700 1 --2- 172.21.15.16:0/2145015335 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f97ec06ee00 0x7f97ec0712c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.639+0000 7f9806227700 1 --2- 172.21.15.16:0/2145015335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9800101850 0x7f980019cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.639+0000 7f9806227700 1 -- 172.21.15.16:0/2145015335 >> 172.21.15.16:0/2145015335 conn(0x7f98000fc750 msgr2=0x7f98000fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:49.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.639+0000 7f9806227700 1 -- 172.21.15.16:0/2145015335 shutdown_connections 2024-04-22T15:53:49.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.639+0000 7f9806227700 1 -- 172.21.15.16:0/2145015335 wait complete. 2024-04-22T15:53:49.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.745+0000 7f1d3147d700 1 -- 172.21.15.16:0/395478513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d2c100ea0 msgr2=0x7f1d2c101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:49.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.745+0000 7f1d3147d700 1 --2- 172.21.15.16:0/395478513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d2c100ea0 0x7f1d2c101280 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c009910 tx=0x7f1d1c009c20 comp rx=0 tx=0).stop 2024-04-22T15:53:49.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.745+0000 7f1d3147d700 1 -- 172.21.15.16:0/395478513 shutdown_connections 2024-04-22T15:53:49.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.746+0000 7f1d3147d700 1 --2- 172.21.15.16:0/395478513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d2c101850 0x7f1d2c1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.746+0000 7f1d3147d700 1 --2- 172.21.15.16:0/395478513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d2c100ea0 0x7f1d2c101280 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.746+0000 7f1d3147d700 1 -- 172.21.15.16:0/395478513 >> 172.21.15.16:0/395478513 conn(0x7f1d2c0fc750 msgr2=0x7f1d2c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:49.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.746+0000 7f1d3147d700 1 -- 172.21.15.16:0/395478513 shutdown_connections 2024-04-22T15:53:49.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.746+0000 7f1d3147d700 1 -- 172.21.15.16:0/395478513 wait complete. 2024-04-22T15:53:49.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.746+0000 7f1d3147d700 1 Processor -- start 2024-04-22T15:53:49.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.746+0000 7f1d3147d700 1 -- start start 2024-04-22T15:53:49.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.747+0000 7f1d3147d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d2c100ea0 0x7f1d2c073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:49.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.747+0000 7f1d3147d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d2c101850 0x7f1d2c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:49.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.747+0000 7f1d3147d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2c073b10 con 0x7f1d2c101850 2024-04-22T15:53:49.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.747+0000 7f1d3147d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2c073c50 con 0x7f1d2c100ea0 2024-04-22T15:53:49.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.747+0000 7f1d2a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d2c101850 0x7f1d2c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:49.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.747+0000 7f1d2a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d2c101850 0x7f1d2c0735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33416/0 (socket says 172.21.15.16:33416) 2024-04-22T15:53:49.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.747+0000 7f1d2a7fc700 1 -- 172.21.15.16:0/1509564414 learned_addr learned my addr 172.21.15.16:0/1509564414 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:53:49.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.747+0000 7f1d2affd700 1 --2- 172.21.15.16:0/1509564414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d2c100ea0 0x7f1d2c073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:49.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.747+0000 7f1d2affd700 1 -- 172.21.15.16:0/1509564414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d2c100ea0 msgr2=0x7f1d2c073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-22T15:53:49.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.747+0000 7f1d2affd700 1 -- 172.21.15.16:0/1509564414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d2c100ea0 msgr2=0x7f1d2c073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T15:53:49.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.747+0000 7f1d2affd700 1 --2- 172.21.15.16:0/1509564414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d2c100ea0 0x7f1d2c073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T15:53:49.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.747+0000 7f1d2affd700 1 --2- 172.21.15.16:0/1509564414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d2c100ea0 0x7f1d2c073090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T15:53:49.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.747+0000 7f1d2a7fc700 1 -- 172.21.15.16:0/1509564414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d2c100ea0 msgr2=0x7f1d2c073090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T15:53:49.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.747+0000 7f1d2a7fc700 1 --2- 172.21.15.16:0/1509564414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d2c100ea0 0x7f1d2c073090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:49.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.747+0000 7f1d2a7fc700 1 -- 172.21.15.16:0/1509564414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d1c0095f0 con 0x7f1d2c101850 2024-04-22T15:53:49.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.747+0000 7f1d2a7fc700 1 --2- 172.21.15.16:0/1509564414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d2c101850 0x7f1d2c0735d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1d2000d900 tx=0x7f1d2000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:49.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.747+0000 7f1d13fff700 1 -- 172.21.15.16:0/1509564414 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d20004a00 con 0x7f1d2c101850 2024-04-22T15:53:49.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.747+0000 7f1d3147d700 1 -- 172.21.15.16:0/1509564414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d2c1a2b10 con 0x7f1d2c101850 2024-04-22T15:53:49.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.747+0000 7f1d13fff700 1 -- 172.21.15.16:0/1509564414 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d200054e0 con 0x7f1d2c101850 2024-04-22T15:53:49.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.747+0000 7f1d13fff700 1 -- 172.21.15.16:0/1509564414 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d20009d00 con 0x7f1d2c101850 2024-04-22T15:53:49.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.747+0000 7f1d3147d700 1 -- 172.21.15.16:0/1509564414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d2c1a2f20 con 0x7f1d2c101850 2024-04-22T15:53:49.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.749+0000 7f1d3147d700 1 -- 172.21.15.16:0/1509564414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d2c109240 con 0x7f1d2c101850 2024-04-22T15:53:49.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.749+0000 7f1d13fff700 1 -- 172.21.15.16:0/1509564414 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f1d20005020 con 0x7f1d2c101850 2024-04-22T15:53:49.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.750+0000 7f1d13fff700 1 --2- 172.21.15.16:0/1509564414 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f1d1406edf0 0x7f1d140712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:53:49.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.750+0000 7f1d13fff700 1 -- 172.21.15.16:0/1509564414 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f1d2008f180 con 0x7f1d2c101850 2024-04-22T15:53:49.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.750+0000 7f1d2affd700 1 --2- 172.21.15.16:0/1509564414 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f1d1406edf0 0x7f1d140712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:53:49.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.750+0000 7f1d2affd700 1 --2- 172.21.15.16:0/1509564414 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f1d1406edf0 0x7f1d140712b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c006010 tx=0x7f1d1c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:53:49.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:49.754+0000 7f1d13fff700 1 -- 172.21.15.16:0/1509564414 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1d20057e40 con 0x7f1d2c101850 2024-04-22T15:53:49.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:49.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: pgmap v12: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 18 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-22T15:53:49.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: from='client.34140 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:49.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: 1: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.smithi027 2024-04-22T15:53:49.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: fsmap cephfs:1 {0=cephfs.smithi027.iobjds=up:active} 1 up:standby-replay 2 up:standby 2024-04-22T15:53:49.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: osdmap e49: 6 total, 6 up, 6 in 2024-04-22T15:53:49.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: mgrmap e37: smithi016.rnsnnr(active, since 18s), standbys: smithi027.ukdgzs 2024-04-22T15:53:49.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:53:49.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:53:49.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:53:49.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: mon.smithi027 calling monitor election 2024-04-22T15:53:49.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: mon.smithi016 calling monitor election 2024-04-22T15:53:49.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-04-22T15:53:49.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:53:49.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: mon.smithi016 is new leader, mons smithi016,smithi027 in quorum (ranks 0,1) 2024-04-22T15:53:49.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: monmap epoch 3 2024-04-22T15:53:49.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: fsid 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T15:53:49.918 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: last_changed 2024-04-22T15:53:48.786768+0000 2024-04-22T15:53:49.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: created 2024-04-22T15:39:17.702487+0000 2024-04-22T15:53:49.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: min_mon_release 19 (squid) 2024-04-22T15:53:49.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: election_strategy: 1 2024-04-22T15:53:49.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.smithi016 2024-04-22T15:53:49.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: 1: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.smithi027 2024-04-22T15:53:49.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: fsmap cephfs:1 {0=cephfs.smithi027.iobjds=up:active} 1 up:standby-replay 2 up:standby 2024-04-22T15:53:49.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: osdmap e49: 6 total, 6 up, 6 in 2024-04-22T15:53:49.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: mgrmap e37: smithi016.rnsnnr(active, since 18s), standbys: smithi027.ukdgzs 2024-04-22T15:53:49.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:53:49.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:53:49.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:53:49.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:49.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:49.920 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:53:49.920 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/2342639461' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:53:49.920 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:49 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/1198761260' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:53:50.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:50.053+0000 7f1d3147d700 1 -- 172.21.15.16:0/1509564414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1d2c04f2e0 con 0x7f1d2c101850 2024-04-22T15:53:50.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:50.053+0000 7f1d13fff700 1 -- 172.21.15.16:0/1509564414 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f1d2005b500 con 0x7f1d2c101850 2024-04-22T15:53:50.056 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:53:50.056 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:53:50.056 INFO:teuthology.orchestra.run.smithi016.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:53:50.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:50.055+0000 7f1d3147d700 1 -- 172.21.15.16:0/1509564414 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f1d1406edf0 msgr2=0x7f1d140712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:50.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:50.055+0000 7f1d3147d700 1 --2- 172.21.15.16:0/1509564414 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f1d1406edf0 0x7f1d140712b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c006010 tx=0x7f1d1c01a040 comp rx=0 tx=0).stop 2024-04-22T15:53:50.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:50.055+0000 7f1d3147d700 1 -- 172.21.15.16:0/1509564414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d2c101850 msgr2=0x7f1d2c0735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:53:50.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:50.055+0000 7f1d3147d700 1 --2- 172.21.15.16:0/1509564414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d2c101850 0x7f1d2c0735d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1d2000d900 tx=0x7f1d2000dcc0 comp rx=0 tx=0).stop 2024-04-22T15:53:50.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:50.055+0000 7f1d3147d700 1 -- 172.21.15.16:0/1509564414 shutdown_connections 2024-04-22T15:53:50.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:50.055+0000 7f1d3147d700 1 --2- 172.21.15.16:0/1509564414 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f1d1406edf0 0x7f1d140712b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:50.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:50.055+0000 7f1d3147d700 1 --2- 172.21.15.16:0/1509564414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d2c100ea0 0x7f1d2c073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:50.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:50.055+0000 7f1d3147d700 1 --2- 172.21.15.16:0/1509564414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d2c101850 0x7f1d2c0735d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:53:50.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:50.055+0000 7f1d3147d700 1 -- 172.21.15.16:0/1509564414 >> 172.21.15.16:0/1509564414 conn(0x7f1d2c0fc750 msgr2=0x7f1d2c0fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:53:50.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:50.055+0000 7f1d3147d700 1 -- 172.21.15.16:0/1509564414 shutdown_connections 2024-04-22T15:53:50.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:53:50.055+0000 7f1d3147d700 1 -- 172.21.15.16:0/1509564414 wait complete. 2024-04-22T15:53:50.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:50.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: pgmap v12: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 18 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-22T15:53:50.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: from='client.34140 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:50.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: 1: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.smithi027 2024-04-22T15:53:50.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: fsmap cephfs:1 {0=cephfs.smithi027.iobjds=up:active} 1 up:standby-replay 2 up:standby 2024-04-22T15:53:50.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: osdmap e49: 6 total, 6 up, 6 in 2024-04-22T15:53:50.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: mgrmap e37: smithi016.rnsnnr(active, since 18s), standbys: smithi027.ukdgzs 2024-04-22T15:53:50.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:53:50.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:53:50.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:53:50.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: mon.smithi027 calling monitor election 2024-04-22T15:53:50.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: mon.smithi016 calling monitor election 2024-04-22T15:53:50.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-04-22T15:53:50.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-22T15:53:50.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: mon.smithi016 is new leader, mons smithi016,smithi027 in quorum (ranks 0,1) 2024-04-22T15:53:50.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: monmap epoch 3 2024-04-22T15:53:50.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: fsid 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T15:53:50.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: last_changed 2024-04-22T15:53:48.786768+0000 2024-04-22T15:53:50.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: created 2024-04-22T15:39:17.702487+0000 2024-04-22T15:53:50.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: min_mon_release 19 (squid) 2024-04-22T15:53:50.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: election_strategy: 1 2024-04-22T15:53:50.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.smithi016 2024-04-22T15:53:50.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: 1: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.smithi027 2024-04-22T15:53:50.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: fsmap cephfs:1 {0=cephfs.smithi027.iobjds=up:active} 1 up:standby-replay 2 up:standby 2024-04-22T15:53:50.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: osdmap e49: 6 total, 6 up, 6 in 2024-04-22T15:53:50.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: mgrmap e37: smithi016.rnsnnr(active, since 18s), standbys: smithi027.ukdgzs 2024-04-22T15:53:50.088 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:53:50.089 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:53:50.089 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:53:50.089 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:50.089 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:50.089 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:53:50.089 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/2342639461' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:53:50.089 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:49 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/1198761260' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:53:51.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:50 smithi027 ceph-mon[83054]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:51.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:50 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/1509564414' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:53:51.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:50 smithi016 ceph-mon[110740]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:53:51.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:50 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/1509564414' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:53:52.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:51 smithi027 ceph-mon[83054]: pgmap v13: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:53:52.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:51 smithi016 ceph-mon[110740]: pgmap v13: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:53:53.159 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:52 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:53.159 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:52 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:53.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:52 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:53.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:52 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:54.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:54 smithi027 ceph-mon[83054]: pgmap v14: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:53:54.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:54 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:54.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:54 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:54.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:54 smithi016 ceph-mon[110740]: pgmap v14: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:53:54.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:54 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:54.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:54 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:56.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:56 smithi027 ceph-mon[83054]: pgmap v15: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:53:56.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:56 smithi016 ceph-mon[110740]: pgmap v15: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:53:57.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:57 smithi027 ceph-mon[83054]: pgmap v16: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:53:57.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:57 smithi027 ceph-mon[83054]: Detected new or changed devices on smithi027 2024-04-22T15:53:57.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:57 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:57.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:57 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:57.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:57 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:53:57.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:57 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:53:57.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:57 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:57.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:57 smithi027 ceph-mon[83054]: Reconfiguring mon.smithi016 (monmap changed)... 2024-04-22T15:53:57.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:57 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:53:57.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:57 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T15:53:57.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:57 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:53:57.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:57 smithi027 ceph-mon[83054]: Reconfiguring daemon mon.smithi016 on smithi016 2024-04-22T15:53:57.901 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:57 smithi016 ceph-mon[110740]: pgmap v16: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:53:57.901 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:57 smithi016 ceph-mon[110740]: Detected new or changed devices on smithi027 2024-04-22T15:53:57.901 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:57 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:57.902 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:57 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:57.902 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:57 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:53:57.902 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:57 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:53:57.902 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:57 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:57.902 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:57 smithi016 ceph-mon[110740]: Reconfiguring mon.smithi016 (monmap changed)... 2024-04-22T15:53:57.902 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:57 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:53:57.902 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:57 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T15:53:57.902 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:57 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:53:57.902 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:57 smithi016 ceph-mon[110740]: Reconfiguring daemon mon.smithi016 on smithi016 2024-04-22T15:53:59.510 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:59 smithi016 ceph-mon[110740]: pgmap v17: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:53:59.510 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:59 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:59.510 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:59 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:59.511 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:59 smithi016 ceph-mon[110740]: Reconfiguring mgr.smithi016.rnsnnr (monmap changed)... 2024-04-22T15:53:59.511 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:59 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi016.rnsnnr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:53:59.511 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:59 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:53:59.511 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:59 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:53:59.511 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:53:59 smithi016 ceph-mon[110740]: Reconfiguring daemon mgr.smithi016.rnsnnr on smithi016 2024-04-22T15:53:59.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:59 smithi027 ceph-mon[83054]: pgmap v17: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:53:59.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:59 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:59.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:59 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:53:59.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:59 smithi027 ceph-mon[83054]: Reconfiguring mgr.smithi016.rnsnnr (monmap changed)... 2024-04-22T15:53:59.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:59 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi016.rnsnnr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:53:59.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:59 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:53:59.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:59 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:53:59.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:53:59 smithi027 ceph-mon[83054]: Reconfiguring daemon mgr.smithi016.rnsnnr on smithi016 2024-04-22T15:54:01.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:00 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:01.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:00 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:01.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:00 smithi027 ceph-mon[83054]: Reconfiguring ceph-exporter.smithi016 (monmap changed)... 2024-04-22T15:54:01.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:00 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi016", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:54:01.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:00 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi016", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:54:01.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:00 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi016", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-22T15:54:01.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:00 smithi027 ceph-mon[83054]: Unable to update caps for client.ceph-exporter.smithi016 2024-04-22T15:54:01.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:00 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi016"}]: dispatch 2024-04-22T15:54:01.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:00 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:01.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:00 smithi027 ceph-mon[83054]: Reconfiguring daemon ceph-exporter.smithi016 on smithi016 2024-04-22T15:54:01.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:00 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:54:01.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:00 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:01.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:00 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:01.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:00 smithi016 ceph-mon[110740]: Reconfiguring ceph-exporter.smithi016 (monmap changed)... 2024-04-22T15:54:01.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:00 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi016", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:54:01.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:00 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi016", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:54:01.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:00 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi016", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-22T15:54:01.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:00 smithi016 ceph-mon[110740]: Unable to update caps for client.ceph-exporter.smithi016 2024-04-22T15:54:01.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:00 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi016"}]: dispatch 2024-04-22T15:54:01.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:00 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:01.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:00 smithi016 ceph-mon[110740]: Reconfiguring daemon ceph-exporter.smithi016 on smithi016 2024-04-22T15:54:01.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:00 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:54:02.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:01 smithi016 ceph-mon[110740]: pgmap v18: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:02.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:01 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:02.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:01 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:02.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:01 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi016", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:54:02.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:01 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:02.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:01 smithi027 ceph-mon[83054]: pgmap v18: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:02.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:01 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:02.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:01 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:02.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:01 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi016", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:54:02.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:01 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:03.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:02 smithi016 ceph-mon[110740]: Reconfiguring crash.smithi016 (monmap changed)... 2024-04-22T15:54:03.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:02 smithi016 ceph-mon[110740]: Reconfiguring daemon crash.smithi016 on smithi016 2024-04-22T15:54:03.257 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:02 smithi027 ceph-mon[83054]: Reconfiguring crash.smithi016 (monmap changed)... 2024-04-22T15:54:03.258 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:02 smithi027 ceph-mon[83054]: Reconfiguring daemon crash.smithi016 on smithi016 2024-04-22T15:54:04.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:03 smithi016 ceph-mon[110740]: pgmap v19: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:54:04.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:03 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:04.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:03 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:04.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:03 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T15:54:04.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:03 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:04.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:03 smithi027 ceph-mon[83054]: pgmap v19: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:54:04.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:03 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:04.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:03 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:04.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:03 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T15:54:04.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:03 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:05.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:04 smithi016 ceph-mon[110740]: Reconfiguring osd.0 (monmap changed)... 2024-04-22T15:54:05.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:04 smithi016 ceph-mon[110740]: Reconfiguring daemon osd.0 on smithi016 2024-04-22T15:54:05.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:04 smithi027 ceph-mon[83054]: Reconfiguring osd.0 (monmap changed)... 2024-04-22T15:54:05.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:04 smithi027 ceph-mon[83054]: Reconfiguring daemon osd.0 on smithi016 2024-04-22T15:54:06.115 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:05 smithi016 ceph-mon[110740]: pgmap v20: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:06.115 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:05 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:06.115 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:05 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:06.115 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:05 smithi016 ceph-mon[110740]: Reconfiguring osd.1 (monmap changed)... 2024-04-22T15:54:06.115 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:05 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T15:54:06.115 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:05 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:06.115 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:05 smithi016 ceph-mon[110740]: Reconfiguring daemon osd.1 on smithi016 2024-04-22T15:54:06.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:05 smithi027 ceph-mon[83054]: pgmap v20: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:06.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:05 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:06.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:05 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:06.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:05 smithi027 ceph-mon[83054]: Reconfiguring osd.1 (monmap changed)... 2024-04-22T15:54:06.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:05 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T15:54:06.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:05 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:06.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:05 smithi027 ceph-mon[83054]: Reconfiguring daemon osd.1 on smithi016 2024-04-22T15:54:07.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:07 smithi027 ceph-mon[83054]: pgmap v21: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:07.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:07 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:07.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:07 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:07.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:07 smithi027 ceph-mon[83054]: Reconfiguring osd.2 (monmap changed)... 2024-04-22T15:54:07.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:07 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T15:54:07.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:07 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:07.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:07 smithi027 ceph-mon[83054]: Reconfiguring daemon osd.2 on smithi016 2024-04-22T15:54:07.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:07 smithi016 ceph-mon[110740]: pgmap v21: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:07.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:07 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:07.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:07 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:07.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:07 smithi016 ceph-mon[110740]: Reconfiguring osd.2 (monmap changed)... 2024-04-22T15:54:07.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:07 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T15:54:07.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:07 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:07.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:07 smithi016 ceph-mon[110740]: Reconfiguring daemon osd.2 on smithi016 2024-04-22T15:54:09.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:09 smithi016 ceph-mon[110740]: pgmap v22: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:54:09.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:09 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:09.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:09 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:09.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:09 smithi016 ceph-mon[110740]: Reconfiguring mds.cephfs.smithi016.fkkejm (monmap changed)... 2024-04-22T15:54:09.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:09 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi016.fkkejm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:54:09.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:09 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:09.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:09 smithi016 ceph-mon[110740]: Reconfiguring daemon mds.cephfs.smithi016.fkkejm on smithi016 2024-04-22T15:54:09.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:09 smithi027 ceph-mon[83054]: pgmap v22: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:54:09.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:09 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:09.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:09 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:09.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:09 smithi027 ceph-mon[83054]: Reconfiguring mds.cephfs.smithi016.fkkejm (monmap changed)... 2024-04-22T15:54:09.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:09 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi016.fkkejm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:54:09.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:09 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:09.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:09 smithi027 ceph-mon[83054]: Reconfiguring daemon mds.cephfs.smithi016.fkkejm on smithi016 2024-04-22T15:54:11.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:10 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:11.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:10 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:11.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:10 smithi016 ceph-mon[110740]: Reconfiguring mds.cephfs.smithi016.pgvphu (monmap changed)... 2024-04-22T15:54:11.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:10 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi016.pgvphu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:54:11.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:10 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:11.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:10 smithi016 ceph-mon[110740]: Reconfiguring daemon mds.cephfs.smithi016.pgvphu on smithi016 2024-04-22T15:54:11.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:10 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:11.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:10 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:11.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:10 smithi027 ceph-mon[83054]: Reconfiguring mds.cephfs.smithi016.pgvphu (monmap changed)... 2024-04-22T15:54:11.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:10 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi016.pgvphu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:54:11.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:10 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:11.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:10 smithi027 ceph-mon[83054]: Reconfiguring daemon mds.cephfs.smithi016.pgvphu on smithi016 2024-04-22T15:54:12.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:11 smithi016 ceph-mon[110740]: pgmap v23: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:12.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:11 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:12.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:11 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:12.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:11 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi027", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:54:12.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:11 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi027", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:54:12.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:11 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi027", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-22T15:54:12.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:11 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi027"}]: dispatch 2024-04-22T15:54:12.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:11 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:12.171 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:11 smithi027 ceph-mon[83054]: pgmap v23: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:12.171 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:11 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:12.171 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:11 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:12.171 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:11 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi027", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:54:12.171 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:11 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi027", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:54:12.171 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:11 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi027", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-22T15:54:12.171 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:11 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi027"}]: dispatch 2024-04-22T15:54:12.171 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:11 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:13.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:12 smithi016 ceph-mon[110740]: Reconfiguring ceph-exporter.smithi027 (monmap changed)... 2024-04-22T15:54:13.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:12 smithi016 ceph-mon[110740]: Unable to update caps for client.ceph-exporter.smithi027 2024-04-22T15:54:13.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:12 smithi016 ceph-mon[110740]: Reconfiguring daemon ceph-exporter.smithi027 on smithi027 2024-04-22T15:54:13.200 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:12 smithi027 ceph-mon[83054]: Reconfiguring ceph-exporter.smithi027 (monmap changed)... 2024-04-22T15:54:13.200 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:12 smithi027 ceph-mon[83054]: Unable to update caps for client.ceph-exporter.smithi027 2024-04-22T15:54:13.200 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:12 smithi027 ceph-mon[83054]: Reconfiguring daemon ceph-exporter.smithi027 on smithi027 2024-04-22T15:54:14.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:13 smithi016 ceph-mon[110740]: pgmap v24: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:54:14.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:13 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:14.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:13 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:14.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:13 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi027", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:54:14.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:13 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:14.177 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:13 smithi027 ceph-mon[83054]: pgmap v24: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:54:14.177 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:13 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:14.177 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:13 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:14.177 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:13 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi027", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:54:14.177 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:13 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:15.116 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:14 smithi027 ceph-mon[83054]: Reconfiguring crash.smithi027 (monmap changed)... 2024-04-22T15:54:15.116 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:14 smithi027 ceph-mon[83054]: Reconfiguring daemon crash.smithi027 on smithi027 2024-04-22T15:54:15.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:14 smithi016 ceph-mon[110740]: Reconfiguring crash.smithi027 (monmap changed)... 2024-04-22T15:54:15.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:14 smithi016 ceph-mon[110740]: Reconfiguring daemon crash.smithi027 on smithi027 2024-04-22T15:54:16.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:15 smithi027 ceph-mon[83054]: pgmap v25: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:16.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:15 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:16.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:15 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:16.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:15 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi027.ukdgzs", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:54:16.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:15 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:54:16.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:15 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:16.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:15 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:54:16.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:15 smithi016 ceph-mon[110740]: pgmap v25: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:16.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:15 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:16.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:15 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:16.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:15 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi027.ukdgzs", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T15:54:16.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:15 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T15:54:16.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:15 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:16.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:15 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:54:17.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:16 smithi027 ceph-mon[83054]: Reconfiguring mgr.smithi027.ukdgzs (monmap changed)... 2024-04-22T15:54:17.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:16 smithi027 ceph-mon[83054]: Reconfiguring daemon mgr.smithi027.ukdgzs on smithi027 2024-04-22T15:54:17.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:16 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:17.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:16 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:17.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:16 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:54:17.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:16 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T15:54:17.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:16 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:17.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:16 smithi016 ceph-mon[110740]: Reconfiguring mgr.smithi027.ukdgzs (monmap changed)... 2024-04-22T15:54:17.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:16 smithi016 ceph-mon[110740]: Reconfiguring daemon mgr.smithi027.ukdgzs on smithi027 2024-04-22T15:54:17.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:16 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:17.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:16 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:17.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:16 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T15:54:17.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:16 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T15:54:17.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:16 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:18.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:17 smithi027 ceph-mon[83054]: pgmap v26: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:54:18.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:17 smithi027 ceph-mon[83054]: Reconfiguring mon.smithi027 (monmap changed)... 2024-04-22T15:54:18.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:17 smithi027 ceph-mon[83054]: Reconfiguring daemon mon.smithi027 on smithi027 2024-04-22T15:54:18.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:17 smithi016 ceph-mon[110740]: pgmap v26: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:54:18.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:17 smithi016 ceph-mon[110740]: Reconfiguring mon.smithi027 (monmap changed)... 2024-04-22T15:54:18.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:17 smithi016 ceph-mon[110740]: Reconfiguring daemon mon.smithi027 on smithi027 2024-04-22T15:54:19.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:19 smithi027 ceph-mon[83054]: pgmap v27: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:54:19.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:19 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:19.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:19 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:19.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:19 smithi027 ceph-mon[83054]: Reconfiguring osd.3 (monmap changed)... 2024-04-22T15:54:19.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:19 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T15:54:19.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:19 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:19.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:19 smithi027 ceph-mon[83054]: Reconfiguring daemon osd.3 on smithi027 2024-04-22T15:54:19.887 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:19 smithi016 ceph-mon[110740]: pgmap v27: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:54:19.888 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:19 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:19.888 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:19 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:19.888 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:19 smithi016 ceph-mon[110740]: Reconfiguring osd.3 (monmap changed)... 2024-04-22T15:54:19.888 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:19 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T15:54:19.888 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:19 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:19.888 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:19 smithi016 ceph-mon[110740]: Reconfiguring daemon osd.3 on smithi027 2024-04-22T15:54:20.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.176+0000 7fcac9aa3700 1 -- 172.21.15.16:0/1995325953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcac4106c00 msgr2=0x7fcac4106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:20.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.176+0000 7fcac9aa3700 1 --2- 172.21.15.16:0/1995325953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcac4106c00 0x7fcac4106fe0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcab4009910 tx=0x7fcab4009c20 comp rx=0 tx=0).stop 2024-04-22T15:54:20.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.176+0000 7fcac9aa3700 1 -- 172.21.15.16:0/1995325953 shutdown_connections 2024-04-22T15:54:20.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.176+0000 7fcac9aa3700 1 --2- 172.21.15.16:0/1995325953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcac4100b50 0x7fcac4100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.176+0000 7fcac9aa3700 1 --2- 172.21.15.16:0/1995325953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcac4106c00 0x7fcac4106fe0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.176+0000 7fcac9aa3700 1 -- 172.21.15.16:0/1995325953 >> 172.21.15.16:0/1995325953 conn(0x7fcac40fc640 msgr2=0x7fcac40fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:20.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.176+0000 7fcac9aa3700 1 -- 172.21.15.16:0/1995325953 shutdown_connections 2024-04-22T15:54:20.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.176+0000 7fcac9aa3700 1 -- 172.21.15.16:0/1995325953 wait complete. 2024-04-22T15:54:20.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.176+0000 7fcac9aa3700 1 Processor -- start 2024-04-22T15:54:20.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.177+0000 7fcac9aa3700 1 -- start start 2024-04-22T15:54:20.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.177+0000 7fcac9aa3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcac4100b50 0x7fcac4196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:20.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.177+0000 7fcac9aa3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcac4106c00 0x7fcac4196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:20.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.177+0000 7fcac9aa3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac4197290 con 0x7fcac4100b50 2024-04-22T15:54:20.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.177+0000 7fcac9aa3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac419b020 con 0x7fcac4106c00 2024-04-22T15:54:20.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.177+0000 7fcac37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcac4100b50 0x7fcac4196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:20.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.177+0000 7fcac37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcac4100b50 0x7fcac4196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48434/0 (socket says 172.21.15.16:48434) 2024-04-22T15:54:20.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.177+0000 7fcac37fe700 1 -- 172.21.15.16:0/2864444176 learned_addr learned my addr 172.21.15.16:0/2864444176 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:54:20.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.177+0000 7fcac2ffd700 1 --2- 172.21.15.16:0/2864444176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcac4106c00 0x7fcac4196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:20.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.177+0000 7fcac37fe700 1 -- 172.21.15.16:0/2864444176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcac4106c00 msgr2=0x7fcac4196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:20.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.177+0000 7fcac37fe700 1 --2- 172.21.15.16:0/2864444176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcac4106c00 0x7fcac4196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.178+0000 7fcac37fe700 1 -- 172.21.15.16:0/2864444176 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab40095f0 con 0x7fcac4100b50 2024-04-22T15:54:20.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.178+0000 7fcac37fe700 1 --2- 172.21.15.16:0/2864444176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcac4100b50 0x7fcac4196670 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcab40092e0 tx=0x7fcab400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:20.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.178+0000 7fcac0ff9700 1 -- 172.21.15.16:0/2864444176 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcab401d070 con 0x7fcac4100b50 2024-04-22T15:54:20.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.178+0000 7fcac0ff9700 1 -- 172.21.15.16:0/2864444176 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcab4005e00 con 0x7fcac4100b50 2024-04-22T15:54:20.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.178+0000 7fcac9aa3700 1 -- 172.21.15.16:0/2864444176 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac419b2a0 con 0x7fcac4100b50 2024-04-22T15:54:20.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.178+0000 7fcac0ff9700 1 -- 172.21.15.16:0/2864444176 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcab4017400 con 0x7fcac4100b50 2024-04-22T15:54:20.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.178+0000 7fcac9aa3700 1 -- 172.21.15.16:0/2864444176 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac419b810 con 0x7fcac4100b50 2024-04-22T15:54:20.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.179+0000 7fcac9aa3700 1 -- 172.21.15.16:0/2864444176 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac4109030 con 0x7fcac4100b50 2024-04-22T15:54:20.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.179+0000 7fcac0ff9700 1 -- 172.21.15.16:0/2864444176 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fcab400f6f0 con 0x7fcac4100b50 2024-04-22T15:54:20.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.180+0000 7fcac0ff9700 1 --2- 172.21.15.16:0/2864444176 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fcaac06edf0 0x7fcaac0712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:20.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.180+0000 7fcac2ffd700 1 --2- 172.21.15.16:0/2864444176 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fcaac06edf0 0x7fcaac0712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:20.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.180+0000 7fcac0ff9700 1 -- 172.21.15.16:0/2864444176 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fcab4090440 con 0x7fcac4100b50 2024-04-22T15:54:20.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.181+0000 7fcac2ffd700 1 --2- 172.21.15.16:0/2864444176 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fcaac06edf0 0x7fcaac0712b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcac4197c90 tx=0x7fcab800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:20.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.185+0000 7fcac0ff9700 1 -- 172.21.15.16:0/2864444176 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fcab4059100 con 0x7fcac4100b50 2024-04-22T15:54:20.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.420+0000 7fcac9aa3700 1 -- 172.21.15.16:0/2864444176 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcac419bb40 con 0x7fcaac06edf0 2024-04-22T15:54:20.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.421+0000 7fcac0ff9700 1 -- 172.21.15.16:0/2864444176 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7fcac419bb40 con 0x7fcaac06edf0 2024-04-22T15:54:20.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.424+0000 7fcac9aa3700 1 -- 172.21.15.16:0/2864444176 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fcaac06edf0 msgr2=0x7fcaac0712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:20.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.424+0000 7fcac9aa3700 1 --2- 172.21.15.16:0/2864444176 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fcaac06edf0 0x7fcaac0712b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcac4197c90 tx=0x7fcab800b410 comp rx=0 tx=0).stop 2024-04-22T15:54:20.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.424+0000 7fcac9aa3700 1 -- 172.21.15.16:0/2864444176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcac4100b50 msgr2=0x7fcac4196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:20.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.424+0000 7fcac9aa3700 1 --2- 172.21.15.16:0/2864444176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcac4100b50 0x7fcac4196670 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcab40092e0 tx=0x7fcab400bd70 comp rx=0 tx=0).stop 2024-04-22T15:54:20.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.424+0000 7fcac9aa3700 1 -- 172.21.15.16:0/2864444176 shutdown_connections 2024-04-22T15:54:20.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.424+0000 7fcac9aa3700 1 --2- 172.21.15.16:0/2864444176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcac4100b50 0x7fcac4196670 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.424+0000 7fcac9aa3700 1 --2- 172.21.15.16:0/2864444176 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fcaac06edf0 0x7fcaac0712b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.424+0000 7fcac9aa3700 1 --2- 172.21.15.16:0/2864444176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcac4106c00 0x7fcac4196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.424+0000 7fcac9aa3700 1 -- 172.21.15.16:0/2864444176 >> 172.21.15.16:0/2864444176 conn(0x7fcac40fc640 msgr2=0x7fcac40fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:20.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.424+0000 7fcac9aa3700 1 -- 172.21.15.16:0/2864444176 shutdown_connections 2024-04-22T15:54:20.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.424+0000 7fcac9aa3700 1 -- 172.21.15.16:0/2864444176 wait complete. 2024-04-22T15:54:20.436 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:54:20.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.531+0000 7f2668e36700 1 -- 172.21.15.16:0/1731829472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26640ff5a0 msgr2=0x7f26640ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.531+0000 7f2668e36700 1 --2- 172.21.15.16:0/1731829472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26640ff5a0 0x7f26640ff980 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2654009910 tx=0x7f2654009c20 comp rx=0 tx=0).stop 2024-04-22T15:54:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.532+0000 7f2668e36700 1 -- 172.21.15.16:0/1731829472 shutdown_connections 2024-04-22T15:54:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.532+0000 7f2668e36700 1 --2- 172.21.15.16:0/1731829472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26640fff50 0x7f2664105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.532+0000 7f2668e36700 1 --2- 172.21.15.16:0/1731829472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26640ff5a0 0x7f26640ff980 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.532+0000 7f2668e36700 1 -- 172.21.15.16:0/1731829472 >> 172.21.15.16:0/1731829472 conn(0x7f2664074e30 msgr2=0x7f2664075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.532+0000 7f2668e36700 1 -- 172.21.15.16:0/1731829472 shutdown_connections 2024-04-22T15:54:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.532+0000 7f2668e36700 1 -- 172.21.15.16:0/1731829472 wait complete. 2024-04-22T15:54:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.532+0000 7f2668e36700 1 Processor -- start 2024-04-22T15:54:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.532+0000 7f2668e36700 1 -- start start 2024-04-22T15:54:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.533+0000 7f2668e36700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26640ff5a0 0x7f2664100520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.533+0000 7f2668e36700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26640fff50 0x7f2664100a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.533+0000 7f2668e36700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2664100fa0 con 0x7f26640ff5a0 2024-04-22T15:54:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.533+0000 7f2668e36700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26641010e0 con 0x7f26640fff50 2024-04-22T15:54:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.533+0000 7f266259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26640ff5a0 0x7f2664100520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.533+0000 7f266259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26640ff5a0 0x7f2664100520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48450/0 (socket says 172.21.15.16:48450) 2024-04-22T15:54:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.533+0000 7f266259c700 1 -- 172.21.15.16:0/3871009973 learned_addr learned my addr 172.21.15.16:0/3871009973 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:54:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.533+0000 7f2661d9b700 1 --2- 172.21.15.16:0/3871009973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26640fff50 0x7f2664100a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.533+0000 7f266259c700 1 -- 172.21.15.16:0/3871009973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26640fff50 msgr2=0x7f2664100a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.533+0000 7f266259c700 1 --2- 172.21.15.16:0/3871009973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26640fff50 0x7f2664100a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.533+0000 7f266259c700 1 -- 172.21.15.16:0/3871009973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26540095f0 con 0x7f26640ff5a0 2024-04-22T15:54:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.533+0000 7f266259c700 1 --2- 172.21.15.16:0/3871009973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26640ff5a0 0x7f2664100520 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2654005f50 tx=0x7f265400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.533+0000 7f26537fe700 1 -- 172.21.15.16:0/3871009973 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f265401d070 con 0x7f26640ff5a0 2024-04-22T15:54:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.533+0000 7f26537fe700 1 -- 172.21.15.16:0/3871009973 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f265400b610 con 0x7f26640ff5a0 2024-04-22T15:54:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.533+0000 7f2668e36700 1 -- 172.21.15.16:0/3871009973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2664101360 con 0x7f26640ff5a0 2024-04-22T15:54:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.533+0000 7f26537fe700 1 -- 172.21.15.16:0/3871009973 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2654022470 con 0x7f26640ff5a0 2024-04-22T15:54:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.533+0000 7f2668e36700 1 -- 172.21.15.16:0/3871009973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26641a2db0 con 0x7f26640ff5a0 2024-04-22T15:54:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.534+0000 7f2668e36700 1 -- 172.21.15.16:0/3871009973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2664109270 con 0x7f26640ff5a0 2024-04-22T15:54:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.535+0000 7f26537fe700 1 -- 172.21.15.16:0/3871009973 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f265400faf0 con 0x7f26640ff5a0 2024-04-22T15:54:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.535+0000 7f26537fe700 1 --2- 172.21.15.16:0/3871009973 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f264c06ee00 0x7f264c0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.535+0000 7f26537fe700 1 -- 172.21.15.16:0/3871009973 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f2654090400 con 0x7f26640ff5a0 2024-04-22T15:54:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.535+0000 7f2661d9b700 1 --2- 172.21.15.16:0/3871009973 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f264c06ee00 0x7f264c0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.536+0000 7f2661d9b700 1 --2- 172.21.15.16:0/3871009973 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f264c06ee00 0x7f264c0712c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2664101d10 tx=0x7f2658009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:20.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.540+0000 7f26537fe700 1 -- 172.21.15.16:0/3871009973 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2654059040 con 0x7f26640ff5a0 2024-04-22T15:54:20.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.771+0000 7f2668e36700 1 -- 172.21.15.16:0/3871009973 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f26640611d0 con 0x7f264c06ee00 2024-04-22T15:54:20.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.772+0000 7f26537fe700 1 -- 172.21.15.16:0/3871009973 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f26640611d0 con 0x7f264c06ee00 2024-04-22T15:54:20.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.774+0000 7f2668e36700 1 -- 172.21.15.16:0/3871009973 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f264c06ee00 msgr2=0x7f264c0712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:20.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.774+0000 7f2668e36700 1 --2- 172.21.15.16:0/3871009973 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f264c06ee00 0x7f264c0712c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2664101d10 tx=0x7f2658009260 comp rx=0 tx=0).stop 2024-04-22T15:54:20.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.774+0000 7f2668e36700 1 -- 172.21.15.16:0/3871009973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26640ff5a0 msgr2=0x7f2664100520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:20.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.774+0000 7f2668e36700 1 --2- 172.21.15.16:0/3871009973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26640ff5a0 0x7f2664100520 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2654005f50 tx=0x7f265400f690 comp rx=0 tx=0).stop 2024-04-22T15:54:20.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.774+0000 7f2668e36700 1 -- 172.21.15.16:0/3871009973 shutdown_connections 2024-04-22T15:54:20.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.774+0000 7f2668e36700 1 --2- 172.21.15.16:0/3871009973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26640ff5a0 0x7f2664100520 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.774+0000 7f2668e36700 1 --2- 172.21.15.16:0/3871009973 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f264c06ee00 0x7f264c0712c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.774+0000 7f2668e36700 1 --2- 172.21.15.16:0/3871009973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26640fff50 0x7f2664100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.774+0000 7f2668e36700 1 -- 172.21.15.16:0/3871009973 >> 172.21.15.16:0/3871009973 conn(0x7f2664074e30 msgr2=0x7f26640fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:20.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.774+0000 7f2668e36700 1 -- 172.21.15.16:0/3871009973 shutdown_connections 2024-04-22T15:54:20.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.774+0000 7f2668e36700 1 -- 172.21.15.16:0/3871009973 wait complete. 2024-04-22T15:54:20.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.883+0000 7f410ab9e700 1 -- 172.21.15.16:0/657990913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4104105bd0 msgr2=0x7f4104105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:20.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.883+0000 7f410ab9e700 1 --2- 172.21.15.16:0/657990913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4104105bd0 0x7f4104105fb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f40f8009910 tx=0x7f40f8009c20 comp rx=0 tx=0).stop 2024-04-22T15:54:20.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.883+0000 7f410ab9e700 1 -- 172.21.15.16:0/657990913 shutdown_connections 2024-04-22T15:54:20.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.883+0000 7f410ab9e700 1 --2- 172.21.15.16:0/657990913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41040ffc20 0x7f4104100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.883+0000 7f410ab9e700 1 --2- 172.21.15.16:0/657990913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4104105bd0 0x7f4104105fb0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.883+0000 7f410ab9e700 1 -- 172.21.15.16:0/657990913 >> 172.21.15.16:0/657990913 conn(0x7f4104075670 msgr2=0x7f4104075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:20.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.883+0000 7f410ab9e700 1 -- 172.21.15.16:0/657990913 shutdown_connections 2024-04-22T15:54:20.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.883+0000 7f410ab9e700 1 -- 172.21.15.16:0/657990913 wait complete. 2024-04-22T15:54:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.884+0000 7f410ab9e700 1 Processor -- start 2024-04-22T15:54:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.884+0000 7f410ab9e700 1 -- start start 2024-04-22T15:54:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.884+0000 7f410ab9e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41040ffc20 0x7f4104101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.884+0000 7f410ab9e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4104105bd0 0x7f4104101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.884+0000 7f410ab9e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41041022c0 con 0x7f41040ffc20 2024-04-22T15:54:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.884+0000 7f410ab9e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4104102400 con 0x7f4104105bd0 2024-04-22T15:54:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.884+0000 7f410893a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41040ffc20 0x7f4104101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.884+0000 7f410893a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41040ffc20 0x7f4104101840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48474/0 (socket says 172.21.15.16:48474) 2024-04-22T15:54:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.884+0000 7f410893a700 1 -- 172.21.15.16:0/612327367 learned_addr learned my addr 172.21.15.16:0/612327367 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:54:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.884+0000 7f4103fff700 1 --2- 172.21.15.16:0/612327367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4104105bd0 0x7f4104101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.884+0000 7f410893a700 1 -- 172.21.15.16:0/612327367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4104105bd0 msgr2=0x7f4104101d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.884+0000 7f410893a700 1 --2- 172.21.15.16:0/612327367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4104105bd0 0x7f4104101d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.884+0000 7f410893a700 1 -- 172.21.15.16:0/612327367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f80095f0 con 0x7f41040ffc20 2024-04-22T15:54:20.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.884+0000 7f4103fff700 1 --2- 172.21.15.16:0/612327367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4104105bd0 0x7f4104101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:54:20.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.885+0000 7f410893a700 1 --2- 172.21.15.16:0/612327367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41040ffc20 0x7f4104101840 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f40f8005f50 tx=0x7f40f800f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:20.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.885+0000 7f4101ffb700 1 -- 172.21.15.16:0/612327367 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40f801d070 con 0x7f41040ffc20 2024-04-22T15:54:20.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.885+0000 7f4101ffb700 1 -- 172.21.15.16:0/612327367 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f40f800b610 con 0x7f41040ffc20 2024-04-22T15:54:20.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.885+0000 7f410ab9e700 1 -- 172.21.15.16:0/612327367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4104102680 con 0x7f41040ffc20 2024-04-22T15:54:20.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.885+0000 7f4101ffb700 1 -- 172.21.15.16:0/612327367 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40f8022470 con 0x7f41040ffc20 2024-04-22T15:54:20.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.885+0000 7f410ab9e700 1 -- 172.21.15.16:0/612327367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41041a2d50 con 0x7f41040ffc20 2024-04-22T15:54:20.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.886+0000 7f410ab9e700 1 -- 172.21.15.16:0/612327367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f410404f2e0 con 0x7f41040ffc20 2024-04-22T15:54:20.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.887+0000 7f4101ffb700 1 -- 172.21.15.16:0/612327367 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f40f800fca0 con 0x7f41040ffc20 2024-04-22T15:54:20.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.887+0000 7f4101ffb700 1 --2- 172.21.15.16:0/612327367 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f40f406ee00 0x7f40f40712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:20.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.887+0000 7f4101ffb700 1 -- 172.21.15.16:0/612327367 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f40f8090490 con 0x7f41040ffc20 2024-04-22T15:54:20.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.887+0000 7f4103fff700 1 --2- 172.21.15.16:0/612327367 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f40f406ee00 0x7f40f40712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:20.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.888+0000 7f4103fff700 1 --2- 172.21.15.16:0/612327367 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f40f406ee00 0x7f40f40712c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4104103030 tx=0x7f40f0009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:20.892+0000 7f4101ffb700 1 -- 172.21.15.16:0/612327367 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f40f80590d0 con 0x7f41040ffc20 2024-04-22T15:54:21.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.108+0000 7f410ab9e700 1 -- 172.21.15.16:0/612327367 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4104068d90 con 0x7f40f406ee00 2024-04-22T15:54:21.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.121+0000 7f4101ffb700 1 -- 172.21.15.16:0/612327367 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f4104068d90 con 0x7f40f406ee00 2024-04-22T15:54:21.123 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:54:21.123 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.smithi016 smithi016 *:9093,9094 running (2m) 45s ago 13m 26.4M - 0.25.0 c8568f914cd2 ea0e0217d6f9 2024-04-22T15:54:21.123 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi016 smithi016 running (13m) 45s ago 13m 7994k - 18.2.1 5be31c24972a f8b627d3f41e 2024-04-22T15:54:21.123 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi027 smithi027 running (12m) 29s ago 12m 8002k - 18.2.1 5be31c24972a f00fd40b4220 2024-04-22T15:54:21.124 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi016 smithi016 running (13m) 45s ago 13m 6660k - 18.2.1 5be31c24972a 6b93db1fa0d9 2024-04-22T15:54:21.124 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi027 smithi027 running (12m) 29s ago 12m 6669k - 18.2.1 5be31c24972a d3bd439f30a0 2024-04-22T15:54:21.124 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.smithi016 smithi016 *:3000 running (2m) 45s ago 13m 75.9M - 9.4.12 2bacad6d85d8 388a3c9c1b3f 2024-04-22T15:54:21.124 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.fkkejm smithi016 running (7m) 45s ago 7m 15.5M - 18.2.1 5be31c24972a 4153ac94cea0 2024-04-22T15:54:21.124 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.pgvphu smithi016 running (7m) 45s ago 7m 92.0M - 18.2.1 5be31c24972a 93dc072068a2 2024-04-22T15:54:21.124 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.dgqyes smithi027 running (7m) 29s ago 7m 18.9M - 18.2.1 5be31c24972a 41a9220430fd 2024-04-22T15:54:21.124 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.iobjds smithi027 running (7m) 29s ago 7m 150M - 18.2.1 5be31c24972a c1c8d6770f02 2024-04-22T15:54:21.124 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi016.rnsnnr smithi016 *:8443,9283,8765 running (4m) 45s ago 14m 472M - 19.0.0-3223-g5883d494 3d983c26e278 09b4fe285bc4 2024-04-22T15:54:21.124 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi027.ukdgzs smithi027 *:8443,9283,8765 running (3m) 29s ago 12m 449M - 19.0.0-3223-g5883d494 3d983c26e278 7548da9b5eb9 2024-04-22T15:54:21.124 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi016 smithi016 running (59s) 45s ago 15m 29.0M 2048M 19.0.0-3223-g5883d494 3d983c26e278 4ae529a67806 2024-04-22T15:54:21.124 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi027 smithi027 running (34s) 29s ago 12m 26.2M 2048M 19.0.0-3223-g5883d494 3d983c26e278 f29fcb888801 2024-04-22T15:54:21.124 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi016 smithi016 *:9100 running (3m) 45s ago 13m 15.9M - 1.5.0 0da6a335fe13 35e46e4d2f3f 2024-04-22T15:54:21.125 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi027 smithi027 *:9100 running (3m) 29s ago 12m 15.5M - 1.5.0 0da6a335fe13 e682a7672e9b 2024-04-22T15:54:21.125 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (11m) 45s ago 11m 201M 1217M 18.2.1 5be31c24972a 3c6ebc4ea041 2024-04-22T15:54:21.125 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (10m) 45s ago 10m 205M 1217M 18.2.1 5be31c24972a 6aa2e6796784 2024-04-22T15:54:21.125 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (10m) 45s ago 10m 176M 1217M 18.2.1 5be31c24972a 3d9d310aa9eb 2024-04-22T15:54:21.125 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi027 running (10m) 29s ago 10m 250M 2240M 18.2.1 5be31c24972a cdf9b7aaf865 2024-04-22T15:54:21.125 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (9m) 29s ago 9m 206M 2240M 18.2.1 5be31c24972a 629397d49c76 2024-04-22T15:54:21.125 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi027 running (9m) 29s ago 9m 193M 2240M 18.2.1 5be31c24972a 23adcf675b04 2024-04-22T15:54:21.125 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.smithi016 smithi016 *:9095 running (2m) 45s ago 12m 67.3M - 2.43.0 a07b618ecd1d 373eaadee82d 2024-04-22T15:54:21.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.124+0000 7f410ab9e700 1 -- 172.21.15.16:0/612327367 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f40f406ee00 msgr2=0x7f40f40712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:21.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.124+0000 7f410ab9e700 1 --2- 172.21.15.16:0/612327367 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f40f406ee00 0x7f40f40712c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4104103030 tx=0x7f40f0009260 comp rx=0 tx=0).stop 2024-04-22T15:54:21.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.124+0000 7f410ab9e700 1 -- 172.21.15.16:0/612327367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41040ffc20 msgr2=0x7f4104101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:21.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.124+0000 7f410ab9e700 1 --2- 172.21.15.16:0/612327367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41040ffc20 0x7f4104101840 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f40f8005f50 tx=0x7f40f800f740 comp rx=0 tx=0).stop 2024-04-22T15:54:21.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.124+0000 7f410ab9e700 1 -- 172.21.15.16:0/612327367 shutdown_connections 2024-04-22T15:54:21.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.124+0000 7f410ab9e700 1 --2- 172.21.15.16:0/612327367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41040ffc20 0x7f4104101840 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.124+0000 7f410ab9e700 1 --2- 172.21.15.16:0/612327367 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f40f406ee00 0x7f40f40712c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.124+0000 7f410ab9e700 1 --2- 172.21.15.16:0/612327367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4104105bd0 0x7f4104101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.124+0000 7f410ab9e700 1 -- 172.21.15.16:0/612327367 >> 172.21.15.16:0/612327367 conn(0x7f4104075670 msgr2=0x7f4104068680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:21.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.124+0000 7f410ab9e700 1 -- 172.21.15.16:0/612327367 shutdown_connections 2024-04-22T15:54:21.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.124+0000 7f410ab9e700 1 -- 172.21.15.16:0/612327367 wait complete. 2024-04-22T15:54:21.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.230+0000 7f750020a700 1 -- 172.21.15.16:0/3884195075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74f8100750 msgr2=0x7f74f81047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:21.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.230+0000 7f750020a700 1 --2- 172.21.15.16:0/3884195075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74f8100750 0x7f74f81047a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f74ec009910 tx=0x7f74ec009c20 comp rx=0 tx=0).stop 2024-04-22T15:54:21.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.230+0000 7f750020a700 1 -- 172.21.15.16:0/3884195075 shutdown_connections 2024-04-22T15:54:21.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.230+0000 7f750020a700 1 --2- 172.21.15.16:0/3884195075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74f8100750 0x7f74f81047a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.230+0000 7f750020a700 1 --2- 172.21.15.16:0/3884195075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74f80ffda0 0x7f74f8100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.230+0000 7f750020a700 1 -- 172.21.15.16:0/3884195075 >> 172.21.15.16:0/3884195075 conn(0x7f74f80fb650 msgr2=0x7f74f80fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:21.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.230+0000 7f750020a700 1 -- 172.21.15.16:0/3884195075 shutdown_connections 2024-04-22T15:54:21.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.230+0000 7f750020a700 1 -- 172.21.15.16:0/3884195075 wait complete. 2024-04-22T15:54:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.231+0000 7f750020a700 1 Processor -- start 2024-04-22T15:54:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.231+0000 7f750020a700 1 -- start start 2024-04-22T15:54:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.231+0000 7f750020a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74f80ffda0 0x7f74f8197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.231+0000 7f750020a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74f8100750 0x7f74f8197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.231+0000 7f750020a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f8198380 con 0x7f74f8100750 2024-04-22T15:54:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.231+0000 7f750020a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f81984c0 con 0x7f74f80ffda0 2024-04-22T15:54:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.231+0000 7f74fd7a5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74f8100750 0x7f74f8197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.231+0000 7f74fd7a5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74f8100750 0x7f74f8197ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48480/0 (socket says 172.21.15.16:48480) 2024-04-22T15:54:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.231+0000 7f74fd7a5700 1 -- 172.21.15.16:0/99798694 learned_addr learned my addr 172.21.15.16:0/99798694 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:54:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.231+0000 7f74fdfa6700 1 --2- 172.21.15.16:0/99798694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74f80ffda0 0x7f74f8197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.232+0000 7f74fd7a5700 1 -- 172.21.15.16:0/99798694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74f80ffda0 msgr2=0x7f74f8197760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.232+0000 7f74fd7a5700 1 --2- 172.21.15.16:0/99798694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74f80ffda0 0x7f74f8197760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.232+0000 7f74fd7a5700 1 -- 172.21.15.16:0/99798694 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ec0095f0 con 0x7f74f8100750 2024-04-22T15:54:21.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.232+0000 7f74fd7a5700 1 --2- 172.21.15.16:0/99798694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74f8100750 0x7f74f8197ca0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f74ec0092e0 tx=0x7f74ec00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:21.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.232+0000 7f74f6ffd700 1 -- 172.21.15.16:0/99798694 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74ec01d070 con 0x7f74f8100750 2024-04-22T15:54:21.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.232+0000 7f750020a700 1 -- 172.21.15.16:0/99798694 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f81a07d0 con 0x7f74f8100750 2024-04-22T15:54:21.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.232+0000 7f750020a700 1 -- 172.21.15.16:0/99798694 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f81a0c40 con 0x7f74f8100750 2024-04-22T15:54:21.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.232+0000 7f74f6ffd700 1 -- 172.21.15.16:0/99798694 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f74ec00f460 con 0x7f74f8100750 2024-04-22T15:54:21.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.232+0000 7f74f6ffd700 1 -- 172.21.15.16:0/99798694 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74ec02b670 con 0x7f74f8100750 2024-04-22T15:54:21.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.233+0000 7f750020a700 1 -- 172.21.15.16:0/99798694 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74e4005320 con 0x7f74f8100750 2024-04-22T15:54:21.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.233+0000 7f74f6ffd700 1 -- 172.21.15.16:0/99798694 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f74ec003a60 con 0x7f74f8100750 2024-04-22T15:54:21.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.234+0000 7f74f6ffd700 1 --2- 172.21.15.16:0/99798694 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f74e006ee00 0x7f74e00712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:21.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.234+0000 7f74f6ffd700 1 -- 172.21.15.16:0/99798694 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f74ec0903d0 con 0x7f74f8100750 2024-04-22T15:54:21.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.234+0000 7f74fdfa6700 1 --2- 172.21.15.16:0/99798694 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f74e006ee00 0x7f74e00712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:21.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.234+0000 7f74fdfa6700 1 --2- 172.21.15.16:0/99798694 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f74e006ee00 0x7f74e00712c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f74e8005950 tx=0x7f74e800a300 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:21.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.239+0000 7f74f6ffd700 1 -- 172.21.15.16:0/99798694 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f74ec059010 con 0x7f74f8100750 2024-04-22T15:54:21.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:21 smithi016 ceph-mon[110740]: pgmap v28: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:21.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:21 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:21.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:21 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:21.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:21 smithi016 ceph-mon[110740]: Reconfiguring osd.4 (monmap changed)... 2024-04-22T15:54:21.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:21 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T15:54:21.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:21 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:21.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:21 smithi016 ceph-mon[110740]: Reconfiguring daemon osd.4 on smithi027 2024-04-22T15:54:21.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:21 smithi016 ceph-mon[110740]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:21.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:21 smithi016 ceph-mon[110740]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:21.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.539+0000 7f750020a700 1 -- 172.21.15.16:0/99798694 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f74e4005cc0 con 0x7f74f8100750 2024-04-22T15:54:21.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.540+0000 7f74f6ffd700 1 -- 172.21.15.16:0/99798694 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+688 (secure 0 0 0) 0x7f74ec05c6d0 con 0x7f74f8100750 2024-04-22T15:54:21.542 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:54:21.542 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-22T15:54:21.542 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:54:21.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:54:21.542 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-22T15:54:21.542 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:54:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:54:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-22T15:54:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-22T15:54:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:54:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-22T15:54:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T15:54:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:54:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-22T15:54:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 10, 2024-04-22T15:54:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:54:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-22T15:54:21.543 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:54:21.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.542+0000 7f750020a700 1 -- 172.21.15.16:0/99798694 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f74e006ee00 msgr2=0x7f74e00712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:21.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.542+0000 7f750020a700 1 --2- 172.21.15.16:0/99798694 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f74e006ee00 0x7f74e00712c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f74e8005950 tx=0x7f74e800a300 comp rx=0 tx=0).stop 2024-04-22T15:54:21.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.542+0000 7f750020a700 1 -- 172.21.15.16:0/99798694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74f8100750 msgr2=0x7f74f8197ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:21.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.542+0000 7f750020a700 1 --2- 172.21.15.16:0/99798694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74f8100750 0x7f74f8197ca0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f74ec0092e0 tx=0x7f74ec00bd70 comp rx=0 tx=0).stop 2024-04-22T15:54:21.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.542+0000 7f750020a700 1 -- 172.21.15.16:0/99798694 shutdown_connections 2024-04-22T15:54:21.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.542+0000 7f750020a700 1 --2- 172.21.15.16:0/99798694 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f74e006ee00 0x7f74e00712c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.542+0000 7f750020a700 1 --2- 172.21.15.16:0/99798694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74f80ffda0 0x7f74f8197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.542+0000 7f750020a700 1 --2- 172.21.15.16:0/99798694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74f8100750 0x7f74f8197ca0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.542+0000 7f750020a700 1 -- 172.21.15.16:0/99798694 >> 172.21.15.16:0/99798694 conn(0x7f74f80fb650 msgr2=0x7f74f80fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:21.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.542+0000 7f750020a700 1 -- 172.21.15.16:0/99798694 shutdown_connections 2024-04-22T15:54:21.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.542+0000 7f750020a700 1 -- 172.21.15.16:0/99798694 wait complete. 2024-04-22T15:54:21.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:21 smithi027 ceph-mon[83054]: pgmap v28: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:21.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:21 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:21.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:21 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:21.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:21 smithi027 ceph-mon[83054]: Reconfiguring osd.4 (monmap changed)... 2024-04-22T15:54:21.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:21 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T15:54:21.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:21 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:21.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:21 smithi027 ceph-mon[83054]: Reconfiguring daemon osd.4 on smithi027 2024-04-22T15:54:21.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:21 smithi027 ceph-mon[83054]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:21.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:21 smithi027 ceph-mon[83054]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:21.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.648+0000 7f38ddfe2700 1 -- 172.21.15.16:0/568354071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38d8101850 msgr2=0x7f38d81058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:21.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.648+0000 7f38ddfe2700 1 --2- 172.21.15.16:0/568354071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38d8101850 0x7f38d81058a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f38cc009910 tx=0x7f38cc009c20 comp rx=0 tx=0).stop 2024-04-22T15:54:21.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.649+0000 7f38ddfe2700 1 -- 172.21.15.16:0/568354071 shutdown_connections 2024-04-22T15:54:21.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.649+0000 7f38ddfe2700 1 --2- 172.21.15.16:0/568354071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38d8101850 0x7f38d81058a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.649+0000 7f38ddfe2700 1 --2- 172.21.15.16:0/568354071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38d8100ea0 0x7f38d8101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.649+0000 7f38ddfe2700 1 -- 172.21.15.16:0/568354071 >> 172.21.15.16:0/568354071 conn(0x7f38d80fc750 msgr2=0x7f38d80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:21.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.649+0000 7f38ddfe2700 1 -- 172.21.15.16:0/568354071 shutdown_connections 2024-04-22T15:54:21.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.649+0000 7f38ddfe2700 1 -- 172.21.15.16:0/568354071 wait complete. 2024-04-22T15:54:21.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.649+0000 7f38ddfe2700 1 Processor -- start 2024-04-22T15:54:21.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.649+0000 7f38ddfe2700 1 -- start start 2024-04-22T15:54:21.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.650+0000 7f38ddfe2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38d8100ea0 0x7f38d819c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:21.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.650+0000 7f38ddfe2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38d8101850 0x7f38d819cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:21.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.650+0000 7f38ddfe2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d819d2d0 con 0x7f38d8100ea0 2024-04-22T15:54:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.650+0000 7f38ddfe2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d8196780 con 0x7f38d8101850 2024-04-22T15:54:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.650+0000 7f38d77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38d8100ea0 0x7f38d819c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.650+0000 7f38d77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38d8100ea0 0x7f38d819c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40772/0 (socket says 172.21.15.16:40772) 2024-04-22T15:54:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.650+0000 7f38d77fe700 1 -- 172.21.15.16:0/3446139948 learned_addr learned my addr 172.21.15.16:0/3446139948 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:54:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.650+0000 7f38d6ffd700 1 --2- 172.21.15.16:0/3446139948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38d8101850 0x7f38d819cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.650+0000 7f38d77fe700 1 -- 172.21.15.16:0/3446139948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38d8101850 msgr2=0x7f38d819cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.650+0000 7f38d77fe700 1 --2- 172.21.15.16:0/3446139948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38d8101850 0x7f38d819cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.650+0000 7f38d77fe700 1 -- 172.21.15.16:0/3446139948 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38cc0095f0 con 0x7f38d8100ea0 2024-04-22T15:54:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.650+0000 7f38d77fe700 1 --2- 172.21.15.16:0/3446139948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38d8100ea0 0x7f38d819c700 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f38c800c8a0 tx=0x7f38c800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.650+0000 7f38d4ff9700 1 -- 172.21.15.16:0/3446139948 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38c800cea0 con 0x7f38d8100ea0 2024-04-22T15:54:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.651+0000 7f38d4ff9700 1 -- 172.21.15.16:0/3446139948 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f38c8004830 con 0x7f38d8100ea0 2024-04-22T15:54:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.651+0000 7f38ddfe2700 1 -- 172.21.15.16:0/3446139948 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d8196a60 con 0x7f38d8100ea0 2024-04-22T15:54:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.651+0000 7f38d4ff9700 1 -- 172.21.15.16:0/3446139948 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38c80055b0 con 0x7f38d8100ea0 2024-04-22T15:54:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.651+0000 7f38ddfe2700 1 -- 172.21.15.16:0/3446139948 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d8196fb0 con 0x7f38d8100ea0 2024-04-22T15:54:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.651+0000 7f38ddfe2700 1 -- 172.21.15.16:0/3446139948 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d81092a0 con 0x7f38d8100ea0 2024-04-22T15:54:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.652+0000 7f38d4ff9700 1 -- 172.21.15.16:0/3446139948 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f38c80049a0 con 0x7f38d8100ea0 2024-04-22T15:54:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.653+0000 7f38d4ff9700 1 --2- 172.21.15.16:0/3446139948 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f38c006ee00 0x7f38c00712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.653+0000 7f38d4ff9700 1 -- 172.21.15.16:0/3446139948 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f38c808ec50 con 0x7f38d8100ea0 2024-04-22T15:54:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.653+0000 7f38d6ffd700 1 --2- 172.21.15.16:0/3446139948 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f38c006ee00 0x7f38c00712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.653+0000 7f38d6ffd700 1 --2- 172.21.15.16:0/3446139948 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f38c006ee00 0x7f38c00712c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f38d8197bf0 tx=0x7f38cc01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:21.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.657+0000 7f38d4ff9700 1 -- 172.21.15.16:0/3446139948 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f38c805b440 con 0x7f38d8100ea0 2024-04-22T15:54:21.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.913+0000 7f38ddfe2700 1 -- 172.21.15.16:0/3446139948 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f38d804f2e0 con 0x7f38d8100ea0 2024-04-22T15:54:21.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.913+0000 7f38d4ff9700 1 -- 172.21.15.16:0/3446139948 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1902 (secure 0 0 0) 0x7f38c801d070 con 0x7f38d8100ea0 2024-04-22T15:54:21.916 INFO:teuthology.orchestra.run.smithi016.stdout:e16 2024-04-22T15:54:21.916 INFO:teuthology.orchestra.run.smithi016.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:54:21.916 INFO:teuthology.orchestra.run.smithi016.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:54:21.916 INFO:teuthology.orchestra.run.smithi016.stdout:legacy client fscid: 1 2024-04-22T15:54:21.916 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:54:21.916 INFO:teuthology.orchestra.run.smithi016.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:54:21.916 INFO:teuthology.orchestra.run.smithi016.stdout:fs_name cephfs 2024-04-22T15:54:21.916 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 16 2024-04-22T15:54:21.916 INFO:teuthology.orchestra.run.smithi016.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T15:54:21.916 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-04-22T15:46:27.774571+0000 2024-04-22T15:54:21.916 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-04-22T15:46:47.381709+0000 2024-04-22T15:54:21.917 INFO:teuthology.orchestra.run.smithi016.stdout:tableserver 0 2024-04-22T15:54:21.917 INFO:teuthology.orchestra.run.smithi016.stdout:root 0 2024-04-22T15:54:21.917 INFO:teuthology.orchestra.run.smithi016.stdout:session_timeout 60 2024-04-22T15:54:21.917 INFO:teuthology.orchestra.run.smithi016.stdout:session_autoclose 300 2024-04-22T15:54:21.917 INFO:teuthology.orchestra.run.smithi016.stdout:max_file_size 1099511627776 2024-04-22T15:54:21.917 INFO:teuthology.orchestra.run.smithi016.stdout:max_xattr_size 65536 2024-04-22T15:54:21.917 INFO:teuthology.orchestra.run.smithi016.stdout:required_client_features {} 2024-04-22T15:54:21.917 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure 0 2024-04-22T15:54:21.917 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure_osd_epoch 46 2024-04-22T15:54:21.917 INFO:teuthology.orchestra.run.smithi016.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:54:21.917 INFO:teuthology.orchestra.run.smithi016.stdout:max_mds 1 2024-04-22T15:54:21.917 INFO:teuthology.orchestra.run.smithi016.stdout:in 0 2024-04-22T15:54:21.917 INFO:teuthology.orchestra.run.smithi016.stdout:up {0=24279} 2024-04-22T15:54:21.917 INFO:teuthology.orchestra.run.smithi016.stdout:failed 2024-04-22T15:54:21.918 INFO:teuthology.orchestra.run.smithi016.stdout:damaged 2024-04-22T15:54:21.918 INFO:teuthology.orchestra.run.smithi016.stdout:stopped 2024-04-22T15:54:21.918 INFO:teuthology.orchestra.run.smithi016.stdout:data_pools [3] 2024-04-22T15:54:21.918 INFO:teuthology.orchestra.run.smithi016.stdout:metadata_pool 2 2024-04-22T15:54:21.918 INFO:teuthology.orchestra.run.smithi016.stdout:inline_data enabled 2024-04-22T15:54:21.918 INFO:teuthology.orchestra.run.smithi016.stdout:balancer 2024-04-22T15:54:21.918 INFO:teuthology.orchestra.run.smithi016.stdout:bal_rank_mask -1 2024-04-22T15:54:21.918 INFO:teuthology.orchestra.run.smithi016.stdout:standby_count_wanted 1 2024-04-22T15:54:21.918 INFO:teuthology.orchestra.run.smithi016.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:54:21.918 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.iobjds{0:24279} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.27:6824/4216277609,v1:172.21.15.27:6825/4216277609] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:54:21.918 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.pgvphu{0:14494} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.16:6828/1006073414,v1:172.21.15.16:6829/1006073414] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:54:21.918 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:54:21.919 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:54:21.919 INFO:teuthology.orchestra.run.smithi016.stdout:Standby daemons: 2024-04-22T15:54:21.919 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:54:21.919 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.fkkejm{-1:14504} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.16:6826/3484907755,v1:172.21.15.16:6827/3484907755] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:54:21.919 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.dgqyes{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.27:6826/476026355,v1:172.21.15.27:6827/476026355] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:54:21.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.917+0000 7f38ddfe2700 1 -- 172.21.15.16:0/3446139948 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f38c006ee00 msgr2=0x7f38c00712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.917+0000 7f38ddfe2700 1 --2- 172.21.15.16:0/3446139948 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f38c006ee00 0x7f38c00712c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f38d8197bf0 tx=0x7f38cc01a040 comp rx=0 tx=0).stop 2024-04-22T15:54:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.917+0000 7f38ddfe2700 1 -- 172.21.15.16:0/3446139948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38d8100ea0 msgr2=0x7f38d819c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.917+0000 7f38ddfe2700 1 --2- 172.21.15.16:0/3446139948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38d8100ea0 0x7f38d819c700 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f38c800c8a0 tx=0x7f38c800cc60 comp rx=0 tx=0).stop 2024-04-22T15:54:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.917+0000 7f38ddfe2700 1 -- 172.21.15.16:0/3446139948 shutdown_connections 2024-04-22T15:54:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.917+0000 7f38ddfe2700 1 --2- 172.21.15.16:0/3446139948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38d8100ea0 0x7f38d819c700 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.917+0000 7f38ddfe2700 1 --2- 172.21.15.16:0/3446139948 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f38c006ee00 0x7f38c00712c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.917+0000 7f38ddfe2700 1 --2- 172.21.15.16:0/3446139948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38d8101850 0x7f38d819cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.917+0000 7f38ddfe2700 1 -- 172.21.15.16:0/3446139948 >> 172.21.15.16:0/3446139948 conn(0x7f38d80fc750 msgr2=0x7f38d80fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.917+0000 7f38ddfe2700 1 -- 172.21.15.16:0/3446139948 shutdown_connections 2024-04-22T15:54:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:21.917+0000 7f38ddfe2700 1 -- 172.21.15.16:0/3446139948 wait complete. 2024-04-22T15:54:21.920 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 16 2024-04-22T15:54:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.024+0000 7f5a7c3d4700 1 -- 172.21.15.16:0/1637059762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a74101850 msgr2=0x7f5a741058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.024+0000 7f5a7c3d4700 1 --2- 172.21.15.16:0/1637059762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a74101850 0x7f5a741058a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5a64009910 tx=0x7f5a64009c20 comp rx=0 tx=0).stop 2024-04-22T15:54:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.025+0000 7f5a7c3d4700 1 -- 172.21.15.16:0/1637059762 shutdown_connections 2024-04-22T15:54:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.025+0000 7f5a7c3d4700 1 --2- 172.21.15.16:0/1637059762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a74101850 0x7f5a741058a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.025+0000 7f5a7c3d4700 1 --2- 172.21.15.16:0/1637059762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a74100ea0 0x7f5a74101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.025+0000 7f5a7c3d4700 1 -- 172.21.15.16:0/1637059762 >> 172.21.15.16:0/1637059762 conn(0x7f5a740fc750 msgr2=0x7f5a740feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.025+0000 7f5a7c3d4700 1 -- 172.21.15.16:0/1637059762 shutdown_connections 2024-04-22T15:54:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.025+0000 7f5a7c3d4700 1 -- 172.21.15.16:0/1637059762 wait complete. 2024-04-22T15:54:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.025+0000 7f5a7c3d4700 1 Processor -- start 2024-04-22T15:54:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.025+0000 7f5a7c3d4700 1 -- start start 2024-04-22T15:54:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.026+0000 7f5a7c3d4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a74100ea0 0x7f5a7419c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.026+0000 7f5a7c3d4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a74101850 0x7f5a7419cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.026+0000 7f5a7c3d4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7419d2d0 con 0x7f5a74101850 2024-04-22T15:54:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.026+0000 7f5a7c3d4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a74196780 con 0x7f5a74100ea0 2024-04-22T15:54:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.026+0000 7f5a7996f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a74101850 0x7f5a7419cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.026+0000 7f5a7996f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a74101850 0x7f5a7419cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40792/0 (socket says 172.21.15.16:40792) 2024-04-22T15:54:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.026+0000 7f5a7996f700 1 -- 172.21.15.16:0/903278697 learned_addr learned my addr 172.21.15.16:0/903278697 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:54:22.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.026+0000 7f5a7a170700 1 --2- 172.21.15.16:0/903278697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a74100ea0 0x7f5a7419c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:22.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.026+0000 7f5a7996f700 1 -- 172.21.15.16:0/903278697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a74100ea0 msgr2=0x7f5a7419c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:22.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.026+0000 7f5a7996f700 1 --2- 172.21.15.16:0/903278697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a74100ea0 0x7f5a7419c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:22.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.026+0000 7f5a7996f700 1 -- 172.21.15.16:0/903278697 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a640095f0 con 0x7f5a74101850 2024-04-22T15:54:22.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.026+0000 7f5a7996f700 1 --2- 172.21.15.16:0/903278697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a74101850 0x7f5a7419cc40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5a64009fd0 tx=0x7f5a6400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:22.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.026+0000 7f5a6b7fe700 1 -- 172.21.15.16:0/903278697 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a6401d070 con 0x7f5a74101850 2024-04-22T15:54:22.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.026+0000 7f5a6b7fe700 1 -- 172.21.15.16:0/903278697 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a64022470 con 0x7f5a74101850 2024-04-22T15:54:22.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.026+0000 7f5a7c3d4700 1 -- 172.21.15.16:0/903278697 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a74196a00 con 0x7f5a74101850 2024-04-22T15:54:22.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.026+0000 7f5a6b7fe700 1 -- 172.21.15.16:0/903278697 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a64017650 con 0x7f5a74101850 2024-04-22T15:54:22.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.026+0000 7f5a7c3d4700 1 -- 172.21.15.16:0/903278697 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a74196e90 con 0x7f5a74101850 2024-04-22T15:54:22.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.028+0000 7f5a7c3d4700 1 -- 172.21.15.16:0/903278697 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a741092a0 con 0x7f5a74101850 2024-04-22T15:54:22.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.028+0000 7f5a6b7fe700 1 -- 172.21.15.16:0/903278697 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f5a64005080 con 0x7f5a74101850 2024-04-22T15:54:22.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.029+0000 7f5a6b7fe700 1 --2- 172.21.15.16:0/903278697 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f5a6006edf0 0x7f5a600712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:22.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.029+0000 7f5a6b7fe700 1 -- 172.21.15.16:0/903278697 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f5a64090470 con 0x7f5a74101850 2024-04-22T15:54:22.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.029+0000 7f5a7a170700 1 --2- 172.21.15.16:0/903278697 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f5a6006edf0 0x7f5a600712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:22.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.029+0000 7f5a7a170700 1 --2- 172.21.15.16:0/903278697 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f5a6006edf0 0x7f5a600712b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5a70009c60 tx=0x7f5a70009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:22.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.033+0000 7f5a6b7fe700 1 -- 172.21.15.16:0/903278697 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5a6405cc60 con 0x7f5a74101850 2024-04-22T15:54:22.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.269+0000 7f5a7c3d4700 1 -- 172.21.15.16:0/903278697 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a74197160 con 0x7f5a6006edf0 2024-04-22T15:54:22.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.270+0000 7f5a6b7fe700 1 -- 172.21.15.16:0/903278697 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f5a74197160 con 0x7f5a6006edf0 2024-04-22T15:54:22.272 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:54:22.272 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:54:22.272 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-22T15:54:22.272 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:54:22.272 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-22T15:54:22.272 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr", 2024-04-22T15:54:22.273 INFO:teuthology.orchestra.run.smithi016.stdout: "mon" 2024-04-22T15:54:22.273 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-22T15:54:22.273 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "4/23 daemons upgraded", 2024-04-22T15:54:22.273 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading mon daemons", 2024-04-22T15:54:22.273 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-22T15:54:22.273 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:54:22.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.272+0000 7f5a7c3d4700 1 -- 172.21.15.16:0/903278697 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f5a6006edf0 msgr2=0x7f5a600712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:22.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.272+0000 7f5a7c3d4700 1 --2- 172.21.15.16:0/903278697 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f5a6006edf0 0x7f5a600712b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5a70009c60 tx=0x7f5a70009260 comp rx=0 tx=0).stop 2024-04-22T15:54:22.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.272+0000 7f5a7c3d4700 1 -- 172.21.15.16:0/903278697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a74101850 msgr2=0x7f5a7419cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:22.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.272+0000 7f5a7c3d4700 1 --2- 172.21.15.16:0/903278697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a74101850 0x7f5a7419cc40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5a64009fd0 tx=0x7f5a6400f690 comp rx=0 tx=0).stop 2024-04-22T15:54:22.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.272+0000 7f5a7c3d4700 1 -- 172.21.15.16:0/903278697 shutdown_connections 2024-04-22T15:54:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.272+0000 7f5a7c3d4700 1 --2- 172.21.15.16:0/903278697 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f5a6006edf0 0x7f5a600712b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.272+0000 7f5a7c3d4700 1 --2- 172.21.15.16:0/903278697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a74100ea0 0x7f5a7419c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.272+0000 7f5a7c3d4700 1 --2- 172.21.15.16:0/903278697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a74101850 0x7f5a7419cc40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.272+0000 7f5a7c3d4700 1 -- 172.21.15.16:0/903278697 >> 172.21.15.16:0/903278697 conn(0x7f5a740fc750 msgr2=0x7f5a740fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.272+0000 7f5a7c3d4700 1 -- 172.21.15.16:0/903278697 shutdown_connections 2024-04-22T15:54:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.272+0000 7f5a7c3d4700 1 -- 172.21.15.16:0/903278697 wait complete. 2024-04-22T15:54:22.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.380+0000 7f59d9e42700 1 -- 172.21.15.16:0/4088440656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59d4101830 msgr2=0x7f59d4105880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:22.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.380+0000 7f59d9e42700 1 --2- 172.21.15.16:0/4088440656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59d4101830 0x7f59d4105880 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f59c8009910 tx=0x7f59c8009c20 comp rx=0 tx=0).stop 2024-04-22T15:54:22.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.380+0000 7f59d9e42700 1 -- 172.21.15.16:0/4088440656 shutdown_connections 2024-04-22T15:54:22.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.380+0000 7f59d9e42700 1 --2- 172.21.15.16:0/4088440656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59d4101830 0x7f59d4105880 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:22.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.380+0000 7f59d9e42700 1 --2- 172.21.15.16:0/4088440656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59d4100e80 0x7f59d4101260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:22.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.380+0000 7f59d9e42700 1 -- 172.21.15.16:0/4088440656 >> 172.21.15.16:0/4088440656 conn(0x7f59d40fc730 msgr2=0x7f59d40feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:22.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.380+0000 7f59d9e42700 1 -- 172.21.15.16:0/4088440656 shutdown_connections 2024-04-22T15:54:22.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.380+0000 7f59d9e42700 1 -- 172.21.15.16:0/4088440656 wait complete. 2024-04-22T15:54:22.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.381+0000 7f59d9e42700 1 Processor -- start 2024-04-22T15:54:22.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.381+0000 7f59d9e42700 1 -- start start 2024-04-22T15:54:22.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.381+0000 7f59d9e42700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59d4100e80 0x7f59d419e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.381+0000 7f59d9e42700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59d4101830 0x7f59d419edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.381+0000 7f59d9e42700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d419f450 con 0x7f59d4101830 2024-04-22T15:54:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.381+0000 7f59d9e42700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d4198900 con 0x7f59d4100e80 2024-04-22T15:54:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.381+0000 7f59d2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59d4101830 0x7f59d419edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.382+0000 7f59d2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59d4101830 0x7f59d419edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40812/0 (socket says 172.21.15.16:40812) 2024-04-22T15:54:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.382+0000 7f59d2ffd700 1 -- 172.21.15.16:0/1632682988 learned_addr learned my addr 172.21.15.16:0/1632682988 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:54:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.382+0000 7f59d37fe700 1 --2- 172.21.15.16:0/1632682988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59d4100e80 0x7f59d419e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.382+0000 7f59d2ffd700 1 -- 172.21.15.16:0/1632682988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59d4100e80 msgr2=0x7f59d419e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.382+0000 7f59d2ffd700 1 --2- 172.21.15.16:0/1632682988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59d4100e80 0x7f59d419e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.382+0000 7f59d2ffd700 1 -- 172.21.15.16:0/1632682988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c80095f0 con 0x7f59d4101830 2024-04-22T15:54:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.382+0000 7f59d37fe700 1 --2- 172.21.15.16:0/1632682988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59d4100e80 0x7f59d419e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:54:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.382+0000 7f59d2ffd700 1 --2- 172.21.15.16:0/1632682988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59d4101830 0x7f59d419edc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f59c80059d0 tx=0x7f59c800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.382+0000 7f59d0ff9700 1 -- 172.21.15.16:0/1632682988 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59c801d070 con 0x7f59d4101830 2024-04-22T15:54:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.382+0000 7f59d0ff9700 1 -- 172.21.15.16:0/1632682988 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59c800b610 con 0x7f59d4101830 2024-04-22T15:54:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.382+0000 7f59d9e42700 1 -- 172.21.15.16:0/1632682988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d4198b80 con 0x7f59d4101830 2024-04-22T15:54:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.382+0000 7f59d0ff9700 1 -- 172.21.15.16:0/1632682988 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59c8022470 con 0x7f59d4101830 2024-04-22T15:54:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.382+0000 7f59d9e42700 1 -- 172.21.15.16:0/1632682988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d4199070 con 0x7f59d4101830 2024-04-22T15:54:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.383+0000 7f59d9e42700 1 -- 172.21.15.16:0/1632682988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d4105000 con 0x7f59d4101830 2024-04-22T15:54:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.384+0000 7f59d0ff9700 1 -- 172.21.15.16:0/1632682988 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f59c8003710 con 0x7f59d4101830 2024-04-22T15:54:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.384+0000 7f59d0ff9700 1 --2- 172.21.15.16:0/1632682988 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f59bc06ee00 0x7f59bc0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.384+0000 7f59d0ff9700 1 -- 172.21.15.16:0/1632682988 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f59c80904d0 con 0x7f59d4101830 2024-04-22T15:54:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.385+0000 7f59d37fe700 1 --2- 172.21.15.16:0/1632682988 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f59bc06ee00 0x7f59bc0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.385+0000 7f59d37fe700 1 --2- 172.21.15.16:0/1632682988 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f59bc06ee00 0x7f59bc0712c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f59c4011490 tx=0x7f59c4009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:22.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.389+0000 7f59d0ff9700 1 -- 172.21.15.16:0/1632682988 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f59c805cc20 con 0x7f59d4101830 2024-04-22T15:54:22.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:22 smithi027 ceph-mon[83054]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:22.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:22 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/99798694' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:54:22.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:22 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/3446139948' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:54:22.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:22 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:22.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:22 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:22.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:22 smithi027 ceph-mon[83054]: Reconfiguring osd.5 (monmap changed)... 2024-04-22T15:54:22.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:22 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T15:54:22.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:22 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:22.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:22 smithi027 ceph-mon[83054]: Reconfiguring daemon osd.5 on smithi027 2024-04-22T15:54:22.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:22 smithi016 ceph-mon[110740]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:22.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:22 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/99798694' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:54:22.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:22 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/3446139948' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:54:22.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:22 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:22.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:22 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:22.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:22 smithi016 ceph-mon[110740]: Reconfiguring osd.5 (monmap changed)... 2024-04-22T15:54:22.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:22 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T15:54:22.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:22 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:22.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:22 smithi016 ceph-mon[110740]: Reconfiguring daemon osd.5 on smithi027 2024-04-22T15:54:22.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.687+0000 7f59d9e42700 1 -- 172.21.15.16:0/1632682988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f59d4199e10 con 0x7f59d4101830 2024-04-22T15:54:22.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.687+0000 7f59d0ff9700 1 -- 172.21.15.16:0/1632682988 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f59c805c7b0 con 0x7f59d4101830 2024-04-22T15:54:22.689 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:54:22.689 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:54:22.690 INFO:teuthology.orchestra.run.smithi016.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:54:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.689+0000 7f59d9e42700 1 -- 172.21.15.16:0/1632682988 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f59bc06ee00 msgr2=0x7f59bc0712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.689+0000 7f59d9e42700 1 --2- 172.21.15.16:0/1632682988 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f59bc06ee00 0x7f59bc0712c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f59c4011490 tx=0x7f59c4009260 comp rx=0 tx=0).stop 2024-04-22T15:54:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.689+0000 7f59d9e42700 1 -- 172.21.15.16:0/1632682988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59d4101830 msgr2=0x7f59d419edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.689+0000 7f59d9e42700 1 --2- 172.21.15.16:0/1632682988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59d4101830 0x7f59d419edc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f59c80059d0 tx=0x7f59c800f690 comp rx=0 tx=0).stop 2024-04-22T15:54:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.690+0000 7f59d9e42700 1 -- 172.21.15.16:0/1632682988 shutdown_connections 2024-04-22T15:54:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.690+0000 7f59d9e42700 1 --2- 172.21.15.16:0/1632682988 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f59bc06ee00 0x7f59bc0712c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.690+0000 7f59d9e42700 1 --2- 172.21.15.16:0/1632682988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59d4100e80 0x7f59d419e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.690+0000 7f59d9e42700 1 --2- 172.21.15.16:0/1632682988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59d4101830 0x7f59d419edc0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.690+0000 7f59d9e42700 1 -- 172.21.15.16:0/1632682988 >> 172.21.15.16:0/1632682988 conn(0x7f59d40fc730 msgr2=0x7f59d40feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.690+0000 7f59d9e42700 1 -- 172.21.15.16:0/1632682988 shutdown_connections 2024-04-22T15:54:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:22.690+0000 7f59d9e42700 1 -- 172.21.15.16:0/1632682988 wait complete. 2024-04-22T15:54:23.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:23 smithi027 ceph-mon[83054]: pgmap v29: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:54:23.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:23 smithi027 ceph-mon[83054]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:23.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:23 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/1632682988' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:54:23.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:23 smithi016 ceph-mon[110740]: pgmap v29: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:54:23.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:23 smithi016 ceph-mon[110740]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:23.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:23 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/1632682988' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:54:25.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:24 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:25.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:24 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:25.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:24 smithi027 ceph-mon[83054]: Reconfiguring mds.cephfs.smithi027.iobjds (monmap changed)... 2024-04-22T15:54:25.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:24 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.iobjds", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:54:25.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:24 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:25.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:24 smithi027 ceph-mon[83054]: Reconfiguring daemon mds.cephfs.smithi027.iobjds on smithi027 2024-04-22T15:54:25.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:24 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:25.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:24 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:25.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:24 smithi016 ceph-mon[110740]: Reconfiguring mds.cephfs.smithi027.iobjds (monmap changed)... 2024-04-22T15:54:25.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:24 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.iobjds", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:54:25.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:24 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:25.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:24 smithi016 ceph-mon[110740]: Reconfiguring daemon mds.cephfs.smithi027.iobjds on smithi027 2024-04-22T15:54:26.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:25 smithi016 ceph-mon[110740]: pgmap v30: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:26.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:25 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:26.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:25 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:26.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:25 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.dgqyes", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:54:26.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:25 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:26.207 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:25 smithi027 ceph-mon[83054]: pgmap v30: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:26.207 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:25 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:26.207 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:25 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:26.207 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:25 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.dgqyes", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:54:26.207 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:25 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:26.978 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:26 smithi027 ceph-mon[83054]: Reconfiguring mds.cephfs.smithi027.dgqyes (monmap changed)... 2024-04-22T15:54:26.978 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:26 smithi027 ceph-mon[83054]: Reconfiguring daemon mds.cephfs.smithi027.dgqyes on smithi027 2024-04-22T15:54:27.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:26 smithi016 ceph-mon[110740]: Reconfiguring mds.cephfs.smithi027.dgqyes (monmap changed)... 2024-04-22T15:54:27.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:26 smithi016 ceph-mon[110740]: Reconfiguring daemon mds.cephfs.smithi027.dgqyes on smithi027 2024-04-22T15:54:28.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:28 smithi027 ceph-mon[83054]: pgmap v31: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:54:28.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:28 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:28.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:28 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:28.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:28 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:54:28.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:28 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:54:28.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:28 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:54:28.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:28 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:28.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:28 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi016"}]: dispatch 2024-04-22T15:54:28.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:28 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi016"}]': finished 2024-04-22T15:54:28.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:28 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi027"}]: dispatch 2024-04-22T15:54:28.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:28 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi027"}]': finished 2024-04-22T15:54:28.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:28 smithi016 ceph-mon[110740]: pgmap v31: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:54:28.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:28 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:28.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:28 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:28.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:28 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:54:28.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:28 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:54:28.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:28 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:54:28.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:28 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:28.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:28 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi016"}]: dispatch 2024-04-22T15:54:28.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:28 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi016"}]': finished 2024-04-22T15:54:28.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:28 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi027"}]: dispatch 2024-04-22T15:54:28.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:28 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi027"}]': finished 2024-04-22T15:54:29.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:29 smithi027 ceph-mon[83054]: Upgrade: Setting container_image for all mon 2024-04-22T15:54:29.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:29 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:29.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:29 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi016", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:54:29.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:29 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:29.381 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:29 smithi016 ceph-mon[110740]: Upgrade: Setting container_image for all mon 2024-04-22T15:54:29.381 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:29 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:29.381 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:29 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi016", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:54:29.381 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:29 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:30.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:30 smithi027 ceph-mon[83054]: pgmap v32: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:30.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:30 smithi027 ceph-mon[83054]: Upgrade: Updating crash.smithi016 (1/2) 2024-04-22T15:54:30.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:30 smithi027 ceph-mon[83054]: Deploying daemon crash.smithi016 on smithi016 2024-04-22T15:54:30.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:30 smithi016 ceph-mon[110740]: pgmap v32: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:30.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:30 smithi016 ceph-mon[110740]: Upgrade: Updating crash.smithi016 (1/2) 2024-04-22T15:54:30.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:30 smithi016 ceph-mon[110740]: Deploying daemon crash.smithi016 on smithi016 2024-04-22T15:54:31.242 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:31 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:54:31.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:31 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:54:32.325 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:32 smithi016 ceph-mon[110740]: pgmap v33: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:32.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:32 smithi027 ceph-mon[83054]: pgmap v33: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:33.813 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:33 smithi016 ceph-mon[110740]: pgmap v34: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:54:33.813 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:33 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:33.813 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:33 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:33.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:33 smithi027 ceph-mon[83054]: pgmap v34: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:54:33.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:33 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:33.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:33 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:35.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:35 smithi016 ceph-mon[110740]: Upgrade: Updating crash.smithi027 (2/2) 2024-04-22T15:54:35.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:35 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:35.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:35 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi027", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:54:35.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:35 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:35.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:35 smithi016 ceph-mon[110740]: Deploying daemon crash.smithi027 on smithi027 2024-04-22T15:54:35.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:35 smithi016 ceph-mon[110740]: pgmap v35: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:35.458 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:35 smithi027 ceph-mon[83054]: Upgrade: Updating crash.smithi027 (2/2) 2024-04-22T15:54:35.458 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:35 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:35.458 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:35 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi027", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T15:54:35.458 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:35 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:35.458 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:35 smithi027 ceph-mon[83054]: Deploying daemon crash.smithi027 on smithi027 2024-04-22T15:54:35.458 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:35 smithi027 ceph-mon[83054]: pgmap v35: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:37.575 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:37 smithi027 ceph-mon[83054]: pgmap v36: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:54:37.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:37 smithi016 ceph-mon[110740]: pgmap v36: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:54:39.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:38 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:39.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:38 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:39.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:38 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:54:39.280 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:38 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:39.280 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:38 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:39.280 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:38 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:54:40.086 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:39 smithi016 ceph-mon[110740]: pgmap v37: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:40.226 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:39 smithi027 ceph-mon[83054]: pgmap v37: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:42.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:41 smithi016 ceph-mon[110740]: pgmap v38: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:42.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:41 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:42.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:41 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:42.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:41 smithi027 ceph-mon[83054]: pgmap v38: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:42.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:41 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:42.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:41 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:43.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:43 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:43.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:43 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:43.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:43 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:43.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:43 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:44.274 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:44 smithi027 ceph-mon[83054]: pgmap v39: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:54:44.274 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:44 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:44.274 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:44 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:44.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:44 smithi016 ceph-mon[110740]: pgmap v39: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:54:44.433 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:44 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:44.433 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:44 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:45.524 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:45 smithi027 ceph-mon[83054]: pgmap v40: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:45.524 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:45 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:45.524 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:45 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:45.524 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:45 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:54:45.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:45 smithi016 ceph-mon[110740]: pgmap v40: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:45.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:45 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:45.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:45 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:45.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:45 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:54:46.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:46.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:46.851 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:46.851 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:47.879 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:47 smithi016 ceph-mon[110740]: pgmap v41: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:54:48.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:47 smithi027 ceph-mon[83054]: pgmap v41: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:54:48.982 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:48 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:48.983 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:48 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:48.983 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:48 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:48.983 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:48 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:54:48.983 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:48 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:48.983 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:48 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:54:48.983 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:48 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:54:48.983 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:48 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:54:48.983 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:48 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:54:48.983 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:48 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:48.983 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:48 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi016"}]: dispatch 2024-04-22T15:54:48.983 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:48 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi016"}]': finished 2024-04-22T15:54:48.984 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:48 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi027"}]: dispatch 2024-04-22T15:54:48.984 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:48 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi027"}]': finished 2024-04-22T15:54:48.984 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:48 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-22T15:54:49.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:48 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:49.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:48 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:49.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:48 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:49.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:48 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:54:49.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:48 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:49.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:48 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:54:49.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:48 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:54:49.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:48 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:54:49.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:48 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:54:49.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:48 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:49.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:48 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi016"}]: dispatch 2024-04-22T15:54:49.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:48 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi016"}]': finished 2024-04-22T15:54:49.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:48 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi027"}]: dispatch 2024-04-22T15:54:49.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:48 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi027"}]': finished 2024-04-22T15:54:49.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:48 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-22T15:54:50.188 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:49 smithi016 ceph-mon[110740]: Upgrade: Setting container_image for all crash 2024-04-22T15:54:50.188 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:49 smithi016 ceph-mon[110740]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-22T15:54:50.188 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:49 smithi016 ceph-mon[110740]: Upgrade: osd.0 is safe to restart 2024-04-22T15:54:50.188 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:49 smithi016 ceph-mon[110740]: pgmap v42: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:50.188 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:49 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:50.188 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:49 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T15:54:50.188 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:49 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:50.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:49 smithi027 ceph-mon[83054]: Upgrade: Setting container_image for all crash 2024-04-22T15:54:50.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:49 smithi027 ceph-mon[83054]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-22T15:54:50.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:49 smithi027 ceph-mon[83054]: Upgrade: osd.0 is safe to restart 2024-04-22T15:54:50.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:49 smithi027 ceph-mon[83054]: pgmap v42: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:50.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:49 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:54:50.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:49 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T15:54:50.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:49 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:54:51.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:50 smithi016 ceph-mon[110740]: Upgrade: Updating osd.0 2024-04-22T15:54:51.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:50 smithi016 ceph-mon[110740]: Deploying daemon osd.0 on smithi016 2024-04-22T15:54:51.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:50 smithi027 ceph-mon[83054]: Upgrade: Updating osd.0 2024-04-22T15:54:51.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:50 smithi027 ceph-mon[83054]: Deploying daemon osd.0 on smithi016 2024-04-22T15:54:52.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:51 smithi016 ceph-mon[110740]: pgmap v43: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:52.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:51 smithi027 ceph-mon[83054]: pgmap v43: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:54:52.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.811+0000 7f76429aa700 1 -- 172.21.15.16:0/1224901227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f763c106db0 msgr2=0x7f763c107190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:52.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.811+0000 7f76429aa700 1 --2- 172.21.15.16:0/1224901227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f763c106db0 0x7f763c107190 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f762c009910 tx=0x7f762c009c20 comp rx=0 tx=0).stop 2024-04-22T15:54:52.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.811+0000 7f76429aa700 1 -- 172.21.15.16:0/1224901227 shutdown_connections 2024-04-22T15:54:52.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.811+0000 7f76429aa700 1 --2- 172.21.15.16:0/1224901227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f763c0ff410 0x7f763c0ff890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:52.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.811+0000 7f76429aa700 1 --2- 172.21.15.16:0/1224901227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f763c106db0 0x7f763c107190 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:52.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.811+0000 7f76429aa700 1 -- 172.21.15.16:0/1224901227 >> 172.21.15.16:0/1224901227 conn(0x7f763c074e30 msgr2=0x7f763c075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:52.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.812+0000 7f76429aa700 1 -- 172.21.15.16:0/1224901227 shutdown_connections 2024-04-22T15:54:52.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.812+0000 7f76429aa700 1 -- 172.21.15.16:0/1224901227 wait complete. 2024-04-22T15:54:52.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.812+0000 7f76429aa700 1 Processor -- start 2024-04-22T15:54:52.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.812+0000 7f76429aa700 1 -- start start 2024-04-22T15:54:52.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.813+0000 7f76429aa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f763c0ff410 0x7f763c19c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:52.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.813+0000 7f76429aa700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f763c106db0 0x7f763c19cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:52.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.813+0000 7f76429aa700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f763c19d370 con 0x7f763c0ff410 2024-04-22T15:54:52.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.813+0000 7f76429aa700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f763c196820 con 0x7f763c106db0 2024-04-22T15:54:52.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.813+0000 7f763bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f763c0ff410 0x7f763c19c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:52.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.813+0000 7f763bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f763c0ff410 0x7f763c19c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42124/0 (socket says 172.21.15.16:42124) 2024-04-22T15:54:52.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.813+0000 7f763bfff700 1 -- 172.21.15.16:0/3663504074 learned_addr learned my addr 172.21.15.16:0/3663504074 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:54:52.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.813+0000 7f763b7fe700 1 --2- 172.21.15.16:0/3663504074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f763c106db0 0x7f763c19cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:52.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.813+0000 7f763bfff700 1 -- 172.21.15.16:0/3663504074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f763c106db0 msgr2=0x7f763c19cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:52.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.813+0000 7f763bfff700 1 --2- 172.21.15.16:0/3663504074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f763c106db0 0x7f763c19cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:52.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.813+0000 7f763bfff700 1 -- 172.21.15.16:0/3663504074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762c0095f0 con 0x7f763c0ff410 2024-04-22T15:54:52.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.813+0000 7f763b7fe700 1 --2- 172.21.15.16:0/3663504074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f763c106db0 0x7f763c19cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:54:52.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.813+0000 7f763bfff700 1 --2- 172.21.15.16:0/3663504074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f763c0ff410 0x7f763c19c7a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f762c006010 tx=0x7f762c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:52.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.814+0000 7f76397fa700 1 -- 172.21.15.16:0/3663504074 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f762c01d070 con 0x7f763c0ff410 2024-04-22T15:54:52.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.814+0000 7f76397fa700 1 -- 172.21.15.16:0/3663504074 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f762c005dc0 con 0x7f763c0ff410 2024-04-22T15:54:52.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.814+0000 7f76429aa700 1 -- 172.21.15.16:0/3663504074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f763c196aa0 con 0x7f763c0ff410 2024-04-22T15:54:52.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.814+0000 7f76397fa700 1 -- 172.21.15.16:0/3663504074 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f762c017400 con 0x7f763c0ff410 2024-04-22T15:54:52.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.814+0000 7f76429aa700 1 -- 172.21.15.16:0/3663504074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f763c196f90 con 0x7f763c0ff410 2024-04-22T15:54:52.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.815+0000 7f76429aa700 1 -- 172.21.15.16:0/3663504074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f763c109300 con 0x7f763c0ff410 2024-04-22T15:54:52.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.816+0000 7f76397fa700 1 -- 172.21.15.16:0/3663504074 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f762c017560 con 0x7f763c0ff410 2024-04-22T15:54:52.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.816+0000 7f76397fa700 1 --2- 172.21.15.16:0/3663504074 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f762406ee00 0x7f76240712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:52.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.816+0000 7f76397fa700 1 -- 172.21.15.16:0/3663504074 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f762c0904b0 con 0x7f763c0ff410 2024-04-22T15:54:52.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.816+0000 7f763b7fe700 1 --2- 172.21.15.16:0/3663504074 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f762406ee00 0x7f76240712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:52.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.817+0000 7f763b7fe700 1 --2- 172.21.15.16:0/3663504074 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f762406ee00 0x7f76240712c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f763c075490 tx=0x7f763000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:52.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:52.822+0000 7f76397fa700 1 -- 172.21.15.16:0/3663504074 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f762c05cc90 con 0x7f763c0ff410 2024-04-22T15:54:53.067 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:52 smithi016 systemd[1]: Stopping Ceph osd.0 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T15:54:53.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.063+0000 7f76429aa700 1 -- 172.21.15.16:0/3663504074 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f763c197df0 con 0x7f762406ee00 2024-04-22T15:54:53.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.065+0000 7f76397fa700 1 -- 172.21.15.16:0/3663504074 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f763c197df0 con 0x7f762406ee00 2024-04-22T15:54:53.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.067+0000 7f76429aa700 1 -- 172.21.15.16:0/3663504074 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f762406ee00 msgr2=0x7f76240712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:53.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.067+0000 7f76429aa700 1 --2- 172.21.15.16:0/3663504074 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f762406ee00 0x7f76240712c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f763c075490 tx=0x7f763000b410 comp rx=0 tx=0).stop 2024-04-22T15:54:53.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.067+0000 7f76429aa700 1 -- 172.21.15.16:0/3663504074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f763c0ff410 msgr2=0x7f763c19c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:53.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.067+0000 7f76429aa700 1 --2- 172.21.15.16:0/3663504074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f763c0ff410 0x7f763c19c7a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f762c006010 tx=0x7f762c00bd70 comp rx=0 tx=0).stop 2024-04-22T15:54:53.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.067+0000 7f76429aa700 1 -- 172.21.15.16:0/3663504074 shutdown_connections 2024-04-22T15:54:53.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.067+0000 7f76429aa700 1 --2- 172.21.15.16:0/3663504074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f763c0ff410 0x7f763c19c7a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.067+0000 7f76429aa700 1 --2- 172.21.15.16:0/3663504074 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f762406ee00 0x7f76240712c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.067+0000 7f76429aa700 1 --2- 172.21.15.16:0/3663504074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f763c106db0 0x7f763c19cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.067+0000 7f76429aa700 1 -- 172.21.15.16:0/3663504074 >> 172.21.15.16:0/3663504074 conn(0x7f763c074e30 msgr2=0x7f763c0fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:53.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.068+0000 7f76429aa700 1 -- 172.21.15.16:0/3663504074 shutdown_connections 2024-04-22T15:54:53.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.068+0000 7f76429aa700 1 -- 172.21.15.16:0/3663504074 wait complete. 2024-04-22T15:54:53.085 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:54:53.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.183+0000 7f2353783700 1 -- 172.21.15.16:0/559197033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f234c100750 msgr2=0x7f234c1047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:53.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.183+0000 7f2353783700 1 --2- 172.21.15.16:0/559197033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f234c100750 0x7f234c1047a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2340009910 tx=0x7f2340009c20 comp rx=0 tx=0).stop 2024-04-22T15:54:53.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.183+0000 7f2353783700 1 -- 172.21.15.16:0/559197033 shutdown_connections 2024-04-22T15:54:53.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.183+0000 7f2353783700 1 --2- 172.21.15.16:0/559197033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f234c100750 0x7f234c1047a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.183+0000 7f2353783700 1 --2- 172.21.15.16:0/559197033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f234c0ffda0 0x7f234c100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.183+0000 7f2353783700 1 -- 172.21.15.16:0/559197033 >> 172.21.15.16:0/559197033 conn(0x7f234c0fb650 msgr2=0x7f234c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:53.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.183+0000 7f2353783700 1 -- 172.21.15.16:0/559197033 shutdown_connections 2024-04-22T15:54:53.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.183+0000 7f2353783700 1 -- 172.21.15.16:0/559197033 wait complete. 2024-04-22T15:54:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.184+0000 7f2353783700 1 Processor -- start 2024-04-22T15:54:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.184+0000 7f2353783700 1 -- start start 2024-04-22T15:54:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.184+0000 7f2353783700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f234c0ffda0 0x7f234c101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.184+0000 7f2353783700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f234c100750 0x7f234c101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.184+0000 7f2353783700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234c1022c0 con 0x7f234c100750 2024-04-22T15:54:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.184+0000 7f2353783700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234c102400 con 0x7f234c0ffda0 2024-04-22T15:54:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.184+0000 7f2350d1e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f234c100750 0x7f234c101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.184+0000 7f2350d1e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f234c100750 0x7f234c101d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42132/0 (socket says 172.21.15.16:42132) 2024-04-22T15:54:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.184+0000 7f2350d1e700 1 -- 172.21.15.16:0/1625300878 learned_addr learned my addr 172.21.15.16:0/1625300878 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:54:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.184+0000 7f235151f700 1 --2- 172.21.15.16:0/1625300878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f234c0ffda0 0x7f234c101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.184+0000 7f2350d1e700 1 -- 172.21.15.16:0/1625300878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f234c0ffda0 msgr2=0x7f234c101840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.184+0000 7f2350d1e700 1 --2- 172.21.15.16:0/1625300878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f234c0ffda0 0x7f234c101840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.185+0000 7f2350d1e700 1 -- 172.21.15.16:0/1625300878 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23400095f0 con 0x7f234c100750 2024-04-22T15:54:53.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.185+0000 7f2350d1e700 1 --2- 172.21.15.16:0/1625300878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f234c100750 0x7f234c101d80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f23400059d0 tx=0x7f234000f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:53.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.185+0000 7f234a7fc700 1 -- 172.21.15.16:0/1625300878 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f234001d070 con 0x7f234c100750 2024-04-22T15:54:53.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.185+0000 7f234a7fc700 1 -- 172.21.15.16:0/1625300878 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f234000b610 con 0x7f234c100750 2024-04-22T15:54:53.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.185+0000 7f2353783700 1 -- 172.21.15.16:0/1625300878 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f234c102680 con 0x7f234c100750 2024-04-22T15:54:53.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.185+0000 7f234a7fc700 1 -- 172.21.15.16:0/1625300878 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2340022470 con 0x7f234c100750 2024-04-22T15:54:53.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.185+0000 7f2353783700 1 -- 172.21.15.16:0/1625300878 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f234c1a2e50 con 0x7f234c100750 2024-04-22T15:54:53.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.186+0000 7f2353783700 1 -- 172.21.15.16:0/1625300878 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f234c04f2e0 con 0x7f234c100750 2024-04-22T15:54:53.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.187+0000 7f234a7fc700 1 -- 172.21.15.16:0/1625300878 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f234000fc70 con 0x7f234c100750 2024-04-22T15:54:53.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.187+0000 7f234a7fc700 1 --2- 172.21.15.16:0/1625300878 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f233406ee00 0x7f23340712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:53.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.187+0000 7f234a7fc700 1 -- 172.21.15.16:0/1625300878 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f2340090440 con 0x7f234c100750 2024-04-22T15:54:53.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.187+0000 7f235151f700 1 --2- 172.21.15.16:0/1625300878 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f233406ee00 0x7f23340712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:53.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.188+0000 7f235151f700 1 --2- 172.21.15.16:0/1625300878 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f233406ee00 0x7f23340712c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2344009380 tx=0x7f234400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:53.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.192+0000 7f234a7fc700 1 -- 172.21.15.16:0/1625300878 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2340059080 con 0x7f234c100750 2024-04-22T15:54:53.417 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:53 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0[45810]: 2024-04-22T15:54:53.170+0000 7f0818f6b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T15:54:53.417 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:53 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0[45810]: 2024-04-22T15:54:53.170+0000 7f0818f6b700 -1 osd.0 49 *** Got signal Terminated *** 2024-04-22T15:54:53.417 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:53 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0[45810]: 2024-04-22T15:54:53.170+0000 7f0818f6b700 -1 osd.0 49 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T15:54:53.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.422+0000 7f2353783700 1 -- 172.21.15.16:0/1625300878 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f234c0fc3d0 con 0x7f233406ee00 2024-04-22T15:54:53.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.423+0000 7f234a7fc700 1 -- 172.21.15.16:0/1625300878 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f234c0fc3d0 con 0x7f233406ee00 2024-04-22T15:54:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.425+0000 7f2353783700 1 -- 172.21.15.16:0/1625300878 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f233406ee00 msgr2=0x7f23340712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.425+0000 7f2353783700 1 --2- 172.21.15.16:0/1625300878 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f233406ee00 0x7f23340712c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2344009380 tx=0x7f234400b040 comp rx=0 tx=0).stop 2024-04-22T15:54:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.425+0000 7f2353783700 1 -- 172.21.15.16:0/1625300878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f234c100750 msgr2=0x7f234c101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.425+0000 7f2353783700 1 --2- 172.21.15.16:0/1625300878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f234c100750 0x7f234c101d80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f23400059d0 tx=0x7f234000f740 comp rx=0 tx=0).stop 2024-04-22T15:54:53.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.425+0000 7f2353783700 1 -- 172.21.15.16:0/1625300878 shutdown_connections 2024-04-22T15:54:53.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.425+0000 7f2353783700 1 --2- 172.21.15.16:0/1625300878 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f233406ee00 0x7f23340712c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.425+0000 7f2353783700 1 --2- 172.21.15.16:0/1625300878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f234c0ffda0 0x7f234c101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.425+0000 7f2353783700 1 --2- 172.21.15.16:0/1625300878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f234c100750 0x7f234c101d80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.425+0000 7f2353783700 1 -- 172.21.15.16:0/1625300878 >> 172.21.15.16:0/1625300878 conn(0x7f234c0fb650 msgr2=0x7f234c103fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:53.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.425+0000 7f2353783700 1 -- 172.21.15.16:0/1625300878 shutdown_connections 2024-04-22T15:54:53.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.425+0000 7f2353783700 1 -- 172.21.15.16:0/1625300878 wait complete. 2024-04-22T15:54:53.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.533+0000 7f8566515700 1 -- 172.21.15.16:0/309147159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8560105bd0 msgr2=0x7f8560105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:53.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.533+0000 7f8566515700 1 --2- 172.21.15.16:0/309147159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8560105bd0 0x7f8560105fb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8550009910 tx=0x7f8550009c20 comp rx=0 tx=0).stop 2024-04-22T15:54:53.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.533+0000 7f8566515700 1 -- 172.21.15.16:0/309147159 shutdown_connections 2024-04-22T15:54:53.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.533+0000 7f8566515700 1 --2- 172.21.15.16:0/309147159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85600ffc20 0x7f8560100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.533+0000 7f8566515700 1 --2- 172.21.15.16:0/309147159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8560105bd0 0x7f8560105fb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.533+0000 7f8566515700 1 -- 172.21.15.16:0/309147159 >> 172.21.15.16:0/309147159 conn(0x7f8560075670 msgr2=0x7f8560075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:53.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.533+0000 7f8566515700 1 -- 172.21.15.16:0/309147159 shutdown_connections 2024-04-22T15:54:53.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.533+0000 7f8566515700 1 -- 172.21.15.16:0/309147159 wait complete. 2024-04-22T15:54:53.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.534+0000 7f8566515700 1 Processor -- start 2024-04-22T15:54:53.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.534+0000 7f8566515700 1 -- start start 2024-04-22T15:54:53.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.534+0000 7f8566515700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85600ffc20 0x7f85601976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:53.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.534+0000 7f8566515700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8560105bd0 0x7f8560197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:53.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.534+0000 7f8566515700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85601982d0 con 0x7f8560105bd0 2024-04-22T15:54:53.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.534+0000 7f8566515700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8560198440 con 0x7f85600ffc20 2024-04-22T15:54:53.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.534+0000 7f855f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8560105bd0 0x7f8560197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:53.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.534+0000 7f855f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8560105bd0 0x7f8560197bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42142/0 (socket says 172.21.15.16:42142) 2024-04-22T15:54:53.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.534+0000 7f855f7fe700 1 -- 172.21.15.16:0/3992182804 learned_addr learned my addr 172.21.15.16:0/3992182804 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:54:53.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.534+0000 7f855ffff700 1 --2- 172.21.15.16:0/3992182804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85600ffc20 0x7f85601976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:53.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.535+0000 7f855f7fe700 1 -- 172.21.15.16:0/3992182804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85600ffc20 msgr2=0x7f85601976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:53.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.535+0000 7f855f7fe700 1 --2- 172.21.15.16:0/3992182804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85600ffc20 0x7f85601976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.535+0000 7f855f7fe700 1 -- 172.21.15.16:0/3992182804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85500095f0 con 0x7f8560105bd0 2024-04-22T15:54:53.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.535+0000 7f855f7fe700 1 --2- 172.21.15.16:0/3992182804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8560105bd0 0x7f8560197bf0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f854800d8d0 tx=0x7f854800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:53.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.535+0000 7f855d7fa700 1 -- 172.21.15.16:0/3992182804 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85480098a0 con 0x7f8560105bd0 2024-04-22T15:54:53.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.535+0000 7f855d7fa700 1 -- 172.21.15.16:0/3992182804 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8548010460 con 0x7f8560105bd0 2024-04-22T15:54:53.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.535+0000 7f855d7fa700 1 -- 172.21.15.16:0/3992182804 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f854800f5f0 con 0x7f8560105bd0 2024-04-22T15:54:53.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.535+0000 7f8566515700 1 -- 172.21.15.16:0/3992182804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85601a0720 con 0x7f8560105bd0 2024-04-22T15:54:53.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.535+0000 7f8566515700 1 -- 172.21.15.16:0/3992182804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85601a0c70 con 0x7f8560105bd0 2024-04-22T15:54:53.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.536+0000 7f8566515700 1 -- 172.21.15.16:0/3992182804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85601080d0 con 0x7f8560105bd0 2024-04-22T15:54:53.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.537+0000 7f855d7fa700 1 -- 172.21.15.16:0/3992182804 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f8548009a00 con 0x7f8560105bd0 2024-04-22T15:54:53.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.537+0000 7f855d7fa700 1 --2- 172.21.15.16:0/3992182804 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f854c06edf0 0x7f854c0712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:53.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.537+0000 7f855d7fa700 1 -- 172.21.15.16:0/3992182804 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f854808ef00 con 0x7f8560105bd0 2024-04-22T15:54:53.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.537+0000 7f855ffff700 1 --2- 172.21.15.16:0/3992182804 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f854c06edf0 0x7f854c0712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:53.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.538+0000 7f855ffff700 1 --2- 172.21.15.16:0/3992182804 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f854c06edf0 0x7f854c0712b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f855000b5c0 tx=0x7f855001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.542+0000 7f855d7fa700 1 -- 172.21.15.16:0/3992182804 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8548057b40 con 0x7f8560105bd0 2024-04-22T15:54:53.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.763+0000 7f8566515700 1 -- 172.21.15.16:0/3992182804 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f85600611d0 con 0x7f854c06edf0 2024-04-22T15:54:53.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.774+0000 7f855d7fa700 1 -- 172.21.15.16:0/3992182804 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f85600611d0 con 0x7f854c06edf0 2024-04-22T15:54:53.777 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:54:53.777 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.smithi016 smithi016 *:9093,9094 running (3m) 10s ago 13m 25.8M - 0.25.0 c8568f914cd2 ea0e0217d6f9 2024-04-22T15:54:53.777 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi016 smithi016 running (14m) 10s ago 14m 8094k - 18.2.1 5be31c24972a f8b627d3f41e 2024-04-22T15:54:53.777 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi027 smithi027 running (13m) 12s ago 13m 8066k - 18.2.1 5be31c24972a f00fd40b4220 2024-04-22T15:54:53.777 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi016 smithi016 running (21s) 10s ago 14m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 1feb008449ed 2024-04-22T15:54:53.778 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi027 smithi027 running (16s) 12s ago 12m 6656k - 19.0.0-3223-g5883d494 3d983c26e278 c7f862c6da00 2024-04-22T15:54:53.778 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.smithi016 smithi016 *:3000 running (2m) 10s ago 13m 79.6M - 9.4.12 2bacad6d85d8 388a3c9c1b3f 2024-04-22T15:54:53.778 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.fkkejm smithi016 running (8m) 10s ago 8m 16.3M - 18.2.1 5be31c24972a 4153ac94cea0 2024-04-22T15:54:53.778 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.pgvphu smithi016 running (8m) 10s ago 8m 91.7M - 18.2.1 5be31c24972a 93dc072068a2 2024-04-22T15:54:53.778 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.dgqyes smithi027 running (8m) 12s ago 8m 19.2M - 18.2.1 5be31c24972a 41a9220430fd 2024-04-22T15:54:53.778 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.iobjds smithi027 running (8m) 12s ago 8m 149M - 18.2.1 5be31c24972a c1c8d6770f02 2024-04-22T15:54:53.778 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi016.rnsnnr smithi016 *:8443,9283,8765 running (5m) 10s ago 15m 500M - 19.0.0-3223-g5883d494 3d983c26e278 09b4fe285bc4 2024-04-22T15:54:53.778 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi027.ukdgzs smithi027 *:8443,9283,8765 running (4m) 12s ago 12m 450M - 19.0.0-3223-g5883d494 3d983c26e278 7548da9b5eb9 2024-04-22T15:54:53.778 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi016 smithi016 running (92s) 10s ago 15m 44.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 4ae529a67806 2024-04-22T15:54:53.778 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi027 smithi027 running (66s) 12s ago 12m 39.0M 2048M 19.0.0-3223-g5883d494 3d983c26e278 f29fcb888801 2024-04-22T15:54:53.778 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi016 smithi016 *:9100 running (4m) 10s ago 13m 16.0M - 1.5.0 0da6a335fe13 35e46e4d2f3f 2024-04-22T15:54:53.778 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi027 smithi027 *:9100 running (3m) 12s ago 12m 15.5M - 1.5.0 0da6a335fe13 e682a7672e9b 2024-04-22T15:54:53.778 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (12m) 10s ago 12m 202M 1217M 18.2.1 5be31c24972a 3c6ebc4ea041 2024-04-22T15:54:53.778 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (11m) 10s ago 11m 206M 1217M 18.2.1 5be31c24972a 6aa2e6796784 2024-04-22T15:54:53.778 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (11m) 10s ago 11m 176M 1217M 18.2.1 5be31c24972a 3d9d310aa9eb 2024-04-22T15:54:53.779 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi027 running (10m) 12s ago 10m 250M 2240M 18.2.1 5be31c24972a cdf9b7aaf865 2024-04-22T15:54:53.779 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (10m) 12s ago 10m 206M 2240M 18.2.1 5be31c24972a 629397d49c76 2024-04-22T15:54:53.779 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi027 running (9m) 12s ago 9m 193M 2240M 18.2.1 5be31c24972a 23adcf675b04 2024-04-22T15:54:53.779 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.smithi016 smithi016 *:9095 running (3m) 10s ago 13m 71.6M - 2.43.0 a07b618ecd1d 373eaadee82d 2024-04-22T15:54:53.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.777+0000 7f8566515700 1 -- 172.21.15.16:0/3992182804 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f854c06edf0 msgr2=0x7f854c0712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:53.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.777+0000 7f8566515700 1 --2- 172.21.15.16:0/3992182804 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f854c06edf0 0x7f854c0712b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f855000b5c0 tx=0x7f855001a040 comp rx=0 tx=0).stop 2024-04-22T15:54:53.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.777+0000 7f8566515700 1 -- 172.21.15.16:0/3992182804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8560105bd0 msgr2=0x7f8560197bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:53.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.777+0000 7f8566515700 1 --2- 172.21.15.16:0/3992182804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8560105bd0 0x7f8560197bf0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f854800d8d0 tx=0x7f854800dbe0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.777+0000 7f8566515700 1 -- 172.21.15.16:0/3992182804 shutdown_connections 2024-04-22T15:54:53.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.777+0000 7f8566515700 1 --2- 172.21.15.16:0/3992182804 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f854c06edf0 0x7f854c0712b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.777+0000 7f8566515700 1 --2- 172.21.15.16:0/3992182804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85600ffc20 0x7f85601976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.777+0000 7f8566515700 1 --2- 172.21.15.16:0/3992182804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8560105bd0 0x7f8560197bf0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.777+0000 7f8566515700 1 -- 172.21.15.16:0/3992182804 >> 172.21.15.16:0/3992182804 conn(0x7f8560075670 msgr2=0x7f85600feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:53.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.777+0000 7f8566515700 1 -- 172.21.15.16:0/3992182804 shutdown_connections 2024-04-22T15:54:53.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.777+0000 7f8566515700 1 -- 172.21.15.16:0/3992182804 wait complete. 2024-04-22T15:54:53.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.883+0000 7fd051923700 1 -- 172.21.15.16:0/647780294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd04c101840 msgr2=0x7fd04c105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:53.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.883+0000 7fd051923700 1 --2- 172.21.15.16:0/647780294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd04c101840 0x7fd04c105890 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd03c009910 tx=0x7fd03c009c20 comp rx=0 tx=0).stop 2024-04-22T15:54:53.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.883+0000 7fd051923700 1 -- 172.21.15.16:0/647780294 shutdown_connections 2024-04-22T15:54:53.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.883+0000 7fd051923700 1 --2- 172.21.15.16:0/647780294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd04c101840 0x7fd04c105890 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.883+0000 7fd051923700 1 --2- 172.21.15.16:0/647780294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd04c100e90 0x7fd04c101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.883+0000 7fd051923700 1 -- 172.21.15.16:0/647780294 >> 172.21.15.16:0/647780294 conn(0x7fd04c0fc740 msgr2=0x7fd04c0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:53.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.883+0000 7fd051923700 1 -- 172.21.15.16:0/647780294 shutdown_connections 2024-04-22T15:54:53.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.883+0000 7fd051923700 1 -- 172.21.15.16:0/647780294 wait complete. 2024-04-22T15:54:53.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.883+0000 7fd051923700 1 Processor -- start 2024-04-22T15:54:53.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.884+0000 7fd051923700 1 -- start start 2024-04-22T15:54:53.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.884+0000 7fd051923700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd04c100e90 0x7fd04c19c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:53.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.884+0000 7fd051923700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd04c101840 0x7fd04c19cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:53.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.884+0000 7fd051923700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c19d240 con 0x7fd04c100e90 2024-04-22T15:54:53.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.884+0000 7fd051923700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c1966f0 con 0x7fd04c101840 2024-04-22T15:54:53.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.884+0000 7fd04affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd04c100e90 0x7fd04c19c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:53.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.884+0000 7fd04affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd04c100e90 0x7fd04c19c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42156/0 (socket says 172.21.15.16:42156) 2024-04-22T15:54:53.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.884+0000 7fd04affd700 1 -- 172.21.15.16:0/2059470006 learned_addr learned my addr 172.21.15.16:0/2059470006 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:54:53.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.884+0000 7fd04a7fc700 1 --2- 172.21.15.16:0/2059470006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd04c101840 0x7fd04c19cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:53.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.884+0000 7fd04affd700 1 -- 172.21.15.16:0/2059470006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd04c101840 msgr2=0x7fd04c19cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:53.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.884+0000 7fd04affd700 1 --2- 172.21.15.16:0/2059470006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd04c101840 0x7fd04c19cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:53.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.884+0000 7fd04affd700 1 -- 172.21.15.16:0/2059470006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03c0095f0 con 0x7fd04c100e90 2024-04-22T15:54:53.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.884+0000 7fd04affd700 1 --2- 172.21.15.16:0/2059470006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd04c100e90 0x7fd04c19c670 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd04000e920 tx=0x7fd04000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:53.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.885+0000 7fd050921700 1 -- 172.21.15.16:0/2059470006 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd04000c9d0 con 0x7fd04c100e90 2024-04-22T15:54:53.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.885+0000 7fd050921700 1 -- 172.21.15.16:0/2059470006 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd040010430 con 0x7fd04c100e90 2024-04-22T15:54:53.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.885+0000 7fd051923700 1 -- 172.21.15.16:0/2059470006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd04c1969d0 con 0x7fd04c100e90 2024-04-22T15:54:53.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.885+0000 7fd050921700 1 -- 172.21.15.16:0/2059470006 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd040018630 con 0x7fd04c100e90 2024-04-22T15:54:53.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.885+0000 7fd051923700 1 -- 172.21.15.16:0/2059470006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd04c196ef0 con 0x7fd04c100e90 2024-04-22T15:54:53.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.886+0000 7fd051923700 1 -- 172.21.15.16:0/2059470006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd04c109290 con 0x7fd04c100e90 2024-04-22T15:54:53.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.886+0000 7fd050921700 1 -- 172.21.15.16:0/2059470006 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fd0400105a0 con 0x7fd04c100e90 2024-04-22T15:54:53.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.887+0000 7fd050921700 1 --2- 172.21.15.16:0/2059470006 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd03406ee00 0x7fd0340712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:53.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.887+0000 7fd050921700 1 -- 172.21.15.16:0/2059470006 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd040014070 con 0x7fd04c100e90 2024-04-22T15:54:53.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.887+0000 7fd04a7fc700 1 --2- 172.21.15.16:0/2059470006 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd03406ee00 0x7fd0340712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:53.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.887+0000 7fd04a7fc700 1 --2- 172.21.15.16:0/2059470006 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd03406ee00 0x7fd0340712c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd03c0059d0 tx=0x7fd03c005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:53.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:53.891+0000 7fd050921700 1 -- 172.21.15.16:0/2059470006 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd040054150 con 0x7fd04c100e90 2024-04-22T15:54:54.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:53 smithi027 ceph-mon[83054]: pgmap v44: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:54:54.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:53 smithi027 ceph-mon[83054]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:54.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:53 smithi027 ceph-mon[83054]: osd.0 marked itself down and dead 2024-04-22T15:54:54.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:53 smithi016 ceph-mon[110740]: pgmap v44: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:54:54.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:53 smithi016 ceph-mon[110740]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:54.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:53 smithi016 ceph-mon[110740]: osd.0 marked itself down and dead 2024-04-22T15:54:54.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.196+0000 7fd051923700 1 -- 172.21.15.16:0/2059470006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd04c197160 con 0x7fd04c100e90 2024-04-22T15:54:54.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.197+0000 7fd050921700 1 -- 172.21.15.16:0/2059470006 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+687 (secure 0 0 0) 0x7fd04005b9d0 con 0x7fd04c100e90 2024-04-22T15:54:54.200 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:54:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-22T15:54:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:54:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:54:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-22T15:54:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:54:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:54:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-22T15:54:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 5 2024-04-22T15:54:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:54:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-22T15:54:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T15:54:54.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:54:54.201 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-22T15:54:54.201 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 9, 2024-04-22T15:54:54.201 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:54:54.201 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-22T15:54:54.201 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:54:54.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.199+0000 7fd051923700 1 -- 172.21.15.16:0/2059470006 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd03406ee00 msgr2=0x7fd0340712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:54.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.199+0000 7fd051923700 1 --2- 172.21.15.16:0/2059470006 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd03406ee00 0x7fd0340712c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd03c0059d0 tx=0x7fd03c005940 comp rx=0 tx=0).stop 2024-04-22T15:54:54.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.199+0000 7fd051923700 1 -- 172.21.15.16:0/2059470006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd04c100e90 msgr2=0x7fd04c19c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:54.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.199+0000 7fd051923700 1 --2- 172.21.15.16:0/2059470006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd04c100e90 0x7fd04c19c670 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd04000e920 tx=0x7fd04000ec30 comp rx=0 tx=0).stop 2024-04-22T15:54:54.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.199+0000 7fd051923700 1 -- 172.21.15.16:0/2059470006 shutdown_connections 2024-04-22T15:54:54.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.199+0000 7fd051923700 1 --2- 172.21.15.16:0/2059470006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd04c100e90 0x7fd04c19c670 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.199+0000 7fd051923700 1 --2- 172.21.15.16:0/2059470006 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd03406ee00 0x7fd0340712c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.199+0000 7fd051923700 1 --2- 172.21.15.16:0/2059470006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd04c101840 0x7fd04c19cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.199+0000 7fd051923700 1 -- 172.21.15.16:0/2059470006 >> 172.21.15.16:0/2059470006 conn(0x7fd04c0fc740 msgr2=0x7fd04c0fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:54.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.200+0000 7fd051923700 1 -- 172.21.15.16:0/2059470006 shutdown_connections 2024-04-22T15:54:54.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.200+0000 7fd051923700 1 -- 172.21.15.16:0/2059470006 wait complete. 2024-04-22T15:54:54.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.308+0000 7f4f135df700 1 -- 172.21.15.16:0/2140145735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f0c100ea0 msgr2=0x7f4f0c101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:54.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.308+0000 7f4f135df700 1 --2- 172.21.15.16:0/2140145735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f0c100ea0 0x7f4f0c101280 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4f08009910 tx=0x7f4f08009c20 comp rx=0 tx=0).stop 2024-04-22T15:54:54.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.308+0000 7f4f135df700 1 -- 172.21.15.16:0/2140145735 shutdown_connections 2024-04-22T15:54:54.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.308+0000 7f4f135df700 1 --2- 172.21.15.16:0/2140145735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f0c101850 0x7f4f0c1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.308+0000 7f4f135df700 1 --2- 172.21.15.16:0/2140145735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f0c100ea0 0x7f4f0c101280 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.308+0000 7f4f135df700 1 -- 172.21.15.16:0/2140145735 >> 172.21.15.16:0/2140145735 conn(0x7f4f0c0fc750 msgr2=0x7f4f0c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:54.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.309+0000 7f4f135df700 1 -- 172.21.15.16:0/2140145735 shutdown_connections 2024-04-22T15:54:54.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.309+0000 7f4f135df700 1 -- 172.21.15.16:0/2140145735 wait complete. 2024-04-22T15:54:54.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.309+0000 7f4f135df700 1 Processor -- start 2024-04-22T15:54:54.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.309+0000 7f4f135df700 1 -- start start 2024-04-22T15:54:54.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.309+0000 7f4f135df700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f0c100ea0 0x7f4f0c073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:54.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.309+0000 7f4f135df700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f0c101850 0x7f4f0c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:54.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.309+0000 7f4f135df700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0c073b10 con 0x7f4f0c100ea0 2024-04-22T15:54:54.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.309+0000 7f4f135df700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0c073c50 con 0x7f4f0c101850 2024-04-22T15:54:54.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.310+0000 7f4f1137b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f0c100ea0 0x7f4f0c073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:54.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.310+0000 7f4f1137b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f0c100ea0 0x7f4f0c073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42164/0 (socket says 172.21.15.16:42164) 2024-04-22T15:54:54.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.310+0000 7f4f1137b700 1 -- 172.21.15.16:0/226278781 learned_addr learned my addr 172.21.15.16:0/226278781 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:54:54.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.310+0000 7f4f10b7a700 1 --2- 172.21.15.16:0/226278781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f0c101850 0x7f4f0c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:54.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.310+0000 7f4f1137b700 1 -- 172.21.15.16:0/226278781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f0c101850 msgr2=0x7f4f0c0735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:54.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.310+0000 7f4f1137b700 1 --2- 172.21.15.16:0/226278781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f0c101850 0x7f4f0c0735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.310+0000 7f4f1137b700 1 -- 172.21.15.16:0/226278781 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f080095f0 con 0x7f4f0c100ea0 2024-04-22T15:54:54.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.310+0000 7f4f1137b700 1 --2- 172.21.15.16:0/226278781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f0c100ea0 0x7f4f0c073090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4f08000c00 tx=0x7f4f0800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:54.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.310+0000 7f4f027fc700 1 -- 172.21.15.16:0/226278781 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f0801d070 con 0x7f4f0c100ea0 2024-04-22T15:54:54.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.310+0000 7f4f027fc700 1 -- 172.21.15.16:0/226278781 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f0800f460 con 0x7f4f0c100ea0 2024-04-22T15:54:54.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.310+0000 7f4f135df700 1 -- 172.21.15.16:0/226278781 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f0c073ed0 con 0x7f4f0c100ea0 2024-04-22T15:54:54.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.310+0000 7f4f027fc700 1 -- 172.21.15.16:0/226278781 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f08017630 con 0x7f4f0c100ea0 2024-04-22T15:54:54.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.310+0000 7f4f135df700 1 -- 172.21.15.16:0/226278781 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f0c1a2e20 con 0x7f4f0c100ea0 2024-04-22T15:54:54.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.311+0000 7f4f135df700 1 -- 172.21.15.16:0/226278781 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f0c109240 con 0x7f4f0c100ea0 2024-04-22T15:54:54.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.312+0000 7f4f027fc700 1 -- 172.21.15.16:0/226278781 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f4f0800f5d0 con 0x7f4f0c100ea0 2024-04-22T15:54:54.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.312+0000 7f4f027fc700 1 --2- 172.21.15.16:0/226278781 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f4ef806ee00 0x7f4ef80712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:54.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.313+0000 7f4f10b7a700 1 --2- 172.21.15.16:0/226278781 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f4ef806ee00 0x7f4ef80712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:54.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.313+0000 7f4f027fc700 1 -- 172.21.15.16:0/226278781 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f4f08090600 con 0x7f4f0c100ea0 2024-04-22T15:54:54.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.313+0000 7f4f10b7a700 1 --2- 172.21.15.16:0/226278781 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f4ef806ee00 0x7f4ef80712c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c074880 tx=0x7f4efc005cf0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:54.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.317+0000 7f4f027fc700 1 -- 172.21.15.16:0/226278781 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4f08059220 con 0x7f4f0c100ea0 2024-04-22T15:54:54.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.586+0000 7f4f135df700 1 -- 172.21.15.16:0/226278781 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4f0c108c20 con 0x7f4f0c100ea0 2024-04-22T15:54:54.590 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:54 smithi016 podman[121465]: 2024-04-22 15:54:54.351144238 +0000 UTC m=+1.262445984 container died 3c6ebc4ea041915eced5b55ed096a280b2668243e7cf0d08a20681964a9d761e (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0, RELEASE=HEAD, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS) 2024-04-22T15:54:54.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.587+0000 7f4f027fc700 1 -- 172.21.15.16:0/226278781 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1902 (secure 0 0 0) 0x7f4f0805c8e0 con 0x7f4f0c100ea0 2024-04-22T15:54:54.590 INFO:teuthology.orchestra.run.smithi016.stdout:e16 2024-04-22T15:54:54.591 INFO:teuthology.orchestra.run.smithi016.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:54:54.591 INFO:teuthology.orchestra.run.smithi016.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:54:54.591 INFO:teuthology.orchestra.run.smithi016.stdout:legacy client fscid: 1 2024-04-22T15:54:54.591 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:54:54.591 INFO:teuthology.orchestra.run.smithi016.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:54:54.591 INFO:teuthology.orchestra.run.smithi016.stdout:fs_name cephfs 2024-04-22T15:54:54.591 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 16 2024-04-22T15:54:54.591 INFO:teuthology.orchestra.run.smithi016.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T15:54:54.591 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-04-22T15:46:27.774571+0000 2024-04-22T15:54:54.591 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-04-22T15:46:47.381709+0000 2024-04-22T15:54:54.591 INFO:teuthology.orchestra.run.smithi016.stdout:tableserver 0 2024-04-22T15:54:54.591 INFO:teuthology.orchestra.run.smithi016.stdout:root 0 2024-04-22T15:54:54.591 INFO:teuthology.orchestra.run.smithi016.stdout:session_timeout 60 2024-04-22T15:54:54.592 INFO:teuthology.orchestra.run.smithi016.stdout:session_autoclose 300 2024-04-22T15:54:54.592 INFO:teuthology.orchestra.run.smithi016.stdout:max_file_size 1099511627776 2024-04-22T15:54:54.592 INFO:teuthology.orchestra.run.smithi016.stdout:max_xattr_size 65536 2024-04-22T15:54:54.592 INFO:teuthology.orchestra.run.smithi016.stdout:required_client_features {} 2024-04-22T15:54:54.592 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure 0 2024-04-22T15:54:54.592 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure_osd_epoch 46 2024-04-22T15:54:54.592 INFO:teuthology.orchestra.run.smithi016.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:54:54.592 INFO:teuthology.orchestra.run.smithi016.stdout:max_mds 1 2024-04-22T15:54:54.592 INFO:teuthology.orchestra.run.smithi016.stdout:in 0 2024-04-22T15:54:54.592 INFO:teuthology.orchestra.run.smithi016.stdout:up {0=24279} 2024-04-22T15:54:54.592 INFO:teuthology.orchestra.run.smithi016.stdout:failed 2024-04-22T15:54:54.592 INFO:teuthology.orchestra.run.smithi016.stdout:damaged 2024-04-22T15:54:54.592 INFO:teuthology.orchestra.run.smithi016.stdout:stopped 2024-04-22T15:54:54.592 INFO:teuthology.orchestra.run.smithi016.stdout:data_pools [3] 2024-04-22T15:54:54.592 INFO:teuthology.orchestra.run.smithi016.stdout:metadata_pool 2 2024-04-22T15:54:54.593 INFO:teuthology.orchestra.run.smithi016.stdout:inline_data enabled 2024-04-22T15:54:54.593 INFO:teuthology.orchestra.run.smithi016.stdout:balancer 2024-04-22T15:54:54.593 INFO:teuthology.orchestra.run.smithi016.stdout:bal_rank_mask -1 2024-04-22T15:54:54.593 INFO:teuthology.orchestra.run.smithi016.stdout:standby_count_wanted 1 2024-04-22T15:54:54.593 INFO:teuthology.orchestra.run.smithi016.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:54:54.593 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.iobjds{0:24279} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.27:6824/4216277609,v1:172.21.15.27:6825/4216277609] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:54:54.593 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.pgvphu{0:14494} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.16:6828/1006073414,v1:172.21.15.16:6829/1006073414] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:54:54.593 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:54:54.593 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:54:54.593 INFO:teuthology.orchestra.run.smithi016.stdout:Standby daemons: 2024-04-22T15:54:54.593 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:54:54.593 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.fkkejm{-1:14504} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.16:6826/3484907755,v1:172.21.15.16:6827/3484907755] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:54:54.593 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.dgqyes{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.27:6826/476026355,v1:172.21.15.27:6827/476026355] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:54:54.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.589+0000 7f4f135df700 1 -- 172.21.15.16:0/226278781 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f4ef806ee00 msgr2=0x7f4ef80712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:54.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.589+0000 7f4f135df700 1 --2- 172.21.15.16:0/226278781 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f4ef806ee00 0x7f4ef80712c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c074880 tx=0x7f4efc005cf0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.589+0000 7f4f135df700 1 -- 172.21.15.16:0/226278781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f0c100ea0 msgr2=0x7f4f0c073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:54.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.589+0000 7f4f135df700 1 --2- 172.21.15.16:0/226278781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f0c100ea0 0x7f4f0c073090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4f08000c00 tx=0x7f4f0800be20 comp rx=0 tx=0).stop 2024-04-22T15:54:54.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.589+0000 7f4f135df700 1 -- 172.21.15.16:0/226278781 shutdown_connections 2024-04-22T15:54:54.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.589+0000 7f4f135df700 1 --2- 172.21.15.16:0/226278781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f0c100ea0 0x7f4f0c073090 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.589+0000 7f4f135df700 1 --2- 172.21.15.16:0/226278781 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f4ef806ee00 0x7f4ef80712c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.589+0000 7f4f135df700 1 --2- 172.21.15.16:0/226278781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f0c101850 0x7f4f0c0735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.589+0000 7f4f135df700 1 -- 172.21.15.16:0/226278781 >> 172.21.15.16:0/226278781 conn(0x7f4f0c0fc750 msgr2=0x7f4f0c0fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:54.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.589+0000 7f4f135df700 1 -- 172.21.15.16:0/226278781 shutdown_connections 2024-04-22T15:54:54.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.589+0000 7f4f135df700 1 -- 172.21.15.16:0/226278781 wait complete. 2024-04-22T15:54:54.595 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 16 2024-04-22T15:54:54.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.699+0000 7f52c49f5700 1 -- 172.21.15.16:0/3994325700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52c0068ea0 msgr2=0x7f52c010d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:54.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.699+0000 7f52c49f5700 1 --2- 172.21.15.16:0/3994325700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52c0068ea0 0x7f52c010d290 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f52b4009910 tx=0x7f52b4009c20 comp rx=0 tx=0).stop 2024-04-22T15:54:54.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.699+0000 7f52c49f5700 1 -- 172.21.15.16:0/3994325700 shutdown_connections 2024-04-22T15:54:54.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.699+0000 7f52c49f5700 1 --2- 172.21.15.16:0/3994325700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52c0068ea0 0x7f52c010d290 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.699+0000 7f52c49f5700 1 --2- 172.21.15.16:0/3994325700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c0068580 0x7f52c0068960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.699+0000 7f52c49f5700 1 -- 172.21.15.16:0/3994325700 >> 172.21.15.16:0/3994325700 conn(0x7f52c0075670 msgr2=0x7f52c0075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:54.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.699+0000 7f52c49f5700 1 -- 172.21.15.16:0/3994325700 shutdown_connections 2024-04-22T15:54:54.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.699+0000 7f52c49f5700 1 -- 172.21.15.16:0/3994325700 wait complete. 2024-04-22T15:54:54.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.700+0000 7f52c49f5700 1 Processor -- start 2024-04-22T15:54:54.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.700+0000 7f52c49f5700 1 -- start start 2024-04-22T15:54:54.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.700+0000 7f52c49f5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c0068580 0x7f52c019e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:54.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.700+0000 7f52c49f5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52c0068ea0 0x7f52c019ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:54.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.700+0000 7f52c49f5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c019f4b0 con 0x7f52c0068ea0 2024-04-22T15:54:54.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.700+0000 7f52c49f5700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c0198960 con 0x7f52c0068580 2024-04-22T15:54:54.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.700+0000 7f52bdd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52c0068ea0 0x7f52c019ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:54.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.700+0000 7f52bdd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52c0068ea0 0x7f52c019ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42176/0 (socket says 172.21.15.16:42176) 2024-04-22T15:54:54.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.700+0000 7f52bdd9b700 1 -- 172.21.15.16:0/4112704570 learned_addr learned my addr 172.21.15.16:0/4112704570 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:54:54.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.700+0000 7f52be59c700 1 --2- 172.21.15.16:0/4112704570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c0068580 0x7f52c019e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:54.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.701+0000 7f52bdd9b700 1 -- 172.21.15.16:0/4112704570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c0068580 msgr2=0x7f52c019e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:54.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.701+0000 7f52bdd9b700 1 --2- 172.21.15.16:0/4112704570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c0068580 0x7f52c019e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.701+0000 7f52bdd9b700 1 -- 172.21.15.16:0/4112704570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b40095f0 con 0x7f52c0068ea0 2024-04-22T15:54:54.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.701+0000 7f52bdd9b700 1 --2- 172.21.15.16:0/4112704570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52c0068ea0 0x7f52c019ee20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f52b4006010 tx=0x7f52b400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:54.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.701+0000 7f52af7fe700 1 -- 172.21.15.16:0/4112704570 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52b401d070 con 0x7f52c0068ea0 2024-04-22T15:54:54.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.701+0000 7f52af7fe700 1 -- 172.21.15.16:0/4112704570 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f52b4005dc0 con 0x7f52c0068ea0 2024-04-22T15:54:54.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.701+0000 7f52c49f5700 1 -- 172.21.15.16:0/4112704570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c0198be0 con 0x7f52c0068ea0 2024-04-22T15:54:54.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.701+0000 7f52af7fe700 1 -- 172.21.15.16:0/4112704570 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52b4017400 con 0x7f52c0068ea0 2024-04-22T15:54:54.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.701+0000 7f52c49f5700 1 -- 172.21.15.16:0/4112704570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c01990d0 con 0x7f52c0068ea0 2024-04-22T15:54:54.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.702+0000 7f52c49f5700 1 -- 172.21.15.16:0/4112704570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c010aa00 con 0x7f52c0068ea0 2024-04-22T15:54:54.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.703+0000 7f52af7fe700 1 -- 172.21.15.16:0/4112704570 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f52b4003a60 con 0x7f52c0068ea0 2024-04-22T15:54:54.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.703+0000 7f52af7fe700 1 --2- 172.21.15.16:0/4112704570 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f52a806ee00 0x7f52a80712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:54.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.703+0000 7f52af7fe700 1 -- 172.21.15.16:0/4112704570 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f52b40903d0 con 0x7f52c0068ea0 2024-04-22T15:54:54.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.703+0000 7f52be59c700 1 --2- 172.21.15.16:0/4112704570 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f52a806ee00 0x7f52a80712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:54.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.704+0000 7f52be59c700 1 --2- 172.21.15.16:0/4112704570 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f52a806ee00 0x7f52a80712c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f52b0005950 tx=0x7f52b000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:54.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.708+0000 7f52af7fe700 1 -- 172.21.15.16:0/4112704570 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f52b405cbc0 con 0x7f52c0068ea0 2024-04-22T15:54:54.845 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:54 smithi016 podman[121465]: 2024-04-22 15:54:54.82153757 +0000 UTC m=+1.732839318 container cleanup 3c6ebc4ea041915eced5b55ed096a280b2668243e7cf0d08a20681964a9d761e (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T15:54:54.845 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:54 smithi016 bash[121465]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0 2024-04-22T15:54:54.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.944+0000 7f52c49f5700 1 -- 172.21.15.16:0/4112704570 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f52c0199e70 con 0x7f52a806ee00 2024-04-22T15:54:54.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.945+0000 7f52af7fe700 1 -- 172.21.15.16:0/4112704570 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f52c0199e70 con 0x7f52a806ee00 2024-04-22T15:54:54.948 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:54:54.948 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:54:54.948 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-22T15:54:54.948 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:54:54.949 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-22T15:54:54.949 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-04-22T15:54:54.949 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr", 2024-04-22T15:54:54.949 INFO:teuthology.orchestra.run.smithi016.stdout: "mon" 2024-04-22T15:54:54.949 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-22T15:54:54.949 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "6/23 daemons upgraded", 2024-04-22T15:54:54.949 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading osd daemons", 2024-04-22T15:54:54.949 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-22T15:54:54.949 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:54:54.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.948+0000 7f52c49f5700 1 -- 172.21.15.16:0/4112704570 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f52a806ee00 msgr2=0x7f52a80712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:54.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.948+0000 7f52c49f5700 1 --2- 172.21.15.16:0/4112704570 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f52a806ee00 0x7f52a80712c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f52b0005950 tx=0x7f52b000b410 comp rx=0 tx=0).stop 2024-04-22T15:54:54.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.948+0000 7f52c49f5700 1 -- 172.21.15.16:0/4112704570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52c0068ea0 msgr2=0x7f52c019ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:54.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.948+0000 7f52c49f5700 1 --2- 172.21.15.16:0/4112704570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52c0068ea0 0x7f52c019ee20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f52b4006010 tx=0x7f52b400bd70 comp rx=0 tx=0).stop 2024-04-22T15:54:54.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.948+0000 7f52c49f5700 1 -- 172.21.15.16:0/4112704570 shutdown_connections 2024-04-22T15:54:54.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.948+0000 7f52c49f5700 1 --2- 172.21.15.16:0/4112704570 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f52a806ee00 0x7f52a80712c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.948+0000 7f52c49f5700 1 --2- 172.21.15.16:0/4112704570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c0068580 0x7f52c019e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.948+0000 7f52c49f5700 1 --2- 172.21.15.16:0/4112704570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52c0068ea0 0x7f52c019ee20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:54.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.948+0000 7f52c49f5700 1 -- 172.21.15.16:0/4112704570 >> 172.21.15.16:0/4112704570 conn(0x7f52c0075670 msgr2=0x7f52c00fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:54.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.948+0000 7f52c49f5700 1 -- 172.21.15.16:0/4112704570 shutdown_connections 2024-04-22T15:54:54.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:54.948+0000 7f52c49f5700 1 -- 172.21.15.16:0/4112704570 wait complete. 2024-04-22T15:54:55.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.062+0000 7fb94ca8a700 1 -- 172.21.15.16:0/2453975628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9480fff50 msgr2=0x7fb948105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:55.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.062+0000 7fb94ca8a700 1 --2- 172.21.15.16:0/2453975628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9480fff50 0x7fb948105840 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb93c009910 tx=0x7fb93c009c20 comp rx=0 tx=0).stop 2024-04-22T15:54:55.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.062+0000 7fb94ca8a700 1 -- 172.21.15.16:0/2453975628 shutdown_connections 2024-04-22T15:54:55.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.062+0000 7fb94ca8a700 1 --2- 172.21.15.16:0/2453975628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9480fff50 0x7fb948105840 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:55.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.062+0000 7fb94ca8a700 1 --2- 172.21.15.16:0/2453975628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9480ff5a0 0x7fb9480ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:55.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.062+0000 7fb94ca8a700 1 -- 172.21.15.16:0/2453975628 >> 172.21.15.16:0/2453975628 conn(0x7fb948074e30 msgr2=0x7fb948075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:55.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.062+0000 7fb94ca8a700 1 -- 172.21.15.16:0/2453975628 shutdown_connections 2024-04-22T15:54:55.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.062+0000 7fb94ca8a700 1 -- 172.21.15.16:0/2453975628 wait complete. 2024-04-22T15:54:55.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.063+0000 7fb94ca8a700 1 Processor -- start 2024-04-22T15:54:55.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.063+0000 7fb94ca8a700 1 -- start start 2024-04-22T15:54:55.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.063+0000 7fb94ca8a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9480ff5a0 0x7fb94819c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:55.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.063+0000 7fb94ca8a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9480fff50 0x7fb94819cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:55.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.063+0000 7fb94ca8a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94819d210 con 0x7fb9480ff5a0 2024-04-22T15:54:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.063+0000 7fb94ca8a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9481966c0 con 0x7fb9480fff50 2024-04-22T15:54:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.063+0000 7fb94659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9480ff5a0 0x7fb94819c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.063+0000 7fb94659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9480ff5a0 0x7fb94819c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42202/0 (socket says 172.21.15.16:42202) 2024-04-22T15:54:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.063+0000 7fb94659c700 1 -- 172.21.15.16:0/1164984886 learned_addr learned my addr 172.21.15.16:0/1164984886 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:54:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.064+0000 7fb945d9b700 1 --2- 172.21.15.16:0/1164984886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9480fff50 0x7fb94819cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.064+0000 7fb94659c700 1 -- 172.21.15.16:0/1164984886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9480fff50 msgr2=0x7fb94819cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.064+0000 7fb94659c700 1 --2- 172.21.15.16:0/1164984886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9480fff50 0x7fb94819cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.064+0000 7fb94659c700 1 -- 172.21.15.16:0/1164984886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb93c0095f0 con 0x7fb9480ff5a0 2024-04-22T15:54:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.064+0000 7fb945d9b700 1 --2- 172.21.15.16:0/1164984886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9480fff50 0x7fb94819cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:54:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.064+0000 7fb94659c700 1 --2- 172.21.15.16:0/1164984886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9480ff5a0 0x7fb94819c640 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb93800d8d0 tx=0x7fb93800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.064+0000 7fb9377fe700 1 -- 172.21.15.16:0/1164984886 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb938009940 con 0x7fb9480ff5a0 2024-04-22T15:54:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.064+0000 7fb94ca8a700 1 -- 172.21.15.16:0/1164984886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9481969a0 con 0x7fb9480ff5a0 2024-04-22T15:54:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.064+0000 7fb9377fe700 1 -- 172.21.15.16:0/1164984886 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb938010460 con 0x7fb9480ff5a0 2024-04-22T15:54:55.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.064+0000 7fb9377fe700 1 -- 172.21.15.16:0/1164984886 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb93800f5f0 con 0x7fb9480ff5a0 2024-04-22T15:54:55.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.064+0000 7fb94ca8a700 1 -- 172.21.15.16:0/1164984886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb948196ef0 con 0x7fb9480ff5a0 2024-04-22T15:54:55.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.065+0000 7fb94ca8a700 1 -- 172.21.15.16:0/1164984886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb948109200 con 0x7fb9480ff5a0 2024-04-22T15:54:55.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.066+0000 7fb9377fe700 1 -- 172.21.15.16:0/1164984886 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fb9380105d0 con 0x7fb9480ff5a0 2024-04-22T15:54:55.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.066+0000 7fb9377fe700 1 --2- 172.21.15.16:0/1164984886 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fb93006ee00 0x7fb9300712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:54:55.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.066+0000 7fb9377fe700 1 -- 172.21.15.16:0/1164984886 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fb93808f050 con 0x7fb9480ff5a0 2024-04-22T15:54:55.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.066+0000 7fb945d9b700 1 --2- 172.21.15.16:0/1164984886 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fb93006ee00 0x7fb9300712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:54:55.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.067+0000 7fb945d9b700 1 --2- 172.21.15.16:0/1164984886 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fb93006ee00 0x7fb9300712c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb93c00b5c0 tx=0x7fb93c0053a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:54:55.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.071+0000 7fb9377fe700 1 -- 172.21.15.16:0/1164984886 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb93805b7c0 con 0x7fb9480ff5a0 2024-04-22T15:54:55.235 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:55 smithi016 podman[121576]: 2024-04-22 15:54:55.081605939 +0000 UTC m=+0.729002954 container remove 3c6ebc4ea041915eced5b55ed096a280b2668243e7cf0d08a20681964a9d761e (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1, GIT_BRANCH=HEAD, RELEASE=HEAD) 2024-04-22T15:54:55.235 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:54 smithi016 ceph-mon[110740]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:55.235 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:54 smithi016 ceph-mon[110740]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:55.235 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:54 smithi016 ceph-mon[110740]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T15:54:55.235 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:54 smithi016 ceph-mon[110740]: osdmap e50: 6 total, 5 up, 6 in 2024-04-22T15:54:55.235 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:54 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/2059470006' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:54:55.235 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:54 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/226278781' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:54:55.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:54 smithi027 ceph-mon[83054]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:55.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:54 smithi027 ceph-mon[83054]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:55.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:54 smithi027 ceph-mon[83054]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T15:54:55.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:54 smithi027 ceph-mon[83054]: osdmap e50: 6 total, 5 up, 6 in 2024-04-22T15:54:55.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:54 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/2059470006' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:54:55.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:54 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/226278781' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:54:55.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.434+0000 7fb94ca8a700 1 -- 172.21.15.16:0/1164984886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb94804f2e0 con 0x7fb9480ff5a0 2024-04-22T15:54:55.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.435+0000 7fb9377fe700 1 -- 172.21.15.16:0/1164984886 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+290 (secure 0 0 0) 0x7fb93805b350 con 0x7fb9480ff5a0 2024-04-22T15:54:55.438 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data; 1 osds down 2024-04-22T15:54:55.438 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:54:55.438 INFO:teuthology.orchestra.run.smithi016.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:54:55.438 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-22T15:54:55.438 INFO:teuthology.orchestra.run.smithi016.stdout: osd.0 (root=default,host=smithi016) is down 2024-04-22T15:54:55.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.437+0000 7fb94ca8a700 1 -- 172.21.15.16:0/1164984886 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fb93006ee00 msgr2=0x7fb9300712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:55.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.437+0000 7fb94ca8a700 1 --2- 172.21.15.16:0/1164984886 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fb93006ee00 0x7fb9300712c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb93c00b5c0 tx=0x7fb93c0053a0 comp rx=0 tx=0).stop 2024-04-22T15:54:55.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.437+0000 7fb94ca8a700 1 -- 172.21.15.16:0/1164984886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9480ff5a0 msgr2=0x7fb94819c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:54:55.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.437+0000 7fb94ca8a700 1 --2- 172.21.15.16:0/1164984886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9480ff5a0 0x7fb94819c640 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb93800d8d0 tx=0x7fb93800dc90 comp rx=0 tx=0).stop 2024-04-22T15:54:55.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.437+0000 7fb94ca8a700 1 -- 172.21.15.16:0/1164984886 shutdown_connections 2024-04-22T15:54:55.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.437+0000 7fb94ca8a700 1 --2- 172.21.15.16:0/1164984886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9480ff5a0 0x7fb94819c640 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:55.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.437+0000 7fb94ca8a700 1 --2- 172.21.15.16:0/1164984886 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fb93006ee00 0x7fb9300712c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:55.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.437+0000 7fb94ca8a700 1 --2- 172.21.15.16:0/1164984886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9480fff50 0x7fb94819cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:54:55.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.437+0000 7fb94ca8a700 1 -- 172.21.15.16:0/1164984886 >> 172.21.15.16:0/1164984886 conn(0x7fb948074e30 msgr2=0x7fb9480fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:54:55.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.437+0000 7fb94ca8a700 1 -- 172.21.15.16:0/1164984886 shutdown_connections 2024-04-22T15:54:55.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:54:55.437+0000 7fb94ca8a700 1 -- 172.21.15.16:0/1164984886 wait complete. 2024-04-22T15:54:55.763 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:55 smithi016 podman[121715]: 2024-04-22 15:54:55.394317297 +0000 UTC m=+0.019044975 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:54:55.764 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:55 smithi016 podman[121715]: 2024-04-22 15:54:55.533473339 +0000 UTC m=+0.158201017 container create 8a8ce197be03d3424bfc790a4f3081b1f1c0cf85baa4f16b531b4c29b11bcb9b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-22T15:54:55.764 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:55 smithi016 podman[121715]: 2024-04-22 15:54:55.761259056 +0000 UTC m=+0.385986751 container init 8a8ce197be03d3424bfc790a4f3081b1f1c0cf85baa4f16b531b4c29b11bcb9b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49) 2024-04-22T15:54:56.073 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:55 smithi016 ceph-mon[110740]: pgmap v46: 65 pgs: 9 stale+active+clean, 56 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2024-04-22T15:54:56.073 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:55 smithi016 ceph-mon[110740]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:56.073 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:55 smithi016 ceph-mon[110740]: osdmap e51: 6 total, 5 up, 6 in 2024-04-22T15:54:56.073 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:55 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/1164984886' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:54:56.073 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:55 smithi016 podman[121715]: 2024-04-22 15:54:55.764768296 +0000 UTC m=+0.389495983 container start 8a8ce197be03d3424bfc790a4f3081b1f1c0cf85baa4f16b531b4c29b11bcb9b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.vendor=CentOS) 2024-04-22T15:54:56.073 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:55 smithi016 podman[121715]: 2024-04-22 15:54:55.86087006 +0000 UTC m=+0.485597753 container attach 8a8ce197be03d3424bfc790a4f3081b1f1c0cf85baa4f16b531b4c29b11bcb9b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240416, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T15:54:56.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:55 smithi027 ceph-mon[83054]: pgmap v46: 65 pgs: 9 stale+active+clean, 56 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2024-04-22T15:54:56.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:55 smithi027 ceph-mon[83054]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:54:56.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:55 smithi027 ceph-mon[83054]: osdmap e51: 6 total, 5 up, 6 in 2024-04-22T15:54:56.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:55 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/1164984886' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:54:56.416 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:56 smithi016 podman[121715]: 2024-04-22 15:54:56.071243937 +0000 UTC m=+0.695971628 container died 8a8ce197be03d3424bfc790a4f3081b1f1c0cf85baa4f16b531b4c29b11bcb9b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-22T15:54:57.042 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:57 smithi016 ceph-mon[110740]: Health check failed: Degraded data redundancy: 49/258 objects degraded (18.992%), 19 pgs degraded (PG_DEGRADED) 2024-04-22T15:54:57.042 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:56 smithi016 podman[121744]: 2024-04-22 15:54:56.782003854 +0000 UTC m=+0.705567405 container remove 8a8ce197be03d3424bfc790a4f3081b1f1c0cf85baa4f16b531b4c29b11bcb9b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T15:54:57.043 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:56 smithi016 systemd[1]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.0.service: Deactivated successfully. 2024-04-22T15:54:57.043 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:56 smithi016 systemd[1]: Stopped Ceph osd.0 for 67ab3a80-00be-11ef-bc93-c7b262605968. 2024-04-22T15:54:57.043 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:56 smithi016 systemd[1]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.0.service: Consumed 54.631s CPU time. 2024-04-22T15:54:57.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:57 smithi027 ceph-mon[83054]: Health check failed: Degraded data redundancy: 49/258 objects degraded (18.992%), 19 pgs degraded (PG_DEGRADED) 2024-04-22T15:54:57.366 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:57 smithi016 systemd[1]: Starting Ceph osd.0 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T15:54:57.366 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:57 smithi016 podman[121848]: 2024-04-22 15:54:57.258747343 +0000 UTC m=+0.114903704 container create 5a3645cd946207894848a25161f76eb7a666eeabac2340774d885eb311f82670 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-activate, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:54:57.366 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:57 smithi016 podman[121848]: 2024-04-22 15:54:57.162576883 +0000 UTC m=+0.018733257 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:54:57.666 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:57 smithi016 podman[121848]: 2024-04-22 15:54:57.444584024 +0000 UTC m=+0.300740397 container init 5a3645cd946207894848a25161f76eb7a666eeabac2340774d885eb311f82670 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-activate, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49) 2024-04-22T15:54:57.666 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:57 smithi016 podman[121848]: 2024-04-22 15:54:57.448503285 +0000 UTC m=+0.304659653 container start 5a3645cd946207894848a25161f76eb7a666eeabac2340774d885eb311f82670 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-22T15:54:57.667 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:57 smithi016 podman[121848]: 2024-04-22 15:54:57.536304572 +0000 UTC m=+0.392460941 container attach 5a3645cd946207894848a25161f76eb7a666eeabac2340774d885eb311f82670 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-activate, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-22T15:54:58.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:54:58 smithi016 ceph-mon[110740]: pgmap v48: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 49/258 objects degraded (18.992%) 2024-04-22T15:54:58.167 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:57 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-activate[121863]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:54:58.167 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:57 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-activate[121863]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:54:58.167 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:57 smithi016 bash[121848]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:54:58.167 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:57 smithi016 bash[121848]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:54:58.167 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:57 smithi016 bash[121848]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-22T15:54:58.167 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:57 smithi016 bash[121848]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-22T15:54:58.168 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:57 smithi016 bash[121848]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-22T15:54:58.168 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:57 smithi016 bash[121848]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-22T15:54:58.168 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:57 smithi016 bash[121848]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-22T15:54:58.168 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:57 smithi016 bash[121848]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-22T15:54:58.168 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:57 smithi016 bash[121848]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-22T15:54:58.168 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:57 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-activate[121863]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-22T15:54:58.168 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:57 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-activate[121863]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-22T15:54:58.168 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:57 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-activate[121863]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-22T15:54:58.168 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:57 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-activate[121863]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-22T15:54:58.168 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:57 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-activate[121863]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-22T15:54:58.168 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:57 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-activate[121863]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-22T15:54:58.168 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:57 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-activate[121863]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-22T15:54:58.168 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:57 smithi016 podman[121848]: 2024-04-22 15:54:57.976876018 +0000 UTC m=+0.833032379 container died 5a3645cd946207894848a25161f76eb7a666eeabac2340774d885eb311f82670 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-activate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-22T15:54:58.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:54:58 smithi027 ceph-mon[83054]: pgmap v48: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 49/258 objects degraded (18.992%) 2024-04-22T15:54:58.991 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:58 smithi016 podman[122011]: 2024-04-22 15:54:58.690435043 +0000 UTC m=+0.706917762 container remove 5a3645cd946207894848a25161f76eb7a666eeabac2340774d885eb311f82670 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416) 2024-04-22T15:54:58.991 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:58 smithi016 podman[122076]: 2024-04-22 15:54:58.900005056 +0000 UTC m=+0.106611733 container create 79217eb383a2dfd422853df80bae709b87ef450c586b17f3bdf8bbbf337fc00f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-22T15:54:58.991 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:58 smithi016 podman[122076]: 2024-04-22 15:54:58.812112522 +0000 UTC m=+0.018719199 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:54:59.416 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:59 smithi016 podman[122076]: 2024-04-22 15:54:59.102883456 +0000 UTC m=+0.309490134 container init 79217eb383a2dfd422853df80bae709b87ef450c586b17f3bdf8bbbf337fc00f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T15:54:59.417 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:59 smithi016 podman[122076]: 2024-04-22 15:54:59.106177716 +0000 UTC m=+0.312784394 container start 79217eb383a2dfd422853df80bae709b87ef450c586b17f3bdf8bbbf337fc00f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2024-04-22T15:54:59.417 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:59 smithi016 bash[122076]: 79217eb383a2dfd422853df80bae709b87ef450c586b17f3bdf8bbbf337fc00f 2024-04-22T15:54:59.417 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:59 smithi016 systemd[1]: Started Ceph osd.0 for 67ab3a80-00be-11ef-bc93-c7b262605968. 2024-04-22T15:54:59.916 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:54:59 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0[122090]: 2024-04-22T15:54:59.645+0000 7f4644a147c0 -1 Falling back to public interface 2024-04-22T15:55:00.283 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:00 smithi016 ceph-mon[110740]: pgmap v49: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 49/258 objects degraded (18.992%) 2024-04-22T15:55:00.284 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:00 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:00.284 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:00 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:00.284 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:00 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:55:00.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:00 smithi027 ceph-mon[83054]: pgmap v49: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 49/258 objects degraded (18.992%) 2024-04-22T15:55:00.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:00 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:00.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:00 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:00.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:00 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:55:01.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:01 smithi027 ceph-mon[83054]: pgmap v50: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 639 B/s rd, 1 op/s; 49/258 objects degraded (18.992%) 2024-04-22T15:55:01.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:01 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:01.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:01 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:55:01.621 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:01 smithi016 ceph-mon[110740]: pgmap v50: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 639 B/s rd, 1 op/s; 49/258 objects degraded (18.992%) 2024-04-22T15:55:01.621 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:01 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:01.621 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:01 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:55:03.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:03 smithi027 ceph-mon[83054]: pgmap v51: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 371 B/s wr, 2 op/s; 49/258 objects degraded (18.992%) 2024-04-22T15:55:03.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:03 smithi027 ceph-mon[83054]: mgrmap e38: smithi016.rnsnnr(active, since 92s), standbys: smithi027.ukdgzs 2024-04-22T15:55:03.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:03 smithi016 ceph-mon[110740]: pgmap v51: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 371 B/s wr, 2 op/s; 49/258 objects degraded (18.992%) 2024-04-22T15:55:03.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:03 smithi016 ceph-mon[110740]: mgrmap e38: smithi016.rnsnnr(active, since 92s), standbys: smithi027.ukdgzs 2024-04-22T15:55:04.416 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:55:04 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0[122090]: 2024-04-22T15:55:04.102+0000 7f4644a147c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-04-22T15:55:04.916 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:55:04 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0[122090]: 2024-04-22T15:55:04.468+0000 7f4644a147c0 -1 osd.0 49 log_to_monitors true 2024-04-22T15:55:05.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:05 smithi027 ceph-mon[83054]: pgmap v52: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 307 B/s wr, 2 op/s; 49/258 objects degraded (18.992%) 2024-04-22T15:55:05.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:05 smithi027 ceph-mon[83054]: from='osd.0 [v2:172.21.15.16:6802/3158080196,v1:172.21.15.16:6803/3158080196]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T15:55:05.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:05 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:05.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:05 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:05.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:05 smithi016 ceph-mon[110740]: pgmap v52: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 307 B/s wr, 2 op/s; 49/258 objects degraded (18.992%) 2024-04-22T15:55:05.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:05 smithi016 ceph-mon[110740]: from='osd.0 [v2:172.21.15.16:6802/3158080196,v1:172.21.15.16:6803/3158080196]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T15:55:05.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:05 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:05.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:05 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:06.416 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 15:55:06 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0[122090]: 2024-04-22T15:55:06.098+0000 7f4631e64700 -1 osd.0 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T15:55:06.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:06 smithi016 ceph-mon[110740]: from='osd.0 [v2:172.21.15.16:6802/3158080196,v1:172.21.15.16:6803/3158080196]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T15:55:06.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:06 smithi016 ceph-mon[110740]: osdmap e52: 6 total, 5 up, 6 in 2024-04-22T15:55:06.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:06 smithi016 ceph-mon[110740]: from='osd.0 [v2:172.21.15.16:6802/3158080196,v1:172.21.15.16:6803/3158080196]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-22T15:55:06.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:06 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:06.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:06 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:06.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:06 smithi016 ceph-mon[110740]: from='osd.0 [v2:172.21.15.16:6802/3158080196,v1:172.21.15.16:6803/3158080196]' entity='osd.0' 2024-04-22T15:55:06.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:06 smithi027 ceph-mon[83054]: from='osd.0 [v2:172.21.15.16:6802/3158080196,v1:172.21.15.16:6803/3158080196]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T15:55:06.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:06 smithi027 ceph-mon[83054]: osdmap e52: 6 total, 5 up, 6 in 2024-04-22T15:55:06.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:06 smithi027 ceph-mon[83054]: from='osd.0 [v2:172.21.15.16:6802/3158080196,v1:172.21.15.16:6803/3158080196]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-22T15:55:06.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:06 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:06.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:06 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:06.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:06 smithi027 ceph-mon[83054]: from='osd.0 [v2:172.21.15.16:6802/3158080196,v1:172.21.15.16:6803/3158080196]' entity='osd.0' 2024-04-22T15:55:07.543 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:07 smithi016 ceph-mon[110740]: pgmap v54: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 307 B/s wr, 2 op/s; 49/258 objects degraded (18.992%) 2024-04-22T15:55:07.543 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:07 smithi016 ceph-mon[110740]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T15:55:07.543 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:07 smithi016 ceph-mon[110740]: osd.0 [v2:172.21.15.16:6802/3158080196,v1:172.21.15.16:6803/3158080196] boot 2024-04-22T15:55:07.543 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:07 smithi016 ceph-mon[110740]: osdmap e53: 6 total, 6 up, 6 in 2024-04-22T15:55:07.543 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:07 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:55:07.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:07 smithi027 ceph-mon[83054]: pgmap v54: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 307 B/s wr, 2 op/s; 49/258 objects degraded (18.992%) 2024-04-22T15:55:07.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:07 smithi027 ceph-mon[83054]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T15:55:07.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:07 smithi027 ceph-mon[83054]: osd.0 [v2:172.21.15.16:6802/3158080196,v1:172.21.15.16:6803/3158080196] boot 2024-04-22T15:55:07.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:07 smithi027 ceph-mon[83054]: osdmap e53: 6 total, 6 up, 6 in 2024-04-22T15:55:07.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:07 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T15:55:09.410 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:09 smithi016 ceph-mon[110740]: osdmap e54: 6 total, 6 up, 6 in 2024-04-22T15:55:09.410 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:09 smithi016 ceph-mon[110740]: pgmap v57: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 49/258 objects degraded (18.992%) 2024-04-22T15:55:09.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:09 smithi027 ceph-mon[83054]: osdmap e54: 6 total, 6 up, 6 in 2024-04-22T15:55:09.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:09 smithi027 ceph-mon[83054]: pgmap v57: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 49/258 objects degraded (18.992%) 2024-04-22T15:55:10.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:10 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:10.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:10 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:10.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:10 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:55:10.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:10 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:55:10.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:10 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:10.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:10 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:55:10.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:10 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:55:10.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:10 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:55:10.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:10 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:55:10.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:10 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-22T15:55:10.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:10 smithi016 ceph-mon[110740]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-22T15:55:10.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:10 smithi016 ceph-mon[110740]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-04-22T15:55:10.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:10 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:10.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:10 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:10.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:10 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:55:10.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:10 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:55:10.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:10 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:10.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:10 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:55:10.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:10 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:55:10.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:10 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:55:10.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:10 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:55:10.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:10 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-22T15:55:10.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:10 smithi027 ceph-mon[83054]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-22T15:55:10.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:10 smithi027 ceph-mon[83054]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-04-22T15:55:11.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:11 smithi027 ceph-mon[83054]: pgmap v58: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 49/258 objects degraded (18.992%) 2024-04-22T15:55:11.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:11 smithi016 ceph-mon[110740]: pgmap v58: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 49/258 objects degraded (18.992%) 2024-04-22T15:55:12.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:12 smithi027 ceph-mon[83054]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 49/258 objects degraded (18.992%), 19 pgs degraded) 2024-04-22T15:55:12.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:12 smithi016 ceph-mon[110740]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 49/258 objects degraded (18.992%), 19 pgs degraded) 2024-04-22T15:55:13.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:13 smithi027 ceph-mon[83054]: pgmap v59: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 2 op/s 2024-04-22T15:55:13.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:13 smithi016 ceph-mon[110740]: pgmap v59: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 2 op/s 2024-04-22T15:55:15.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:15 smithi027 ceph-mon[83054]: pgmap v60: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 639 B/s rd, 1 op/s 2024-04-22T15:55:15.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:15 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:15.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:15 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:55:15.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:15 smithi016 ceph-mon[110740]: pgmap v60: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 639 B/s rd, 1 op/s 2024-04-22T15:55:15.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:15 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:15.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:15 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:55:17.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:17 smithi027 ceph-mon[83054]: pgmap v61: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-22T15:55:17.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:17 smithi016 ceph-mon[110740]: pgmap v61: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-22T15:55:19.821 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:19 smithi027 ceph-mon[83054]: pgmap v62: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1011 B/s rd, 1 op/s 2024-04-22T15:55:19.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:19 smithi016 ceph-mon[110740]: pgmap v62: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1011 B/s rd, 1 op/s 2024-04-22T15:55:21.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:21 smithi027 ceph-mon[83054]: pgmap v63: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:55:21.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:21 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:21.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:21 smithi016 ceph-mon[110740]: pgmap v63: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:55:21.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:21 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:23.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:23 smithi027 ceph-mon[83054]: pgmap v64: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:55:23.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:23 smithi016 ceph-mon[110740]: pgmap v64: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:55:25.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.552+0000 7f0519bd6700 1 -- 172.21.15.16:0/990089249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0514102720 msgr2=0x7f051410ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:25.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.552+0000 7f0519bd6700 1 --2- 172.21.15.16:0/990089249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0514102720 0x7f051410ac10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f04fc009910 tx=0x7f04fc009c20 comp rx=0 tx=0).stop 2024-04-22T15:55:25.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.552+0000 7f0519bd6700 1 -- 172.21.15.16:0/990089249 shutdown_connections 2024-04-22T15:55:25.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.552+0000 7f0519bd6700 1 --2- 172.21.15.16:0/990089249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0514102720 0x7f051410ac10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.552+0000 7f0519bd6700 1 --2- 172.21.15.16:0/990089249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0514101e00 0x7f05141021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.552+0000 7f0519bd6700 1 -- 172.21.15.16:0/990089249 >> 172.21.15.16:0/990089249 conn(0x7f05140fb650 msgr2=0x7f05140fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:25.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.552+0000 7f0519bd6700 1 -- 172.21.15.16:0/990089249 shutdown_connections 2024-04-22T15:55:25.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.552+0000 7f0519bd6700 1 -- 172.21.15.16:0/990089249 wait complete. 2024-04-22T15:55:25.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:25 smithi016 ceph-mon[110740]: pgmap v65: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:55:25.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:25 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-22T15:55:25.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:25 smithi016 ceph-mon[110740]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-22T15:55:25.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:25 smithi016 ceph-mon[110740]: Upgrade: osd.1 is safe to restart 2024-04-22T15:55:25.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.553+0000 7f0519bd6700 1 Processor -- start 2024-04-22T15:55:25.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.553+0000 7f0519bd6700 1 -- start start 2024-04-22T15:55:25.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.553+0000 7f0519bd6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0514101e00 0x7f051419c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:25.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.553+0000 7f0519bd6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0514102720 0x7f051419cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:25.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.553+0000 7f0519bd6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051419d400 con 0x7f0514102720 2024-04-22T15:55:25.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.553+0000 7f0519bd6700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05141968b0 con 0x7f0514101e00 2024-04-22T15:55:25.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.553+0000 7f0512ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0514102720 0x7f051419cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:25.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.553+0000 7f0512ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0514102720 0x7f051419cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40402/0 (socket says 172.21.15.16:40402) 2024-04-22T15:55:25.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.553+0000 7f0512ffd700 1 -- 172.21.15.16:0/937226851 learned_addr learned my addr 172.21.15.16:0/937226851 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:55:25.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.553+0000 7f05137fe700 1 --2- 172.21.15.16:0/937226851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0514101e00 0x7f051419c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:25.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.554+0000 7f0512ffd700 1 -- 172.21.15.16:0/937226851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0514101e00 msgr2=0x7f051419c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:25.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.554+0000 7f0512ffd700 1 --2- 172.21.15.16:0/937226851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0514101e00 0x7f051419c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.554+0000 7f0512ffd700 1 -- 172.21.15.16:0/937226851 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04fc0095f0 con 0x7f0514102720 2024-04-22T15:55:25.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.554+0000 7f0512ffd700 1 --2- 172.21.15.16:0/937226851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0514102720 0x7f051419cd70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f04fc00bd70 tx=0x7f04fc00bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:25.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.554+0000 7f0510ff9700 1 -- 172.21.15.16:0/937226851 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04fc01d070 con 0x7f0514102720 2024-04-22T15:55:25.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.554+0000 7f0510ff9700 1 -- 172.21.15.16:0/937226851 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f04fc00f460 con 0x7f0514102720 2024-04-22T15:55:25.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.554+0000 7f0519bd6700 1 -- 172.21.15.16:0/937226851 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0514196b30 con 0x7f0514102720 2024-04-22T15:55:25.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.554+0000 7f0510ff9700 1 -- 172.21.15.16:0/937226851 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04fc017630 con 0x7f0514102720 2024-04-22T15:55:25.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.554+0000 7f0519bd6700 1 -- 172.21.15.16:0/937226851 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0514197020 con 0x7f0514102720 2024-04-22T15:55:25.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.555+0000 7f0519bd6700 1 -- 172.21.15.16:0/937226851 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0514108310 con 0x7f0514102720 2024-04-22T15:55:25.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.556+0000 7f0510ff9700 1 -- 172.21.15.16:0/937226851 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f04fc017850 con 0x7f0514102720 2024-04-22T15:55:25.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.556+0000 7f0510ff9700 1 --2- 172.21.15.16:0/937226851 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f050006ecc0 0x7f0500071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:25.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.556+0000 7f0510ff9700 1 -- 172.21.15.16:0/937226851 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f04fc0901a0 con 0x7f0514102720 2024-04-22T15:55:25.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.556+0000 7f05137fe700 1 --2- 172.21.15.16:0/937226851 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f050006ecc0 0x7f0500071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:25.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.557+0000 7f05137fe700 1 --2- 172.21.15.16:0/937226851 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f050006ecc0 0x7f0500071180 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0504009a10 tx=0x7f0504009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:25.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.561+0000 7f0510ff9700 1 -- 172.21.15.16:0/937226851 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f04fc05cc20 con 0x7f0514102720 2024-04-22T15:55:25.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:25 smithi027 ceph-mon[83054]: pgmap v65: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:55:25.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:25 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-22T15:55:25.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:25 smithi027 ceph-mon[83054]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-22T15:55:25.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:25 smithi027 ceph-mon[83054]: Upgrade: osd.1 is safe to restart 2024-04-22T15:55:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.802+0000 7f0519bd6700 1 -- 172.21.15.16:0/937226851 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0514197dc0 con 0x7f050006ecc0 2024-04-22T15:55:25.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.803+0000 7f0510ff9700 1 -- 172.21.15.16:0/937226851 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f0514197dc0 con 0x7f050006ecc0 2024-04-22T15:55:25.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.805+0000 7f0519bd6700 1 -- 172.21.15.16:0/937226851 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f050006ecc0 msgr2=0x7f0500071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.805+0000 7f0519bd6700 1 --2- 172.21.15.16:0/937226851 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f050006ecc0 0x7f0500071180 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0504009a10 tx=0x7f0504009190 comp rx=0 tx=0).stop 2024-04-22T15:55:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.805+0000 7f0519bd6700 1 -- 172.21.15.16:0/937226851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0514102720 msgr2=0x7f051419cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.805+0000 7f0519bd6700 1 --2- 172.21.15.16:0/937226851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0514102720 0x7f051419cd70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f04fc00bd70 tx=0x7f04fc00bda0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.805+0000 7f0519bd6700 1 -- 172.21.15.16:0/937226851 shutdown_connections 2024-04-22T15:55:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.805+0000 7f0519bd6700 1 --2- 172.21.15.16:0/937226851 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f050006ecc0 0x7f0500071180 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.805+0000 7f0519bd6700 1 --2- 172.21.15.16:0/937226851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0514101e00 0x7f051419c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.805+0000 7f0519bd6700 1 --2- 172.21.15.16:0/937226851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0514102720 0x7f051419cd70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.805+0000 7f0519bd6700 1 -- 172.21.15.16:0/937226851 >> 172.21.15.16:0/937226851 conn(0x7f05140fb650 msgr2=0x7f0514105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.805+0000 7f0519bd6700 1 -- 172.21.15.16:0/937226851 shutdown_connections 2024-04-22T15:55:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.806+0000 7f0519bd6700 1 -- 172.21.15.16:0/937226851 wait complete. 2024-04-22T15:55:25.820 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:55:25.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.916+0000 7f9557eaa700 1 -- 172.21.15.16:0/607087922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95500fff50 msgr2=0x7f9550105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:25.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.916+0000 7f9557eaa700 1 --2- 172.21.15.16:0/607087922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95500fff50 0x7f9550105840 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9540009910 tx=0x7f9540009c20 comp rx=0 tx=0).stop 2024-04-22T15:55:25.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.916+0000 7f9557eaa700 1 -- 172.21.15.16:0/607087922 shutdown_connections 2024-04-22T15:55:25.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.916+0000 7f9557eaa700 1 --2- 172.21.15.16:0/607087922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95500fff50 0x7f9550105840 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.916+0000 7f9557eaa700 1 --2- 172.21.15.16:0/607087922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95500ff5a0 0x7f95500ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.916+0000 7f9557eaa700 1 -- 172.21.15.16:0/607087922 >> 172.21.15.16:0/607087922 conn(0x7f9550074e30 msgr2=0x7f9550075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:25.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.916+0000 7f9557eaa700 1 -- 172.21.15.16:0/607087922 shutdown_connections 2024-04-22T15:55:25.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.916+0000 7f9557eaa700 1 -- 172.21.15.16:0/607087922 wait complete. 2024-04-22T15:55:25.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.917+0000 7f9557eaa700 1 Processor -- start 2024-04-22T15:55:25.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.917+0000 7f9557eaa700 1 -- start start 2024-04-22T15:55:25.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.917+0000 7f9557eaa700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95500ff5a0 0x7f9550100520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:25.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.917+0000 7f9557eaa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95500fff50 0x7f9550100a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:25.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.917+0000 7f9557eaa700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9550100fa0 con 0x7f95500fff50 2024-04-22T15:55:25.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.917+0000 7f9557eaa700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95501010e0 con 0x7f95500ff5a0 2024-04-22T15:55:25.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.917+0000 7f9555445700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95500fff50 0x7f9550100a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:25.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.917+0000 7f9555445700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95500fff50 0x7f9550100a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40414/0 (socket says 172.21.15.16:40414) 2024-04-22T15:55:25.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.917+0000 7f9555445700 1 -- 172.21.15.16:0/173203175 learned_addr learned my addr 172.21.15.16:0/173203175 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:55:25.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.917+0000 7f9555c46700 1 --2- 172.21.15.16:0/173203175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95500ff5a0 0x7f9550100520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:25.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.917+0000 7f9555445700 1 -- 172.21.15.16:0/173203175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95500ff5a0 msgr2=0x7f9550100520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:25.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.917+0000 7f9555445700 1 --2- 172.21.15.16:0/173203175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95500ff5a0 0x7f9550100520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:25.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.917+0000 7f9555445700 1 -- 172.21.15.16:0/173203175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95400095f0 con 0x7f95500fff50 2024-04-22T15:55:25.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.918+0000 7f9555445700 1 --2- 172.21.15.16:0/173203175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95500fff50 0x7f9550100a60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f95400092e0 tx=0x7f954000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:25.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.918+0000 7f9546ffd700 1 -- 172.21.15.16:0/173203175 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f954001d070 con 0x7f95500fff50 2024-04-22T15:55:25.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.918+0000 7f9557eaa700 1 -- 172.21.15.16:0/173203175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9550101360 con 0x7f95500fff50 2024-04-22T15:55:25.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.918+0000 7f9546ffd700 1 -- 172.21.15.16:0/173203175 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9540005dc0 con 0x7f95500fff50 2024-04-22T15:55:25.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.918+0000 7f9546ffd700 1 -- 172.21.15.16:0/173203175 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9540017400 con 0x7f95500fff50 2024-04-22T15:55:25.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.918+0000 7f9557eaa700 1 -- 172.21.15.16:0/173203175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95501a2dc0 con 0x7f95500fff50 2024-04-22T15:55:25.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.919+0000 7f9557eaa700 1 -- 172.21.15.16:0/173203175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9550109200 con 0x7f95500fff50 2024-04-22T15:55:25.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.919+0000 7f9546ffd700 1 -- 172.21.15.16:0/173203175 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f954000f6f0 con 0x7f95500fff50 2024-04-22T15:55:25.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.920+0000 7f9546ffd700 1 --2- 172.21.15.16:0/173203175 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f953c06ecc0 0x7f953c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:25.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.920+0000 7f9546ffd700 1 -- 172.21.15.16:0/173203175 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f95400901a0 con 0x7f95500fff50 2024-04-22T15:55:25.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.920+0000 7f9555c46700 1 --2- 172.21.15.16:0/173203175 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f953c06ecc0 0x7f953c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:25.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.921+0000 7f9555c46700 1 --2- 172.21.15.16:0/173203175 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f953c06ecc0 0x7f953c071180 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f954c005950 tx=0x7f954c00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:25.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:25.925+0000 7f9546ffd700 1 -- 172.21.15.16:0/173203175 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f9540059000 con 0x7f95500fff50 2024-04-22T15:55:26.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.154+0000 7f9557eaa700 1 -- 172.21.15.16:0/173203175 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f955002cce0 con 0x7f953c06ecc0 2024-04-22T15:55:26.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.155+0000 7f9546ffd700 1 -- 172.21.15.16:0/173203175 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f955002cce0 con 0x7f953c06ecc0 2024-04-22T15:55:26.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.157+0000 7f9557eaa700 1 -- 172.21.15.16:0/173203175 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f953c06ecc0 msgr2=0x7f953c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:26.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.157+0000 7f9557eaa700 1 --2- 172.21.15.16:0/173203175 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f953c06ecc0 0x7f953c071180 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f954c005950 tx=0x7f954c00b410 comp rx=0 tx=0).stop 2024-04-22T15:55:26.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.157+0000 7f9557eaa700 1 -- 172.21.15.16:0/173203175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95500fff50 msgr2=0x7f9550100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:26.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.157+0000 7f9557eaa700 1 --2- 172.21.15.16:0/173203175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95500fff50 0x7f9550100a60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f95400092e0 tx=0x7f954000bd70 comp rx=0 tx=0).stop 2024-04-22T15:55:26.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.157+0000 7f9557eaa700 1 -- 172.21.15.16:0/173203175 shutdown_connections 2024-04-22T15:55:26.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.157+0000 7f9557eaa700 1 --2- 172.21.15.16:0/173203175 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f953c06ecc0 0x7f953c071180 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.157+0000 7f9557eaa700 1 --2- 172.21.15.16:0/173203175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95500ff5a0 0x7f9550100520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.157+0000 7f9557eaa700 1 --2- 172.21.15.16:0/173203175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95500fff50 0x7f9550100a60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.157+0000 7f9557eaa700 1 -- 172.21.15.16:0/173203175 >> 172.21.15.16:0/173203175 conn(0x7f9550074e30 msgr2=0x7f95500fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:26.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.157+0000 7f9557eaa700 1 -- 172.21.15.16:0/173203175 shutdown_connections 2024-04-22T15:55:26.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.157+0000 7f9557eaa700 1 -- 172.21.15.16:0/173203175 wait complete. 2024-04-22T15:55:26.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.266+0000 7fd6c37fe700 1 -- 172.21.15.16:0/1786600124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6bc101850 msgr2=0x7fd6bc1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:26.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.266+0000 7fd6c37fe700 1 --2- 172.21.15.16:0/1786600124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6bc101850 0x7fd6bc1058a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8009910 tx=0x7fd6b8009c20 comp rx=0 tx=0).stop 2024-04-22T15:55:26.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.266+0000 7fd6c37fe700 1 -- 172.21.15.16:0/1786600124 shutdown_connections 2024-04-22T15:55:26.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.266+0000 7fd6c37fe700 1 --2- 172.21.15.16:0/1786600124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6bc101850 0x7fd6bc1058a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.266+0000 7fd6c37fe700 1 --2- 172.21.15.16:0/1786600124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6bc100ea0 0x7fd6bc101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.266+0000 7fd6c37fe700 1 -- 172.21.15.16:0/1786600124 >> 172.21.15.16:0/1786600124 conn(0x7fd6bc0fc750 msgr2=0x7fd6bc0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:26.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.267+0000 7fd6c37fe700 1 -- 172.21.15.16:0/1786600124 shutdown_connections 2024-04-22T15:55:26.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.267+0000 7fd6c37fe700 1 -- 172.21.15.16:0/1786600124 wait complete. 2024-04-22T15:55:26.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.267+0000 7fd6c37fe700 1 Processor -- start 2024-04-22T15:55:26.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.267+0000 7fd6c37fe700 1 -- start start 2024-04-22T15:55:26.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.267+0000 7fd6c37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6bc100ea0 0x7fd6bc19c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:26.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.267+0000 7fd6c37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6bc101850 0x7fd6bc19cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:26.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.267+0000 7fd6c37fe700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc19d2d0 con 0x7fd6bc100ea0 2024-04-22T15:55:26.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.267+0000 7fd6c37fe700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc196780 con 0x7fd6bc101850 2024-04-22T15:55:26.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.268+0000 7fd6c159a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6bc100ea0 0x7fd6bc19c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:26.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.268+0000 7fd6c159a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6bc100ea0 0x7fd6bc19c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40432/0 (socket says 172.21.15.16:40432) 2024-04-22T15:55:26.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.268+0000 7fd6c159a700 1 -- 172.21.15.16:0/3806743534 learned_addr learned my addr 172.21.15.16:0/3806743534 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:55:26.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.268+0000 7fd6c0d99700 1 --2- 172.21.15.16:0/3806743534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6bc101850 0x7fd6bc19cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:26.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.268+0000 7fd6c159a700 1 -- 172.21.15.16:0/3806743534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6bc101850 msgr2=0x7fd6bc19cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:26.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.268+0000 7fd6c159a700 1 --2- 172.21.15.16:0/3806743534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6bc101850 0x7fd6bc19cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.268+0000 7fd6c159a700 1 -- 172.21.15.16:0/3806743534 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b80095f0 con 0x7fd6bc100ea0 2024-04-22T15:55:26.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.268+0000 7fd6c0d99700 1 --2- 172.21.15.16:0/3806743534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6bc101850 0x7fd6bc19cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:55:26.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.268+0000 7fd6c159a700 1 --2- 172.21.15.16:0/3806743534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6bc100ea0 0x7fd6bc19c700 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac00e920 tx=0x7fd6ac00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:26.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.268+0000 7fd6b27fc700 1 -- 172.21.15.16:0/3806743534 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6ac009790 con 0x7fd6bc100ea0 2024-04-22T15:55:26.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.268+0000 7fd6b27fc700 1 -- 172.21.15.16:0/3806743534 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd6ac00cba0 con 0x7fd6bc100ea0 2024-04-22T15:55:26.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.269+0000 7fd6b27fc700 1 -- 172.21.15.16:0/3806743534 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6ac018650 con 0x7fd6bc100ea0 2024-04-22T15:55:26.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.269+0000 7fd6c37fe700 1 -- 172.21.15.16:0/3806743534 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6bc196a60 con 0x7fd6bc100ea0 2024-04-22T15:55:26.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.269+0000 7fd6c37fe700 1 -- 172.21.15.16:0/3806743534 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6bc196f30 con 0x7fd6bc100ea0 2024-04-22T15:55:26.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.270+0000 7fd6c37fe700 1 -- 172.21.15.16:0/3806743534 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6bc1092a0 con 0x7fd6bc100ea0 2024-04-22T15:55:26.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.271+0000 7fd6b27fc700 1 -- 172.21.15.16:0/3806743534 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd6ac00cd10 con 0x7fd6bc100ea0 2024-04-22T15:55:26.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.271+0000 7fd6b27fc700 1 --2- 172.21.15.16:0/3806743534 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd6a806eab0 0x7fd6a8070f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:26.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.271+0000 7fd6b27fc700 1 -- 172.21.15.16:0/3806743534 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd6ac014070 con 0x7fd6bc100ea0 2024-04-22T15:55:26.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.271+0000 7fd6c0d99700 1 --2- 172.21.15.16:0/3806743534 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd6a806eab0 0x7fd6a8070f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:26.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.272+0000 7fd6c0d99700 1 --2- 172.21.15.16:0/3806743534 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd6a806eab0 0x7fd6a8070f70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8000c00 tx=0x7fd6b8009f70 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:26.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.276+0000 7fd6b27fc700 1 -- 172.21.15.16:0/3806743534 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd6ac05b9b0 con 0x7fd6bc100ea0 2024-04-22T15:55:26.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:26 smithi016 ceph-mon[110740]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.501+0000 7fd6c37fe700 1 -- 172.21.15.16:0/3806743534 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6bc197d00 con 0x7fd6a806eab0 2024-04-22T15:55:26.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.522+0000 7fd6b27fc700 1 -- 172.21.15.16:0/3806743534 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fd6bc197d00 con 0x7fd6a806eab0 2024-04-22T15:55:26.525 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:55:26.525 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.smithi016 smithi016 *:9093,9094 running (3m) 22s ago 14m 25.8M - 0.25.0 c8568f914cd2 ea0e0217d6f9 2024-04-22T15:55:26.525 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi016 smithi016 running (14m) 22s ago 14m 8120k - 18.2.1 5be31c24972a f8b627d3f41e 2024-04-22T15:55:26.525 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi027 smithi027 running (13m) 45s ago 13m 8066k - 18.2.1 5be31c24972a f00fd40b4220 2024-04-22T15:55:26.525 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi016 smithi016 running (54s) 22s ago 14m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 1feb008449ed 2024-04-22T15:55:26.526 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi027 smithi027 running (49s) 45s ago 13m 6656k - 19.0.0-3223-g5883d494 3d983c26e278 c7f862c6da00 2024-04-22T15:55:26.526 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.smithi016 smithi016 *:3000 running (3m) 22s ago 14m 82.1M - 9.4.12 2bacad6d85d8 388a3c9c1b3f 2024-04-22T15:55:26.526 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.fkkejm smithi016 running (8m) 22s ago 8m 16.5M - 18.2.1 5be31c24972a 4153ac94cea0 2024-04-22T15:55:26.526 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.pgvphu smithi016 running (8m) 22s ago 8m 91.9M - 18.2.1 5be31c24972a 93dc072068a2 2024-04-22T15:55:26.526 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.dgqyes smithi027 running (8m) 45s ago 8m 19.2M - 18.2.1 5be31c24972a 41a9220430fd 2024-04-22T15:55:26.526 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.iobjds smithi027 running (8m) 45s ago 8m 149M - 18.2.1 5be31c24972a c1c8d6770f02 2024-04-22T15:55:26.526 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi016.rnsnnr smithi016 *:8443,9283,8765 running (5m) 22s ago 15m 502M - 19.0.0-3223-g5883d494 3d983c26e278 09b4fe285bc4 2024-04-22T15:55:26.526 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi027.ukdgzs smithi027 *:8443,9283,8765 running (4m) 45s ago 13m 450M - 19.0.0-3223-g5883d494 3d983c26e278 7548da9b5eb9 2024-04-22T15:55:26.526 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi016 smithi016 running (2m) 22s ago 16m 45.5M 2048M 19.0.0-3223-g5883d494 3d983c26e278 4ae529a67806 2024-04-22T15:55:26.526 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi027 smithi027 running (99s) 45s ago 13m 39.0M 2048M 19.0.0-3223-g5883d494 3d983c26e278 f29fcb888801 2024-04-22T15:55:26.526 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi016 smithi016 *:9100 running (4m) 22s ago 14m 15.6M - 1.5.0 0da6a335fe13 35e46e4d2f3f 2024-04-22T15:55:26.526 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi027 smithi027 *:9100 running (4m) 45s ago 13m 15.5M - 1.5.0 0da6a335fe13 e682a7672e9b 2024-04-22T15:55:26.526 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (27s) 22s ago 12m 12.7M 1217M 19.0.0-3223-g5883d494 3d983c26e278 79217eb383a2 2024-04-22T15:55:26.526 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (12m) 22s ago 12m 206M 1217M 18.2.1 5be31c24972a 6aa2e6796784 2024-04-22T15:55:26.527 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (11m) 22s ago 11m 177M 1217M 18.2.1 5be31c24972a 3d9d310aa9eb 2024-04-22T15:55:26.527 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi027 running (11m) 45s ago 11m 250M 2240M 18.2.1 5be31c24972a cdf9b7aaf865 2024-04-22T15:55:26.527 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (10m) 45s ago 10m 206M 2240M 18.2.1 5be31c24972a 629397d49c76 2024-04-22T15:55:26.527 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi027 running (10m) 45s ago 10m 193M 2240M 18.2.1 5be31c24972a 23adcf675b04 2024-04-22T15:55:26.527 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.smithi016 smithi016 *:9095 running (4m) 22s ago 13m 71.6M - 2.43.0 a07b618ecd1d 373eaadee82d 2024-04-22T15:55:26.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.524+0000 7fd6c37fe700 1 -- 172.21.15.16:0/3806743534 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd6a806eab0 msgr2=0x7fd6a8070f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:26.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.524+0000 7fd6c37fe700 1 --2- 172.21.15.16:0/3806743534 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd6a806eab0 0x7fd6a8070f70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8000c00 tx=0x7fd6b8009f70 comp rx=0 tx=0).stop 2024-04-22T15:55:26.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.524+0000 7fd6c37fe700 1 -- 172.21.15.16:0/3806743534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6bc100ea0 msgr2=0x7fd6bc19c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:26.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.524+0000 7fd6c37fe700 1 --2- 172.21.15.16:0/3806743534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6bc100ea0 0x7fd6bc19c700 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac00e920 tx=0x7fd6ac00ece0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.524+0000 7fd6c37fe700 1 -- 172.21.15.16:0/3806743534 shutdown_connections 2024-04-22T15:55:26.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.524+0000 7fd6c37fe700 1 --2- 172.21.15.16:0/3806743534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6bc100ea0 0x7fd6bc19c700 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.524+0000 7fd6c37fe700 1 --2- 172.21.15.16:0/3806743534 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd6a806eab0 0x7fd6a8070f70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.524+0000 7fd6c37fe700 1 --2- 172.21.15.16:0/3806743534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6bc101850 0x7fd6bc19cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.524+0000 7fd6c37fe700 1 -- 172.21.15.16:0/3806743534 >> 172.21.15.16:0/3806743534 conn(0x7fd6bc0fc750 msgr2=0x7fd6bc0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:26.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.525+0000 7fd6c37fe700 1 -- 172.21.15.16:0/3806743534 shutdown_connections 2024-04-22T15:55:26.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.525+0000 7fd6c37fe700 1 -- 172.21.15.16:0/3806743534 wait complete. 2024-04-22T15:55:26.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:26 smithi027 ceph-mon[83054]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.633+0000 7ff21ef81700 1 -- 172.21.15.16:0/752900609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff218101060 msgr2=0x7ff2181047b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.633+0000 7ff21ef81700 1 --2- 172.21.15.16:0/752900609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff218101060 0x7ff2181047b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff204009910 tx=0x7ff204009c20 comp rx=0 tx=0).stop 2024-04-22T15:55:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.633+0000 7ff21ef81700 1 -- 172.21.15.16:0/752900609 shutdown_connections 2024-04-22T15:55:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.633+0000 7ff21ef81700 1 --2- 172.21.15.16:0/752900609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff218101060 0x7ff2181047b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.633+0000 7ff21ef81700 1 --2- 172.21.15.16:0/752900609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2181006b0 0x7ff218100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.633+0000 7ff21ef81700 1 -- 172.21.15.16:0/752900609 >> 172.21.15.16:0/752900609 conn(0x7ff218074e30 msgr2=0x7ff218075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.633+0000 7ff21ef81700 1 -- 172.21.15.16:0/752900609 shutdown_connections 2024-04-22T15:55:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.633+0000 7ff21ef81700 1 -- 172.21.15.16:0/752900609 wait complete. 2024-04-22T15:55:26.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.634+0000 7ff21ef81700 1 Processor -- start 2024-04-22T15:55:26.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.634+0000 7ff21ef81700 1 -- start start 2024-04-22T15:55:26.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.634+0000 7ff21ef81700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2181006b0 0x7ff21819b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.634+0000 7ff21ef81700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff218101060 0x7ff21819ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.634+0000 7ff21ef81700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21819c120 con 0x7ff218101060 2024-04-22T15:55:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.634+0000 7ff21ef81700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2181955d0 con 0x7ff2181006b0 2024-04-22T15:55:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.634+0000 7ff217fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff218101060 0x7ff21819ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.634+0000 7ff217fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff218101060 0x7ff21819ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40442/0 (socket says 172.21.15.16:40442) 2024-04-22T15:55:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.634+0000 7ff217fff700 1 -- 172.21.15.16:0/3100609553 learned_addr learned my addr 172.21.15.16:0/3100609553 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:55:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.634+0000 7ff21cd1d700 1 --2- 172.21.15.16:0/3100609553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2181006b0 0x7ff21819b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.635+0000 7ff217fff700 1 -- 172.21.15.16:0/3100609553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2181006b0 msgr2=0x7ff21819b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.635+0000 7ff217fff700 1 --2- 172.21.15.16:0/3100609553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2181006b0 0x7ff21819b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.635+0000 7ff217fff700 1 -- 172.21.15.16:0/3100609553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2040095f0 con 0x7ff218101060 2024-04-22T15:55:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.635+0000 7ff217fff700 1 --2- 172.21.15.16:0/3100609553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff218101060 0x7ff21819ba90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff20400bd70 tx=0x7ff20400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.635+0000 7ff215ffb700 1 -- 172.21.15.16:0/3100609553 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff20401d070 con 0x7ff218101060 2024-04-22T15:55:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.635+0000 7ff215ffb700 1 -- 172.21.15.16:0/3100609553 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff204005dc0 con 0x7ff218101060 2024-04-22T15:55:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.635+0000 7ff21ef81700 1 -- 172.21.15.16:0/3100609553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff218195850 con 0x7ff218101060 2024-04-22T15:55:26.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.635+0000 7ff215ffb700 1 -- 172.21.15.16:0/3100609553 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff204017400 con 0x7ff218101060 2024-04-22T15:55:26.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.635+0000 7ff21ef81700 1 -- 172.21.15.16:0/3100609553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff218195d40 con 0x7ff218101060 2024-04-22T15:55:26.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.636+0000 7ff21ef81700 1 -- 172.21.15.16:0/3100609553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff218108070 con 0x7ff218101060 2024-04-22T15:55:26.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.637+0000 7ff215ffb700 1 -- 172.21.15.16:0/3100609553 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff20400f460 con 0x7ff218101060 2024-04-22T15:55:26.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.638+0000 7ff215ffb700 1 --2- 172.21.15.16:0/3100609553 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ff20806ecc0 0x7ff208071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:26.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.638+0000 7ff215ffb700 1 -- 172.21.15.16:0/3100609553 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff204090130 con 0x7ff218101060 2024-04-22T15:55:26.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.638+0000 7ff21cd1d700 1 --2- 172.21.15.16:0/3100609553 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ff20806ecc0 0x7ff208071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:26.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.638+0000 7ff21cd1d700 1 --2- 172.21.15.16:0/3100609553 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ff20806ecc0 0x7ff208071180 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff20c004630 tx=0x7ff20c00b3c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:26.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.643+0000 7ff215ffb700 1 -- 172.21.15.16:0/3100609553 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff20405cb40 con 0x7ff218101060 2024-04-22T15:55:26.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.947+0000 7ff21ef81700 1 -- 172.21.15.16:0/3100609553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff21804f2e0 con 0x7ff218101060 2024-04-22T15:55:26.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.947+0000 7ff215ffb700 1 -- 172.21.15.16:0/3100609553 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7ff20405c6d0 con 0x7ff218101060 2024-04-22T15:55:26.950 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:55:26.950 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-22T15:55:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:55:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:55:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-22T15:55:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:55:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:55:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-22T15:55:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 5, 2024-04-22T15:55:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 1 2024-04-22T15:55:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:55:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-22T15:55:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T15:55:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:55:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-22T15:55:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 9, 2024-04-22T15:55:26.952 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 5 2024-04-22T15:55:26.952 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-22T15:55:26.952 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:55:26.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.949+0000 7ff21ef81700 1 -- 172.21.15.16:0/3100609553 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ff20806ecc0 msgr2=0x7ff208071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:26.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.949+0000 7ff21ef81700 1 --2- 172.21.15.16:0/3100609553 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ff20806ecc0 0x7ff208071180 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff20c004630 tx=0x7ff20c00b3c0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.949+0000 7ff21ef81700 1 -- 172.21.15.16:0/3100609553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff218101060 msgr2=0x7ff21819ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:26.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.949+0000 7ff21ef81700 1 --2- 172.21.15.16:0/3100609553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff218101060 0x7ff21819ba90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff20400bd70 tx=0x7ff20400bda0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.949+0000 7ff21ef81700 1 -- 172.21.15.16:0/3100609553 shutdown_connections 2024-04-22T15:55:26.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.949+0000 7ff21ef81700 1 --2- 172.21.15.16:0/3100609553 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ff20806ecc0 0x7ff208071180 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.949+0000 7ff21ef81700 1 --2- 172.21.15.16:0/3100609553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2181006b0 0x7ff21819b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.949+0000 7ff21ef81700 1 --2- 172.21.15.16:0/3100609553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff218101060 0x7ff21819ba90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:26.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.949+0000 7ff21ef81700 1 -- 172.21.15.16:0/3100609553 >> 172.21.15.16:0/3100609553 conn(0x7ff218074e30 msgr2=0x7ff218104020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:26.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.949+0000 7ff21ef81700 1 -- 172.21.15.16:0/3100609553 shutdown_connections 2024-04-22T15:55:26.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:26.950+0000 7ff21ef81700 1 -- 172.21.15.16:0/3100609553 wait complete. 2024-04-22T15:55:27.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.058+0000 7f2bcedab700 1 -- 172.21.15.16:0/1226470775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bc8100510 msgr2=0x7f2bc81049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:27.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.058+0000 7f2bcedab700 1 --2- 172.21.15.16:0/1226470775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bc8100510 0x7f2bc81049e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4009910 tx=0x7f2bb4009c20 comp rx=0 tx=0).stop 2024-04-22T15:55:27.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.058+0000 7f2bcedab700 1 -- 172.21.15.16:0/1226470775 shutdown_connections 2024-04-22T15:55:27.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.058+0000 7f2bcedab700 1 --2- 172.21.15.16:0/1226470775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bc8100510 0x7f2bc81049e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:27.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.058+0000 7f2bcedab700 1 --2- 172.21.15.16:0/1226470775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bc80ffbf0 0x7f2bc80fffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:27.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.058+0000 7f2bcedab700 1 -- 172.21.15.16:0/1226470775 >> 172.21.15.16:0/1226470775 conn(0x7f2bc80fb650 msgr2=0x7f2bc80fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:27.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.058+0000 7f2bcedab700 1 -- 172.21.15.16:0/1226470775 shutdown_connections 2024-04-22T15:55:27.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.058+0000 7f2bcedab700 1 -- 172.21.15.16:0/1226470775 wait complete. 2024-04-22T15:55:27.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.058+0000 7f2bcedab700 1 Processor -- start 2024-04-22T15:55:27.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.058+0000 7f2bcedab700 1 -- start start 2024-04-22T15:55:27.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.059+0000 7f2bcedab700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bc80ffbf0 0x7f2bc819cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:27.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.059+0000 7f2bcedab700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bc8100510 0x7f2bc819d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:27.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.059+0000 7f2bcedab700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc819dbe0 con 0x7f2bc8100510 2024-04-22T15:55:27.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.059+0000 7f2bcedab700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc81a1970 con 0x7f2bc80ffbf0 2024-04-22T15:55:27.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.059+0000 7f2bc7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bc8100510 0x7f2bc819d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:27.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.059+0000 7f2bc7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bc8100510 0x7f2bc819d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40450/0 (socket says 172.21.15.16:40450) 2024-04-22T15:55:27.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.059+0000 7f2bc7fff700 1 -- 172.21.15.16:0/2756935035 learned_addr learned my addr 172.21.15.16:0/2756935035 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:55:27.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.059+0000 7f2bccb47700 1 --2- 172.21.15.16:0/2756935035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bc80ffbf0 0x7f2bc819cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:27.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.059+0000 7f2bc7fff700 1 -- 172.21.15.16:0/2756935035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bc80ffbf0 msgr2=0x7f2bc819cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:27.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.059+0000 7f2bc7fff700 1 --2- 172.21.15.16:0/2756935035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bc80ffbf0 0x7f2bc819cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:27.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.059+0000 7f2bc7fff700 1 -- 172.21.15.16:0/2756935035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb40095f0 con 0x7f2bc8100510 2024-04-22T15:55:27.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.059+0000 7f2bccb47700 1 --2- 172.21.15.16:0/2756935035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bc80ffbf0 0x7f2bc819cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:55:27.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.059+0000 7f2bc7fff700 1 --2- 172.21.15.16:0/2756935035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bc8100510 0x7f2bc819d500 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2bb40092e0 tx=0x7f2bb400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:27.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.059+0000 7f2bc5ffb700 1 -- 172.21.15.16:0/2756935035 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bb401d070 con 0x7f2bc8100510 2024-04-22T15:55:27.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.059+0000 7f2bc5ffb700 1 -- 172.21.15.16:0/2756935035 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2bb400f460 con 0x7f2bc8100510 2024-04-22T15:55:27.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.059+0000 7f2bcedab700 1 -- 172.21.15.16:0/2756935035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc81a1bf0 con 0x7f2bc8100510 2024-04-22T15:55:27.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.060+0000 7f2bcedab700 1 -- 172.21.15.16:0/2756935035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc81a20e0 con 0x7f2bc8100510 2024-04-22T15:55:27.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.060+0000 7f2bc5ffb700 1 -- 172.21.15.16:0/2756935035 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bb402b610 con 0x7f2bc8100510 2024-04-22T15:55:27.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.061+0000 7f2bcedab700 1 -- 172.21.15.16:0/2756935035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc81020d0 con 0x7f2bc8100510 2024-04-22T15:55:27.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.061+0000 7f2bc5ffb700 1 -- 172.21.15.16:0/2756935035 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2bb400f5d0 con 0x7f2bc8100510 2024-04-22T15:55:27.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.061+0000 7f2bc5ffb700 1 --2- 172.21.15.16:0/2756935035 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f2bb806ecc0 0x7f2bb8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:27.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.062+0000 7f2bc5ffb700 1 -- 172.21.15.16:0/2756935035 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f2bb40902f0 con 0x7f2bc8100510 2024-04-22T15:55:27.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.062+0000 7f2bccb47700 1 --2- 172.21.15.16:0/2756935035 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f2bb806ecc0 0x7f2bb8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:27.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.062+0000 7f2bccb47700 1 --2- 172.21.15.16:0/2756935035 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f2bb806ecc0 0x7f2bb8071180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc005fd0 tx=0x7f2bbc005dc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:27.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.066+0000 7f2bc5ffb700 1 -- 172.21.15.16:0/2756935035 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2bb4059150 con 0x7f2bc8100510 2024-04-22T15:55:27.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.349+0000 7f2bcedab700 1 -- 172.21.15.16:0/2756935035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2bc819e320 con 0x7f2bc8100510 2024-04-22T15:55:27.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.349+0000 7f2bc5ffb700 1 -- 172.21.15.16:0/2756935035 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1902 (secure 0 0 0) 0x7f2bb405c810 con 0x7f2bc8100510 2024-04-22T15:55:27.352 INFO:teuthology.orchestra.run.smithi016.stdout:e16 2024-04-22T15:55:27.353 INFO:teuthology.orchestra.run.smithi016.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:55:27.353 INFO:teuthology.orchestra.run.smithi016.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:55:27.353 INFO:teuthology.orchestra.run.smithi016.stdout:legacy client fscid: 1 2024-04-22T15:55:27.353 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:55:27.353 INFO:teuthology.orchestra.run.smithi016.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:55:27.353 INFO:teuthology.orchestra.run.smithi016.stdout:fs_name cephfs 2024-04-22T15:55:27.353 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 16 2024-04-22T15:55:27.353 INFO:teuthology.orchestra.run.smithi016.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T15:55:27.353 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-04-22T15:46:27.774571+0000 2024-04-22T15:55:27.353 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-04-22T15:46:47.381709+0000 2024-04-22T15:55:27.353 INFO:teuthology.orchestra.run.smithi016.stdout:tableserver 0 2024-04-22T15:55:27.353 INFO:teuthology.orchestra.run.smithi016.stdout:root 0 2024-04-22T15:55:27.353 INFO:teuthology.orchestra.run.smithi016.stdout:session_timeout 60 2024-04-22T15:55:27.353 INFO:teuthology.orchestra.run.smithi016.stdout:session_autoclose 300 2024-04-22T15:55:27.354 INFO:teuthology.orchestra.run.smithi016.stdout:max_file_size 1099511627776 2024-04-22T15:55:27.354 INFO:teuthology.orchestra.run.smithi016.stdout:max_xattr_size 65536 2024-04-22T15:55:27.354 INFO:teuthology.orchestra.run.smithi016.stdout:required_client_features {} 2024-04-22T15:55:27.354 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure 0 2024-04-22T15:55:27.354 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure_osd_epoch 46 2024-04-22T15:55:27.354 INFO:teuthology.orchestra.run.smithi016.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:55:27.354 INFO:teuthology.orchestra.run.smithi016.stdout:max_mds 1 2024-04-22T15:55:27.354 INFO:teuthology.orchestra.run.smithi016.stdout:in 0 2024-04-22T15:55:27.354 INFO:teuthology.orchestra.run.smithi016.stdout:up {0=24279} 2024-04-22T15:55:27.354 INFO:teuthology.orchestra.run.smithi016.stdout:failed 2024-04-22T15:55:27.354 INFO:teuthology.orchestra.run.smithi016.stdout:damaged 2024-04-22T15:55:27.354 INFO:teuthology.orchestra.run.smithi016.stdout:stopped 2024-04-22T15:55:27.354 INFO:teuthology.orchestra.run.smithi016.stdout:data_pools [3] 2024-04-22T15:55:27.354 INFO:teuthology.orchestra.run.smithi016.stdout:metadata_pool 2 2024-04-22T15:55:27.355 INFO:teuthology.orchestra.run.smithi016.stdout:inline_data enabled 2024-04-22T15:55:27.355 INFO:teuthology.orchestra.run.smithi016.stdout:balancer 2024-04-22T15:55:27.355 INFO:teuthology.orchestra.run.smithi016.stdout:bal_rank_mask -1 2024-04-22T15:55:27.355 INFO:teuthology.orchestra.run.smithi016.stdout:standby_count_wanted 1 2024-04-22T15:55:27.355 INFO:teuthology.orchestra.run.smithi016.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:55:27.355 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.iobjds{0:24279} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.27:6824/4216277609,v1:172.21.15.27:6825/4216277609] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:55:27.355 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.pgvphu{0:14494} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.16:6828/1006073414,v1:172.21.15.16:6829/1006073414] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:55:27.355 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:55:27.355 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:55:27.355 INFO:teuthology.orchestra.run.smithi016.stdout:Standby daemons: 2024-04-22T15:55:27.355 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:55:27.355 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.fkkejm{-1:14504} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.16:6826/3484907755,v1:172.21.15.16:6827/3484907755] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:55:27.355 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.dgqyes{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.27:6826/476026355,v1:172.21.15.27:6827/476026355] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:55:27.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.352+0000 7f2bcedab700 1 -- 172.21.15.16:0/2756935035 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f2bb806ecc0 msgr2=0x7f2bb8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:27.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.352+0000 7f2bcedab700 1 --2- 172.21.15.16:0/2756935035 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f2bb806ecc0 0x7f2bb8071180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc005fd0 tx=0x7f2bbc005dc0 comp rx=0 tx=0).stop 2024-04-22T15:55:27.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.352+0000 7f2bcedab700 1 -- 172.21.15.16:0/2756935035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bc8100510 msgr2=0x7f2bc819d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:27.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.352+0000 7f2bcedab700 1 --2- 172.21.15.16:0/2756935035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bc8100510 0x7f2bc819d500 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2bb40092e0 tx=0x7f2bb400be20 comp rx=0 tx=0).stop 2024-04-22T15:55:27.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.352+0000 7f2bcedab700 1 -- 172.21.15.16:0/2756935035 shutdown_connections 2024-04-22T15:55:27.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.352+0000 7f2bcedab700 1 --2- 172.21.15.16:0/2756935035 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f2bb806ecc0 0x7f2bb8071180 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:27.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.352+0000 7f2bcedab700 1 --2- 172.21.15.16:0/2756935035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bc80ffbf0 0x7f2bc819cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:27.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.352+0000 7f2bcedab700 1 --2- 172.21.15.16:0/2756935035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bc8100510 0x7f2bc819d500 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:27.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.352+0000 7f2bcedab700 1 -- 172.21.15.16:0/2756935035 >> 172.21.15.16:0/2756935035 conn(0x7f2bc80fb650 msgr2=0x7f2bc8111060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:27.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.352+0000 7f2bcedab700 1 -- 172.21.15.16:0/2756935035 shutdown_connections 2024-04-22T15:55:27.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.352+0000 7f2bcedab700 1 -- 172.21.15.16:0/2756935035 wait complete. 2024-04-22T15:55:27.357 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 16 2024-04-22T15:55:27.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.461+0000 7f7e38f6f700 1 -- 172.21.15.16:0/3920333152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e340ffda0 msgr2=0x7f7e34100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:27.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.461+0000 7f7e38f6f700 1 --2- 172.21.15.16:0/3920333152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e340ffda0 0x7f7e34100180 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7e24009910 tx=0x7f7e24009c20 comp rx=0 tx=0).stop 2024-04-22T15:55:27.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.461+0000 7f7e38f6f700 1 -- 172.21.15.16:0/3920333152 shutdown_connections 2024-04-22T15:55:27.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.461+0000 7f7e38f6f700 1 --2- 172.21.15.16:0/3920333152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e34100750 0x7f7e341047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:27.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.461+0000 7f7e38f6f700 1 --2- 172.21.15.16:0/3920333152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e340ffda0 0x7f7e34100180 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:27.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.461+0000 7f7e38f6f700 1 -- 172.21.15.16:0/3920333152 >> 172.21.15.16:0/3920333152 conn(0x7f7e340fb650 msgr2=0x7f7e340fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:27.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.461+0000 7f7e38f6f700 1 -- 172.21.15.16:0/3920333152 shutdown_connections 2024-04-22T15:55:27.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.461+0000 7f7e38f6f700 1 -- 172.21.15.16:0/3920333152 wait complete. 2024-04-22T15:55:27.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.462+0000 7f7e38f6f700 1 Processor -- start 2024-04-22T15:55:27.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.462+0000 7f7e38f6f700 1 -- start start 2024-04-22T15:55:27.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.462+0000 7f7e38f6f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e340ffda0 0x7f7e3419c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:27.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.462+0000 7f7e38f6f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e34100750 0x7f7e3419cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:27.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.462+0000 7f7e38f6f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3419d220 con 0x7f7e34100750 2024-04-22T15:55:27.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.462+0000 7f7e38f6f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e34196760 con 0x7f7e340ffda0 2024-04-22T15:55:27.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.462+0000 7f7e31d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e34100750 0x7f7e3419cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:27.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.462+0000 7f7e31d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e34100750 0x7f7e3419cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40468/0 (socket says 172.21.15.16:40468) 2024-04-22T15:55:27.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.462+0000 7f7e31d9b700 1 -- 172.21.15.16:0/4021757886 learned_addr learned my addr 172.21.15.16:0/4021757886 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:55:27.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.462+0000 7f7e3259c700 1 --2- 172.21.15.16:0/4021757886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e340ffda0 0x7f7e3419c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:27.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.462+0000 7f7e31d9b700 1 -- 172.21.15.16:0/4021757886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e340ffda0 msgr2=0x7f7e3419c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:27.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.462+0000 7f7e31d9b700 1 --2- 172.21.15.16:0/4021757886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e340ffda0 0x7f7e3419c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:27.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.462+0000 7f7e31d9b700 1 -- 172.21.15.16:0/4021757886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e240095f0 con 0x7f7e34100750 2024-04-22T15:55:27.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.462+0000 7f7e3259c700 1 --2- 172.21.15.16:0/4021757886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e340ffda0 0x7f7e3419c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:55:27.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.463+0000 7f7e31d9b700 1 --2- 172.21.15.16:0/4021757886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e34100750 0x7f7e3419cc20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800c960 tx=0x7f7e2800cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:27.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.463+0000 7f7e237fe700 1 -- 172.21.15.16:0/4021757886 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e28007a70 con 0x7f7e34100750 2024-04-22T15:55:27.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.463+0000 7f7e237fe700 1 -- 172.21.15.16:0/4021757886 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e28007bd0 con 0x7f7e34100750 2024-04-22T15:55:27.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.463+0000 7f7e38f6f700 1 -- 172.21.15.16:0/4021757886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e34196a40 con 0x7f7e34100750 2024-04-22T15:55:27.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.463+0000 7f7e237fe700 1 -- 172.21.15.16:0/4021757886 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e280186c0 con 0x7f7e34100750 2024-04-22T15:55:27.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.463+0000 7f7e38f6f700 1 -- 172.21.15.16:0/4021757886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e34196f90 con 0x7f7e34100750 2024-04-22T15:55:27.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.464+0000 7f7e38f6f700 1 -- 172.21.15.16:0/4021757886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e3404f2e0 con 0x7f7e34100750 2024-04-22T15:55:27.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.464+0000 7f7e237fe700 1 -- 172.21.15.16:0/4021757886 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7e2800f450 con 0x7f7e34100750 2024-04-22T15:55:27.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.465+0000 7f7e237fe700 1 --2- 172.21.15.16:0/4021757886 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f7e1c06ecc0 0x7f7e1c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:27.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.465+0000 7f7e237fe700 1 -- 172.21.15.16:0/4021757886 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f7e2808ee00 con 0x7f7e34100750 2024-04-22T15:55:27.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.465+0000 7f7e3259c700 1 --2- 172.21.15.16:0/4021757886 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f7e1c06ecc0 0x7f7e1c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:27.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.465+0000 7f7e3259c700 1 --2- 172.21.15.16:0/4021757886 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f7e1c06ecc0 0x7f7e1c071180 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7e2400b5c0 tx=0x7f7e2401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:27.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.470+0000 7f7e237fe700 1 -- 172.21.15.16:0/4021757886 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7e2805b500 con 0x7f7e34100750 2024-04-22T15:55:27.473 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:27 smithi016 ceph-mon[110740]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:27.473 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:27 smithi016 ceph-mon[110740]: pgmap v66: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:55:27.473 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:27 smithi016 ceph-mon[110740]: Upgrade: Updating osd.1 2024-04-22T15:55:27.473 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:27 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:27.473 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:27 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T15:55:27.473 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:27 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:55:27.473 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:27 smithi016 ceph-mon[110740]: Deploying daemon osd.1 on smithi016 2024-04-22T15:55:27.473 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:27 smithi016 ceph-mon[110740]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:27.473 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:27 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/3100609553' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:55:27.473 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:27 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/2756935035' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:55:27.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.704+0000 7f7e38f6f700 1 -- 172.21.15.16:0/4021757886 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e34197210 con 0x7f7e1c06ecc0 2024-04-22T15:55:27.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.705+0000 7f7e237fe700 1 -- 172.21.15.16:0/4021757886 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f7e34197210 con 0x7f7e1c06ecc0 2024-04-22T15:55:27.708 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:55:27.709 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:55:27.709 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-22T15:55:27.709 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:55:27.709 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-22T15:55:27.709 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-04-22T15:55:27.709 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr", 2024-04-22T15:55:27.709 INFO:teuthology.orchestra.run.smithi016.stdout: "mon" 2024-04-22T15:55:27.709 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-22T15:55:27.709 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "7/23 daemons upgraded", 2024-04-22T15:55:27.709 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading osd daemons", 2024-04-22T15:55:27.710 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-22T15:55:27.710 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:55:27.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.707+0000 7f7e38f6f700 1 -- 172.21.15.16:0/4021757886 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f7e1c06ecc0 msgr2=0x7f7e1c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:27.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.707+0000 7f7e38f6f700 1 --2- 172.21.15.16:0/4021757886 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f7e1c06ecc0 0x7f7e1c071180 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7e2400b5c0 tx=0x7f7e2401a040 comp rx=0 tx=0).stop 2024-04-22T15:55:27.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.707+0000 7f7e38f6f700 1 -- 172.21.15.16:0/4021757886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e34100750 msgr2=0x7f7e3419cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:27.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.707+0000 7f7e38f6f700 1 --2- 172.21.15.16:0/4021757886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e34100750 0x7f7e3419cc20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800c960 tx=0x7f7e2800cd20 comp rx=0 tx=0).stop 2024-04-22T15:55:27.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.707+0000 7f7e38f6f700 1 -- 172.21.15.16:0/4021757886 shutdown_connections 2024-04-22T15:55:27.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.707+0000 7f7e38f6f700 1 --2- 172.21.15.16:0/4021757886 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f7e1c06ecc0 0x7f7e1c071180 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.707+0000 7f7e38f6f700 1 --2- 172.21.15.16:0/4021757886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e340ffda0 0x7f7e3419c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.707+0000 7f7e38f6f700 1 --2- 172.21.15.16:0/4021757886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e34100750 0x7f7e3419cc20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.707+0000 7f7e38f6f700 1 -- 172.21.15.16:0/4021757886 >> 172.21.15.16:0/4021757886 conn(0x7f7e340fb650 msgr2=0x7f7e34103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.707+0000 7f7e38f6f700 1 -- 172.21.15.16:0/4021757886 shutdown_connections 2024-04-22T15:55:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.707+0000 7f7e38f6f700 1 -- 172.21.15.16:0/4021757886 wait complete. 2024-04-22T15:55:27.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.816+0000 7f85d9387700 1 -- 172.21.15.16:0/566658283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85d4101060 msgr2=0x7f85d41047b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:27.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.816+0000 7f85d9387700 1 --2- 172.21.15.16:0/566658283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85d4101060 0x7f85d41047b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f85c8009910 tx=0x7f85c8009c20 comp rx=0 tx=0).stop 2024-04-22T15:55:27.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.816+0000 7f85d9387700 1 -- 172.21.15.16:0/566658283 shutdown_connections 2024-04-22T15:55:27.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.816+0000 7f85d9387700 1 --2- 172.21.15.16:0/566658283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85d4101060 0x7f85d41047b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:27.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.816+0000 7f85d9387700 1 --2- 172.21.15.16:0/566658283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85d41006b0 0x7f85d4100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:27.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.816+0000 7f85d9387700 1 -- 172.21.15.16:0/566658283 >> 172.21.15.16:0/566658283 conn(0x7f85d4074e30 msgr2=0x7f85d4075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:27.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.816+0000 7f85d9387700 1 -- 172.21.15.16:0/566658283 shutdown_connections 2024-04-22T15:55:27.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.816+0000 7f85d9387700 1 -- 172.21.15.16:0/566658283 wait complete. 2024-04-22T15:55:27.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.817+0000 7f85d9387700 1 Processor -- start 2024-04-22T15:55:27.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.817+0000 7f85d9387700 1 -- start start 2024-04-22T15:55:27.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.817+0000 7f85d9387700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85d41006b0 0x7f85d41976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:27.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.817+0000 7f85d9387700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85d4101060 0x7f85d4197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:27.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.817+0000 7f85d9387700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d4198250 con 0x7f85d4101060 2024-04-22T15:55:27.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.817+0000 7f85d9387700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d4198390 con 0x7f85d41006b0 2024-04-22T15:55:27.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.818+0000 7f85d27fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85d4101060 0x7f85d4197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:27.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.818+0000 7f85d27fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85d4101060 0x7f85d4197c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40486/0 (socket says 172.21.15.16:40486) 2024-04-22T15:55:27.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.818+0000 7f85d27fc700 1 -- 172.21.15.16:0/2028612058 learned_addr learned my addr 172.21.15.16:0/2028612058 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:55:27.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.818+0000 7f85d2ffd700 1 --2- 172.21.15.16:0/2028612058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85d41006b0 0x7f85d41976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:27.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.818+0000 7f85d27fc700 1 -- 172.21.15.16:0/2028612058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85d41006b0 msgr2=0x7f85d41976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:27.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.818+0000 7f85d27fc700 1 --2- 172.21.15.16:0/2028612058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85d41006b0 0x7f85d41976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:27.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.818+0000 7f85d27fc700 1 -- 172.21.15.16:0/2028612058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c80095f0 con 0x7f85d4101060 2024-04-22T15:55:27.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.818+0000 7f85d27fc700 1 --2- 172.21.15.16:0/2028612058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85d4101060 0x7f85d4197c30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f85c80092e0 tx=0x7f85c800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:27.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.818+0000 7f85bbfff700 1 -- 172.21.15.16:0/2028612058 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85c801d070 con 0x7f85d4101060 2024-04-22T15:55:27.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.818+0000 7f85bbfff700 1 -- 172.21.15.16:0/2028612058 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f85c8005dc0 con 0x7f85d4101060 2024-04-22T15:55:27.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.818+0000 7f85bbfff700 1 -- 172.21.15.16:0/2028612058 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85c8017400 con 0x7f85d4101060 2024-04-22T15:55:27.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.818+0000 7f85d9387700 1 -- 172.21.15.16:0/2028612058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85d41a0760 con 0x7f85d4101060 2024-04-22T15:55:27.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.818+0000 7f85d9387700 1 -- 172.21.15.16:0/2028612058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d41a0b70 con 0x7f85d4101060 2024-04-22T15:55:27.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.819+0000 7f85d9387700 1 -- 172.21.15.16:0/2028612058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85d4108090 con 0x7f85d4101060 2024-04-22T15:55:27.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.820+0000 7f85bbfff700 1 -- 172.21.15.16:0/2028612058 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f85c800f460 con 0x7f85d4101060 2024-04-22T15:55:27.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.820+0000 7f85bbfff700 1 --2- 172.21.15.16:0/2028612058 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f85bc06ecc0 0x7f85bc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:27.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.821+0000 7f85d2ffd700 1 --2- 172.21.15.16:0/2028612058 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f85bc06ecc0 0x7f85bc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:27.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.821+0000 7f85bbfff700 1 -- 172.21.15.16:0/2028612058 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f85c8090210 con 0x7f85d4101060 2024-04-22T15:55:27.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.821+0000 7f85d2ffd700 1 --2- 172.21.15.16:0/2028612058 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f85bc06ecc0 0x7f85bc071180 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f85c4005fd0 tx=0x7f85c4005f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:27.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:27.825+0000 7f85bbfff700 1 -- 172.21.15.16:0/2028612058 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f85c80590f0 con 0x7f85d4101060 2024-04-22T15:55:27.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:27 smithi027 ceph-mon[83054]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:27.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:27 smithi027 ceph-mon[83054]: pgmap v66: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:55:27.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:27 smithi027 ceph-mon[83054]: Upgrade: Updating osd.1 2024-04-22T15:55:27.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:27 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:27.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:27 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T15:55:27.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:27 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:55:27.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:27 smithi027 ceph-mon[83054]: Deploying daemon osd.1 on smithi016 2024-04-22T15:55:27.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:27 smithi027 ceph-mon[83054]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:27.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:27 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/3100609553' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:55:27.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:27 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/2756935035' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:55:28.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:28.132+0000 7f85d9387700 1 -- 172.21.15.16:0/2028612058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f85d4107a70 con 0x7f85d4101060 2024-04-22T15:55:28.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:28.132+0000 7f85bbfff700 1 -- 172.21.15.16:0/2028612058 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f85c805c7b0 con 0x7f85d4101060 2024-04-22T15:55:28.136 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:55:28.136 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:55:28.136 INFO:teuthology.orchestra.run.smithi016.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:55:28.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:28.135+0000 7f85d9387700 1 -- 172.21.15.16:0/2028612058 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f85bc06ecc0 msgr2=0x7f85bc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:28.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:28.135+0000 7f85d9387700 1 --2- 172.21.15.16:0/2028612058 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f85bc06ecc0 0x7f85bc071180 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f85c4005fd0 tx=0x7f85c4005f60 comp rx=0 tx=0).stop 2024-04-22T15:55:28.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:28.135+0000 7f85d9387700 1 -- 172.21.15.16:0/2028612058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85d4101060 msgr2=0x7f85d4197c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:28.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:28.135+0000 7f85d9387700 1 --2- 172.21.15.16:0/2028612058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85d4101060 0x7f85d4197c30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f85c80092e0 tx=0x7f85c800be20 comp rx=0 tx=0).stop 2024-04-22T15:55:28.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:28.135+0000 7f85d9387700 1 -- 172.21.15.16:0/2028612058 shutdown_connections 2024-04-22T15:55:28.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:28.135+0000 7f85d9387700 1 --2- 172.21.15.16:0/2028612058 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f85bc06ecc0 0x7f85bc071180 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:28.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:28.135+0000 7f85d9387700 1 --2- 172.21.15.16:0/2028612058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85d41006b0 0x7f85d41976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:28.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:28.135+0000 7f85d9387700 1 --2- 172.21.15.16:0/2028612058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85d4101060 0x7f85d4197c30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:28.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:28.135+0000 7f85d9387700 1 -- 172.21.15.16:0/2028612058 >> 172.21.15.16:0/2028612058 conn(0x7f85d4074e30 msgr2=0x7f85d40fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:28.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:28.135+0000 7f85d9387700 1 -- 172.21.15.16:0/2028612058 shutdown_connections 2024-04-22T15:55:28.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:28.135+0000 7f85d9387700 1 -- 172.21.15.16:0/2028612058 wait complete. 2024-04-22T15:55:28.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:28 smithi016 ceph-mon[110740]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:28.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:28 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/2028612058' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:55:28.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:28 smithi027 ceph-mon[83054]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:28.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:28 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/2028612058' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:55:29.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:29 smithi016 ceph-mon[110740]: pgmap v67: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:55:29.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:29 smithi027 ceph-mon[83054]: pgmap v67: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:55:30.073 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:29 smithi016 systemd[1]: Stopping Ceph osd.1 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T15:55:30.383 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:30 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1[53304]: 2024-04-22T15:55:30.070+0000 7f29d32fe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T15:55:30.383 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:30 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1[53304]: 2024-04-22T15:55:30.070+0000 7f29d32fe700 -1 osd.1 54 *** Got signal Terminated *** 2024-04-22T15:55:30.383 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:30 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1[53304]: 2024-04-22T15:55:30.070+0000 7f29d32fe700 -1 osd.1 54 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T15:55:30.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:30 smithi016 ceph-mon[110740]: osd.1 marked itself down and dead 2024-04-22T15:55:30.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:55:30.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:30 smithi027 ceph-mon[83054]: osd.1 marked itself down and dead 2024-04-22T15:55:30.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:30 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:55:31.066 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:30 smithi016 podman[127462]: 2024-04-22 15:55:30.796249337 +0000 UTC m=+0.834471271 container died 6aa2e6796784b581091d885ae90ff7f00aaed6f20df21cb4455549772e9ded7a (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240222, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-22T15:55:31.384 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:31 smithi016 podman[127462]: 2024-04-22 15:55:31.153817116 +0000 UTC m=+1.192039036 container cleanup 6aa2e6796784b581091d885ae90ff7f00aaed6f20df21cb4455549772e9ded7a (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, io.buildah.version=1.29.1) 2024-04-22T15:55:31.384 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:31 smithi016 bash[127462]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1 2024-04-22T15:55:31.666 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:31 smithi016 podman[127480]: 2024-04-22 15:55:31.455281808 +0000 UTC m=+0.653361083 container remove 6aa2e6796784b581091d885ae90ff7f00aaed6f20df21cb4455549772e9ded7a (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222) 2024-04-22T15:55:31.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:31 smithi016 ceph-mon[110740]: pgmap v68: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:55:31.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:31 smithi016 ceph-mon[110740]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T15:55:31.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:31 smithi016 ceph-mon[110740]: osdmap e55: 6 total, 5 up, 6 in 2024-04-22T15:55:31.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:31 smithi027 ceph-mon[83054]: pgmap v68: 65 pgs: 65 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:55:31.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:31 smithi027 ceph-mon[83054]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T15:55:31.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:31 smithi027 ceph-mon[83054]: osdmap e55: 6 total, 5 up, 6 in 2024-04-22T15:55:32.061 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:31 smithi016 podman[127570]: 2024-04-22 15:55:31.765033765 +0000 UTC m=+0.099329603 container create b27dc426a6b88ac6ddd3fb850a9e92b000db8dfe2455e96405f40779775c43e9 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T15:55:32.061 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:31 smithi016 podman[127570]: 2024-04-22 15:55:31.685286993 +0000 UTC m=+0.019582801 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:55:32.061 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:31 smithi016 podman[127570]: 2024-04-22 15:55:31.975932184 +0000 UTC m=+0.310227980 container init b27dc426a6b88ac6ddd3fb850a9e92b000db8dfe2455e96405f40779775c43e9 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-22T15:55:32.061 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:31 smithi016 podman[127570]: 2024-04-22 15:55:31.979294836 +0000 UTC m=+0.313590641 container start b27dc426a6b88ac6ddd3fb850a9e92b000db8dfe2455e96405f40779775c43e9 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:55:32.061 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:32 smithi016 podman[127570]: 2024-04-22 15:55:32.059121634 +0000 UTC m=+0.393417451 container attach b27dc426a6b88ac6ddd3fb850a9e92b000db8dfe2455e96405f40779775c43e9 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T15:55:32.392 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:32 smithi016 podman[127570]: 2024-04-22 15:55:32.293844883 +0000 UTC m=+0.628140690 container died b27dc426a6b88ac6ddd3fb850a9e92b000db8dfe2455e96405f40779775c43e9 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49) 2024-04-22T15:55:32.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:32 smithi016 ceph-mon[110740]: osdmap e56: 6 total, 5 up, 6 in 2024-04-22T15:55:32.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:32 smithi027 ceph-mon[83054]: osdmap e56: 6 total, 5 up, 6 in 2024-04-22T15:55:33.094 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:32 smithi016 podman[127595]: 2024-04-22 15:55:32.854477037 +0000 UTC m=+0.555545733 container remove b27dc426a6b88ac6ddd3fb850a9e92b000db8dfe2455e96405f40779775c43e9 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.schema-version=1.0) 2024-04-22T15:55:33.094 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:32 smithi016 systemd[1]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.1.service: Deactivated successfully. 2024-04-22T15:55:33.094 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:32 smithi016 systemd[1]: Stopped Ceph osd.1 for 67ab3a80-00be-11ef-bc93-c7b262605968. 2024-04-22T15:55:33.094 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:32 smithi016 systemd[1]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.1.service: Consumed 58.519s CPU time. 2024-04-22T15:55:33.394 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:33 smithi016 systemd[1]: Starting Ceph osd.1 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T15:55:33.394 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:33 smithi016 podman[127697]: 2024-04-22 15:55:33.305959026 +0000 UTC m=+0.110157547 container create ce82492a98ad01bd500948ab1006df45a8d5362d69ccdb02bfd97bc77dfef59c (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-activate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-22T15:55:33.394 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:33 smithi016 podman[127697]: 2024-04-22 15:55:33.214579285 +0000 UTC m=+0.018777782 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:55:33.652 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:33 smithi016 ceph-mon[110740]: pgmap v71: 65 pgs: 29 peering, 3 stale+active+clean, 33 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-04-22T15:55:33.652 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:33 smithi016 ceph-mon[110740]: Health check failed: Reduced data availability: 9 pgs peering (PG_AVAILABILITY) 2024-04-22T15:55:33.652 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:33 smithi016 podman[127697]: 2024-04-22 15:55:33.551123022 +0000 UTC m=+0.355321514 container init ce82492a98ad01bd500948ab1006df45a8d5362d69ccdb02bfd97bc77dfef59c (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-22T15:55:33.652 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:33 smithi016 podman[127697]: 2024-04-22 15:55:33.55498713 +0000 UTC m=+0.359185609 container start ce82492a98ad01bd500948ab1006df45a8d5362d69ccdb02bfd97bc77dfef59c (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-activate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-22T15:55:33.652 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:33 smithi016 podman[127697]: 2024-04-22 15:55:33.650419323 +0000 UTC m=+0.454617810 container attach ce82492a98ad01bd500948ab1006df45a8d5362d69ccdb02bfd97bc77dfef59c (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49) 2024-04-22T15:55:33.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:33 smithi027 ceph-mon[83054]: pgmap v71: 65 pgs: 29 peering, 3 stale+active+clean, 33 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-04-22T15:55:33.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:33 smithi027 ceph-mon[83054]: Health check failed: Reduced data availability: 9 pgs peering (PG_AVAILABILITY) 2024-04-22T15:55:34.365 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:34 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-activate[127711]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:55:34.365 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:34 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-activate[127711]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:55:34.365 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:34 smithi016 bash[127697]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:55:34.365 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:34 smithi016 bash[127697]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:55:34.365 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:34 smithi016 bash[127697]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-22T15:55:34.365 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:34 smithi016 bash[127697]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-22T15:55:34.365 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:34 smithi016 bash[127697]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-22T15:55:34.365 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:34 smithi016 bash[127697]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-22T15:55:34.365 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:34 smithi016 bash[127697]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-22T15:55:34.366 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:34 smithi016 bash[127697]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-22T15:55:34.366 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:34 smithi016 bash[127697]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-22T15:55:34.366 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:34 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-activate[127711]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-22T15:55:34.366 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:34 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-activate[127711]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-22T15:55:34.366 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:34 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-activate[127711]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-22T15:55:34.366 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:34 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-activate[127711]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-22T15:55:34.366 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:34 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-activate[127711]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-22T15:55:34.366 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:34 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-activate[127711]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-22T15:55:34.366 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:34 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-activate[127711]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-22T15:55:34.366 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:34 smithi016 podman[127697]: 2024-04-22 15:55:34.076533485 +0000 UTC m=+0.880731962 container died ce82492a98ad01bd500948ab1006df45a8d5362d69ccdb02bfd97bc77dfef59c (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:55:34.916 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:34 smithi016 podman[127857]: 2024-04-22 15:55:34.638089471 +0000 UTC m=+0.554353494 container remove ce82492a98ad01bd500948ab1006df45a8d5362d69ccdb02bfd97bc77dfef59c (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-22T15:55:34.917 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:34 smithi016 podman[127922]: 2024-04-22 15:55:34.758373229 +0000 UTC m=+0.018907674 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:55:34.917 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:34 smithi016 podman[127922]: 2024-04-22 15:55:34.864026927 +0000 UTC m=+0.124561352 container create ec1a45b3c53ece2baa204840f17d0fdb536387d4f285cdc7bc7dbb3560775f9f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-22T15:55:35.397 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:35 smithi016 podman[127922]: 2024-04-22 15:55:35.058504134 +0000 UTC m=+0.319038562 container init ec1a45b3c53ece2baa204840f17d0fdb536387d4f285cdc7bc7dbb3560775f9f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:55:35.397 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:35 smithi016 podman[127922]: 2024-04-22 15:55:35.061544286 +0000 UTC m=+0.322078720 container start ec1a45b3c53ece2baa204840f17d0fdb536387d4f285cdc7bc7dbb3560775f9f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-22T15:55:35.397 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:35 smithi016 ceph-osd[127941]: -- 172.21.15.16:0/1880425164 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55beb4b91680 con 0x55beb4b5ac00 2024-04-22T15:55:35.397 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:35 smithi016 bash[127922]: ec1a45b3c53ece2baa204840f17d0fdb536387d4f285cdc7bc7dbb3560775f9f 2024-04-22T15:55:35.397 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:35 smithi016 systemd[1]: Started Ceph osd.1 for 67ab3a80-00be-11ef-bc93-c7b262605968. 2024-04-22T15:55:35.666 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:35 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1[127937]: 2024-04-22T15:55:35.601+0000 7ff0164047c0 -1 Falling back to public interface 2024-04-22T15:55:35.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:35 smithi016 ceph-mon[110740]: pgmap v72: 65 pgs: 29 peering, 3 stale+active+clean, 33 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-04-22T15:55:35.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:35 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:35.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:35 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:35.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:35 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:55:35.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:35 smithi027 ceph-mon[83054]: pgmap v72: 65 pgs: 29 peering, 3 stale+active+clean, 33 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-04-22T15:55:35.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:35 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:35.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:35 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:35.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:35 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:55:36.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:36 smithi027 ceph-mon[83054]: Health check failed: Degraded data redundancy: 34/258 objects degraded (13.178%), 13 pgs degraded (PG_DEGRADED) 2024-04-22T15:55:36.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:36 smithi027 ceph-mon[83054]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs peering) 2024-04-22T15:55:36.851 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:36 smithi016 ceph-mon[110740]: Health check failed: Degraded data redundancy: 34/258 objects degraded (13.178%), 13 pgs degraded (PG_DEGRADED) 2024-04-22T15:55:36.851 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:36 smithi016 ceph-mon[110740]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs peering) 2024-04-22T15:55:37.703 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:37 smithi016 ceph-mon[110740]: pgmap v73: 65 pgs: 13 active+undersized+degraded, 21 active+undersized, 31 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 34/258 objects degraded (13.178%) 2024-04-22T15:55:37.737 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:37 smithi027 ceph-mon[83054]: pgmap v73: 65 pgs: 13 active+undersized+degraded, 21 active+undersized, 31 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 34/258 objects degraded (13.178%) 2024-04-22T15:55:39.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:39 smithi027 ceph-mon[83054]: pgmap v74: 65 pgs: 13 active+undersized+degraded, 21 active+undersized, 31 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 34/258 objects degraded (13.178%) 2024-04-22T15:55:39.768 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:39 smithi016 ceph-mon[110740]: pgmap v74: 65 pgs: 13 active+undersized+degraded, 21 active+undersized, 31 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 34/258 objects degraded (13.178%) 2024-04-22T15:55:40.416 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:40 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1[127937]: 2024-04-22T15:55:40.103+0000 7ff0164047c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-04-22T15:55:40.916 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:40 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1[127937]: 2024-04-22T15:55:40.517+0000 7ff0164047c0 -1 osd.1 54 log_to_monitors true 2024-04-22T15:55:41.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:41 smithi027 ceph-mon[83054]: pgmap v75: 65 pgs: 13 active+undersized+degraded, 21 active+undersized, 31 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 34/258 objects degraded (13.178%) 2024-04-22T15:55:41.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:41 smithi027 ceph-mon[83054]: from='osd.1 [v2:172.21.15.16:6810/3524132843,v1:172.21.15.16:6811/3524132843]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T15:55:41.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:41 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:41.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:41 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:41.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:41 smithi016 ceph-mon[110740]: pgmap v75: 65 pgs: 13 active+undersized+degraded, 21 active+undersized, 31 active+clean; 250 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 34/258 objects degraded (13.178%) 2024-04-22T15:55:41.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:41 smithi016 ceph-mon[110740]: from='osd.1 [v2:172.21.15.16:6810/3524132843,v1:172.21.15.16:6811/3524132843]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T15:55:41.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:41 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:41.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:41 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:42.560 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 15:55:42 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1[127937]: 2024-04-22T15:55:42.250+0000 7ff003854700 -1 osd.1 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T15:55:42.560 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:42 smithi016 ceph-mon[110740]: from='osd.1 [v2:172.21.15.16:6810/3524132843,v1:172.21.15.16:6811/3524132843]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T15:55:42.560 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:42 smithi016 ceph-mon[110740]: osdmap e57: 6 total, 5 up, 6 in 2024-04-22T15:55:42.561 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:42 smithi016 ceph-mon[110740]: from='osd.1 [v2:172.21.15.16:6810/3524132843,v1:172.21.15.16:6811/3524132843]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-22T15:55:42.561 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:42 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:42.561 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:42 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:42.561 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:42 smithi016 ceph-mon[110740]: from='osd.1 [v2:172.21.15.16:6810/3524132843,v1:172.21.15.16:6811/3524132843]' entity='osd.1' 2024-04-22T15:55:42.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:42 smithi027 ceph-mon[83054]: from='osd.1 [v2:172.21.15.16:6810/3524132843,v1:172.21.15.16:6811/3524132843]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T15:55:42.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:42 smithi027 ceph-mon[83054]: osdmap e57: 6 total, 5 up, 6 in 2024-04-22T15:55:42.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:42 smithi027 ceph-mon[83054]: from='osd.1 [v2:172.21.15.16:6810/3524132843,v1:172.21.15.16:6811/3524132843]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-22T15:55:42.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:42 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:42.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:42 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:42.838 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:42 smithi027 ceph-mon[83054]: from='osd.1 [v2:172.21.15.16:6810/3524132843,v1:172.21.15.16:6811/3524132843]' entity='osd.1' 2024-04-22T15:55:43.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:43 smithi016 ceph-mon[110740]: pgmap v77: 65 pgs: 13 active+undersized+degraded, 21 active+undersized, 31 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 34/258 objects degraded (13.178%) 2024-04-22T15:55:43.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:43 smithi016 ceph-mon[110740]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T15:55:43.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:43 smithi016 ceph-mon[110740]: osd.1 [v2:172.21.15.16:6810/3524132843,v1:172.21.15.16:6811/3524132843] boot 2024-04-22T15:55:43.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:43 smithi016 ceph-mon[110740]: osdmap e58: 6 total, 6 up, 6 in 2024-04-22T15:55:43.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:43 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:55:44.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:43 smithi027 ceph-mon[83054]: pgmap v77: 65 pgs: 13 active+undersized+degraded, 21 active+undersized, 31 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 34/258 objects degraded (13.178%) 2024-04-22T15:55:44.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:43 smithi027 ceph-mon[83054]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T15:55:44.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:43 smithi027 ceph-mon[83054]: osd.1 [v2:172.21.15.16:6810/3524132843,v1:172.21.15.16:6811/3524132843] boot 2024-04-22T15:55:44.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:43 smithi027 ceph-mon[83054]: osdmap e58: 6 total, 6 up, 6 in 2024-04-22T15:55:44.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:43 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T15:55:45.576 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:45 smithi016 ceph-mon[110740]: pgmap v79: 65 pgs: 13 active+undersized+degraded, 21 active+undersized, 31 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 639 B/s rd, 1 op/s; 34/258 objects degraded (13.178%) 2024-04-22T15:55:45.576 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:45 smithi016 ceph-mon[110740]: osdmap e59: 6 total, 6 up, 6 in 2024-04-22T15:55:45.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:45 smithi027 ceph-mon[83054]: pgmap v79: 65 pgs: 13 active+undersized+degraded, 21 active+undersized, 31 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 639 B/s rd, 1 op/s; 34/258 objects degraded (13.178%) 2024-04-22T15:55:45.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:45 smithi027 ceph-mon[83054]: osdmap e59: 6 total, 6 up, 6 in 2024-04-22T15:55:46.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:46.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:55:46.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:46.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:46.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:55:46.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:55:46.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:46.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:55:46.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:55:46.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:55:46.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:55:46.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-22T15:55:46.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:46 smithi027 ceph-mon[83054]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-22T15:55:46.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:46 smithi027 ceph-mon[83054]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-22T15:55:46.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:46.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:55:46.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:46.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:46.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:55:46.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:55:46.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:46.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:55:46.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:55:46.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:55:46.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:55:46.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-22T15:55:46.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:46 smithi016 ceph-mon[110740]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-22T15:55:46.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:46 smithi016 ceph-mon[110740]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-22T15:55:47.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:47 smithi027 ceph-mon[83054]: pgmap v81: 65 pgs: 7 active+undersized+degraded, 5 active+undersized, 53 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 18/258 objects degraded (6.977%) 2024-04-22T15:55:47.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:47 smithi027 ceph-mon[83054]: Health check update: Degraded data redundancy: 18/258 objects degraded (6.977%), 7 pgs degraded (PG_DEGRADED) 2024-04-22T15:55:47.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:47 smithi016 ceph-mon[110740]: pgmap v81: 65 pgs: 7 active+undersized+degraded, 5 active+undersized, 53 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 18/258 objects degraded (6.977%) 2024-04-22T15:55:47.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:47 smithi016 ceph-mon[110740]: Health check update: Degraded data redundancy: 18/258 objects degraded (6.977%), 7 pgs degraded (PG_DEGRADED) 2024-04-22T15:55:48.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:48 smithi027 ceph-mon[83054]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 18/258 objects degraded (6.977%), 7 pgs degraded) 2024-04-22T15:55:48.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:48 smithi016 ceph-mon[110740]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 18/258 objects degraded (6.977%), 7 pgs degraded) 2024-04-22T15:55:49.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:49 smithi027 ceph-mon[83054]: pgmap v82: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 756 B/s rd, 1 op/s 2024-04-22T15:55:49.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:49 smithi016 ceph-mon[110740]: pgmap v82: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 756 B/s rd, 1 op/s 2024-04-22T15:55:51.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:51 smithi027 ceph-mon[83054]: pgmap v83: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 639 B/s rd, 1 op/s 2024-04-22T15:55:51.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:51 smithi016 ceph-mon[110740]: pgmap v83: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 639 B/s rd, 1 op/s 2024-04-22T15:55:53.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:53 smithi027 ceph-mon[83054]: pgmap v84: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-22T15:55:53.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:53 smithi016 ceph-mon[110740]: pgmap v84: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-22T15:55:55.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:55 smithi027 ceph-mon[83054]: pgmap v85: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2024-04-22T15:55:55.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:55 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:55.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:55 smithi016 ceph-mon[110740]: pgmap v85: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2024-04-22T15:55:55.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:55 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:55:57.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:57 smithi027 ceph-mon[83054]: pgmap v86: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s 2024-04-22T15:55:57.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:57 smithi016 ceph-mon[110740]: pgmap v86: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s 2024-04-22T15:55:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.258+0000 7f538dd34700 1 -- 172.21.15.16:0/4052325469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5388101850 msgr2=0x7f53881058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.258+0000 7f538dd34700 1 --2- 172.21.15.16:0/4052325469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5388101850 0x7f53881058a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5370009910 tx=0x7f5370009c20 comp rx=0 tx=0).stop 2024-04-22T15:55:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.259+0000 7f538dd34700 1 -- 172.21.15.16:0/4052325469 shutdown_connections 2024-04-22T15:55:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.259+0000 7f538dd34700 1 --2- 172.21.15.16:0/4052325469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5388101850 0x7f53881058a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.259+0000 7f538dd34700 1 --2- 172.21.15.16:0/4052325469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5388100ea0 0x7f5388101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.259+0000 7f538dd34700 1 -- 172.21.15.16:0/4052325469 >> 172.21.15.16:0/4052325469 conn(0x7f53880fc750 msgr2=0x7f53880feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.259+0000 7f538dd34700 1 -- 172.21.15.16:0/4052325469 shutdown_connections 2024-04-22T15:55:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.259+0000 7f538dd34700 1 -- 172.21.15.16:0/4052325469 wait complete. 2024-04-22T15:55:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.259+0000 7f538dd34700 1 Processor -- start 2024-04-22T15:55:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.259+0000 7f538dd34700 1 -- start start 2024-04-22T15:55:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.260+0000 7f538dd34700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5388100ea0 0x7f5388073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.260+0000 7f538dd34700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5388101850 0x7f53880735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.260+0000 7f538dd34700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5388073b10 con 0x7f5388100ea0 2024-04-22T15:55:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.260+0000 7f538dd34700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5388073c50 con 0x7f5388101850 2024-04-22T15:55:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.260+0000 7f53877fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5388100ea0 0x7f5388073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.260+0000 7f53877fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5388100ea0 0x7f5388073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47304/0 (socket says 172.21.15.16:47304) 2024-04-22T15:55:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.260+0000 7f53877fe700 1 -- 172.21.15.16:0/1236135109 learned_addr learned my addr 172.21.15.16:0/1236135109 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:55:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.260+0000 7f5386ffd700 1 --2- 172.21.15.16:0/1236135109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5388101850 0x7f53880735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.260+0000 7f53877fe700 1 -- 172.21.15.16:0/1236135109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5388101850 msgr2=0x7f53880735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.260+0000 7f53877fe700 1 --2- 172.21.15.16:0/1236135109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5388101850 0x7f53880735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.260+0000 7f53877fe700 1 -- 172.21.15.16:0/1236135109 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53700095f0 con 0x7f5388100ea0 2024-04-22T15:55:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.261+0000 7f53877fe700 1 --2- 172.21.15.16:0/1236135109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5388100ea0 0x7f5388073090 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f537800ef10 tx=0x7f537800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.261+0000 7f5384ff9700 1 -- 172.21.15.16:0/1236135109 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f537800c9f0 con 0x7f5388100ea0 2024-04-22T15:55:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.261+0000 7f5384ff9700 1 -- 172.21.15.16:0/1236135109 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f537800f040 con 0x7f5388100ea0 2024-04-22T15:55:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.261+0000 7f5384ff9700 1 -- 172.21.15.16:0/1236135109 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53780146b0 con 0x7f5388100ea0 2024-04-22T15:55:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.261+0000 7f538dd34700 1 -- 172.21.15.16:0/1236135109 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5388073ed0 con 0x7f5388100ea0 2024-04-22T15:55:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.261+0000 7f538dd34700 1 -- 172.21.15.16:0/1236135109 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53881a2e90 con 0x7f5388100ea0 2024-04-22T15:55:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.262+0000 7f538dd34700 1 -- 172.21.15.16:0/1236135109 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53881092a0 con 0x7f5388100ea0 2024-04-22T15:55:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.262+0000 7f5384ff9700 1 -- 172.21.15.16:0/1236135109 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f537800cb50 con 0x7f5388100ea0 2024-04-22T15:55:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.263+0000 7f5384ff9700 1 --2- 172.21.15.16:0/1236135109 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f537406ecc0 0x7f5374071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.263+0000 7f5384ff9700 1 -- 172.21.15.16:0/1236135109 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f537808f0e0 con 0x7f5388100ea0 2024-04-22T15:55:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.263+0000 7f5386ffd700 1 --2- 172.21.15.16:0/1236135109 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f537406ecc0 0x7f5374071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.263+0000 7f5386ffd700 1 --2- 172.21.15.16:0/1236135109 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f537406ecc0 0x7f5374071180 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5370009fd0 tx=0x7f5370005a90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:58.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.268+0000 7f5384ff9700 1 -- 172.21.15.16:0/1236135109 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5378057f40 con 0x7f5388100ea0 2024-04-22T15:55:58.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.498+0000 7f538dd34700 1 -- 172.21.15.16:0/1236135109 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f53880611d0 con 0x7f537406ecc0 2024-04-22T15:55:58.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.499+0000 7f5384ff9700 1 -- 172.21.15.16:0/1236135109 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f53880611d0 con 0x7f537406ecc0 2024-04-22T15:55:58.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.501+0000 7f538dd34700 1 -- 172.21.15.16:0/1236135109 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f537406ecc0 msgr2=0x7f5374071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:58.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.501+0000 7f538dd34700 1 --2- 172.21.15.16:0/1236135109 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f537406ecc0 0x7f5374071180 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5370009fd0 tx=0x7f5370005a90 comp rx=0 tx=0).stop 2024-04-22T15:55:58.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.501+0000 7f538dd34700 1 -- 172.21.15.16:0/1236135109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5388100ea0 msgr2=0x7f5388073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:58.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.501+0000 7f538dd34700 1 --2- 172.21.15.16:0/1236135109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5388100ea0 0x7f5388073090 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f537800ef10 tx=0x7f537800c3a0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.501+0000 7f538dd34700 1 -- 172.21.15.16:0/1236135109 shutdown_connections 2024-04-22T15:55:58.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.501+0000 7f538dd34700 1 --2- 172.21.15.16:0/1236135109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5388100ea0 0x7f5388073090 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.502+0000 7f538dd34700 1 --2- 172.21.15.16:0/1236135109 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f537406ecc0 0x7f5374071180 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.502+0000 7f538dd34700 1 --2- 172.21.15.16:0/1236135109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5388101850 0x7f53880735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.502+0000 7f538dd34700 1 -- 172.21.15.16:0/1236135109 >> 172.21.15.16:0/1236135109 conn(0x7f53880fc750 msgr2=0x7f53880fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:58.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.502+0000 7f538dd34700 1 -- 172.21.15.16:0/1236135109 shutdown_connections 2024-04-22T15:55:58.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.502+0000 7f538dd34700 1 -- 172.21.15.16:0/1236135109 wait complete. 2024-04-22T15:55:58.516 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:55:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.611+0000 7fc8b3e68700 1 -- 172.21.15.16:0/232882272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8ac101e20 msgr2=0x7fc8ac102200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.611+0000 7fc8b3e68700 1 --2- 172.21.15.16:0/232882272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8ac101e20 0x7fc8ac102200 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc8a8009910 tx=0x7fc8a8009c20 comp rx=0 tx=0).stop 2024-04-22T15:55:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.611+0000 7fc8b3e68700 1 -- 172.21.15.16:0/232882272 shutdown_connections 2024-04-22T15:55:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.611+0000 7fc8b3e68700 1 --2- 172.21.15.16:0/232882272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8ac102740 0x7fc8ac10ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.611+0000 7fc8b3e68700 1 --2- 172.21.15.16:0/232882272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8ac101e20 0x7fc8ac102200 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.611+0000 7fc8b3e68700 1 -- 172.21.15.16:0/232882272 >> 172.21.15.16:0/232882272 conn(0x7fc8ac0fb650 msgr2=0x7fc8ac0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.611+0000 7fc8b3e68700 1 -- 172.21.15.16:0/232882272 shutdown_connections 2024-04-22T15:55:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.611+0000 7fc8b3e68700 1 -- 172.21.15.16:0/232882272 wait complete. 2024-04-22T15:55:58.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.611+0000 7fc8b3e68700 1 Processor -- start 2024-04-22T15:55:58.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.612+0000 7fc8b3e68700 1 -- start start 2024-04-22T15:55:58.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.612+0000 7fc8b3e68700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8ac101e20 0x7fc8ac19c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:58.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.612+0000 7fc8b3e68700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8ac102740 0x7fc8ac19cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:58.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.612+0000 7fc8b3e68700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ac19d260 con 0x7fc8ac101e20 2024-04-22T15:55:58.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.612+0000 7fc8b3e68700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ac196760 con 0x7fc8ac102740 2024-04-22T15:55:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.612+0000 7fc8b1c04700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8ac101e20 0x7fc8ac19c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.612+0000 7fc8b1c04700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8ac101e20 0x7fc8ac19c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47312/0 (socket says 172.21.15.16:47312) 2024-04-22T15:55:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.612+0000 7fc8b1c04700 1 -- 172.21.15.16:0/3461782617 learned_addr learned my addr 172.21.15.16:0/3461782617 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:55:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.612+0000 7fc8b1403700 1 --2- 172.21.15.16:0/3461782617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8ac102740 0x7fc8ac19cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.612+0000 7fc8b1c04700 1 -- 172.21.15.16:0/3461782617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8ac102740 msgr2=0x7fc8ac19cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.612+0000 7fc8b1c04700 1 --2- 172.21.15.16:0/3461782617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8ac102740 0x7fc8ac19cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.612+0000 7fc8b1c04700 1 -- 172.21.15.16:0/3461782617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a80095f0 con 0x7fc8ac101e20 2024-04-22T15:55:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.612+0000 7fc8b1c04700 1 --2- 172.21.15.16:0/3461782617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8ac101e20 0x7fc8ac19c690 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc8a80092e0 tx=0x7fc8a800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.613+0000 7fc8a2ffd700 1 -- 172.21.15.16:0/3461782617 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8a801d070 con 0x7fc8ac101e20 2024-04-22T15:55:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.613+0000 7fc8a2ffd700 1 -- 172.21.15.16:0/3461782617 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc8a8005e00 con 0x7fc8ac101e20 2024-04-22T15:55:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.613+0000 7fc8b3e68700 1 -- 172.21.15.16:0/3461782617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8ac1969e0 con 0x7fc8ac101e20 2024-04-22T15:55:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.613+0000 7fc8a2ffd700 1 -- 172.21.15.16:0/3461782617 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8a8017400 con 0x7fc8ac101e20 2024-04-22T15:55:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.613+0000 7fc8b3e68700 1 -- 172.21.15.16:0/3461782617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8ac196ed0 con 0x7fc8ac101e20 2024-04-22T15:55:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.614+0000 7fc8b3e68700 1 -- 172.21.15.16:0/3461782617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8ac108330 con 0x7fc8ac101e20 2024-04-22T15:55:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.614+0000 7fc8a2ffd700 1 -- 172.21.15.16:0/3461782617 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc8a800f460 con 0x7fc8ac101e20 2024-04-22T15:55:58.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.615+0000 7fc8a2ffd700 1 --2- 172.21.15.16:0/3461782617 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fc89806ecc0 0x7fc898071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:58.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.615+0000 7fc8a2ffd700 1 -- 172.21.15.16:0/3461782617 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc8a80901e0 con 0x7fc8ac101e20 2024-04-22T15:55:58.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.615+0000 7fc8b1403700 1 --2- 172.21.15.16:0/3461782617 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fc89806ecc0 0x7fc898071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:58.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.615+0000 7fc8b1403700 1 --2- 172.21.15.16:0/3461782617 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fc89806ecc0 0x7fc898071180 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc89c005950 tx=0x7fc89c0058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:58.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.620+0000 7fc8a2ffd700 1 -- 172.21.15.16:0/3461782617 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc8a805cbf0 con 0x7fc8ac101e20 2024-04-22T15:55:58.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.852+0000 7fc8b3e68700 1 -- 172.21.15.16:0/3461782617 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8ac197d30 con 0x7fc89806ecc0 2024-04-22T15:55:58.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.853+0000 7fc8a2ffd700 1 -- 172.21.15.16:0/3461782617 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fc8ac197d30 con 0x7fc89806ecc0 2024-04-22T15:55:58.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.856+0000 7fc8b3e68700 1 -- 172.21.15.16:0/3461782617 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fc89806ecc0 msgr2=0x7fc898071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:58.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.856+0000 7fc8b3e68700 1 --2- 172.21.15.16:0/3461782617 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fc89806ecc0 0x7fc898071180 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc89c005950 tx=0x7fc89c0058e0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.856+0000 7fc8b3e68700 1 -- 172.21.15.16:0/3461782617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8ac101e20 msgr2=0x7fc8ac19c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:58.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.856+0000 7fc8b3e68700 1 --2- 172.21.15.16:0/3461782617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8ac101e20 0x7fc8ac19c690 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc8a80092e0 tx=0x7fc8a800be20 comp rx=0 tx=0).stop 2024-04-22T15:55:58.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.856+0000 7fc8b3e68700 1 -- 172.21.15.16:0/3461782617 shutdown_connections 2024-04-22T15:55:58.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.856+0000 7fc8b3e68700 1 --2- 172.21.15.16:0/3461782617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8ac101e20 0x7fc8ac19c690 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.856+0000 7fc8b3e68700 1 --2- 172.21.15.16:0/3461782617 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fc89806ecc0 0x7fc898071180 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.856+0000 7fc8b3e68700 1 --2- 172.21.15.16:0/3461782617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8ac102740 0x7fc8ac19cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.856+0000 7fc8b3e68700 1 -- 172.21.15.16:0/3461782617 >> 172.21.15.16:0/3461782617 conn(0x7fc8ac0fb650 msgr2=0x7fc8ac105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:58.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.856+0000 7fc8b3e68700 1 -- 172.21.15.16:0/3461782617 shutdown_connections 2024-04-22T15:55:58.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.856+0000 7fc8b3e68700 1 -- 172.21.15.16:0/3461782617 wait complete. 2024-04-22T15:55:58.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.964+0000 7f190fcd2700 1 -- 172.21.15.16:0/3245265313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1908101bb0 msgr2=0x7f1908101f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:58.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.964+0000 7f190fcd2700 1 --2- 172.21.15.16:0/3245265313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1908101bb0 0x7f1908101f90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f18fc009910 tx=0x7f18fc009c20 comp rx=0 tx=0).stop 2024-04-22T15:55:58.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.964+0000 7f190fcd2700 1 -- 172.21.15.16:0/3245265313 shutdown_connections 2024-04-22T15:55:58.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.964+0000 7f190fcd2700 1 --2- 172.21.15.16:0/3245265313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19080ff340 0x7f19080ff7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.964+0000 7f190fcd2700 1 --2- 172.21.15.16:0/3245265313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1908101bb0 0x7f1908101f90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.964+0000 7f190fcd2700 1 -- 172.21.15.16:0/3245265313 >> 172.21.15.16:0/3245265313 conn(0x7f1908074e30 msgr2=0x7f1908075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:58.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.964+0000 7f190fcd2700 1 -- 172.21.15.16:0/3245265313 shutdown_connections 2024-04-22T15:55:58.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.964+0000 7f190fcd2700 1 -- 172.21.15.16:0/3245265313 wait complete. 2024-04-22T15:55:58.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.964+0000 7f190fcd2700 1 Processor -- start 2024-04-22T15:55:58.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.964+0000 7f190fcd2700 1 -- start start 2024-04-22T15:55:58.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.965+0000 7f190fcd2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19080ff340 0x7f190819e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:58.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.965+0000 7f190fcd2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1908101bb0 0x7f190819ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:58.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.965+0000 7f190fcd2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190819f410 con 0x7f1908101bb0 2024-04-22T15:55:58.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.965+0000 7f190fcd2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19081988c0 con 0x7f19080ff340 2024-04-22T15:55:58.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.965+0000 7f190d26d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1908101bb0 0x7f190819ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:58.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.965+0000 7f190da6e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19080ff340 0x7f190819e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:58.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.965+0000 7f190d26d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1908101bb0 0x7f190819ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47334/0 (socket says 172.21.15.16:47334) 2024-04-22T15:55:58.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.965+0000 7f190d26d700 1 -- 172.21.15.16:0/2514519511 learned_addr learned my addr 172.21.15.16:0/2514519511 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:55:58.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.965+0000 7f190d26d700 1 -- 172.21.15.16:0/2514519511 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19080ff340 msgr2=0x7f190819e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:58.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.965+0000 7f190d26d700 1 --2- 172.21.15.16:0/2514519511 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19080ff340 0x7f190819e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:58.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.965+0000 7f190d26d700 1 -- 172.21.15.16:0/2514519511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18fc0095f0 con 0x7f1908101bb0 2024-04-22T15:55:58.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.965+0000 7f190da6e700 1 --2- 172.21.15.16:0/2514519511 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19080ff340 0x7f190819e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:55:58.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.965+0000 7f190d26d700 1 --2- 172.21.15.16:0/2514519511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1908101bb0 0x7f190819ed80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f190400d8d0 tx=0x7f190400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:58.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.965+0000 7f18faffd700 1 -- 172.21.15.16:0/2514519511 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19040098a0 con 0x7f1908101bb0 2024-04-22T15:55:58.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.965+0000 7f190fcd2700 1 -- 172.21.15.16:0/2514519511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1908198ba0 con 0x7f1908101bb0 2024-04-22T15:55:58.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.965+0000 7f18faffd700 1 -- 172.21.15.16:0/2514519511 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1904010460 con 0x7f1908101bb0 2024-04-22T15:55:58.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.965+0000 7f18faffd700 1 -- 172.21.15.16:0/2514519511 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f190400f5f0 con 0x7f1908101bb0 2024-04-22T15:55:58.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.966+0000 7f190fcd2700 1 -- 172.21.15.16:0/2514519511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19081990f0 con 0x7f1908101bb0 2024-04-22T15:55:58.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.966+0000 7f190fcd2700 1 -- 172.21.15.16:0/2514519511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f190804f2e0 con 0x7f1908101bb0 2024-04-22T15:55:58.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.967+0000 7f18faffd700 1 -- 172.21.15.16:0/2514519511 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1904009a00 con 0x7f1908101bb0 2024-04-22T15:55:58.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.967+0000 7f18faffd700 1 --2- 172.21.15.16:0/2514519511 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f18f406ecc0 0x7f18f4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:58.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.968+0000 7f18faffd700 1 -- 172.21.15.16:0/2514519511 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f190408ec80 con 0x7f1908101bb0 2024-04-22T15:55:58.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.968+0000 7f190da6e700 1 --2- 172.21.15.16:0/2514519511 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f18f406ecc0 0x7f18f4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:58.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.968+0000 7f190da6e700 1 --2- 172.21.15.16:0/2514519511 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f18f406ecc0 0x7f18f4071180 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f18fc006010 tx=0x7f18fc01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:58.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:58.972+0000 7f18faffd700 1 -- 172.21.15.16:0/2514519511 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f190405b610 con 0x7f1908101bb0 2024-04-22T15:55:59.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.189+0000 7f190fcd2700 1 -- 172.21.15.16:0/2514519511 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1908199430 con 0x7f18f406ecc0 2024-04-22T15:55:59.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.200+0000 7f18faffd700 1 -- 172.21.15.16:0/2514519511 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f1908199430 con 0x7f18f406ecc0 2024-04-22T15:55:59.203 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:55:59.203 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.smithi016 smithi016 *:9093,9094 running (4m) 18s ago 14m 25.8M - 0.25.0 c8568f914cd2 ea0e0217d6f9 2024-04-22T15:55:59.203 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi016 smithi016 running (15m) 18s ago 15m 8350k - 18.2.1 5be31c24972a f8b627d3f41e 2024-04-22T15:55:59.204 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi027 smithi027 running (14m) 78s ago 14m 8066k - 18.2.1 5be31c24972a f00fd40b4220 2024-04-22T15:55:59.204 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi016 smithi016 running (87s) 18s ago 15m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 1feb008449ed 2024-04-22T15:55:59.204 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi027 smithi027 running (81s) 78s ago 14m 6656k - 19.0.0-3223-g5883d494 3d983c26e278 c7f862c6da00 2024-04-22T15:55:59.204 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.smithi016 smithi016 *:3000 running (3m) 18s ago 14m 82.3M - 9.4.12 2bacad6d85d8 388a3c9c1b3f 2024-04-22T15:55:59.204 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.fkkejm smithi016 running (9m) 18s ago 9m 16.7M - 18.2.1 5be31c24972a 4153ac94cea0 2024-04-22T15:55:59.204 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.pgvphu smithi016 running (9m) 18s ago 9m 92.0M - 18.2.1 5be31c24972a 93dc072068a2 2024-04-22T15:55:59.204 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.dgqyes smithi027 running (9m) 78s ago 9m 19.2M - 18.2.1 5be31c24972a 41a9220430fd 2024-04-22T15:55:59.204 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.iobjds smithi027 running (9m) 78s ago 9m 149M - 18.2.1 5be31c24972a c1c8d6770f02 2024-04-22T15:55:59.204 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi016.rnsnnr smithi016 *:8443,9283,8765 running (6m) 18s ago 16m 504M - 19.0.0-3223-g5883d494 3d983c26e278 09b4fe285bc4 2024-04-22T15:55:59.204 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi027.ukdgzs smithi027 *:8443,9283,8765 running (5m) 78s ago 13m 450M - 19.0.0-3223-g5883d494 3d983c26e278 7548da9b5eb9 2024-04-22T15:55:59.204 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi016 smithi016 running (2m) 18s ago 16m 48.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 4ae529a67806 2024-04-22T15:55:59.204 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi027 smithi027 running (2m) 78s ago 13m 39.0M 2048M 19.0.0-3223-g5883d494 3d983c26e278 f29fcb888801 2024-04-22T15:55:59.204 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi016 smithi016 *:9100 running (5m) 18s ago 15m 15.5M - 1.5.0 0da6a335fe13 35e46e4d2f3f 2024-04-22T15:55:59.205 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi027 smithi027 *:9100 running (5m) 78s ago 13m 15.5M - 1.5.0 0da6a335fe13 e682a7672e9b 2024-04-22T15:55:59.205 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (60s) 18s ago 13m 88.0M 1217M 19.0.0-3223-g5883d494 3d983c26e278 79217eb383a2 2024-04-22T15:55:59.205 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (24s) 18s ago 12m 14.1M 1217M 19.0.0-3223-g5883d494 3d983c26e278 ec1a45b3c53e 2024-04-22T15:55:59.205 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (12m) 18s ago 12m 177M 1217M 18.2.1 5be31c24972a 3d9d310aa9eb 2024-04-22T15:55:59.205 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi027 running (11m) 78s ago 11m 250M 2240M 18.2.1 5be31c24972a cdf9b7aaf865 2024-04-22T15:55:59.205 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (11m) 78s ago 11m 206M 2240M 18.2.1 5be31c24972a 629397d49c76 2024-04-22T15:55:59.205 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi027 running (10m) 78s ago 10m 193M 2240M 18.2.1 5be31c24972a 23adcf675b04 2024-04-22T15:55:59.205 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.smithi016 smithi016 *:9095 running (4m) 18s ago 14m 71.7M - 2.43.0 a07b618ecd1d 373eaadee82d 2024-04-22T15:55:59.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.202+0000 7f190fcd2700 1 -- 172.21.15.16:0/2514519511 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f18f406ecc0 msgr2=0x7f18f4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:59.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.202+0000 7f190fcd2700 1 --2- 172.21.15.16:0/2514519511 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f18f406ecc0 0x7f18f4071180 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f18fc006010 tx=0x7f18fc01a040 comp rx=0 tx=0).stop 2024-04-22T15:55:59.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.202+0000 7f190fcd2700 1 -- 172.21.15.16:0/2514519511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1908101bb0 msgr2=0x7f190819ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:59.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.202+0000 7f190fcd2700 1 --2- 172.21.15.16:0/2514519511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1908101bb0 0x7f190819ed80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f190400d8d0 tx=0x7f190400dbe0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.202+0000 7f190fcd2700 1 -- 172.21.15.16:0/2514519511 shutdown_connections 2024-04-22T15:55:59.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.202+0000 7f190fcd2700 1 --2- 172.21.15.16:0/2514519511 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f18f406ecc0 0x7f18f4071180 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.202+0000 7f190fcd2700 1 --2- 172.21.15.16:0/2514519511 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19080ff340 0x7f190819e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.202+0000 7f190fcd2700 1 --2- 172.21.15.16:0/2514519511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1908101bb0 0x7f190819ed80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.202+0000 7f190fcd2700 1 -- 172.21.15.16:0/2514519511 >> 172.21.15.16:0/2514519511 conn(0x7f1908074e30 msgr2=0x7f19080fd2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:59.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.202+0000 7f190fcd2700 1 -- 172.21.15.16:0/2514519511 shutdown_connections 2024-04-22T15:55:59.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.202+0000 7f190fcd2700 1 -- 172.21.15.16:0/2514519511 wait complete. 2024-04-22T15:55:59.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.308+0000 7fc1c5349700 1 -- 172.21.15.16:0/2538033094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1c0101850 msgr2=0x7fc1c01058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:59.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.308+0000 7fc1c5349700 1 --2- 172.21.15.16:0/2538033094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1c0101850 0x7fc1c01058a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0009910 tx=0x7fc1b0009c20 comp rx=0 tx=0).stop 2024-04-22T15:55:59.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.308+0000 7fc1c5349700 1 -- 172.21.15.16:0/2538033094 shutdown_connections 2024-04-22T15:55:59.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.308+0000 7fc1c5349700 1 --2- 172.21.15.16:0/2538033094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1c0101850 0x7fc1c01058a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.308+0000 7fc1c5349700 1 --2- 172.21.15.16:0/2538033094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1c0100ea0 0x7fc1c0101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.308+0000 7fc1c5349700 1 -- 172.21.15.16:0/2538033094 >> 172.21.15.16:0/2538033094 conn(0x7fc1c00fc750 msgr2=0x7fc1c00feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:59.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.308+0000 7fc1c5349700 1 -- 172.21.15.16:0/2538033094 shutdown_connections 2024-04-22T15:55:59.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.308+0000 7fc1c5349700 1 -- 172.21.15.16:0/2538033094 wait complete. 2024-04-22T15:55:59.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.309+0000 7fc1c5349700 1 Processor -- start 2024-04-22T15:55:59.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.309+0000 7fc1c5349700 1 -- start start 2024-04-22T15:55:59.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.309+0000 7fc1c5349700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1c0100ea0 0x7fc1c019c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:59.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.309+0000 7fc1c5349700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1c0101850 0x7fc1c019cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:59.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.309+0000 7fc1c5349700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c019d2d0 con 0x7fc1c0100ea0 2024-04-22T15:55:59.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.309+0000 7fc1c5349700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c0196780 con 0x7fc1c0101850 2024-04-22T15:55:59.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.309+0000 7fc1beffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1c0100ea0 0x7fc1c019c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:59.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.309+0000 7fc1beffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1c0100ea0 0x7fc1c019c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47360/0 (socket says 172.21.15.16:47360) 2024-04-22T15:55:59.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.309+0000 7fc1beffd700 1 -- 172.21.15.16:0/3101588728 learned_addr learned my addr 172.21.15.16:0/3101588728 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:55:59.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.310+0000 7fc1be7fc700 1 --2- 172.21.15.16:0/3101588728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1c0101850 0x7fc1c019cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:59.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.310+0000 7fc1beffd700 1 -- 172.21.15.16:0/3101588728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1c0101850 msgr2=0x7fc1c019cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:59.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.310+0000 7fc1beffd700 1 --2- 172.21.15.16:0/3101588728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1c0101850 0x7fc1c019cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.310+0000 7fc1beffd700 1 -- 172.21.15.16:0/3101588728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b00095f0 con 0x7fc1c0100ea0 2024-04-22T15:55:59.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.310+0000 7fc1beffd700 1 --2- 172.21.15.16:0/3101588728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1c0100ea0 0x7fc1c019c700 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc1a800e920 tx=0x7fc1a800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:59.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.310+0000 7fc1b7fff700 1 -- 172.21.15.16:0/3101588728 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1a8009790 con 0x7fc1c0100ea0 2024-04-22T15:55:59.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.310+0000 7fc1c5349700 1 -- 172.21.15.16:0/3101588728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c0196a60 con 0x7fc1c0100ea0 2024-04-22T15:55:59.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.310+0000 7fc1b7fff700 1 -- 172.21.15.16:0/3101588728 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc1a800cba0 con 0x7fc1c0100ea0 2024-04-22T15:55:59.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.310+0000 7fc1b7fff700 1 -- 172.21.15.16:0/3101588728 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1a8018650 con 0x7fc1c0100ea0 2024-04-22T15:55:59.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.310+0000 7fc1c5349700 1 -- 172.21.15.16:0/3101588728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c0196fb0 con 0x7fc1c0100ea0 2024-04-22T15:55:59.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.311+0000 7fc1c5349700 1 -- 172.21.15.16:0/3101588728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c01092a0 con 0x7fc1c0100ea0 2024-04-22T15:55:59.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.312+0000 7fc1b7fff700 1 -- 172.21.15.16:0/3101588728 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc1a800cd10 con 0x7fc1c0100ea0 2024-04-22T15:55:59.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.312+0000 7fc1b7fff700 1 --2- 172.21.15.16:0/3101588728 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fc1ac06ecc0 0x7fc1ac071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:59.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.312+0000 7fc1b7fff700 1 -- 172.21.15.16:0/3101588728 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc1a8014070 con 0x7fc1c0100ea0 2024-04-22T15:55:59.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.312+0000 7fc1be7fc700 1 --2- 172.21.15.16:0/3101588728 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fc1ac06ecc0 0x7fc1ac071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:59.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.313+0000 7fc1be7fc700 1 --2- 172.21.15.16:0/3101588728 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fc1ac06ecc0 0x7fc1ac071180 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0009fd0 tx=0x7fc1b0005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:59.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.317+0000 7fc1b7fff700 1 -- 172.21.15.16:0/3101588728 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc1a8054150 con 0x7fc1c0100ea0 2024-04-22T15:55:59.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.612+0000 7fc1c5349700 1 -- 172.21.15.16:0/3101588728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc1c004f2e0 con 0x7fc1c0100ea0 2024-04-22T15:55:59.615 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:59 smithi016 ceph-mon[110740]: pgmap v87: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:55:59.615 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:59 smithi016 ceph-mon[110740]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:59.615 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:55:59 smithi016 ceph-mon[110740]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:59.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.612+0000 7fc1b7fff700 1 -- 172.21.15.16:0/3101588728 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7fc1a805b8c0 con 0x7fc1c0100ea0 2024-04-22T15:55:59.616 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:55:59.616 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-22T15:55:59.616 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:55:59.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:55:59.616 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-22T15:55:59.616 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:55:59.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:55:59.616 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-22T15:55:59.617 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4, 2024-04-22T15:55:59.617 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:55:59.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:55:59.617 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-22T15:55:59.618 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T15:55:59.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:55:59.618 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-22T15:55:59.618 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 8, 2024-04-22T15:55:59.618 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:55:59.618 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-22T15:55:59.618 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:55:59.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.614+0000 7fc1c5349700 1 -- 172.21.15.16:0/3101588728 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fc1ac06ecc0 msgr2=0x7fc1ac071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:59.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.614+0000 7fc1c5349700 1 --2- 172.21.15.16:0/3101588728 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fc1ac06ecc0 0x7fc1ac071180 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0009fd0 tx=0x7fc1b0005940 comp rx=0 tx=0).stop 2024-04-22T15:55:59.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.614+0000 7fc1c5349700 1 -- 172.21.15.16:0/3101588728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1c0100ea0 msgr2=0x7fc1c019c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:59.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.614+0000 7fc1c5349700 1 --2- 172.21.15.16:0/3101588728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1c0100ea0 0x7fc1c019c700 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc1a800e920 tx=0x7fc1a800ece0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.614+0000 7fc1c5349700 1 -- 172.21.15.16:0/3101588728 shutdown_connections 2024-04-22T15:55:59.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.614+0000 7fc1c5349700 1 --2- 172.21.15.16:0/3101588728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1c0100ea0 0x7fc1c019c700 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.614+0000 7fc1c5349700 1 --2- 172.21.15.16:0/3101588728 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fc1ac06ecc0 0x7fc1ac071180 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.614+0000 7fc1c5349700 1 --2- 172.21.15.16:0/3101588728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1c0101850 0x7fc1c019cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.614+0000 7fc1c5349700 1 -- 172.21.15.16:0/3101588728 >> 172.21.15.16:0/3101588728 conn(0x7fc1c00fc750 msgr2=0x7fc1c00fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:59.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.615+0000 7fc1c5349700 1 -- 172.21.15.16:0/3101588728 shutdown_connections 2024-04-22T15:55:59.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.615+0000 7fc1c5349700 1 -- 172.21.15.16:0/3101588728 wait complete. 2024-04-22T15:55:59.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.722+0000 7faf95fed700 1 -- 172.21.15.16:0/1205114324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf900ffc20 msgr2=0x7faf90100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:59.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.722+0000 7faf95fed700 1 --2- 172.21.15.16:0/1205114324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf900ffc20 0x7faf90100080 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7faf84009910 tx=0x7faf84009c20 comp rx=0 tx=0).stop 2024-04-22T15:55:59.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.722+0000 7faf95fed700 1 -- 172.21.15.16:0/1205114324 shutdown_connections 2024-04-22T15:55:59.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.722+0000 7faf95fed700 1 --2- 172.21.15.16:0/1205114324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf900ffc20 0x7faf90100080 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.722+0000 7faf95fed700 1 --2- 172.21.15.16:0/1205114324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf90105bd0 0x7faf90105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.722+0000 7faf95fed700 1 -- 172.21.15.16:0/1205114324 >> 172.21.15.16:0/1205114324 conn(0x7faf90075670 msgr2=0x7faf90075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:59.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.722+0000 7faf95fed700 1 -- 172.21.15.16:0/1205114324 shutdown_connections 2024-04-22T15:55:59.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.722+0000 7faf95fed700 1 -- 172.21.15.16:0/1205114324 wait complete. 2024-04-22T15:55:59.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.723+0000 7faf95fed700 1 Processor -- start 2024-04-22T15:55:59.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.723+0000 7faf95fed700 1 -- start start 2024-04-22T15:55:59.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.723+0000 7faf95fed700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf900ffc20 0x7faf9019c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:59.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.723+0000 7faf95fed700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf90105bd0 0x7faf9019cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:59.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.723+0000 7faf95fed700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9019d230 con 0x7faf90105bd0 2024-04-22T15:55:59.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.723+0000 7faf95fed700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf901966e0 con 0x7faf900ffc20 2024-04-22T15:55:59.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.723+0000 7faf8effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf90105bd0 0x7faf9019cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:59.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.723+0000 7faf8effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf90105bd0 0x7faf9019cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47378/0 (socket says 172.21.15.16:47378) 2024-04-22T15:55:59.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.723+0000 7faf8effd700 1 -- 172.21.15.16:0/574283874 learned_addr learned my addr 172.21.15.16:0/574283874 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:55:59.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.724+0000 7faf8f7fe700 1 --2- 172.21.15.16:0/574283874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf900ffc20 0x7faf9019c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:59.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.724+0000 7faf8effd700 1 -- 172.21.15.16:0/574283874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf900ffc20 msgr2=0x7faf9019c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:59.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.724+0000 7faf8effd700 1 --2- 172.21.15.16:0/574283874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf900ffc20 0x7faf9019c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.724+0000 7faf8effd700 1 -- 172.21.15.16:0/574283874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf840095f0 con 0x7faf90105bd0 2024-04-22T15:55:59.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.724+0000 7faf8effd700 1 --2- 172.21.15.16:0/574283874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf90105bd0 0x7faf9019cba0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7faf84006010 tx=0x7faf8400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:59.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.724+0000 7faf8cff9700 1 -- 172.21.15.16:0/574283874 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf8401d070 con 0x7faf90105bd0 2024-04-22T15:55:59.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.724+0000 7faf8cff9700 1 -- 172.21.15.16:0/574283874 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf84005dc0 con 0x7faf90105bd0 2024-04-22T15:55:59.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.724+0000 7faf95fed700 1 -- 172.21.15.16:0/574283874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf90196960 con 0x7faf90105bd0 2024-04-22T15:55:59.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.724+0000 7faf8cff9700 1 -- 172.21.15.16:0/574283874 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf84017400 con 0x7faf90105bd0 2024-04-22T15:55:59.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.724+0000 7faf95fed700 1 -- 172.21.15.16:0/574283874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf90196e50 con 0x7faf90105bd0 2024-04-22T15:55:59.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.725+0000 7faf95fed700 1 -- 172.21.15.16:0/574283874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf9004f2e0 con 0x7faf90105bd0 2024-04-22T15:55:59.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.726+0000 7faf8cff9700 1 -- 172.21.15.16:0/574283874 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7faf84003a60 con 0x7faf90105bd0 2024-04-22T15:55:59.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.726+0000 7faf8cff9700 1 --2- 172.21.15.16:0/574283874 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7faf7806ecc0 0x7faf78071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:55:59.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.726+0000 7faf8cff9700 1 -- 172.21.15.16:0/574283874 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6328+0+0 (secure 0 0 0) 0x7faf84090150 con 0x7faf90105bd0 2024-04-22T15:55:59.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.726+0000 7faf8f7fe700 1 --2- 172.21.15.16:0/574283874 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7faf7806ecc0 0x7faf78071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:55:59.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.727+0000 7faf8f7fe700 1 --2- 172.21.15.16:0/574283874 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7faf7806ecc0 0x7faf78071180 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faf80005950 tx=0x7faf8000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:55:59.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.731+0000 7faf8cff9700 1 -- 172.21.15.16:0/574283874 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7faf8405cb60 con 0x7faf90105bd0 2024-04-22T15:55:59.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:59 smithi027 ceph-mon[83054]: pgmap v87: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:55:59.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:59 smithi027 ceph-mon[83054]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:59.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:55:59 smithi027 ceph-mon[83054]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:55:59.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.987+0000 7faf95fed700 1 -- 172.21.15.16:0/574283874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7faf90197bf0 con 0x7faf90105bd0 2024-04-22T15:55:59.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.987+0000 7faf8cff9700 1 -- 172.21.15.16:0/574283874 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1902 (secure 0 0 0) 0x7faf8405c6f0 con 0x7faf90105bd0 2024-04-22T15:55:59.990 INFO:teuthology.orchestra.run.smithi016.stdout:e16 2024-04-22T15:55:59.991 INFO:teuthology.orchestra.run.smithi016.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:55:59.991 INFO:teuthology.orchestra.run.smithi016.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:55:59.991 INFO:teuthology.orchestra.run.smithi016.stdout:legacy client fscid: 1 2024-04-22T15:55:59.991 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:55:59.991 INFO:teuthology.orchestra.run.smithi016.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:55:59.991 INFO:teuthology.orchestra.run.smithi016.stdout:fs_name cephfs 2024-04-22T15:55:59.991 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 16 2024-04-22T15:55:59.991 INFO:teuthology.orchestra.run.smithi016.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T15:55:59.991 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-04-22T15:46:27.774571+0000 2024-04-22T15:55:59.991 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-04-22T15:46:47.381709+0000 2024-04-22T15:55:59.991 INFO:teuthology.orchestra.run.smithi016.stdout:tableserver 0 2024-04-22T15:55:59.991 INFO:teuthology.orchestra.run.smithi016.stdout:root 0 2024-04-22T15:55:59.991 INFO:teuthology.orchestra.run.smithi016.stdout:session_timeout 60 2024-04-22T15:55:59.992 INFO:teuthology.orchestra.run.smithi016.stdout:session_autoclose 300 2024-04-22T15:55:59.992 INFO:teuthology.orchestra.run.smithi016.stdout:max_file_size 1099511627776 2024-04-22T15:55:59.992 INFO:teuthology.orchestra.run.smithi016.stdout:max_xattr_size 65536 2024-04-22T15:55:59.992 INFO:teuthology.orchestra.run.smithi016.stdout:required_client_features {} 2024-04-22T15:55:59.992 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure 0 2024-04-22T15:55:59.992 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure_osd_epoch 46 2024-04-22T15:55:59.992 INFO:teuthology.orchestra.run.smithi016.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:55:59.992 INFO:teuthology.orchestra.run.smithi016.stdout:max_mds 1 2024-04-22T15:55:59.992 INFO:teuthology.orchestra.run.smithi016.stdout:in 0 2024-04-22T15:55:59.992 INFO:teuthology.orchestra.run.smithi016.stdout:up {0=24279} 2024-04-22T15:55:59.992 INFO:teuthology.orchestra.run.smithi016.stdout:failed 2024-04-22T15:55:59.992 INFO:teuthology.orchestra.run.smithi016.stdout:damaged 2024-04-22T15:55:59.992 INFO:teuthology.orchestra.run.smithi016.stdout:stopped 2024-04-22T15:55:59.992 INFO:teuthology.orchestra.run.smithi016.stdout:data_pools [3] 2024-04-22T15:55:59.992 INFO:teuthology.orchestra.run.smithi016.stdout:metadata_pool 2 2024-04-22T15:55:59.993 INFO:teuthology.orchestra.run.smithi016.stdout:inline_data enabled 2024-04-22T15:55:59.993 INFO:teuthology.orchestra.run.smithi016.stdout:balancer 2024-04-22T15:55:59.993 INFO:teuthology.orchestra.run.smithi016.stdout:bal_rank_mask -1 2024-04-22T15:55:59.993 INFO:teuthology.orchestra.run.smithi016.stdout:standby_count_wanted 1 2024-04-22T15:55:59.993 INFO:teuthology.orchestra.run.smithi016.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:55:59.993 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.iobjds{0:24279} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.27:6824/4216277609,v1:172.21.15.27:6825/4216277609] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:55:59.993 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.pgvphu{0:14494} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.16:6828/1006073414,v1:172.21.15.16:6829/1006073414] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:55:59.993 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:55:59.993 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:55:59.993 INFO:teuthology.orchestra.run.smithi016.stdout:Standby daemons: 2024-04-22T15:55:59.993 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:55:59.993 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.fkkejm{-1:14504} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.16:6826/3484907755,v1:172.21.15.16:6827/3484907755] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:55:59.993 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.dgqyes{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.27:6826/476026355,v1:172.21.15.27:6827/476026355] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:55:59.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.989+0000 7faf95fed700 1 -- 172.21.15.16:0/574283874 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7faf7806ecc0 msgr2=0x7faf78071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:59.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.989+0000 7faf95fed700 1 --2- 172.21.15.16:0/574283874 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7faf7806ecc0 0x7faf78071180 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faf80005950 tx=0x7faf8000b410 comp rx=0 tx=0).stop 2024-04-22T15:55:59.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.989+0000 7faf95fed700 1 -- 172.21.15.16:0/574283874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf90105bd0 msgr2=0x7faf9019cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:55:59.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.989+0000 7faf95fed700 1 --2- 172.21.15.16:0/574283874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf90105bd0 0x7faf9019cba0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7faf84006010 tx=0x7faf8400bd70 comp rx=0 tx=0).stop 2024-04-22T15:55:59.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.989+0000 7faf95fed700 1 -- 172.21.15.16:0/574283874 shutdown_connections 2024-04-22T15:55:59.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.989+0000 7faf95fed700 1 --2- 172.21.15.16:0/574283874 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7faf7806ecc0 0x7faf78071180 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.989+0000 7faf95fed700 1 --2- 172.21.15.16:0/574283874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf900ffc20 0x7faf9019c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.989+0000 7faf95fed700 1 --2- 172.21.15.16:0/574283874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf90105bd0 0x7faf9019cba0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:55:59.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.989+0000 7faf95fed700 1 -- 172.21.15.16:0/574283874 >> 172.21.15.16:0/574283874 conn(0x7faf90075670 msgr2=0x7faf900686a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:55:59.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.990+0000 7faf95fed700 1 -- 172.21.15.16:0/574283874 shutdown_connections 2024-04-22T15:55:59.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:55:59.990+0000 7faf95fed700 1 -- 172.21.15.16:0/574283874 wait complete. 2024-04-22T15:55:59.995 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 16 2024-04-22T15:56:00.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.098+0000 7f703e805700 1 -- 172.21.15.16:0/2665436055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7038105bd0 msgr2=0x7f7038105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:00.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.098+0000 7f703e805700 1 --2- 172.21.15.16:0/2665436055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7038105bd0 0x7f7038105fb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f702c009910 tx=0x7f702c009c20 comp rx=0 tx=0).stop 2024-04-22T15:56:00.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.098+0000 7f703e805700 1 -- 172.21.15.16:0/2665436055 shutdown_connections 2024-04-22T15:56:00.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.098+0000 7f703e805700 1 --2- 172.21.15.16:0/2665436055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70380ffc20 0x7f7038100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:00.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.098+0000 7f703e805700 1 --2- 172.21.15.16:0/2665436055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7038105bd0 0x7f7038105fb0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:00.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.098+0000 7f703e805700 1 -- 172.21.15.16:0/2665436055 >> 172.21.15.16:0/2665436055 conn(0x7f7038075670 msgr2=0x7f7038075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:00.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.098+0000 7f703e805700 1 -- 172.21.15.16:0/2665436055 shutdown_connections 2024-04-22T15:56:00.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.098+0000 7f703e805700 1 -- 172.21.15.16:0/2665436055 wait complete. 2024-04-22T15:56:00.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.099+0000 7f703e805700 1 Processor -- start 2024-04-22T15:56:00.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.099+0000 7f703e805700 1 -- start start 2024-04-22T15:56:00.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.099+0000 7f703e805700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70380ffc20 0x7f703819b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.099+0000 7f703e805700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7038105bd0 0x7f703819bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.099+0000 7f703e805700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7038195590 con 0x7f70380ffc20 2024-04-22T15:56:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.099+0000 7f703e805700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7038195700 con 0x7f7038105bd0 2024-04-22T15:56:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.099+0000 7f7037fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70380ffc20 0x7f703819b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.099+0000 7f7037fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70380ffc20 0x7f703819b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47386/0 (socket says 172.21.15.16:47386) 2024-04-22T15:56:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.099+0000 7f7037fff700 1 -- 172.21.15.16:0/3376875279 learned_addr learned my addr 172.21.15.16:0/3376875279 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:56:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.099+0000 7f70377fe700 1 --2- 172.21.15.16:0/3376875279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7038105bd0 0x7f703819bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.100+0000 7f7037fff700 1 -- 172.21.15.16:0/3376875279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7038105bd0 msgr2=0x7f703819bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.100+0000 7f7037fff700 1 --2- 172.21.15.16:0/3376875279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7038105bd0 0x7f703819bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.100+0000 7f7037fff700 1 -- 172.21.15.16:0/3376875279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f702c0095f0 con 0x7f70380ffc20 2024-04-22T15:56:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.100+0000 7f70377fe700 1 --2- 172.21.15.16:0/3376875279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7038105bd0 0x7f703819bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:56:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.100+0000 7f7037fff700 1 --2- 172.21.15.16:0/3376875279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70380ffc20 0x7f703819b5a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f702c00f690 tx=0x7f702c00f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.100+0000 7f70357fa700 1 -- 172.21.15.16:0/3376875279 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f702c01d070 con 0x7f70380ffc20 2024-04-22T15:56:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.100+0000 7f70357fa700 1 -- 172.21.15.16:0/3376875279 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f702c0050d0 con 0x7f70380ffc20 2024-04-22T15:56:00.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.100+0000 7f703e805700 1 -- 172.21.15.16:0/3376875279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7038195980 con 0x7f70380ffc20 2024-04-22T15:56:00.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.100+0000 7f70357fa700 1 -- 172.21.15.16:0/3376875279 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f702c022470 con 0x7f70380ffc20 2024-04-22T15:56:00.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.100+0000 7f703e805700 1 -- 172.21.15.16:0/3376875279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7038195e70 con 0x7f70380ffc20 2024-04-22T15:56:00.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.101+0000 7f703e805700 1 -- 172.21.15.16:0/3376875279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f703804f2e0 con 0x7f70380ffc20 2024-04-22T15:56:00.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.101+0000 7f70357fa700 1 -- 172.21.15.16:0/3376875279 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f702c003680 con 0x7f70380ffc20 2024-04-22T15:56:00.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.102+0000 7f70357fa700 1 --2- 172.21.15.16:0/3376875279 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f7020070ed0 0x7f7020073390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:00.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.102+0000 7f70357fa700 1 -- 172.21.15.16:0/3376875279 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f702c0901e0 con 0x7f70380ffc20 2024-04-22T15:56:00.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.102+0000 7f70377fe700 1 --2- 172.21.15.16:0/3376875279 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f7020070ed0 0x7f7020073390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:00.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.102+0000 7f70377fe700 1 --2- 172.21.15.16:0/3376875279 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f7020070ed0 0x7f7020073390 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7038196e30 tx=0x7f7028009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:00.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.107+0000 7f70357fa700 1 -- 172.21.15.16:0/3376875279 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f702c059060 con 0x7f70380ffc20 2024-04-22T15:56:00.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.344+0000 7f703e805700 1 -- 172.21.15.16:0/3376875279 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f70381960e0 con 0x7f7020070ed0 2024-04-22T15:56:00.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.520+0000 7f70357fa700 1 -- 172.21.15.16:0/3376875279 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f70381960e0 con 0x7f7020070ed0 2024-04-22T15:56:00.523 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:56:00.523 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:56:00.523 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-22T15:56:00.524 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:56:00.524 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-22T15:56:00.524 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-04-22T15:56:00.524 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr", 2024-04-22T15:56:00.524 INFO:teuthology.orchestra.run.smithi016.stdout: "mon" 2024-04-22T15:56:00.524 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-22T15:56:00.524 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "8/23 daemons upgraded", 2024-04-22T15:56:00.524 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading osd daemons", 2024-04-22T15:56:00.524 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-22T15:56:00.524 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:56:00.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.522+0000 7f703e805700 1 -- 172.21.15.16:0/3376875279 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f7020070ed0 msgr2=0x7f7020073390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:00.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.522+0000 7f703e805700 1 --2- 172.21.15.16:0/3376875279 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f7020070ed0 0x7f7020073390 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7038196e30 tx=0x7f7028009260 comp rx=0 tx=0).stop 2024-04-22T15:56:00.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.522+0000 7f703e805700 1 -- 172.21.15.16:0/3376875279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70380ffc20 msgr2=0x7f703819b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:00.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.522+0000 7f703e805700 1 --2- 172.21.15.16:0/3376875279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70380ffc20 0x7f703819b5a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f702c00f690 tx=0x7f702c00f770 comp rx=0 tx=0).stop 2024-04-22T15:56:00.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.523+0000 7f703e805700 1 -- 172.21.15.16:0/3376875279 shutdown_connections 2024-04-22T15:56:00.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.523+0000 7f703e805700 1 --2- 172.21.15.16:0/3376875279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70380ffc20 0x7f703819b5a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:00.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.523+0000 7f703e805700 1 --2- 172.21.15.16:0/3376875279 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f7020070ed0 0x7f7020073390 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:00.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.523+0000 7f703e805700 1 --2- 172.21.15.16:0/3376875279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7038105bd0 0x7f703819bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:00.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.523+0000 7f703e805700 1 -- 172.21.15.16:0/3376875279 >> 172.21.15.16:0/3376875279 conn(0x7f7038075670 msgr2=0x7f70380feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:00.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.523+0000 7f703e805700 1 -- 172.21.15.16:0/3376875279 shutdown_connections 2024-04-22T15:56:00.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.523+0000 7f703e805700 1 -- 172.21.15.16:0/3376875279 wait complete. 2024-04-22T15:56:00.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.632+0000 7f9c750c1700 1 -- 172.21.15.16:0/1622789345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c700ff5a0 msgr2=0x7f9c700ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:00.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.632+0000 7f9c750c1700 1 --2- 172.21.15.16:0/1622789345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c700ff5a0 0x7f9c700ff980 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9c60009910 tx=0x7f9c60009c20 comp rx=0 tx=0).stop 2024-04-22T15:56:00.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.632+0000 7f9c750c1700 1 -- 172.21.15.16:0/1622789345 shutdown_connections 2024-04-22T15:56:00.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.632+0000 7f9c750c1700 1 --2- 172.21.15.16:0/1622789345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c700fff50 0x7f9c70105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:00.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.632+0000 7f9c750c1700 1 --2- 172.21.15.16:0/1622789345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c700ff5a0 0x7f9c700ff980 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:00.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.632+0000 7f9c750c1700 1 -- 172.21.15.16:0/1622789345 >> 172.21.15.16:0/1622789345 conn(0x7f9c70074e30 msgr2=0x7f9c70075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:00.636 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:00 smithi016 ceph-mon[110740]: from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:56:00.636 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:00 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/3101588728' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:56:00.636 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:00 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/574283874' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:56:00.636 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:00 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:00.636 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:00 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:56:00.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.632+0000 7f9c750c1700 1 -- 172.21.15.16:0/1622789345 shutdown_connections 2024-04-22T15:56:00.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.632+0000 7f9c750c1700 1 -- 172.21.15.16:0/1622789345 wait complete. 2024-04-22T15:56:00.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.633+0000 7f9c750c1700 1 Processor -- start 2024-04-22T15:56:00.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.633+0000 7f9c750c1700 1 -- start start 2024-04-22T15:56:00.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.633+0000 7f9c750c1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c700ff5a0 0x7f9c70100520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:00.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.633+0000 7f9c750c1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c700fff50 0x7f9c70100a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:00.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.633+0000 7f9c750c1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c70100fa0 con 0x7f9c700fff50 2024-04-22T15:56:00.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.633+0000 7f9c750c1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c701010e0 con 0x7f9c700ff5a0 2024-04-22T15:56:00.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.634+0000 7f9c6e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c700fff50 0x7f9c70100a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:00.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.634+0000 7f9c6e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c700fff50 0x7f9c70100a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47404/0 (socket says 172.21.15.16:47404) 2024-04-22T15:56:00.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.634+0000 7f9c6e59c700 1 -- 172.21.15.16:0/654805187 learned_addr learned my addr 172.21.15.16:0/654805187 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:56:00.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.634+0000 7f9c6ed9d700 1 --2- 172.21.15.16:0/654805187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c700ff5a0 0x7f9c70100520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.634+0000 7f9c6e59c700 1 -- 172.21.15.16:0/654805187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c700ff5a0 msgr2=0x7f9c70100520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.634+0000 7f9c6e59c700 1 --2- 172.21.15.16:0/654805187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c700ff5a0 0x7f9c70100520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.634+0000 7f9c6e59c700 1 -- 172.21.15.16:0/654805187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c600095f0 con 0x7f9c700fff50 2024-04-22T15:56:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.634+0000 7f9c6e59c700 1 --2- 172.21.15.16:0/654805187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c700fff50 0x7f9c70100a60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9c6400c8f0 tx=0x7f9c6400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.634+0000 7f9c57fff700 1 -- 172.21.15.16:0/654805187 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c640078a0 con 0x7f9c700fff50 2024-04-22T15:56:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.635+0000 7f9c57fff700 1 -- 172.21.15.16:0/654805187 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c6400f450 con 0x7f9c700fff50 2024-04-22T15:56:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.635+0000 7f9c57fff700 1 -- 172.21.15.16:0/654805187 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c64018610 con 0x7f9c700fff50 2024-04-22T15:56:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.635+0000 7f9c750c1700 1 -- 172.21.15.16:0/654805187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c701a2a80 con 0x7f9c700fff50 2024-04-22T15:56:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.635+0000 7f9c750c1700 1 -- 172.21.15.16:0/654805187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c701a2e30 con 0x7f9c700fff50 2024-04-22T15:56:00.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.636+0000 7f9c750c1700 1 -- 172.21.15.16:0/654805187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c70109270 con 0x7f9c700fff50 2024-04-22T15:56:00.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.637+0000 7f9c57fff700 1 -- 172.21.15.16:0/654805187 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9c6400f5c0 con 0x7f9c700fff50 2024-04-22T15:56:00.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.637+0000 7f9c57fff700 1 --2- 172.21.15.16:0/654805187 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f9c5806ecc0 0x7f9c58071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:00.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.637+0000 7f9c57fff700 1 -- 172.21.15.16:0/654805187 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f9c6408ec90 con 0x7f9c700fff50 2024-04-22T15:56:00.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.637+0000 7f9c6ed9d700 1 --2- 172.21.15.16:0/654805187 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f9c5806ecc0 0x7f9c58071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:00.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.638+0000 7f9c6ed9d700 1 --2- 172.21.15.16:0/654805187 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f9c5806ecc0 0x7f9c58071180 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9c60005f50 tx=0x7f9c6001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:00.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.642+0000 7f9c57fff700 1 -- 172.21.15.16:0/654805187 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f9c64057b70 con 0x7f9c700fff50 2024-04-22T15:56:00.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:00 smithi027 ceph-mon[83054]: from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:56:00.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:00 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/3101588728' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:56:00.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:00 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/574283874' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:56:00.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:00 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:00.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:00 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:56:00.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.943+0000 7f9c750c1700 1 -- 172.21.15.16:0/654805187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9c7004f2e0 con 0x7f9c700fff50 2024-04-22T15:56:00.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.943+0000 7f9c57fff700 1 -- 172.21.15.16:0/654805187 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f9c6405b230 con 0x7f9c700fff50 2024-04-22T15:56:00.946 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:56:00.947 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:56:00.947 INFO:teuthology.orchestra.run.smithi016.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:56:00.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.946+0000 7f9c750c1700 1 -- 172.21.15.16:0/654805187 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f9c5806ecc0 msgr2=0x7f9c58071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:00.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.946+0000 7f9c750c1700 1 --2- 172.21.15.16:0/654805187 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f9c5806ecc0 0x7f9c58071180 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9c60005f50 tx=0x7f9c6001a040 comp rx=0 tx=0).stop 2024-04-22T15:56:00.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.946+0000 7f9c750c1700 1 -- 172.21.15.16:0/654805187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c700fff50 msgr2=0x7f9c70100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:00.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.946+0000 7f9c750c1700 1 --2- 172.21.15.16:0/654805187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c700fff50 0x7f9c70100a60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9c6400c8f0 tx=0x7f9c6400cc00 comp rx=0 tx=0).stop 2024-04-22T15:56:00.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.946+0000 7f9c750c1700 1 -- 172.21.15.16:0/654805187 shutdown_connections 2024-04-22T15:56:00.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.946+0000 7f9c750c1700 1 --2- 172.21.15.16:0/654805187 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f9c5806ecc0 0x7f9c58071180 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:00.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.946+0000 7f9c750c1700 1 --2- 172.21.15.16:0/654805187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c700ff5a0 0x7f9c70100520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:00.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.946+0000 7f9c750c1700 1 --2- 172.21.15.16:0/654805187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c700fff50 0x7f9c70100a60 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:00.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.946+0000 7f9c750c1700 1 -- 172.21.15.16:0/654805187 >> 172.21.15.16:0/654805187 conn(0x7f9c70074e30 msgr2=0x7f9c700fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:00.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.946+0000 7f9c750c1700 1 -- 172.21.15.16:0/654805187 shutdown_connections 2024-04-22T15:56:00.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:00.946+0000 7f9c750c1700 1 -- 172.21.15.16:0/654805187 wait complete. 2024-04-22T15:56:01.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:01 smithi016 ceph-mon[110740]: pgmap v88: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:56:01.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:01 smithi016 ceph-mon[110740]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:56:01.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:01 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-22T15:56:01.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:01 smithi016 ceph-mon[110740]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-22T15:56:01.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:01 smithi016 ceph-mon[110740]: Upgrade: osd.2 is safe to restart 2024-04-22T15:56:01.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:01 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/654805187' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:56:01.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:01 smithi027 ceph-mon[83054]: pgmap v88: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:56:01.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:01 smithi027 ceph-mon[83054]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:56:01.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:01 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-22T15:56:01.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:01 smithi027 ceph-mon[83054]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-22T15:56:01.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:01 smithi027 ceph-mon[83054]: Upgrade: osd.2 is safe to restart 2024-04-22T15:56:01.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:01 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/654805187' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:56:03.312 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:03 smithi016 ceph-mon[110740]: Upgrade: Updating osd.2 2024-04-22T15:56:03.313 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:03 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:03.313 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:03 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T15:56:03.313 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:03 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:56:03.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:03 smithi027 ceph-mon[83054]: Upgrade: Updating osd.2 2024-04-22T15:56:03.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:03 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:03.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:03 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T15:56:03.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:03 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:56:04.298 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:04 smithi016 ceph-mon[110740]: Deploying daemon osd.2 on smithi016 2024-04-22T15:56:04.299 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:04 smithi016 ceph-mon[110740]: pgmap v89: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:56:04.472 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:04 smithi027 ceph-mon[83054]: Deploying daemon osd.2 on smithi016 2024-04-22T15:56:04.473 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:04 smithi027 ceph-mon[83054]: pgmap v89: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:56:05.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:05 smithi016 ceph-mon[110740]: pgmap v90: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:56:05.417 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:05 smithi016 systemd[1]: Stopping Ceph osd.2 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T15:56:05.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:05 smithi027 ceph-mon[83054]: pgmap v90: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:56:05.916 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:05 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2[60684]: 2024-04-22T15:56:05.471+0000 7f9ab1001700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T15:56:05.917 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:05 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2[60684]: 2024-04-22T15:56:05.471+0000 7f9ab1001700 -1 osd.2 59 *** Got signal Terminated *** 2024-04-22T15:56:05.917 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:05 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2[60684]: 2024-04-22T15:56:05.471+0000 7f9ab1001700 -1 osd.2 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T15:56:06.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:06 smithi016 ceph-mon[110740]: osd.2 marked itself down and dead 2024-04-22T15:56:06.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:06 smithi027 ceph-mon[83054]: osd.2 marked itself down and dead 2024-04-22T15:56:06.933 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:06 smithi016 podman[133023]: 2024-04-22 15:56:06.668562324 +0000 UTC m=+1.305426999 container died 3d9d310aa9eb3d36aed7f6f122320450879eedd59bcdefd363ed5ae8f522617d (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_BRANCH=HEAD, RELEASE=HEAD) 2024-04-22T15:56:07.317 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:07 smithi016 ceph-mon[110740]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T15:56:07.317 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:07 smithi016 ceph-mon[110740]: osdmap e60: 6 total, 5 up, 6 in 2024-04-22T15:56:07.317 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:07 smithi016 ceph-mon[110740]: pgmap v92: 65 pgs: 12 peering, 7 stale+active+clean, 46 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 921 B/s rd, 1 op/s 2024-04-22T15:56:07.317 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:07 smithi016 podman[133023]: 2024-04-22 15:56:07.10517104 +0000 UTC m=+1.742035712 container cleanup 3d9d310aa9eb3d36aed7f6f122320450879eedd59bcdefd363ed5ae8f522617d (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240222) 2024-04-22T15:56:07.318 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:07 smithi016 bash[133023]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2 2024-04-22T15:56:07.488 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:07 smithi027 ceph-mon[83054]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T15:56:07.488 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:07 smithi027 ceph-mon[83054]: osdmap e60: 6 total, 5 up, 6 in 2024-04-22T15:56:07.488 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:07 smithi027 ceph-mon[83054]: pgmap v92: 65 pgs: 12 peering, 7 stale+active+clean, 46 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 921 B/s rd, 1 op/s 2024-04-22T15:56:07.666 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:07 smithi016 podman[133041]: 2024-04-22 15:56:07.315259147 +0000 UTC m=+0.642246569 container remove 3d9d310aa9eb3d36aed7f6f122320450879eedd59bcdefd363ed5ae8f522617d (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.license=GPLv2) 2024-04-22T15:56:08.119 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:07 smithi016 podman[133134]: 2024-04-22 15:56:07.639341908 +0000 UTC m=+0.018403195 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:56:08.119 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:07 smithi016 podman[133134]: 2024-04-22 15:56:07.741270392 +0000 UTC m=+0.120331704 container create d7f66e63fed66b44cf0c75dcf48bfdf766b6a5d7b8edac525dbd0d9601c4f5cb (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:56:08.119 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:07 smithi016 podman[133134]: 2024-04-22 15:56:07.902067157 +0000 UTC m=+0.281128424 container init d7f66e63fed66b44cf0c75dcf48bfdf766b6a5d7b8edac525dbd0d9601c4f5cb (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-22T15:56:08.119 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:07 smithi016 podman[133134]: 2024-04-22 15:56:07.905366944 +0000 UTC m=+0.284428224 container start d7f66e63fed66b44cf0c75dcf48bfdf766b6a5d7b8edac525dbd0d9601c4f5cb (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-22T15:56:08.120 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:07 smithi016 podman[133134]: 2024-04-22 15:56:07.985198175 +0000 UTC m=+0.364259458 container attach d7f66e63fed66b44cf0c75dcf48bfdf766b6a5d7b8edac525dbd0d9601c4f5cb (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49) 2024-04-22T15:56:08.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:08 smithi016 ceph-mon[110740]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-22T15:56:08.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:08 smithi016 ceph-mon[110740]: osdmap e61: 6 total, 5 up, 6 in 2024-04-22T15:56:08.417 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:08 smithi016 podman[133134]: 2024-04-22 15:56:08.218160377 +0000 UTC m=+0.597221649 container died d7f66e63fed66b44cf0c75dcf48bfdf766b6a5d7b8edac525dbd0d9601c4f5cb (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-22T15:56:08.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:08 smithi027 ceph-mon[83054]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-22T15:56:08.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:08 smithi027 ceph-mon[83054]: osdmap e61: 6 total, 5 up, 6 in 2024-04-22T15:56:09.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:09 smithi016 ceph-mon[110740]: pgmap v94: 65 pgs: 19 peering, 3 stale+active+clean, 43 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 895 B/s rd, 1 op/s 2024-04-22T15:56:09.167 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:08 smithi016 podman[133159]: 2024-04-22 15:56:08.889635825 +0000 UTC m=+0.661035944 container remove d7f66e63fed66b44cf0c75dcf48bfdf766b6a5d7b8edac525dbd0d9601c4f5cb (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-22T15:56:09.167 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:08 smithi016 systemd[1]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.2.service: Deactivated successfully. 2024-04-22T15:56:09.167 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:08 smithi016 systemd[1]: Stopped Ceph osd.2 for 67ab3a80-00be-11ef-bc93-c7b262605968. 2024-04-22T15:56:09.167 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:08 smithi016 systemd[1]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.2.service: Consumed 49.326s CPU time. 2024-04-22T15:56:09.167 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:09 smithi016 systemd[1]: Starting Ceph osd.2 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T15:56:09.523 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:09 smithi027 ceph-mon[83054]: pgmap v94: 65 pgs: 19 peering, 3 stale+active+clean, 43 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 895 B/s rd, 1 op/s 2024-04-22T15:56:09.595 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:09 smithi016 podman[133263]: 2024-04-22 15:56:09.34069838 +0000 UTC m=+0.102621700 container create 67068e5007fe904f0a8ad64b211ba694b771c049f04147c206651fb1b135de10 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-activate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:56:09.595 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:09 smithi016 podman[133263]: 2024-04-22 15:56:09.256865723 +0000 UTC m=+0.018789015 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:56:09.595 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:09 smithi016 podman[133263]: 2024-04-22 15:56:09.50969417 +0000 UTC m=+0.271617448 container init 67068e5007fe904f0a8ad64b211ba694b771c049f04147c206651fb1b135de10 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-activate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True) 2024-04-22T15:56:09.595 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:09 smithi016 podman[133263]: 2024-04-22 15:56:09.512986117 +0000 UTC m=+0.274909397 container start 67068e5007fe904f0a8ad64b211ba694b771c049f04147c206651fb1b135de10 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-activate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:56:09.595 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:09 smithi016 podman[133263]: 2024-04-22 15:56:09.593021653 +0000 UTC m=+0.354944946 container attach 67068e5007fe904f0a8ad64b211ba694b771c049f04147c206651fb1b135de10 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-22T15:56:10.383 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:10 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-activate[133277]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:56:10.383 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:10 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-activate[133277]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:56:10.383 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:10 smithi016 bash[133263]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:56:10.383 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:10 smithi016 bash[133263]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:56:10.383 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:10 smithi016 bash[133263]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-22T15:56:10.383 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:10 smithi016 bash[133263]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-22T15:56:10.383 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:10 smithi016 bash[133263]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-22T15:56:10.383 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:10 smithi016 bash[133263]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-22T15:56:10.384 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:10 smithi016 bash[133263]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-22T15:56:10.384 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:10 smithi016 bash[133263]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-22T15:56:10.384 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:10 smithi016 bash[133263]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-22T15:56:10.384 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:10 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-activate[133277]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-22T15:56:10.384 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:10 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-activate[133277]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-22T15:56:10.384 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:10 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-activate[133277]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-22T15:56:10.384 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:10 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-activate[133277]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-22T15:56:10.384 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:10 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-activate[133277]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-22T15:56:10.384 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:10 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-activate[133277]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-22T15:56:10.384 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:10 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-activate[133277]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-22T15:56:10.384 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:10 smithi016 podman[133263]: 2024-04-22 15:56:10.033558847 +0000 UTC m=+0.795482127 container died 67068e5007fe904f0a8ad64b211ba694b771c049f04147c206651fb1b135de10 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-activate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:56:11.015 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:10 smithi016 podman[133423]: 2024-04-22 15:56:10.680569811 +0000 UTC m=+0.637103898 container remove 67068e5007fe904f0a8ad64b211ba694b771c049f04147c206651fb1b135de10 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:56:11.015 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:10 smithi016 podman[133490]: 2024-04-22 15:56:10.802172544 +0000 UTC m=+0.019022847 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:56:11.015 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:10 smithi016 podman[133490]: 2024-04-22 15:56:10.914954035 +0000 UTC m=+0.131804323 container create 48bb459fd021a9aa416678c2047ee8afc67f7d599ecc8677256372aebf7dad8b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T15:56:11.286 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:11 smithi016 podman[133490]: 2024-04-22 15:56:11.117347302 +0000 UTC m=+0.334197591 container init 48bb459fd021a9aa416678c2047ee8afc67f7d599ecc8677256372aebf7dad8b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416) 2024-04-22T15:56:11.286 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:11 smithi016 podman[133490]: 2024-04-22 15:56:11.120496652 +0000 UTC m=+0.337346941 container start 48bb459fd021a9aa416678c2047ee8afc67f7d599ecc8677256372aebf7dad8b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-22T15:56:11.286 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:11 smithi016 ceph-osd[133509]: -- 172.21.15.16:0/3732398873 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5577c6961680 con 0x5577c692b000 2024-04-22T15:56:11.286 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:11 smithi016 bash[133490]: 48bb459fd021a9aa416678c2047ee8afc67f7d599ecc8677256372aebf7dad8b 2024-04-22T15:56:11.286 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:11 smithi016 systemd[1]: Started Ceph osd.2 for 67ab3a80-00be-11ef-bc93-c7b262605968. 2024-04-22T15:56:11.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:11 smithi027 ceph-mon[83054]: pgmap v95: 65 pgs: 19 peering, 3 stale+active+clean, 43 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 255 B/s rd, 0 op/s 2024-04-22T15:56:11.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:11 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:11.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:11 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:11.662 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:11 smithi016 ceph-mon[110740]: pgmap v95: 65 pgs: 19 peering, 3 stale+active+clean, 43 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 255 B/s rd, 0 op/s 2024-04-22T15:56:11.662 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:11 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:11.662 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:11 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:11.916 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:11 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2[133505]: 2024-04-22T15:56:11.658+0000 7f279531c7c0 -1 Falling back to public interface 2024-04-22T15:56:12.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:12 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:56:12.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:12 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:56:13.571 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:13 smithi016 ceph-mon[110740]: pgmap v96: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 895 B/s rd, 1 op/s; 33/258 objects degraded (12.791%) 2024-04-22T15:56:13.571 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:13 smithi016 ceph-mon[110740]: Health check failed: Degraded data redundancy: 33/258 objects degraded (12.791%), 12 pgs degraded (PG_DEGRADED) 2024-04-22T15:56:13.571 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:13 smithi016 ceph-mon[110740]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs peering) 2024-04-22T15:56:13.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:13 smithi027 ceph-mon[83054]: pgmap v96: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 895 B/s rd, 1 op/s; 33/258 objects degraded (12.791%) 2024-04-22T15:56:13.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:13 smithi027 ceph-mon[83054]: Health check failed: Degraded data redundancy: 33/258 objects degraded (12.791%), 12 pgs degraded (PG_DEGRADED) 2024-04-22T15:56:13.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:13 smithi027 ceph-mon[83054]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs peering) 2024-04-22T15:56:15.579 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:15 smithi016 ceph-mon[110740]: pgmap v97: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 883 B/s rd, 1 op/s; 33/258 objects degraded (12.791%) 2024-04-22T15:56:15.579 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:15 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:15.579 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:15 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:56:15.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:15 smithi027 ceph-mon[83054]: pgmap v97: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 883 B/s rd, 1 op/s; 33/258 objects degraded (12.791%) 2024-04-22T15:56:15.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:15 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:15.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:15 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:56:15.830 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:15 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2[133505]: 2024-04-22T15:56:15.574+0000 7f279531c7c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-04-22T15:56:16.100 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:15 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2[133505]: 2024-04-22T15:56:15.854+0000 7f279531c7c0 -1 osd.2 59 log_to_monitors true 2024-04-22T15:56:16.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:16 smithi016 ceph-mon[110740]: from='osd.2 [v2:172.21.15.16:6818/2085469945,v1:172.21.15.16:6819/2085469945]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T15:56:16.417 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 15:56:16 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2[133505]: 2024-04-22T15:56:16.318+0000 7f2786774700 -1 osd.2 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T15:56:16.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:16 smithi027 ceph-mon[83054]: from='osd.2 [v2:172.21.15.16:6818/2085469945,v1:172.21.15.16:6819/2085469945]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T15:56:17.451 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:17 smithi016 ceph-mon[110740]: pgmap v98: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 819 B/s rd, 1 op/s; 33/258 objects degraded (12.791%) 2024-04-22T15:56:17.451 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:17 smithi016 ceph-mon[110740]: from='osd.2 [v2:172.21.15.16:6818/2085469945,v1:172.21.15.16:6819/2085469945]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T15:56:17.451 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:17 smithi016 ceph-mon[110740]: osdmap e62: 6 total, 5 up, 6 in 2024-04-22T15:56:17.451 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:17 smithi016 ceph-mon[110740]: from='osd.2 [v2:172.21.15.16:6818/2085469945,v1:172.21.15.16:6819/2085469945]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-22T15:56:17.452 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:17 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:17.452 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:17 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:17.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:17 smithi027 ceph-mon[83054]: pgmap v98: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 819 B/s rd, 1 op/s; 33/258 objects degraded (12.791%) 2024-04-22T15:56:17.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:17 smithi027 ceph-mon[83054]: from='osd.2 [v2:172.21.15.16:6818/2085469945,v1:172.21.15.16:6819/2085469945]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T15:56:17.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:17 smithi027 ceph-mon[83054]: osdmap e62: 6 total, 5 up, 6 in 2024-04-22T15:56:17.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:17 smithi027 ceph-mon[83054]: from='osd.2 [v2:172.21.15.16:6818/2085469945,v1:172.21.15.16:6819/2085469945]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-22T15:56:17.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:17 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:17.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:17 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:18.562 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:18 smithi016 ceph-mon[110740]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T15:56:18.562 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:18 smithi016 ceph-mon[110740]: osd.2 [v2:172.21.15.16:6818/2085469945,v1:172.21.15.16:6819/2085469945] boot 2024-04-22T15:56:18.562 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:18 smithi016 ceph-mon[110740]: osdmap e63: 6 total, 6 up, 6 in 2024-04-22T15:56:18.562 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:18 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:56:18.562 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:18 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:18.562 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:18 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:18.562 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:18 smithi016 ceph-mon[110740]: osdmap e64: 6 total, 6 up, 6 in 2024-04-22T15:56:18.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:18 smithi027 ceph-mon[83054]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T15:56:18.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:18 smithi027 ceph-mon[83054]: osd.2 [v2:172.21.15.16:6818/2085469945,v1:172.21.15.16:6819/2085469945] boot 2024-04-22T15:56:18.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:18 smithi027 ceph-mon[83054]: osdmap e63: 6 total, 6 up, 6 in 2024-04-22T15:56:18.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:18 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T15:56:18.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:18 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:18.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:18 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:18.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:18 smithi027 ceph-mon[83054]: osdmap e64: 6 total, 6 up, 6 in 2024-04-22T15:56:19.930 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:19 smithi016 ceph-mon[110740]: pgmap v101: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 33/258 objects degraded (12.791%) 2024-04-22T15:56:20.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:19 smithi027 ceph-mon[83054]: pgmap v101: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 33/258 objects degraded (12.791%) 2024-04-22T15:56:22.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:21 smithi027 ceph-mon[83054]: pgmap v103: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 33/258 objects degraded (12.791%) 2024-04-22T15:56:22.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:21 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:22.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:21 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:22.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:21 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:56:22.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:21 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:56:22.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:21 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:22.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:21 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:56:22.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:21 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:56:22.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:21 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:56:22.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:21 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:56:22.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:21 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-22T15:56:22.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:21 smithi016 ceph-mon[110740]: pgmap v103: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 33/258 objects degraded (12.791%) 2024-04-22T15:56:22.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:21 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:22.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:21 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:22.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:21 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:56:22.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:21 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:56:22.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:21 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:22.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:21 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:56:22.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:21 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:56:22.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:21 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:56:22.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:21 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:56:22.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:21 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-22T15:56:23.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:22 smithi027 ceph-mon[83054]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-22T15:56:23.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:22 smithi027 ceph-mon[83054]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-22T15:56:23.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:22 smithi016 ceph-mon[110740]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-22T15:56:23.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:22 smithi016 ceph-mon[110740]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-22T15:56:24.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:23 smithi027 ceph-mon[83054]: pgmap v104: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 682 B/s rd, 1 op/s 2024-04-22T15:56:24.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:23 smithi027 ceph-mon[83054]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 33/258 objects degraded (12.791%), 12 pgs degraded) 2024-04-22T15:56:24.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:23 smithi016 ceph-mon[110740]: pgmap v104: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 682 B/s rd, 1 op/s 2024-04-22T15:56:24.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:23 smithi016 ceph-mon[110740]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 33/258 objects degraded (12.791%), 12 pgs degraded) 2024-04-22T15:56:26.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:25 smithi027 ceph-mon[83054]: pgmap v105: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 516 B/s rd, 1 op/s 2024-04-22T15:56:26.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:25 smithi016 ceph-mon[110740]: pgmap v105: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 516 B/s rd, 1 op/s 2024-04-22T15:56:28.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:27 smithi027 ceph-mon[83054]: pgmap v106: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-22T15:56:28.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:27 smithi016 ceph-mon[110740]: pgmap v106: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-22T15:56:30.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:29 smithi027 ceph-mon[83054]: pgmap v107: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 921 B/s rd, 1 op/s 2024-04-22T15:56:30.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:29 smithi016 ceph-mon[110740]: pgmap v107: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 921 B/s rd, 1 op/s 2024-04-22T15:56:31.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.064+0000 7f269d796700 1 -- 172.21.15.16:0/4195642805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2698068580 msgr2=0x7f26980689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:31.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.064+0000 7f269d796700 1 --2- 172.21.15.16:0/4195642805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2698068580 0x7f26980689e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f268c009910 tx=0x7f268c009c20 comp rx=0 tx=0).stop 2024-04-22T15:56:31.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.064+0000 7f269d796700 1 -- 172.21.15.16:0/4195642805 shutdown_connections 2024-04-22T15:56:31.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.064+0000 7f269d796700 1 --2- 172.21.15.16:0/4195642805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2698068580 0x7f26980689e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.064+0000 7f269d796700 1 --2- 172.21.15.16:0/4195642805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2698106cd0 0x7f26981070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.064+0000 7f269d796700 1 -- 172.21.15.16:0/4195642805 >> 172.21.15.16:0/4195642805 conn(0x7f2698075670 msgr2=0x7f2698075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:31.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.064+0000 7f269d796700 1 -- 172.21.15.16:0/4195642805 shutdown_connections 2024-04-22T15:56:31.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.064+0000 7f269d796700 1 -- 172.21.15.16:0/4195642805 wait complete. 2024-04-22T15:56:31.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.065+0000 7f269d796700 1 Processor -- start 2024-04-22T15:56:31.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.065+0000 7f269d796700 1 -- start start 2024-04-22T15:56:31.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.065+0000 7f269d796700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2698068580 0x7f2698196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:31.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.065+0000 7f269d796700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2698106cd0 0x7f2698196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:31.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.065+0000 7f269d796700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2698197550 con 0x7f2698106cd0 2024-04-22T15:56:31.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.065+0000 7f269d796700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269819b360 con 0x7f2698068580 2024-04-22T15:56:31.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.065+0000 7f26967fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2698106cd0 0x7f2698196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:31.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.066+0000 7f26967fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2698106cd0 0x7f2698196ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54338/0 (socket says 172.21.15.16:54338) 2024-04-22T15:56:31.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.066+0000 7f26967fc700 1 -- 172.21.15.16:0/3049500020 learned_addr learned my addr 172.21.15.16:0/3049500020 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:56:31.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.066+0000 7f2696ffd700 1 --2- 172.21.15.16:0/3049500020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2698068580 0x7f2698196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:31.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.066+0000 7f26967fc700 1 -- 172.21.15.16:0/3049500020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2698068580 msgr2=0x7f2698196980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:31.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.066+0000 7f26967fc700 1 --2- 172.21.15.16:0/3049500020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2698068580 0x7f2698196980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.066+0000 7f26967fc700 1 -- 172.21.15.16:0/3049500020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268c0095f0 con 0x7f2698106cd0 2024-04-22T15:56:31.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.066+0000 7f26967fc700 1 --2- 172.21.15.16:0/3049500020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2698106cd0 0x7f2698196ec0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f268c006010 tx=0x7f268c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:31.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.066+0000 7f267ffff700 1 -- 172.21.15.16:0/3049500020 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f268c01d070 con 0x7f2698106cd0 2024-04-22T15:56:31.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.066+0000 7f269d796700 1 -- 172.21.15.16:0/3049500020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f269819b5e0 con 0x7f2698106cd0 2024-04-22T15:56:31.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.066+0000 7f267ffff700 1 -- 172.21.15.16:0/3049500020 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f268c004fb0 con 0x7f2698106cd0 2024-04-22T15:56:31.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.066+0000 7f267ffff700 1 -- 172.21.15.16:0/3049500020 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f268c022ea0 con 0x7f2698106cd0 2024-04-22T15:56:31.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.066+0000 7f269d796700 1 -- 172.21.15.16:0/3049500020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f269819bad0 con 0x7f2698106cd0 2024-04-22T15:56:31.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.067+0000 7f269d796700 1 -- 172.21.15.16:0/3049500020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26981091f0 con 0x7f2698106cd0 2024-04-22T15:56:31.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.068+0000 7f267ffff700 1 -- 172.21.15.16:0/3049500020 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f268c022470 con 0x7f2698106cd0 2024-04-22T15:56:31.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.068+0000 7f267ffff700 1 --2- 172.21.15.16:0/3049500020 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f268006ecc0 0x7f2680071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:31.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.068+0000 7f267ffff700 1 -- 172.21.15.16:0/3049500020 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f268c090340 con 0x7f2698106cd0 2024-04-22T15:56:31.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.068+0000 7f2696ffd700 1 --2- 172.21.15.16:0/3049500020 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f268006ecc0 0x7f2680071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:31.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.069+0000 7f2696ffd700 1 --2- 172.21.15.16:0/3049500020 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f268006ecc0 0x7f2680071180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2688006fd0 tx=0x7f2688008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:31.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.073+0000 7f267ffff700 1 -- 172.21.15.16:0/3049500020 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f268c05cd80 con 0x7f2698106cd0 2024-04-22T15:56:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.322+0000 7f269d796700 1 -- 172.21.15.16:0/3049500020 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2698197da0 con 0x7f268006ecc0 2024-04-22T15:56:31.325 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:31 smithi016 ceph-mon[110740]: pgmap v108: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 772 B/s rd, 1 op/s 2024-04-22T15:56:31.325 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:31 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:31.325 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:31 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:56:31.325 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:31 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.323+0000 7f267ffff700 1 -- 172.21.15.16:0/3049500020 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f2698197da0 con 0x7f268006ecc0 2024-04-22T15:56:31.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.326+0000 7f269d796700 1 -- 172.21.15.16:0/3049500020 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f268006ecc0 msgr2=0x7f2680071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:31.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.326+0000 7f269d796700 1 --2- 172.21.15.16:0/3049500020 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f268006ecc0 0x7f2680071180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2688006fd0 tx=0x7f2688008040 comp rx=0 tx=0).stop 2024-04-22T15:56:31.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.326+0000 7f269d796700 1 -- 172.21.15.16:0/3049500020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2698106cd0 msgr2=0x7f2698196ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:31.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.326+0000 7f269d796700 1 --2- 172.21.15.16:0/3049500020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2698106cd0 0x7f2698196ec0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f268c006010 tx=0x7f268c00be20 comp rx=0 tx=0).stop 2024-04-22T15:56:31.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.326+0000 7f269d796700 1 -- 172.21.15.16:0/3049500020 shutdown_connections 2024-04-22T15:56:31.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.326+0000 7f269d796700 1 --2- 172.21.15.16:0/3049500020 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f268006ecc0 0x7f2680071180 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.326+0000 7f269d796700 1 --2- 172.21.15.16:0/3049500020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2698068580 0x7f2698196980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.326+0000 7f269d796700 1 --2- 172.21.15.16:0/3049500020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2698106cd0 0x7f2698196ec0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.326+0000 7f269d796700 1 -- 172.21.15.16:0/3049500020 >> 172.21.15.16:0/3049500020 conn(0x7f2698075670 msgr2=0x7f26980fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:31.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.326+0000 7f269d796700 1 -- 172.21.15.16:0/3049500020 shutdown_connections 2024-04-22T15:56:31.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.326+0000 7f269d796700 1 -- 172.21.15.16:0/3049500020 wait complete. 2024-04-22T15:56:31.341 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:56:31.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.435+0000 7ffb45ba5700 1 -- 172.21.15.16:0/3721774847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb40068580 msgr2=0x7ffb400689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:31.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.435+0000 7ffb45ba5700 1 --2- 172.21.15.16:0/3721774847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb40068580 0x7ffb400689e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ffb34009910 tx=0x7ffb34009c20 comp rx=0 tx=0).stop 2024-04-22T15:56:31.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.435+0000 7ffb45ba5700 1 -- 172.21.15.16:0/3721774847 shutdown_connections 2024-04-22T15:56:31.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.435+0000 7ffb45ba5700 1 --2- 172.21.15.16:0/3721774847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb40068580 0x7ffb400689e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.435+0000 7ffb45ba5700 1 --2- 172.21.15.16:0/3721774847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb40106cd0 0x7ffb401070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.435+0000 7ffb45ba5700 1 -- 172.21.15.16:0/3721774847 >> 172.21.15.16:0/3721774847 conn(0x7ffb40075670 msgr2=0x7ffb40075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:31.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.435+0000 7ffb45ba5700 1 -- 172.21.15.16:0/3721774847 shutdown_connections 2024-04-22T15:56:31.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.435+0000 7ffb45ba5700 1 -- 172.21.15.16:0/3721774847 wait complete. 2024-04-22T15:56:31.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.435+0000 7ffb45ba5700 1 Processor -- start 2024-04-22T15:56:31.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.436+0000 7ffb45ba5700 1 -- start start 2024-04-22T15:56:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.436+0000 7ffb45ba5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb40068580 0x7ffb40196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.436+0000 7ffb45ba5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb40106cd0 0x7ffb40196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.436+0000 7ffb45ba5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb40197550 con 0x7ffb40068580 2024-04-22T15:56:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.436+0000 7ffb45ba5700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4019b360 con 0x7ffb40106cd0 2024-04-22T15:56:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.436+0000 7ffb3f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb40068580 0x7ffb40196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.436+0000 7ffb3f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb40068580 0x7ffb40196980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54364/0 (socket says 172.21.15.16:54364) 2024-04-22T15:56:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.436+0000 7ffb3f7fe700 1 -- 172.21.15.16:0/296330253 learned_addr learned my addr 172.21.15.16:0/296330253 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:56:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.436+0000 7ffb3effd700 1 --2- 172.21.15.16:0/296330253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb40106cd0 0x7ffb40196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.436+0000 7ffb3f7fe700 1 -- 172.21.15.16:0/296330253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb40106cd0 msgr2=0x7ffb40196ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.436+0000 7ffb3f7fe700 1 --2- 172.21.15.16:0/296330253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb40106cd0 0x7ffb40196ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.436+0000 7ffb3f7fe700 1 -- 172.21.15.16:0/296330253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb340095f0 con 0x7ffb40068580 2024-04-22T15:56:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.437+0000 7ffb3f7fe700 1 --2- 172.21.15.16:0/296330253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb40068580 0x7ffb40196980 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ffb3000b4b0 tx=0x7ffb3000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.437+0000 7ffb3cff9700 1 -- 172.21.15.16:0/296330253 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb30011800 con 0x7ffb40068580 2024-04-22T15:56:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.437+0000 7ffb45ba5700 1 -- 172.21.15.16:0/296330253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb4019b640 con 0x7ffb40068580 2024-04-22T15:56:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.437+0000 7ffb3cff9700 1 -- 172.21.15.16:0/296330253 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffb30011e20 con 0x7ffb40068580 2024-04-22T15:56:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.437+0000 7ffb3cff9700 1 -- 172.21.15.16:0/296330253 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb3000f2e0 con 0x7ffb40068580 2024-04-22T15:56:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.437+0000 7ffb45ba5700 1 -- 172.21.15.16:0/296330253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb4019bb90 con 0x7ffb40068580 2024-04-22T15:56:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.438+0000 7ffb45ba5700 1 -- 172.21.15.16:0/296330253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb401091d0 con 0x7ffb40068580 2024-04-22T15:56:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.438+0000 7ffb3cff9700 1 -- 172.21.15.16:0/296330253 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ffb30010380 con 0x7ffb40068580 2024-04-22T15:56:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.439+0000 7ffb3cff9700 1 --2- 172.21.15.16:0/296330253 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ffb2806ecc0 0x7ffb28071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.439+0000 7ffb3effd700 1 --2- 172.21.15.16:0/296330253 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ffb2806ecc0 0x7ffb28071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.439+0000 7ffb3cff9700 1 -- 172.21.15.16:0/296330253 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ffb3008e8d0 con 0x7ffb40068580 2024-04-22T15:56:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.440+0000 7ffb3effd700 1 --2- 172.21.15.16:0/296330253 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ffb2806ecc0 0x7ffb28071180 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffb40197fa0 tx=0x7ffb3401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:31.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.444+0000 7ffb3cff9700 1 -- 172.21.15.16:0/296330253 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ffb3005b260 con 0x7ffb40068580 2024-04-22T15:56:31.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:31 smithi027 ceph-mon[83054]: pgmap v108: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 772 B/s rd, 1 op/s 2024-04-22T15:56:31.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:31 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:31.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:31 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:56:31.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:31 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:31.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.676+0000 7ffb45ba5700 1 -- 172.21.15.16:0/296330253 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb40197d50 con 0x7ffb2806ecc0 2024-04-22T15:56:31.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.677+0000 7ffb3cff9700 1 -- 172.21.15.16:0/296330253 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7ffb40197d50 con 0x7ffb2806ecc0 2024-04-22T15:56:31.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.679+0000 7ffb45ba5700 1 -- 172.21.15.16:0/296330253 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ffb2806ecc0 msgr2=0x7ffb28071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:31.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.679+0000 7ffb45ba5700 1 --2- 172.21.15.16:0/296330253 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ffb2806ecc0 0x7ffb28071180 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffb40197fa0 tx=0x7ffb3401a040 comp rx=0 tx=0).stop 2024-04-22T15:56:31.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.679+0000 7ffb45ba5700 1 -- 172.21.15.16:0/296330253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb40068580 msgr2=0x7ffb40196980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:31.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.679+0000 7ffb45ba5700 1 --2- 172.21.15.16:0/296330253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb40068580 0x7ffb40196980 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ffb3000b4b0 tx=0x7ffb3000b7c0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.680+0000 7ffb45ba5700 1 -- 172.21.15.16:0/296330253 shutdown_connections 2024-04-22T15:56:31.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.680+0000 7ffb45ba5700 1 --2- 172.21.15.16:0/296330253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb40068580 0x7ffb40196980 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.680+0000 7ffb45ba5700 1 --2- 172.21.15.16:0/296330253 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ffb2806ecc0 0x7ffb28071180 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.680+0000 7ffb45ba5700 1 --2- 172.21.15.16:0/296330253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb40106cd0 0x7ffb40196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.680+0000 7ffb45ba5700 1 -- 172.21.15.16:0/296330253 >> 172.21.15.16:0/296330253 conn(0x7ffb40075670 msgr2=0x7ffb400fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:31.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.680+0000 7ffb45ba5700 1 -- 172.21.15.16:0/296330253 shutdown_connections 2024-04-22T15:56:31.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.680+0000 7ffb45ba5700 1 -- 172.21.15.16:0/296330253 wait complete. 2024-04-22T15:56:31.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.785+0000 7fad40dad700 1 -- 172.21.15.16:0/1729190009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad3c100ea0 msgr2=0x7fad3c101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:31.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.785+0000 7fad40dad700 1 --2- 172.21.15.16:0/1729190009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad3c100ea0 0x7fad3c101280 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fad2c009910 tx=0x7fad2c009c20 comp rx=0 tx=0).stop 2024-04-22T15:56:31.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.786+0000 7fad40dad700 1 -- 172.21.15.16:0/1729190009 shutdown_connections 2024-04-22T15:56:31.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.786+0000 7fad40dad700 1 --2- 172.21.15.16:0/1729190009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad3c101850 0x7fad3c1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.786+0000 7fad40dad700 1 --2- 172.21.15.16:0/1729190009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad3c100ea0 0x7fad3c101280 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.786+0000 7fad40dad700 1 -- 172.21.15.16:0/1729190009 >> 172.21.15.16:0/1729190009 conn(0x7fad3c0fc750 msgr2=0x7fad3c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:31.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.786+0000 7fad40dad700 1 -- 172.21.15.16:0/1729190009 shutdown_connections 2024-04-22T15:56:31.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.786+0000 7fad40dad700 1 -- 172.21.15.16:0/1729190009 wait complete. 2024-04-22T15:56:31.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.786+0000 7fad40dad700 1 Processor -- start 2024-04-22T15:56:31.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.787+0000 7fad40dad700 1 -- start start 2024-04-22T15:56:31.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.787+0000 7fad40dad700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad3c100ea0 0x7fad3c19c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:31.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.787+0000 7fad40dad700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad3c101850 0x7fad3c19cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:31.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.787+0000 7fad40dad700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c19d2c0 con 0x7fad3c100ea0 2024-04-22T15:56:31.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.787+0000 7fad40dad700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c196770 con 0x7fad3c101850 2024-04-22T15:56:31.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.787+0000 7fad3a59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad3c100ea0 0x7fad3c19c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:31.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.787+0000 7fad3a59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad3c100ea0 0x7fad3c19c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45526/0 (socket says 172.21.15.16:45526) 2024-04-22T15:56:31.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.787+0000 7fad3a59c700 1 -- 172.21.15.16:0/137751349 learned_addr learned my addr 172.21.15.16:0/137751349 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:56:31.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.787+0000 7fad39d9b700 1 --2- 172.21.15.16:0/137751349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad3c101850 0x7fad3c19cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:31.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.787+0000 7fad3a59c700 1 -- 172.21.15.16:0/137751349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad3c101850 msgr2=0x7fad3c19cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:31.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.787+0000 7fad3a59c700 1 --2- 172.21.15.16:0/137751349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad3c101850 0x7fad3c19cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:31.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.787+0000 7fad3a59c700 1 -- 172.21.15.16:0/137751349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c0095f0 con 0x7fad3c100ea0 2024-04-22T15:56:31.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.787+0000 7fad3a59c700 1 --2- 172.21.15.16:0/137751349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad3c100ea0 0x7fad3c19c6f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fad2c005f50 tx=0x7fad2c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:31.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.788+0000 7fad2b7fe700 1 -- 172.21.15.16:0/137751349 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad2c01d070 con 0x7fad3c100ea0 2024-04-22T15:56:31.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.788+0000 7fad40dad700 1 -- 172.21.15.16:0/137751349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad3c1969f0 con 0x7fad3c100ea0 2024-04-22T15:56:31.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.788+0000 7fad2b7fe700 1 -- 172.21.15.16:0/137751349 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fad2c00b610 con 0x7fad3c100ea0 2024-04-22T15:56:31.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.788+0000 7fad2b7fe700 1 -- 172.21.15.16:0/137751349 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad2c022470 con 0x7fad3c100ea0 2024-04-22T15:56:31.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.788+0000 7fad40dad700 1 -- 172.21.15.16:0/137751349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad3c196ee0 con 0x7fad3c100ea0 2024-04-22T15:56:31.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.789+0000 7fad40dad700 1 -- 172.21.15.16:0/137751349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad3c109240 con 0x7fad3c100ea0 2024-04-22T15:56:31.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.789+0000 7fad2b7fe700 1 -- 172.21.15.16:0/137751349 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fad2c003780 con 0x7fad3c100ea0 2024-04-22T15:56:31.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.790+0000 7fad2b7fe700 1 --2- 172.21.15.16:0/137751349 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fad2406ecc0 0x7fad24071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:31.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.790+0000 7fad2b7fe700 1 -- 172.21.15.16:0/137751349 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fad2c090090 con 0x7fad3c100ea0 2024-04-22T15:56:31.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.790+0000 7fad39d9b700 1 --2- 172.21.15.16:0/137751349 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fad2406ecc0 0x7fad24071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:31.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.790+0000 7fad39d9b700 1 --2- 172.21.15.16:0/137751349 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fad2406ecc0 0x7fad24071180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fad3c197be0 tx=0x7fad30009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:31.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:31.795+0000 7fad2b7fe700 1 -- 172.21.15.16:0/137751349 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fad2c05caa0 con 0x7fad3c100ea0 2024-04-22T15:56:32.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.014+0000 7fad40dad700 1 -- 172.21.15.16:0/137751349 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fad3c197150 con 0x7fad2406ecc0 2024-04-22T15:56:32.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.025+0000 7fad2b7fe700 1 -- 172.21.15.16:0/137751349 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fad3c197150 con 0x7fad2406ecc0 2024-04-22T15:56:32.028 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:56:32.028 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.smithi016 smithi016 *:9093,9094 running (4m) 15s ago 15m 25.6M - 0.25.0 c8568f914cd2 ea0e0217d6f9 2024-04-22T15:56:32.028 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi016 smithi016 running (15m) 15s ago 15m 8640k - 18.2.1 5be31c24972a f8b627d3f41e 2024-04-22T15:56:32.028 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi027 smithi027 running (14m) 111s ago 14m 8066k - 18.2.1 5be31c24972a f00fd40b4220 2024-04-22T15:56:32.028 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi016 smithi016 running (119s) 15s ago 15m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 1feb008449ed 2024-04-22T15:56:32.028 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi027 smithi027 running (114s) 111s ago 14m 6656k - 19.0.0-3223-g5883d494 3d983c26e278 c7f862c6da00 2024-04-22T15:56:32.028 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.smithi016 smithi016 *:3000 running (4m) 15s ago 15m 78.9M - 9.4.12 2bacad6d85d8 388a3c9c1b3f 2024-04-22T15:56:32.028 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.fkkejm smithi016 running (9m) 15s ago 9m 17.0M - 18.2.1 5be31c24972a 4153ac94cea0 2024-04-22T15:56:32.028 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.pgvphu smithi016 running (9m) 15s ago 9m 92.0M - 18.2.1 5be31c24972a 93dc072068a2 2024-04-22T15:56:32.028 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.dgqyes smithi027 running (9m) 111s ago 9m 19.2M - 18.2.1 5be31c24972a 41a9220430fd 2024-04-22T15:56:32.028 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.iobjds smithi027 running (9m) 111s ago 9m 149M - 18.2.1 5be31c24972a c1c8d6770f02 2024-04-22T15:56:32.028 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi016.rnsnnr smithi016 *:8443,9283,8765 running (6m) 15s ago 17m 505M - 19.0.0-3223-g5883d494 3d983c26e278 09b4fe285bc4 2024-04-22T15:56:32.029 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi027.ukdgzs smithi027 *:8443,9283,8765 running (6m) 111s ago 14m 450M - 19.0.0-3223-g5883d494 3d983c26e278 7548da9b5eb9 2024-04-22T15:56:32.029 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi016 smithi016 running (3m) 15s ago 17m 49.6M 2048M 19.0.0-3223-g5883d494 3d983c26e278 4ae529a67806 2024-04-22T15:56:32.029 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi027 smithi027 running (2m) 111s ago 14m 39.0M 2048M 19.0.0-3223-g5883d494 3d983c26e278 f29fcb888801 2024-04-22T15:56:32.029 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi016 smithi016 *:9100 running (5m) 15s ago 15m 16.1M - 1.5.0 0da6a335fe13 35e46e4d2f3f 2024-04-22T15:56:32.029 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi027 smithi027 *:9100 running (5m) 111s ago 14m 15.5M - 1.5.0 0da6a335fe13 e682a7672e9b 2024-04-22T15:56:32.029 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (93s) 15s ago 13m 88.0M 1217M 19.0.0-3223-g5883d494 3d983c26e278 79217eb383a2 2024-04-22T15:56:32.029 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (57s) 15s ago 13m 89.0M 1217M 19.0.0-3223-g5883d494 3d983c26e278 ec1a45b3c53e 2024-04-22T15:56:32.029 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (21s) 15s ago 12m 12.6M 1217M 19.0.0-3223-g5883d494 3d983c26e278 48bb459fd021 2024-04-22T15:56:32.029 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi027 running (12m) 111s ago 12m 250M 2240M 18.2.1 5be31c24972a cdf9b7aaf865 2024-04-22T15:56:32.029 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (11m) 111s ago 11m 206M 2240M 18.2.1 5be31c24972a 629397d49c76 2024-04-22T15:56:32.029 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi027 running (11m) 111s ago 11m 193M 2240M 18.2.1 5be31c24972a 23adcf675b04 2024-04-22T15:56:32.029 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.smithi016 smithi016 *:9095 running (5m) 15s ago 14m 73.8M - 2.43.0 a07b618ecd1d 373eaadee82d 2024-04-22T15:56:32.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.027+0000 7fad40dad700 1 -- 172.21.15.16:0/137751349 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fad2406ecc0 msgr2=0x7fad24071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:32.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.027+0000 7fad40dad700 1 --2- 172.21.15.16:0/137751349 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fad2406ecc0 0x7fad24071180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fad3c197be0 tx=0x7fad30009260 comp rx=0 tx=0).stop 2024-04-22T15:56:32.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.027+0000 7fad40dad700 1 -- 172.21.15.16:0/137751349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad3c100ea0 msgr2=0x7fad3c19c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:32.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.027+0000 7fad40dad700 1 --2- 172.21.15.16:0/137751349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad3c100ea0 0x7fad3c19c6f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fad2c005f50 tx=0x7fad2c00f690 comp rx=0 tx=0).stop 2024-04-22T15:56:32.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.027+0000 7fad40dad700 1 -- 172.21.15.16:0/137751349 shutdown_connections 2024-04-22T15:56:32.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.027+0000 7fad40dad700 1 --2- 172.21.15.16:0/137751349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad3c100ea0 0x7fad3c19c6f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:32.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.027+0000 7fad40dad700 1 --2- 172.21.15.16:0/137751349 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fad2406ecc0 0x7fad24071180 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:32.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.027+0000 7fad40dad700 1 --2- 172.21.15.16:0/137751349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad3c101850 0x7fad3c19cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:32.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.027+0000 7fad40dad700 1 -- 172.21.15.16:0/137751349 >> 172.21.15.16:0/137751349 conn(0x7fad3c0fc750 msgr2=0x7fad3c0fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:32.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.027+0000 7fad40dad700 1 -- 172.21.15.16:0/137751349 shutdown_connections 2024-04-22T15:56:32.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.027+0000 7fad40dad700 1 -- 172.21.15.16:0/137751349 wait complete. 2024-04-22T15:56:32.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.133+0000 7fe07eaf1700 1 -- 172.21.15.16:0/2996760746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0780fff50 msgr2=0x7fe078105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:32.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.133+0000 7fe07eaf1700 1 --2- 172.21.15.16:0/2996760746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0780fff50 0x7fe078105840 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe064009910 tx=0x7fe064009c20 comp rx=0 tx=0).stop 2024-04-22T15:56:32.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.133+0000 7fe07eaf1700 1 -- 172.21.15.16:0/2996760746 shutdown_connections 2024-04-22T15:56:32.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.133+0000 7fe07eaf1700 1 --2- 172.21.15.16:0/2996760746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0780fff50 0x7fe078105840 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:32.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.133+0000 7fe07eaf1700 1 --2- 172.21.15.16:0/2996760746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0780ff5a0 0x7fe0780ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:32.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.133+0000 7fe07eaf1700 1 -- 172.21.15.16:0/2996760746 >> 172.21.15.16:0/2996760746 conn(0x7fe078074e30 msgr2=0x7fe078075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:32.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.133+0000 7fe07eaf1700 1 -- 172.21.15.16:0/2996760746 shutdown_connections 2024-04-22T15:56:32.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.133+0000 7fe07eaf1700 1 -- 172.21.15.16:0/2996760746 wait complete. 2024-04-22T15:56:32.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.134+0000 7fe07eaf1700 1 Processor -- start 2024-04-22T15:56:32.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.134+0000 7fe07eaf1700 1 -- start start 2024-04-22T15:56:32.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.134+0000 7fe07eaf1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0780ff5a0 0x7fe078100520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:32.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.134+0000 7fe07eaf1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0780fff50 0x7fe078100a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:32.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.134+0000 7fe07eaf1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe078100fa0 con 0x7fe0780fff50 2024-04-22T15:56:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.134+0000 7fe07eaf1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0781010e0 con 0x7fe0780ff5a0 2024-04-22T15:56:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.134+0000 7fe077fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0780fff50 0x7fe078100a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.134+0000 7fe077fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0780fff50 0x7fe078100a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45546/0 (socket says 172.21.15.16:45546) 2024-04-22T15:56:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.134+0000 7fe077fff700 1 -- 172.21.15.16:0/1575224088 learned_addr learned my addr 172.21.15.16:0/1575224088 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:56:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.134+0000 7fe07c88d700 1 --2- 172.21.15.16:0/1575224088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0780ff5a0 0x7fe078100520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.134+0000 7fe077fff700 1 -- 172.21.15.16:0/1575224088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0780ff5a0 msgr2=0x7fe078100520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.134+0000 7fe077fff700 1 --2- 172.21.15.16:0/1575224088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0780ff5a0 0x7fe078100520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.134+0000 7fe077fff700 1 -- 172.21.15.16:0/1575224088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0640095f0 con 0x7fe0780fff50 2024-04-22T15:56:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.134+0000 7fe077fff700 1 --2- 172.21.15.16:0/1575224088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0780fff50 0x7fe078100a60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe0640092e0 tx=0x7fe06400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.135+0000 7fe075ffb700 1 -- 172.21.15.16:0/1575224088 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe06401d070 con 0x7fe0780fff50 2024-04-22T15:56:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.135+0000 7fe075ffb700 1 -- 172.21.15.16:0/1575224088 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe06400f460 con 0x7fe0780fff50 2024-04-22T15:56:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.135+0000 7fe07eaf1700 1 -- 172.21.15.16:0/1575224088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe078101360 con 0x7fe0780fff50 2024-04-22T15:56:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.135+0000 7fe075ffb700 1 -- 172.21.15.16:0/1575224088 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0640175d0 con 0x7fe0780fff50 2024-04-22T15:56:32.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.135+0000 7fe07eaf1700 1 -- 172.21.15.16:0/1575224088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0781a2dc0 con 0x7fe0780fff50 2024-04-22T15:56:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.136+0000 7fe07eaf1700 1 -- 172.21.15.16:0/1575224088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe078109200 con 0x7fe0780fff50 2024-04-22T15:56:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.136+0000 7fe075ffb700 1 -- 172.21.15.16:0/1575224088 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe06400f5d0 con 0x7fe0780fff50 2024-04-22T15:56:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.137+0000 7fe075ffb700 1 --2- 172.21.15.16:0/1575224088 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe06806ecc0 0x7fe068071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:32.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.137+0000 7fe075ffb700 1 -- 172.21.15.16:0/1575224088 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fe0640902f0 con 0x7fe0780fff50 2024-04-22T15:56:32.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.137+0000 7fe07c88d700 1 --2- 172.21.15.16:0/1575224088 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe06806ecc0 0x7fe068071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:32.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.137+0000 7fe07c88d700 1 --2- 172.21.15.16:0/1575224088 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe06806ecc0 0x7fe068071180 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe06c005950 tx=0x7fe06c00a300 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:32.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.142+0000 7fe075ffb700 1 -- 172.21.15.16:0/1575224088 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe0640590b0 con 0x7fe0780fff50 2024-04-22T15:56:32.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.447+0000 7fe07eaf1700 1 -- 172.21.15.16:0/1575224088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe07804f2e0 con 0x7fe0780fff50 2024-04-22T15:56:32.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.447+0000 7fe075ffb700 1 -- 172.21.15.16:0/1575224088 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7fe06405c770 con 0x7fe0780fff50 2024-04-22T15:56:32.450 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:56:32.450 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-22T15:56:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:56:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:56:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-22T15:56:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:56:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:56:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-22T15:56:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 3, 2024-04-22T15:56:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 3 2024-04-22T15:56:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:56:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-22T15:56:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T15:56:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:56:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-22T15:56:32.452 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 7, 2024-04-22T15:56:32.452 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 7 2024-04-22T15:56:32.452 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-22T15:56:32.452 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:56:32.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.449+0000 7fe07eaf1700 1 -- 172.21.15.16:0/1575224088 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe06806ecc0 msgr2=0x7fe068071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:32.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.449+0000 7fe07eaf1700 1 --2- 172.21.15.16:0/1575224088 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe06806ecc0 0x7fe068071180 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe06c005950 tx=0x7fe06c00a300 comp rx=0 tx=0).stop 2024-04-22T15:56:32.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.449+0000 7fe07eaf1700 1 -- 172.21.15.16:0/1575224088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0780fff50 msgr2=0x7fe078100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:32.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.449+0000 7fe07eaf1700 1 --2- 172.21.15.16:0/1575224088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0780fff50 0x7fe078100a60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe0640092e0 tx=0x7fe06400bd70 comp rx=0 tx=0).stop 2024-04-22T15:56:32.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.449+0000 7fe07eaf1700 1 -- 172.21.15.16:0/1575224088 shutdown_connections 2024-04-22T15:56:32.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.449+0000 7fe07eaf1700 1 --2- 172.21.15.16:0/1575224088 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe06806ecc0 0x7fe068071180 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:32.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.449+0000 7fe07eaf1700 1 --2- 172.21.15.16:0/1575224088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0780ff5a0 0x7fe078100520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:32.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.449+0000 7fe07eaf1700 1 --2- 172.21.15.16:0/1575224088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0780fff50 0x7fe078100a60 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:32.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.449+0000 7fe07eaf1700 1 -- 172.21.15.16:0/1575224088 >> 172.21.15.16:0/1575224088 conn(0x7fe078074e30 msgr2=0x7fe0780fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:32.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.450+0000 7fe07eaf1700 1 -- 172.21.15.16:0/1575224088 shutdown_connections 2024-04-22T15:56:32.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.450+0000 7fe07eaf1700 1 -- 172.21.15.16:0/1575224088 wait complete. 2024-04-22T15:56:32.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.557+0000 7fda0533a700 1 -- 172.21.15.16:0/3957015758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda00101ea0 msgr2=0x7fda00102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:32.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.557+0000 7fda0533a700 1 --2- 172.21.15.16:0/3957015758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda00101ea0 0x7fda00102280 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0009910 tx=0x7fd9f0009c20 comp rx=0 tx=0).stop 2024-04-22T15:56:32.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.557+0000 7fda0533a700 1 -- 172.21.15.16:0/3957015758 shutdown_connections 2024-04-22T15:56:32.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.557+0000 7fda0533a700 1 --2- 172.21.15.16:0/3957015758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda001027c0 0x7fda0010acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:32.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.557+0000 7fda0533a700 1 --2- 172.21.15.16:0/3957015758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda00101ea0 0x7fda00102280 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:32.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.557+0000 7fda0533a700 1 -- 172.21.15.16:0/3957015758 >> 172.21.15.16:0/3957015758 conn(0x7fda000fb650 msgr2=0x7fda000fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:32.561 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:32 smithi016 ceph-mon[110740]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:56:32.561 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:32 smithi016 ceph-mon[110740]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:56:32.561 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:32 smithi016 ceph-mon[110740]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:56:32.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.557+0000 7fda0533a700 1 -- 172.21.15.16:0/3957015758 shutdown_connections 2024-04-22T15:56:32.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.557+0000 7fda0533a700 1 -- 172.21.15.16:0/3957015758 wait complete. 2024-04-22T15:56:32.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.558+0000 7fda0533a700 1 Processor -- start 2024-04-22T15:56:32.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.558+0000 7fda0533a700 1 -- start start 2024-04-22T15:56:32.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.558+0000 7fda0533a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda00101ea0 0x7fda0019e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:32.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.558+0000 7fda0533a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda001027c0 0x7fda0019edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:32.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.558+0000 7fda0533a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0019f440 con 0x7fda001027c0 2024-04-22T15:56:32.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.558+0000 7fda0533a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda001988f0 con 0x7fda00101ea0 2024-04-22T15:56:32.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.558+0000 7fd9fe7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda001027c0 0x7fda0019edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:32.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.558+0000 7fd9fe7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda001027c0 0x7fda0019edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45556/0 (socket says 172.21.15.16:45556) 2024-04-22T15:56:32.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.558+0000 7fd9fe7fc700 1 -- 172.21.15.16:0/2550736055 learned_addr learned my addr 172.21.15.16:0/2550736055 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:56:32.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.558+0000 7fd9feffd700 1 --2- 172.21.15.16:0/2550736055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda00101ea0 0x7fda0019e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:32.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.559+0000 7fd9fe7fc700 1 -- 172.21.15.16:0/2550736055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda00101ea0 msgr2=0x7fda0019e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:32.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.559+0000 7fd9fe7fc700 1 --2- 172.21.15.16:0/2550736055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda00101ea0 0x7fda0019e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:32.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.559+0000 7fd9fe7fc700 1 -- 172.21.15.16:0/2550736055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f00095f0 con 0x7fda001027c0 2024-04-22T15:56:32.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.559+0000 7fd9feffd700 1 --2- 172.21.15.16:0/2550736055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda00101ea0 0x7fda0019e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:56:32.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.559+0000 7fd9fe7fc700 1 --2- 172.21.15.16:0/2550736055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda001027c0 0x7fda0019edb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd9f400c8f0 tx=0x7fd9f400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:32.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.559+0000 7fd9e7fff700 1 -- 172.21.15.16:0/2550736055 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9f4007a50 con 0x7fda001027c0 2024-04-22T15:56:32.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.559+0000 7fd9e7fff700 1 -- 172.21.15.16:0/2550736055 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd9f4007bb0 con 0x7fda001027c0 2024-04-22T15:56:32.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.559+0000 7fda0533a700 1 -- 172.21.15.16:0/2550736055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda00198bd0 con 0x7fda001027c0 2024-04-22T15:56:32.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.559+0000 7fd9e7fff700 1 -- 172.21.15.16:0/2550736055 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9f4018630 con 0x7fda001027c0 2024-04-22T15:56:32.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.559+0000 7fda0533a700 1 -- 172.21.15.16:0/2550736055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda00199120 con 0x7fda001027c0 2024-04-22T15:56:32.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.560+0000 7fda0533a700 1 -- 172.21.15.16:0/2550736055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda0010a7b0 con 0x7fda001027c0 2024-04-22T15:56:32.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.561+0000 7fd9e7fff700 1 -- 172.21.15.16:0/2550736055 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd9f4007d20 con 0x7fda001027c0 2024-04-22T15:56:32.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.561+0000 7fd9e7fff700 1 --2- 172.21.15.16:0/2550736055 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd9e806ecc0 0x7fd9e8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:32.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.561+0000 7fd9e7fff700 1 -- 172.21.15.16:0/2550736055 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd9f408eda0 con 0x7fda001027c0 2024-04-22T15:56:32.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.561+0000 7fd9feffd700 1 --2- 172.21.15.16:0/2550736055 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd9e806ecc0 0x7fd9e8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:32.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.562+0000 7fd9feffd700 1 --2- 172.21.15.16:0/2550736055 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd9e806ecc0 0x7fd9e8071180 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0005100 tx=0x7fd9f001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:32.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.566+0000 7fd9e7fff700 1 -- 172.21.15.16:0/2550736055 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd9f405b730 con 0x7fda001027c0 2024-04-22T15:56:32.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:32 smithi027 ceph-mon[83054]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:56:32.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:32 smithi027 ceph-mon[83054]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:56:32.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:32 smithi027 ceph-mon[83054]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:56:32.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.821+0000 7fda0533a700 1 -- 172.21.15.16:0/2550736055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fda001993a0 con 0x7fda001027c0 2024-04-22T15:56:32.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.822+0000 7fd9e7fff700 1 -- 172.21.15.16:0/2550736055 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1902 (secure 0 0 0) 0x7fd9f405b2c0 con 0x7fda001027c0 2024-04-22T15:56:32.825 INFO:teuthology.orchestra.run.smithi016.stdout:e16 2024-04-22T15:56:32.825 INFO:teuthology.orchestra.run.smithi016.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:56:32.825 INFO:teuthology.orchestra.run.smithi016.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:56:32.825 INFO:teuthology.orchestra.run.smithi016.stdout:legacy client fscid: 1 2024-04-22T15:56:32.825 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:56:32.825 INFO:teuthology.orchestra.run.smithi016.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:56:32.825 INFO:teuthology.orchestra.run.smithi016.stdout:fs_name cephfs 2024-04-22T15:56:32.825 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 16 2024-04-22T15:56:32.826 INFO:teuthology.orchestra.run.smithi016.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T15:56:32.826 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-04-22T15:46:27.774571+0000 2024-04-22T15:56:32.826 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-04-22T15:46:47.381709+0000 2024-04-22T15:56:32.826 INFO:teuthology.orchestra.run.smithi016.stdout:tableserver 0 2024-04-22T15:56:32.826 INFO:teuthology.orchestra.run.smithi016.stdout:root 0 2024-04-22T15:56:32.826 INFO:teuthology.orchestra.run.smithi016.stdout:session_timeout 60 2024-04-22T15:56:32.826 INFO:teuthology.orchestra.run.smithi016.stdout:session_autoclose 300 2024-04-22T15:56:32.826 INFO:teuthology.orchestra.run.smithi016.stdout:max_file_size 1099511627776 2024-04-22T15:56:32.826 INFO:teuthology.orchestra.run.smithi016.stdout:max_xattr_size 65536 2024-04-22T15:56:32.826 INFO:teuthology.orchestra.run.smithi016.stdout:required_client_features {} 2024-04-22T15:56:32.826 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure 0 2024-04-22T15:56:32.826 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure_osd_epoch 46 2024-04-22T15:56:32.826 INFO:teuthology.orchestra.run.smithi016.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:56:32.826 INFO:teuthology.orchestra.run.smithi016.stdout:max_mds 1 2024-04-22T15:56:32.826 INFO:teuthology.orchestra.run.smithi016.stdout:in 0 2024-04-22T15:56:32.827 INFO:teuthology.orchestra.run.smithi016.stdout:up {0=24279} 2024-04-22T15:56:32.827 INFO:teuthology.orchestra.run.smithi016.stdout:failed 2024-04-22T15:56:32.827 INFO:teuthology.orchestra.run.smithi016.stdout:damaged 2024-04-22T15:56:32.827 INFO:teuthology.orchestra.run.smithi016.stdout:stopped 2024-04-22T15:56:32.827 INFO:teuthology.orchestra.run.smithi016.stdout:data_pools [3] 2024-04-22T15:56:32.827 INFO:teuthology.orchestra.run.smithi016.stdout:metadata_pool 2 2024-04-22T15:56:32.827 INFO:teuthology.orchestra.run.smithi016.stdout:inline_data enabled 2024-04-22T15:56:32.827 INFO:teuthology.orchestra.run.smithi016.stdout:balancer 2024-04-22T15:56:32.827 INFO:teuthology.orchestra.run.smithi016.stdout:bal_rank_mask -1 2024-04-22T15:56:32.827 INFO:teuthology.orchestra.run.smithi016.stdout:standby_count_wanted 1 2024-04-22T15:56:32.827 INFO:teuthology.orchestra.run.smithi016.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:56:32.827 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.iobjds{0:24279} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.27:6824/4216277609,v1:172.21.15.27:6825/4216277609] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:56:32.827 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.pgvphu{0:14494} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.16:6828/1006073414,v1:172.21.15.16:6829/1006073414] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:56:32.827 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:56:32.827 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:56:32.828 INFO:teuthology.orchestra.run.smithi016.stdout:Standby daemons: 2024-04-22T15:56:32.828 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:56:32.828 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.fkkejm{-1:14504} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.16:6826/3484907755,v1:172.21.15.16:6827/3484907755] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:56:32.828 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.dgqyes{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.27:6826/476026355,v1:172.21.15.27:6827/476026355] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:56:32.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.824+0000 7fda0533a700 1 -- 172.21.15.16:0/2550736055 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd9e806ecc0 msgr2=0x7fd9e8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:32.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.824+0000 7fda0533a700 1 --2- 172.21.15.16:0/2550736055 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd9e806ecc0 0x7fd9e8071180 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0005100 tx=0x7fd9f001a040 comp rx=0 tx=0).stop 2024-04-22T15:56:32.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.824+0000 7fda0533a700 1 -- 172.21.15.16:0/2550736055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda001027c0 msgr2=0x7fda0019edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:32.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.824+0000 7fda0533a700 1 --2- 172.21.15.16:0/2550736055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda001027c0 0x7fda0019edb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd9f400c8f0 tx=0x7fd9f400ccb0 comp rx=0 tx=0).stop 2024-04-22T15:56:32.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.824+0000 7fda0533a700 1 -- 172.21.15.16:0/2550736055 shutdown_connections 2024-04-22T15:56:32.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.824+0000 7fda0533a700 1 --2- 172.21.15.16:0/2550736055 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd9e806ecc0 0x7fd9e8071180 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:32.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.824+0000 7fda0533a700 1 --2- 172.21.15.16:0/2550736055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda00101ea0 0x7fda0019e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:32.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.824+0000 7fda0533a700 1 --2- 172.21.15.16:0/2550736055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda001027c0 0x7fda0019edb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:32.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.824+0000 7fda0533a700 1 -- 172.21.15.16:0/2550736055 >> 172.21.15.16:0/2550736055 conn(0x7fda000fb650 msgr2=0x7fda001002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:32.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.824+0000 7fda0533a700 1 -- 172.21.15.16:0/2550736055 shutdown_connections 2024-04-22T15:56:32.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.824+0000 7fda0533a700 1 -- 172.21.15.16:0/2550736055 wait complete. 2024-04-22T15:56:32.829 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 16 2024-04-22T15:56:32.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.932+0000 7f0bd9f41700 1 -- 172.21.15.16:0/4198521667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd40ffda0 msgr2=0x7f0bd4100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:32.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.932+0000 7f0bd9f41700 1 --2- 172.21.15.16:0/4198521667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd40ffda0 0x7f0bd4100180 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8009910 tx=0x7f0bc8009c20 comp rx=0 tx=0).stop 2024-04-22T15:56:32.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.932+0000 7f0bd9f41700 1 -- 172.21.15.16:0/4198521667 shutdown_connections 2024-04-22T15:56:32.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.932+0000 7f0bd9f41700 1 --2- 172.21.15.16:0/4198521667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bd4100750 0x7f0bd41047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:32.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.932+0000 7f0bd9f41700 1 --2- 172.21.15.16:0/4198521667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd40ffda0 0x7f0bd4100180 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:32.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.932+0000 7f0bd9f41700 1 -- 172.21.15.16:0/4198521667 >> 172.21.15.16:0/4198521667 conn(0x7f0bd40fb650 msgr2=0x7f0bd40fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:32.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.933+0000 7f0bd9f41700 1 -- 172.21.15.16:0/4198521667 shutdown_connections 2024-04-22T15:56:32.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.933+0000 7f0bd9f41700 1 -- 172.21.15.16:0/4198521667 wait complete. 2024-04-22T15:56:32.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.933+0000 7f0bd9f41700 1 Processor -- start 2024-04-22T15:56:32.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.933+0000 7f0bd9f41700 1 -- start start 2024-04-22T15:56:32.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.933+0000 7f0bd9f41700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bd40ffda0 0x7f0bd410ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:32.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.934+0000 7f0bd9f41700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd4100750 0x7f0bd410b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:32.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.934+0000 7f0bd9f41700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd410b590 con 0x7f0bd4100750 2024-04-22T15:56:32.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.934+0000 7f0bd9f41700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd410b6d0 con 0x7f0bd40ffda0 2024-04-22T15:56:32.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.934+0000 7f0bd2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd4100750 0x7f0bd410b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:32.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.934+0000 7f0bd2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd4100750 0x7f0bd410b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45570/0 (socket says 172.21.15.16:45570) 2024-04-22T15:56:32.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.934+0000 7f0bd2ffd700 1 -- 172.21.15.16:0/1341542614 learned_addr learned my addr 172.21.15.16:0/1341542614 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:56:32.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.934+0000 7f0bd37fe700 1 --2- 172.21.15.16:0/1341542614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bd40ffda0 0x7f0bd410ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:32.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.934+0000 7f0bd2ffd700 1 -- 172.21.15.16:0/1341542614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bd40ffda0 msgr2=0x7f0bd410ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:32.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.934+0000 7f0bd2ffd700 1 --2- 172.21.15.16:0/1341542614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bd40ffda0 0x7f0bd410ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:32.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.934+0000 7f0bd2ffd700 1 -- 172.21.15.16:0/1341542614 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc80095f0 con 0x7f0bd4100750 2024-04-22T15:56:32.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.934+0000 7f0bd2ffd700 1 --2- 172.21.15.16:0/1341542614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd4100750 0x7f0bd410b050 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400d900 tx=0x7f0bc400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:32.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.934+0000 7f0bd0ff9700 1 -- 172.21.15.16:0/1341542614 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bc4009900 con 0x7f0bd4100750 2024-04-22T15:56:32.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.934+0000 7f0bd0ff9700 1 -- 172.21.15.16:0/1341542614 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0bc4010460 con 0x7f0bd4100750 2024-04-22T15:56:32.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.934+0000 7f0bd9f41700 1 -- 172.21.15.16:0/1341542614 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd41a2a60 con 0x7f0bd4100750 2024-04-22T15:56:32.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.934+0000 7f0bd0ff9700 1 -- 172.21.15.16:0/1341542614 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bc400f5f0 con 0x7f0bd4100750 2024-04-22T15:56:32.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.935+0000 7f0bd9f41700 1 -- 172.21.15.16:0/1341542614 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd41a2e70 con 0x7f0bd4100750 2024-04-22T15:56:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.936+0000 7f0bd9f41700 1 -- 172.21.15.16:0/1341542614 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd4108140 con 0x7f0bd4100750 2024-04-22T15:56:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.936+0000 7f0bd0ff9700 1 -- 172.21.15.16:0/1341542614 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0bc40105d0 con 0x7f0bd4100750 2024-04-22T15:56:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.936+0000 7f0bd0ff9700 1 --2- 172.21.15.16:0/1341542614 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0bbc06ecc0 0x7f0bbc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.937+0000 7f0bd0ff9700 1 -- 172.21.15.16:0/1341542614 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f0bc408ec00 con 0x7f0bd4100750 2024-04-22T15:56:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.937+0000 7f0bd37fe700 1 --2- 172.21.15.16:0/1341542614 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0bbc06ecc0 0x7f0bbc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.937+0000 7f0bd37fe700 1 --2- 172.21.15.16:0/1341542614 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0bbc06ecc0 0x7f0bbc071180 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0bc800b5c0 tx=0x7f0bc801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:32.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:32.941+0000 7f0bd0ff9700 1 -- 172.21.15.16:0/1341542614 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0bc4057a60 con 0x7f0bd4100750 2024-04-22T15:56:33.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.172+0000 7f0bd9f41700 1 -- 172.21.15.16:0/1341542614 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0bd40611d0 con 0x7f0bbc06ecc0 2024-04-22T15:56:33.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.173+0000 7f0bd0ff9700 1 -- 172.21.15.16:0/1341542614 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f0bd40611d0 con 0x7f0bbc06ecc0 2024-04-22T15:56:33.176 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:56:33.177 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:56:33.177 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-22T15:56:33.177 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:56:33.177 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-22T15:56:33.177 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-04-22T15:56:33.177 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr", 2024-04-22T15:56:33.177 INFO:teuthology.orchestra.run.smithi016.stdout: "mon" 2024-04-22T15:56:33.177 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-22T15:56:33.177 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "9/23 daemons upgraded", 2024-04-22T15:56:33.177 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading osd daemons", 2024-04-22T15:56:33.177 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-22T15:56:33.177 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:56:33.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.176+0000 7f0bd9f41700 1 -- 172.21.15.16:0/1341542614 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0bbc06ecc0 msgr2=0x7f0bbc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:33.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.176+0000 7f0bd9f41700 1 --2- 172.21.15.16:0/1341542614 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0bbc06ecc0 0x7f0bbc071180 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0bc800b5c0 tx=0x7f0bc801a040 comp rx=0 tx=0).stop 2024-04-22T15:56:33.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.176+0000 7f0bd9f41700 1 -- 172.21.15.16:0/1341542614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd4100750 msgr2=0x7f0bd410b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:33.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.176+0000 7f0bd9f41700 1 --2- 172.21.15.16:0/1341542614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd4100750 0x7f0bd410b050 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400d900 tx=0x7f0bc400dcc0 comp rx=0 tx=0).stop 2024-04-22T15:56:33.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.176+0000 7f0bd9f41700 1 -- 172.21.15.16:0/1341542614 shutdown_connections 2024-04-22T15:56:33.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.176+0000 7f0bd9f41700 1 --2- 172.21.15.16:0/1341542614 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0bbc06ecc0 0x7f0bbc071180 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:33.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.176+0000 7f0bd9f41700 1 --2- 172.21.15.16:0/1341542614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bd40ffda0 0x7f0bd410ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:33.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.176+0000 7f0bd9f41700 1 --2- 172.21.15.16:0/1341542614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd4100750 0x7f0bd410b050 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:33.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.176+0000 7f0bd9f41700 1 -- 172.21.15.16:0/1341542614 >> 172.21.15.16:0/1341542614 conn(0x7f0bd40fb650 msgr2=0x7f0bd40fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:33.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.176+0000 7f0bd9f41700 1 -- 172.21.15.16:0/1341542614 shutdown_connections 2024-04-22T15:56:33.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.176+0000 7f0bd9f41700 1 -- 172.21.15.16:0/1341542614 wait complete. 2024-04-22T15:56:33.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.284+0000 7f059dc34700 1 -- 172.21.15.16:0/878466000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05980ff340 msgr2=0x7f05980ff7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:33.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.284+0000 7f059dc34700 1 --2- 172.21.15.16:0/878466000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05980ff340 0x7f05980ff7a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f058c009910 tx=0x7f058c009c20 comp rx=0 tx=0).stop 2024-04-22T15:56:33.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.284+0000 7f059dc34700 1 -- 172.21.15.16:0/878466000 shutdown_connections 2024-04-22T15:56:33.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.284+0000 7f059dc34700 1 --2- 172.21.15.16:0/878466000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05980ff340 0x7f05980ff7a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:33.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.284+0000 7f059dc34700 1 --2- 172.21.15.16:0/878466000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0598105350 0x7f0598105730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:33.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.284+0000 7f059dc34700 1 -- 172.21.15.16:0/878466000 >> 172.21.15.16:0/878466000 conn(0x7f0598074e30 msgr2=0x7f0598075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:33.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.284+0000 7f059dc34700 1 -- 172.21.15.16:0/878466000 shutdown_connections 2024-04-22T15:56:33.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.284+0000 7f059dc34700 1 -- 172.21.15.16:0/878466000 wait complete. 2024-04-22T15:56:33.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.284+0000 7f059dc34700 1 Processor -- start 2024-04-22T15:56:33.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.285+0000 7f059dc34700 1 -- start start 2024-04-22T15:56:33.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.285+0000 7f059dc34700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05980ff340 0x7f059819b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:33.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.285+0000 7f059dc34700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0598105350 0x7f059819bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:33.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.285+0000 7f059dc34700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0598195600 con 0x7f0598105350 2024-04-22T15:56:33.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.285+0000 7f059dc34700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0598195770 con 0x7f05980ff340 2024-04-22T15:56:33.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.285+0000 7f0596ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0598105350 0x7f059819bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:33.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.285+0000 7f0596ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0598105350 0x7f059819bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45584/0 (socket says 172.21.15.16:45584) 2024-04-22T15:56:33.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.285+0000 7f0596ffd700 1 -- 172.21.15.16:0/2982656254 learned_addr learned my addr 172.21.15.16:0/2982656254 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:56:33.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.285+0000 7f05977fe700 1 --2- 172.21.15.16:0/2982656254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05980ff340 0x7f059819b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:33.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.285+0000 7f0596ffd700 1 -- 172.21.15.16:0/2982656254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05980ff340 msgr2=0x7f059819b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:33.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.285+0000 7f0596ffd700 1 --2- 172.21.15.16:0/2982656254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05980ff340 0x7f059819b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:33.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.285+0000 7f0596ffd700 1 -- 172.21.15.16:0/2982656254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058c0095f0 con 0x7f0598105350 2024-04-22T15:56:33.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.285+0000 7f0596ffd700 1 --2- 172.21.15.16:0/2982656254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0598105350 0x7f059819bb50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f058c006010 tx=0x7f058c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:33.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.286+0000 7f0594ff9700 1 -- 172.21.15.16:0/2982656254 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f058c01d070 con 0x7f0598105350 2024-04-22T15:56:33.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.286+0000 7f0594ff9700 1 -- 172.21.15.16:0/2982656254 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f058c00f460 con 0x7f0598105350 2024-04-22T15:56:33.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.286+0000 7f0594ff9700 1 -- 172.21.15.16:0/2982656254 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f058c017610 con 0x7f0598105350 2024-04-22T15:56:33.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.286+0000 7f059dc34700 1 -- 172.21.15.16:0/2982656254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05981959f0 con 0x7f0598105350 2024-04-22T15:56:33.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.286+0000 7f059dc34700 1 -- 172.21.15.16:0/2982656254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0598195e90 con 0x7f0598105350 2024-04-22T15:56:33.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.287+0000 7f059dc34700 1 -- 172.21.15.16:0/2982656254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f059804f2e0 con 0x7f0598105350 2024-04-22T15:56:33.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.287+0000 7f0594ff9700 1 -- 172.21.15.16:0/2982656254 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f058c00f5d0 con 0x7f0598105350 2024-04-22T15:56:33.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.288+0000 7f0594ff9700 1 --2- 172.21.15.16:0/2982656254 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f058006ecc0 0x7f0580071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:56:33.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.288+0000 7f0594ff9700 1 -- 172.21.15.16:0/2982656254 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f058c090300 con 0x7f0598105350 2024-04-22T15:56:33.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.288+0000 7f05977fe700 1 --2- 172.21.15.16:0/2982656254 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f058006ecc0 0x7f0580071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:56:33.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.288+0000 7f05977fe700 1 --2- 172.21.15.16:0/2982656254 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f058006ecc0 0x7f0580071180 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0588005950 tx=0x7f058800a300 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:56:33.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.293+0000 7f0594ff9700 1 -- 172.21.15.16:0/2982656254 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f058c059160 con 0x7f0598105350 2024-04-22T15:56:33.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:33 smithi027 ceph-mon[83054]: pgmap v109: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:56:33.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:33 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/1575224088' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:56:33.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:33 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/2550736055' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:56:33.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.587+0000 7f059dc34700 1 -- 172.21.15.16:0/2982656254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0598196100 con 0x7f0598105350 2024-04-22T15:56:33.591 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:33 smithi016 ceph-mon[110740]: pgmap v109: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:56:33.591 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:33 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/1575224088' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:56:33.591 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:33 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/2550736055' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:56:33.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.588+0000 7f0594ff9700 1 -- 172.21.15.16:0/2982656254 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f058c05c820 con 0x7f0598105350 2024-04-22T15:56:33.591 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:56:33.591 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:56:33.591 INFO:teuthology.orchestra.run.smithi016.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:56:33.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.591+0000 7f059dc34700 1 -- 172.21.15.16:0/2982656254 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f058006ecc0 msgr2=0x7f0580071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:33.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.591+0000 7f059dc34700 1 --2- 172.21.15.16:0/2982656254 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f058006ecc0 0x7f0580071180 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0588005950 tx=0x7f058800a300 comp rx=0 tx=0).stop 2024-04-22T15:56:33.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.591+0000 7f059dc34700 1 -- 172.21.15.16:0/2982656254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0598105350 msgr2=0x7f059819bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:56:33.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.591+0000 7f059dc34700 1 --2- 172.21.15.16:0/2982656254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0598105350 0x7f059819bb50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f058c006010 tx=0x7f058c00bd70 comp rx=0 tx=0).stop 2024-04-22T15:56:33.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.591+0000 7f059dc34700 1 -- 172.21.15.16:0/2982656254 shutdown_connections 2024-04-22T15:56:33.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.591+0000 7f059dc34700 1 --2- 172.21.15.16:0/2982656254 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f058006ecc0 0x7f0580071180 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:33.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.591+0000 7f059dc34700 1 --2- 172.21.15.16:0/2982656254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05980ff340 0x7f059819b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:33.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.591+0000 7f059dc34700 1 --2- 172.21.15.16:0/2982656254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0598105350 0x7f059819bb50 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:56:33.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.591+0000 7f059dc34700 1 -- 172.21.15.16:0/2982656254 >> 172.21.15.16:0/2982656254 conn(0x7f0598074e30 msgr2=0x7f0598103a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:56:33.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.591+0000 7f059dc34700 1 -- 172.21.15.16:0/2982656254 shutdown_connections 2024-04-22T15:56:33.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:56:33.591+0000 7f059dc34700 1 -- 172.21.15.16:0/2982656254 wait complete. 2024-04-22T15:56:34.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:34 smithi027 ceph-mon[83054]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:56:34.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:34 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/2982656254' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:56:34.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:34 smithi016 ceph-mon[110740]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:56:34.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:34 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/2982656254' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:56:35.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:35 smithi027 ceph-mon[83054]: pgmap v110: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:56:35.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:35 smithi016 ceph-mon[110740]: pgmap v110: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:56:37.388 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:37 smithi027 ceph-mon[83054]: pgmap v111: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:56:37.388 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:37 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-22T15:56:37.388 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:37 smithi027 ceph-mon[83054]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-22T15:56:37.388 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:37 smithi027 ceph-mon[83054]: Upgrade: osd.3 is safe to restart 2024-04-22T15:56:37.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:37 smithi016 ceph-mon[110740]: pgmap v111: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:56:37.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:37 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-22T15:56:37.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:37 smithi016 ceph-mon[110740]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-22T15:56:37.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:37 smithi016 ceph-mon[110740]: Upgrade: osd.3 is safe to restart 2024-04-22T15:56:39.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:39 smithi027 ceph-mon[83054]: Upgrade: Updating osd.3 2024-04-22T15:56:39.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:39 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:39.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:39 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T15:56:39.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:39 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:56:39.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:39 smithi027 ceph-mon[83054]: Deploying daemon osd.3 on smithi027 2024-04-22T15:56:39.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:39 smithi027 ceph-mon[83054]: pgmap v112: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:56:39.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:39 smithi016 ceph-mon[110740]: Upgrade: Updating osd.3 2024-04-22T15:56:39.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:39 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:39.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:39 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T15:56:39.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:39 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:56:39.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:39 smithi016 ceph-mon[110740]: Deploying daemon osd.3 on smithi027 2024-04-22T15:56:39.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:39 smithi016 ceph-mon[110740]: pgmap v112: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:56:41.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:41 smithi027 ceph-mon[83054]: pgmap v113: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:56:41.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:41 smithi016 ceph-mon[110740]: pgmap v113: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:56:41.874 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:41 smithi027 systemd[1]: Stopping Ceph osd.3 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T15:56:41.874 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:41 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3[40990]: 2024-04-22T15:56:41.842+0000 7f8e37f07700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T15:56:41.874 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:41 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3[40990]: 2024-04-22T15:56:41.842+0000 7f8e37f07700 -1 osd.3 64 *** Got signal Terminated *** 2024-04-22T15:56:41.874 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:41 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3[40990]: 2024-04-22T15:56:41.842+0000 7f8e37f07700 -1 osd.3 64 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T15:56:42.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:42 smithi027 ceph-mon[83054]: osd.3 marked itself down and dead 2024-04-22T15:56:42.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:42 smithi016 ceph-mon[110740]: osd.3 marked itself down and dead 2024-04-22T15:56:43.137 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:42 smithi027 podman[91882]: 2024-04-22 15:56:42.859835702 +0000 UTC m=+1.098649861 container died cdf9b7aaf865d8e4cdabd2959ff21fc43b60954f0368c7a380907d1cb30b8da2 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.vendor=CentOS, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1) 2024-04-22T15:56:43.586 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:43 smithi027 podman[91882]: 2024-04-22 15:56:43.226517517 +0000 UTC m=+1.465331675 container cleanup cdf9b7aaf865d8e4cdabd2959ff21fc43b60954f0368c7a380907d1cb30b8da2 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS) 2024-04-22T15:56:43.586 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:43 smithi027 bash[91882]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3 2024-04-22T15:56:43.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:43 smithi027 ceph-mon[83054]: pgmap v114: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:56:43.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:43 smithi027 ceph-mon[83054]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T15:56:43.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:43 smithi027 ceph-mon[83054]: osdmap e65: 6 total, 5 up, 6 in 2024-04-22T15:56:43.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:43 smithi016 ceph-mon[110740]: pgmap v114: 65 pgs: 65 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:56:43.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:43 smithi016 ceph-mon[110740]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T15:56:43.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:43 smithi016 ceph-mon[110740]: osdmap e65: 6 total, 5 up, 6 in 2024-04-22T15:56:44.014 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:43 smithi027 podman[91912]: 2024-04-22 15:56:43.611169297 +0000 UTC m=+0.749861261 container remove cdf9b7aaf865d8e4cdabd2959ff21fc43b60954f0368c7a380907d1cb30b8da2 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-22T15:56:44.311 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:44 smithi027 podman[92002]: 2024-04-22 15:56:44.012119303 +0000 UTC m=+0.115326012 container create 4734cab89d55dff25a286453ab15dc466b095d5b9c180acfbb6593a9bb47af1b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-22T15:56:44.311 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:44 smithi027 podman[92002]: 2024-04-22 15:56:43.916326395 +0000 UTC m=+0.019533113 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:56:44.311 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:44 smithi027 podman[92002]: 2024-04-22 15:56:44.256632306 +0000 UTC m=+0.359839023 container init 4734cab89d55dff25a286453ab15dc466b095d5b9c180acfbb6593a9bb47af1b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, maintainer=Guillaume Abrioux , ceph=True) 2024-04-22T15:56:44.312 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:44 smithi027 podman[92002]: 2024-04-22 15:56:44.259988149 +0000 UTC m=+0.363194856 container start 4734cab89d55dff25a286453ab15dc466b095d5b9c180acfbb6593a9bb47af1b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:56:44.565 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:44 smithi027 ceph-mon[83054]: osdmap e66: 6 total, 5 up, 6 in 2024-04-22T15:56:44.565 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:44 smithi027 podman[92002]: 2024-04-22 15:56:44.331420228 +0000 UTC m=+0.434626941 container attach 4734cab89d55dff25a286453ab15dc466b095d5b9c180acfbb6593a9bb47af1b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-22T15:56:44.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:44 smithi016 ceph-mon[110740]: osdmap e66: 6 total, 5 up, 6 in 2024-04-22T15:56:44.836 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:44 smithi027 podman[92002]: 2024-04-22 15:56:44.563545259 +0000 UTC m=+0.666751971 container died 4734cab89d55dff25a286453ab15dc466b095d5b9c180acfbb6593a9bb47af1b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-deactivate, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-22T15:56:45.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:45 smithi027 ceph-mon[83054]: pgmap v117: 65 pgs: 16 stale+active+clean, 49 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 639 B/s rd, 1 op/s 2024-04-22T15:56:45.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:45 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:45.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:45 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:56:45.587 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:45 smithi027 podman[92028]: 2024-04-22 15:56:45.218385618 +0000 UTC m=+0.645206241 container remove 4734cab89d55dff25a286453ab15dc466b095d5b9c180acfbb6593a9bb47af1b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T15:56:45.587 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:45 smithi027 systemd[1]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.3.service: Deactivated successfully. 2024-04-22T15:56:45.587 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:45 smithi027 systemd[1]: Stopped Ceph osd.3 for 67ab3a80-00be-11ef-bc93-c7b262605968. 2024-04-22T15:56:45.587 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:45 smithi027 systemd[1]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.3.service: Consumed 1min 7.552s CPU time. 2024-04-22T15:56:45.587 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:45 smithi027 systemd[1]: Starting Ceph osd.3 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T15:56:45.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:45 smithi016 ceph-mon[110740]: pgmap v117: 65 pgs: 16 stale+active+clean, 49 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 639 B/s rd, 1 op/s 2024-04-22T15:56:45.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:45 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:45.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:45 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:56:45.982 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:45 smithi027 podman[92132]: 2024-04-22 15:56:45.587395496 +0000 UTC m=+0.019201241 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:56:45.982 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:45 smithi027 podman[92132]: 2024-04-22 15:56:45.686098275 +0000 UTC m=+0.117904016 container create b6f18e651907e5219a256b561517991900d29522a42623a9db7f7d600ea16242 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-activate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-22T15:56:45.982 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:45 smithi027 podman[92132]: 2024-04-22 15:56:45.896949261 +0000 UTC m=+0.328755003 container init b6f18e651907e5219a256b561517991900d29522a42623a9db7f7d600ea16242 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-activate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-22T15:56:45.982 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:45 smithi027 podman[92132]: 2024-04-22 15:56:45.9003401 +0000 UTC m=+0.332145843 container start b6f18e651907e5219a256b561517991900d29522a42623a9db7f7d600ea16242 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:56:45.982 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:45 smithi027 podman[92132]: 2024-04-22 15:56:45.980455077 +0000 UTC m=+0.412260832 container attach b6f18e651907e5219a256b561517991900d29522a42623a9db7f7d600ea16242 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-activate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-22T15:56:46.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:46 smithi016 ceph-mon[110740]: Health check failed: Degraded data redundancy: 39/258 objects degraded (15.116%), 11 pgs degraded (PG_DEGRADED) 2024-04-22T15:56:46.710 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:46 smithi027 ceph-mon[83054]: Health check failed: Degraded data redundancy: 39/258 objects degraded (15.116%), 11 pgs degraded (PG_DEGRADED) 2024-04-22T15:56:46.710 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:46 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-activate[92147]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:56:46.710 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:46 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-activate[92147]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:56:46.710 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:46 smithi027 bash[92132]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:56:46.710 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:46 smithi027 bash[92132]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:56:46.711 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:46 smithi027 bash[92132]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-22T15:56:46.711 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:46 smithi027 bash[92132]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-22T15:56:46.711 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:46 smithi027 bash[92132]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-22T15:56:46.711 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:46 smithi027 bash[92132]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-22T15:56:46.711 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:46 smithi027 bash[92132]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-04-22T15:56:46.711 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:46 smithi027 bash[92132]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-22T15:56:46.711 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:46 smithi027 bash[92132]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-22T15:56:46.711 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:46 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-activate[92147]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-22T15:56:46.711 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:46 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-activate[92147]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-22T15:56:46.711 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:46 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-activate[92147]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-22T15:56:46.711 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:46 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-activate[92147]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-22T15:56:46.711 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:46 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-activate[92147]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-04-22T15:56:46.711 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:46 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-activate[92147]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-22T15:56:46.711 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:46 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-activate[92147]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-22T15:56:46.712 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:46 smithi027 podman[92132]: 2024-04-22 15:56:46.424354617 +0000 UTC m=+0.856160362 container died b6f18e651907e5219a256b561517991900d29522a42623a9db7f7d600ea16242 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-activate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T15:56:47.336 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:47 smithi027 podman[92289]: 2024-04-22 15:56:47.025633701 +0000 UTC m=+0.596097942 container remove b6f18e651907e5219a256b561517991900d29522a42623a9db7f7d600ea16242 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-activate, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-22T15:56:47.336 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:47 smithi027 podman[92350]: 2024-04-22 15:56:47.243096346 +0000 UTC m=+0.111005923 container create 81034bf22d3f0e38ac4e75e6e3c51825cb23b87f19f420a7ef5279197c33250b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-22T15:56:47.336 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:47 smithi027 podman[92350]: 2024-04-22 15:56:47.150843966 +0000 UTC m=+0.018753520 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:56:47.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:47 smithi016 ceph-mon[110740]: pgmap v118: 65 pgs: 11 active+undersized+degraded, 3 stale+active+clean, 14 active+undersized, 37 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 39/258 objects degraded (15.116%) 2024-04-22T15:56:47.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:47 smithi027 ceph-mon[83054]: pgmap v118: 65 pgs: 11 active+undersized+degraded, 3 stale+active+clean, 14 active+undersized, 37 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 39/258 objects degraded (15.116%) 2024-04-22T15:56:47.837 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:47 smithi027 podman[92350]: 2024-04-22 15:56:47.470582596 +0000 UTC m=+0.338492141 container init 81034bf22d3f0e38ac4e75e6e3c51825cb23b87f19f420a7ef5279197c33250b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49) 2024-04-22T15:56:47.837 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:47 smithi027 podman[92350]: 2024-04-22 15:56:47.473833213 +0000 UTC m=+0.341742748 container start 81034bf22d3f0e38ac4e75e6e3c51825cb23b87f19f420a7ef5279197c33250b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-22T15:56:47.837 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:47 smithi027 ceph-osd[92370]: -- 172.21.15.27:0/1176255836 <== mon.1 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5608443e7680 con 0x5608443b1000 2024-04-22T15:56:47.837 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:47 smithi027 bash[92350]: 81034bf22d3f0e38ac4e75e6e3c51825cb23b87f19f420a7ef5279197c33250b 2024-04-22T15:56:47.837 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:47 smithi027 systemd[1]: Started Ceph osd.3 for 67ab3a80-00be-11ef-bc93-c7b262605968. 2024-04-22T15:56:48.336 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:48 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3[92366]: 2024-04-22T15:56:48.014+0000 7f74648be7c0 -1 Falling back to public interface 2024-04-22T15:56:48.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:48 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:48.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:48 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:48.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:48 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:56:48.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:48 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:48.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:48 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:48.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:48 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:56:49.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:49 smithi016 ceph-mon[110740]: pgmap v119: 65 pgs: 18 active+undersized+degraded, 20 active+undersized, 27 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 58/258 objects degraded (22.481%) 2024-04-22T15:56:49.879 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:49 smithi027 ceph-mon[83054]: pgmap v119: 65 pgs: 18 active+undersized+degraded, 20 active+undersized, 27 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 58/258 objects degraded (22.481%) 2024-04-22T15:56:51.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:51 smithi027 ceph-mon[83054]: pgmap v120: 65 pgs: 18 active+undersized+degraded, 20 active+undersized, 27 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 58/258 objects degraded (22.481%) 2024-04-22T15:56:51.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:51 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:51.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:51 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:51.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:51 smithi016 ceph-mon[110740]: pgmap v120: 65 pgs: 18 active+undersized+degraded, 20 active+undersized, 27 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 58/258 objects degraded (22.481%) 2024-04-22T15:56:51.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:51 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:51.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:51 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:52.584 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:52 smithi027 ceph-mon[83054]: Health check update: Degraded data redundancy: 58/258 objects degraded (22.481%), 18 pgs degraded (PG_DEGRADED) 2024-04-22T15:56:52.584 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:52 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:52.584 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:52 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:52.836 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:52 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3[92366]: 2024-04-22T15:56:52.580+0000 7f74648be7c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-04-22T15:56:52.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:52 smithi016 ceph-mon[110740]: Health check update: Degraded data redundancy: 58/258 objects degraded (22.481%), 18 pgs degraded (PG_DEGRADED) 2024-04-22T15:56:52.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:52 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:52.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:52 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:53.336 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:53 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3[92366]: 2024-04-22T15:56:53.025+0000 7f74648be7c0 -1 osd.3 64 log_to_monitors true 2024-04-22T15:56:54.086 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 15:56:53 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3[92366]: 2024-04-22T15:56:53.784+0000 7f7455d16700 -1 osd.3 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T15:56:54.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:53 smithi027 ceph-mon[83054]: pgmap v121: 65 pgs: 18 active+undersized+degraded, 20 active+undersized, 27 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 58/258 objects degraded (22.481%) 2024-04-22T15:56:54.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:53 smithi027 ceph-mon[83054]: from='osd.3 [v2:172.21.15.27:6800/365345197,v1:172.21.15.27:6801/365345197]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T15:56:54.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:53 smithi027 ceph-mon[83054]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T15:56:54.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:53 smithi016 ceph-mon[110740]: pgmap v121: 65 pgs: 18 active+undersized+degraded, 20 active+undersized, 27 active+clean; 250 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 58/258 objects degraded (22.481%) 2024-04-22T15:56:54.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:53 smithi016 ceph-mon[110740]: from='osd.3 [v2:172.21.15.27:6800/365345197,v1:172.21.15.27:6801/365345197]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T15:56:54.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:53 smithi016 ceph-mon[110740]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T15:56:54.898 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:54 smithi027 ceph-mon[83054]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T15:56:54.898 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:54 smithi027 ceph-mon[83054]: from='osd.3 [v2:172.21.15.27:6800/365345197,v1:172.21.15.27:6801/365345197]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-22T15:56:54.898 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:54 smithi027 ceph-mon[83054]: osdmap e67: 6 total, 5 up, 6 in 2024-04-22T15:56:54.898 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:54 smithi027 ceph-mon[83054]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-22T15:56:55.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:54 smithi016 ceph-mon[110740]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T15:56:55.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:54 smithi016 ceph-mon[110740]: from='osd.3 [v2:172.21.15.27:6800/365345197,v1:172.21.15.27:6801/365345197]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-22T15:56:55.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:54 smithi016 ceph-mon[110740]: osdmap e67: 6 total, 5 up, 6 in 2024-04-22T15:56:55.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:54 smithi016 ceph-mon[110740]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-22T15:56:55.784 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:55 smithi016 ceph-mon[110740]: pgmap v123: 65 pgs: 18 active+undersized+degraded, 20 active+undersized, 27 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 58/258 objects degraded (22.481%) 2024-04-22T15:56:55.784 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:55 smithi016 ceph-mon[110740]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T15:56:55.784 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:55 smithi016 ceph-mon[110740]: osd.3 [v2:172.21.15.27:6800/365345197,v1:172.21.15.27:6801/365345197] boot 2024-04-22T15:56:55.784 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:55 smithi016 ceph-mon[110740]: osdmap e68: 6 total, 6 up, 6 in 2024-04-22T15:56:55.784 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:55 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:56:55.784 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:55 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:55.784 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:55 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:55.784 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:55 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:56:55.784 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:55 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:56:55.785 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:55 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:55.785 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:55 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:56:55.785 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:55 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:56:55.785 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:55 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:56:55.785 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:55 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:56:55.785 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:55 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-22T15:56:56.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:55 smithi027 ceph-mon[83054]: pgmap v123: 65 pgs: 18 active+undersized+degraded, 20 active+undersized, 27 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 58/258 objects degraded (22.481%) 2024-04-22T15:56:56.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:55 smithi027 ceph-mon[83054]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T15:56:56.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:55 smithi027 ceph-mon[83054]: osd.3 [v2:172.21.15.27:6800/365345197,v1:172.21.15.27:6801/365345197] boot 2024-04-22T15:56:56.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:55 smithi027 ceph-mon[83054]: osdmap e68: 6 total, 6 up, 6 in 2024-04-22T15:56:56.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:55 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T15:56:56.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:55 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:56.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:55 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:56.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:55 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:56:56.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:55 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:56:56.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:55 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:56:56.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:55 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:56:56.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:55 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:56:56.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:55 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:56:56.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:55 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:56:56.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:55 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-22T15:56:56.849 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:56 smithi016 ceph-mon[110740]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-22T15:56:56.849 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:56 smithi016 ceph-mon[110740]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-04-22T15:56:56.849 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:56 smithi016 ceph-mon[110740]: osdmap e69: 6 total, 6 up, 6 in 2024-04-22T15:56:57.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:56 smithi027 ceph-mon[83054]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-22T15:56:57.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:56 smithi027 ceph-mon[83054]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-04-22T15:56:57.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:56 smithi027 ceph-mon[83054]: osdmap e69: 6 total, 6 up, 6 in 2024-04-22T15:56:58.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:57 smithi027 ceph-mon[83054]: pgmap v126: 65 pgs: 4 peering, 15 active+undersized+degraded, 11 active+undersized, 35 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 47/258 objects degraded (18.217%) 2024-04-22T15:56:58.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:57 smithi027 ceph-mon[83054]: Health check update: Degraded data redundancy: 47/258 objects degraded (18.217%), 15 pgs degraded (PG_DEGRADED) 2024-04-22T15:56:58.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:57 smithi016 ceph-mon[110740]: pgmap v126: 65 pgs: 4 peering, 15 active+undersized+degraded, 11 active+undersized, 35 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 47/258 objects degraded (18.217%) 2024-04-22T15:56:58.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:57 smithi016 ceph-mon[110740]: Health check update: Degraded data redundancy: 47/258 objects degraded (18.217%), 15 pgs degraded (PG_DEGRADED) 2024-04-22T15:57:00.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:56:59 smithi027 ceph-mon[83054]: pgmap v127: 65 pgs: 4 peering, 9 active+undersized+degraded, 7 active+undersized, 45 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 29/258 objects degraded (11.240%) 2024-04-22T15:57:00.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:56:59 smithi016 ceph-mon[110740]: pgmap v127: 65 pgs: 4 peering, 9 active+undersized+degraded, 7 active+undersized, 45 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 29/258 objects degraded (11.240%) 2024-04-22T15:57:01.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:01 smithi027 ceph-mon[83054]: pgmap v128: 65 pgs: 4 peering, 61 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 949 B/s rd, 1 op/s 2024-04-22T15:57:01.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:01 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:01.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:01 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:57:01.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:01 smithi027 ceph-mon[83054]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/258 objects degraded (11.240%), 9 pgs degraded) 2024-04-22T15:57:01.627 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:01 smithi016 ceph-mon[110740]: pgmap v128: 65 pgs: 4 peering, 61 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 949 B/s rd, 1 op/s 2024-04-22T15:57:01.627 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:01 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:01.627 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:01 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:57:01.627 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:01 smithi016 ceph-mon[110740]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/258 objects degraded (11.240%), 9 pgs degraded) 2024-04-22T15:57:03.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:03 smithi027 ceph-mon[83054]: pgmap v129: 65 pgs: 65 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 767 B/s rd, 1 op/s 2024-04-22T15:57:03.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:03 smithi016 ceph-mon[110740]: pgmap v129: 65 pgs: 65 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 767 B/s rd, 1 op/s 2024-04-22T15:57:03.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.715+0000 7f62955be700 1 -- 172.21.15.16:0/1553335821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6290101850 msgr2=0x7f62901058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:03.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.715+0000 7f62955be700 1 --2- 172.21.15.16:0/1553335821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6290101850 0x7f62901058a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6284009910 tx=0x7f6284009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:03.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.715+0000 7f62955be700 1 -- 172.21.15.16:0/1553335821 shutdown_connections 2024-04-22T15:57:03.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.715+0000 7f62955be700 1 --2- 172.21.15.16:0/1553335821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6290101850 0x7f62901058a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:03.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.715+0000 7f62955be700 1 --2- 172.21.15.16:0/1553335821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6290100ea0 0x7f6290101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:03.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.715+0000 7f62955be700 1 -- 172.21.15.16:0/1553335821 >> 172.21.15.16:0/1553335821 conn(0x7f62900fc750 msgr2=0x7f62900feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:03.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.715+0000 7f62955be700 1 -- 172.21.15.16:0/1553335821 shutdown_connections 2024-04-22T15:57:03.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.715+0000 7f62955be700 1 -- 172.21.15.16:0/1553335821 wait complete. 2024-04-22T15:57:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.715+0000 7f62955be700 1 Processor -- start 2024-04-22T15:57:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.716+0000 7f62955be700 1 -- start start 2024-04-22T15:57:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.716+0000 7f62955be700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6290100ea0 0x7f629019c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.716+0000 7f62955be700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6290101850 0x7f629019ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.716+0000 7f62955be700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629019d360 con 0x7f6290101850 2024-04-22T15:57:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.716+0000 7f62955be700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6290196810 con 0x7f6290100ea0 2024-04-22T15:57:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.716+0000 7f628e7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6290101850 0x7f629019ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.716+0000 7f628e7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6290101850 0x7f629019ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52612/0 (socket says 172.21.15.16:52612) 2024-04-22T15:57:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.716+0000 7f628e7fc700 1 -- 172.21.15.16:0/2608235300 learned_addr learned my addr 172.21.15.16:0/2608235300 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:57:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.716+0000 7f628effd700 1 --2- 172.21.15.16:0/2608235300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6290100ea0 0x7f629019c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.716+0000 7f628e7fc700 1 -- 172.21.15.16:0/2608235300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6290100ea0 msgr2=0x7f629019c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:03.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.716+0000 7f628e7fc700 1 --2- 172.21.15.16:0/2608235300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6290100ea0 0x7f629019c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:03.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.716+0000 7f628e7fc700 1 -- 172.21.15.16:0/2608235300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62840095f0 con 0x7f6290101850 2024-04-22T15:57:03.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.717+0000 7f628e7fc700 1 --2- 172.21.15.16:0/2608235300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6290101850 0x7f629019ccd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f62840092e0 tx=0x7f628400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:03.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.717+0000 7f6277fff700 1 -- 172.21.15.16:0/2608235300 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f628401d070 con 0x7f6290101850 2024-04-22T15:57:03.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.717+0000 7f62955be700 1 -- 172.21.15.16:0/2608235300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6290196a90 con 0x7f6290101850 2024-04-22T15:57:03.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.717+0000 7f6277fff700 1 -- 172.21.15.16:0/2608235300 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f628400f460 con 0x7f6290101850 2024-04-22T15:57:03.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.717+0000 7f6277fff700 1 -- 172.21.15.16:0/2608235300 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62840175f0 con 0x7f6290101850 2024-04-22T15:57:03.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.717+0000 7f62955be700 1 -- 172.21.15.16:0/2608235300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6290196f80 con 0x7f6290101850 2024-04-22T15:57:03.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.718+0000 7f62955be700 1 -- 172.21.15.16:0/2608235300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62901091f0 con 0x7f6290101850 2024-04-22T15:57:03.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.718+0000 7f6277fff700 1 -- 172.21.15.16:0/2608235300 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f628400f5d0 con 0x7f6290101850 2024-04-22T15:57:03.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.719+0000 7f6277fff700 1 --2- 172.21.15.16:0/2608235300 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f627806ecc0 0x7f6278071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:03.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.719+0000 7f6277fff700 1 -- 172.21.15.16:0/2608235300 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6284090310 con 0x7f6290101850 2024-04-22T15:57:03.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.719+0000 7f628effd700 1 --2- 172.21.15.16:0/2608235300 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f627806ecc0 0x7f6278071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:03.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.719+0000 7f628effd700 1 --2- 172.21.15.16:0/2608235300 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f627806ecc0 0x7f6278071180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6280006fd0 tx=0x7f6280008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:03.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.724+0000 7f6277fff700 1 -- 172.21.15.16:0/2608235300 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f628405cca0 con 0x7f6290101850 2024-04-22T15:57:03.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.955+0000 7f62955be700 1 -- 172.21.15.16:0/2608235300 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f62901971a0 con 0x7f627806ecc0 2024-04-22T15:57:03.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.956+0000 7f6277fff700 1 -- 172.21.15.16:0/2608235300 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f62901971a0 con 0x7f627806ecc0 2024-04-22T15:57:03.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.958+0000 7f62955be700 1 -- 172.21.15.16:0/2608235300 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f627806ecc0 msgr2=0x7f6278071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:03.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.958+0000 7f62955be700 1 --2- 172.21.15.16:0/2608235300 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f627806ecc0 0x7f6278071180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6280006fd0 tx=0x7f6280008040 comp rx=0 tx=0).stop 2024-04-22T15:57:03.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.958+0000 7f62955be700 1 -- 172.21.15.16:0/2608235300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6290101850 msgr2=0x7f629019ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:03.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.958+0000 7f62955be700 1 --2- 172.21.15.16:0/2608235300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6290101850 0x7f629019ccd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f62840092e0 tx=0x7f628400be20 comp rx=0 tx=0).stop 2024-04-22T15:57:03.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.958+0000 7f62955be700 1 -- 172.21.15.16:0/2608235300 shutdown_connections 2024-04-22T15:57:03.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.958+0000 7f62955be700 1 --2- 172.21.15.16:0/2608235300 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f627806ecc0 0x7f6278071180 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:03.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.958+0000 7f62955be700 1 --2- 172.21.15.16:0/2608235300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6290100ea0 0x7f629019c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:03.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.958+0000 7f62955be700 1 --2- 172.21.15.16:0/2608235300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6290101850 0x7f629019ccd0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:03.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.958+0000 7f62955be700 1 -- 172.21.15.16:0/2608235300 >> 172.21.15.16:0/2608235300 conn(0x7f62900fc750 msgr2=0x7f62900fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:03.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.959+0000 7f62955be700 1 -- 172.21.15.16:0/2608235300 shutdown_connections 2024-04-22T15:57:03.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:03.959+0000 7f62955be700 1 -- 172.21.15.16:0/2608235300 wait complete. 2024-04-22T15:57:03.972 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:57:04.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.067+0000 7f40cf7bb700 1 -- 172.21.15.16:0/620011201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40c80ff5a0 msgr2=0x7f40c80ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:04.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.067+0000 7f40cf7bb700 1 --2- 172.21.15.16:0/620011201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40c80ff5a0 0x7f40c80ff980 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f40c4009910 tx=0x7f40c4009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:04.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.067+0000 7f40cf7bb700 1 -- 172.21.15.16:0/620011201 shutdown_connections 2024-04-22T15:57:04.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.067+0000 7f40cf7bb700 1 --2- 172.21.15.16:0/620011201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40c80fff50 0x7f40c8105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:04.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.067+0000 7f40cf7bb700 1 --2- 172.21.15.16:0/620011201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40c80ff5a0 0x7f40c80ff980 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:04.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.067+0000 7f40cf7bb700 1 -- 172.21.15.16:0/620011201 >> 172.21.15.16:0/620011201 conn(0x7f40c8074e30 msgr2=0x7f40c8075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:04.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.067+0000 7f40cf7bb700 1 -- 172.21.15.16:0/620011201 shutdown_connections 2024-04-22T15:57:04.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.067+0000 7f40cf7bb700 1 -- 172.21.15.16:0/620011201 wait complete. 2024-04-22T15:57:04.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.068+0000 7f40cf7bb700 1 Processor -- start 2024-04-22T15:57:04.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.068+0000 7f40cf7bb700 1 -- start start 2024-04-22T15:57:04.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.068+0000 7f40cf7bb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40c80ff5a0 0x7f40c819c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:04.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.068+0000 7f40cf7bb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40c80fff50 0x7f40c819cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:04.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.068+0000 7f40cf7bb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c819d250 con 0x7f40c80ff5a0 2024-04-22T15:57:04.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.068+0000 7f40cf7bb700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c8196700 con 0x7f40c80fff50 2024-04-22T15:57:04.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.068+0000 7f40cd557700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40c80ff5a0 0x7f40c819c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:04.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.068+0000 7f40cd557700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40c80ff5a0 0x7f40c819c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52624/0 (socket says 172.21.15.16:52624) 2024-04-22T15:57:04.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.068+0000 7f40cd557700 1 -- 172.21.15.16:0/3163813188 learned_addr learned my addr 172.21.15.16:0/3163813188 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:57:04.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.068+0000 7f40ccd56700 1 --2- 172.21.15.16:0/3163813188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40c80fff50 0x7f40c819cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:04.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.068+0000 7f40cd557700 1 -- 172.21.15.16:0/3163813188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40c80fff50 msgr2=0x7f40c819cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:04.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.069+0000 7f40cd557700 1 --2- 172.21.15.16:0/3163813188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40c80fff50 0x7f40c819cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:04.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.069+0000 7f40cd557700 1 -- 172.21.15.16:0/3163813188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c40095f0 con 0x7f40c80ff5a0 2024-04-22T15:57:04.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.069+0000 7f40cd557700 1 --2- 172.21.15.16:0/3163813188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40c80ff5a0 0x7f40c819c680 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f40c4006010 tx=0x7f40c400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:04.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.069+0000 7f40be7fc700 1 -- 172.21.15.16:0/3163813188 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40c401d070 con 0x7f40c80ff5a0 2024-04-22T15:57:04.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.069+0000 7f40be7fc700 1 -- 172.21.15.16:0/3163813188 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f40c4005dc0 con 0x7f40c80ff5a0 2024-04-22T15:57:04.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.069+0000 7f40cf7bb700 1 -- 172.21.15.16:0/3163813188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c8196980 con 0x7f40c80ff5a0 2024-04-22T15:57:04.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.069+0000 7f40be7fc700 1 -- 172.21.15.16:0/3163813188 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40c4017400 con 0x7f40c80ff5a0 2024-04-22T15:57:04.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.069+0000 7f40cf7bb700 1 -- 172.21.15.16:0/3163813188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c8196e70 con 0x7f40c80ff5a0 2024-04-22T15:57:04.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.070+0000 7f40cf7bb700 1 -- 172.21.15.16:0/3163813188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40c8109270 con 0x7f40c80ff5a0 2024-04-22T15:57:04.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.070+0000 7f40be7fc700 1 -- 172.21.15.16:0/3163813188 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f40c400f460 con 0x7f40c80ff5a0 2024-04-22T15:57:04.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.071+0000 7f40be7fc700 1 --2- 172.21.15.16:0/3163813188 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f40b406ecc0 0x7f40b4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:04.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.071+0000 7f40be7fc700 1 -- 172.21.15.16:0/3163813188 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f40c4090190 con 0x7f40c80ff5a0 2024-04-22T15:57:04.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.071+0000 7f40ccd56700 1 --2- 172.21.15.16:0/3163813188 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f40b406ecc0 0x7f40b4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:04.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.071+0000 7f40ccd56700 1 --2- 172.21.15.16:0/3163813188 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f40b406ecc0 0x7f40b4071180 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f40c8197b70 tx=0x7f40b800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:04.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.076+0000 7f40be7fc700 1 -- 172.21.15.16:0/3163813188 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f40c405cbe0 con 0x7f40c80ff5a0 2024-04-22T15:57:04.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.309+0000 7f40cf7bb700 1 -- 172.21.15.16:0/3163813188 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f40c8197090 con 0x7f40b406ecc0 2024-04-22T15:57:04.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.310+0000 7f40be7fc700 1 -- 172.21.15.16:0/3163813188 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f40c8197090 con 0x7f40b406ecc0 2024-04-22T15:57:04.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.312+0000 7f40cf7bb700 1 -- 172.21.15.16:0/3163813188 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f40b406ecc0 msgr2=0x7f40b4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:04.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.312+0000 7f40cf7bb700 1 --2- 172.21.15.16:0/3163813188 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f40b406ecc0 0x7f40b4071180 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f40c8197b70 tx=0x7f40b800b410 comp rx=0 tx=0).stop 2024-04-22T15:57:04.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.312+0000 7f40cf7bb700 1 -- 172.21.15.16:0/3163813188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40c80ff5a0 msgr2=0x7f40c819c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:04.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.312+0000 7f40cf7bb700 1 --2- 172.21.15.16:0/3163813188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40c80ff5a0 0x7f40c819c680 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f40c4006010 tx=0x7f40c400bd70 comp rx=0 tx=0).stop 2024-04-22T15:57:04.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.312+0000 7f40cf7bb700 1 -- 172.21.15.16:0/3163813188 shutdown_connections 2024-04-22T15:57:04.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.312+0000 7f40cf7bb700 1 --2- 172.21.15.16:0/3163813188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40c80ff5a0 0x7f40c819c680 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:04.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.312+0000 7f40cf7bb700 1 --2- 172.21.15.16:0/3163813188 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f40b406ecc0 0x7f40b4071180 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:04.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.312+0000 7f40cf7bb700 1 --2- 172.21.15.16:0/3163813188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40c80fff50 0x7f40c819cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:04.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.312+0000 7f40cf7bb700 1 -- 172.21.15.16:0/3163813188 >> 172.21.15.16:0/3163813188 conn(0x7f40c8074e30 msgr2=0x7f40c80fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:04.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.312+0000 7f40cf7bb700 1 -- 172.21.15.16:0/3163813188 shutdown_connections 2024-04-22T15:57:04.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.312+0000 7f40cf7bb700 1 -- 172.21.15.16:0/3163813188 wait complete. 2024-04-22T15:57:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.420+0000 7f461ff7f700 1 -- 172.21.15.16:0/1347888423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4618101060 msgr2=0x7f46181047b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.420+0000 7f461ff7f700 1 --2- 172.21.15.16:0/1347888423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4618101060 0x7f46181047b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4608009910 tx=0x7f4608009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.420+0000 7f461ff7f700 1 -- 172.21.15.16:0/1347888423 shutdown_connections 2024-04-22T15:57:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.420+0000 7f461ff7f700 1 --2- 172.21.15.16:0/1347888423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4618101060 0x7f46181047b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.420+0000 7f461ff7f700 1 --2- 172.21.15.16:0/1347888423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46181006b0 0x7f4618100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.420+0000 7f461ff7f700 1 -- 172.21.15.16:0/1347888423 >> 172.21.15.16:0/1347888423 conn(0x7f4618074e30 msgr2=0x7f4618075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.420+0000 7f461ff7f700 1 -- 172.21.15.16:0/1347888423 shutdown_connections 2024-04-22T15:57:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.420+0000 7f461ff7f700 1 -- 172.21.15.16:0/1347888423 wait complete. 2024-04-22T15:57:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.420+0000 7f461ff7f700 1 Processor -- start 2024-04-22T15:57:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.420+0000 7f461ff7f700 1 -- start start 2024-04-22T15:57:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.421+0000 7f461ff7f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46181006b0 0x7f4618101850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.421+0000 7f461ff7f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4618101060 0x7f4618101d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.421+0000 7f461ff7f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46181022d0 con 0x7f4618101060 2024-04-22T15:57:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.421+0000 7f461ff7f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4618102410 con 0x7f46181006b0 2024-04-22T15:57:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.421+0000 7f461d51a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4618101060 0x7f4618101d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.421+0000 7f461d51a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4618101060 0x7f4618101d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52634/0 (socket says 172.21.15.16:52634) 2024-04-22T15:57:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.421+0000 7f461d51a700 1 -- 172.21.15.16:0/1358626140 learned_addr learned my addr 172.21.15.16:0/1358626140 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:57:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.421+0000 7f461dd1b700 1 --2- 172.21.15.16:0/1358626140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46181006b0 0x7f4618101850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.421+0000 7f461d51a700 1 -- 172.21.15.16:0/1358626140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46181006b0 msgr2=0x7f4618101850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.421+0000 7f461d51a700 1 --2- 172.21.15.16:0/1358626140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46181006b0 0x7f4618101850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.421+0000 7f461d51a700 1 -- 172.21.15.16:0/1358626140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46080095f0 con 0x7f4618101060 2024-04-22T15:57:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.421+0000 7f461d51a700 1 --2- 172.21.15.16:0/1358626140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4618101060 0x7f4618101d90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f46080092e0 tx=0x7f460800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.421+0000 7f460effd700 1 -- 172.21.15.16:0/1358626140 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f460801d070 con 0x7f4618101060 2024-04-22T15:57:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.421+0000 7f461ff7f700 1 -- 172.21.15.16:0/1358626140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4618102690 con 0x7f4618101060 2024-04-22T15:57:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.421+0000 7f460effd700 1 -- 172.21.15.16:0/1358626140 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f460800f460 con 0x7f4618101060 2024-04-22T15:57:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.421+0000 7f460effd700 1 -- 172.21.15.16:0/1358626140 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46080175d0 con 0x7f4618101060 2024-04-22T15:57:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.422+0000 7f461ff7f700 1 -- 172.21.15.16:0/1358626140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46181a2e60 con 0x7f4618101060 2024-04-22T15:57:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.422+0000 7f461ff7f700 1 -- 172.21.15.16:0/1358626140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f461804f2e0 con 0x7f4618101060 2024-04-22T15:57:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.423+0000 7f460effd700 1 -- 172.21.15.16:0/1358626140 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f460800f5d0 con 0x7f4618101060 2024-04-22T15:57:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.423+0000 7f460effd700 1 --2- 172.21.15.16:0/1358626140 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f460406ecc0 0x7f4604071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.424+0000 7f460effd700 1 -- 172.21.15.16:0/1358626140 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f4608090240 con 0x7f4618101060 2024-04-22T15:57:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.424+0000 7f461dd1b700 1 --2- 172.21.15.16:0/1358626140 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f460406ecc0 0x7f4604071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.424+0000 7f461dd1b700 1 --2- 172.21.15.16:0/1358626140 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f460406ecc0 0x7f4604071180 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4614009a10 tx=0x7f4614009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:04.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.428+0000 7f460effd700 1 -- 172.21.15.16:0/1358626140 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f46080590a0 con 0x7f4618101060 2024-04-22T15:57:04.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:04 smithi027 ceph-mon[83054]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:04.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.647+0000 7f461ff7f700 1 -- 172.21.15.16:0/1358626140 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4618103970 con 0x7f460406ecc0 2024-04-22T15:57:04.650 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:04 smithi016 ceph-mon[110740]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:04.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.658+0000 7f460effd700 1 -- 172.21.15.16:0/1358626140 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f4618103970 con 0x7f460406ecc0 2024-04-22T15:57:04.661 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:57:04.661 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.smithi016 smithi016 *:9093,9094 running (5m) 48s ago 16m 25.6M - 0.25.0 c8568f914cd2 ea0e0217d6f9 2024-04-22T15:57:04.661 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi016 smithi016 running (16m) 48s ago 16m 8640k - 18.2.1 5be31c24972a f8b627d3f41e 2024-04-22T15:57:04.661 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi027 smithi027 running (15m) 14s ago 15m 8238k - 18.2.1 5be31c24972a f00fd40b4220 2024-04-22T15:57:04.661 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi016 smithi016 running (2m) 48s ago 16m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 1feb008449ed 2024-04-22T15:57:04.661 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi027 smithi027 running (2m) 14s ago 15m 6656k - 19.0.0-3223-g5883d494 3d983c26e278 c7f862c6da00 2024-04-22T15:57:04.661 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.smithi016 smithi016 *:3000 running (4m) 48s ago 15m 78.9M - 9.4.12 2bacad6d85d8 388a3c9c1b3f 2024-04-22T15:57:04.661 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.fkkejm smithi016 running (10m) 48s ago 10m 17.0M - 18.2.1 5be31c24972a 4153ac94cea0 2024-04-22T15:57:04.661 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.pgvphu smithi016 running (10m) 48s ago 10m 92.0M - 18.2.1 5be31c24972a 93dc072068a2 2024-04-22T15:57:04.662 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.dgqyes smithi027 running (10m) 14s ago 10m 20.1M - 18.2.1 5be31c24972a 41a9220430fd 2024-04-22T15:57:04.662 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.iobjds smithi027 running (10m) 14s ago 10m 149M - 18.2.1 5be31c24972a c1c8d6770f02 2024-04-22T15:57:04.662 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi016.rnsnnr smithi016 *:8443,9283,8765 running (7m) 48s ago 17m 505M - 19.0.0-3223-g5883d494 3d983c26e278 09b4fe285bc4 2024-04-22T15:57:04.662 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi027.ukdgzs smithi027 *:8443,9283,8765 running (6m) 14s ago 15m 451M - 19.0.0-3223-g5883d494 3d983c26e278 7548da9b5eb9 2024-04-22T15:57:04.662 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi016 smithi016 running (3m) 48s ago 17m 49.6M 2048M 19.0.0-3223-g5883d494 3d983c26e278 4ae529a67806 2024-04-22T15:57:04.662 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi027 smithi027 running (3m) 14s ago 14m 39.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 f29fcb888801 2024-04-22T15:57:04.662 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi016 smithi016 *:9100 running (6m) 48s ago 16m 16.1M - 1.5.0 0da6a335fe13 35e46e4d2f3f 2024-04-22T15:57:04.662 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi027 smithi027 *:9100 running (6m) 14s ago 15m 16.1M - 1.5.0 0da6a335fe13 e682a7672e9b 2024-04-22T15:57:04.662 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (2m) 48s ago 14m 88.0M 1217M 19.0.0-3223-g5883d494 3d983c26e278 79217eb383a2 2024-04-22T15:57:04.662 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (89s) 48s ago 13m 89.0M 1217M 19.0.0-3223-g5883d494 3d983c26e278 ec1a45b3c53e 2024-04-22T15:57:04.662 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (53s) 48s ago 13m 12.6M 1217M 19.0.0-3223-g5883d494 3d983c26e278 48bb459fd021 2024-04-22T15:57:04.662 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi027 running (17s) 14s ago 12m 14.9M 2240M 19.0.0-3223-g5883d494 3d983c26e278 81034bf22d3f 2024-04-22T15:57:04.662 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (12m) 14s ago 12m 209M 2240M 18.2.1 5be31c24972a 629397d49c76 2024-04-22T15:57:04.662 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi027 running (12m) 14s ago 11m 195M 2240M 18.2.1 5be31c24972a 23adcf675b04 2024-04-22T15:57:04.663 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.smithi016 smithi016 *:9095 running (5m) 48s ago 15m 73.8M - 2.43.0 a07b618ecd1d 373eaadee82d 2024-04-22T15:57:04.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.660+0000 7f461ff7f700 1 -- 172.21.15.16:0/1358626140 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f460406ecc0 msgr2=0x7f4604071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:04.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.660+0000 7f461ff7f700 1 --2- 172.21.15.16:0/1358626140 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f460406ecc0 0x7f4604071180 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4614009a10 tx=0x7f4614009190 comp rx=0 tx=0).stop 2024-04-22T15:57:04.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.660+0000 7f461ff7f700 1 -- 172.21.15.16:0/1358626140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4618101060 msgr2=0x7f4618101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:04.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.660+0000 7f461ff7f700 1 --2- 172.21.15.16:0/1358626140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4618101060 0x7f4618101d90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f46080092e0 tx=0x7f460800bd70 comp rx=0 tx=0).stop 2024-04-22T15:57:04.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.660+0000 7f461ff7f700 1 -- 172.21.15.16:0/1358626140 shutdown_connections 2024-04-22T15:57:04.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.660+0000 7f461ff7f700 1 --2- 172.21.15.16:0/1358626140 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f460406ecc0 0x7f4604071180 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:04.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.660+0000 7f461ff7f700 1 --2- 172.21.15.16:0/1358626140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46181006b0 0x7f4618101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:04.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.661+0000 7f461ff7f700 1 --2- 172.21.15.16:0/1358626140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4618101060 0x7f4618101d90 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:04.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.661+0000 7f461ff7f700 1 -- 172.21.15.16:0/1358626140 >> 172.21.15.16:0/1358626140 conn(0x7f4618074e30 msgr2=0x7f46180fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:04.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.661+0000 7f461ff7f700 1 -- 172.21.15.16:0/1358626140 shutdown_connections 2024-04-22T15:57:04.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.661+0000 7f461ff7f700 1 -- 172.21.15.16:0/1358626140 wait complete. 2024-04-22T15:57:04.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.766+0000 7f2c584a2700 1 -- 172.21.15.16:0/4107287498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c500ffda0 msgr2=0x7f2c50100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.766+0000 7f2c584a2700 1 --2- 172.21.15.16:0/4107287498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c500ffda0 0x7f2c50100180 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2c40009910 tx=0x7f2c40009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.767+0000 7f2c584a2700 1 -- 172.21.15.16:0/4107287498 shutdown_connections 2024-04-22T15:57:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.767+0000 7f2c584a2700 1 --2- 172.21.15.16:0/4107287498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c50100750 0x7f2c501047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.767+0000 7f2c584a2700 1 --2- 172.21.15.16:0/4107287498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c500ffda0 0x7f2c50100180 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.767+0000 7f2c584a2700 1 -- 172.21.15.16:0/4107287498 >> 172.21.15.16:0/4107287498 conn(0x7f2c500fb650 msgr2=0x7f2c500fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.767+0000 7f2c584a2700 1 -- 172.21.15.16:0/4107287498 shutdown_connections 2024-04-22T15:57:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.767+0000 7f2c584a2700 1 -- 172.21.15.16:0/4107287498 wait complete. 2024-04-22T15:57:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.767+0000 7f2c584a2700 1 Processor -- start 2024-04-22T15:57:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.767+0000 7f2c584a2700 1 -- start start 2024-04-22T15:57:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.767+0000 7f2c584a2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c500ffda0 0x7f2c5019c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.768+0000 7f2c584a2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c50100750 0x7f2c5019cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.768+0000 7f2c584a2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5019d210 con 0x7f2c50100750 2024-04-22T15:57:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.768+0000 7f2c584a2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c501966c0 con 0x7f2c500ffda0 2024-04-22T15:57:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.768+0000 7f2c55a3d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c50100750 0x7f2c5019cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.768+0000 7f2c55a3d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c50100750 0x7f2c5019cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52638/0 (socket says 172.21.15.16:52638) 2024-04-22T15:57:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.768+0000 7f2c55a3d700 1 -- 172.21.15.16:0/2950592236 learned_addr learned my addr 172.21.15.16:0/2950592236 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:57:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.768+0000 7f2c5623e700 1 --2- 172.21.15.16:0/2950592236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c500ffda0 0x7f2c5019c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.768+0000 7f2c55a3d700 1 -- 172.21.15.16:0/2950592236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c500ffda0 msgr2=0x7f2c5019c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.768+0000 7f2c55a3d700 1 --2- 172.21.15.16:0/2950592236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c500ffda0 0x7f2c5019c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.768+0000 7f2c55a3d700 1 -- 172.21.15.16:0/2950592236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c400095f0 con 0x7f2c50100750 2024-04-22T15:57:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.768+0000 7f2c55a3d700 1 --2- 172.21.15.16:0/2950592236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c50100750 0x7f2c5019cb80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00d900 tx=0x7f2c4c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.768+0000 7f2c477fe700 1 -- 172.21.15.16:0/2950592236 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c4c009900 con 0x7f2c50100750 2024-04-22T15:57:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.768+0000 7f2c477fe700 1 -- 172.21.15.16:0/2950592236 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c4c010460 con 0x7f2c50100750 2024-04-22T15:57:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.768+0000 7f2c584a2700 1 -- 172.21.15.16:0/2950592236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c501969a0 con 0x7f2c50100750 2024-04-22T15:57:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.768+0000 7f2c477fe700 1 -- 172.21.15.16:0/2950592236 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c4c00f5f0 con 0x7f2c50100750 2024-04-22T15:57:04.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.769+0000 7f2c584a2700 1 -- 172.21.15.16:0/2950592236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c50196ef0 con 0x7f2c50100750 2024-04-22T15:57:04.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.769+0000 7f2c584a2700 1 -- 172.21.15.16:0/2950592236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c5004f2e0 con 0x7f2c50100750 2024-04-22T15:57:04.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.770+0000 7f2c477fe700 1 -- 172.21.15.16:0/2950592236 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2c4c0105d0 con 0x7f2c50100750 2024-04-22T15:57:04.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.771+0000 7f2c477fe700 1 --2- 172.21.15.16:0/2950592236 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f2c3c06ecc0 0x7f2c3c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:04.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.771+0000 7f2c5623e700 1 --2- 172.21.15.16:0/2950592236 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f2c3c06ecc0 0x7f2c3c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:04.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.771+0000 7f2c477fe700 1 -- 172.21.15.16:0/2950592236 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f2c4c08e860 con 0x7f2c50100750 2024-04-22T15:57:04.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.771+0000 7f2c5623e700 1 --2- 172.21.15.16:0/2950592236 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f2c3c06ecc0 0x7f2c3c071180 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2c40006010 tx=0x7f2c4001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:04.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:04.775+0000 7f2c477fe700 1 -- 172.21.15.16:0/2950592236 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2c4c05b270 con 0x7f2c50100750 2024-04-22T15:57:05.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.070+0000 7f2c584a2700 1 -- 172.21.15.16:0/2950592236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2c50196b30 con 0x7f2c50100750 2024-04-22T15:57:05.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.071+0000 7f2c477fe700 1 -- 172.21.15.16:0/2950592236 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f2c50196b30 con 0x7f2c50100750 2024-04-22T15:57:05.074 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:57:05.074 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-22T15:57:05.074 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:57:05.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:57:05.074 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-22T15:57:05.074 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:57:05.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:57:05.074 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-22T15:57:05.074 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2, 2024-04-22T15:57:05.075 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:57:05.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:57:05.075 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-22T15:57:05.075 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T15:57:05.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:57:05.075 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-22T15:57:05.075 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6, 2024-04-22T15:57:05.075 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 8 2024-04-22T15:57:05.075 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-22T15:57:05.075 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:57:05.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.073+0000 7f2c584a2700 1 -- 172.21.15.16:0/2950592236 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f2c3c06ecc0 msgr2=0x7f2c3c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:05.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.073+0000 7f2c584a2700 1 --2- 172.21.15.16:0/2950592236 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f2c3c06ecc0 0x7f2c3c071180 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2c40006010 tx=0x7f2c4001a040 comp rx=0 tx=0).stop 2024-04-22T15:57:05.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.073+0000 7f2c584a2700 1 -- 172.21.15.16:0/2950592236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c50100750 msgr2=0x7f2c5019cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:05.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.073+0000 7f2c584a2700 1 --2- 172.21.15.16:0/2950592236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c50100750 0x7f2c5019cb80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00d900 tx=0x7f2c4c00dc10 comp rx=0 tx=0).stop 2024-04-22T15:57:05.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.073+0000 7f2c584a2700 1 -- 172.21.15.16:0/2950592236 shutdown_connections 2024-04-22T15:57:05.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.073+0000 7f2c584a2700 1 --2- 172.21.15.16:0/2950592236 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f2c3c06ecc0 0x7f2c3c071180 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:05.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.073+0000 7f2c584a2700 1 --2- 172.21.15.16:0/2950592236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c500ffda0 0x7f2c5019c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:05.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.073+0000 7f2c584a2700 1 --2- 172.21.15.16:0/2950592236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c50100750 0x7f2c5019cb80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:05.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.073+0000 7f2c584a2700 1 -- 172.21.15.16:0/2950592236 >> 172.21.15.16:0/2950592236 conn(0x7f2c500fb650 msgr2=0x7f2c50103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:05.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.073+0000 7f2c584a2700 1 -- 172.21.15.16:0/2950592236 shutdown_connections 2024-04-22T15:57:05.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.073+0000 7f2c584a2700 1 -- 172.21.15.16:0/2950592236 wait complete. 2024-04-22T15:57:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.180+0000 7f81431a2700 1 -- 172.21.15.16:0/1090766465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f813c106cd0 msgr2=0x7f813c1070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.180+0000 7f81431a2700 1 --2- 172.21.15.16:0/1090766465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f813c106cd0 0x7f813c1070b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8130009910 tx=0x7f8130009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.180+0000 7f81431a2700 1 -- 172.21.15.16:0/1090766465 shutdown_connections 2024-04-22T15:57:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.180+0000 7f81431a2700 1 --2- 172.21.15.16:0/1090766465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f813c068580 0x7f813c0689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.180+0000 7f81431a2700 1 --2- 172.21.15.16:0/1090766465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f813c106cd0 0x7f813c1070b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.180+0000 7f81431a2700 1 -- 172.21.15.16:0/1090766465 >> 172.21.15.16:0/1090766465 conn(0x7f813c075670 msgr2=0x7f813c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:05.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.180+0000 7f81431a2700 1 -- 172.21.15.16:0/1090766465 shutdown_connections 2024-04-22T15:57:05.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.180+0000 7f81431a2700 1 -- 172.21.15.16:0/1090766465 wait complete. 2024-04-22T15:57:05.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.181+0000 7f81431a2700 1 Processor -- start 2024-04-22T15:57:05.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.181+0000 7f81431a2700 1 -- start start 2024-04-22T15:57:05.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.181+0000 7f81431a2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f813c068580 0x7f813c19c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:05.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.181+0000 7f81431a2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f813c106cd0 0x7f813c19cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:05.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.181+0000 7f81431a2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c19d230 con 0x7f813c106cd0 2024-04-22T15:57:05.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.181+0000 7f81431a2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c1966e0 con 0x7f813c068580 2024-04-22T15:57:05.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.181+0000 7f813bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f813c106cd0 0x7f813c19cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.181+0000 7f813bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f813c106cd0 0x7f813c19cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52654/0 (socket says 172.21.15.16:52654) 2024-04-22T15:57:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.181+0000 7f813bfff700 1 -- 172.21.15.16:0/771908616 learned_addr learned my addr 172.21.15.16:0/771908616 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:57:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.181+0000 7f8140f3e700 1 --2- 172.21.15.16:0/771908616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f813c068580 0x7f813c19c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.181+0000 7f813bfff700 1 -- 172.21.15.16:0/771908616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f813c068580 msgr2=0x7f813c19c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.181+0000 7f813bfff700 1 --2- 172.21.15.16:0/771908616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f813c068580 0x7f813c19c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.181+0000 7f813bfff700 1 -- 172.21.15.16:0/771908616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81300095f0 con 0x7f813c106cd0 2024-04-22T15:57:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.181+0000 7f8140f3e700 1 --2- 172.21.15.16:0/771908616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f813c068580 0x7f813c19c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:57:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.181+0000 7f813bfff700 1 --2- 172.21.15.16:0/771908616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f813c106cd0 0x7f813c19cba0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f812800d8d0 tx=0x7f812800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.182+0000 7f8139ffb700 1 -- 172.21.15.16:0/771908616 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8128009940 con 0x7f813c106cd0 2024-04-22T15:57:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.182+0000 7f81431a2700 1 -- 172.21.15.16:0/771908616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f813c1969c0 con 0x7f813c106cd0 2024-04-22T15:57:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.182+0000 7f8139ffb700 1 -- 172.21.15.16:0/771908616 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8128010460 con 0x7f813c106cd0 2024-04-22T15:57:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.182+0000 7f8139ffb700 1 -- 172.21.15.16:0/771908616 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f812800f5f0 con 0x7f813c106cd0 2024-04-22T15:57:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.182+0000 7f81431a2700 1 -- 172.21.15.16:0/771908616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f813c196f10 con 0x7f813c106cd0 2024-04-22T15:57:05.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.183+0000 7f81431a2700 1 -- 172.21.15.16:0/771908616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813c1091d0 con 0x7f813c106cd0 2024-04-22T15:57:05.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.183+0000 7f8139ffb700 1 -- 172.21.15.16:0/771908616 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f81280105d0 con 0x7f813c106cd0 2024-04-22T15:57:05.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.184+0000 7f8139ffb700 1 --2- 172.21.15.16:0/771908616 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f812c06ecc0 0x7f812c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:05.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.184+0000 7f8139ffb700 1 -- 172.21.15.16:0/771908616 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f812808ee70 con 0x7f813c106cd0 2024-04-22T15:57:05.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.184+0000 7f8140f3e700 1 --2- 172.21.15.16:0/771908616 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f812c06ecc0 0x7f812c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:05.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.184+0000 7f8140f3e700 1 --2- 172.21.15.16:0/771908616 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f812c06ecc0 0x7f812c071180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8130005b40 tx=0x7f813001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:05.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.188+0000 7f8139ffb700 1 -- 172.21.15.16:0/771908616 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f812805b760 con 0x7f813c106cd0 2024-04-22T15:57:05.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:05 smithi027 ceph-mon[83054]: pgmap v130: 65 pgs: 65 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:57:05.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:05 smithi027 ceph-mon[83054]: from='client.34318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:05.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:05 smithi027 ceph-mon[83054]: from='client.34322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:05.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:05 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/2950592236' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:57:05.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:05 smithi016 ceph-mon[110740]: pgmap v130: 65 pgs: 65 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:57:05.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:05 smithi016 ceph-mon[110740]: from='client.34318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:05.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:05 smithi016 ceph-mon[110740]: from='client.34322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:05.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:05 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/2950592236' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:57:05.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.442+0000 7f81431a2700 1 -- 172.21.15.16:0/771908616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f813c197130 con 0x7f813c106cd0 2024-04-22T15:57:05.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.442+0000 7f8139ffb700 1 -- 172.21.15.16:0/771908616 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1902 (secure 0 0 0) 0x7f812805b2f0 con 0x7f813c106cd0 2024-04-22T15:57:05.445 INFO:teuthology.orchestra.run.smithi016.stdout:e16 2024-04-22T15:57:05.445 INFO:teuthology.orchestra.run.smithi016.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:57:05.445 INFO:teuthology.orchestra.run.smithi016.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:57:05.445 INFO:teuthology.orchestra.run.smithi016.stdout:legacy client fscid: 1 2024-04-22T15:57:05.445 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:57:05.445 INFO:teuthology.orchestra.run.smithi016.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:57:05.445 INFO:teuthology.orchestra.run.smithi016.stdout:fs_name cephfs 2024-04-22T15:57:05.446 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 16 2024-04-22T15:57:05.446 INFO:teuthology.orchestra.run.smithi016.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T15:57:05.446 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-04-22T15:46:27.774571+0000 2024-04-22T15:57:05.446 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-04-22T15:46:47.381709+0000 2024-04-22T15:57:05.446 INFO:teuthology.orchestra.run.smithi016.stdout:tableserver 0 2024-04-22T15:57:05.446 INFO:teuthology.orchestra.run.smithi016.stdout:root 0 2024-04-22T15:57:05.446 INFO:teuthology.orchestra.run.smithi016.stdout:session_timeout 60 2024-04-22T15:57:05.446 INFO:teuthology.orchestra.run.smithi016.stdout:session_autoclose 300 2024-04-22T15:57:05.446 INFO:teuthology.orchestra.run.smithi016.stdout:max_file_size 1099511627776 2024-04-22T15:57:05.446 INFO:teuthology.orchestra.run.smithi016.stdout:max_xattr_size 65536 2024-04-22T15:57:05.446 INFO:teuthology.orchestra.run.smithi016.stdout:required_client_features {} 2024-04-22T15:57:05.446 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure 0 2024-04-22T15:57:05.446 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure_osd_epoch 46 2024-04-22T15:57:05.446 INFO:teuthology.orchestra.run.smithi016.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:57:05.447 INFO:teuthology.orchestra.run.smithi016.stdout:max_mds 1 2024-04-22T15:57:05.447 INFO:teuthology.orchestra.run.smithi016.stdout:in 0 2024-04-22T15:57:05.447 INFO:teuthology.orchestra.run.smithi016.stdout:up {0=24279} 2024-04-22T15:57:05.447 INFO:teuthology.orchestra.run.smithi016.stdout:failed 2024-04-22T15:57:05.447 INFO:teuthology.orchestra.run.smithi016.stdout:damaged 2024-04-22T15:57:05.447 INFO:teuthology.orchestra.run.smithi016.stdout:stopped 2024-04-22T15:57:05.447 INFO:teuthology.orchestra.run.smithi016.stdout:data_pools [3] 2024-04-22T15:57:05.447 INFO:teuthology.orchestra.run.smithi016.stdout:metadata_pool 2 2024-04-22T15:57:05.447 INFO:teuthology.orchestra.run.smithi016.stdout:inline_data enabled 2024-04-22T15:57:05.447 INFO:teuthology.orchestra.run.smithi016.stdout:balancer 2024-04-22T15:57:05.447 INFO:teuthology.orchestra.run.smithi016.stdout:bal_rank_mask -1 2024-04-22T15:57:05.447 INFO:teuthology.orchestra.run.smithi016.stdout:standby_count_wanted 1 2024-04-22T15:57:05.447 INFO:teuthology.orchestra.run.smithi016.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:57:05.447 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.iobjds{0:24279} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.27:6824/4216277609,v1:172.21.15.27:6825/4216277609] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:57:05.448 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.pgvphu{0:14494} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.16:6828/1006073414,v1:172.21.15.16:6829/1006073414] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:57:05.448 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:57:05.448 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:57:05.448 INFO:teuthology.orchestra.run.smithi016.stdout:Standby daemons: 2024-04-22T15:57:05.448 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:57:05.448 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.fkkejm{-1:14504} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.16:6826/3484907755,v1:172.21.15.16:6827/3484907755] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:57:05.448 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.dgqyes{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.27:6826/476026355,v1:172.21.15.27:6827/476026355] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:57:05.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.445+0000 7f81431a2700 1 -- 172.21.15.16:0/771908616 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f812c06ecc0 msgr2=0x7f812c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:05.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.445+0000 7f81431a2700 1 --2- 172.21.15.16:0/771908616 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f812c06ecc0 0x7f812c071180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8130005b40 tx=0x7f813001a040 comp rx=0 tx=0).stop 2024-04-22T15:57:05.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.445+0000 7f81431a2700 1 -- 172.21.15.16:0/771908616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f813c106cd0 msgr2=0x7f813c19cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:05.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.445+0000 7f81431a2700 1 --2- 172.21.15.16:0/771908616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f813c106cd0 0x7f813c19cba0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f812800d8d0 tx=0x7f812800dc90 comp rx=0 tx=0).stop 2024-04-22T15:57:05.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.445+0000 7f81431a2700 1 -- 172.21.15.16:0/771908616 shutdown_connections 2024-04-22T15:57:05.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.445+0000 7f81431a2700 1 --2- 172.21.15.16:0/771908616 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f812c06ecc0 0x7f812c071180 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:05.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.445+0000 7f81431a2700 1 --2- 172.21.15.16:0/771908616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f813c068580 0x7f813c19c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:05.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.445+0000 7f81431a2700 1 --2- 172.21.15.16:0/771908616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f813c106cd0 0x7f813c19cba0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:05.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.445+0000 7f81431a2700 1 -- 172.21.15.16:0/771908616 >> 172.21.15.16:0/771908616 conn(0x7f813c075670 msgr2=0x7f813c0fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:05.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.445+0000 7f81431a2700 1 -- 172.21.15.16:0/771908616 shutdown_connections 2024-04-22T15:57:05.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.445+0000 7f81431a2700 1 -- 172.21.15.16:0/771908616 wait complete. 2024-04-22T15:57:05.449 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 16 2024-04-22T15:57:05.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.553+0000 7fbdefa55700 1 -- 172.21.15.16:0/3386461337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbde8100ea0 msgr2=0x7fbde8101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:05.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.553+0000 7fbdefa55700 1 --2- 172.21.15.16:0/3386461337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbde8100ea0 0x7fbde8101280 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8009910 tx=0x7fbdd8009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:05.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.553+0000 7fbdefa55700 1 -- 172.21.15.16:0/3386461337 shutdown_connections 2024-04-22T15:57:05.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.553+0000 7fbdefa55700 1 --2- 172.21.15.16:0/3386461337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbde8101850 0x7fbde81058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:05.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.553+0000 7fbdefa55700 1 --2- 172.21.15.16:0/3386461337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbde8100ea0 0x7fbde8101280 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:05.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.553+0000 7fbdefa55700 1 -- 172.21.15.16:0/3386461337 >> 172.21.15.16:0/3386461337 conn(0x7fbde80fc750 msgr2=0x7fbde80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:05.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.553+0000 7fbdefa55700 1 -- 172.21.15.16:0/3386461337 shutdown_connections 2024-04-22T15:57:05.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.553+0000 7fbdefa55700 1 -- 172.21.15.16:0/3386461337 wait complete. 2024-04-22T15:57:05.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.554+0000 7fbdefa55700 1 Processor -- start 2024-04-22T15:57:05.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.554+0000 7fbdefa55700 1 -- start start 2024-04-22T15:57:05.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.554+0000 7fbdefa55700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbde8100ea0 0x7fbde819c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:05.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.554+0000 7fbdefa55700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbde8101850 0x7fbde819cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:05.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.554+0000 7fbdefa55700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde819d220 con 0x7fbde8101850 2024-04-22T15:57:05.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.554+0000 7fbdefa55700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde81966d0 con 0x7fbde8100ea0 2024-04-22T15:57:05.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.554+0000 7fbdecff0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbde8101850 0x7fbde819cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:05.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.554+0000 7fbdecff0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbde8101850 0x7fbde819cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52668/0 (socket says 172.21.15.16:52668) 2024-04-22T15:57:05.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.554+0000 7fbdecff0700 1 -- 172.21.15.16:0/3436114335 learned_addr learned my addr 172.21.15.16:0/3436114335 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:57:05.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.554+0000 7fbded7f1700 1 --2- 172.21.15.16:0/3436114335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbde8100ea0 0x7fbde819c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:05.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.554+0000 7fbdecff0700 1 -- 172.21.15.16:0/3436114335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbde8100ea0 msgr2=0x7fbde819c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:05.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.554+0000 7fbdecff0700 1 --2- 172.21.15.16:0/3436114335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbde8100ea0 0x7fbde819c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:05.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.554+0000 7fbdecff0700 1 -- 172.21.15.16:0/3436114335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd80095f0 con 0x7fbde8101850 2024-04-22T15:57:05.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.554+0000 7fbdecff0700 1 --2- 172.21.15.16:0/3436114335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbde8101850 0x7fbde819cb90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbde400c8f0 tx=0x7fbde400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:05.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.555+0000 7fbdde7fc700 1 -- 172.21.15.16:0/3436114335 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbde40078a0 con 0x7fbde8101850 2024-04-22T15:57:05.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.555+0000 7fbdde7fc700 1 -- 172.21.15.16:0/3436114335 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbde400f450 con 0x7fbde8101850 2024-04-22T15:57:05.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.555+0000 7fbdefa55700 1 -- 172.21.15.16:0/3436114335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde81969b0 con 0x7fbde8101850 2024-04-22T15:57:05.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.555+0000 7fbdde7fc700 1 -- 172.21.15.16:0/3436114335 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbde4018610 con 0x7fbde8101850 2024-04-22T15:57:05.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.555+0000 7fbdefa55700 1 -- 172.21.15.16:0/3436114335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde8196f00 con 0x7fbde8101850 2024-04-22T15:57:05.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.556+0000 7fbdefa55700 1 -- 172.21.15.16:0/3436114335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbde8109240 con 0x7fbde8101850 2024-04-22T15:57:05.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.556+0000 7fbdde7fc700 1 -- 172.21.15.16:0/3436114335 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbde400f5c0 con 0x7fbde8101850 2024-04-22T15:57:05.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.557+0000 7fbdde7fc700 1 --2- 172.21.15.16:0/3436114335 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fbdd406ecc0 0x7fbdd4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:05.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.557+0000 7fbdde7fc700 1 -- 172.21.15.16:0/3436114335 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fbde408ec90 con 0x7fbde8101850 2024-04-22T15:57:05.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.557+0000 7fbded7f1700 1 --2- 172.21.15.16:0/3436114335 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fbdd406ecc0 0x7fbdd4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:05.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.557+0000 7fbded7f1700 1 --2- 172.21.15.16:0/3436114335 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fbdd406ecc0 0x7fbdd4071180 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8006010 tx=0x7fbdd801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:05.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.562+0000 7fbdde7fc700 1 -- 172.21.15.16:0/3436114335 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbde405b6a0 con 0x7fbde8101850 2024-04-22T15:57:05.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.792+0000 7fbdefa55700 1 -- 172.21.15.16:0/3436114335 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbde81971d0 con 0x7fbdd406ecc0 2024-04-22T15:57:05.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.793+0000 7fbdde7fc700 1 -- 172.21.15.16:0/3436114335 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fbde81971d0 con 0x7fbdd406ecc0 2024-04-22T15:57:05.796 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:57:05.796 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:57:05.796 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-22T15:57:05.797 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:57:05.797 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-22T15:57:05.797 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-04-22T15:57:05.797 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr", 2024-04-22T15:57:05.797 INFO:teuthology.orchestra.run.smithi016.stdout: "mon" 2024-04-22T15:57:05.797 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-22T15:57:05.797 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "10/23 daemons upgraded", 2024-04-22T15:57:05.797 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading osd daemons", 2024-04-22T15:57:05.797 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-22T15:57:05.797 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:57:05.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.796+0000 7fbdefa55700 1 -- 172.21.15.16:0/3436114335 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fbdd406ecc0 msgr2=0x7fbdd4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:05.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.796+0000 7fbdefa55700 1 --2- 172.21.15.16:0/3436114335 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fbdd406ecc0 0x7fbdd4071180 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8006010 tx=0x7fbdd801a040 comp rx=0 tx=0).stop 2024-04-22T15:57:05.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.796+0000 7fbdefa55700 1 -- 172.21.15.16:0/3436114335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbde8101850 msgr2=0x7fbde819cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:05.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.796+0000 7fbdefa55700 1 --2- 172.21.15.16:0/3436114335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbde8101850 0x7fbde819cb90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbde400c8f0 tx=0x7fbde400cc00 comp rx=0 tx=0).stop 2024-04-22T15:57:05.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.796+0000 7fbdefa55700 1 -- 172.21.15.16:0/3436114335 shutdown_connections 2024-04-22T15:57:05.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.796+0000 7fbdefa55700 1 --2- 172.21.15.16:0/3436114335 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fbdd406ecc0 0x7fbdd4071180 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:05.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.796+0000 7fbdefa55700 1 --2- 172.21.15.16:0/3436114335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbde8100ea0 0x7fbde819c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:05.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.796+0000 7fbdefa55700 1 --2- 172.21.15.16:0/3436114335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbde8101850 0x7fbde819cb90 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:05.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.796+0000 7fbdefa55700 1 -- 172.21.15.16:0/3436114335 >> 172.21.15.16:0/3436114335 conn(0x7fbde80fc750 msgr2=0x7fbde80fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:05.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.796+0000 7fbdefa55700 1 -- 172.21.15.16:0/3436114335 shutdown_connections 2024-04-22T15:57:05.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.796+0000 7fbdefa55700 1 -- 172.21.15.16:0/3436114335 wait complete. 2024-04-22T15:57:05.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.905+0000 7f0f7646c700 1 -- 172.21.15.16:0/672370963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f70100340 msgr2=0x7f0f7010d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:05.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.905+0000 7f0f7646c700 1 --2- 172.21.15.16:0/672370963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f70100340 0x7f0f7010d000 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0f60009910 tx=0x7f0f60009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:05.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.905+0000 7f0f7646c700 1 -- 172.21.15.16:0/672370963 shutdown_connections 2024-04-22T15:57:05.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.905+0000 7f0f7646c700 1 --2- 172.21.15.16:0/672370963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f70100340 0x7f0f7010d000 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:05.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.905+0000 7f0f7646c700 1 --2- 172.21.15.16:0/672370963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f700ff990 0x7f0f700ffd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:05.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.905+0000 7f0f7646c700 1 -- 172.21.15.16:0/672370963 >> 172.21.15.16:0/672370963 conn(0x7f0f700fb650 msgr2=0x7f0f700fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:05.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.905+0000 7f0f7646c700 1 -- 172.21.15.16:0/672370963 shutdown_connections 2024-04-22T15:57:05.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.905+0000 7f0f7646c700 1 -- 172.21.15.16:0/672370963 wait complete. 2024-04-22T15:57:05.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.905+0000 7f0f7646c700 1 Processor -- start 2024-04-22T15:57:05.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.905+0000 7f0f7646c700 1 -- start start 2024-04-22T15:57:05.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.906+0000 7f0f7646c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f700ff990 0x7f0f7019e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:05.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.906+0000 7f0f7646c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f70100340 0x7f0f7019eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:05.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.906+0000 7f0f7646c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7019f540 con 0x7f0f70100340 2024-04-22T15:57:05.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.906+0000 7f0f7646c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f701989f0 con 0x7f0f700ff990 2024-04-22T15:57:05.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.906+0000 7f0f6f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f70100340 0x7f0f7019eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:05.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.906+0000 7f0f6f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f70100340 0x7f0f7019eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52688/0 (socket says 172.21.15.16:52688) 2024-04-22T15:57:05.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.906+0000 7f0f6f7fe700 1 -- 172.21.15.16:0/3909582512 learned_addr learned my addr 172.21.15.16:0/3909582512 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:57:05.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.906+0000 7f0f6ffff700 1 --2- 172.21.15.16:0/3909582512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f700ff990 0x7f0f7019e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:05.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.906+0000 7f0f6f7fe700 1 -- 172.21.15.16:0/3909582512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f700ff990 msgr2=0x7f0f7019e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:05.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.906+0000 7f0f6f7fe700 1 --2- 172.21.15.16:0/3909582512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f700ff990 0x7f0f7019e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:05.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.906+0000 7f0f6f7fe700 1 -- 172.21.15.16:0/3909582512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f600095f0 con 0x7f0f70100340 2024-04-22T15:57:05.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.906+0000 7f0f6f7fe700 1 --2- 172.21.15.16:0/3909582512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f70100340 0x7f0f7019eeb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0f60006010 tx=0x7f0f600056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:05.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.906+0000 7f0f6d7fa700 1 -- 172.21.15.16:0/3909582512 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f6001d070 con 0x7f0f70100340 2024-04-22T15:57:05.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.907+0000 7f0f6d7fa700 1 -- 172.21.15.16:0/3909582512 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f6000f460 con 0x7f0f70100340 2024-04-22T15:57:05.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.907+0000 7f0f7646c700 1 -- 172.21.15.16:0/3909582512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f70198c70 con 0x7f0f70100340 2024-04-22T15:57:05.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.907+0000 7f0f6d7fa700 1 -- 172.21.15.16:0/3909582512 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f60021640 con 0x7f0f70100340 2024-04-22T15:57:05.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.907+0000 7f0f7646c700 1 -- 172.21.15.16:0/3909582512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f701991c0 con 0x7f0f70100340 2024-04-22T15:57:05.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.908+0000 7f0f7646c700 1 -- 172.21.15.16:0/3909582512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f7010a770 con 0x7f0f70100340 2024-04-22T15:57:05.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.908+0000 7f0f6d7fa700 1 -- 172.21.15.16:0/3909582512 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0f6000f5d0 con 0x7f0f70100340 2024-04-22T15:57:05.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.909+0000 7f0f6d7fa700 1 --2- 172.21.15.16:0/3909582512 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0f5c06ecc0 0x7f0f5c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:05.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.909+0000 7f0f6ffff700 1 --2- 172.21.15.16:0/3909582512 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0f5c06ecc0 0x7f0f5c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:05.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.909+0000 7f0f6d7fa700 1 -- 172.21.15.16:0/3909582512 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f0f60090080 con 0x7f0f70100340 2024-04-22T15:57:05.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.909+0000 7f0f6ffff700 1 --2- 172.21.15.16:0/3909582512 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0f5c06ecc0 0x7f0f5c071180 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0f58006fd0 tx=0x7f0f58008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:05.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:05.913+0000 7f0f6d7fa700 1 -- 172.21.15.16:0/3909582512 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0f6005ca10 con 0x7f0f70100340 2024-04-22T15:57:06.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:06.214+0000 7f0f7646c700 1 -- 172.21.15.16:0/3909582512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0f7004f2e0 con 0x7f0f70100340 2024-04-22T15:57:06.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:06.214+0000 7f0f6d7fa700 1 -- 172.21.15.16:0/3909582512 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f0f6005c5a0 con 0x7f0f70100340 2024-04-22T15:57:06.218 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:57:06.218 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:57:06.218 INFO:teuthology.orchestra.run.smithi016.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:57:06.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:06.216+0000 7f0f7646c700 1 -- 172.21.15.16:0/3909582512 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0f5c06ecc0 msgr2=0x7f0f5c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:06.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:06.216+0000 7f0f7646c700 1 --2- 172.21.15.16:0/3909582512 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0f5c06ecc0 0x7f0f5c071180 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0f58006fd0 tx=0x7f0f58008040 comp rx=0 tx=0).stop 2024-04-22T15:57:06.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:06.216+0000 7f0f7646c700 1 -- 172.21.15.16:0/3909582512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f70100340 msgr2=0x7f0f7019eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:06.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:06.216+0000 7f0f7646c700 1 --2- 172.21.15.16:0/3909582512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f70100340 0x7f0f7019eeb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0f60006010 tx=0x7f0f600056a0 comp rx=0 tx=0).stop 2024-04-22T15:57:06.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:06.217+0000 7f0f7646c700 1 -- 172.21.15.16:0/3909582512 shutdown_connections 2024-04-22T15:57:06.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:06.217+0000 7f0f7646c700 1 --2- 172.21.15.16:0/3909582512 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0f5c06ecc0 0x7f0f5c071180 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:06.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:06.217+0000 7f0f7646c700 1 --2- 172.21.15.16:0/3909582512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f700ff990 0x7f0f7019e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:06.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:06.217+0000 7f0f7646c700 1 --2- 172.21.15.16:0/3909582512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f70100340 0x7f0f7019eeb0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:06.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:06.217+0000 7f0f7646c700 1 -- 172.21.15.16:0/3909582512 >> 172.21.15.16:0/3909582512 conn(0x7f0f700fb650 msgr2=0x7f0f700fcd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:06.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:06.217+0000 7f0f7646c700 1 -- 172.21.15.16:0/3909582512 shutdown_connections 2024-04-22T15:57:06.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:06.217+0000 7f0f7646c700 1 -- 172.21.15.16:0/3909582512 wait complete. 2024-04-22T15:57:06.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:06 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/771908616' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:57:06.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:06 smithi027 ceph-mon[83054]: from='client.34334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:06.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:06 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/3909582512' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:57:06.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:06 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/771908616' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:57:06.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:06 smithi016 ceph-mon[110740]: from='client.34334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:06.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:06 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/3909582512' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:57:07.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:07 smithi027 ceph-mon[83054]: pgmap v131: 65 pgs: 65 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-22T15:57:07.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:07 smithi016 ceph-mon[110740]: pgmap v131: 65 pgs: 65 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-22T15:57:09.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:09 smithi027 ceph-mon[83054]: pgmap v132: 65 pgs: 65 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 938 B/s rd, 1 op/s 2024-04-22T15:57:09.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:09 smithi016 ceph-mon[110740]: pgmap v132: 65 pgs: 65 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 938 B/s rd, 1 op/s 2024-04-22T15:57:11.402 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:11 smithi027 ceph-mon[83054]: pgmap v133: 65 pgs: 65 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s 2024-04-22T15:57:11.402 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:11 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-22T15:57:11.402 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:11 smithi027 ceph-mon[83054]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-22T15:57:11.402 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:11 smithi027 ceph-mon[83054]: Upgrade: osd.4 is safe to restart 2024-04-22T15:57:11.402 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:11 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:11.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:11 smithi016 ceph-mon[110740]: pgmap v133: 65 pgs: 65 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s 2024-04-22T15:57:11.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:11 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-22T15:57:11.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:11 smithi016 ceph-mon[110740]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-22T15:57:11.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:11 smithi016 ceph-mon[110740]: Upgrade: osd.4 is safe to restart 2024-04-22T15:57:11.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:11 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:13.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:12 smithi027 ceph-mon[83054]: Upgrade: Updating osd.4 2024-04-22T15:57:13.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:12 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:13.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:12 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T15:57:13.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:12 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:57:13.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:12 smithi027 ceph-mon[83054]: Deploying daemon osd.4 on smithi027 2024-04-22T15:57:13.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:12 smithi016 ceph-mon[110740]: Upgrade: Updating osd.4 2024-04-22T15:57:13.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:12 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:13.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:12 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T15:57:13.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:12 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:57:13.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:12 smithi016 ceph-mon[110740]: Deploying daemon osd.4 on smithi027 2024-04-22T15:57:14.227 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:13 smithi027 ceph-mon[83054]: pgmap v134: 65 pgs: 65 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:57:14.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:13 smithi016 ceph-mon[110740]: pgmap v134: 65 pgs: 65 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:57:15.586 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:15 smithi027 systemd[1]: Stopping Ceph osd.4 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T15:57:15.985 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:15 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[47751]: 2024-04-22T15:57:15.630+0000 7f321b677700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T15:57:15.986 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:15 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[47751]: 2024-04-22T15:57:15.630+0000 7f321b677700 -1 osd.4 69 *** Got signal Terminated *** 2024-04-22T15:57:15.986 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:15 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[47751]: 2024-04-22T15:57:15.630+0000 7f321b677700 -1 osd.4 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T15:57:16.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:15 smithi027 ceph-mon[83054]: pgmap v135: 65 pgs: 65 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:57:16.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:15 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:16.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:15 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:57:16.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:15 smithi027 ceph-mon[83054]: osd.4 marked itself down and dead 2024-04-22T15:57:16.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:15 smithi016 ceph-mon[110740]: pgmap v135: 65 pgs: 65 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:57:16.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:15 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:16.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:15 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:57:16.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:15 smithi016 ceph-mon[110740]: osd.4 marked itself down and dead 2024-04-22T15:57:17.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:16 smithi027 ceph-mon[83054]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T15:57:17.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:16 smithi027 ceph-mon[83054]: osdmap e70: 6 total, 5 up, 6 in 2024-04-22T15:57:17.087 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:16 smithi027 podman[97111]: 2024-04-22 15:57:16.818909742 +0000 UTC m=+1.297345868 container died 629397d49c76b8518315447e166e20da1960271d6c1bcf3254f9b31d07ffadce (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-22T15:57:17.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:16 smithi016 ceph-mon[110740]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T15:57:17.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:16 smithi016 ceph-mon[110740]: osdmap e70: 6 total, 5 up, 6 in 2024-04-22T15:57:17.475 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:17 smithi027 podman[97111]: 2024-04-22 15:57:17.180453285 +0000 UTC m=+1.658889401 container cleanup 629397d49c76b8518315447e166e20da1960271d6c1bcf3254f9b31d07ffadce (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=HEAD) 2024-04-22T15:57:17.475 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:17 smithi027 bash[97111]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4 2024-04-22T15:57:17.822 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:17 smithi027 podman[97129]: 2024-04-22 15:57:17.47342735 +0000 UTC m=+0.648171646 container remove 629397d49c76b8518315447e166e20da1960271d6c1bcf3254f9b31d07ffadce (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, RELEASE=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222) 2024-04-22T15:57:18.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:17 smithi027 ceph-mon[83054]: pgmap v136: 65 pgs: 65 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:57:18.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:17 smithi027 ceph-mon[83054]: osdmap e71: 6 total, 5 up, 6 in 2024-04-22T15:57:18.087 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:17 smithi027 podman[97218]: 2024-04-22 15:57:17.723932582 +0000 UTC m=+0.018679056 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:57:18.087 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:17 smithi027 podman[97218]: 2024-04-22 15:57:17.824333762 +0000 UTC m=+0.119080218 container create b0a58bed82281eb5c72f2c8c4d4c5ad8f24369c31e644028d67f07802db6f2fa (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-22T15:57:18.087 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:18 smithi027 podman[97218]: 2024-04-22 15:57:18.027005634 +0000 UTC m=+0.321752093 container init b0a58bed82281eb5c72f2c8c4d4c5ad8f24369c31e644028d67f07802db6f2fa (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T15:57:18.087 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:18 smithi027 podman[97218]: 2024-04-22 15:57:18.030357152 +0000 UTC m=+0.325103609 container start b0a58bed82281eb5c72f2c8c4d4c5ad8f24369c31e644028d67f07802db6f2fa (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-22T15:57:18.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:17 smithi016 ceph-mon[110740]: pgmap v136: 65 pgs: 65 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:57:18.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:17 smithi016 ceph-mon[110740]: osdmap e71: 6 total, 5 up, 6 in 2024-04-22T15:57:18.586 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:18 smithi027 podman[97218]: 2024-04-22 15:57:18.111171893 +0000 UTC m=+0.405918349 container attach b0a58bed82281eb5c72f2c8c4d4c5ad8f24369c31e644028d67f07802db6f2fa (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49) 2024-04-22T15:57:18.586 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:18 smithi027 podman[97218]: 2024-04-22 15:57:18.347924359 +0000 UTC m=+0.642670818 container died b0a58bed82281eb5c72f2c8c4d4c5ad8f24369c31e644028d67f07802db6f2fa (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD) 2024-04-22T15:57:19.044 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:18 smithi027 podman[97244]: 2024-04-22 15:57:18.847089598 +0000 UTC m=+0.493838541 container remove b0a58bed82281eb5c72f2c8c4d4c5ad8f24369c31e644028d67f07802db6f2fa (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-22T15:57:19.044 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:18 smithi027 systemd[1]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.4.service: Deactivated successfully. 2024-04-22T15:57:19.044 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:18 smithi027 systemd[1]: Stopped Ceph osd.4 for 67ab3a80-00be-11ef-bc93-c7b262605968. 2024-04-22T15:57:19.044 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:18 smithi027 systemd[1]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.4.service: Consumed 54.110s CPU time. 2024-04-22T15:57:19.319 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:19 smithi027 ceph-mon[83054]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-22T15:57:19.319 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:19 smithi027 systemd[1]: Starting Ceph osd.4 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T15:57:19.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:19 smithi016 ceph-mon[110740]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-22T15:57:19.586 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:19 smithi027 podman[97350]: 2024-04-22 15:57:19.219912021 +0000 UTC m=+0.018835423 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:57:19.586 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:19 smithi027 podman[97350]: 2024-04-22 15:57:19.331382233 +0000 UTC m=+0.130305621 container create 5ead97ef5bcbf7967c6358c9d20e5b5596e04814d269926c4226f3a4ba05aeed (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-activate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-22T15:57:19.586 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:19 smithi027 podman[97350]: 2024-04-22 15:57:19.525680389 +0000 UTC m=+0.324603772 container init 5ead97ef5bcbf7967c6358c9d20e5b5596e04814d269926c4226f3a4ba05aeed (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T15:57:19.586 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:19 smithi027 podman[97350]: 2024-04-22 15:57:19.528858439 +0000 UTC m=+0.327781823 container start 5ead97ef5bcbf7967c6358c9d20e5b5596e04814d269926c4226f3a4ba05aeed (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-activate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T15:57:20.034 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:19 smithi027 podman[97350]: 2024-04-22 15:57:19.675579533 +0000 UTC m=+0.474503162 container attach 5ead97ef5bcbf7967c6358c9d20e5b5596e04814d269926c4226f3a4ba05aeed (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-activate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-22T15:57:20.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:20 smithi027 ceph-mon[83054]: pgmap v139: 65 pgs: 10 peering, 4 stale+active+clean, 51 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 639 B/s rd, 1 op/s 2024-04-22T15:57:20.337 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:20 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-activate[97365]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:57:20.337 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:20 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-activate[97365]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:57:20.337 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:20 smithi027 bash[97350]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:57:20.337 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:20 smithi027 bash[97350]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:57:20.337 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:20 smithi027 bash[97350]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-22T15:57:20.337 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:20 smithi027 bash[97350]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-22T15:57:20.337 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:20 smithi027 bash[97350]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-22T15:57:20.337 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:20 smithi027 bash[97350]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-22T15:57:20.337 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:20 smithi027 bash[97350]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-04-22T15:57:20.337 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:20 smithi027 bash[97350]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-22T15:57:20.337 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:20 smithi027 bash[97350]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-22T15:57:20.338 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:20 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-activate[97365]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-22T15:57:20.338 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:20 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-activate[97365]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-22T15:57:20.338 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:20 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-activate[97365]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-22T15:57:20.338 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:20 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-activate[97365]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-22T15:57:20.338 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:20 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-activate[97365]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-04-22T15:57:20.338 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:20 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-activate[97365]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-22T15:57:20.338 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:20 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-activate[97365]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-22T15:57:20.338 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:20 smithi027 podman[97350]: 2024-04-22 15:57:20.058845331 +0000 UTC m=+0.857768706 container died 5ead97ef5bcbf7967c6358c9d20e5b5596e04814d269926c4226f3a4ba05aeed (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-activate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True) 2024-04-22T15:57:20.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:20 smithi016 ceph-mon[110740]: pgmap v139: 65 pgs: 10 peering, 4 stale+active+clean, 51 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 639 B/s rd, 1 op/s 2024-04-22T15:57:20.917 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:20 smithi027 podman[97492]: 2024-04-22 15:57:20.594391344 +0000 UTC m=+0.530163691 container remove 5ead97ef5bcbf7967c6358c9d20e5b5596e04814d269926c4226f3a4ba05aeed (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-activate, org.label-schema.build-date=20240416, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:57:20.917 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:20 smithi027 podman[97558]: 2024-04-22 15:57:20.797542865 +0000 UTC m=+0.101545614 container create 93b660e5ecb615a1808923519ebfb6684cd16c14ff1969e5be5b346d78b32bab (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True) 2024-04-22T15:57:20.917 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:20 smithi027 podman[97558]: 2024-04-22 15:57:20.714586852 +0000 UTC m=+0.018589613 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:57:21.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:21 smithi027 ceph-mon[83054]: Health check failed: Degraded data redundancy: 22/258 objects degraded (8.527%), 5 pgs degraded (PG_DEGRADED) 2024-04-22T15:57:21.336 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:21 smithi027 podman[97558]: 2024-04-22 15:57:21.008462187 +0000 UTC m=+0.312464942 container init 93b660e5ecb615a1808923519ebfb6684cd16c14ff1969e5be5b346d78b32bab (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416) 2024-04-22T15:57:21.336 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:21 smithi027 podman[97558]: 2024-04-22 15:57:21.011781508 +0000 UTC m=+0.315784255 container start 93b660e5ecb615a1808923519ebfb6684cd16c14ff1969e5be5b346d78b32bab (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416) 2024-04-22T15:57:21.336 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:21 smithi027 ceph-osd[97577]: -- 172.21.15.27:0/1532070467 <== mon.1 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5638c07c3680 con 0x5638c078d000 2024-04-22T15:57:21.337 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:21 smithi027 bash[97558]: 93b660e5ecb615a1808923519ebfb6684cd16c14ff1969e5be5b346d78b32bab 2024-04-22T15:57:21.337 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:21 smithi027 systemd[1]: Started Ceph osd.4 for 67ab3a80-00be-11ef-bc93-c7b262605968. 2024-04-22T15:57:21.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:21 smithi016 ceph-mon[110740]: Health check failed: Degraded data redundancy: 22/258 objects degraded (8.527%), 5 pgs degraded (PG_DEGRADED) 2024-04-22T15:57:21.836 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:21 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[97573]: 2024-04-22T15:57:21.548+0000 7f818c6a77c0 -1 Falling back to public interface 2024-04-22T15:57:22.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:22 smithi016 ceph-mon[110740]: pgmap v140: 65 pgs: 5 active+undersized+degraded, 10 peering, 3 stale+active+clean, 6 active+undersized, 41 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 639 B/s rd, 1 op/s; 22/258 objects degraded (8.527%) 2024-04-22T15:57:22.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:22 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:22.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:22 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:22.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:22 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:57:22.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:22 smithi027 ceph-mon[83054]: pgmap v140: 65 pgs: 5 active+undersized+degraded, 10 peering, 3 stale+active+clean, 6 active+undersized, 41 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 639 B/s rd, 1 op/s; 22/258 objects degraded (8.527%) 2024-04-22T15:57:22.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:22 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:22.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:22 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:22.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:22 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:57:23.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:23 smithi027 ceph-mon[83054]: pgmap v141: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 44/258 objects degraded (17.054%) 2024-04-22T15:57:23.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:23 smithi027 ceph-mon[83054]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-22T15:57:23.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:23 smithi016 ceph-mon[110740]: pgmap v141: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 44/258 objects degraded (17.054%) 2024-04-22T15:57:23.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:23 smithi016 ceph-mon[110740]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-22T15:57:25.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:25 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:25.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:25 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:25.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:25 smithi016 ceph-mon[110740]: pgmap v142: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 44/258 objects degraded (17.054%) 2024-04-22T15:57:25.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:25 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:25.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:25 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:25.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:25 smithi027 ceph-mon[83054]: pgmap v142: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 44/258 objects degraded (17.054%) 2024-04-22T15:57:26.336 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:26 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[97573]: 2024-04-22T15:57:26.101+0000 7f818c6a77c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-04-22T15:57:26.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:26 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:26.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:26 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:26.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:26 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:26.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:26 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:26.837 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:26 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[97573]: 2024-04-22T15:57:26.497+0000 7f818c6a77c0 -1 osd.4 69 log_to_monitors true 2024-04-22T15:57:27.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:27 smithi027 ceph-mon[83054]: pgmap v143: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 44/258 objects degraded (17.054%) 2024-04-22T15:57:27.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:27 smithi027 ceph-mon[83054]: from='osd.4 [v2:172.21.15.27:6808/84770413,v1:172.21.15.27:6809/84770413]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T15:57:27.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:27 smithi027 ceph-mon[83054]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T15:57:27.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:27 smithi027 ceph-mon[83054]: Health check update: Degraded data redundancy: 44/258 objects degraded (17.054%), 15 pgs degraded (PG_DEGRADED) 2024-04-22T15:57:27.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:27 smithi016 ceph-mon[110740]: pgmap v143: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 250 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 44/258 objects degraded (17.054%) 2024-04-22T15:57:27.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:27 smithi016 ceph-mon[110740]: from='osd.4 [v2:172.21.15.27:6808/84770413,v1:172.21.15.27:6809/84770413]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T15:57:27.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:27 smithi016 ceph-mon[110740]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T15:57:27.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:27 smithi016 ceph-mon[110740]: Health check update: Degraded data redundancy: 44/258 objects degraded (17.054%), 15 pgs degraded (PG_DEGRADED) 2024-04-22T15:57:28.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:28 smithi027 ceph-mon[83054]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T15:57:28.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:28 smithi027 ceph-mon[83054]: osdmap e72: 6 total, 5 up, 6 in 2024-04-22T15:57:28.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:28 smithi027 ceph-mon[83054]: from='osd.4 [v2:172.21.15.27:6808/84770413,v1:172.21.15.27:6809/84770413]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-22T15:57:28.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:28 smithi027 ceph-mon[83054]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-22T15:57:28.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:28 smithi027 ceph-mon[83054]: from='osd.4 ' entity='osd.4' 2024-04-22T15:57:28.587 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 15:57:28 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[97573]: 2024-04-22T15:57:28.303+0000 7f8179af7700 -1 osd.4 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T15:57:28.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:28 smithi016 ceph-mon[110740]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T15:57:28.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:28 smithi016 ceph-mon[110740]: osdmap e72: 6 total, 5 up, 6 in 2024-04-22T15:57:28.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:28 smithi016 ceph-mon[110740]: from='osd.4 [v2:172.21.15.27:6808/84770413,v1:172.21.15.27:6809/84770413]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-22T15:57:28.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:28 smithi016 ceph-mon[110740]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-22T15:57:28.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:28 smithi016 ceph-mon[110740]: from='osd.4 ' entity='osd.4' 2024-04-22T15:57:30.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:30 smithi027 ceph-mon[83054]: pgmap v145: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 250 MiB data, 1010 MiB used, 535 GiB / 536 GiB avail; 44/258 objects degraded (17.054%) 2024-04-22T15:57:30.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:30 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:30.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:30 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:30.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:30 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:57:30.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:30 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:57:30.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:30 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:30.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:30 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:57:30.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:30 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:57:30.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:30 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:57:30.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:30 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:57:30.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:30 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-22T15:57:30.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:30 smithi027 ceph-mon[83054]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T15:57:30.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:30 smithi027 ceph-mon[83054]: osd.4 [v2:172.21.15.27:6808/84770413,v1:172.21.15.27:6809/84770413] boot 2024-04-22T15:57:30.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:30 smithi027 ceph-mon[83054]: osdmap e73: 6 total, 6 up, 6 in 2024-04-22T15:57:30.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:30 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:57:30.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:30 smithi016 ceph-mon[110740]: pgmap v145: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 250 MiB data, 1010 MiB used, 535 GiB / 536 GiB avail; 44/258 objects degraded (17.054%) 2024-04-22T15:57:30.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:30.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:30.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:57:30.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:57:30.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:30.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:57:30.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:57:30.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:57:30.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:57:30.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-22T15:57:30.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:30 smithi016 ceph-mon[110740]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T15:57:30.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:30 smithi016 ceph-mon[110740]: osd.4 [v2:172.21.15.27:6808/84770413,v1:172.21.15.27:6809/84770413] boot 2024-04-22T15:57:30.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:30 smithi016 ceph-mon[110740]: osdmap e73: 6 total, 6 up, 6 in 2024-04-22T15:57:30.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T15:57:31.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:31 smithi027 ceph-mon[83054]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-22T15:57:31.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:31 smithi027 ceph-mon[83054]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-22T15:57:31.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:31 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:31.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:31 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:57:31.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:31 smithi027 ceph-mon[83054]: osdmap e74: 6 total, 6 up, 6 in 2024-04-22T15:57:31.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:31 smithi016 ceph-mon[110740]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-22T15:57:31.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:31 smithi016 ceph-mon[110740]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-22T15:57:31.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:31 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:31.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:31 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:57:31.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:31 smithi016 ceph-mon[110740]: osdmap e74: 6 total, 6 up, 6 in 2024-04-22T15:57:32.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:32 smithi027 ceph-mon[83054]: pgmap v147: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 250 MiB data, 1010 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 44/258 objects degraded (17.054%) 2024-04-22T15:57:32.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:32 smithi016 ceph-mon[110740]: pgmap v147: 65 pgs: 15 active+undersized+degraded, 19 active+undersized, 31 active+clean; 250 MiB data, 1010 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 44/258 objects degraded (17.054%) 2024-04-22T15:57:33.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:33 smithi016 ceph-mon[110740]: pgmap v149: 65 pgs: 6 peering, 7 active+undersized+degraded, 9 active+undersized, 43 active+clean; 250 MiB data, 1010 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 27/258 objects degraded (10.465%) 2024-04-22T15:57:33.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:33 smithi016 ceph-mon[110740]: Health check update: Degraded data redundancy: 27/258 objects degraded (10.465%), 7 pgs degraded (PG_DEGRADED) 2024-04-22T15:57:33.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:33 smithi027 ceph-mon[83054]: pgmap v149: 65 pgs: 6 peering, 7 active+undersized+degraded, 9 active+undersized, 43 active+clean; 250 MiB data, 1010 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 27/258 objects degraded (10.465%) 2024-04-22T15:57:33.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:33 smithi027 ceph-mon[83054]: Health check update: Degraded data redundancy: 27/258 objects degraded (10.465%), 7 pgs degraded (PG_DEGRADED) 2024-04-22T15:57:34.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:34 smithi016 ceph-mon[110740]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 27/258 objects degraded (10.465%), 7 pgs degraded) 2024-04-22T15:57:34.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:34 smithi027 ceph-mon[83054]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 27/258 objects degraded (10.465%), 7 pgs degraded) 2024-04-22T15:57:35.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:35 smithi016 ceph-mon[110740]: pgmap v150: 65 pgs: 6 peering, 59 active+clean; 250 MiB data, 1014 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:57:35.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:35 smithi027 ceph-mon[83054]: pgmap v150: 65 pgs: 6 peering, 59 active+clean; 250 MiB data, 1014 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:57:36.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.340+0000 7fddf7754700 1 -- 172.21.15.16:0/542778918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf0100510 msgr2=0x7fddf01049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:36.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.340+0000 7fddf7754700 1 --2- 172.21.15.16:0/542778918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf0100510 0x7fddf01049e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009910 tx=0x7fdde0009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:36.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.340+0000 7fddf7754700 1 -- 172.21.15.16:0/542778918 shutdown_connections 2024-04-22T15:57:36.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.340+0000 7fddf7754700 1 --2- 172.21.15.16:0/542778918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf0100510 0x7fddf01049e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.340+0000 7fddf7754700 1 --2- 172.21.15.16:0/542778918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddf00ffbf0 0x7fddf00fffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.340+0000 7fddf7754700 1 -- 172.21.15.16:0/542778918 >> 172.21.15.16:0/542778918 conn(0x7fddf00fb650 msgr2=0x7fddf00fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:36.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.340+0000 7fddf7754700 1 -- 172.21.15.16:0/542778918 shutdown_connections 2024-04-22T15:57:36.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.340+0000 7fddf7754700 1 -- 172.21.15.16:0/542778918 wait complete. 2024-04-22T15:57:36.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.341+0000 7fddf7754700 1 Processor -- start 2024-04-22T15:57:36.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.341+0000 7fddf7754700 1 -- start start 2024-04-22T15:57:36.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.341+0000 7fddf7754700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf00ffbf0 0x7fddf019cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:36.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.341+0000 7fddf7754700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddf0100510 0x7fddf019d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:36.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.341+0000 7fddf7754700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf019db50 con 0x7fddf00ffbf0 2024-04-22T15:57:36.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.341+0000 7fddf7754700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf019dc90 con 0x7fddf0100510 2024-04-22T15:57:36.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.341+0000 7fddf54f0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf00ffbf0 0x7fddf019cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:36.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.341+0000 7fddf54f0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf00ffbf0 0x7fddf019cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57328/0 (socket says 172.21.15.16:57328) 2024-04-22T15:57:36.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.341+0000 7fddf54f0700 1 -- 172.21.15.16:0/892523462 learned_addr learned my addr 172.21.15.16:0/892523462 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:57:36.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.341+0000 7fddf4cef700 1 --2- 172.21.15.16:0/892523462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddf0100510 0x7fddf019d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:36.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.341+0000 7fddf54f0700 1 -- 172.21.15.16:0/892523462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddf0100510 msgr2=0x7fddf019d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:36.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.341+0000 7fddf54f0700 1 --2- 172.21.15.16:0/892523462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddf0100510 0x7fddf019d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.341+0000 7fddf54f0700 1 -- 172.21.15.16:0/892523462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde00095f0 con 0x7fddf00ffbf0 2024-04-22T15:57:36.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.341+0000 7fddf54f0700 1 --2- 172.21.15.16:0/892523462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf00ffbf0 0x7fddf019cfc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fddec00c930 tx=0x7fddec00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:36.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.342+0000 7fdde67fc700 1 -- 172.21.15.16:0/892523462 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddec007ad0 con 0x7fddf00ffbf0 2024-04-22T15:57:36.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.342+0000 7fdde67fc700 1 -- 172.21.15.16:0/892523462 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fddec007c30 con 0x7fddf00ffbf0 2024-04-22T15:57:36.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.342+0000 7fddf7754700 1 -- 172.21.15.16:0/892523462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf01a1ae0 con 0x7fddf00ffbf0 2024-04-22T15:57:36.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.342+0000 7fdde67fc700 1 -- 172.21.15.16:0/892523462 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddec018610 con 0x7fddf00ffbf0 2024-04-22T15:57:36.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.342+0000 7fddf7754700 1 -- 172.21.15.16:0/892523462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf01a2030 con 0x7fddf00ffbf0 2024-04-22T15:57:36.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.343+0000 7fddf7754700 1 -- 172.21.15.16:0/892523462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf0102090 con 0x7fddf00ffbf0 2024-04-22T15:57:36.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.343+0000 7fdde67fc700 1 -- 172.21.15.16:0/892523462 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fddec00f450 con 0x7fddf00ffbf0 2024-04-22T15:57:36.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.344+0000 7fdde67fc700 1 --2- 172.21.15.16:0/892523462 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdddc06ecc0 0x7fdddc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:36.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.344+0000 7fdde67fc700 1 -- 172.21.15.16:0/892523462 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fddec08efd0 con 0x7fddf00ffbf0 2024-04-22T15:57:36.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.344+0000 7fddf4cef700 1 --2- 172.21.15.16:0/892523462 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdddc06ecc0 0x7fdddc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:36.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.344+0000 7fddf4cef700 1 --2- 172.21.15.16:0/892523462 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdddc06ecc0 0x7fdddc071180 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fddf019e5e0 tx=0x7fdde001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.349+0000 7fdde67fc700 1 -- 172.21.15.16:0/892523462 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fddec057e30 con 0x7fddf00ffbf0 2024-04-22T15:57:36.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.579+0000 7fddf7754700 1 -- 172.21.15.16:0/892523462 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fddf019e460 con 0x7fdddc06ecc0 2024-04-22T15:57:36.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.580+0000 7fdde67fc700 1 -- 172.21.15.16:0/892523462 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fdde00056e0 con 0x7fdddc06ecc0 2024-04-22T15:57:36.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.582+0000 7fddf7754700 1 -- 172.21.15.16:0/892523462 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdddc06ecc0 msgr2=0x7fdddc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:36.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.582+0000 7fddf7754700 1 --2- 172.21.15.16:0/892523462 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdddc06ecc0 0x7fdddc071180 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fddf019e5e0 tx=0x7fdde001a040 comp rx=0 tx=0).stop 2024-04-22T15:57:36.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.582+0000 7fddf7754700 1 -- 172.21.15.16:0/892523462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf00ffbf0 msgr2=0x7fddf019cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:36.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.582+0000 7fddf7754700 1 --2- 172.21.15.16:0/892523462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf00ffbf0 0x7fddf019cfc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fddec00c930 tx=0x7fddec00ccf0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.582+0000 7fddf7754700 1 -- 172.21.15.16:0/892523462 shutdown_connections 2024-04-22T15:57:36.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.582+0000 7fddf7754700 1 --2- 172.21.15.16:0/892523462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf00ffbf0 0x7fddf019cfc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.582+0000 7fddf7754700 1 --2- 172.21.15.16:0/892523462 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdddc06ecc0 0x7fdddc071180 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.582+0000 7fddf7754700 1 --2- 172.21.15.16:0/892523462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddf0100510 0x7fddf019d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.582+0000 7fddf7754700 1 -- 172.21.15.16:0/892523462 >> 172.21.15.16:0/892523462 conn(0x7fddf00fb650 msgr2=0x7fddf0111060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:36.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.582+0000 7fddf7754700 1 -- 172.21.15.16:0/892523462 shutdown_connections 2024-04-22T15:57:36.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.582+0000 7fddf7754700 1 -- 172.21.15.16:0/892523462 wait complete. 2024-04-22T15:57:36.596 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:57:36.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.690+0000 7f38af6ad700 1 -- 172.21.15.16:0/2921438497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a8100ea0 msgr2=0x7f38a8101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:36.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.690+0000 7f38af6ad700 1 --2- 172.21.15.16:0/2921438497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a8100ea0 0x7f38a8101280 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f38a4009910 tx=0x7f38a4009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:36.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.690+0000 7f38af6ad700 1 -- 172.21.15.16:0/2921438497 shutdown_connections 2024-04-22T15:57:36.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.690+0000 7f38af6ad700 1 --2- 172.21.15.16:0/2921438497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38a8101850 0x7f38a81058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.690+0000 7f38af6ad700 1 --2- 172.21.15.16:0/2921438497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a8100ea0 0x7f38a8101280 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.690+0000 7f38af6ad700 1 -- 172.21.15.16:0/2921438497 >> 172.21.15.16:0/2921438497 conn(0x7f38a80fc750 msgr2=0x7f38a80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:36.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.690+0000 7f38af6ad700 1 -- 172.21.15.16:0/2921438497 shutdown_connections 2024-04-22T15:57:36.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.690+0000 7f38af6ad700 1 -- 172.21.15.16:0/2921438497 wait complete. 2024-04-22T15:57:36.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.690+0000 7f38af6ad700 1 Processor -- start 2024-04-22T15:57:36.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.690+0000 7f38af6ad700 1 -- start start 2024-04-22T15:57:36.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.691+0000 7f38af6ad700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a8100ea0 0x7f38a819c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:36.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.691+0000 7f38af6ad700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38a8101850 0x7f38a819ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:36.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.691+0000 7f38af6ad700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a819d350 con 0x7f38a8100ea0 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.691+0000 7f38af6ad700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a8196800 con 0x7f38a8101850 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.691+0000 7f38ad449700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a8100ea0 0x7f38a819c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.691+0000 7f38ad449700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a8100ea0 0x7f38a819c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57344/0 (socket says 172.21.15.16:57344) 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.691+0000 7f38ad449700 1 -- 172.21.15.16:0/3718141909 learned_addr learned my addr 172.21.15.16:0/3718141909 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.691+0000 7f38acc48700 1 --2- 172.21.15.16:0/3718141909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38a8101850 0x7f38a819ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.691+0000 7f38ad449700 1 -- 172.21.15.16:0/3718141909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38a8101850 msgr2=0x7f38a819ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.691+0000 7f38ad449700 1 --2- 172.21.15.16:0/3718141909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38a8101850 0x7f38a819ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.691+0000 7f38ad449700 1 -- 172.21.15.16:0/3718141909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a40095f0 con 0x7f38a8100ea0 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.691+0000 7f38ad449700 1 --2- 172.21.15.16:0/3718141909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a8100ea0 0x7f38a819c780 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f38a4006010 tx=0x7f38a400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.691+0000 7f389e7fc700 1 -- 172.21.15.16:0/3718141909 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38a401d070 con 0x7f38a8100ea0 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.691+0000 7f389e7fc700 1 -- 172.21.15.16:0/3718141909 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f38a400f460 con 0x7f38a8100ea0 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.691+0000 7f38af6ad700 1 -- 172.21.15.16:0/3718141909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a8196a80 con 0x7f38a8100ea0 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.691+0000 7f389e7fc700 1 -- 172.21.15.16:0/3718141909 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38a4017610 con 0x7f38a8100ea0 2024-04-22T15:57:36.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.692+0000 7f38af6ad700 1 -- 172.21.15.16:0/3718141909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a8196f70 con 0x7f38a8100ea0 2024-04-22T15:57:36.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.692+0000 7f38af6ad700 1 -- 172.21.15.16:0/3718141909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a8109240 con 0x7f38a8100ea0 2024-04-22T15:57:36.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.693+0000 7f389e7fc700 1 -- 172.21.15.16:0/3718141909 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f38a400f5d0 con 0x7f38a8100ea0 2024-04-22T15:57:36.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.694+0000 7f389e7fc700 1 --2- 172.21.15.16:0/3718141909 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f389406ecc0 0x7f3894071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:36.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.694+0000 7f389e7fc700 1 -- 172.21.15.16:0/3718141909 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f38a4090250 con 0x7f38a8100ea0 2024-04-22T15:57:36.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.694+0000 7f38acc48700 1 --2- 172.21.15.16:0/3718141909 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f389406ecc0 0x7f3894071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:36.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.694+0000 7f38acc48700 1 --2- 172.21.15.16:0/3718141909 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f389406ecc0 0x7f3894071180 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f38a80fd010 tx=0x7f3898009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:36.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.699+0000 7f389e7fc700 1 -- 172.21.15.16:0/3718141909 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f38a405cc60 con 0x7f38a8100ea0 2024-04-22T15:57:36.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.929+0000 7f38af6ad700 1 -- 172.21.15.16:0/3718141909 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f38a81972a0 con 0x7f389406ecc0 2024-04-22T15:57:36.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.930+0000 7f389e7fc700 1 -- 172.21.15.16:0/3718141909 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f38a81972a0 con 0x7f389406ecc0 2024-04-22T15:57:36.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.932+0000 7f38af6ad700 1 -- 172.21.15.16:0/3718141909 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f389406ecc0 msgr2=0x7f3894071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.932+0000 7f38af6ad700 1 --2- 172.21.15.16:0/3718141909 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f389406ecc0 0x7f3894071180 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f38a80fd010 tx=0x7f3898009190 comp rx=0 tx=0).stop 2024-04-22T15:57:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.932+0000 7f38af6ad700 1 -- 172.21.15.16:0/3718141909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a8100ea0 msgr2=0x7f38a819c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.932+0000 7f38af6ad700 1 --2- 172.21.15.16:0/3718141909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a8100ea0 0x7f38a819c780 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f38a4006010 tx=0x7f38a400bd70 comp rx=0 tx=0).stop 2024-04-22T15:57:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.932+0000 7f38af6ad700 1 -- 172.21.15.16:0/3718141909 shutdown_connections 2024-04-22T15:57:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.932+0000 7f38af6ad700 1 --2- 172.21.15.16:0/3718141909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38a8100ea0 0x7f38a819c780 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.932+0000 7f38af6ad700 1 --2- 172.21.15.16:0/3718141909 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f389406ecc0 0x7f3894071180 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.932+0000 7f38af6ad700 1 --2- 172.21.15.16:0/3718141909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38a8101850 0x7f38a819ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.932+0000 7f38af6ad700 1 -- 172.21.15.16:0/3718141909 >> 172.21.15.16:0/3718141909 conn(0x7f38a80fc750 msgr2=0x7f38a80fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.932+0000 7f38af6ad700 1 -- 172.21.15.16:0/3718141909 shutdown_connections 2024-04-22T15:57:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:36.932+0000 7f38af6ad700 1 -- 172.21.15.16:0/3718141909 wait complete. 2024-04-22T15:57:37.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.040+0000 7f1943135700 1 -- 172.21.15.16:0/2034390471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f193c068580 msgr2=0x7f193c0689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:37.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.040+0000 7f1943135700 1 --2- 172.21.15.16:0/2034390471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f193c068580 0x7f193c0689e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1928009910 tx=0x7f1928009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:37.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.040+0000 7f1943135700 1 -- 172.21.15.16:0/2034390471 shutdown_connections 2024-04-22T15:57:37.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.040+0000 7f1943135700 1 --2- 172.21.15.16:0/2034390471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f193c068580 0x7f193c0689e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:37.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.040+0000 7f1943135700 1 --2- 172.21.15.16:0/2034390471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f193c106cd0 0x7f193c1070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:37.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.040+0000 7f1943135700 1 -- 172.21.15.16:0/2034390471 >> 172.21.15.16:0/2034390471 conn(0x7f193c075670 msgr2=0x7f193c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:37.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.040+0000 7f1943135700 1 -- 172.21.15.16:0/2034390471 shutdown_connections 2024-04-22T15:57:37.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.040+0000 7f1943135700 1 -- 172.21.15.16:0/2034390471 wait complete. 2024-04-22T15:57:37.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.041+0000 7f1943135700 1 Processor -- start 2024-04-22T15:57:37.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.041+0000 7f1943135700 1 -- start start 2024-04-22T15:57:37.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.041+0000 7f1943135700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f193c068580 0x7f193c19c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:37.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.041+0000 7f1943135700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f193c106cd0 0x7f193c19cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:37.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.041+0000 7f1943135700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c19d120 con 0x7f193c068580 2024-04-22T15:57:37.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.041+0000 7f1943135700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c1966b0 con 0x7f193c106cd0 2024-04-22T15:57:37.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.041+0000 7f1940ed1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f193c068580 0x7f193c19c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:37.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.041+0000 7f1940ed1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f193c068580 0x7f193c19c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57370/0 (socket says 172.21.15.16:57370) 2024-04-22T15:57:37.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.041+0000 7f1940ed1700 1 -- 172.21.15.16:0/2193060007 learned_addr learned my addr 172.21.15.16:0/2193060007 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:57:37.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.041+0000 7f193bfff700 1 --2- 172.21.15.16:0/2193060007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f193c106cd0 0x7f193c19cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:37.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.042+0000 7f1940ed1700 1 -- 172.21.15.16:0/2193060007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f193c106cd0 msgr2=0x7f193c19cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:37.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.042+0000 7f1940ed1700 1 --2- 172.21.15.16:0/2193060007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f193c106cd0 0x7f193c19cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:37.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.042+0000 7f1940ed1700 1 -- 172.21.15.16:0/2193060007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19280095f0 con 0x7f193c068580 2024-04-22T15:57:37.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.042+0000 7f193bfff700 1 --2- 172.21.15.16:0/2193060007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f193c106cd0 0x7f193c19cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:57:37.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.042+0000 7f1940ed1700 1 --2- 172.21.15.16:0/2193060007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f193c068580 0x7f193c19c5c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f193000c8a0 tx=0x7f193000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:37.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.042+0000 7f1939ffb700 1 -- 172.21.15.16:0/2193060007 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f193000cea0 con 0x7f193c068580 2024-04-22T15:57:37.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.042+0000 7f1939ffb700 1 -- 172.21.15.16:0/2193060007 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1930004830 con 0x7f193c068580 2024-04-22T15:57:37.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.042+0000 7f1943135700 1 -- 172.21.15.16:0/2193060007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f193c196990 con 0x7f193c068580 2024-04-22T15:57:37.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.042+0000 7f1939ffb700 1 -- 172.21.15.16:0/2193060007 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19300055b0 con 0x7f193c068580 2024-04-22T15:57:37.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.042+0000 7f1943135700 1 -- 172.21.15.16:0/2193060007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f193c196ee0 con 0x7f193c068580 2024-04-22T15:57:37.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.043+0000 7f1943135700 1 -- 172.21.15.16:0/2193060007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f193c1091d0 con 0x7f193c068580 2024-04-22T15:57:37.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.044+0000 7f1939ffb700 1 -- 172.21.15.16:0/2193060007 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f19300049a0 con 0x7f193c068580 2024-04-22T15:57:37.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.044+0000 7f1939ffb700 1 --2- 172.21.15.16:0/2193060007 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f192c06ecc0 0x7f192c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:37.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.044+0000 7f1939ffb700 1 -- 172.21.15.16:0/2193060007 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f193008ea70 con 0x7f193c068580 2024-04-22T15:57:37.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.044+0000 7f193bfff700 1 --2- 172.21.15.16:0/2193060007 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f192c06ecc0 0x7f192c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:37.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.045+0000 7f193bfff700 1 --2- 172.21.15.16:0/2193060007 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f192c06ecc0 0x7f192c071180 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f192800b5c0 tx=0x7f1928005480 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.049+0000 7f1939ffb700 1 -- 172.21.15.16:0/2193060007 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f193005b400 con 0x7f193c068580 2024-04-22T15:57:37.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.266+0000 7f1943135700 1 -- 172.21.15.16:0/2193060007 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f193c197160 con 0x7f192c06ecc0 2024-04-22T15:57:37.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.277+0000 7f1939ffb700 1 -- 172.21.15.16:0/2193060007 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f193c197160 con 0x7f192c06ecc0 2024-04-22T15:57:37.280 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:57:37.280 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.smithi016 smithi016 *:9093,9094 running (5m) 80s ago 16m 25.6M - 0.25.0 c8568f914cd2 ea0e0217d6f9 2024-04-22T15:57:37.280 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi016 smithi016 running (16m) 80s ago 16m 8640k - 18.2.1 5be31c24972a f8b627d3f41e 2024-04-22T15:57:37.280 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi027 smithi027 running (15m) 13s ago 15m 8442k - 18.2.1 5be31c24972a f00fd40b4220 2024-04-22T15:57:37.280 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi016 smithi016 running (3m) 80s ago 16m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 1feb008449ed 2024-04-22T15:57:37.280 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi027 smithi027 running (2m) 13s ago 15m 6656k - 19.0.0-3223-g5883d494 3d983c26e278 c7f862c6da00 2024-04-22T15:57:37.280 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.smithi016 smithi016 *:3000 running (5m) 80s ago 16m 78.9M - 9.4.12 2bacad6d85d8 388a3c9c1b3f 2024-04-22T15:57:37.280 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.fkkejm smithi016 running (11m) 80s ago 11m 17.0M - 18.2.1 5be31c24972a 4153ac94cea0 2024-04-22T15:57:37.280 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.pgvphu smithi016 running (10m) 80s ago 10m 92.0M - 18.2.1 5be31c24972a 93dc072068a2 2024-04-22T15:57:37.280 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.dgqyes smithi027 running (10m) 13s ago 10m 20.3M - 18.2.1 5be31c24972a 41a9220430fd 2024-04-22T15:57:37.280 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.iobjds smithi027 running (11m) 13s ago 11m 149M - 18.2.1 5be31c24972a c1c8d6770f02 2024-04-22T15:57:37.281 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi016.rnsnnr smithi016 *:8443,9283,8765 running (7m) 80s ago 18m 505M - 19.0.0-3223-g5883d494 3d983c26e278 09b4fe285bc4 2024-04-22T15:57:37.281 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi027.ukdgzs smithi027 *:8443,9283,8765 running (7m) 13s ago 15m 451M - 19.0.0-3223-g5883d494 3d983c26e278 7548da9b5eb9 2024-04-22T15:57:37.281 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi016 smithi016 running (4m) 80s ago 18m 49.6M 2048M 19.0.0-3223-g5883d494 3d983c26e278 4ae529a67806 2024-04-22T15:57:37.281 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi027 smithi027 running (3m) 13s ago 15m 40.6M 2048M 19.0.0-3223-g5883d494 3d983c26e278 f29fcb888801 2024-04-22T15:57:37.281 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi016 smithi016 *:9100 running (6m) 80s ago 16m 16.1M - 1.5.0 0da6a335fe13 35e46e4d2f3f 2024-04-22T15:57:37.281 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi027 smithi027 *:9100 running (6m) 13s ago 15m 15.6M - 1.5.0 0da6a335fe13 e682a7672e9b 2024-04-22T15:57:37.281 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (2m) 80s ago 14m 88.0M 1217M 19.0.0-3223-g5883d494 3d983c26e278 79217eb383a2 2024-04-22T15:57:37.281 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (2m) 80s ago 14m 89.0M 1217M 19.0.0-3223-g5883d494 3d983c26e278 ec1a45b3c53e 2024-04-22T15:57:37.281 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (86s) 80s ago 13m 12.6M 1217M 19.0.0-3223-g5883d494 3d983c26e278 48bb459fd021 2024-04-22T15:57:37.281 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi027 running (50s) 13s ago 13m 94.1M 2240M 19.0.0-3223-g5883d494 3d983c26e278 81034bf22d3f 2024-04-22T15:57:37.281 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (16s) 13s ago 12m 14.8M 2240M 19.0.0-3223-g5883d494 3d983c26e278 93b660e5ecb6 2024-04-22T15:57:37.281 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi027 running (12m) 13s ago 12m 195M 2240M 18.2.1 5be31c24972a 23adcf675b04 2024-04-22T15:57:37.281 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.smithi016 smithi016 *:9095 running (6m) 80s ago 16m 73.8M - 2.43.0 a07b618ecd1d 373eaadee82d 2024-04-22T15:57:37.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.279+0000 7f1943135700 1 -- 172.21.15.16:0/2193060007 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f192c06ecc0 msgr2=0x7f192c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:37.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.279+0000 7f1943135700 1 --2- 172.21.15.16:0/2193060007 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f192c06ecc0 0x7f192c071180 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f192800b5c0 tx=0x7f1928005480 comp rx=0 tx=0).stop 2024-04-22T15:57:37.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.279+0000 7f1943135700 1 -- 172.21.15.16:0/2193060007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f193c068580 msgr2=0x7f193c19c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:37.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.279+0000 7f1943135700 1 --2- 172.21.15.16:0/2193060007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f193c068580 0x7f193c19c5c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f193000c8a0 tx=0x7f193000cc60 comp rx=0 tx=0).stop 2024-04-22T15:57:37.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.279+0000 7f1943135700 1 -- 172.21.15.16:0/2193060007 shutdown_connections 2024-04-22T15:57:37.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.279+0000 7f1943135700 1 --2- 172.21.15.16:0/2193060007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f193c068580 0x7f193c19c5c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:37.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.279+0000 7f1943135700 1 --2- 172.21.15.16:0/2193060007 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f192c06ecc0 0x7f192c071180 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:37.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.279+0000 7f1943135700 1 --2- 172.21.15.16:0/2193060007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f193c106cd0 0x7f193c19cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:37.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.279+0000 7f1943135700 1 -- 172.21.15.16:0/2193060007 >> 172.21.15.16:0/2193060007 conn(0x7f193c075670 msgr2=0x7f193c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:37.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.279+0000 7f1943135700 1 -- 172.21.15.16:0/2193060007 shutdown_connections 2024-04-22T15:57:37.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.279+0000 7f1943135700 1 -- 172.21.15.16:0/2193060007 wait complete. 2024-04-22T15:57:37.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.385+0000 7f83a9353700 1 -- 172.21.15.16:0/1363016606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a4102700 msgr2=0x7f83a410abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:37.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.385+0000 7f83a9353700 1 --2- 172.21.15.16:0/1363016606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a4102700 0x7f83a410abf0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8394009910 tx=0x7f8394009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:37.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.385+0000 7f83a9353700 1 -- 172.21.15.16:0/1363016606 shutdown_connections 2024-04-22T15:57:37.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.385+0000 7f83a9353700 1 --2- 172.21.15.16:0/1363016606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a4102700 0x7f83a410abf0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:37.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.385+0000 7f83a9353700 1 --2- 172.21.15.16:0/1363016606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83a4101de0 0x7f83a41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:37.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.385+0000 7f83a9353700 1 -- 172.21.15.16:0/1363016606 >> 172.21.15.16:0/1363016606 conn(0x7f83a40fb650 msgr2=0x7f83a40fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:37.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.385+0000 7f83a9353700 1 -- 172.21.15.16:0/1363016606 shutdown_connections 2024-04-22T15:57:37.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.385+0000 7f83a9353700 1 -- 172.21.15.16:0/1363016606 wait complete. 2024-04-22T15:57:37.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.386+0000 7f83a9353700 1 Processor -- start 2024-04-22T15:57:37.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.386+0000 7f83a9353700 1 -- start start 2024-04-22T15:57:37.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.386+0000 7f83a9353700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83a4101de0 0x7f83a419e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:37.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.386+0000 7f83a9353700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a4102700 0x7f83a419edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:37.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.386+0000 7f83a9353700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a419f450 con 0x7f83a4102700 2024-04-22T15:57:37.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.386+0000 7f83a9353700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a4198900 con 0x7f83a4101de0 2024-04-22T15:57:37.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.387+0000 7f83a27fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a4102700 0x7f83a419edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:37.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.387+0000 7f83a27fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a4102700 0x7f83a419edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57382/0 (socket says 172.21.15.16:57382) 2024-04-22T15:57:37.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.387+0000 7f83a27fc700 1 -- 172.21.15.16:0/2625513546 learned_addr learned my addr 172.21.15.16:0/2625513546 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:57:37.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.387+0000 7f83a2ffd700 1 --2- 172.21.15.16:0/2625513546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83a4101de0 0x7f83a419e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:37.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.387+0000 7f83a27fc700 1 -- 172.21.15.16:0/2625513546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83a4101de0 msgr2=0x7f83a419e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:37.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.387+0000 7f83a27fc700 1 --2- 172.21.15.16:0/2625513546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83a4101de0 0x7f83a419e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:37.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.387+0000 7f83a27fc700 1 -- 172.21.15.16:0/2625513546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83940095f0 con 0x7f83a4102700 2024-04-22T15:57:37.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.387+0000 7f83a27fc700 1 --2- 172.21.15.16:0/2625513546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a4102700 0x7f83a419edc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f83940059d0 tx=0x7f839400b5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:37.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.387+0000 7f839bfff700 1 -- 172.21.15.16:0/2625513546 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f839401d070 con 0x7f83a4102700 2024-04-22T15:57:37.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.387+0000 7f839bfff700 1 -- 172.21.15.16:0/2625513546 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f83940052a0 con 0x7f83a4102700 2024-04-22T15:57:37.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.387+0000 7f83a9353700 1 -- 172.21.15.16:0/2625513546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83a4198b80 con 0x7f83a4102700 2024-04-22T15:57:37.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.387+0000 7f839bfff700 1 -- 172.21.15.16:0/2625513546 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8394022470 con 0x7f83a4102700 2024-04-22T15:57:37.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.387+0000 7f83a9353700 1 -- 172.21.15.16:0/2625513546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83a4199070 con 0x7f83a4102700 2024-04-22T15:57:37.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.388+0000 7f83a9353700 1 -- 172.21.15.16:0/2625513546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83a410a750 con 0x7f83a4102700 2024-04-22T15:57:37.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.389+0000 7f839bfff700 1 -- 172.21.15.16:0/2625513546 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f839400f6f0 con 0x7f83a4102700 2024-04-22T15:57:37.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.389+0000 7f839bfff700 1 --2- 172.21.15.16:0/2625513546 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f839006ecc0 0x7f8390071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:37.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.390+0000 7f839bfff700 1 -- 172.21.15.16:0/2625513546 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f839408ffe0 con 0x7f83a4102700 2024-04-22T15:57:37.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.390+0000 7f83a2ffd700 1 --2- 172.21.15.16:0/2625513546 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f839006ecc0 0x7f8390071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:37.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.390+0000 7f83a2ffd700 1 --2- 172.21.15.16:0/2625513546 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f839006ecc0 0x7f8390071180 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f838c009c60 tx=0x7f838c009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.394+0000 7f839bfff700 1 -- 172.21.15.16:0/2625513546 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f839405c9f0 con 0x7f83a4102700 2024-04-22T15:57:37.638 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:37 smithi027 ceph-mon[83054]: pgmap v151: 65 pgs: 65 active+clean; 250 MiB data, 1014 MiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2024-04-22T15:57:37.638 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:37 smithi027 ceph-mon[83054]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:37.638 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:37 smithi027 ceph-mon[83054]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:37.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:37 smithi016 ceph-mon[110740]: pgmap v151: 65 pgs: 65 active+clean; 250 MiB data, 1014 MiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2024-04-22T15:57:37.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:37 smithi016 ceph-mon[110740]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:37.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:37 smithi016 ceph-mon[110740]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:37.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.693+0000 7f83a9353700 1 -- 172.21.15.16:0/2625513546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f83a404f2e0 con 0x7f83a4102700 2024-04-22T15:57:37.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.694+0000 7f839bfff700 1 -- 172.21.15.16:0/2625513546 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f839405c580 con 0x7f83a4102700 2024-04-22T15:57:37.697 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:57:37.697 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-22T15:57:37.697 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:57:37.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:57:37.697 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-22T15:57:37.697 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:57:37.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:57:37.698 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-22T15:57:37.698 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T15:57:37.698 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 5 2024-04-22T15:57:37.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:57:37.698 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-22T15:57:37.698 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T15:57:37.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:57:37.698 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-22T15:57:37.698 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 5, 2024-04-22T15:57:37.698 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 9 2024-04-22T15:57:37.698 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-22T15:57:37.699 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:57:37.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.696+0000 7f83a9353700 1 -- 172.21.15.16:0/2625513546 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f839006ecc0 msgr2=0x7f8390071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:37.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.696+0000 7f83a9353700 1 --2- 172.21.15.16:0/2625513546 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f839006ecc0 0x7f8390071180 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f838c009c60 tx=0x7f838c009260 comp rx=0 tx=0).stop 2024-04-22T15:57:37.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.696+0000 7f83a9353700 1 -- 172.21.15.16:0/2625513546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a4102700 msgr2=0x7f83a419edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:37.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.696+0000 7f83a9353700 1 --2- 172.21.15.16:0/2625513546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a4102700 0x7f83a419edc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f83940059d0 tx=0x7f839400b5a0 comp rx=0 tx=0).stop 2024-04-22T15:57:37.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.696+0000 7f83a9353700 1 -- 172.21.15.16:0/2625513546 shutdown_connections 2024-04-22T15:57:37.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.696+0000 7f83a9353700 1 --2- 172.21.15.16:0/2625513546 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f839006ecc0 0x7f8390071180 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:37.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.696+0000 7f83a9353700 1 --2- 172.21.15.16:0/2625513546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83a4101de0 0x7f83a419e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:37.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.696+0000 7f83a9353700 1 --2- 172.21.15.16:0/2625513546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a4102700 0x7f83a419edc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:37.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.696+0000 7f83a9353700 1 -- 172.21.15.16:0/2625513546 >> 172.21.15.16:0/2625513546 conn(0x7f83a40fb650 msgr2=0x7f83a4100200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:37.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.696+0000 7f83a9353700 1 -- 172.21.15.16:0/2625513546 shutdown_connections 2024-04-22T15:57:37.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.696+0000 7f83a9353700 1 -- 172.21.15.16:0/2625513546 wait complete. 2024-04-22T15:57:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.802+0000 7ff0776ed700 1 -- 172.21.15.16:0/2165857193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff070101e60 msgr2=0x7ff070102240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.802+0000 7ff0776ed700 1 --2- 172.21.15.16:0/2165857193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff070101e60 0x7ff070102240 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff06c009910 tx=0x7ff06c009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.803+0000 7ff0776ed700 1 -- 172.21.15.16:0/2165857193 shutdown_connections 2024-04-22T15:57:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.803+0000 7ff0776ed700 1 --2- 172.21.15.16:0/2165857193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff070102780 0x7ff07010ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.803+0000 7ff0776ed700 1 --2- 172.21.15.16:0/2165857193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff070101e60 0x7ff070102240 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.803+0000 7ff0776ed700 1 -- 172.21.15.16:0/2165857193 >> 172.21.15.16:0/2165857193 conn(0x7ff0700fb650 msgr2=0x7ff0700fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.803+0000 7ff0776ed700 1 -- 172.21.15.16:0/2165857193 shutdown_connections 2024-04-22T15:57:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.803+0000 7ff0776ed700 1 -- 172.21.15.16:0/2165857193 wait complete. 2024-04-22T15:57:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.803+0000 7ff0776ed700 1 Processor -- start 2024-04-22T15:57:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.803+0000 7ff0776ed700 1 -- start start 2024-04-22T15:57:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.804+0000 7ff0776ed700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff070101e60 0x7ff0700ffac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.804+0000 7ff0776ed700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff070102780 0x7ff070100000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.804+0000 7ff0776ed700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff070100540 con 0x7ff070102780 2024-04-22T15:57:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.804+0000 7ff0776ed700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff070100680 con 0x7ff070101e60 2024-04-22T15:57:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.804+0000 7ff074c88700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff070102780 0x7ff070100000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.804+0000 7ff074c88700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff070102780 0x7ff070100000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57410/0 (socket says 172.21.15.16:57410) 2024-04-22T15:57:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.804+0000 7ff074c88700 1 -- 172.21.15.16:0/1790006485 learned_addr learned my addr 172.21.15.16:0/1790006485 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:57:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.804+0000 7ff075489700 1 --2- 172.21.15.16:0/1790006485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff070101e60 0x7ff0700ffac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.804+0000 7ff074c88700 1 -- 172.21.15.16:0/1790006485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff070101e60 msgr2=0x7ff0700ffac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.804+0000 7ff074c88700 1 --2- 172.21.15.16:0/1790006485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff070101e60 0x7ff0700ffac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.804+0000 7ff074c88700 1 -- 172.21.15.16:0/1790006485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06c0095f0 con 0x7ff070102780 2024-04-22T15:57:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.804+0000 7ff074c88700 1 --2- 172.21.15.16:0/1790006485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff070102780 0x7ff070100000 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff06000d900 tx=0x7ff06000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.804+0000 7ff0667fc700 1 -- 172.21.15.16:0/1790006485 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff060009900 con 0x7ff070102780 2024-04-22T15:57:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.804+0000 7ff0667fc700 1 -- 172.21.15.16:0/1790006485 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff060010460 con 0x7ff070102780 2024-04-22T15:57:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.804+0000 7ff0776ed700 1 -- 172.21.15.16:0/1790006485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0701a2aa0 con 0x7ff070102780 2024-04-22T15:57:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.804+0000 7ff0667fc700 1 -- 172.21.15.16:0/1790006485 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff06000f5f0 con 0x7ff070102780 2024-04-22T15:57:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.805+0000 7ff0776ed700 1 -- 172.21.15.16:0/1790006485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0701a2e50 con 0x7ff070102780 2024-04-22T15:57:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.805+0000 7ff0776ed700 1 -- 172.21.15.16:0/1790006485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff070108310 con 0x7ff070102780 2024-04-22T15:57:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.806+0000 7ff0667fc700 1 -- 172.21.15.16:0/1790006485 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff0600105d0 con 0x7ff070102780 2024-04-22T15:57:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.806+0000 7ff0667fc700 1 --2- 172.21.15.16:0/1790006485 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ff05c06ecc0 0x7ff05c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.807+0000 7ff0667fc700 1 -- 172.21.15.16:0/1790006485 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff06008e820 con 0x7ff070102780 2024-04-22T15:57:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.807+0000 7ff075489700 1 --2- 172.21.15.16:0/1790006485 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ff05c06ecc0 0x7ff05c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.807+0000 7ff075489700 1 --2- 172.21.15.16:0/1790006485 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ff05c06ecc0 0x7ff05c071180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff06c006010 tx=0x7ff06c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:37.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:37.811+0000 7ff0667fc700 1 -- 172.21.15.16:0/1790006485 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff060057700 con 0x7ff070102780 2024-04-22T15:57:38.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.067+0000 7ff0776ed700 1 -- 172.21.15.16:0/1790006485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff07004f2e0 con 0x7ff070102780 2024-04-22T15:57:38.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.068+0000 7ff0667fc700 1 -- 172.21.15.16:0/1790006485 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1902 (secure 0 0 0) 0x7ff06005adc0 con 0x7ff070102780 2024-04-22T15:57:38.071 INFO:teuthology.orchestra.run.smithi016.stdout:e16 2024-04-22T15:57:38.071 INFO:teuthology.orchestra.run.smithi016.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:57:38.071 INFO:teuthology.orchestra.run.smithi016.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:57:38.071 INFO:teuthology.orchestra.run.smithi016.stdout:legacy client fscid: 1 2024-04-22T15:57:38.072 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:57:38.072 INFO:teuthology.orchestra.run.smithi016.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:57:38.072 INFO:teuthology.orchestra.run.smithi016.stdout:fs_name cephfs 2024-04-22T15:57:38.072 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 16 2024-04-22T15:57:38.072 INFO:teuthology.orchestra.run.smithi016.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T15:57:38.072 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-04-22T15:46:27.774571+0000 2024-04-22T15:57:38.072 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-04-22T15:46:47.381709+0000 2024-04-22T15:57:38.072 INFO:teuthology.orchestra.run.smithi016.stdout:tableserver 0 2024-04-22T15:57:38.072 INFO:teuthology.orchestra.run.smithi016.stdout:root 0 2024-04-22T15:57:38.072 INFO:teuthology.orchestra.run.smithi016.stdout:session_timeout 60 2024-04-22T15:57:38.072 INFO:teuthology.orchestra.run.smithi016.stdout:session_autoclose 300 2024-04-22T15:57:38.072 INFO:teuthology.orchestra.run.smithi016.stdout:max_file_size 1099511627776 2024-04-22T15:57:38.072 INFO:teuthology.orchestra.run.smithi016.stdout:max_xattr_size 65536 2024-04-22T15:57:38.072 INFO:teuthology.orchestra.run.smithi016.stdout:required_client_features {} 2024-04-22T15:57:38.072 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure 0 2024-04-22T15:57:38.073 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure_osd_epoch 46 2024-04-22T15:57:38.073 INFO:teuthology.orchestra.run.smithi016.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:57:38.073 INFO:teuthology.orchestra.run.smithi016.stdout:max_mds 1 2024-04-22T15:57:38.073 INFO:teuthology.orchestra.run.smithi016.stdout:in 0 2024-04-22T15:57:38.073 INFO:teuthology.orchestra.run.smithi016.stdout:up {0=24279} 2024-04-22T15:57:38.073 INFO:teuthology.orchestra.run.smithi016.stdout:failed 2024-04-22T15:57:38.073 INFO:teuthology.orchestra.run.smithi016.stdout:damaged 2024-04-22T15:57:38.073 INFO:teuthology.orchestra.run.smithi016.stdout:stopped 2024-04-22T15:57:38.073 INFO:teuthology.orchestra.run.smithi016.stdout:data_pools [3] 2024-04-22T15:57:38.073 INFO:teuthology.orchestra.run.smithi016.stdout:metadata_pool 2 2024-04-22T15:57:38.073 INFO:teuthology.orchestra.run.smithi016.stdout:inline_data enabled 2024-04-22T15:57:38.073 INFO:teuthology.orchestra.run.smithi016.stdout:balancer 2024-04-22T15:57:38.073 INFO:teuthology.orchestra.run.smithi016.stdout:bal_rank_mask -1 2024-04-22T15:57:38.073 INFO:teuthology.orchestra.run.smithi016.stdout:standby_count_wanted 1 2024-04-22T15:57:38.073 INFO:teuthology.orchestra.run.smithi016.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:57:38.074 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.iobjds{0:24279} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.27:6824/4216277609,v1:172.21.15.27:6825/4216277609] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:57:38.074 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.pgvphu{0:14494} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.16:6828/1006073414,v1:172.21.15.16:6829/1006073414] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:57:38.074 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:57:38.074 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:57:38.074 INFO:teuthology.orchestra.run.smithi016.stdout:Standby daemons: 2024-04-22T15:57:38.074 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:57:38.074 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.fkkejm{-1:14504} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.16:6826/3484907755,v1:172.21.15.16:6827/3484907755] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:57:38.074 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.dgqyes{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.27:6826/476026355,v1:172.21.15.27:6827/476026355] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:57:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.071+0000 7ff0776ed700 1 -- 172.21.15.16:0/1790006485 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ff05c06ecc0 msgr2=0x7ff05c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.071+0000 7ff0776ed700 1 --2- 172.21.15.16:0/1790006485 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ff05c06ecc0 0x7ff05c071180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff06c006010 tx=0x7ff06c01a040 comp rx=0 tx=0).stop 2024-04-22T15:57:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.071+0000 7ff0776ed700 1 -- 172.21.15.16:0/1790006485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff070102780 msgr2=0x7ff070100000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.071+0000 7ff0776ed700 1 --2- 172.21.15.16:0/1790006485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff070102780 0x7ff070100000 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff06000d900 tx=0x7ff06000dc10 comp rx=0 tx=0).stop 2024-04-22T15:57:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.071+0000 7ff0776ed700 1 -- 172.21.15.16:0/1790006485 shutdown_connections 2024-04-22T15:57:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.071+0000 7ff0776ed700 1 --2- 172.21.15.16:0/1790006485 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ff05c06ecc0 0x7ff05c071180 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.071+0000 7ff0776ed700 1 --2- 172.21.15.16:0/1790006485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff070101e60 0x7ff0700ffac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.071+0000 7ff0776ed700 1 --2- 172.21.15.16:0/1790006485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff070102780 0x7ff070100000 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.071+0000 7ff0776ed700 1 -- 172.21.15.16:0/1790006485 >> 172.21.15.16:0/1790006485 conn(0x7ff0700fb650 msgr2=0x7ff070105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.071+0000 7ff0776ed700 1 -- 172.21.15.16:0/1790006485 shutdown_connections 2024-04-22T15:57:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.071+0000 7ff0776ed700 1 -- 172.21.15.16:0/1790006485 wait complete. 2024-04-22T15:57:38.076 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 16 2024-04-22T15:57:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.180+0000 7fe90e086700 1 -- 172.21.15.16:0/1135352478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe908105bd0 msgr2=0x7fe908105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.180+0000 7fe90e086700 1 --2- 172.21.15.16:0/1135352478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe908105bd0 0x7fe908105fb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc009910 tx=0x7fe8fc009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.180+0000 7fe90e086700 1 -- 172.21.15.16:0/1135352478 shutdown_connections 2024-04-22T15:57:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.180+0000 7fe90e086700 1 --2- 172.21.15.16:0/1135352478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9080ffc20 0x7fe908100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.180+0000 7fe90e086700 1 --2- 172.21.15.16:0/1135352478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe908105bd0 0x7fe908105fb0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.180+0000 7fe90e086700 1 -- 172.21.15.16:0/1135352478 >> 172.21.15.16:0/1135352478 conn(0x7fe908075670 msgr2=0x7fe908075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.180+0000 7fe90e086700 1 -- 172.21.15.16:0/1135352478 shutdown_connections 2024-04-22T15:57:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.180+0000 7fe90e086700 1 -- 172.21.15.16:0/1135352478 wait complete. 2024-04-22T15:57:38.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.181+0000 7fe90e086700 1 Processor -- start 2024-04-22T15:57:38.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.181+0000 7fe90e086700 1 -- start start 2024-04-22T15:57:38.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.181+0000 7fe90e086700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9080ffc20 0x7fe9081976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:38.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.181+0000 7fe90e086700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe908105bd0 0x7fe908197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:38.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.181+0000 7fe90e086700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9081982f0 con 0x7fe908105bd0 2024-04-22T15:57:38.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.181+0000 7fe90e086700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe908198460 con 0x7fe9080ffc20 2024-04-22T15:57:38.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.181+0000 7fe906ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe908105bd0 0x7fe908197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:38.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.181+0000 7fe906ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe908105bd0 0x7fe908197c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57424/0 (socket says 172.21.15.16:57424) 2024-04-22T15:57:38.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.181+0000 7fe906ffd700 1 -- 172.21.15.16:0/3724551208 learned_addr learned my addr 172.21.15.16:0/3724551208 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:57:38.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.181+0000 7fe9077fe700 1 --2- 172.21.15.16:0/3724551208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9080ffc20 0x7fe9081976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:38.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.181+0000 7fe906ffd700 1 -- 172.21.15.16:0/3724551208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9080ffc20 msgr2=0x7fe9081976d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:38.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.182+0000 7fe906ffd700 1 --2- 172.21.15.16:0/3724551208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9080ffc20 0x7fe9081976d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:38.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.182+0000 7fe906ffd700 1 -- 172.21.15.16:0/3724551208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8fc0095f0 con 0x7fe908105bd0 2024-04-22T15:57:38.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.182+0000 7fe906ffd700 1 --2- 172.21.15.16:0/3724551208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe908105bd0 0x7fe908197c10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe8f800d8d0 tx=0x7fe8f800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:38.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.182+0000 7fe904ff9700 1 -- 172.21.15.16:0/3724551208 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8f80098a0 con 0x7fe908105bd0 2024-04-22T15:57:38.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.182+0000 7fe904ff9700 1 -- 172.21.15.16:0/3724551208 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe8f8010460 con 0x7fe908105bd0 2024-04-22T15:57:38.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.182+0000 7fe90e086700 1 -- 172.21.15.16:0/3724551208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9081a0740 con 0x7fe908105bd0 2024-04-22T15:57:38.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.182+0000 7fe904ff9700 1 -- 172.21.15.16:0/3724551208 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8f800f5f0 con 0x7fe908105bd0 2024-04-22T15:57:38.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.182+0000 7fe90e086700 1 -- 172.21.15.16:0/3724551208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9081a0c90 con 0x7fe908105bd0 2024-04-22T15:57:38.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.183+0000 7fe90e086700 1 -- 172.21.15.16:0/3724551208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9081080d0 con 0x7fe908105bd0 2024-04-22T15:57:38.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.183+0000 7fe904ff9700 1 -- 172.21.15.16:0/3724551208 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe8f8009a00 con 0x7fe908105bd0 2024-04-22T15:57:38.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.184+0000 7fe904ff9700 1 --2- 172.21.15.16:0/3724551208 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe8f006ecc0 0x7fe8f0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:38.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.184+0000 7fe904ff9700 1 -- 172.21.15.16:0/3724551208 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fe8f808ec80 con 0x7fe908105bd0 2024-04-22T15:57:38.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.184+0000 7fe9077fe700 1 --2- 172.21.15.16:0/3724551208 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe8f006ecc0 0x7fe8f0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:38.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.184+0000 7fe9077fe700 1 --2- 172.21.15.16:0/3724551208 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe8f006ecc0 0x7fe8f0071180 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc00b5c0 tx=0x7fe8fc01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:38.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.189+0000 7fe904ff9700 1 -- 172.21.15.16:0/3724551208 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe8f8057ae0 con 0x7fe908105bd0 2024-04-22T15:57:38.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.426+0000 7fe90e086700 1 -- 172.21.15.16:0/3724551208 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9080611d0 con 0x7fe8f006ecc0 2024-04-22T15:57:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.427+0000 7fe904ff9700 1 -- 172.21.15.16:0/3724551208 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fe9080611d0 con 0x7fe8f006ecc0 2024-04-22T15:57:38.430 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:57:38.430 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:57:38.430 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-22T15:57:38.430 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:57:38.431 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-22T15:57:38.431 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-04-22T15:57:38.431 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr", 2024-04-22T15:57:38.431 INFO:teuthology.orchestra.run.smithi016.stdout: "mon" 2024-04-22T15:57:38.431 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-22T15:57:38.431 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "11/23 daemons upgraded", 2024-04-22T15:57:38.431 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading osd daemons", 2024-04-22T15:57:38.431 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-22T15:57:38.431 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:57:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.429+0000 7fe90e086700 1 -- 172.21.15.16:0/3724551208 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe8f006ecc0 msgr2=0x7fe8f0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.429+0000 7fe90e086700 1 --2- 172.21.15.16:0/3724551208 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe8f006ecc0 0x7fe8f0071180 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc00b5c0 tx=0x7fe8fc01a040 comp rx=0 tx=0).stop 2024-04-22T15:57:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.429+0000 7fe90e086700 1 -- 172.21.15.16:0/3724551208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe908105bd0 msgr2=0x7fe908197c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:38.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.429+0000 7fe90e086700 1 --2- 172.21.15.16:0/3724551208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe908105bd0 0x7fe908197c10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe8f800d8d0 tx=0x7fe8f800dbe0 comp rx=0 tx=0).stop 2024-04-22T15:57:38.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.429+0000 7fe90e086700 1 -- 172.21.15.16:0/3724551208 shutdown_connections 2024-04-22T15:57:38.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.429+0000 7fe90e086700 1 --2- 172.21.15.16:0/3724551208 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe8f006ecc0 0x7fe8f0071180 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:38.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.429+0000 7fe90e086700 1 --2- 172.21.15.16:0/3724551208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9080ffc20 0x7fe9081976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:38.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.429+0000 7fe90e086700 1 --2- 172.21.15.16:0/3724551208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe908105bd0 0x7fe908197c10 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:38.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.429+0000 7fe90e086700 1 -- 172.21.15.16:0/3724551208 >> 172.21.15.16:0/3724551208 conn(0x7fe908075670 msgr2=0x7fe9080feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:38.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.430+0000 7fe90e086700 1 -- 172.21.15.16:0/3724551208 shutdown_connections 2024-04-22T15:57:38.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.430+0000 7fe90e086700 1 -- 172.21.15.16:0/3724551208 wait complete. 2024-04-22T15:57:38.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.536+0000 7fd21d91a700 1 -- 172.21.15.16:0/3410301053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2180730b0 msgr2=0x7fd218073490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.536+0000 7fd21d91a700 1 --2- 172.21.15.16:0/3410301053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2180730b0 0x7fd218073490 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd208009910 tx=0x7fd208009c20 comp rx=0 tx=0).stop 2024-04-22T15:57:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.536+0000 7fd21d91a700 1 -- 172.21.15.16:0/3410301053 shutdown_connections 2024-04-22T15:57:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.536+0000 7fd21d91a700 1 --2- 172.21.15.16:0/3410301053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2180739d0 0x7fd21810cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.536+0000 7fd21d91a700 1 --2- 172.21.15.16:0/3410301053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2180730b0 0x7fd218073490 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.536+0000 7fd21d91a700 1 -- 172.21.15.16:0/3410301053 >> 172.21.15.16:0/3410301053 conn(0x7fd2180fc750 msgr2=0x7fd2180feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.536+0000 7fd21d91a700 1 -- 172.21.15.16:0/3410301053 shutdown_connections 2024-04-22T15:57:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.536+0000 7fd21d91a700 1 -- 172.21.15.16:0/3410301053 wait complete. 2024-04-22T15:57:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.537+0000 7fd21d91a700 1 Processor -- start 2024-04-22T15:57:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.537+0000 7fd21d91a700 1 -- start start 2024-04-22T15:57:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.537+0000 7fd21d91a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2180730b0 0x7fd21819e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.537+0000 7fd21d91a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2180739d0 0x7fd21819ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.537+0000 7fd21d91a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd21819f410 con 0x7fd2180739d0 2024-04-22T15:57:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.537+0000 7fd21d91a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2181988c0 con 0x7fd2180730b0 2024-04-22T15:57:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.538+0000 7fd2167fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2180739d0 0x7fd21819ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.538+0000 7fd2167fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2180739d0 0x7fd21819ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57446/0 (socket says 172.21.15.16:57446) 2024-04-22T15:57:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.538+0000 7fd2167fc700 1 -- 172.21.15.16:0/1288589519 learned_addr learned my addr 172.21.15.16:0/1288589519 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:57:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.538+0000 7fd216ffd700 1 --2- 172.21.15.16:0/1288589519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2180730b0 0x7fd21819e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.538+0000 7fd2167fc700 1 -- 172.21.15.16:0/1288589519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2180730b0 msgr2=0x7fd21819e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.538+0000 7fd2167fc700 1 --2- 172.21.15.16:0/1288589519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2180730b0 0x7fd21819e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.538+0000 7fd2167fc700 1 -- 172.21.15.16:0/1288589519 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2080095f0 con 0x7fd2180739d0 2024-04-22T15:57:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.538+0000 7fd2167fc700 1 --2- 172.21.15.16:0/1288589519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2180739d0 0x7fd21819ed80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd20c00d8d0 tx=0x7fd20c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.538+0000 7fd21c918700 1 -- 172.21.15.16:0/1288589519 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd20c0098a0 con 0x7fd2180739d0 2024-04-22T15:57:38.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.538+0000 7fd21d91a700 1 -- 172.21.15.16:0/1288589519 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd218198ba0 con 0x7fd2180739d0 2024-04-22T15:57:38.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.538+0000 7fd21c918700 1 -- 172.21.15.16:0/1288589519 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd20c010460 con 0x7fd2180739d0 2024-04-22T15:57:38.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.538+0000 7fd21c918700 1 -- 172.21.15.16:0/1288589519 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd20c00f5f0 con 0x7fd2180739d0 2024-04-22T15:57:38.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.538+0000 7fd21d91a700 1 -- 172.21.15.16:0/1288589519 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2181990f0 con 0x7fd2180739d0 2024-04-22T15:57:38.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.539+0000 7fd21d91a700 1 -- 172.21.15.16:0/1288589519 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd21810a6c0 con 0x7fd2180739d0 2024-04-22T15:57:38.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.540+0000 7fd21c918700 1 -- 172.21.15.16:0/1288589519 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd20c009a00 con 0x7fd2180739d0 2024-04-22T15:57:38.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.540+0000 7fd21c918700 1 --2- 172.21.15.16:0/1288589519 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd20006ecc0 0x7fd200071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:57:38.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.540+0000 7fd21c918700 1 -- 172.21.15.16:0/1288589519 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd20c08ec80 con 0x7fd2180739d0 2024-04-22T15:57:38.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.541+0000 7fd216ffd700 1 --2- 172.21.15.16:0/1288589519 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd20006ecc0 0x7fd200071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:57:38.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.541+0000 7fd216ffd700 1 --2- 172.21.15.16:0/1288589519 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd20006ecc0 0x7fd200071180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd208006010 tx=0x7fd20801a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:57:38.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.545+0000 7fd21c918700 1 -- 172.21.15.16:0/1288589519 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd20c05b610 con 0x7fd2180739d0 2024-04-22T15:57:38.658 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:38 smithi027 ceph-mon[83054]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:38.658 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:38 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/2625513546' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:57:38.658 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:38 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/1790006485' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:57:38.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:38 smithi016 ceph-mon[110740]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:38.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:38 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/2625513546' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:57:38.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:38 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/1790006485' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:57:38.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.842+0000 7fd21d91a700 1 -- 172.21.15.16:0/1288589519 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd218199dd0 con 0x7fd2180739d0 2024-04-22T15:57:38.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.843+0000 7fd21c918700 1 -- 172.21.15.16:0/1288589519 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7fd20c05b1a0 con 0x7fd2180739d0 2024-04-22T15:57:38.846 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:57:38.846 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:57:38.847 INFO:teuthology.orchestra.run.smithi016.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:57:38.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.845+0000 7fd21d91a700 1 -- 172.21.15.16:0/1288589519 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd20006ecc0 msgr2=0x7fd200071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.845+0000 7fd21d91a700 1 --2- 172.21.15.16:0/1288589519 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd20006ecc0 0x7fd200071180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd208006010 tx=0x7fd20801a040 comp rx=0 tx=0).stop 2024-04-22T15:57:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.845+0000 7fd21d91a700 1 -- 172.21.15.16:0/1288589519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2180739d0 msgr2=0x7fd21819ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:57:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.845+0000 7fd21d91a700 1 --2- 172.21.15.16:0/1288589519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2180739d0 0x7fd21819ed80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd20c00d8d0 tx=0x7fd20c00dbe0 comp rx=0 tx=0).stop 2024-04-22T15:57:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.845+0000 7fd21d91a700 1 -- 172.21.15.16:0/1288589519 shutdown_connections 2024-04-22T15:57:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.845+0000 7fd21d91a700 1 --2- 172.21.15.16:0/1288589519 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd20006ecc0 0x7fd200071180 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.845+0000 7fd21d91a700 1 --2- 172.21.15.16:0/1288589519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2180730b0 0x7fd21819e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.845+0000 7fd21d91a700 1 --2- 172.21.15.16:0/1288589519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2180739d0 0x7fd21819ed80 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:57:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.845+0000 7fd21d91a700 1 -- 172.21.15.16:0/1288589519 >> 172.21.15.16:0/1288589519 conn(0x7fd2180fc750 msgr2=0x7fd218107800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:57:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.845+0000 7fd21d91a700 1 -- 172.21.15.16:0/1288589519 shutdown_connections 2024-04-22T15:57:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:57:38.845+0000 7fd21d91a700 1 -- 172.21.15.16:0/1288589519 wait complete. 2024-04-22T15:57:39.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:39 smithi016 ceph-mon[110740]: pgmap v152: 65 pgs: 65 active+clean; 250 MiB data, 1014 MiB used, 535 GiB / 536 GiB avail; 343 B/s rd, 0 op/s 2024-04-22T15:57:39.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:39 smithi016 ceph-mon[110740]: from='client.34364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:39.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:39 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/1288589519' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:57:39.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:39 smithi027 ceph-mon[83054]: pgmap v152: 65 pgs: 65 active+clean; 250 MiB data, 1014 MiB used, 535 GiB / 536 GiB avail; 343 B/s rd, 0 op/s 2024-04-22T15:57:39.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:39 smithi027 ceph-mon[83054]: from='client.34364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:57:39.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:39 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/1288589519' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:57:41.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:41 smithi016 ceph-mon[110740]: pgmap v153: 65 pgs: 65 active+clean; 250 MiB data, 1014 MiB used, 535 GiB / 536 GiB avail; 818 B/s rd, 1 op/s 2024-04-22T15:57:41.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:41 smithi027 ceph-mon[83054]: pgmap v153: 65 pgs: 65 active+clean; 250 MiB data, 1014 MiB used, 535 GiB / 536 GiB avail; 818 B/s rd, 1 op/s 2024-04-22T15:57:43.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:43 smithi016 ceph-mon[110740]: pgmap v154: 65 pgs: 65 active+clean; 250 MiB data, 1014 MiB used, 535 GiB / 536 GiB avail; 685 B/s rd, 1 op/s 2024-04-22T15:57:43.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:43 smithi027 ceph-mon[83054]: pgmap v154: 65 pgs: 65 active+clean; 250 MiB data, 1014 MiB used, 535 GiB / 536 GiB avail; 685 B/s rd, 1 op/s 2024-04-22T15:57:44.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:44 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-22T15:57:44.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:44 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-22T15:57:45.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:45 smithi027 ceph-mon[83054]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-22T15:57:45.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:45 smithi027 ceph-mon[83054]: Upgrade: osd.5 is safe to restart 2024-04-22T15:57:45.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:45 smithi027 ceph-mon[83054]: pgmap v155: 65 pgs: 65 active+clean; 250 MiB data, 1014 MiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-22T15:57:45.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:45 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:45.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:45 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:57:45.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:45 smithi016 ceph-mon[110740]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-22T15:57:45.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:45 smithi016 ceph-mon[110740]: Upgrade: osd.5 is safe to restart 2024-04-22T15:57:45.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:45 smithi016 ceph-mon[110740]: pgmap v155: 65 pgs: 65 active+clean; 250 MiB data, 1014 MiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-22T15:57:45.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:45 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:45.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:45 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:57:46.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:46.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:46 smithi027 ceph-mon[83054]: Upgrade: Updating osd.5 2024-04-22T15:57:46.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:46.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T15:57:46.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:57:46.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:46 smithi027 ceph-mon[83054]: Deploying daemon osd.5 on smithi027 2024-04-22T15:57:46.850 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:46.850 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:46 smithi016 ceph-mon[110740]: Upgrade: Updating osd.5 2024-04-22T15:57:46.850 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:46.850 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T15:57:46.850 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:57:46.851 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:46 smithi016 ceph-mon[110740]: Deploying daemon osd.5 on smithi027 2024-04-22T15:57:48.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:48 smithi027 ceph-mon[83054]: pgmap v156: 65 pgs: 65 active+clean; 250 MiB data, 1014 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:57:48.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:48 smithi016 ceph-mon[110740]: pgmap v156: 65 pgs: 65 active+clean; 250 MiB data, 1014 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:57:49.836 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:49 smithi027 systemd[1]: Stopping Ceph osd.5 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T15:57:49.836 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:49 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[54558]: 2024-04-22T15:57:49.594+0000 7f17c9f45700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T15:57:49.837 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:49 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[54558]: 2024-04-22T15:57:49.594+0000 7f17c9f45700 -1 osd.5 74 *** Got signal Terminated *** 2024-04-22T15:57:49.837 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:49 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[54558]: 2024-04-22T15:57:49.594+0000 7f17c9f45700 -1 osd.5 74 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T15:57:50.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:50 smithi027 ceph-mon[83054]: pgmap v157: 65 pgs: 65 active+clean; 250 MiB data, 1014 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:57:50.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:50 smithi027 ceph-mon[83054]: osd.5 marked itself down and dead 2024-04-22T15:57:50.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:50 smithi016 ceph-mon[110740]: pgmap v157: 65 pgs: 65 active+clean; 250 MiB data, 1014 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:57:50.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:50 smithi016 ceph-mon[110740]: osd.5 marked itself down and dead 2024-04-22T15:57:50.836 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:50 smithi027 podman[102468]: 2024-04-22 15:57:50.58651106 +0000 UTC m=+1.091660876 container died 23adcf675b04b9f452d077909b97f7f477bcff83996055bbe90db2fa0844fe54 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux ) 2024-04-22T15:57:51.213 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:51 smithi027 ceph-mon[83054]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T15:57:51.213 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:51 smithi027 ceph-mon[83054]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-22T15:57:51.213 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:51 smithi027 ceph-mon[83054]: osdmap e75: 6 total, 5 up, 6 in 2024-04-22T15:57:51.213 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:51 smithi027 ceph-mon[83054]: osdmap e76: 6 total, 5 up, 6 in 2024-04-22T15:57:51.214 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:50 smithi027 podman[102468]: 2024-04-22 15:57:50.943642503 +0000 UTC m=+1.448792318 container cleanup 23adcf675b04b9f452d077909b97f7f477bcff83996055bbe90db2fa0844fe54 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-22T15:57:51.214 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:50 smithi027 bash[102468]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5 2024-04-22T15:57:51.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:51 smithi016 ceph-mon[110740]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T15:57:51.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:51 smithi016 ceph-mon[110740]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-22T15:57:51.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:51 smithi016 ceph-mon[110740]: osdmap e75: 6 total, 5 up, 6 in 2024-04-22T15:57:51.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:51 smithi016 ceph-mon[110740]: osdmap e76: 6 total, 5 up, 6 in 2024-04-22T15:57:51.538 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:51 smithi027 podman[102487]: 2024-04-22 15:57:51.21166281 +0000 UTC m=+0.619398446 container remove 23adcf675b04b9f452d077909b97f7f477bcff83996055bbe90db2fa0844fe54 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD) 2024-04-22T15:57:51.825 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:51 smithi027 podman[102581]: 2024-04-22 15:57:51.439750293 +0000 UTC m=+0.017876303 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:57:51.825 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:51 smithi027 podman[102581]: 2024-04-22 15:57:51.537473974 +0000 UTC m=+0.115599982 container create f0a45db8f62b586ab544c951a656f5ba3e4aa321386d2778677a91e21fc4777f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49) 2024-04-22T15:57:51.825 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:51 smithi027 podman[102581]: 2024-04-22 15:57:51.74844802 +0000 UTC m=+0.326574030 container init f0a45db8f62b586ab544c951a656f5ba3e4aa321386d2778677a91e21fc4777f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:57:51.825 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:51 smithi027 podman[102581]: 2024-04-22 15:57:51.751668918 +0000 UTC m=+0.329794936 container start f0a45db8f62b586ab544c951a656f5ba3e4aa321386d2778677a91e21fc4777f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:57:51.825 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:51 smithi027 podman[102581]: 2024-04-22 15:57:51.823150119 +0000 UTC m=+0.401276128 container attach f0a45db8f62b586ab544c951a656f5ba3e4aa321386d2778677a91e21fc4777f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-22T15:57:52.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:52 smithi027 ceph-mon[83054]: pgmap v159: 65 pgs: 13 stale+active+clean, 52 active+clean; 250 MiB data, 1014 MiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2024-04-22T15:57:52.086 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:52 smithi027 podman[102581]: 2024-04-22 15:57:52.061854948 +0000 UTC m=+0.639980958 container died f0a45db8f62b586ab544c951a656f5ba3e4aa321386d2778677a91e21fc4777f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416) 2024-04-22T15:57:52.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:52 smithi016 ceph-mon[110740]: pgmap v159: 65 pgs: 13 stale+active+clean, 52 active+clean; 250 MiB data, 1014 MiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2024-04-22T15:57:53.086 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:52 smithi027 podman[102606]: 2024-04-22 15:57:52.669027838 +0000 UTC m=+0.601776912 container remove f0a45db8f62b586ab544c951a656f5ba3e4aa321386d2778677a91e21fc4777f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-22T15:57:53.086 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:52 smithi027 systemd[1]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.5.service: Deactivated successfully. 2024-04-22T15:57:53.086 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:52 smithi027 systemd[1]: Stopped Ceph osd.5 for 67ab3a80-00be-11ef-bc93-c7b262605968. 2024-04-22T15:57:53.086 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:52 smithi027 systemd[1]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.5.service: Consumed 55.368s CPU time. 2024-04-22T15:57:53.087 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:52 smithi027 systemd[1]: Starting Ceph osd.5 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T15:57:53.396 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:53 smithi027 ceph-mon[83054]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-22T15:57:53.396 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:53 smithi027 ceph-mon[83054]: Health check failed: Degraded data redundancy: 7/258 objects degraded (2.713%), 4 pgs degraded (PG_DEGRADED) 2024-04-22T15:57:53.396 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:53 smithi027 podman[102709]: 2024-04-22 15:57:53.035188215 +0000 UTC m=+0.018648540 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:57:53.396 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:53 smithi027 podman[102709]: 2024-04-22 15:57:53.161633252 +0000 UTC m=+0.145093550 container create 20c742fafa87858cfd6c3a844cc4d45ddd8a78ed10af087a63ac257032f2171a (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-activate, ceph=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-22T15:57:53.396 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:53 smithi027 podman[102709]: 2024-04-22 15:57:53.394376521 +0000 UTC m=+0.377836819 container init 20c742fafa87858cfd6c3a844cc4d45ddd8a78ed10af087a63ac257032f2171a (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-activate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T15:57:53.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:53 smithi016 ceph-mon[110740]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-22T15:57:53.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:53 smithi016 ceph-mon[110740]: Health check failed: Degraded data redundancy: 7/258 objects degraded (2.713%), 4 pgs degraded (PG_DEGRADED) 2024-04-22T15:57:53.836 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:53 smithi027 podman[102709]: 2024-04-22 15:57:53.397639114 +0000 UTC m=+0.381099415 container start 20c742fafa87858cfd6c3a844cc4d45ddd8a78ed10af087a63ac257032f2171a (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T15:57:53.836 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:53 smithi027 podman[102709]: 2024-04-22 15:57:53.481036264 +0000 UTC m=+0.464496565 container attach 20c742fafa87858cfd6c3a844cc4d45ddd8a78ed10af087a63ac257032f2171a (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-activate, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-22T15:57:54.203 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:54 smithi027 ceph-mon[83054]: pgmap v161: 65 pgs: 4 active+undersized+degraded, 3 peering, 7 stale+active+clean, 4 active+undersized, 47 active+clean; 250 MiB data, 1014 MiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 7/258 objects degraded (2.713%) 2024-04-22T15:57:54.203 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:53 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-activate[102724]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:57:54.204 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:53 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-activate[102724]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:57:54.204 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:53 smithi027 bash[102709]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:57:54.204 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:53 smithi027 bash[102709]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T15:57:54.204 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:53 smithi027 bash[102709]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-22T15:57:54.204 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:53 smithi027 bash[102709]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-22T15:57:54.204 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:53 smithi027 bash[102709]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-22T15:57:54.204 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:53 smithi027 bash[102709]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-22T15:57:54.204 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:53 smithi027 bash[102709]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-04-22T15:57:54.204 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:53 smithi027 bash[102709]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-22T15:57:54.204 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:53 smithi027 bash[102709]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-22T15:57:54.204 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:53 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-activate[102724]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-22T15:57:54.204 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:53 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-activate[102724]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-22T15:57:54.204 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:53 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-activate[102724]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-22T15:57:54.204 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:53 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-activate[102724]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-22T15:57:54.204 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:53 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-activate[102724]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-04-22T15:57:54.205 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:53 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-activate[102724]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-22T15:57:54.205 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:53 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-activate[102724]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-22T15:57:54.205 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:53 smithi027 podman[102709]: 2024-04-22 15:57:53.919805276 +0000 UTC m=+0.903265578 container died 20c742fafa87858cfd6c3a844cc4d45ddd8a78ed10af087a63ac257032f2171a (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-activate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-22T15:57:54.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:54 smithi016 ceph-mon[110740]: pgmap v161: 65 pgs: 4 active+undersized+degraded, 3 peering, 7 stale+active+clean, 4 active+undersized, 47 active+clean; 250 MiB data, 1014 MiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 7/258 objects degraded (2.713%) 2024-04-22T15:57:54.836 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:54 smithi027 podman[102866]: 2024-04-22 15:57:54.527085813 +0000 UTC m=+0.601965058 container remove 20c742fafa87858cfd6c3a844cc4d45ddd8a78ed10af087a63ac257032f2171a (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-activate, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-22T15:57:54.836 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:54 smithi027 podman[102933]: 2024-04-22 15:57:54.646604751 +0000 UTC m=+0.018553420 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T15:57:54.836 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:54 smithi027 podman[102933]: 2024-04-22 15:57:54.752952757 +0000 UTC m=+0.124901459 container create d857c152994a0f6879196dd78b66368fc88aeb7cbb3d26f31c7a0a5e79e2a341 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, CEPH_POINT_RELEASE=) 2024-04-22T15:57:55.336 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:54 smithi027 podman[102933]: 2024-04-22 15:57:54.980229343 +0000 UTC m=+0.352178000 container init d857c152994a0f6879196dd78b66368fc88aeb7cbb3d26f31c7a0a5e79e2a341 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-22T15:57:55.336 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:54 smithi027 podman[102933]: 2024-04-22 15:57:54.983378408 +0000 UTC m=+0.355327064 container start d857c152994a0f6879196dd78b66368fc88aeb7cbb3d26f31c7a0a5e79e2a341 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-22T15:57:55.336 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:55 smithi027 bash[102933]: d857c152994a0f6879196dd78b66368fc88aeb7cbb3d26f31c7a0a5e79e2a341 2024-04-22T15:57:55.336 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:55 smithi027 systemd[1]: Started Ceph osd.5 for 67ab3a80-00be-11ef-bc93-c7b262605968. 2024-04-22T15:57:55.836 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:55 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T15:57:55.528+0000 7f94eb6f27c0 -1 Falling back to public interface 2024-04-22T15:57:56.235 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:56 smithi027 ceph-mon[83054]: pgmap v162: 65 pgs: 13 active+undersized+degraded, 3 peering, 12 active+undersized, 37 active+clean; 250 MiB data, 1018 MiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 40/258 objects degraded (15.504%) 2024-04-22T15:57:56.235 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:56 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:56.235 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:56 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:56.235 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:56 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:57:56.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:56 smithi016 ceph-mon[110740]: pgmap v162: 65 pgs: 13 active+undersized+degraded, 3 peering, 12 active+undersized, 37 active+clean; 250 MiB data, 1018 MiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 40/258 objects degraded (15.504%) 2024-04-22T15:57:56.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:56 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:56.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:56 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:56.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:56 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:57:57.486 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:57 smithi027 ceph-mon[83054]: pgmap v163: 65 pgs: 13 active+undersized+degraded, 15 active+undersized, 37 active+clean; 250 MiB data, 1018 MiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 40/258 objects degraded (15.504%) 2024-04-22T15:57:57.487 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:57 smithi027 ceph-mon[83054]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-22T15:57:57.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:57 smithi016 ceph-mon[110740]: pgmap v163: 65 pgs: 13 active+undersized+degraded, 15 active+undersized, 37 active+clean; 250 MiB data, 1018 MiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 40/258 objects degraded (15.504%) 2024-04-22T15:57:57.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:57 smithi016 ceph-mon[110740]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-22T15:57:59.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:59 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:59.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:59 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:59.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:57:59 smithi016 ceph-mon[110740]: pgmap v164: 65 pgs: 13 active+undersized+degraded, 15 active+undersized, 37 active+clean; 250 MiB data, 1018 MiB used, 535 GiB / 536 GiB avail; 40/258 objects degraded (15.504%) 2024-04-22T15:57:59.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:59 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:59.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:59 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:57:59.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:57:59 smithi027 ceph-mon[83054]: pgmap v164: 65 pgs: 13 active+undersized+degraded, 15 active+undersized, 37 active+clean; 250 MiB data, 1018 MiB used, 535 GiB / 536 GiB avail; 40/258 objects degraded (15.504%) 2024-04-22T15:58:00.294 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:57:59 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T15:57:59.972+0000 7f94eb6f27c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-04-22T15:58:00.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:00 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:00.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:00 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:00.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:00 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:00.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:00 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:58:00.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:00 smithi027 ceph-mon[83054]: from='osd.5 [v2:172.21.15.27:6816/3875730087,v1:172.21.15.27:6817/3875730087]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T15:58:00.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:00 smithi027 ceph-mon[83054]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T15:58:00.587 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:58:00 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T15:58:00.299+0000 7f94eb6f27c0 -1 osd.5 74 log_to_monitors true 2024-04-22T15:58:00.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:00 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:00.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:00 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:00.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:00 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:00.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:00 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:58:00.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:00 smithi016 ceph-mon[110740]: from='osd.5 [v2:172.21.15.27:6816/3875730087,v1:172.21.15.27:6817/3875730087]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T15:58:00.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:00 smithi016 ceph-mon[110740]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T15:58:01.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:01 smithi016 ceph-mon[110740]: pgmap v165: 65 pgs: 13 active+undersized+degraded, 15 active+undersized, 37 active+clean; 250 MiB data, 1018 MiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 40/258 objects degraded (15.504%) 2024-04-22T15:58:01.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:01 smithi016 ceph-mon[110740]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T15:58:01.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:01 smithi016 ceph-mon[110740]: osdmap e77: 6 total, 5 up, 6 in 2024-04-22T15:58:01.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:01 smithi016 ceph-mon[110740]: from='osd.5 [v2:172.21.15.27:6816/3875730087,v1:172.21.15.27:6817/3875730087]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-22T15:58:01.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:01 smithi016 ceph-mon[110740]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-22T15:58:01.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:01 smithi027 ceph-mon[83054]: pgmap v165: 65 pgs: 13 active+undersized+degraded, 15 active+undersized, 37 active+clean; 250 MiB data, 1018 MiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 40/258 objects degraded (15.504%) 2024-04-22T15:58:01.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:01 smithi027 ceph-mon[83054]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T15:58:01.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:01 smithi027 ceph-mon[83054]: osdmap e77: 6 total, 5 up, 6 in 2024-04-22T15:58:01.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:01 smithi027 ceph-mon[83054]: from='osd.5 [v2:172.21.15.27:6816/3875730087,v1:172.21.15.27:6817/3875730087]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-22T15:58:01.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:01 smithi027 ceph-mon[83054]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-22T15:58:02.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:02 smithi027 ceph-mon[83054]: Health check update: Degraded data redundancy: 40/258 objects degraded (15.504%), 13 pgs degraded (PG_DEGRADED) 2024-04-22T15:58:02.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:02 smithi027 ceph-mon[83054]: from='osd.5 ' entity='osd.5' 2024-04-22T15:58:02.587 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 15:58:02 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T15:58:02.196+0000 7f94d8b42700 -1 osd.5 74 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T15:58:02.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:02 smithi016 ceph-mon[110740]: Health check update: Degraded data redundancy: 40/258 objects degraded (15.504%), 13 pgs degraded (PG_DEGRADED) 2024-04-22T15:58:02.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:02 smithi016 ceph-mon[110740]: from='osd.5 ' entity='osd.5' 2024-04-22T15:58:04.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: pgmap v167: 65 pgs: 13 active+undersized+degraded, 15 active+undersized, 37 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 511 B/s rd, 0 op/s; 40/258 objects degraded (15.504%) 2024-04-22T15:58:04.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:04.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:04.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:58:04.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:58:04.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:04.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T15:58:04.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: osd.5 [v2:172.21.15.27:6816/3875730087,v1:172.21.15.27:6817/3875730087] boot 2024-04-22T15:58:04.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: osdmap e78: 6 total, 6 up, 6 in 2024-04-22T15:58:04.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:58:04.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:58:04.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:04.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:04.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:04.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:04.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:04.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-22T15:58:04.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-22T15:58:04.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-22T15:58:04.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-22T15:58:04.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-22T15:58:04.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-22T15:58:04.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-22T15:58:04.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-22T15:58:04.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-22T15:58:04.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-22T15:58:04.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-22T15:58:04.339 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-22T15:58:04.339 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:04 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-22T15:58:04.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: pgmap v167: 65 pgs: 13 active+undersized+degraded, 15 active+undersized, 37 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 511 B/s rd, 0 op/s; 40/258 objects degraded (15.504%) 2024-04-22T15:58:04.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:04.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:04.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:58:04.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:58:04.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:04.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T15:58:04.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: osd.5 [v2:172.21.15.27:6816/3875730087,v1:172.21.15.27:6817/3875730087] boot 2024-04-22T15:58:04.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: osdmap e78: 6 total, 6 up, 6 in 2024-04-22T15:58:04.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T15:58:04.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:58:04.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:04.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:04.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:04.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:04.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:04.419 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-22T15:58:04.419 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-22T15:58:04.419 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-22T15:58:04.419 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-22T15:58:04.419 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-22T15:58:04.419 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-22T15:58:04.419 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-22T15:58:04.419 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-22T15:58:04.419 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-22T15:58:04.419 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-22T15:58:04.419 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-22T15:58:04.419 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-22T15:58:04.419 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:04 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-22T15:58:05.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:05 smithi027 ceph-mon[83054]: Upgrade: Setting container_image for all osd 2024-04-22T15:58:05.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:05 smithi027 ceph-mon[83054]: Upgrade: Setting require_osd_release to 19 squid 2024-04-22T15:58:05.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:05 smithi027 ceph-mon[83054]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-22T15:58:05.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:05 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-22T15:58:05.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:05 smithi027 ceph-mon[83054]: osdmap e79: 6 total, 6 up, 6 in 2024-04-22T15:58:05.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:05 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:05.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:05 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-04-22T15:58:05.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:05 smithi016 ceph-mon[110740]: Upgrade: Setting container_image for all osd 2024-04-22T15:58:05.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:05 smithi016 ceph-mon[110740]: Upgrade: Setting require_osd_release to 19 squid 2024-04-22T15:58:05.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:05 smithi016 ceph-mon[110740]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-22T15:58:05.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:05 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-22T15:58:05.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:05 smithi016 ceph-mon[110740]: osdmap e79: 6 total, 6 up, 6 in 2024-04-22T15:58:05.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:05 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:05.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:05 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-04-22T15:58:06.260 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:06 smithi016 ceph-mon[110740]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-04-22T15:58:06.260 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:06 smithi016 ceph-mon[110740]: pgmap v170: 65 pgs: 7 peering, 8 active+undersized+degraded, 13 active+undersized, 37 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 20/258 objects degraded (7.752%) 2024-04-22T15:58:06.261 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:06 smithi016 ceph-mon[110740]: osdmap e80: 6 total, 6 up, 6 in 2024-04-22T15:58:06.261 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:06 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-04-22T15:58:06.261 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:06 smithi016 ceph-mon[110740]: fsmap cephfs:1 {0=cephfs.smithi027.iobjds=up:active} 2 up:standby 2024-04-22T15:58:06.261 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:06 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:58:06.261 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:06 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:58:06.261 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:06 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:58:06.261 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:06 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:06.261 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:06 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:58:06.261 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:06 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:06.261 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:06 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:06.261 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:06 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:06.261 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:06 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:06.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:06 smithi027 ceph-mon[83054]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-04-22T15:58:06.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:06 smithi027 ceph-mon[83054]: pgmap v170: 65 pgs: 7 peering, 8 active+undersized+degraded, 13 active+undersized, 37 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 20/258 objects degraded (7.752%) 2024-04-22T15:58:06.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:06 smithi027 ceph-mon[83054]: osdmap e80: 6 total, 6 up, 6 in 2024-04-22T15:58:06.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:06 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-04-22T15:58:06.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:06 smithi027 ceph-mon[83054]: fsmap cephfs:1 {0=cephfs.smithi027.iobjds=up:active} 2 up:standby 2024-04-22T15:58:06.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:06 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:58:06.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:06 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:58:06.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:06 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:58:06.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:06 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:06.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:06 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:58:06.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:06 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:06.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:06 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:06.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:06 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:06.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:06 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:07.375 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:07 smithi016 ceph-mon[110740]: mds.? [v2:172.21.15.16:6828/3518018286,v1:172.21.15.16:6829/3518018286] up:boot 2024-04-22T15:58:07.375 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:07 smithi016 ceph-mon[110740]: fsmap cephfs:1 {0=cephfs.smithi027.iobjds=up:active} 3 up:standby 2024-04-22T15:58:07.375 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:07 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.pgvphu"}]: dispatch 2024-04-22T15:58:07.376 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:07 smithi016 ceph-mon[110740]: pgmap v172: 65 pgs: 7 peering, 8 active+undersized+degraded, 12 active+undersized, 38 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 615 B/s rd, 1 op/s; 20/258 objects degraded (7.752%) 2024-04-22T15:58:07.376 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:07 smithi016 ceph-mon[110740]: Health check update: Degraded data redundancy: 20/258 objects degraded (7.752%), 8 pgs degraded (PG_DEGRADED) 2024-04-22T15:58:07.376 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:07 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:07.376 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:07 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi016.fkkejm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:58:07.376 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:07 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:58:07.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:07 smithi027 ceph-mon[83054]: mds.? [v2:172.21.15.16:6828/3518018286,v1:172.21.15.16:6829/3518018286] up:boot 2024-04-22T15:58:07.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:07 smithi027 ceph-mon[83054]: fsmap cephfs:1 {0=cephfs.smithi027.iobjds=up:active} 3 up:standby 2024-04-22T15:58:07.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:07 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.pgvphu"}]: dispatch 2024-04-22T15:58:07.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:07 smithi027 ceph-mon[83054]: pgmap v172: 65 pgs: 7 peering, 8 active+undersized+degraded, 12 active+undersized, 38 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 615 B/s rd, 1 op/s; 20/258 objects degraded (7.752%) 2024-04-22T15:58:07.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:07 smithi027 ceph-mon[83054]: Health check update: Degraded data redundancy: 20/258 objects degraded (7.752%), 8 pgs degraded (PG_DEGRADED) 2024-04-22T15:58:07.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:07 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:07.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:07 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi016.fkkejm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:58:07.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:07 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:58:08.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:08 smithi016 ceph-mon[110740]: Upgrade: Updating mds.cephfs.smithi016.fkkejm 2024-04-22T15:58:08.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:08 smithi016 ceph-mon[110740]: Deploying daemon mds.cephfs.smithi016.fkkejm on smithi016 2024-04-22T15:58:08.519 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:08 smithi027 ceph-mon[83054]: Upgrade: Updating mds.cephfs.smithi016.fkkejm 2024-04-22T15:58:08.519 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:08 smithi027 ceph-mon[83054]: Deploying daemon mds.cephfs.smithi016.fkkejm on smithi016 2024-04-22T15:58:08.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.968+0000 7f81bbb98700 1 -- 172.21.15.16:0/249705052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81b4106cd0 msgr2=0x7f81b41070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:08.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.968+0000 7f81bbb98700 1 --2- 172.21.15.16:0/249705052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81b4106cd0 0x7f81b41070b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f81b0009910 tx=0x7f81b0009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:08.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.968+0000 7f81bbb98700 1 -- 172.21.15.16:0/249705052 shutdown_connections 2024-04-22T15:58:08.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.968+0000 7f81bbb98700 1 --2- 172.21.15.16:0/249705052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81b4068580 0x7f81b40689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:08.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.968+0000 7f81bbb98700 1 --2- 172.21.15.16:0/249705052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81b4106cd0 0x7f81b41070b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.968+0000 7f81bbb98700 1 -- 172.21.15.16:0/249705052 >> 172.21.15.16:0/249705052 conn(0x7f81b4075670 msgr2=0x7f81b4075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.969+0000 7f81bbb98700 1 -- 172.21.15.16:0/249705052 shutdown_connections 2024-04-22T15:58:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.969+0000 7f81bbb98700 1 -- 172.21.15.16:0/249705052 wait complete. 2024-04-22T15:58:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.969+0000 7f81bbb98700 1 Processor -- start 2024-04-22T15:58:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.969+0000 7f81bbb98700 1 -- start start 2024-04-22T15:58:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.969+0000 7f81bbb98700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81b4068580 0x7f81b419c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.969+0000 7f81bbb98700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81b4106cd0 0x7f81b419cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.969+0000 7f81bbb98700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b419d240 con 0x7f81b4068580 2024-04-22T15:58:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.970+0000 7f81bbb98700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b41966f0 con 0x7f81b4106cd0 2024-04-22T15:58:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.970+0000 7f81b9934700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81b4068580 0x7f81b419c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.970+0000 7f81b9934700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81b4068580 0x7f81b419c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59082/0 (socket says 172.21.15.16:59082) 2024-04-22T15:58:08.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.970+0000 7f81b9934700 1 -- 172.21.15.16:0/1533591741 learned_addr learned my addr 172.21.15.16:0/1533591741 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:58:08.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.970+0000 7f81b9133700 1 --2- 172.21.15.16:0/1533591741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81b4106cd0 0x7f81b419cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:08.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.970+0000 7f81b9934700 1 -- 172.21.15.16:0/1533591741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81b4106cd0 msgr2=0x7f81b419cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:08.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.970+0000 7f81b9934700 1 --2- 172.21.15.16:0/1533591741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81b4106cd0 0x7f81b419cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:08.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.970+0000 7f81b9934700 1 -- 172.21.15.16:0/1533591741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b00095f0 con 0x7f81b4068580 2024-04-22T15:58:08.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.970+0000 7f81b9934700 1 --2- 172.21.15.16:0/1533591741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81b4068580 0x7f81b419c670 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f81b0005f50 tx=0x7f81b000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:08.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.970+0000 7f81aaffd700 1 -- 172.21.15.16:0/1533591741 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81b001d070 con 0x7f81b4068580 2024-04-22T15:58:08.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.970+0000 7f81aaffd700 1 -- 172.21.15.16:0/1533591741 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f81b000b610 con 0x7f81b4068580 2024-04-22T15:58:08.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.970+0000 7f81bbb98700 1 -- 172.21.15.16:0/1533591741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b4196970 con 0x7f81b4068580 2024-04-22T15:58:08.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.970+0000 7f81aaffd700 1 -- 172.21.15.16:0/1533591741 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81b0022470 con 0x7f81b4068580 2024-04-22T15:58:08.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.970+0000 7f81bbb98700 1 -- 172.21.15.16:0/1533591741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b4196e60 con 0x7f81b4068580 2024-04-22T15:58:08.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.972+0000 7f81bbb98700 1 -- 172.21.15.16:0/1533591741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81b41091f0 con 0x7f81b4068580 2024-04-22T15:58:08.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.972+0000 7f81aaffd700 1 -- 172.21.15.16:0/1533591741 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f81b0003780 con 0x7f81b4068580 2024-04-22T15:58:08.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.972+0000 7f81aaffd700 1 --2- 172.21.15.16:0/1533591741 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f81a006ecc0 0x7f81a0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:08.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.973+0000 7f81aaffd700 1 -- 172.21.15.16:0/1533591741 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f81b0090060 con 0x7f81b4068580 2024-04-22T15:58:08.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.973+0000 7f81b9133700 1 --2- 172.21.15.16:0/1533591741 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f81a006ecc0 0x7f81a0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:08.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.973+0000 7f81b9133700 1 --2- 172.21.15.16:0/1533591741 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f81a006ecc0 0x7f81a0071180 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f81b4197b60 tx=0x7f81a4009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:08.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:08.978+0000 7f81aaffd700 1 -- 172.21.15.16:0/1533591741 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f81b005ca70 con 0x7f81b4068580 2024-04-22T15:58:09.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.234+0000 7f81bbb98700 1 -- 172.21.15.16:0/1533591741 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f81b4197c70 con 0x7f81a006ecc0 2024-04-22T15:58:09.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.236+0000 7f81aaffd700 1 -- 172.21.15.16:0/1533591741 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f81b4197c70 con 0x7f81a006ecc0 2024-04-22T15:58:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.238+0000 7f81bbb98700 1 -- 172.21.15.16:0/1533591741 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f81a006ecc0 msgr2=0x7f81a0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.238+0000 7f81bbb98700 1 --2- 172.21.15.16:0/1533591741 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f81a006ecc0 0x7f81a0071180 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f81b4197b60 tx=0x7f81a4009260 comp rx=0 tx=0).stop 2024-04-22T15:58:09.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.238+0000 7f81bbb98700 1 -- 172.21.15.16:0/1533591741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81b4068580 msgr2=0x7f81b419c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:09.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.238+0000 7f81bbb98700 1 --2- 172.21.15.16:0/1533591741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81b4068580 0x7f81b419c670 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f81b0005f50 tx=0x7f81b000f690 comp rx=0 tx=0).stop 2024-04-22T15:58:09.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.238+0000 7f81bbb98700 1 -- 172.21.15.16:0/1533591741 shutdown_connections 2024-04-22T15:58:09.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.238+0000 7f81bbb98700 1 --2- 172.21.15.16:0/1533591741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81b4068580 0x7f81b419c670 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:09.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.238+0000 7f81bbb98700 1 --2- 172.21.15.16:0/1533591741 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f81a006ecc0 0x7f81a0071180 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:09.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.238+0000 7f81bbb98700 1 --2- 172.21.15.16:0/1533591741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81b4106cd0 0x7f81b419cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:09.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.238+0000 7f81bbb98700 1 -- 172.21.15.16:0/1533591741 >> 172.21.15.16:0/1533591741 conn(0x7f81b4075670 msgr2=0x7f81b40fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:09.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.238+0000 7f81bbb98700 1 -- 172.21.15.16:0/1533591741 shutdown_connections 2024-04-22T15:58:09.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.238+0000 7f81bbb98700 1 -- 172.21.15.16:0/1533591741 wait complete. 2024-04-22T15:58:09.252 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:58:09.320 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:09 smithi016 ceph-mon[110740]: pgmap v173: 65 pgs: 7 peering, 58 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2024-04-22T15:58:09.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.356+0000 7f0e112ec700 1 -- 172.21.15.16:0/4256482956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e0c0ffc20 msgr2=0x7f0e0c100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:09.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.356+0000 7f0e112ec700 1 --2- 172.21.15.16:0/4256482956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e0c0ffc20 0x7f0e0c100080 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0e00009910 tx=0x7f0e00009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:09.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.356+0000 7f0e112ec700 1 -- 172.21.15.16:0/4256482956 shutdown_connections 2024-04-22T15:58:09.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.356+0000 7f0e112ec700 1 --2- 172.21.15.16:0/4256482956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e0c0ffc20 0x7f0e0c100080 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:09.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.356+0000 7f0e112ec700 1 --2- 172.21.15.16:0/4256482956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e0c105bd0 0x7f0e0c105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:09.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.356+0000 7f0e112ec700 1 -- 172.21.15.16:0/4256482956 >> 172.21.15.16:0/4256482956 conn(0x7f0e0c075670 msgr2=0x7f0e0c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:09.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.356+0000 7f0e112ec700 1 -- 172.21.15.16:0/4256482956 shutdown_connections 2024-04-22T15:58:09.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.356+0000 7f0e112ec700 1 -- 172.21.15.16:0/4256482956 wait complete. 2024-04-22T15:58:09.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.356+0000 7f0e112ec700 1 Processor -- start 2024-04-22T15:58:09.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.357+0000 7f0e112ec700 1 -- start start 2024-04-22T15:58:09.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.357+0000 7f0e112ec700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e0c0ffc20 0x7f0e0c197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:09.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.357+0000 7f0e112ec700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e0c105bd0 0x7f0e0c197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:09.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.357+0000 7f0e112ec700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c198380 con 0x7f0e0c105bd0 2024-04-22T15:58:09.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.357+0000 7f0e112ec700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c1984f0 con 0x7f0e0c0ffc20 2024-04-22T15:58:09.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.357+0000 7f0e0a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e0c105bd0 0x7f0e0c197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:09.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.357+0000 7f0e0affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e0c0ffc20 0x7f0e0c197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:09.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.357+0000 7f0e0a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e0c105bd0 0x7f0e0c197ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59106/0 (socket says 172.21.15.16:59106) 2024-04-22T15:58:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.357+0000 7f0e0a7fc700 1 -- 172.21.15.16:0/2783916813 learned_addr learned my addr 172.21.15.16:0/2783916813 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:58:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.357+0000 7f0e0a7fc700 1 -- 172.21.15.16:0/2783916813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e0c0ffc20 msgr2=0x7f0e0c197760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.357+0000 7f0e0a7fc700 1 --2- 172.21.15.16:0/2783916813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e0c0ffc20 0x7f0e0c197760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.357+0000 7f0e0a7fc700 1 -- 172.21.15.16:0/2783916813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e000095f0 con 0x7f0e0c105bd0 2024-04-22T15:58:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.357+0000 7f0e0affd700 1 --2- 172.21.15.16:0/2783916813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e0c0ffc20 0x7f0e0c197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:58:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.358+0000 7f0e0a7fc700 1 --2- 172.21.15.16:0/2783916813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e0c105bd0 0x7f0e0c197ca0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0e00005f50 tx=0x7f0e0000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.358+0000 7f0df3fff700 1 -- 172.21.15.16:0/2783916813 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e0001d070 con 0x7f0e0c105bd0 2024-04-22T15:58:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.358+0000 7f0df3fff700 1 -- 172.21.15.16:0/2783916813 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e0000b610 con 0x7f0e0c105bd0 2024-04-22T15:58:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.358+0000 7f0e112ec700 1 -- 172.21.15.16:0/2783916813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e0c1a07d0 con 0x7f0e0c105bd0 2024-04-22T15:58:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.358+0000 7f0df3fff700 1 -- 172.21.15.16:0/2783916813 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e00022470 con 0x7f0e0c105bd0 2024-04-22T15:58:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.358+0000 7f0e112ec700 1 -- 172.21.15.16:0/2783916813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e0c1a0c40 con 0x7f0e0c105bd0 2024-04-22T15:58:09.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.359+0000 7f0e112ec700 1 -- 172.21.15.16:0/2783916813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e0c1080d0 con 0x7f0e0c105bd0 2024-04-22T15:58:09.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.359+0000 7f0df3fff700 1 -- 172.21.15.16:0/2783916813 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0e000225d0 con 0x7f0e0c105bd0 2024-04-22T15:58:09.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.360+0000 7f0df3fff700 1 --2- 172.21.15.16:0/2783916813 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0df406ecc0 0x7f0df4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:09.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.360+0000 7f0df3fff700 1 -- 172.21.15.16:0/2783916813 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f0e00090180 con 0x7f0e0c105bd0 2024-04-22T15:58:09.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.360+0000 7f0e0affd700 1 --2- 172.21.15.16:0/2783916813 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0df406ecc0 0x7f0df4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:09.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.360+0000 7f0e0affd700 1 --2- 172.21.15.16:0/2783916813 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0df406ecc0 0x7f0df4071180 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc011490 tx=0x7f0dfc009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:09.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.365+0000 7f0df3fff700 1 -- 172.21.15.16:0/2783916813 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0e00058fe0 con 0x7f0e0c105bd0 2024-04-22T15:58:09.493 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:09 smithi027 ceph-mon[83054]: pgmap v173: 65 pgs: 7 peering, 58 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2024-04-22T15:58:09.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.642+0000 7f0e112ec700 1 -- 172.21.15.16:0/2783916813 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e0c068c50 con 0x7f0df406ecc0 2024-04-22T15:58:09.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.643+0000 7f0df3fff700 1 -- 172.21.15.16:0/2783916813 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0e0c068c50 con 0x7f0df406ecc0 2024-04-22T15:58:09.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.645+0000 7f0e112ec700 1 -- 172.21.15.16:0/2783916813 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0df406ecc0 msgr2=0x7f0df4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:09.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.645+0000 7f0e112ec700 1 --2- 172.21.15.16:0/2783916813 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0df406ecc0 0x7f0df4071180 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc011490 tx=0x7f0dfc009260 comp rx=0 tx=0).stop 2024-04-22T15:58:09.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.645+0000 7f0e112ec700 1 -- 172.21.15.16:0/2783916813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e0c105bd0 msgr2=0x7f0e0c197ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:09.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.645+0000 7f0e112ec700 1 --2- 172.21.15.16:0/2783916813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e0c105bd0 0x7f0e0c197ca0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0e00005f50 tx=0x7f0e0000f690 comp rx=0 tx=0).stop 2024-04-22T15:58:09.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.645+0000 7f0e112ec700 1 -- 172.21.15.16:0/2783916813 shutdown_connections 2024-04-22T15:58:09.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.645+0000 7f0e112ec700 1 --2- 172.21.15.16:0/2783916813 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0df406ecc0 0x7f0df4071180 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:09.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.645+0000 7f0e112ec700 1 --2- 172.21.15.16:0/2783916813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e0c0ffc20 0x7f0e0c197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:09.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.645+0000 7f0e112ec700 1 --2- 172.21.15.16:0/2783916813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e0c105bd0 0x7f0e0c197ca0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:09.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.645+0000 7f0e112ec700 1 -- 172.21.15.16:0/2783916813 >> 172.21.15.16:0/2783916813 conn(0x7f0e0c075670 msgr2=0x7f0e0c0fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:09.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.646+0000 7f0e112ec700 1 -- 172.21.15.16:0/2783916813 shutdown_connections 2024-04-22T15:58:09.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.646+0000 7f0e112ec700 1 -- 172.21.15.16:0/2783916813 wait complete. 2024-04-22T15:58:09.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.753+0000 7f438f477700 1 -- 172.21.15.16:0/4186289420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43881027a0 msgr2=0x7f438810ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:09.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.753+0000 7f438f477700 1 --2- 172.21.15.16:0/4186289420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43881027a0 0x7f438810ac90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f437c009910 tx=0x7f437c009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:09.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.753+0000 7f438f477700 1 -- 172.21.15.16:0/4186289420 shutdown_connections 2024-04-22T15:58:09.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.753+0000 7f438f477700 1 --2- 172.21.15.16:0/4186289420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43881027a0 0x7f438810ac90 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:09.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.753+0000 7f438f477700 1 --2- 172.21.15.16:0/4186289420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4388101e80 0x7f4388102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:09.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.753+0000 7f438f477700 1 -- 172.21.15.16:0/4186289420 >> 172.21.15.16:0/4186289420 conn(0x7f43880fb650 msgr2=0x7f43880fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:09.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.753+0000 7f438f477700 1 -- 172.21.15.16:0/4186289420 shutdown_connections 2024-04-22T15:58:09.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.753+0000 7f438f477700 1 -- 172.21.15.16:0/4186289420 wait complete. 2024-04-22T15:58:09.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.754+0000 7f438f477700 1 Processor -- start 2024-04-22T15:58:09.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.754+0000 7f438f477700 1 -- start start 2024-04-22T15:58:09.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.754+0000 7f438f477700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4388101e80 0x7f438819c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:09.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.754+0000 7f438f477700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43881027a0 0x7f438819cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:09.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.754+0000 7f438f477700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438819d220 con 0x7f43881027a0 2024-04-22T15:58:09.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.754+0000 7f438f477700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43881966d0 con 0x7f4388101e80 2024-04-22T15:58:09.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.754+0000 7f438ca12700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43881027a0 0x7f438819cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:09.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.754+0000 7f438ca12700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43881027a0 0x7f438819cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59120/0 (socket says 172.21.15.16:59120) 2024-04-22T15:58:09.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.754+0000 7f438ca12700 1 -- 172.21.15.16:0/2052814409 learned_addr learned my addr 172.21.15.16:0/2052814409 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:58:09.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.754+0000 7f438d213700 1 --2- 172.21.15.16:0/2052814409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4388101e80 0x7f438819c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:09.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.755+0000 7f438ca12700 1 -- 172.21.15.16:0/2052814409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4388101e80 msgr2=0x7f438819c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:09.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.755+0000 7f438ca12700 1 --2- 172.21.15.16:0/2052814409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4388101e80 0x7f438819c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:09.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.755+0000 7f438ca12700 1 -- 172.21.15.16:0/2052814409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f437c0095f0 con 0x7f43881027a0 2024-04-22T15:58:09.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.755+0000 7f438ca12700 1 --2- 172.21.15.16:0/2052814409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43881027a0 0x7f438819cb90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f437c0092e0 tx=0x7f437c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:09.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.755+0000 7f43867fc700 1 -- 172.21.15.16:0/2052814409 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f437c01d070 con 0x7f43881027a0 2024-04-22T15:58:09.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.755+0000 7f438f477700 1 -- 172.21.15.16:0/2052814409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4388196950 con 0x7f43881027a0 2024-04-22T15:58:09.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.755+0000 7f43867fc700 1 -- 172.21.15.16:0/2052814409 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f437c005dc0 con 0x7f43881027a0 2024-04-22T15:58:09.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.755+0000 7f43867fc700 1 -- 172.21.15.16:0/2052814409 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f437c017400 con 0x7f43881027a0 2024-04-22T15:58:09.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.755+0000 7f438f477700 1 -- 172.21.15.16:0/2052814409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4388196e40 con 0x7f43881027a0 2024-04-22T15:58:09.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.756+0000 7f438f477700 1 -- 172.21.15.16:0/2052814409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4388108320 con 0x7f43881027a0 2024-04-22T15:58:09.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.756+0000 7f43867fc700 1 -- 172.21.15.16:0/2052814409 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f437c00f6f0 con 0x7f43881027a0 2024-04-22T15:58:09.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.757+0000 7f43867fc700 1 --2- 172.21.15.16:0/2052814409 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f437006ecc0 0x7f4370071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:09.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.757+0000 7f43867fc700 1 -- 172.21.15.16:0/2052814409 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f437c0901a0 con 0x7f43881027a0 2024-04-22T15:58:09.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.757+0000 7f438d213700 1 --2- 172.21.15.16:0/2052814409 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f437006ecc0 0x7f4370071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:09.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.758+0000 7f438d213700 1 --2- 172.21.15.16:0/2052814409 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f437006ecc0 0x7f4370071180 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4380009550 tx=0x7f43800091b0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:09.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.762+0000 7f43867fc700 1 -- 172.21.15.16:0/2052814409 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f437c05cb30 con 0x7f43881027a0 2024-04-22T15:58:09.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.982+0000 7f438f477700 1 -- 172.21.15.16:0/2052814409 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4388197060 con 0x7f437006ecc0 2024-04-22T15:58:09.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.993+0000 7f43867fc700 1 -- 172.21.15.16:0/2052814409 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f4388197060 con 0x7f437006ecc0 2024-04-22T15:58:09.996 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:58:09.996 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.smithi016 smithi016 *:9093,9094 running (6m) 113s ago 17m 25.6M - 0.25.0 c8568f914cd2 ea0e0217d6f9 2024-04-22T15:58:09.996 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi016 smithi016 running (17m) 113s ago 17m 8640k - 18.2.1 5be31c24972a f8b627d3f41e 2024-04-22T15:58:09.996 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi027 smithi027 running (16m) 11s ago 16m 8690k - 18.2.1 5be31c24972a f00fd40b4220 2024-04-22T15:58:09.996 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi016 smithi016 running (3m) 113s ago 17m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 1feb008449ed 2024-04-22T15:58:09.996 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi027 smithi027 running (3m) 11s ago 16m 6656k - 19.0.0-3223-g5883d494 3d983c26e278 c7f862c6da00 2024-04-22T15:58:09.996 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.smithi016 smithi016 *:3000 running (6m) 113s ago 16m 78.9M - 9.4.12 2bacad6d85d8 388a3c9c1b3f 2024-04-22T15:58:09.997 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.fkkejm smithi016 running (11m) 113s ago 11m 17.0M - 18.2.1 5be31c24972a 4153ac94cea0 2024-04-22T15:58:09.997 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.pgvphu smithi016 running (11m) 113s ago 11m 92.0M - 18.2.1 5be31c24972a 93dc072068a2 2024-04-22T15:58:09.997 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.dgqyes smithi027 running (11m) 11s ago 11m 20.6M - 18.2.1 5be31c24972a 41a9220430fd 2024-04-22T15:58:09.997 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.iobjds smithi027 running (11m) 11s ago 11m 149M - 18.2.1 5be31c24972a c1c8d6770f02 2024-04-22T15:58:09.997 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi016.rnsnnr smithi016 *:8443,9283,8765 running (8m) 113s ago 18m 505M - 19.0.0-3223-g5883d494 3d983c26e278 09b4fe285bc4 2024-04-22T15:58:09.997 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi027.ukdgzs smithi027 *:8443,9283,8765 running (7m) 11s ago 16m 451M - 19.0.0-3223-g5883d494 3d983c26e278 7548da9b5eb9 2024-04-22T15:58:09.997 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi016 smithi016 running (4m) 113s ago 18m 49.6M 2048M 19.0.0-3223-g5883d494 3d983c26e278 4ae529a67806 2024-04-22T15:58:09.997 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi027 smithi027 running (4m) 11s ago 16m 41.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 f29fcb888801 2024-04-22T15:58:09.997 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi016 smithi016 *:9100 running (7m) 113s ago 17m 16.1M - 1.5.0 0da6a335fe13 35e46e4d2f3f 2024-04-22T15:58:09.997 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi027 smithi027 *:9100 running (7m) 11s ago 16m 15.0M - 1.5.0 0da6a335fe13 e682a7672e9b 2024-04-22T15:58:09.997 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (3m) 113s ago 15m 88.0M 1217M 19.0.0-3223-g5883d494 3d983c26e278 79217eb383a2 2024-04-22T15:58:09.997 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (2m) 113s ago 14m 89.0M 1217M 19.0.0-3223-g5883d494 3d983c26e278 ec1a45b3c53e 2024-04-22T15:58:09.997 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (119s) 113s ago 14m 12.6M 1217M 19.0.0-3223-g5883d494 3d983c26e278 48bb459fd021 2024-04-22T15:58:09.997 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi027 running (82s) 11s ago 13m 97.2M 2240M 19.0.0-3223-g5883d494 3d983c26e278 81034bf22d3f 2024-04-22T15:58:09.997 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (49s) 11s ago 13m 94.9M 2240M 19.0.0-3223-g5883d494 3d983c26e278 93b660e5ecb6 2024-04-22T15:58:09.998 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi027 running (15s) 11s ago 13m 14.2M 2240M 19.0.0-3223-g5883d494 3d983c26e278 d857c152994a 2024-04-22T15:58:09.998 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.smithi016 smithi016 *:9095 running (6m) 113s ago 16m 73.8M - 2.43.0 a07b618ecd1d 373eaadee82d 2024-04-22T15:58:09.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.995+0000 7f438f477700 1 -- 172.21.15.16:0/2052814409 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f437006ecc0 msgr2=0x7f4370071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:09.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.995+0000 7f438f477700 1 --2- 172.21.15.16:0/2052814409 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f437006ecc0 0x7f4370071180 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4380009550 tx=0x7f43800091b0 comp rx=0 tx=0).stop 2024-04-22T15:58:09.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.995+0000 7f438f477700 1 -- 172.21.15.16:0/2052814409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43881027a0 msgr2=0x7f438819cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:09.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.995+0000 7f438f477700 1 --2- 172.21.15.16:0/2052814409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43881027a0 0x7f438819cb90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f437c0092e0 tx=0x7f437c00bd70 comp rx=0 tx=0).stop 2024-04-22T15:58:09.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.995+0000 7f438f477700 1 -- 172.21.15.16:0/2052814409 shutdown_connections 2024-04-22T15:58:09.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.995+0000 7f438f477700 1 --2- 172.21.15.16:0/2052814409 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f437006ecc0 0x7f4370071180 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:09.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.995+0000 7f438f477700 1 --2- 172.21.15.16:0/2052814409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4388101e80 0x7f438819c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:09.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.995+0000 7f438f477700 1 --2- 172.21.15.16:0/2052814409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43881027a0 0x7f438819cb90 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:09.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.995+0000 7f438f477700 1 -- 172.21.15.16:0/2052814409 >> 172.21.15.16:0/2052814409 conn(0x7f43880fb650 msgr2=0x7f4388105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:09.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.996+0000 7f438f477700 1 -- 172.21.15.16:0/2052814409 shutdown_connections 2024-04-22T15:58:10.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:09.996+0000 7f438f477700 1 -- 172.21.15.16:0/2052814409 wait complete. 2024-04-22T15:58:10.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.101+0000 7fa19e9d8700 1 -- 172.21.15.16:0/3963123759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa198101840 msgr2=0x7fa198105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:10.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.101+0000 7fa19e9d8700 1 --2- 172.21.15.16:0/3963123759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa198101840 0x7fa198105890 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa18c009910 tx=0x7fa18c009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:10.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.101+0000 7fa19e9d8700 1 -- 172.21.15.16:0/3963123759 shutdown_connections 2024-04-22T15:58:10.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.101+0000 7fa19e9d8700 1 --2- 172.21.15.16:0/3963123759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa198101840 0x7fa198105890 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:10.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.101+0000 7fa19e9d8700 1 --2- 172.21.15.16:0/3963123759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa198100e90 0x7fa198101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:10.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.101+0000 7fa19e9d8700 1 -- 172.21.15.16:0/3963123759 >> 172.21.15.16:0/3963123759 conn(0x7fa1980fc740 msgr2=0x7fa1980feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:10.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.102+0000 7fa19e9d8700 1 -- 172.21.15.16:0/3963123759 shutdown_connections 2024-04-22T15:58:10.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.102+0000 7fa19e9d8700 1 -- 172.21.15.16:0/3963123759 wait complete. 2024-04-22T15:58:10.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.102+0000 7fa19e9d8700 1 Processor -- start 2024-04-22T15:58:10.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.102+0000 7fa19e9d8700 1 -- start start 2024-04-22T15:58:10.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.102+0000 7fa19e9d8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa198100e90 0x7fa198073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:10.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.102+0000 7fa19e9d8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa198101840 0x7fa1980735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:10.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.102+0000 7fa19e9d8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa198073b10 con 0x7fa198101840 2024-04-22T15:58:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.102+0000 7fa19e9d8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa198073c50 con 0x7fa198100e90 2024-04-22T15:58:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.103+0000 7fa1977fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa198101840 0x7fa1980735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.103+0000 7fa1977fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa198101840 0x7fa1980735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59146/0 (socket says 172.21.15.16:59146) 2024-04-22T15:58:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.103+0000 7fa197fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa198100e90 0x7fa198073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.103+0000 7fa1977fe700 1 -- 172.21.15.16:0/1087330398 learned_addr learned my addr 172.21.15.16:0/1087330398 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:58:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.103+0000 7fa1977fe700 1 -- 172.21.15.16:0/1087330398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa198100e90 msgr2=0x7fa198073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.103+0000 7fa1977fe700 1 --2- 172.21.15.16:0/1087330398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa198100e90 0x7fa198073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.103+0000 7fa1977fe700 1 -- 172.21.15.16:0/1087330398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa18c0095f0 con 0x7fa198101840 2024-04-22T15:58:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.103+0000 7fa197fff700 1 --2- 172.21.15.16:0/1087330398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa198100e90 0x7fa198073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:58:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.103+0000 7fa1977fe700 1 --2- 172.21.15.16:0/1087330398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa198101840 0x7fa1980735d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa18c0092e0 tx=0x7fa18c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.103+0000 7fa1957fa700 1 -- 172.21.15.16:0/1087330398 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa18c01d070 con 0x7fa198101840 2024-04-22T15:58:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.103+0000 7fa1957fa700 1 -- 172.21.15.16:0/1087330398 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa18c005dc0 con 0x7fa198101840 2024-04-22T15:58:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.103+0000 7fa1957fa700 1 -- 172.21.15.16:0/1087330398 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa18c017400 con 0x7fa198101840 2024-04-22T15:58:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.103+0000 7fa19e9d8700 1 -- 172.21.15.16:0/1087330398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa198073ed0 con 0x7fa198101840 2024-04-22T15:58:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.103+0000 7fa19e9d8700 1 -- 172.21.15.16:0/1087330398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1981a2e40 con 0x7fa198101840 2024-04-22T15:58:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.104+0000 7fa19e9d8700 1 -- 172.21.15.16:0/1087330398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa198109290 con 0x7fa198101840 2024-04-22T15:58:10.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.105+0000 7fa1957fa700 1 -- 172.21.15.16:0/1087330398 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa18c00f6f0 con 0x7fa198101840 2024-04-22T15:58:10.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.105+0000 7fa1957fa700 1 --2- 172.21.15.16:0/1087330398 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fa18006ecc0 0x7fa180071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:10.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.106+0000 7fa1957fa700 1 -- 172.21.15.16:0/1087330398 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa18c0901a0 con 0x7fa198101840 2024-04-22T15:58:10.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.106+0000 7fa197fff700 1 --2- 172.21.15.16:0/1087330398 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fa18006ecc0 0x7fa180071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:10.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.106+0000 7fa197fff700 1 --2- 172.21.15.16:0/1087330398 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fa18006ecc0 0x7fa180071180 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa188005950 tx=0x7fa18800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.110+0000 7fa1957fa700 1 -- 172.21.15.16:0/1087330398 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa18c059000 con 0x7fa198101840 2024-04-22T15:58:10.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.418+0000 7fa19e9d8700 1 -- 172.21.15.16:0/1087330398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa19804f2e0 con 0x7fa198101840 2024-04-22T15:58:10.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.419+0000 7fa1957fa700 1 -- 172.21.15.16:0/1087330398 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+701 (secure 0 0 0) 0x7fa18c05c6c0 con 0x7fa198101840 2024-04-22T15:58:10.422 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:58:10.422 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-22T15:58:10.422 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:58:10.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:58:10.422 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-22T15:58:10.422 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:58:10.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:58:10.423 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-22T15:58:10.423 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:58:10.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:58:10.423 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-22T15:58:10.423 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 3 2024-04-22T15:58:10.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:58:10.423 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-22T15:58:10.423 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 3, 2024-04-22T15:58:10.423 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 10 2024-04-22T15:58:10.423 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-22T15:58:10.423 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:58:10.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.421+0000 7fa19e9d8700 1 -- 172.21.15.16:0/1087330398 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fa18006ecc0 msgr2=0x7fa180071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:10.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.421+0000 7fa19e9d8700 1 --2- 172.21.15.16:0/1087330398 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fa18006ecc0 0x7fa180071180 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa188005950 tx=0x7fa18800b410 comp rx=0 tx=0).stop 2024-04-22T15:58:10.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.421+0000 7fa19e9d8700 1 -- 172.21.15.16:0/1087330398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa198101840 msgr2=0x7fa1980735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:10.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.421+0000 7fa19e9d8700 1 --2- 172.21.15.16:0/1087330398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa198101840 0x7fa1980735d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa18c0092e0 tx=0x7fa18c00bd70 comp rx=0 tx=0).stop 2024-04-22T15:58:10.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.421+0000 7fa19e9d8700 1 -- 172.21.15.16:0/1087330398 shutdown_connections 2024-04-22T15:58:10.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.421+0000 7fa19e9d8700 1 --2- 172.21.15.16:0/1087330398 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fa18006ecc0 0x7fa180071180 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:10.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.421+0000 7fa19e9d8700 1 --2- 172.21.15.16:0/1087330398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa198100e90 0x7fa198073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:10.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.421+0000 7fa19e9d8700 1 --2- 172.21.15.16:0/1087330398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa198101840 0x7fa1980735d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:10.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.421+0000 7fa19e9d8700 1 -- 172.21.15.16:0/1087330398 >> 172.21.15.16:0/1087330398 conn(0x7fa1980fc740 msgr2=0x7fa1980fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:10.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.421+0000 7fa19e9d8700 1 -- 172.21.15.16:0/1087330398 shutdown_connections 2024-04-22T15:58:10.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.421+0000 7fa19e9d8700 1 -- 172.21.15.16:0/1087330398 wait complete. 2024-04-22T15:58:10.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.528+0000 7fdb84f32700 1 -- 172.21.15.16:0/4282785485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb80101850 msgr2=0x7fdb801058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:10.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.528+0000 7fdb84f32700 1 --2- 172.21.15.16:0/4282785485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb80101850 0x7fdb801058a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdb70009910 tx=0x7fdb70009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:10.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.528+0000 7fdb84f32700 1 -- 172.21.15.16:0/4282785485 shutdown_connections 2024-04-22T15:58:10.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.528+0000 7fdb84f32700 1 --2- 172.21.15.16:0/4282785485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb80101850 0x7fdb801058a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:10.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.528+0000 7fdb84f32700 1 --2- 172.21.15.16:0/4282785485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb80100ea0 0x7fdb80101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:10.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.528+0000 7fdb84f32700 1 -- 172.21.15.16:0/4282785485 >> 172.21.15.16:0/4282785485 conn(0x7fdb800fc750 msgr2=0x7fdb800feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:10.532 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:10 smithi016 ceph-mon[110740]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/258 objects degraded (7.752%), 8 pgs degraded) 2024-04-22T15:58:10.532 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:10 smithi016 ceph-mon[110740]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:10.532 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:10 smithi016 ceph-mon[110740]: from='client.34384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:10.532 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:10 smithi016 ceph-mon[110740]: from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.528+0000 7fdb84f32700 1 -- 172.21.15.16:0/4282785485 shutdown_connections 2024-04-22T15:58:10.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.528+0000 7fdb84f32700 1 -- 172.21.15.16:0/4282785485 wait complete. 2024-04-22T15:58:10.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.529+0000 7fdb84f32700 1 Processor -- start 2024-04-22T15:58:10.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.529+0000 7fdb84f32700 1 -- start start 2024-04-22T15:58:10.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.529+0000 7fdb84f32700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb80100ea0 0x7fdb8019c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:10.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.529+0000 7fdb84f32700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb80101850 0x7fdb8019cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:10.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.529+0000 7fdb84f32700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8019d220 con 0x7fdb80100ea0 2024-04-22T15:58:10.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.529+0000 7fdb84f32700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb801966d0 con 0x7fdb80101850 2024-04-22T15:58:10.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.529+0000 7fdb7e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb80100ea0 0x7fdb8019c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:10.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.529+0000 7fdb7e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb80100ea0 0x7fdb8019c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59156/0 (socket says 172.21.15.16:59156) 2024-04-22T15:58:10.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.529+0000 7fdb7e59c700 1 -- 172.21.15.16:0/2568286277 learned_addr learned my addr 172.21.15.16:0/2568286277 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:58:10.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.529+0000 7fdb7dd9b700 1 --2- 172.21.15.16:0/2568286277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb80101850 0x7fdb8019cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:10.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.530+0000 7fdb7e59c700 1 -- 172.21.15.16:0/2568286277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb80101850 msgr2=0x7fdb8019cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:10.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.530+0000 7fdb7e59c700 1 --2- 172.21.15.16:0/2568286277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb80101850 0x7fdb8019cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:10.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.530+0000 7fdb7e59c700 1 -- 172.21.15.16:0/2568286277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb700095f0 con 0x7fdb80100ea0 2024-04-22T15:58:10.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.530+0000 7fdb7e59c700 1 --2- 172.21.15.16:0/2568286277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb80100ea0 0x7fdb8019c650 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdb7400d900 tx=0x7fdb7400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:10.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.530+0000 7fdb6f7fe700 1 -- 172.21.15.16:0/2568286277 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb74004a00 con 0x7fdb80100ea0 2024-04-22T15:58:10.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.530+0000 7fdb6f7fe700 1 -- 172.21.15.16:0/2568286277 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb740054e0 con 0x7fdb80100ea0 2024-04-22T15:58:10.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.530+0000 7fdb84f32700 1 -- 172.21.15.16:0/2568286277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb801969b0 con 0x7fdb80100ea0 2024-04-22T15:58:10.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.530+0000 7fdb6f7fe700 1 -- 172.21.15.16:0/2568286277 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb74009d00 con 0x7fdb80100ea0 2024-04-22T15:58:10.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.530+0000 7fdb84f32700 1 -- 172.21.15.16:0/2568286277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb80196f00 con 0x7fdb80100ea0 2024-04-22T15:58:10.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.531+0000 7fdb84f32700 1 -- 172.21.15.16:0/2568286277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb801091b0 con 0x7fdb80100ea0 2024-04-22T15:58:10.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.532+0000 7fdb6f7fe700 1 -- 172.21.15.16:0/2568286277 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdb74005020 con 0x7fdb80100ea0 2024-04-22T15:58:10.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.532+0000 7fdb6f7fe700 1 --2- 172.21.15.16:0/2568286277 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdb6806ecc0 0x7fdb68071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:10.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.532+0000 7fdb6f7fe700 1 -- 172.21.15.16:0/2568286277 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fdb7408ed00 con 0x7fdb80100ea0 2024-04-22T15:58:10.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.532+0000 7fdb7dd9b700 1 --2- 172.21.15.16:0/2568286277 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdb6806ecc0 0x7fdb68071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:10.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.533+0000 7fdb7dd9b700 1 --2- 172.21.15.16:0/2568286277 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdb6806ecc0 0x7fdb68071180 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdb80197b40 tx=0x7fdb7001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:10.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.537+0000 7fdb6f7fe700 1 -- 172.21.15.16:0/2568286277 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fdb74092030 con 0x7fdb80100ea0 2024-04-22T15:58:10.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:10 smithi027 ceph-mon[83054]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/258 objects degraded (7.752%), 8 pgs degraded) 2024-04-22T15:58:10.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:10 smithi027 ceph-mon[83054]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:10.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:10 smithi027 ceph-mon[83054]: from='client.34384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:10.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:10 smithi027 ceph-mon[83054]: from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:10.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.793+0000 7fdb84f32700 1 -- 172.21.15.16:0/2568286277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fdb8004f2e0 con 0x7fdb80100ea0 2024-04-22T15:58:10.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.793+0000 7fdb6f7fe700 1 -- 172.21.15.16:0/2568286277 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 19 v19) v1 ==== 76+0+1707 (secure 0 0 0) 0x7fdb740926c0 con 0x7fdb80100ea0 2024-04-22T15:58:10.797 INFO:teuthology.orchestra.run.smithi016.stdout:e19 2024-04-22T15:58:10.797 INFO:teuthology.orchestra.run.smithi016.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:58:10.797 INFO:teuthology.orchestra.run.smithi016.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:58:10.797 INFO:teuthology.orchestra.run.smithi016.stdout:legacy client fscid: 1 2024-04-22T15:58:10.797 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:58:10.797 INFO:teuthology.orchestra.run.smithi016.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:58:10.797 INFO:teuthology.orchestra.run.smithi016.stdout:fs_name cephfs 2024-04-22T15:58:10.797 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 17 2024-04-22T15:58:10.797 INFO:teuthology.orchestra.run.smithi016.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-22T15:58:10.797 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-04-22T15:46:27.774571+0000 2024-04-22T15:58:10.797 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-04-22T15:58:05.206233+0000 2024-04-22T15:58:10.797 INFO:teuthology.orchestra.run.smithi016.stdout:tableserver 0 2024-04-22T15:58:10.798 INFO:teuthology.orchestra.run.smithi016.stdout:root 0 2024-04-22T15:58:10.798 INFO:teuthology.orchestra.run.smithi016.stdout:session_timeout 60 2024-04-22T15:58:10.798 INFO:teuthology.orchestra.run.smithi016.stdout:session_autoclose 300 2024-04-22T15:58:10.798 INFO:teuthology.orchestra.run.smithi016.stdout:max_file_size 1099511627776 2024-04-22T15:58:10.798 INFO:teuthology.orchestra.run.smithi016.stdout:max_xattr_size 65536 2024-04-22T15:58:10.798 INFO:teuthology.orchestra.run.smithi016.stdout:required_client_features {} 2024-04-22T15:58:10.798 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure 0 2024-04-22T15:58:10.798 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure_osd_epoch 0 2024-04-22T15:58:10.798 INFO:teuthology.orchestra.run.smithi016.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:58:10.798 INFO:teuthology.orchestra.run.smithi016.stdout:max_mds 1 2024-04-22T15:58:10.799 INFO:teuthology.orchestra.run.smithi016.stdout:in 0 2024-04-22T15:58:10.799 INFO:teuthology.orchestra.run.smithi016.stdout:up {0=24279} 2024-04-22T15:58:10.799 INFO:teuthology.orchestra.run.smithi016.stdout:failed 2024-04-22T15:58:10.799 INFO:teuthology.orchestra.run.smithi016.stdout:damaged 2024-04-22T15:58:10.799 INFO:teuthology.orchestra.run.smithi016.stdout:stopped 2024-04-22T15:58:10.799 INFO:teuthology.orchestra.run.smithi016.stdout:data_pools [3] 2024-04-22T15:58:10.799 INFO:teuthology.orchestra.run.smithi016.stdout:metadata_pool 2 2024-04-22T15:58:10.799 INFO:teuthology.orchestra.run.smithi016.stdout:inline_data enabled 2024-04-22T15:58:10.799 INFO:teuthology.orchestra.run.smithi016.stdout:balancer 2024-04-22T15:58:10.799 INFO:teuthology.orchestra.run.smithi016.stdout:bal_rank_mask -1 2024-04-22T15:58:10.799 INFO:teuthology.orchestra.run.smithi016.stdout:standby_count_wanted 1 2024-04-22T15:58:10.799 INFO:teuthology.orchestra.run.smithi016.stdout:qdb_cluster leader: 24279 members: 24279 2024-04-22T15:58:10.800 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.iobjds{0:24279} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.27:6824/4216277609,v1:172.21.15.27:6825/4216277609] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:58:10.800 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:58:10.800 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:58:10.800 INFO:teuthology.orchestra.run.smithi016.stdout:Standby daemons: 2024-04-22T15:58:10.800 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:58:10.800 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.dgqyes{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.27:6826/476026355,v1:172.21.15.27:6827/476026355] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:58:10.800 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.pgvphu{-1:34376} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.16:6828/3518018286,v1:172.21.15.16:6829/3518018286] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:58:10.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.796+0000 7fdb84f32700 1 -- 172.21.15.16:0/2568286277 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdb6806ecc0 msgr2=0x7fdb68071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:10.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.796+0000 7fdb84f32700 1 --2- 172.21.15.16:0/2568286277 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdb6806ecc0 0x7fdb68071180 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdb80197b40 tx=0x7fdb7001a040 comp rx=0 tx=0).stop 2024-04-22T15:58:10.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.796+0000 7fdb84f32700 1 -- 172.21.15.16:0/2568286277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb80100ea0 msgr2=0x7fdb8019c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:10.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.796+0000 7fdb84f32700 1 --2- 172.21.15.16:0/2568286277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb80100ea0 0x7fdb8019c650 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdb7400d900 tx=0x7fdb7400dc10 comp rx=0 tx=0).stop 2024-04-22T15:58:10.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.796+0000 7fdb84f32700 1 -- 172.21.15.16:0/2568286277 shutdown_connections 2024-04-22T15:58:10.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.796+0000 7fdb84f32700 1 --2- 172.21.15.16:0/2568286277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb80100ea0 0x7fdb8019c650 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:10.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.796+0000 7fdb84f32700 1 --2- 172.21.15.16:0/2568286277 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdb6806ecc0 0x7fdb68071180 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:10.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.796+0000 7fdb84f32700 1 --2- 172.21.15.16:0/2568286277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb80101850 0x7fdb8019cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:10.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.796+0000 7fdb84f32700 1 -- 172.21.15.16:0/2568286277 >> 172.21.15.16:0/2568286277 conn(0x7fdb800fc750 msgr2=0x7fdb80105110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:10.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.796+0000 7fdb84f32700 1 -- 172.21.15.16:0/2568286277 shutdown_connections 2024-04-22T15:58:10.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.796+0000 7fdb84f32700 1 -- 172.21.15.16:0/2568286277 wait complete. 2024-04-22T15:58:10.802 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 19 2024-04-22T15:58:10.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.904+0000 7f97da68c700 1 -- 172.21.15.16:0/2546680273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97d40ff600 msgr2=0x7f97d40ff9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:10.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.904+0000 7f97da68c700 1 --2- 172.21.15.16:0/2546680273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97d40ff600 0x7f97d40ff9e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f97c4009910 tx=0x7f97c4009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:10.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.904+0000 7f97da68c700 1 -- 172.21.15.16:0/2546680273 shutdown_connections 2024-04-22T15:58:10.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.904+0000 7f97da68c700 1 --2- 172.21.15.16:0/2546680273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d40fffb0 0x7f97d41058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:10.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.904+0000 7f97da68c700 1 --2- 172.21.15.16:0/2546680273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97d40ff600 0x7f97d40ff9e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:10.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.904+0000 7f97da68c700 1 -- 172.21.15.16:0/2546680273 >> 172.21.15.16:0/2546680273 conn(0x7f97d4074df0 msgr2=0x7f97d4075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:10.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.905+0000 7f97da68c700 1 -- 172.21.15.16:0/2546680273 shutdown_connections 2024-04-22T15:58:10.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.905+0000 7f97da68c700 1 -- 172.21.15.16:0/2546680273 wait complete. 2024-04-22T15:58:10.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.905+0000 7f97da68c700 1 Processor -- start 2024-04-22T15:58:10.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.905+0000 7f97da68c700 1 -- start start 2024-04-22T15:58:10.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.905+0000 7f97da68c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97d40ff600 0x7f97d4100580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:10.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.905+0000 7f97da68c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d40fffb0 0x7f97d4100ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:10.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.906+0000 7f97da68c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d4101000 con 0x7f97d40ff600 2024-04-22T15:58:10.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.906+0000 7f97da68c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d4101140 con 0x7f97d40fffb0 2024-04-22T15:58:10.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.906+0000 7f97d3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97d40ff600 0x7f97d4100580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:10.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.906+0000 7f97d3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97d40ff600 0x7f97d4100580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59164/0 (socket says 172.21.15.16:59164) 2024-04-22T15:58:10.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.906+0000 7f97d3fff700 1 -- 172.21.15.16:0/3547314341 learned_addr learned my addr 172.21.15.16:0/3547314341 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:58:10.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.906+0000 7f97d37fe700 1 --2- 172.21.15.16:0/3547314341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d40fffb0 0x7f97d4100ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:10.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.906+0000 7f97d3fff700 1 -- 172.21.15.16:0/3547314341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d40fffb0 msgr2=0x7f97d4100ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:10.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.906+0000 7f97d3fff700 1 --2- 172.21.15.16:0/3547314341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d40fffb0 0x7f97d4100ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:10.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.906+0000 7f97d3fff700 1 -- 172.21.15.16:0/3547314341 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c40095f0 con 0x7f97d40ff600 2024-04-22T15:58:10.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.906+0000 7f97d3fff700 1 --2- 172.21.15.16:0/3547314341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97d40ff600 0x7f97d4100580 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f97c400b5c0 tx=0x7f97c400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:10.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.906+0000 7f97d17fa700 1 -- 172.21.15.16:0/3547314341 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97c401d070 con 0x7f97d40ff600 2024-04-22T15:58:10.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.906+0000 7f97d17fa700 1 -- 172.21.15.16:0/3547314341 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f97c400f460 con 0x7f97d40ff600 2024-04-22T15:58:10.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.906+0000 7f97da68c700 1 -- 172.21.15.16:0/3547314341 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d41013c0 con 0x7f97d40ff600 2024-04-22T15:58:10.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.906+0000 7f97d17fa700 1 -- 172.21.15.16:0/3547314341 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97c4017630 con 0x7f97d40ff600 2024-04-22T15:58:10.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.906+0000 7f97da68c700 1 -- 172.21.15.16:0/3547314341 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97d41a2dd0 con 0x7f97d40ff600 2024-04-22T15:58:10.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.908+0000 7f97da68c700 1 -- 172.21.15.16:0/3547314341 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97d4109270 con 0x7f97d40ff600 2024-04-22T15:58:10.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.908+0000 7f97d17fa700 1 -- 172.21.15.16:0/3547314341 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f97c400f5d0 con 0x7f97d40ff600 2024-04-22T15:58:10.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.908+0000 7f97d17fa700 1 --2- 172.21.15.16:0/3547314341 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f97bc06ecc0 0x7f97bc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:10.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.908+0000 7f97d17fa700 1 -- 172.21.15.16:0/3547314341 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f97c4090210 con 0x7f97d40ff600 2024-04-22T15:58:10.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.908+0000 7f97d37fe700 1 --2- 172.21.15.16:0/3547314341 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f97bc06ecc0 0x7f97bc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:10.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.909+0000 7f97d37fe700 1 --2- 172.21.15.16:0/3547314341 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f97bc06ecc0 0x7f97bc071180 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f97d4101d70 tx=0x7f97c800a380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:10.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:10.913+0000 7f97d17fa700 1 -- 172.21.15.16:0/3547314341 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f97c40590f0 con 0x7f97d40ff600 2024-04-22T15:58:11.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.141+0000 7f97da68c700 1 -- 172.21.15.16:0/3547314341 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f97d40611d0 con 0x7f97bc06ecc0 2024-04-22T15:58:11.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.143+0000 7f97d17fa700 1 -- 172.21.15.16:0/3547314341 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f97d40611d0 con 0x7f97bc06ecc0 2024-04-22T15:58:11.146 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:58:11.146 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:58:11.146 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-22T15:58:11.146 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:58:11.146 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-22T15:58:11.146 INFO:teuthology.orchestra.run.smithi016.stdout: "osd", 2024-04-22T15:58:11.146 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-04-22T15:58:11.146 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr", 2024-04-22T15:58:11.146 INFO:teuthology.orchestra.run.smithi016.stdout: "mon" 2024-04-22T15:58:11.146 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-22T15:58:11.147 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "12/23 daemons upgraded", 2024-04-22T15:58:11.147 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:58:11.147 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-22T15:58:11.147 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:58:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.145+0000 7f97da68c700 1 -- 172.21.15.16:0/3547314341 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f97bc06ecc0 msgr2=0x7f97bc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.145+0000 7f97da68c700 1 --2- 172.21.15.16:0/3547314341 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f97bc06ecc0 0x7f97bc071180 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f97d4101d70 tx=0x7f97c800a380 comp rx=0 tx=0).stop 2024-04-22T15:58:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.145+0000 7f97da68c700 1 -- 172.21.15.16:0/3547314341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97d40ff600 msgr2=0x7f97d4100580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.145+0000 7f97da68c700 1 --2- 172.21.15.16:0/3547314341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97d40ff600 0x7f97d4100580 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f97c400b5c0 tx=0x7f97c400bd50 comp rx=0 tx=0).stop 2024-04-22T15:58:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.145+0000 7f97da68c700 1 -- 172.21.15.16:0/3547314341 shutdown_connections 2024-04-22T15:58:11.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.145+0000 7f97da68c700 1 --2- 172.21.15.16:0/3547314341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97d40ff600 0x7f97d4100580 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:11.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.145+0000 7f97da68c700 1 --2- 172.21.15.16:0/3547314341 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f97bc06ecc0 0x7f97bc071180 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:11.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.145+0000 7f97da68c700 1 --2- 172.21.15.16:0/3547314341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d40fffb0 0x7f97d4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:11.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.145+0000 7f97da68c700 1 -- 172.21.15.16:0/3547314341 >> 172.21.15.16:0/3547314341 conn(0x7f97d4074df0 msgr2=0x7f97d40fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:11.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.145+0000 7f97da68c700 1 -- 172.21.15.16:0/3547314341 shutdown_connections 2024-04-22T15:58:11.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.146+0000 7f97da68c700 1 -- 172.21.15.16:0/3547314341 wait complete. 2024-04-22T15:58:11.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.256+0000 7f7bef65e700 1 -- 172.21.15.16:0/3230678110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7be8100ea0 msgr2=0x7f7be8101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:11.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.256+0000 7f7bef65e700 1 --2- 172.21.15.16:0/3230678110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7be8100ea0 0x7f7be8101280 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7be4009910 tx=0x7f7be4009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:11.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.256+0000 7f7bef65e700 1 -- 172.21.15.16:0/3230678110 shutdown_connections 2024-04-22T15:58:11.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.256+0000 7f7bef65e700 1 --2- 172.21.15.16:0/3230678110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7be8101850 0x7f7be81058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:11.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.256+0000 7f7bef65e700 1 --2- 172.21.15.16:0/3230678110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7be8100ea0 0x7f7be8101280 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:11.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.256+0000 7f7bef65e700 1 -- 172.21.15.16:0/3230678110 >> 172.21.15.16:0/3230678110 conn(0x7f7be80fc750 msgr2=0x7f7be80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:11.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.256+0000 7f7bef65e700 1 -- 172.21.15.16:0/3230678110 shutdown_connections 2024-04-22T15:58:11.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.256+0000 7f7bef65e700 1 -- 172.21.15.16:0/3230678110 wait complete. 2024-04-22T15:58:11.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.257+0000 7f7bef65e700 1 Processor -- start 2024-04-22T15:58:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.257+0000 7f7bef65e700 1 -- start start 2024-04-22T15:58:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.257+0000 7f7bef65e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7be8100ea0 0x7f7be8073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.257+0000 7f7bef65e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7be8101850 0x7f7be80735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.257+0000 7f7bef65e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be8073b10 con 0x7f7be8100ea0 2024-04-22T15:58:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.257+0000 7f7bef65e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be8073c50 con 0x7f7be8101850 2024-04-22T15:58:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.257+0000 7f7bed3fa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7be8100ea0 0x7f7be8073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.257+0000 7f7bed3fa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7be8100ea0 0x7f7be8073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59178/0 (socket says 172.21.15.16:59178) 2024-04-22T15:58:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.257+0000 7f7bed3fa700 1 -- 172.21.15.16:0/365390956 learned_addr learned my addr 172.21.15.16:0/365390956 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:58:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.257+0000 7f7becbf9700 1 --2- 172.21.15.16:0/365390956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7be8101850 0x7f7be80735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.257+0000 7f7bed3fa700 1 -- 172.21.15.16:0/365390956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7be8101850 msgr2=0x7f7be80735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.257+0000 7f7bed3fa700 1 --2- 172.21.15.16:0/365390956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7be8101850 0x7f7be80735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.257+0000 7f7bed3fa700 1 -- 172.21.15.16:0/365390956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be40095f0 con 0x7f7be8100ea0 2024-04-22T15:58:11.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.258+0000 7f7bed3fa700 1 --2- 172.21.15.16:0/365390956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7be8100ea0 0x7f7be8073090 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7be40092e0 tx=0x7f7be400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:11.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.258+0000 7f7bde7fc700 1 -- 172.21.15.16:0/365390956 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7be401d070 con 0x7f7be8100ea0 2024-04-22T15:58:11.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.258+0000 7f7bde7fc700 1 -- 172.21.15.16:0/365390956 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7be4005e00 con 0x7f7be8100ea0 2024-04-22T15:58:11.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.258+0000 7f7bde7fc700 1 -- 172.21.15.16:0/365390956 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7be4017400 con 0x7f7be8100ea0 2024-04-22T15:58:11.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.258+0000 7f7bef65e700 1 -- 172.21.15.16:0/365390956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7be8073ed0 con 0x7f7be8100ea0 2024-04-22T15:58:11.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.258+0000 7f7bef65e700 1 -- 172.21.15.16:0/365390956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7be81a2e40 con 0x7f7be8100ea0 2024-04-22T15:58:11.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.259+0000 7f7bef65e700 1 -- 172.21.15.16:0/365390956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7be8109240 con 0x7f7be8100ea0 2024-04-22T15:58:11.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.259+0000 7f7bde7fc700 1 -- 172.21.15.16:0/365390956 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7be400f6f0 con 0x7f7be8100ea0 2024-04-22T15:58:11.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.260+0000 7f7bde7fc700 1 --2- 172.21.15.16:0/365390956 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f7bd406ecc0 0x7f7bd4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:11.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.260+0000 7f7bde7fc700 1 -- 172.21.15.16:0/365390956 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f7be4090130 con 0x7f7be8100ea0 2024-04-22T15:58:11.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.260+0000 7f7becbf9700 1 --2- 172.21.15.16:0/365390956 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f7bd406ecc0 0x7f7bd4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:11.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.260+0000 7f7becbf9700 1 --2- 172.21.15.16:0/365390956 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f7bd406ecc0 0x7f7bd4071180 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7be8074880 tx=0x7f7bd800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:11.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.265+0000 7f7bde7fc700 1 -- 172.21.15.16:0/365390956 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7be4059010 con 0x7f7be8100ea0 2024-04-22T15:58:11.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:11 smithi016 ceph-mon[110740]: osdmap e81: 6 total, 6 up, 6 in 2024-04-22T15:58:11.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:11 smithi016 ceph-mon[110740]: fsmap cephfs:1 {0=cephfs.smithi027.iobjds=up:active} 2 up:standby 2024-04-22T15:58:11.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:11 smithi016 ceph-mon[110740]: pgmap v175: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 168 B/s rd, 0 op/s 2024-04-22T15:58:11.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:11 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/1087330398' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:11.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:11 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/2568286277' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:58:11.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:11 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.fkkejm"}]: dispatch 2024-04-22T15:58:11.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.562+0000 7f7bef65e700 1 -- 172.21.15.16:0/365390956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7be80746b0 con 0x7f7be8100ea0 2024-04-22T15:58:11.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.563+0000 7f7bde7fc700 1 -- 172.21.15.16:0/365390956 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f7be405c6d0 con 0x7f7be8100ea0 2024-04-22T15:58:11.566 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:58:11.566 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:58:11.566 INFO:teuthology.orchestra.run.smithi016.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:58:11.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.565+0000 7f7bef65e700 1 -- 172.21.15.16:0/365390956 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f7bd406ecc0 msgr2=0x7f7bd4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:11.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.565+0000 7f7bef65e700 1 --2- 172.21.15.16:0/365390956 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f7bd406ecc0 0x7f7bd4071180 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7be8074880 tx=0x7f7bd800b410 comp rx=0 tx=0).stop 2024-04-22T15:58:11.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.565+0000 7f7bef65e700 1 -- 172.21.15.16:0/365390956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7be8100ea0 msgr2=0x7f7be8073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:11.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.565+0000 7f7bef65e700 1 --2- 172.21.15.16:0/365390956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7be8100ea0 0x7f7be8073090 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7be40092e0 tx=0x7f7be400bd70 comp rx=0 tx=0).stop 2024-04-22T15:58:11.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.565+0000 7f7bef65e700 1 -- 172.21.15.16:0/365390956 shutdown_connections 2024-04-22T15:58:11.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.565+0000 7f7bef65e700 1 --2- 172.21.15.16:0/365390956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7be8100ea0 0x7f7be8073090 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:11.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.565+0000 7f7bef65e700 1 --2- 172.21.15.16:0/365390956 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f7bd406ecc0 0x7f7bd4071180 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:11.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.565+0000 7f7bef65e700 1 --2- 172.21.15.16:0/365390956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7be8101850 0x7f7be80735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:11.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.565+0000 7f7bef65e700 1 -- 172.21.15.16:0/365390956 >> 172.21.15.16:0/365390956 conn(0x7f7be80fc750 msgr2=0x7f7be80fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:11.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.565+0000 7f7bef65e700 1 -- 172.21.15.16:0/365390956 shutdown_connections 2024-04-22T15:58:11.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:11.565+0000 7f7bef65e700 1 -- 172.21.15.16:0/365390956 wait complete. 2024-04-22T15:58:11.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:11 smithi027 ceph-mon[83054]: osdmap e81: 6 total, 6 up, 6 in 2024-04-22T15:58:11.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:11 smithi027 ceph-mon[83054]: fsmap cephfs:1 {0=cephfs.smithi027.iobjds=up:active} 2 up:standby 2024-04-22T15:58:11.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:11 smithi027 ceph-mon[83054]: pgmap v175: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 168 B/s rd, 0 op/s 2024-04-22T15:58:11.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:11 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/1087330398' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:11.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:11 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/2568286277' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:58:11.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:11 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.fkkejm"}]: dispatch 2024-04-22T15:58:12.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:12 smithi027 ceph-mon[83054]: from='client.34400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:12.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:12 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/365390956' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:58:12.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:12 smithi016 ceph-mon[110740]: from='client.34400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:12.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:12 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/365390956' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:58:13.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:13 smithi016 ceph-mon[110740]: pgmap v176: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 127 B/s rd, 0 op/s 2024-04-22T15:58:13.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:13 smithi027 ceph-mon[83054]: pgmap v176: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 127 B/s rd, 0 op/s 2024-04-22T15:58:15.064 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:15 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:15.064 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:15 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:15.065 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:15 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:58:15.065 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:15 smithi016 ceph-mon[110740]: mds.? [v2:172.21.15.16:6826/2871308685,v1:172.21.15.16:6827/2871308685] up:boot 2024-04-22T15:58:15.065 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:15 smithi016 ceph-mon[110740]: fsmap cephfs:1 {0=cephfs.smithi027.iobjds=up:active} 3 up:standby 2024-04-22T15:58:15.065 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:15 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.fkkejm"}]: dispatch 2024-04-22T15:58:15.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:15 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:15.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:15 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:15.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:15 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:58:15.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:15 smithi027 ceph-mon[83054]: mds.? [v2:172.21.15.16:6826/2871308685,v1:172.21.15.16:6827/2871308685] up:boot 2024-04-22T15:58:15.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:15 smithi027 ceph-mon[83054]: fsmap cephfs:1 {0=cephfs.smithi027.iobjds=up:active} 3 up:standby 2024-04-22T15:58:15.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:15 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.fkkejm"}]: dispatch 2024-04-22T15:58:16.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:16 smithi016 ceph-mon[110740]: pgmap v177: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 111 B/s rd, 0 op/s 2024-04-22T15:58:16.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:16 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:16.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:16 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:58:16.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:16 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:16.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:16 smithi027 ceph-mon[83054]: pgmap v177: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 111 B/s rd, 0 op/s 2024-04-22T15:58:16.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:16 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:16.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:16 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:58:16.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:16 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:18.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:18 smithi027 ceph-mon[83054]: pgmap v178: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2024-04-22T15:58:18.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:18 smithi016 ceph-mon[110740]: pgmap v178: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2024-04-22T15:58:20.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:20 smithi027 ceph-mon[83054]: pgmap v179: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2024-04-22T15:58:20.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:20 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:20.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:20 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:20.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:20 smithi016 ceph-mon[110740]: pgmap v179: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2024-04-22T15:58:20.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:20 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:20.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:20 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:21.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:21 smithi016 ceph-mon[110740]: pgmap v180: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:21.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:21 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:21.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:21 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:21.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:21 smithi027 ceph-mon[83054]: pgmap v180: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:21.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:21 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:21.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:21 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:23.549 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:23 smithi016 ceph-mon[110740]: pgmap v181: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:23.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:23 smithi027 ceph-mon[83054]: pgmap v181: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:24.933 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:24 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:24.933 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:24 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:24.933 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:24 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:58:24.933 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:24 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:58:24.933 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:24 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:24.934 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:24 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:58:24.934 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:24 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:24.934 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:24 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:24.934 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:24 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:24.934 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:24 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:25.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:24 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:25.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:24 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:25.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:24 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:58:25.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:24 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:58:25.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:24 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:25.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:24 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:58:25.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:24 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:25.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:24 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:25.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:24 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:25.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:24 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:26.196 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:25 smithi016 ceph-mon[110740]: pgmap v182: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:26.197 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:25 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:26.197 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:25 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi016.pgvphu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:58:26.197 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:25 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:58:26.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:25 smithi027 ceph-mon[83054]: pgmap v182: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:26.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:25 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:26.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:25 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi016.pgvphu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:58:26.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:25 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:58:27.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:26 smithi016 ceph-mon[110740]: Upgrade: Updating mds.cephfs.smithi016.pgvphu 2024-04-22T15:58:27.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:26 smithi016 ceph-mon[110740]: Deploying daemon mds.cephfs.smithi016.pgvphu on smithi016 2024-04-22T15:58:27.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:26 smithi027 ceph-mon[83054]: Upgrade: Updating mds.cephfs.smithi016.pgvphu 2024-04-22T15:58:27.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:26 smithi027 ceph-mon[83054]: Deploying daemon mds.cephfs.smithi016.pgvphu on smithi016 2024-04-22T15:58:28.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:27 smithi016 ceph-mon[110740]: pgmap v183: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:28.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:27 smithi027 ceph-mon[83054]: pgmap v183: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:29.280 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:28 smithi016 ceph-mon[110740]: osdmap e82: 6 total, 6 up, 6 in 2024-04-22T15:58:29.280 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:28 smithi016 ceph-mon[110740]: fsmap cephfs:1 {0=cephfs.smithi027.iobjds=up:active} 2 up:standby 2024-04-22T15:58:29.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:28 smithi027 ceph-mon[83054]: osdmap e82: 6 total, 6 up, 6 in 2024-04-22T15:58:29.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:28 smithi027 ceph-mon[83054]: fsmap cephfs:1 {0=cephfs.smithi027.iobjds=up:active} 2 up:standby 2024-04-22T15:58:30.298 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:29 smithi016 ceph-mon[110740]: pgmap v185: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:30.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:29 smithi027 ceph-mon[83054]: pgmap v185: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:31.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:58:31.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:31.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:31.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:58:31.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:30 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:58:31.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:30 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:31.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:30 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:31.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:30 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:58:32.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:31 smithi016 ceph-mon[110740]: pgmap v186: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:32.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:31 smithi016 ceph-mon[110740]: mds.? [v2:172.21.15.16:6828/4228489800,v1:172.21.15.16:6829/4228489800] up:boot 2024-04-22T15:58:32.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:31 smithi016 ceph-mon[110740]: fsmap cephfs:1 {0=cephfs.smithi027.iobjds=up:active} 3 up:standby 2024-04-22T15:58:32.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:31 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.pgvphu"}]: dispatch 2024-04-22T15:58:32.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:31 smithi027 ceph-mon[83054]: pgmap v186: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:32.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:31 smithi027 ceph-mon[83054]: mds.? [v2:172.21.15.16:6828/4228489800,v1:172.21.15.16:6829/4228489800] up:boot 2024-04-22T15:58:32.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:31 smithi027 ceph-mon[83054]: fsmap cephfs:1 {0=cephfs.smithi027.iobjds=up:active} 3 up:standby 2024-04-22T15:58:32.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:31 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi016.pgvphu"}]: dispatch 2024-04-22T15:58:34.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:33 smithi027 ceph-mon[83054]: pgmap v187: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:34.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:33 smithi016 ceph-mon[110740]: pgmap v187: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:36.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:35 smithi027 ceph-mon[83054]: pgmap v188: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:36.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:35 smithi016 ceph-mon[110740]: pgmap v188: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:37.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:37 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:37.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:37 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:37.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:37 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:37.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:37 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:38.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:38 smithi027 ceph-mon[83054]: pgmap v189: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:38.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:38 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:38.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:38 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:38.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:38 smithi016 ceph-mon[110740]: pgmap v189: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:38.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:38 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:38.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:38 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:40.306 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:40 smithi016 ceph-mon[110740]: pgmap v190: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:40.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:40 smithi027 ceph-mon[83054]: pgmap v190: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:41.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.690+0000 7fdf9b160700 1 -- 172.21.15.16:0/1147062406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf94105bf0 msgr2=0x7fdf94105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:41.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.690+0000 7fdf9b160700 1 --2- 172.21.15.16:0/1147062406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf94105bf0 0x7fdf94105fd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdf88009910 tx=0x7fdf88009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:41.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.690+0000 7fdf9b160700 1 -- 172.21.15.16:0/1147062406 shutdown_connections 2024-04-22T15:58:41.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.690+0000 7fdf9b160700 1 --2- 172.21.15.16:0/1147062406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf940ffb40 0x7fdf940fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:41.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.690+0000 7fdf9b160700 1 --2- 172.21.15.16:0/1147062406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf94105bf0 0x7fdf94105fd0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:41.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.690+0000 7fdf9b160700 1 -- 172.21.15.16:0/1147062406 >> 172.21.15.16:0/1147062406 conn(0x7fdf940fb610 msgr2=0x7fdf940fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:41.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.690+0000 7fdf9b160700 1 -- 172.21.15.16:0/1147062406 shutdown_connections 2024-04-22T15:58:41.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.690+0000 7fdf9b160700 1 -- 172.21.15.16:0/1147062406 wait complete. 2024-04-22T15:58:41.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.691+0000 7fdf9b160700 1 Processor -- start 2024-04-22T15:58:41.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.691+0000 7fdf9b160700 1 -- start start 2024-04-22T15:58:41.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.691+0000 7fdf9b160700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf940ffb40 0x7fdf94195550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:41.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.691+0000 7fdf9b160700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf94105bf0 0x7fdf94195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:41.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.691+0000 7fdf9b160700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf94195fd0 con 0x7fdf940ffb40 2024-04-22T15:58:41.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.691+0000 7fdf9b160700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf94196140 con 0x7fdf94105bf0 2024-04-22T15:58:41.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.691+0000 7fdf98efc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf940ffb40 0x7fdf94195550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:41.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.691+0000 7fdf98efc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf940ffb40 0x7fdf94195550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33272/0 (socket says 172.21.15.16:33272) 2024-04-22T15:58:41.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.691+0000 7fdf98efc700 1 -- 172.21.15.16:0/835523412 learned_addr learned my addr 172.21.15.16:0/835523412 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:58:41.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.691+0000 7fdf93fff700 1 --2- 172.21.15.16:0/835523412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf94105bf0 0x7fdf94195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:41.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.691+0000 7fdf98efc700 1 -- 172.21.15.16:0/835523412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf94105bf0 msgr2=0x7fdf94195a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:41.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.691+0000 7fdf98efc700 1 --2- 172.21.15.16:0/835523412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf94105bf0 0x7fdf94195a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:41.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.691+0000 7fdf98efc700 1 -- 172.21.15.16:0/835523412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf880095f0 con 0x7fdf940ffb40 2024-04-22T15:58:41.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.692+0000 7fdf98efc700 1 --2- 172.21.15.16:0/835523412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf940ffb40 0x7fdf94195550 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdf88006010 tx=0x7fdf8800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:41.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.692+0000 7fdf91ffb700 1 -- 172.21.15.16:0/835523412 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf8801d070 con 0x7fdf940ffb40 2024-04-22T15:58:41.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.692+0000 7fdf9b160700 1 -- 172.21.15.16:0/835523412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf941963c0 con 0x7fdf940ffb40 2024-04-22T15:58:41.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.692+0000 7fdf91ffb700 1 -- 172.21.15.16:0/835523412 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf8800f460 con 0x7fdf940ffb40 2024-04-22T15:58:41.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.692+0000 7fdf91ffb700 1 -- 172.21.15.16:0/835523412 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf88017610 con 0x7fdf940ffb40 2024-04-22T15:58:41.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.692+0000 7fdf9b160700 1 -- 172.21.15.16:0/835523412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf941a2e40 con 0x7fdf940ffb40 2024-04-22T15:58:41.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.693+0000 7fdf9b160700 1 -- 172.21.15.16:0/835523412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf94108020 con 0x7fdf940ffb40 2024-04-22T15:58:41.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.694+0000 7fdf91ffb700 1 -- 172.21.15.16:0/835523412 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdf8800f5d0 con 0x7fdf940ffb40 2024-04-22T15:58:41.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.694+0000 7fdf91ffb700 1 --2- 172.21.15.16:0/835523412 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdf8406ecc0 0x7fdf84071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:41.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.694+0000 7fdf91ffb700 1 -- 172.21.15.16:0/835523412 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fdf880900c0 con 0x7fdf940ffb40 2024-04-22T15:58:41.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.695+0000 7fdf93fff700 1 --2- 172.21.15.16:0/835523412 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdf8406ecc0 0x7fdf84071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:41.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.695+0000 7fdf93fff700 1 --2- 172.21.15.16:0/835523412 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdf8406ecc0 0x7fdf84071180 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdf94069770 tx=0x7fdf80009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:41.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.699+0000 7fdf91ffb700 1 -- 172.21.15.16:0/835523412 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fdf8805cad0 con 0x7fdf940ffb40 2024-04-22T15:58:41.817 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:41 smithi027 ceph-mon[83054]: pgmap v191: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:41.817 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:41 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:41.817 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:41 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:41.817 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:41 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:58:41.818 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:41 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:58:41.818 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:41 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:41.818 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:41 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:58:41.818 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:41 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:41.818 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:41 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:41.818 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:41 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:41.818 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:41 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:41.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.930+0000 7fdf9b160700 1 -- 172.21.15.16:0/835523412 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf941963c0 con 0x7fdf8406ecc0 2024-04-22T15:58:41.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.932+0000 7fdf91ffb700 1 -- 172.21.15.16:0/835523412 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fdf941963c0 con 0x7fdf8406ecc0 2024-04-22T15:58:41.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.934+0000 7fdf9b160700 1 -- 172.21.15.16:0/835523412 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdf8406ecc0 msgr2=0x7fdf84071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:41.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.934+0000 7fdf9b160700 1 --2- 172.21.15.16:0/835523412 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdf8406ecc0 0x7fdf84071180 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdf94069770 tx=0x7fdf80009190 comp rx=0 tx=0).stop 2024-04-22T15:58:41.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.934+0000 7fdf9b160700 1 -- 172.21.15.16:0/835523412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf940ffb40 msgr2=0x7fdf94195550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:41.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.934+0000 7fdf9b160700 1 --2- 172.21.15.16:0/835523412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf940ffb40 0x7fdf94195550 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdf88006010 tx=0x7fdf8800bd70 comp rx=0 tx=0).stop 2024-04-22T15:58:41.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.934+0000 7fdf9b160700 1 -- 172.21.15.16:0/835523412 shutdown_connections 2024-04-22T15:58:41.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.934+0000 7fdf9b160700 1 --2- 172.21.15.16:0/835523412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf940ffb40 0x7fdf94195550 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:41.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.934+0000 7fdf9b160700 1 --2- 172.21.15.16:0/835523412 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdf8406ecc0 0x7fdf84071180 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:41.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.934+0000 7fdf9b160700 1 --2- 172.21.15.16:0/835523412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf94105bf0 0x7fdf94195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:41.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.934+0000 7fdf9b160700 1 -- 172.21.15.16:0/835523412 >> 172.21.15.16:0/835523412 conn(0x7fdf940fb610 msgr2=0x7fdf940fd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:41.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.934+0000 7fdf9b160700 1 -- 172.21.15.16:0/835523412 shutdown_connections 2024-04-22T15:58:41.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:41.934+0000 7fdf9b160700 1 -- 172.21.15.16:0/835523412 wait complete. 2024-04-22T15:58:41.948 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:58:41.948 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:41 smithi016 ceph-mon[110740]: pgmap v191: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:41.948 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:41 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:41.948 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:41 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:41.948 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:41 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:58:41.948 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:41 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:58:41.948 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:41 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:41.949 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:41 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:58:41.949 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:41 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:41.949 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:41 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:41.949 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:41 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:41.949 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:41 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.042+0000 7f284b724700 1 -- 172.21.15.16:0/4195782771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2844100ea0 msgr2=0x7f2844101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.042+0000 7f284b724700 1 --2- 172.21.15.16:0/4195782771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2844100ea0 0x7f2844101280 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2840009910 tx=0x7f2840009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.042+0000 7f284b724700 1 -- 172.21.15.16:0/4195782771 shutdown_connections 2024-04-22T15:58:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.042+0000 7f284b724700 1 --2- 172.21.15.16:0/4195782771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2844101850 0x7f28441058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.042+0000 7f284b724700 1 --2- 172.21.15.16:0/4195782771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2844100ea0 0x7f2844101280 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.042+0000 7f284b724700 1 -- 172.21.15.16:0/4195782771 >> 172.21.15.16:0/4195782771 conn(0x7f28440fc750 msgr2=0x7f28440feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.042+0000 7f284b724700 1 -- 172.21.15.16:0/4195782771 shutdown_connections 2024-04-22T15:58:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.043+0000 7f284b724700 1 -- 172.21.15.16:0/4195782771 wait complete. 2024-04-22T15:58:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.043+0000 7f284b724700 1 Processor -- start 2024-04-22T15:58:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.043+0000 7f284b724700 1 -- start start 2024-04-22T15:58:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.043+0000 7f284b724700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2844100ea0 0x7f284419e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.043+0000 7f284b724700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2844101850 0x7f284419ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.043+0000 7f284b724700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284419f4b0 con 0x7f2844101850 2024-04-22T15:58:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.043+0000 7f284b724700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2844198960 con 0x7f2844100ea0 2024-04-22T15:58:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.044+0000 7f2848cbf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2844101850 0x7f284419ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.044+0000 7f2848cbf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2844101850 0x7f284419ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33278/0 (socket says 172.21.15.16:33278) 2024-04-22T15:58:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.044+0000 7f2848cbf700 1 -- 172.21.15.16:0/3166594457 learned_addr learned my addr 172.21.15.16:0/3166594457 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:58:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.044+0000 7f28494c0700 1 --2- 172.21.15.16:0/3166594457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2844100ea0 0x7f284419e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.044+0000 7f2848cbf700 1 -- 172.21.15.16:0/3166594457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2844100ea0 msgr2=0x7f284419e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.044+0000 7f2848cbf700 1 --2- 172.21.15.16:0/3166594457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2844100ea0 0x7f284419e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.044+0000 7f2848cbf700 1 -- 172.21.15.16:0/3166594457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28400095f0 con 0x7f2844101850 2024-04-22T15:58:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.044+0000 7f2848cbf700 1 --2- 172.21.15.16:0/3166594457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2844101850 0x7f284419ee20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f283400d900 tx=0x7f283400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.044+0000 7f283a7fc700 1 -- 172.21.15.16:0/3166594457 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2834009900 con 0x7f2844101850 2024-04-22T15:58:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.044+0000 7f283a7fc700 1 -- 172.21.15.16:0/3166594457 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2834010460 con 0x7f2844101850 2024-04-22T15:58:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.044+0000 7f284b724700 1 -- 172.21.15.16:0/3166594457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2844198c40 con 0x7f2844101850 2024-04-22T15:58:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.044+0000 7f283a7fc700 1 -- 172.21.15.16:0/3166594457 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f283400f5f0 con 0x7f2844101850 2024-04-22T15:58:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.044+0000 7f284b724700 1 -- 172.21.15.16:0/3166594457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2844199190 con 0x7f2844101850 2024-04-22T15:58:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.045+0000 7f284b724700 1 -- 172.21.15.16:0/3166594457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2844105020 con 0x7f2844101850 2024-04-22T15:58:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.046+0000 7f283a7fc700 1 -- 172.21.15.16:0/3166594457 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f28340105d0 con 0x7f2844101850 2024-04-22T15:58:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.046+0000 7f283a7fc700 1 --2- 172.21.15.16:0/3166594457 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f283006ecc0 0x7f2830071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.046+0000 7f283a7fc700 1 -- 172.21.15.16:0/3166594457 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f283408e950 con 0x7f2844101850 2024-04-22T15:58:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.046+0000 7f28494c0700 1 --2- 172.21.15.16:0/3166594457 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f283006ecc0 0x7f2830071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.047+0000 7f28494c0700 1 --2- 172.21.15.16:0/3166594457 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f283006ecc0 0x7f2830071180 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2840006010 tx=0x7f284001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.051+0000 7f283a7fc700 1 -- 172.21.15.16:0/3166594457 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f283405b2e0 con 0x7f2844101850 2024-04-22T15:58:42.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.280+0000 7f284b724700 1 -- 172.21.15.16:0/3166594457 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2844199e70 con 0x7f283006ecc0 2024-04-22T15:58:42.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.282+0000 7f283a7fc700 1 -- 172.21.15.16:0/3166594457 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2844199e70 con 0x7f283006ecc0 2024-04-22T15:58:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.284+0000 7f284b724700 1 -- 172.21.15.16:0/3166594457 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f283006ecc0 msgr2=0x7f2830071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.284+0000 7f284b724700 1 --2- 172.21.15.16:0/3166594457 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f283006ecc0 0x7f2830071180 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2840006010 tx=0x7f284001a040 comp rx=0 tx=0).stop 2024-04-22T15:58:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.284+0000 7f284b724700 1 -- 172.21.15.16:0/3166594457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2844101850 msgr2=0x7f284419ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.284+0000 7f284b724700 1 --2- 172.21.15.16:0/3166594457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2844101850 0x7f284419ee20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f283400d900 tx=0x7f283400dc10 comp rx=0 tx=0).stop 2024-04-22T15:58:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.284+0000 7f284b724700 1 -- 172.21.15.16:0/3166594457 shutdown_connections 2024-04-22T15:58:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.284+0000 7f284b724700 1 --2- 172.21.15.16:0/3166594457 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f283006ecc0 0x7f2830071180 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.284+0000 7f284b724700 1 --2- 172.21.15.16:0/3166594457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2844100ea0 0x7f284419e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.284+0000 7f284b724700 1 --2- 172.21.15.16:0/3166594457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2844101850 0x7f284419ee20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.284+0000 7f284b724700 1 -- 172.21.15.16:0/3166594457 >> 172.21.15.16:0/3166594457 conn(0x7f28440fc750 msgr2=0x7f28440fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.284+0000 7f284b724700 1 -- 172.21.15.16:0/3166594457 shutdown_connections 2024-04-22T15:58:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.284+0000 7f284b724700 1 -- 172.21.15.16:0/3166594457 wait complete. 2024-04-22T15:58:42.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.393+0000 7fbc15c45700 1 -- 172.21.15.16:0/2935432639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc10101e00 msgr2=0x7fbc101021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:42.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.393+0000 7fbc15c45700 1 --2- 172.21.15.16:0/2935432639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc10101e00 0x7fbc101021e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbc00009910 tx=0x7fbc00009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:42.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.393+0000 7fbc15c45700 1 -- 172.21.15.16:0/2935432639 shutdown_connections 2024-04-22T15:58:42.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.393+0000 7fbc15c45700 1 --2- 172.21.15.16:0/2935432639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc10102720 0x7fbc1010ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:42.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.393+0000 7fbc15c45700 1 --2- 172.21.15.16:0/2935432639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc10101e00 0x7fbc101021e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:42.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.393+0000 7fbc15c45700 1 -- 172.21.15.16:0/2935432639 >> 172.21.15.16:0/2935432639 conn(0x7fbc100fb650 msgr2=0x7fbc100fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:42.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.393+0000 7fbc15c45700 1 -- 172.21.15.16:0/2935432639 shutdown_connections 2024-04-22T15:58:42.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.393+0000 7fbc15c45700 1 -- 172.21.15.16:0/2935432639 wait complete. 2024-04-22T15:58:42.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.394+0000 7fbc15c45700 1 Processor -- start 2024-04-22T15:58:42.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.394+0000 7fbc15c45700 1 -- start start 2024-04-22T15:58:42.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.394+0000 7fbc15c45700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc10101e00 0x7fbc1019c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:42.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.394+0000 7fbc15c45700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc10102720 0x7fbc1019cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:42.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.394+0000 7fbc15c45700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1019d260 con 0x7fbc10102720 2024-04-22T15:58:42.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.394+0000 7fbc15c45700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc10196710 con 0x7fbc10101e00 2024-04-22T15:58:42.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.394+0000 7fbc0effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc10102720 0x7fbc1019cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:42.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.394+0000 7fbc0effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc10102720 0x7fbc1019cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33294/0 (socket says 172.21.15.16:33294) 2024-04-22T15:58:42.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.394+0000 7fbc0effd700 1 -- 172.21.15.16:0/1469253353 learned_addr learned my addr 172.21.15.16:0/1469253353 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:58:42.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.394+0000 7fbc0f7fe700 1 --2- 172.21.15.16:0/1469253353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc10101e00 0x7fbc1019c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:42.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.394+0000 7fbc0effd700 1 -- 172.21.15.16:0/1469253353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc10101e00 msgr2=0x7fbc1019c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:42.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.394+0000 7fbc0effd700 1 --2- 172.21.15.16:0/1469253353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc10101e00 0x7fbc1019c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:42.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.394+0000 7fbc0effd700 1 -- 172.21.15.16:0/1469253353 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc000095f0 con 0x7fbc10102720 2024-04-22T15:58:42.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.394+0000 7fbc0f7fe700 1 --2- 172.21.15.16:0/1469253353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc10101e00 0x7fbc1019c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:58:42.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.395+0000 7fbc0effd700 1 --2- 172.21.15.16:0/1469253353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc10102720 0x7fbc1019cbd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbc0400d8d0 tx=0x7fbc0400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:42.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.395+0000 7fbc0cff9700 1 -- 172.21.15.16:0/1469253353 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc040041f0 con 0x7fbc10102720 2024-04-22T15:58:42.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.395+0000 7fbc0cff9700 1 -- 172.21.15.16:0/1469253353 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbc04004350 con 0x7fbc10102720 2024-04-22T15:58:42.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.395+0000 7fbc15c45700 1 -- 172.21.15.16:0/1469253353 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc101969f0 con 0x7fbc10102720 2024-04-22T15:58:42.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.395+0000 7fbc15c45700 1 -- 172.21.15.16:0/1469253353 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc10196f40 con 0x7fbc10102720 2024-04-22T15:58:42.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.395+0000 7fbc0cff9700 1 -- 172.21.15.16:0/1469253353 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc04003d60 con 0x7fbc10102720 2024-04-22T15:58:42.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.396+0000 7fbc15c45700 1 -- 172.21.15.16:0/1469253353 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc10108310 con 0x7fbc10102720 2024-04-22T15:58:42.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.396+0000 7fbc0cff9700 1 -- 172.21.15.16:0/1469253353 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbc04009730 con 0x7fbc10102720 2024-04-22T15:58:42.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.397+0000 7fbc0cff9700 1 --2- 172.21.15.16:0/1469253353 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fbbf806ed10 0x7fbbf80711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:42.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.397+0000 7fbc0f7fe700 1 --2- 172.21.15.16:0/1469253353 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fbbf806ed10 0x7fbbf80711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:42.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.397+0000 7fbc0cff9700 1 -- 172.21.15.16:0/1469253353 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fbc0408e9d0 con 0x7fbc10102720 2024-04-22T15:58:42.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.398+0000 7fbc0f7fe700 1 --2- 172.21.15.16:0/1469253353 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fbbf806ed10 0x7fbbf80711d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbc00005b40 tx=0x7fbc0001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:42.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.402+0000 7fbc0cff9700 1 -- 172.21.15.16:0/1469253353 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbc0405b340 con 0x7fbc10102720 2024-04-22T15:58:42.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.620+0000 7fbc15c45700 1 -- 172.21.15.16:0/1469253353 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc101971b0 con 0x7fbbf806ed10 2024-04-22T15:58:42.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.631+0000 7fbc0cff9700 1 -- 172.21.15.16:0/1469253353 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fbc101971b0 con 0x7fbbf806ed10 2024-04-22T15:58:42.634 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:58:42.634 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.smithi016 smithi016 *:9093,9094 running (6m) 6s ago 17m 25.8M - 0.25.0 c8568f914cd2 ea0e0217d6f9 2024-04-22T15:58:42.634 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi016 smithi016 running (17m) 6s ago 17m 8980k - 18.2.1 5be31c24972a f8b627d3f41e 2024-04-22T15:58:42.634 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi027 smithi027 running (16m) 44s ago 16m 8690k - 18.2.1 5be31c24972a f00fd40b4220 2024-04-22T15:58:42.634 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi016 smithi016 running (4m) 6s ago 17m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 1feb008449ed 2024-04-22T15:58:42.634 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi027 smithi027 running (4m) 44s ago 16m 6656k - 19.0.0-3223-g5883d494 3d983c26e278 c7f862c6da00 2024-04-22T15:58:42.634 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.smithi016 smithi016 *:3000 running (6m) 6s ago 17m 81.9M - 9.4.12 2bacad6d85d8 388a3c9c1b3f 2024-04-22T15:58:42.635 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.fkkejm smithi016 running (29s) 6s ago 12m 16.0M - 19.0.0-3223-g5883d494 3d983c26e278 e23d9fc30059 2024-04-22T15:58:42.635 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.pgvphu smithi016 running (12s) 6s ago 12m 11.6M - 19.0.0-3223-g5883d494 3d983c26e278 4777936b5956 2024-04-22T15:58:42.635 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.dgqyes smithi027 running (12m) 44s ago 12m 20.6M - 18.2.1 5be31c24972a 41a9220430fd 2024-04-22T15:58:42.635 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.iobjds smithi027 running (12m) 44s ago 12m 149M - 18.2.1 5be31c24972a c1c8d6770f02 2024-04-22T15:58:42.635 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi016.rnsnnr smithi016 *:8443,9283,8765 running (9m) 6s ago 19m 510M - 19.0.0-3223-g5883d494 3d983c26e278 09b4fe285bc4 2024-04-22T15:58:42.635 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi027.ukdgzs smithi027 *:8443,9283,8765 running (8m) 44s ago 16m 451M - 19.0.0-3223-g5883d494 3d983c26e278 7548da9b5eb9 2024-04-22T15:58:42.635 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi016 smithi016 running (5m) 6s ago 19m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 4ae529a67806 2024-04-22T15:58:42.635 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi027 smithi027 running (4m) 44s ago 16m 41.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 f29fcb888801 2024-04-22T15:58:42.635 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi016 smithi016 *:9100 running (7m) 6s ago 17m 15.5M - 1.5.0 0da6a335fe13 35e46e4d2f3f 2024-04-22T15:58:42.635 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi027 smithi027 *:9100 running (7m) 44s ago 16m 15.0M - 1.5.0 0da6a335fe13 e682a7672e9b 2024-04-22T15:58:42.635 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (3m) 6s ago 15m 92.0M 1217M 19.0.0-3223-g5883d494 3d983c26e278 79217eb383a2 2024-04-22T15:58:42.635 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (3m) 6s ago 15m 92.0M 1217M 19.0.0-3223-g5883d494 3d983c26e278 ec1a45b3c53e 2024-04-22T15:58:42.635 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (2m) 6s ago 14m 77.8M 1217M 19.0.0-3223-g5883d494 3d983c26e278 48bb459fd021 2024-04-22T15:58:42.635 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi027 running (115s) 44s ago 14m 97.2M 2240M 19.0.0-3223-g5883d494 3d983c26e278 81034bf22d3f 2024-04-22T15:58:42.635 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (81s) 44s ago 14m 94.9M 2240M 19.0.0-3223-g5883d494 3d983c26e278 93b660e5ecb6 2024-04-22T15:58:42.636 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi027 running (47s) 44s ago 13m 14.2M 2240M 19.0.0-3223-g5883d494 3d983c26e278 d857c152994a 2024-04-22T15:58:42.636 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.smithi016 smithi016 *:9095 running (7m) 6s ago 17m 76.0M - 2.43.0 a07b618ecd1d 373eaadee82d 2024-04-22T15:58:42.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.633+0000 7fbc15c45700 1 -- 172.21.15.16:0/1469253353 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fbbf806ed10 msgr2=0x7fbbf80711d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:42.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.633+0000 7fbc15c45700 1 --2- 172.21.15.16:0/1469253353 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fbbf806ed10 0x7fbbf80711d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbc00005b40 tx=0x7fbc0001a040 comp rx=0 tx=0).stop 2024-04-22T15:58:42.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.633+0000 7fbc15c45700 1 -- 172.21.15.16:0/1469253353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc10102720 msgr2=0x7fbc1019cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:42.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.633+0000 7fbc15c45700 1 --2- 172.21.15.16:0/1469253353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc10102720 0x7fbc1019cbd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbc0400d8d0 tx=0x7fbc0400dc90 comp rx=0 tx=0).stop 2024-04-22T15:58:42.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.633+0000 7fbc15c45700 1 -- 172.21.15.16:0/1469253353 shutdown_connections 2024-04-22T15:58:42.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.633+0000 7fbc15c45700 1 --2- 172.21.15.16:0/1469253353 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fbbf806ed10 0x7fbbf80711d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:42.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.633+0000 7fbc15c45700 1 --2- 172.21.15.16:0/1469253353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc10101e00 0x7fbc1019c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:42.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.633+0000 7fbc15c45700 1 --2- 172.21.15.16:0/1469253353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc10102720 0x7fbc1019cbd0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:42.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.633+0000 7fbc15c45700 1 -- 172.21.15.16:0/1469253353 >> 172.21.15.16:0/1469253353 conn(0x7fbc100fb650 msgr2=0x7fbc10105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:42.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.633+0000 7fbc15c45700 1 -- 172.21.15.16:0/1469253353 shutdown_connections 2024-04-22T15:58:42.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.633+0000 7fbc15c45700 1 -- 172.21.15.16:0/1469253353 wait complete. 2024-04-22T15:58:42.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.741+0000 7fed11645700 1 -- 172.21.15.16:0/913369912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed0c0ffda0 msgr2=0x7fed0c100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:42.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.741+0000 7fed11645700 1 --2- 172.21.15.16:0/913369912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed0c0ffda0 0x7fed0c100180 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009910 tx=0x7fecfc009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:42.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.741+0000 7fed11645700 1 -- 172.21.15.16:0/913369912 shutdown_connections 2024-04-22T15:58:42.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.741+0000 7fed11645700 1 --2- 172.21.15.16:0/913369912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed0c100750 0x7fed0c1047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:42.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.741+0000 7fed11645700 1 --2- 172.21.15.16:0/913369912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed0c0ffda0 0x7fed0c100180 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:42.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.741+0000 7fed11645700 1 -- 172.21.15.16:0/913369912 >> 172.21.15.16:0/913369912 conn(0x7fed0c0fb650 msgr2=0x7fed0c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:42.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.741+0000 7fed11645700 1 -- 172.21.15.16:0/913369912 shutdown_connections 2024-04-22T15:58:42.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.741+0000 7fed11645700 1 -- 172.21.15.16:0/913369912 wait complete. 2024-04-22T15:58:42.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.742+0000 7fed11645700 1 Processor -- start 2024-04-22T15:58:42.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.742+0000 7fed11645700 1 -- start start 2024-04-22T15:58:42.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.742+0000 7fed11645700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed0c0ffda0 0x7fed0c19c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:42.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.742+0000 7fed11645700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed0c100750 0x7fed0c19cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:42.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.742+0000 7fed11645700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c19d200 con 0x7fed0c100750 2024-04-22T15:58:42.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.742+0000 7fed11645700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c1966b0 con 0x7fed0c0ffda0 2024-04-22T15:58:42.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.742+0000 7fed0a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed0c100750 0x7fed0c19cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:42.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.742+0000 7fed0a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed0c100750 0x7fed0c19cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33316/0 (socket says 172.21.15.16:33316) 2024-04-22T15:58:42.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.742+0000 7fed0a7fc700 1 -- 172.21.15.16:0/2279628284 learned_addr learned my addr 172.21.15.16:0/2279628284 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:58:42.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.742+0000 7fed0affd700 1 --2- 172.21.15.16:0/2279628284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed0c0ffda0 0x7fed0c19c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:42.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.743+0000 7fed0a7fc700 1 -- 172.21.15.16:0/2279628284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed0c0ffda0 msgr2=0x7fed0c19c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:42.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.743+0000 7fed0a7fc700 1 --2- 172.21.15.16:0/2279628284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed0c0ffda0 0x7fed0c19c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:42.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.743+0000 7fed0a7fc700 1 -- 172.21.15.16:0/2279628284 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecfc0095f0 con 0x7fed0c100750 2024-04-22T15:58:42.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.743+0000 7fed0affd700 1 --2- 172.21.15.16:0/2279628284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed0c0ffda0 0x7fed0c19c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T15:58:42.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.743+0000 7fed0a7fc700 1 --2- 172.21.15.16:0/2279628284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed0c100750 0x7fed0c19cb70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fed0000e8f0 tx=0x7fed0000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:42.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.743+0000 7fecf3fff700 1 -- 172.21.15.16:0/2279628284 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed00009790 con 0x7fed0c100750 2024-04-22T15:58:42.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.743+0000 7fed11645700 1 -- 172.21.15.16:0/2279628284 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed0c196990 con 0x7fed0c100750 2024-04-22T15:58:42.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.743+0000 7fecf3fff700 1 -- 172.21.15.16:0/2279628284 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed0000c810 con 0x7fed0c100750 2024-04-22T15:58:42.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.743+0000 7fecf3fff700 1 -- 172.21.15.16:0/2279628284 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed00005580 con 0x7fed0c100750 2024-04-22T15:58:42.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.743+0000 7fed11645700 1 -- 172.21.15.16:0/2279628284 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0c196ee0 con 0x7fed0c100750 2024-04-22T15:58:42.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.744+0000 7fed11645700 1 -- 172.21.15.16:0/2279628284 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed0c04f2e0 con 0x7fed0c100750 2024-04-22T15:58:42.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.745+0000 7fecf3fff700 1 -- 172.21.15.16:0/2279628284 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fed0000c980 con 0x7fed0c100750 2024-04-22T15:58:42.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.745+0000 7fecf3fff700 1 --2- 172.21.15.16:0/2279628284 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fecf406ecc0 0x7fecf4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:42.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.745+0000 7fecf3fff700 1 -- 172.21.15.16:0/2279628284 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fed00014070 con 0x7fed0c100750 2024-04-22T15:58:42.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.745+0000 7fed0affd700 1 --2- 172.21.15.16:0/2279628284 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fecf406ecc0 0x7fecf4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:42.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.746+0000 7fed0affd700 1 --2- 172.21.15.16:0/2279628284 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fecf406ecc0 0x7fecf4071180 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fecfc00b760 tx=0x7fecfc005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:42.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:42.750+0000 7fecf3fff700 1 -- 172.21.15.16:0/2279628284 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fed0005ba00 con 0x7fed0c100750 2024-04-22T15:58:42.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:42 smithi016 ceph-mon[110740]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:42.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:42 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:42.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:42 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.iobjds", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:58:42.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:42 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:58:43.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.056+0000 7fed11645700 1 -- 172.21.15.16:0/2279628284 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fed0c197bc0 con 0x7fed0c100750 2024-04-22T15:58:43.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.057+0000 7fecf3fff700 1 -- 172.21.15.16:0/2279628284 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fed00062070 con 0x7fed0c100750 2024-04-22T15:58:43.060 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:58:43.060 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-22T15:58:43.060 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:58:43.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:58:43.060 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-22T15:58:43.060 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:58:43.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:58:43.060 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-22T15:58:43.060 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:58:43.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:58:43.060 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-22T15:58:43.060 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2, 2024-04-22T15:58:43.060 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:58:43.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:58:43.061 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-22T15:58:43.061 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2, 2024-04-22T15:58:43.061 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 12 2024-04-22T15:58:43.061 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-22T15:58:43.061 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:58:43.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.059+0000 7fed11645700 1 -- 172.21.15.16:0/2279628284 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fecf406ecc0 msgr2=0x7fecf4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:43.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.059+0000 7fed11645700 1 --2- 172.21.15.16:0/2279628284 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fecf406ecc0 0x7fecf4071180 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fecfc00b760 tx=0x7fecfc005940 comp rx=0 tx=0).stop 2024-04-22T15:58:43.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.059+0000 7fed11645700 1 -- 172.21.15.16:0/2279628284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed0c100750 msgr2=0x7fed0c19cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:43.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.059+0000 7fed11645700 1 --2- 172.21.15.16:0/2279628284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed0c100750 0x7fed0c19cb70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fed0000e8f0 tx=0x7fed0000ec00 comp rx=0 tx=0).stop 2024-04-22T15:58:43.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.060+0000 7fed11645700 1 -- 172.21.15.16:0/2279628284 shutdown_connections 2024-04-22T15:58:43.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.060+0000 7fed11645700 1 --2- 172.21.15.16:0/2279628284 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fecf406ecc0 0x7fecf4071180 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:43.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.060+0000 7fed11645700 1 --2- 172.21.15.16:0/2279628284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed0c0ffda0 0x7fed0c19c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:43.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.060+0000 7fed11645700 1 --2- 172.21.15.16:0/2279628284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed0c100750 0x7fed0c19cb70 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:43.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.060+0000 7fed11645700 1 -- 172.21.15.16:0/2279628284 >> 172.21.15.16:0/2279628284 conn(0x7fed0c0fb650 msgr2=0x7fed0c103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:43.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.060+0000 7fed11645700 1 -- 172.21.15.16:0/2279628284 shutdown_connections 2024-04-22T15:58:43.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.060+0000 7fed11645700 1 -- 172.21.15.16:0/2279628284 wait complete. 2024-04-22T15:58:43.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:42 smithi027 ceph-mon[83054]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:43.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:42 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:43.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:42 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.iobjds", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:58:43.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:42 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:58:43.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.166+0000 7f08dee2e700 1 -- 172.21.15.16:0/3726283829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08d8100ea0 msgr2=0x7f08d8101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:43.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.166+0000 7f08dee2e700 1 --2- 172.21.15.16:0/3726283829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08d8100ea0 0x7f08d8101280 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f08cc009910 tx=0x7f08cc009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:43.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.166+0000 7f08dee2e700 1 -- 172.21.15.16:0/3726283829 shutdown_connections 2024-04-22T15:58:43.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.166+0000 7f08dee2e700 1 --2- 172.21.15.16:0/3726283829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d8101850 0x7f08d81058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:43.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.166+0000 7f08dee2e700 1 --2- 172.21.15.16:0/3726283829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08d8100ea0 0x7f08d8101280 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:43.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.166+0000 7f08dee2e700 1 -- 172.21.15.16:0/3726283829 >> 172.21.15.16:0/3726283829 conn(0x7f08d80fc750 msgr2=0x7f08d80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:43.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.166+0000 7f08dee2e700 1 -- 172.21.15.16:0/3726283829 shutdown_connections 2024-04-22T15:58:43.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.166+0000 7f08dee2e700 1 -- 172.21.15.16:0/3726283829 wait complete. 2024-04-22T15:58:43.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.167+0000 7f08dee2e700 1 Processor -- start 2024-04-22T15:58:43.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.167+0000 7f08dee2e700 1 -- start start 2024-04-22T15:58:43.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.167+0000 7f08dee2e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08d8100ea0 0x7f08d8196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:43.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.167+0000 7f08dee2e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d8101850 0x7f08d8196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.167+0000 7f08dee2e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d8197560 con 0x7f08d8100ea0 2024-04-22T15:58:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.167+0000 7f08dee2e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d819b370 con 0x7f08d8101850 2024-04-22T15:58:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.167+0000 7f08dcbca700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08d8100ea0 0x7f08d8196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.167+0000 7f08dcbca700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08d8100ea0 0x7f08d8196990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33332/0 (socket says 172.21.15.16:33332) 2024-04-22T15:58:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.167+0000 7f08dcbca700 1 -- 172.21.15.16:0/2562735161 learned_addr learned my addr 172.21.15.16:0/2562735161 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:58:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.167+0000 7f08d7fff700 1 --2- 172.21.15.16:0/2562735161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d8101850 0x7f08d8196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.168+0000 7f08dcbca700 1 -- 172.21.15.16:0/2562735161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d8101850 msgr2=0x7f08d8196ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.168+0000 7f08dcbca700 1 --2- 172.21.15.16:0/2562735161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d8101850 0x7f08d8196ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.168+0000 7f08dcbca700 1 -- 172.21.15.16:0/2562735161 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08cc0095f0 con 0x7f08d8100ea0 2024-04-22T15:58:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.168+0000 7f08dcbca700 1 --2- 172.21.15.16:0/2562735161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08d8100ea0 0x7f08d8196990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f08cc006010 tx=0x7f08cc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.168+0000 7f08d5ffb700 1 -- 172.21.15.16:0/2562735161 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08cc01d070 con 0x7f08d8100ea0 2024-04-22T15:58:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.168+0000 7f08d5ffb700 1 -- 172.21.15.16:0/2562735161 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f08cc00f460 con 0x7f08d8100ea0 2024-04-22T15:58:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.168+0000 7f08dee2e700 1 -- 172.21.15.16:0/2562735161 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d819b5f0 con 0x7f08d8100ea0 2024-04-22T15:58:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.168+0000 7f08d5ffb700 1 -- 172.21.15.16:0/2562735161 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08cc017610 con 0x7f08d8100ea0 2024-04-22T15:58:43.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.168+0000 7f08dee2e700 1 -- 172.21.15.16:0/2562735161 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d819bae0 con 0x7f08d8100ea0 2024-04-22T15:58:43.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.169+0000 7f08dee2e700 1 -- 172.21.15.16:0/2562735161 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08d8109240 con 0x7f08d8100ea0 2024-04-22T15:58:43.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.170+0000 7f08d5ffb700 1 -- 172.21.15.16:0/2562735161 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f08cc003a60 con 0x7f08d8100ea0 2024-04-22T15:58:43.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.170+0000 7f08d5ffb700 1 --2- 172.21.15.16:0/2562735161 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f08c806ecc0 0x7f08c8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:43.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.170+0000 7f08d5ffb700 1 -- 172.21.15.16:0/2562735161 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f08cc090310 con 0x7f08d8100ea0 2024-04-22T15:58:43.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.170+0000 7f08d7fff700 1 --2- 172.21.15.16:0/2562735161 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f08c806ecc0 0x7f08c8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:43.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.171+0000 7f08d7fff700 1 --2- 172.21.15.16:0/2562735161 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f08c806ecc0 0x7f08c8071180 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f08d8197fb0 tx=0x7f08c4009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:43.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.175+0000 7f08d5ffb700 1 -- 172.21.15.16:0/2562735161 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f08cc05cca0 con 0x7f08d8100ea0 2024-04-22T15:58:43.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.432+0000 7f08dee2e700 1 -- 172.21.15.16:0/2562735161 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f08d8197cf0 con 0x7f08d8100ea0 2024-04-22T15:58:43.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.433+0000 7f08d5ffb700 1 -- 172.21.15.16:0/2562735161 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1885 (secure 0 0 0) 0x7f08cc05c830 con 0x7f08d8100ea0 2024-04-22T15:58:43.436 INFO:teuthology.orchestra.run.smithi016.stdout:e22 2024-04-22T15:58:43.436 INFO:teuthology.orchestra.run.smithi016.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:58:43.436 INFO:teuthology.orchestra.run.smithi016.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:58:43.436 INFO:teuthology.orchestra.run.smithi016.stdout:legacy client fscid: 1 2024-04-22T15:58:43.436 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:58:43.437 INFO:teuthology.orchestra.run.smithi016.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:58:43.437 INFO:teuthology.orchestra.run.smithi016.stdout:fs_name cephfs 2024-04-22T15:58:43.437 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 17 2024-04-22T15:58:43.437 INFO:teuthology.orchestra.run.smithi016.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-22T15:58:43.437 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-04-22T15:46:27.774571+0000 2024-04-22T15:58:43.437 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-04-22T15:58:05.206233+0000 2024-04-22T15:58:43.437 INFO:teuthology.orchestra.run.smithi016.stdout:tableserver 0 2024-04-22T15:58:43.437 INFO:teuthology.orchestra.run.smithi016.stdout:root 0 2024-04-22T15:58:43.437 INFO:teuthology.orchestra.run.smithi016.stdout:session_timeout 60 2024-04-22T15:58:43.437 INFO:teuthology.orchestra.run.smithi016.stdout:session_autoclose 300 2024-04-22T15:58:43.437 INFO:teuthology.orchestra.run.smithi016.stdout:max_file_size 1099511627776 2024-04-22T15:58:43.437 INFO:teuthology.orchestra.run.smithi016.stdout:max_xattr_size 65536 2024-04-22T15:58:43.437 INFO:teuthology.orchestra.run.smithi016.stdout:required_client_features {} 2024-04-22T15:58:43.437 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure 0 2024-04-22T15:58:43.437 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure_osd_epoch 0 2024-04-22T15:58:43.437 INFO:teuthology.orchestra.run.smithi016.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:58:43.438 INFO:teuthology.orchestra.run.smithi016.stdout:max_mds 1 2024-04-22T15:58:43.438 INFO:teuthology.orchestra.run.smithi016.stdout:in 0 2024-04-22T15:58:43.438 INFO:teuthology.orchestra.run.smithi016.stdout:up {0=24279} 2024-04-22T15:58:43.438 INFO:teuthology.orchestra.run.smithi016.stdout:failed 2024-04-22T15:58:43.438 INFO:teuthology.orchestra.run.smithi016.stdout:damaged 2024-04-22T15:58:43.438 INFO:teuthology.orchestra.run.smithi016.stdout:stopped 2024-04-22T15:58:43.438 INFO:teuthology.orchestra.run.smithi016.stdout:data_pools [3] 2024-04-22T15:58:43.438 INFO:teuthology.orchestra.run.smithi016.stdout:metadata_pool 2 2024-04-22T15:58:43.438 INFO:teuthology.orchestra.run.smithi016.stdout:inline_data enabled 2024-04-22T15:58:43.438 INFO:teuthology.orchestra.run.smithi016.stdout:balancer 2024-04-22T15:58:43.438 INFO:teuthology.orchestra.run.smithi016.stdout:bal_rank_mask -1 2024-04-22T15:58:43.438 INFO:teuthology.orchestra.run.smithi016.stdout:standby_count_wanted 1 2024-04-22T15:58:43.438 INFO:teuthology.orchestra.run.smithi016.stdout:qdb_cluster leader: 24279 members: 24279 2024-04-22T15:58:43.439 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.iobjds{0:24279} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.27:6824/4216277609,v1:172.21.15.27:6825/4216277609] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:58:43.439 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:58:43.439 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:58:43.439 INFO:teuthology.orchestra.run.smithi016.stdout:Standby daemons: 2024-04-22T15:58:43.439 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:58:43.439 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.dgqyes{-1:24293} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.27:6826/476026355,v1:172.21.15.27:6827/476026355] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T15:58:43.439 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.fkkejm{-1:34408} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.16:6826/2871308685,v1:172.21.15.16:6827/2871308685] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:58:43.439 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.pgvphu{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.16:6828/4228489800,v1:172.21.15.16:6829/4228489800] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:58:43.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.435+0000 7f08dee2e700 1 -- 172.21.15.16:0/2562735161 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f08c806ecc0 msgr2=0x7f08c8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:43.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.435+0000 7f08dee2e700 1 --2- 172.21.15.16:0/2562735161 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f08c806ecc0 0x7f08c8071180 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f08d8197fb0 tx=0x7f08c4009190 comp rx=0 tx=0).stop 2024-04-22T15:58:43.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.435+0000 7f08dee2e700 1 -- 172.21.15.16:0/2562735161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08d8100ea0 msgr2=0x7f08d8196990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:43.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.435+0000 7f08dee2e700 1 --2- 172.21.15.16:0/2562735161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08d8100ea0 0x7f08d8196990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f08cc006010 tx=0x7f08cc00bd70 comp rx=0 tx=0).stop 2024-04-22T15:58:43.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.435+0000 7f08dee2e700 1 -- 172.21.15.16:0/2562735161 shutdown_connections 2024-04-22T15:58:43.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.435+0000 7f08dee2e700 1 --2- 172.21.15.16:0/2562735161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08d8100ea0 0x7f08d8196990 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:43.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.435+0000 7f08dee2e700 1 --2- 172.21.15.16:0/2562735161 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f08c806ecc0 0x7f08c8071180 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:43.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.435+0000 7f08dee2e700 1 --2- 172.21.15.16:0/2562735161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d8101850 0x7f08d8196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:43.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.435+0000 7f08dee2e700 1 -- 172.21.15.16:0/2562735161 >> 172.21.15.16:0/2562735161 conn(0x7f08d80fc750 msgr2=0x7f08d80fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:43.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.435+0000 7f08dee2e700 1 -- 172.21.15.16:0/2562735161 shutdown_connections 2024-04-22T15:58:43.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.435+0000 7f08dee2e700 1 -- 172.21.15.16:0/2562735161 wait complete. 2024-04-22T15:58:43.441 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 22 2024-04-22T15:58:43.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.544+0000 7fa51ea6e700 1 -- 172.21.15.16:0/3853081484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa518105bd0 msgr2=0x7fa518105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:43.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.544+0000 7fa51ea6e700 1 --2- 172.21.15.16:0/3853081484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa518105bd0 0x7fa518105fb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa50c009910 tx=0x7fa50c009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:43.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.544+0000 7fa51ea6e700 1 -- 172.21.15.16:0/3853081484 shutdown_connections 2024-04-22T15:58:43.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.544+0000 7fa51ea6e700 1 --2- 172.21.15.16:0/3853081484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5180ffc20 0x7fa518100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:43.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.544+0000 7fa51ea6e700 1 --2- 172.21.15.16:0/3853081484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa518105bd0 0x7fa518105fb0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:43.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.544+0000 7fa51ea6e700 1 -- 172.21.15.16:0/3853081484 >> 172.21.15.16:0/3853081484 conn(0x7fa518075670 msgr2=0x7fa518075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:43.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.544+0000 7fa51ea6e700 1 -- 172.21.15.16:0/3853081484 shutdown_connections 2024-04-22T15:58:43.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.544+0000 7fa51ea6e700 1 -- 172.21.15.16:0/3853081484 wait complete. 2024-04-22T15:58:43.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.545+0000 7fa51ea6e700 1 Processor -- start 2024-04-22T15:58:43.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.545+0000 7fa51ea6e700 1 -- start start 2024-04-22T15:58:43.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.545+0000 7fa51ea6e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5180ffc20 0x7fa518101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:43.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.545+0000 7fa51ea6e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa518105bd0 0x7fa518101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:43.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.545+0000 7fa51ea6e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5181022c0 con 0x7fa518105bd0 2024-04-22T15:58:43.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.545+0000 7fa51ea6e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa518102400 con 0x7fa5180ffc20 2024-04-22T15:58:43.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.545+0000 7fa517fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa518105bd0 0x7fa518101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:43.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.545+0000 7fa517fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa518105bd0 0x7fa518101d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33338/0 (socket says 172.21.15.16:33338) 2024-04-22T15:58:43.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.545+0000 7fa517fff700 1 -- 172.21.15.16:0/298748216 learned_addr learned my addr 172.21.15.16:0/298748216 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:58:43.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.545+0000 7fa51c80a700 1 --2- 172.21.15.16:0/298748216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5180ffc20 0x7fa518101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:43.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.545+0000 7fa517fff700 1 -- 172.21.15.16:0/298748216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5180ffc20 msgr2=0x7fa518101840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:43.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.545+0000 7fa517fff700 1 --2- 172.21.15.16:0/298748216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5180ffc20 0x7fa518101840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:43.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.545+0000 7fa517fff700 1 -- 172.21.15.16:0/298748216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa50c0095f0 con 0x7fa518105bd0 2024-04-22T15:58:43.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.545+0000 7fa51c80a700 1 --2- 172.21.15.16:0/298748216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5180ffc20 0x7fa518101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T15:58:43.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.546+0000 7fa517fff700 1 --2- 172.21.15.16:0/298748216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa518105bd0 0x7fa518101d80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa50400d8d0 tx=0x7fa50400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:43.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.546+0000 7fa515ffb700 1 -- 172.21.15.16:0/298748216 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa504009940 con 0x7fa518105bd0 2024-04-22T15:58:43.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.546+0000 7fa515ffb700 1 -- 172.21.15.16:0/298748216 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa504010460 con 0x7fa518105bd0 2024-04-22T15:58:43.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.546+0000 7fa51ea6e700 1 -- 172.21.15.16:0/298748216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5181a2a20 con 0x7fa518105bd0 2024-04-22T15:58:43.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.546+0000 7fa515ffb700 1 -- 172.21.15.16:0/298748216 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa50400f5f0 con 0x7fa518105bd0 2024-04-22T15:58:43.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.546+0000 7fa51ea6e700 1 -- 172.21.15.16:0/298748216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5181a2e30 con 0x7fa518105bd0 2024-04-22T15:58:43.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.547+0000 7fa51ea6e700 1 -- 172.21.15.16:0/298748216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa51804f2e0 con 0x7fa518105bd0 2024-04-22T15:58:43.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.547+0000 7fa515ffb700 1 -- 172.21.15.16:0/298748216 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa5040105d0 con 0x7fa518105bd0 2024-04-22T15:58:43.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.548+0000 7fa515ffb700 1 --2- 172.21.15.16:0/298748216 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fa50806ecc0 0x7fa508071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:43.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.548+0000 7fa515ffb700 1 -- 172.21.15.16:0/298748216 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa50408ecc0 con 0x7fa518105bd0 2024-04-22T15:58:43.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.548+0000 7fa51c80a700 1 --2- 172.21.15.16:0/298748216 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fa50806ecc0 0x7fa508071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:43.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.548+0000 7fa51c80a700 1 --2- 172.21.15.16:0/298748216 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fa50806ecc0 0x7fa508071180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa50c005b40 tx=0x7fa50c01a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:43.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.553+0000 7fa515ffb700 1 -- 172.21.15.16:0/298748216 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa504057ba0 con 0x7fa518105bd0 2024-04-22T15:58:43.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.782+0000 7fa51ea6e700 1 -- 172.21.15.16:0/298748216 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa518068dd0 con 0x7fa50806ecc0 2024-04-22T15:58:43.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.783+0000 7fa515ffb700 1 -- 172.21.15.16:0/298748216 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa518068dd0 con 0x7fa50806ecc0 2024-04-22T15:58:43.786 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:58:43.786 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:58:43.786 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-22T15:58:43.786 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:58:43.786 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-22T15:58:43.786 INFO:teuthology.orchestra.run.smithi016.stdout: "osd", 2024-04-22T15:58:43.787 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-04-22T15:58:43.787 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr", 2024-04-22T15:58:43.787 INFO:teuthology.orchestra.run.smithi016.stdout: "mon" 2024-04-22T15:58:43.787 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-22T15:58:43.787 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "14/23 daemons upgraded", 2024-04-22T15:58:43.787 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:58:43.787 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-22T15:58:43.787 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:58:43.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.786+0000 7fa51ea6e700 1 -- 172.21.15.16:0/298748216 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fa50806ecc0 msgr2=0x7fa508071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:43.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.786+0000 7fa51ea6e700 1 --2- 172.21.15.16:0/298748216 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fa50806ecc0 0x7fa508071180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa50c005b40 tx=0x7fa50c01a040 comp rx=0 tx=0).stop 2024-04-22T15:58:43.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.786+0000 7fa51ea6e700 1 -- 172.21.15.16:0/298748216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa518105bd0 msgr2=0x7fa518101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:43.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.786+0000 7fa51ea6e700 1 --2- 172.21.15.16:0/298748216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa518105bd0 0x7fa518101d80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa50400d8d0 tx=0x7fa50400dc90 comp rx=0 tx=0).stop 2024-04-22T15:58:43.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.786+0000 7fa51ea6e700 1 -- 172.21.15.16:0/298748216 shutdown_connections 2024-04-22T15:58:43.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.786+0000 7fa51ea6e700 1 --2- 172.21.15.16:0/298748216 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fa50806ecc0 0x7fa508071180 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:43.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.786+0000 7fa51ea6e700 1 --2- 172.21.15.16:0/298748216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5180ffc20 0x7fa518101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:43.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.786+0000 7fa51ea6e700 1 --2- 172.21.15.16:0/298748216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa518105bd0 0x7fa518101d80 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:43.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.786+0000 7fa51ea6e700 1 -- 172.21.15.16:0/298748216 >> 172.21.15.16:0/298748216 conn(0x7fa518075670 msgr2=0x7fa5180686c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:43.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.786+0000 7fa51ea6e700 1 -- 172.21.15.16:0/298748216 shutdown_connections 2024-04-22T15:58:43.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.786+0000 7fa51ea6e700 1 -- 172.21.15.16:0/298748216 wait complete. 2024-04-22T15:58:43.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.895+0000 7facde980700 1 -- 172.21.15.16:0/3556440726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facd80ff340 msgr2=0x7facd80ff7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:43.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.895+0000 7facde980700 1 --2- 172.21.15.16:0/3556440726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facd80ff340 0x7facd80ff7a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7faccc009910 tx=0x7faccc009c20 comp rx=0 tx=0).stop 2024-04-22T15:58:43.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.895+0000 7facde980700 1 -- 172.21.15.16:0/3556440726 shutdown_connections 2024-04-22T15:58:43.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.895+0000 7facde980700 1 --2- 172.21.15.16:0/3556440726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facd80ff340 0x7facd80ff7a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:43.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.895+0000 7facde980700 1 --2- 172.21.15.16:0/3556440726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facd8105350 0x7facd8105730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:43.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.895+0000 7facde980700 1 -- 172.21.15.16:0/3556440726 >> 172.21.15.16:0/3556440726 conn(0x7facd8074e30 msgr2=0x7facd8075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:43.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.895+0000 7facde980700 1 -- 172.21.15.16:0/3556440726 shutdown_connections 2024-04-22T15:58:43.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.895+0000 7facde980700 1 -- 172.21.15.16:0/3556440726 wait complete. 2024-04-22T15:58:43.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.895+0000 7facde980700 1 Processor -- start 2024-04-22T15:58:43.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.895+0000 7facde980700 1 -- start start 2024-04-22T15:58:43.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.896+0000 7facde980700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facd80ff340 0x7facd8197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:43.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.896+0000 7facde980700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facd8105350 0x7facd8197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:43.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.896+0000 7facde980700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd81982b0 con 0x7facd8105350 2024-04-22T15:58:43.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.896+0000 7facde980700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd81983f0 con 0x7facd80ff340 2024-04-22T15:58:43.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.896+0000 7facd77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facd8105350 0x7facd8197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:43.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.896+0000 7facd77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facd8105350 0x7facd8197bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33364/0 (socket says 172.21.15.16:33364) 2024-04-22T15:58:43.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.896+0000 7facd7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facd80ff340 0x7facd8197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:43.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.896+0000 7facd77fe700 1 -- 172.21.15.16:0/2893799996 learned_addr learned my addr 172.21.15.16:0/2893799996 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:58:43.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.896+0000 7facd77fe700 1 -- 172.21.15.16:0/2893799996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facd80ff340 msgr2=0x7facd8197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:43.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.896+0000 7facd77fe700 1 --2- 172.21.15.16:0/2893799996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facd80ff340 0x7facd8197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:43.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.896+0000 7facd77fe700 1 -- 172.21.15.16:0/2893799996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc0095f0 con 0x7facd8105350 2024-04-22T15:58:43.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.896+0000 7facd77fe700 1 --2- 172.21.15.16:0/2893799996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facd8105350 0x7facd8197bd0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7faccc006010 tx=0x7faccc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:43.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.896+0000 7facd57fa700 1 -- 172.21.15.16:0/2893799996 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faccc01d070 con 0x7facd8105350 2024-04-22T15:58:43.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.896+0000 7facd57fa700 1 -- 172.21.15.16:0/2893799996 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faccc005dc0 con 0x7facd8105350 2024-04-22T15:58:43.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.896+0000 7facde980700 1 -- 172.21.15.16:0/2893799996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd81a0700 con 0x7facd8105350 2024-04-22T15:58:43.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.897+0000 7facde980700 1 -- 172.21.15.16:0/2893799996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd81a0b70 con 0x7facd8105350 2024-04-22T15:58:43.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.897+0000 7facd57fa700 1 -- 172.21.15.16:0/2893799996 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faccc017400 con 0x7facd8105350 2024-04-22T15:58:43.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.898+0000 7facde980700 1 -- 172.21.15.16:0/2893799996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd8107850 con 0x7facd8105350 2024-04-22T15:58:43.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.898+0000 7facd57fa700 1 -- 172.21.15.16:0/2893799996 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7faccc003a60 con 0x7facd8105350 2024-04-22T15:58:43.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.899+0000 7facd57fa700 1 --2- 172.21.15.16:0/2893799996 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7facc006ecc0 0x7facc0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:58:43.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.899+0000 7facd57fa700 1 -- 172.21.15.16:0/2893799996 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6328+0+0 (secure 0 0 0) 0x7faccc090240 con 0x7facd8105350 2024-04-22T15:58:43.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.899+0000 7facd7fff700 1 --2- 172.21.15.16:0/2893799996 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7facc006ecc0 0x7facc0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:58:43.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.899+0000 7facd7fff700 1 --2- 172.21.15.16:0/2893799996 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7facc006ecc0 0x7facc0071180 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7facc8005950 tx=0x7facc800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:58:43.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:43.903+0000 7facd57fa700 1 -- 172.21.15.16:0/2893799996 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7faccc0590a0 con 0x7facd8105350 2024-04-22T15:58:44.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:43 smithi027 ceph-mon[83054]: pgmap v192: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:44.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:43 smithi027 ceph-mon[83054]: from='client.34420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:44.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:43 smithi027 ceph-mon[83054]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:44.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:43 smithi027 ceph-mon[83054]: Upgrade: Updating mds.cephfs.smithi027.iobjds 2024-04-22T15:58:44.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:43 smithi027 ceph-mon[83054]: Deploying daemon mds.cephfs.smithi027.iobjds on smithi027 2024-04-22T15:58:44.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:43 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/2279628284' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:44.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:43 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/2562735161' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:58:44.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:43 smithi016 ceph-mon[110740]: pgmap v192: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:44.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:43 smithi016 ceph-mon[110740]: from='client.34420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:44.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:43 smithi016 ceph-mon[110740]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:44.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:43 smithi016 ceph-mon[110740]: Upgrade: Updating mds.cephfs.smithi027.iobjds 2024-04-22T15:58:44.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:43 smithi016 ceph-mon[110740]: Deploying daemon mds.cephfs.smithi027.iobjds on smithi027 2024-04-22T15:58:44.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:43 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/2279628284' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:58:44.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:43 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/2562735161' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:58:44.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:44.201+0000 7facde980700 1 -- 172.21.15.16:0/2893799996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7facd804f2e0 con 0x7facd8105350 2024-04-22T15:58:44.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:44.202+0000 7facd57fa700 1 -- 172.21.15.16:0/2893799996 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7faccc05c760 con 0x7facd8105350 2024-04-22T15:58:44.205 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:58:44.205 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:58:44.205 INFO:teuthology.orchestra.run.smithi016.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:58:44.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:44.204+0000 7facde980700 1 -- 172.21.15.16:0/2893799996 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7facc006ecc0 msgr2=0x7facc0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:44.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:44.204+0000 7facde980700 1 --2- 172.21.15.16:0/2893799996 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7facc006ecc0 0x7facc0071180 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7facc8005950 tx=0x7facc800b410 comp rx=0 tx=0).stop 2024-04-22T15:58:44.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:44.204+0000 7facde980700 1 -- 172.21.15.16:0/2893799996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facd8105350 msgr2=0x7facd8197bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:58:44.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:44.204+0000 7facde980700 1 --2- 172.21.15.16:0/2893799996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facd8105350 0x7facd8197bd0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7faccc006010 tx=0x7faccc00bd70 comp rx=0 tx=0).stop 2024-04-22T15:58:44.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:44.204+0000 7facde980700 1 -- 172.21.15.16:0/2893799996 shutdown_connections 2024-04-22T15:58:44.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:44.204+0000 7facde980700 1 --2- 172.21.15.16:0/2893799996 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7facc006ecc0 0x7facc0071180 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:44.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:44.204+0000 7facde980700 1 --2- 172.21.15.16:0/2893799996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facd80ff340 0x7facd8197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:44.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:44.204+0000 7facde980700 1 --2- 172.21.15.16:0/2893799996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facd8105350 0x7facd8197bd0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:58:44.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:44.204+0000 7facde980700 1 -- 172.21.15.16:0/2893799996 >> 172.21.15.16:0/2893799996 conn(0x7facd8074e30 msgr2=0x7facd80fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:58:44.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:44.204+0000 7facde980700 1 -- 172.21.15.16:0/2893799996 shutdown_connections 2024-04-22T15:58:44.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:58:44.204+0000 7facde980700 1 -- 172.21.15.16:0/2893799996 wait complete. 2024-04-22T15:58:44.934 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:44 smithi027 ceph-mon[83054]: from='client.34436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:44.935 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:44 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/2893799996' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:58:45.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:44 smithi016 ceph-mon[110740]: from='client.34436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:58:45.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:44 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/2893799996' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:58:45.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:44 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016[110736]: 2024-04-22T15:58:44.932+0000 7f2925f97700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T15:58:46.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:45 smithi027 ceph-mon[83054]: pgmap v193: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:46.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:45 smithi027 ceph-mon[83054]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T15:58:46.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:45 smithi027 ceph-mon[83054]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T15:58:46.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:45 smithi027 ceph-mon[83054]: osdmap e83: 6 total, 6 up, 6 in 2024-04-22T15:58:46.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:45 smithi027 ceph-mon[83054]: Standby daemon mds.cephfs.smithi027.dgqyes assigned to filesystem cephfs as rank 0 2024-04-22T15:58:46.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:45 smithi027 ceph-mon[83054]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T15:58:46.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:45 smithi027 ceph-mon[83054]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-04-22T15:58:46.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:45 smithi027 ceph-mon[83054]: fsmap cephfs:1/1 {0=cephfs.smithi027.dgqyes=up:replay} 2 up:standby 2024-04-22T15:58:46.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:45 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:46.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:45 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:58:46.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:45 smithi016 ceph-mon[110740]: pgmap v193: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail 2024-04-22T15:58:46.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:45 smithi016 ceph-mon[110740]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T15:58:46.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:45 smithi016 ceph-mon[110740]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T15:58:46.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:45 smithi016 ceph-mon[110740]: osdmap e83: 6 total, 6 up, 6 in 2024-04-22T15:58:46.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:45 smithi016 ceph-mon[110740]: Standby daemon mds.cephfs.smithi027.dgqyes assigned to filesystem cephfs as rank 0 2024-04-22T15:58:46.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:45 smithi016 ceph-mon[110740]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T15:58:46.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:45 smithi016 ceph-mon[110740]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-04-22T15:58:46.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:45 smithi016 ceph-mon[110740]: fsmap cephfs:1/1 {0=cephfs.smithi027.dgqyes=up:replay} 2 up:standby 2024-04-22T15:58:46.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:45 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:46.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:45 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:58:47.068 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.iobjds"}]: dispatch 2024-04-22T15:58:47.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.iobjds"}]: dispatch 2024-04-22T15:58:48.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:47 smithi027 ceph-mon[83054]: pgmap v195: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 410 KiB/s rd, 0 op/s 2024-04-22T15:58:48.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:47 smithi016 ceph-mon[110740]: pgmap v195: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 410 KiB/s rd, 0 op/s 2024-04-22T15:58:50.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:49 smithi027 ceph-mon[83054]: pgmap v196: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 5.2 MiB/s rd, 2 op/s 2024-04-22T15:58:50.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:49 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:50.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:49 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:50.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:49 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:58:50.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:49 smithi016 ceph-mon[110740]: pgmap v196: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 5.2 MiB/s rd, 2 op/s 2024-04-22T15:58:50.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:49 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:50.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:49 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:50.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:49 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:58:51.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:50 smithi027 ceph-mon[83054]: mds.? [v2:172.21.15.27:6824/3921606817,v1:172.21.15.27:6825/3921606817] up:boot 2024-04-22T15:58:51.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:50 smithi027 ceph-mon[83054]: fsmap cephfs:1/1 {0=cephfs.smithi027.dgqyes=up:replay} 3 up:standby 2024-04-22T15:58:51.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:50 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.iobjds"}]: dispatch 2024-04-22T15:58:51.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:50 smithi016 ceph-mon[110740]: mds.? [v2:172.21.15.27:6824/3921606817,v1:172.21.15.27:6825/3921606817] up:boot 2024-04-22T15:58:51.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:50 smithi016 ceph-mon[110740]: fsmap cephfs:1/1 {0=cephfs.smithi027.dgqyes=up:replay} 3 up:standby 2024-04-22T15:58:51.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:50 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.iobjds"}]: dispatch 2024-04-22T15:58:52.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:51 smithi027 ceph-mon[83054]: pgmap v197: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 11 MiB/s rd, 3 op/s 2024-04-22T15:58:52.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:51 smithi016 ceph-mon[110740]: pgmap v197: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 11 MiB/s rd, 3 op/s 2024-04-22T15:58:54.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:54 smithi016 ceph-mon[110740]: pgmap v198: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 19 MiB/s rd, 6 op/s 2024-04-22T15:58:54.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:54 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:54.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:54 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:54.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:54 smithi027 ceph-mon[83054]: pgmap v198: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 19 MiB/s rd, 6 op/s 2024-04-22T15:58:54.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:54 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:54.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:54 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:55.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:55 smithi027 ceph-mon[83054]: pgmap v199: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 6 op/s 2024-04-22T15:58:55.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:55 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:55.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:55 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:55.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:55 smithi016 ceph-mon[110740]: pgmap v199: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 6 op/s 2024-04-22T15:58:55.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:55 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:55.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:55 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:58:56.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:56 smithi027 ceph-mon[83054]: reconnect by client.24341 192.168.0.1:0/2832093308 after 0.000999975 2024-04-22T15:58:56.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:56 smithi027 ceph-mon[83054]: reconnect by client.14544 192.168.0.1:0/3830921175 after 0.000999975 2024-04-22T15:58:56.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:56 smithi027 ceph-mon[83054]: mds.? [v2:172.21.15.27:6826/476026355,v1:172.21.15.27:6827/476026355] up:reconnect 2024-04-22T15:58:56.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:56 smithi027 ceph-mon[83054]: fsmap cephfs:1/1 {0=cephfs.smithi027.dgqyes=up:reconnect} 3 up:standby 2024-04-22T15:58:56.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:56 smithi016 ceph-mon[110740]: reconnect by client.24341 192.168.0.1:0/2832093308 after 0.000999975 2024-04-22T15:58:56.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:56 smithi016 ceph-mon[110740]: reconnect by client.14544 192.168.0.1:0/3830921175 after 0.000999975 2024-04-22T15:58:56.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:56 smithi016 ceph-mon[110740]: mds.? [v2:172.21.15.27:6826/476026355,v1:172.21.15.27:6827/476026355] up:reconnect 2024-04-22T15:58:56.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:56 smithi016 ceph-mon[110740]: fsmap cephfs:1/1 {0=cephfs.smithi027.dgqyes=up:reconnect} 3 up:standby 2024-04-22T15:58:57.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:57 smithi027 ceph-mon[83054]: pgmap v200: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 6 op/s 2024-04-22T15:58:57.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:57 smithi027 ceph-mon[83054]: mds.? [v2:172.21.15.27:6826/476026355,v1:172.21.15.27:6827/476026355] up:rejoin 2024-04-22T15:58:57.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:57 smithi027 ceph-mon[83054]: fsmap cephfs:1/1 {0=cephfs.smithi027.dgqyes=up:rejoin} 3 up:standby 2024-04-22T15:58:57.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:57 smithi027 ceph-mon[83054]: daemon mds.cephfs.smithi027.dgqyes is now active in filesystem cephfs as rank 0 2024-04-22T15:58:57.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:57 smithi016 ceph-mon[110740]: pgmap v200: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 6 op/s 2024-04-22T15:58:57.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:57 smithi016 ceph-mon[110740]: mds.? [v2:172.21.15.27:6826/476026355,v1:172.21.15.27:6827/476026355] up:rejoin 2024-04-22T15:58:57.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:57 smithi016 ceph-mon[110740]: fsmap cephfs:1/1 {0=cephfs.smithi027.dgqyes=up:rejoin} 3 up:standby 2024-04-22T15:58:57.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:57 smithi016 ceph-mon[110740]: daemon mds.cephfs.smithi027.dgqyes is now active in filesystem cephfs as rank 0 2024-04-22T15:58:58.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:58 smithi016 ceph-mon[110740]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T15:58:58.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:58 smithi016 ceph-mon[110740]: mds.? [v2:172.21.15.27:6826/476026355,v1:172.21.15.27:6827/476026355] up:active 2024-04-22T15:58:58.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:58 smithi016 ceph-mon[110740]: fsmap cephfs:1 {0=cephfs.smithi027.dgqyes=up:active} 3 up:standby 2024-04-22T15:58:58.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:58 smithi027 ceph-mon[83054]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T15:58:58.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:58 smithi027 ceph-mon[83054]: mds.? [v2:172.21.15.27:6826/476026355,v1:172.21.15.27:6827/476026355] up:active 2024-04-22T15:58:58.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:58 smithi027 ceph-mon[83054]: fsmap cephfs:1 {0=cephfs.smithi027.dgqyes=up:active} 3 up:standby 2024-04-22T15:58:59.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:58:59 smithi016 ceph-mon[110740]: pgmap v201: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 6 op/s 2024-04-22T15:58:59.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:58:59 smithi027 ceph-mon[83054]: pgmap v201: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 6 op/s 2024-04-22T15:59:01.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:01 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:01.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:01 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:59:01.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:01 smithi027 ceph-mon[83054]: pgmap v202: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 16 MiB/s rd, 170 B/s wr, 5 op/s 2024-04-22T15:59:01.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:01 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:01.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:01 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:59:01.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:01 smithi016 ceph-mon[110740]: pgmap v202: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 16 MiB/s rd, 170 B/s wr, 5 op/s 2024-04-22T15:59:03.459 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:03 smithi027 ceph-mon[83054]: pgmap v203: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 12 MiB/s rd, 426 B/s wr, 6 op/s 2024-04-22T15:59:03.459 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:03 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:03.460 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:03 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:03.460 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:03 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:59:03.460 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:03 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:59:03.460 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:03 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:03.460 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:03 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:59:03.460 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:03 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:03.460 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:03 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:03.460 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:03 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:03.460 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:03 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:03.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:03 smithi016 ceph-mon[110740]: pgmap v203: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 12 MiB/s rd, 426 B/s wr, 6 op/s 2024-04-22T15:59:03.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:03 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:03.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:03 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:03.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:03 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:59:03.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:03 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:59:03.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:03 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:03.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:03 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:59:03.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:03 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:03.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:03 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:03.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:03 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:03.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:03 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:05.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:05 smithi027 ceph-mon[83054]: Upgrade: Updating mds.cephfs.smithi027.dgqyes 2024-04-22T15:59:05.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:05 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:05.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:05 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.dgqyes", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:59:05.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:05 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:59:05.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:05 smithi027 ceph-mon[83054]: Deploying daemon mds.cephfs.smithi027.dgqyes on smithi027 2024-04-22T15:59:05.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:05 smithi016 ceph-mon[110740]: Upgrade: Updating mds.cephfs.smithi027.dgqyes 2024-04-22T15:59:05.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:05 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:05.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:05 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.dgqyes", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T15:59:05.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:05 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:59:05.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:05 smithi016 ceph-mon[110740]: Deploying daemon mds.cephfs.smithi027.dgqyes on smithi027 2024-04-22T15:59:06.183 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:06 smithi027 ceph-mon[83054]: pgmap v204: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 4.6 MiB/s rd, 4.6 KiB/s wr, 6 op/s 2024-04-22T15:59:06.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:06 smithi016 ceph-mon[110740]: pgmap v204: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 4.6 MiB/s rd, 4.6 KiB/s wr, 6 op/s 2024-04-22T15:59:06.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:06 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016[110736]: 2024-04-22T15:59:06.180+0000 7f2925f97700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T15:59:07.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:07 smithi016 ceph-mon[110740]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T15:59:07.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:07 smithi016 ceph-mon[110740]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T15:59:07.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:07 smithi016 ceph-mon[110740]: osdmap e84: 6 total, 6 up, 6 in 2024-04-22T15:59:07.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:07 smithi016 ceph-mon[110740]: Standby daemon mds.cephfs.smithi016.fkkejm assigned to filesystem cephfs as rank 0 2024-04-22T15:59:07.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:07 smithi016 ceph-mon[110740]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T15:59:07.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:07 smithi016 ceph-mon[110740]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-04-22T15:59:07.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:07 smithi016 ceph-mon[110740]: fsmap cephfs:1/1 {0=cephfs.smithi016.fkkejm=up:replay} 2 up:standby 2024-04-22T15:59:07.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:07 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.dgqyes"}]: dispatch 2024-04-22T15:59:07.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:07 smithi027 ceph-mon[83054]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T15:59:07.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:07 smithi027 ceph-mon[83054]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T15:59:07.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:07 smithi027 ceph-mon[83054]: osdmap e84: 6 total, 6 up, 6 in 2024-04-22T15:59:07.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:07 smithi027 ceph-mon[83054]: Standby daemon mds.cephfs.smithi016.fkkejm assigned to filesystem cephfs as rank 0 2024-04-22T15:59:07.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:07 smithi027 ceph-mon[83054]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T15:59:07.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:07 smithi027 ceph-mon[83054]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-04-22T15:59:07.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:07 smithi027 ceph-mon[83054]: fsmap cephfs:1/1 {0=cephfs.smithi016.fkkejm=up:replay} 2 up:standby 2024-04-22T15:59:07.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:07 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.dgqyes"}]: dispatch 2024-04-22T15:59:08.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:08 smithi027 ceph-mon[83054]: pgmap v206: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 2.4 MiB/s rd, 5.5 KiB/s wr, 7 op/s 2024-04-22T15:59:08.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:08 smithi016 ceph-mon[110740]: pgmap v206: 65 pgs: 65 active+clean; 250 MiB data, 976 MiB used, 535 GiB / 536 GiB avail; 2.4 MiB/s rd, 5.5 KiB/s wr, 7 op/s 2024-04-22T15:59:10.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:10 smithi016 ceph-mon[110740]: pgmap v207: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 2.8 MiB/s rd, 5.5 KiB/s wr, 7 op/s 2024-04-22T15:59:10.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:10 smithi027 ceph-mon[83054]: pgmap v207: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 2.8 MiB/s rd, 5.5 KiB/s wr, 7 op/s 2024-04-22T15:59:12.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:12 smithi016 ceph-mon[110740]: pgmap v208: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 7.6 MiB/s rd, 5.3 KiB/s wr, 8 op/s 2024-04-22T15:59:12.444 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:12 smithi027 ceph-mon[83054]: pgmap v208: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 7.6 MiB/s rd, 5.3 KiB/s wr, 8 op/s 2024-04-22T15:59:14.165 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:14 smithi027 ceph-mon[83054]: pgmap v209: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 17 MiB/s rd, 5.0 KiB/s wr, 7 op/s 2024-04-22T15:59:14.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.318+0000 7f11d8d10700 1 -- 172.21.15.16:0/3212275222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d4107d90 msgr2=0x7f11d4108170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:14.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.318+0000 7f11d8d10700 1 --2- 172.21.15.16:0/3212275222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d4107d90 0x7f11d4108170 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f11c8009910 tx=0x7f11c8009c20 comp rx=0 tx=0).stop 2024-04-22T15:59:14.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.318+0000 7f11d8d10700 1 -- 172.21.15.16:0/3212275222 shutdown_connections 2024-04-22T15:59:14.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.318+0000 7f11d8d10700 1 --2- 172.21.15.16:0/3212275222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11d41086b0 0x7f11d41139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:14.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.318+0000 7f11d8d10700 1 --2- 172.21.15.16:0/3212275222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d4107d90 0x7f11d4108170 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:14.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.318+0000 7f11d8d10700 1 -- 172.21.15.16:0/3212275222 >> 172.21.15.16:0/3212275222 conn(0x7f11d406cd90 msgr2=0x7f11d406d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:14.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.318+0000 7f11d8d10700 1 -- 172.21.15.16:0/3212275222 shutdown_connections 2024-04-22T15:59:14.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.318+0000 7f11d8d10700 1 -- 172.21.15.16:0/3212275222 wait complete. 2024-04-22T15:59:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.319+0000 7f11d8d10700 1 Processor -- start 2024-04-22T15:59:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.319+0000 7f11d8d10700 1 -- start start 2024-04-22T15:59:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.319+0000 7f11d8d10700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11d4107d90 0x7f11d4109820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.319+0000 7f11d8d10700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d41086b0 0x7f11d4109d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.319+0000 7f11d8d10700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d410a2a0 con 0x7f11d41086b0 2024-04-22T15:59:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.319+0000 7f11d8d10700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d410a3e0 con 0x7f11d4107d90 2024-04-22T15:59:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.320+0000 7f11d259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11d4107d90 0x7f11d4109820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.320+0000 7f11d259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11d4107d90 0x7f11d4109820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:45196/0 (socket says 172.21.15.16:45196) 2024-04-22T15:59:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.320+0000 7f11d259c700 1 -- 172.21.15.16:0/3788917188 learned_addr learned my addr 172.21.15.16:0/3788917188 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:59:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.320+0000 7f11d1d9b700 1 --2- 172.21.15.16:0/3788917188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d41086b0 0x7f11d4109d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.320+0000 7f11d259c700 1 -- 172.21.15.16:0/3788917188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d41086b0 msgr2=0x7f11d4109d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.320+0000 7f11d259c700 1 --2- 172.21.15.16:0/3788917188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d41086b0 0x7f11d4109d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.320+0000 7f11d259c700 1 -- 172.21.15.16:0/3788917188 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c80095f0 con 0x7f11d4107d90 2024-04-22T15:59:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.320+0000 7f11d1d9b700 1 --2- 172.21.15.16:0/3788917188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d41086b0 0x7f11d4109d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:59:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.321+0000 7f11d259c700 1 --2- 172.21.15.16:0/3788917188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11d4107d90 0x7f11d4109820 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f11c8005f50 tx=0x7f11c800f740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.321+0000 7f11c37fe700 1 -- 172.21.15.16:0/3788917188 <== mon.1 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11c801d070 con 0x7f11d4107d90 2024-04-22T15:59:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.321+0000 7f11c37fe700 1 -- 172.21.15.16:0/3788917188 <== mon.1 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f11c800b610 con 0x7f11d4107d90 2024-04-22T15:59:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.321+0000 7f11d8d10700 1 -- 172.21.15.16:0/3788917188 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d410a660 con 0x7f11d4107d90 2024-04-22T15:59:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.321+0000 7f11d8d10700 1 -- 172.21.15.16:0/3788917188 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d41ab500 con 0x7f11d4107d90 2024-04-22T15:59:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.321+0000 7f11c37fe700 1 -- 172.21.15.16:0/3788917188 <== mon.1 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11c8022470 con 0x7f11d4107d90 2024-04-22T15:59:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.322+0000 7f11d8d10700 1 -- 172.21.15.16:0/3788917188 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b4005320 con 0x7f11d4107d90 2024-04-22T15:59:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.323+0000 7f11c37fe700 1 -- 172.21.15.16:0/3788917188 <== mon.1 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f11c80225d0 con 0x7f11d4107d90 2024-04-22T15:59:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.324+0000 7f11c37fe700 1 --2- 172.21.15.16:0/3788917188 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f11bc06ecc0 0x7f11bc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.324+0000 7f11c37fe700 1 -- 172.21.15.16:0/3788917188 <== mon.1 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f11c8090190 con 0x7f11d4107d90 2024-04-22T15:59:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.324+0000 7f11d1d9b700 1 --2- 172.21.15.16:0/3788917188 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f11bc06ecc0 0x7f11bc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.324+0000 7f11d1d9b700 1 --2- 172.21.15.16:0/3788917188 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f11bc06ecc0 0x7f11bc071180 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f11d410b010 tx=0x7f11c4009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:14.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.329+0000 7f11c37fe700 1 -- 172.21.15.16:0/3788917188 <== mon.1 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f11c8058fc0 con 0x7f11d4107d90 2024-04-22T15:59:14.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:14 smithi016 ceph-mon[110740]: pgmap v209: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 17 MiB/s rd, 5.0 KiB/s wr, 7 op/s 2024-04-22T15:59:14.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.593+0000 7f11d8d10700 1 -- 172.21.15.16:0/3788917188 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f11b4000bf0 con 0x7f11bc06ecc0 2024-04-22T15:59:14.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.595+0000 7f11c37fe700 1 -- 172.21.15.16:0/3788917188 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f11b4000bf0 con 0x7f11bc06ecc0 2024-04-22T15:59:14.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.597+0000 7f11d8d10700 1 -- 172.21.15.16:0/3788917188 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f11bc06ecc0 msgr2=0x7f11bc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:14.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.597+0000 7f11d8d10700 1 --2- 172.21.15.16:0/3788917188 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f11bc06ecc0 0x7f11bc071180 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f11d410b010 tx=0x7f11c4009260 comp rx=0 tx=0).stop 2024-04-22T15:59:14.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.597+0000 7f11d8d10700 1 -- 172.21.15.16:0/3788917188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11d4107d90 msgr2=0x7f11d4109820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:14.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.597+0000 7f11d8d10700 1 --2- 172.21.15.16:0/3788917188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11d4107d90 0x7f11d4109820 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f11c8005f50 tx=0x7f11c800f740 comp rx=0 tx=0).stop 2024-04-22T15:59:14.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.597+0000 7f11d8d10700 1 -- 172.21.15.16:0/3788917188 shutdown_connections 2024-04-22T15:59:14.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.597+0000 7f11d8d10700 1 --2- 172.21.15.16:0/3788917188 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f11bc06ecc0 0x7f11bc071180 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:14.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.597+0000 7f11d8d10700 1 --2- 172.21.15.16:0/3788917188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11d4107d90 0x7f11d4109820 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:14.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.597+0000 7f11d8d10700 1 --2- 172.21.15.16:0/3788917188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d41086b0 0x7f11d4109d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:14.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.597+0000 7f11d8d10700 1 -- 172.21.15.16:0/3788917188 >> 172.21.15.16:0/3788917188 conn(0x7f11d406cd90 msgr2=0x7f11d410e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:14.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.597+0000 7f11d8d10700 1 -- 172.21.15.16:0/3788917188 shutdown_connections 2024-04-22T15:59:14.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.597+0000 7f11d8d10700 1 -- 172.21.15.16:0/3788917188 wait complete. 2024-04-22T15:59:14.616 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T15:59:14.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.712+0000 7fb0179db700 1 -- 172.21.15.16:0/337394037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb010073a80 msgr2=0x7fb01010cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:14.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.712+0000 7fb0179db700 1 --2- 172.21.15.16:0/337394037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb010073a80 0x7fb01010cf90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb000009910 tx=0x7fb000009c20 comp rx=0 tx=0).stop 2024-04-22T15:59:14.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.712+0000 7fb0179db700 1 -- 172.21.15.16:0/337394037 shutdown_connections 2024-04-22T15:59:14.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.712+0000 7fb0179db700 1 --2- 172.21.15.16:0/337394037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb010073a80 0x7fb01010cf90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:14.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.712+0000 7fb0179db700 1 --2- 172.21.15.16:0/337394037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb010073160 0x7fb010073540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:14.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.712+0000 7fb0179db700 1 -- 172.21.15.16:0/337394037 >> 172.21.15.16:0/337394037 conn(0x7fb0100fc740 msgr2=0x7fb0100feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.713+0000 7fb0179db700 1 -- 172.21.15.16:0/337394037 shutdown_connections 2024-04-22T15:59:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.713+0000 7fb0179db700 1 -- 172.21.15.16:0/337394037 wait complete. 2024-04-22T15:59:14.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.713+0000 7fb0179db700 1 Processor -- start 2024-04-22T15:59:14.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.713+0000 7fb0179db700 1 -- start start 2024-04-22T15:59:14.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.714+0000 7fb0179db700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb010073160 0x7fb01019e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:14.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.714+0000 7fb0179db700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb010073a80 0x7fb01019ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:14.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.714+0000 7fb0179db700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01019f3f0 con 0x7fb010073a80 2024-04-22T15:59:14.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.714+0000 7fb0179db700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0101988f0 con 0x7fb010073160 2024-04-22T15:59:14.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.714+0000 7fb014f76700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb010073a80 0x7fb01019ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:14.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.714+0000 7fb015777700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb010073160 0x7fb01019e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:14.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.714+0000 7fb014f76700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb010073a80 0x7fb01019ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45952/0 (socket says 172.21.15.16:45952) 2024-04-22T15:59:14.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.714+0000 7fb014f76700 1 -- 172.21.15.16:0/1074678191 learned_addr learned my addr 172.21.15.16:0/1074678191 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:59:14.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.714+0000 7fb014f76700 1 -- 172.21.15.16:0/1074678191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb010073160 msgr2=0x7fb01019e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:14.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.714+0000 7fb014f76700 1 --2- 172.21.15.16:0/1074678191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb010073160 0x7fb01019e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:14.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.714+0000 7fb014f76700 1 -- 172.21.15.16:0/1074678191 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0000095f0 con 0x7fb010073a80 2024-04-22T15:59:14.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.714+0000 7fb014f76700 1 --2- 172.21.15.16:0/1074678191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb010073a80 0x7fb01019ed60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb00000b7d0 tx=0x7fb00000bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:14.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.715+0000 7fb0067fc700 1 -- 172.21.15.16:0/1074678191 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb00001d070 con 0x7fb010073a80 2024-04-22T15:59:14.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.715+0000 7fb0067fc700 1 -- 172.21.15.16:0/1074678191 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb0000052a0 con 0x7fb010073a80 2024-04-22T15:59:14.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.715+0000 7fb0179db700 1 -- 172.21.15.16:0/1074678191 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb010198b70 con 0x7fb010073a80 2024-04-22T15:59:14.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.715+0000 7fb0067fc700 1 -- 172.21.15.16:0/1074678191 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb000022470 con 0x7fb010073a80 2024-04-22T15:59:14.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.715+0000 7fb0179db700 1 -- 172.21.15.16:0/1074678191 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb010199060 con 0x7fb010073a80 2024-04-22T15:59:14.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.716+0000 7fb0179db700 1 -- 172.21.15.16:0/1074678191 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb01010a690 con 0x7fb010073a80 2024-04-22T15:59:14.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.716+0000 7fb0067fc700 1 -- 172.21.15.16:0/1074678191 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb00000f850 con 0x7fb010073a80 2024-04-22T15:59:14.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.717+0000 7fb0067fc700 1 --2- 172.21.15.16:0/1074678191 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7faffc06ecc0 0x7faffc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:14.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.717+0000 7fb0067fc700 1 -- 172.21.15.16:0/1074678191 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb000090150 con 0x7fb010073a80 2024-04-22T15:59:14.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.717+0000 7fb015777700 1 --2- 172.21.15.16:0/1074678191 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7faffc06ecc0 0x7faffc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:14.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.717+0000 7fb015777700 1 --2- 172.21.15.16:0/1074678191 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7faffc06ecc0 0x7faffc071180 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb00c009c60 tx=0x7fb00c009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:14.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.722+0000 7fb0067fc700 1 -- 172.21.15.16:0/1074678191 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb00005cab0 con 0x7fb010073a80 2024-04-22T15:59:14.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.951+0000 7fb0179db700 1 -- 172.21.15.16:0/1074678191 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0101992d0 con 0x7faffc06ecc0 2024-04-22T15:59:14.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.953+0000 7fb0067fc700 1 -- 172.21.15.16:0/1074678191 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb0101992d0 con 0x7faffc06ecc0 2024-04-22T15:59:14.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.956+0000 7fb0179db700 1 -- 172.21.15.16:0/1074678191 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7faffc06ecc0 msgr2=0x7faffc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:14.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.956+0000 7fb0179db700 1 --2- 172.21.15.16:0/1074678191 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7faffc06ecc0 0x7faffc071180 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb00c009c60 tx=0x7fb00c009260 comp rx=0 tx=0).stop 2024-04-22T15:59:14.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.956+0000 7fb0179db700 1 -- 172.21.15.16:0/1074678191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb010073a80 msgr2=0x7fb01019ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:14.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.956+0000 7fb0179db700 1 --2- 172.21.15.16:0/1074678191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb010073a80 0x7fb01019ed60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb00000b7d0 tx=0x7fb00000bf90 comp rx=0 tx=0).stop 2024-04-22T15:59:14.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.956+0000 7fb0179db700 1 -- 172.21.15.16:0/1074678191 shutdown_connections 2024-04-22T15:59:14.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.956+0000 7fb0179db700 1 --2- 172.21.15.16:0/1074678191 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7faffc06ecc0 0x7faffc071180 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:14.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.956+0000 7fb0179db700 1 --2- 172.21.15.16:0/1074678191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb010073160 0x7fb01019e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:14.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.956+0000 7fb0179db700 1 --2- 172.21.15.16:0/1074678191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb010073a80 0x7fb01019ed60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:14.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.956+0000 7fb0179db700 1 -- 172.21.15.16:0/1074678191 >> 172.21.15.16:0/1074678191 conn(0x7fb0100fc740 msgr2=0x7fb0101077d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:14.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.956+0000 7fb0179db700 1 -- 172.21.15.16:0/1074678191 shutdown_connections 2024-04-22T15:59:14.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:14.956+0000 7fb0179db700 1 -- 172.21.15.16:0/1074678191 wait complete. 2024-04-22T15:59:15.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.063+0000 7fd30d157700 1 -- 172.21.15.16:0/474393317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd308106cd0 msgr2=0x7fd3081070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:15.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.063+0000 7fd30d157700 1 --2- 172.21.15.16:0/474393317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd308106cd0 0x7fd3081070b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8009910 tx=0x7fd2f8009c20 comp rx=0 tx=0).stop 2024-04-22T15:59:15.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.063+0000 7fd30d157700 1 -- 172.21.15.16:0/474393317 shutdown_connections 2024-04-22T15:59:15.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.063+0000 7fd30d157700 1 --2- 172.21.15.16:0/474393317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd308068580 0x7fd3080689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:15.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.063+0000 7fd30d157700 1 --2- 172.21.15.16:0/474393317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd308106cd0 0x7fd3081070b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:15.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.063+0000 7fd30d157700 1 -- 172.21.15.16:0/474393317 >> 172.21.15.16:0/474393317 conn(0x7fd308075670 msgr2=0x7fd308075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:15.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.063+0000 7fd30d157700 1 -- 172.21.15.16:0/474393317 shutdown_connections 2024-04-22T15:59:15.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.063+0000 7fd30d157700 1 -- 172.21.15.16:0/474393317 wait complete. 2024-04-22T15:59:15.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.064+0000 7fd30d157700 1 Processor -- start 2024-04-22T15:59:15.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.064+0000 7fd30d157700 1 -- start start 2024-04-22T15:59:15.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.065+0000 7fd30d157700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd308068580 0x7fd3080ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:15.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.065+0000 7fd30d157700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd308106cd0 0x7fd308100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:15.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.065+0000 7fd30d157700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd308100770 con 0x7fd308068580 2024-04-22T15:59:15.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.065+0000 7fd30d157700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3081008b0 con 0x7fd308106cd0 2024-04-22T15:59:15.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.065+0000 7fd306d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd308068580 0x7fd3080ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:15.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.065+0000 7fd30659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd308106cd0 0x7fd308100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:15.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.065+0000 7fd30659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd308106cd0 0x7fd308100230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:45236/0 (socket says 172.21.15.16:45236) 2024-04-22T15:59:15.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.065+0000 7fd30659c700 1 -- 172.21.15.16:0/1299922539 learned_addr learned my addr 172.21.15.16:0/1299922539 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:59:15.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.065+0000 7fd306d9d700 1 -- 172.21.15.16:0/1299922539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd308106cd0 msgr2=0x7fd308100230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:15.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.065+0000 7fd306d9d700 1 --2- 172.21.15.16:0/1299922539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd308106cd0 0x7fd308100230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:15.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.065+0000 7fd306d9d700 1 -- 172.21.15.16:0/1299922539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f80095f0 con 0x7fd308068580 2024-04-22T15:59:15.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.065+0000 7fd306d9d700 1 --2- 172.21.15.16:0/1299922539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd308068580 0x7fd3080ffcf0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8009fd0 tx=0x7fd2f800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:15.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.065+0000 7fd2effff700 1 -- 172.21.15.16:0/1299922539 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2f801d070 con 0x7fd308068580 2024-04-22T15:59:15.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.066+0000 7fd30d157700 1 -- 172.21.15.16:0/1299922539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd308100b30 con 0x7fd308068580 2024-04-22T15:59:15.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.066+0000 7fd2effff700 1 -- 172.21.15.16:0/1299922539 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd2f80050d0 con 0x7fd308068580 2024-04-22T15:59:15.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.066+0000 7fd2effff700 1 -- 172.21.15.16:0/1299922539 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2f8022470 con 0x7fd308068580 2024-04-22T15:59:15.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.066+0000 7fd30d157700 1 -- 172.21.15.16:0/1299922539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3081a2d90 con 0x7fd308068580 2024-04-22T15:59:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.067+0000 7fd30d157700 1 -- 172.21.15.16:0/1299922539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3081091f0 con 0x7fd308068580 2024-04-22T15:59:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.067+0000 7fd2effff700 1 -- 172.21.15.16:0/1299922539 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd2f8017400 con 0x7fd308068580 2024-04-22T15:59:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.068+0000 7fd2effff700 1 --2- 172.21.15.16:0/1299922539 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd2f006ecc0 0x7fd2f0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.068+0000 7fd2effff700 1 -- 172.21.15.16:0/1299922539 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd2f80902f0 con 0x7fd308068580 2024-04-22T15:59:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.068+0000 7fd30659c700 1 --2- 172.21.15.16:0/1299922539 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd2f006ecc0 0x7fd2f0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.069+0000 7fd30659c700 1 --2- 172.21.15.16:0/1299922539 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd2f006ecc0 0x7fd2f0071180 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd3081014e0 tx=0x7fd2fc009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:15.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.073+0000 7fd2effff700 1 -- 172.21.15.16:0/1299922539 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd2f8059190 con 0x7fd308068580 2024-04-22T15:59:15.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.333+0000 7fd30d157700 1 -- 172.21.15.16:0/1299922539 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd308101330 con 0x7fd2f006ecc0 2024-04-22T15:59:15.336 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:15 smithi016 ceph-mon[110740]: mds.? [v2:172.21.15.27:6826/4073572546,v1:172.21.15.27:6827/4073572546] up:boot 2024-04-22T15:59:15.337 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:15 smithi016 ceph-mon[110740]: fsmap cephfs:1/1 {0=cephfs.smithi016.fkkejm=up:replay} 3 up:standby 2024-04-22T15:59:15.337 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:15 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.dgqyes"}]: dispatch 2024-04-22T15:59:15.337 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:15 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:15.337 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:15 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:15.337 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:15 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:59:15.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.354+0000 7fd2effff700 1 -- 172.21.15.16:0/1299922539 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fd308101330 con 0x7fd2f006ecc0 2024-04-22T15:59:15.357 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:59:15.357 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.smithi016 smithi016 *:9093,9094 running (7m) 39s ago 18m 25.8M - 0.25.0 c8568f914cd2 ea0e0217d6f9 2024-04-22T15:59:15.357 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi016 smithi016 running (18m) 39s ago 18m 8980k - 18.2.1 5be31c24972a f8b627d3f41e 2024-04-22T15:59:15.357 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi027 smithi027 running (17m) 22s ago 17m 8984k - 18.2.1 5be31c24972a f00fd40b4220 2024-04-22T15:59:15.358 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi016 smithi016 running (4m) 39s ago 18m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 1feb008449ed 2024-04-22T15:59:15.358 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi027 smithi027 running (4m) 22s ago 17m 6656k - 19.0.0-3223-g5883d494 3d983c26e278 c7f862c6da00 2024-04-22T15:59:15.358 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.smithi016 smithi016 *:3000 running (7m) 39s ago 17m 81.9M - 9.4.12 2bacad6d85d8 388a3c9c1b3f 2024-04-22T15:59:15.358 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.fkkejm smithi016 running (61s) 39s ago 12m 16.0M - 19.0.0-3223-g5883d494 3d983c26e278 e23d9fc30059 2024-04-22T15:59:15.358 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.pgvphu smithi016 running (44s) 39s ago 12m 11.6M - 19.0.0-3223-g5883d494 3d983c26e278 4777936b5956 2024-04-22T15:59:15.358 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.dgqyes smithi027 starting - - - - 2024-04-22T15:59:15.358 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.iobjds smithi027 running (26s) 22s ago 12m 14.0M - 19.0.0-3223-g5883d494 3d983c26e278 8dd739415c15 2024-04-22T15:59:15.358 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi016.rnsnnr smithi016 *:8443,9283,8765 running (9m) 39s ago 19m 510M - 19.0.0-3223-g5883d494 3d983c26e278 09b4fe285bc4 2024-04-22T15:59:15.358 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi027.ukdgzs smithi027 *:8443,9283,8765 running (8m) 22s ago 17m 451M - 19.0.0-3223-g5883d494 3d983c26e278 7548da9b5eb9 2024-04-22T15:59:15.358 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi016 smithi016 running (5m) 39s ago 19m 54.8M 2048M 19.0.0-3223-g5883d494 3d983c26e278 4ae529a67806 2024-04-22T15:59:15.359 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi027 smithi027 running (5m) 22s ago 17m 47.3M 2048M 19.0.0-3223-g5883d494 3d983c26e278 f29fcb888801 2024-04-22T15:59:15.359 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi016 smithi016 *:9100 running (8m) 39s ago 18m 15.5M - 1.5.0 0da6a335fe13 35e46e4d2f3f 2024-04-22T15:59:15.359 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi027 smithi027 *:9100 running (8m) 22s ago 17m 15.3M - 1.5.0 0da6a335fe13 e682a7672e9b 2024-04-22T15:59:15.359 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (4m) 39s ago 16m 92.0M 1217M 19.0.0-3223-g5883d494 3d983c26e278 79217eb383a2 2024-04-22T15:59:15.359 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (3m) 39s ago 15m 92.0M 1217M 19.0.0-3223-g5883d494 3d983c26e278 ec1a45b3c53e 2024-04-22T15:59:15.359 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (3m) 39s ago 15m 77.8M 1217M 19.0.0-3223-g5883d494 3d983c26e278 48bb459fd021 2024-04-22T15:59:15.359 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi027 running (2m) 22s ago 15m 106M 2240M 19.0.0-3223-g5883d494 3d983c26e278 81034bf22d3f 2024-04-22T15:59:15.359 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (114s) 22s ago 14m 94.6M 2240M 19.0.0-3223-g5883d494 3d983c26e278 93b660e5ecb6 2024-04-22T15:59:15.359 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi027 running (80s) 22s ago 14m 92.5M 2240M 19.0.0-3223-g5883d494 3d983c26e278 d857c152994a 2024-04-22T15:59:15.360 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.smithi016 smithi016 *:9095 running (7m) 39s ago 17m 76.0M - 2.43.0 a07b618ecd1d 373eaadee82d 2024-04-22T15:59:15.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.356+0000 7fd30d157700 1 -- 172.21.15.16:0/1299922539 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd2f006ecc0 msgr2=0x7fd2f0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:15.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.356+0000 7fd30d157700 1 --2- 172.21.15.16:0/1299922539 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd2f006ecc0 0x7fd2f0071180 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd3081014e0 tx=0x7fd2fc009260 comp rx=0 tx=0).stop 2024-04-22T15:59:15.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.356+0000 7fd30d157700 1 -- 172.21.15.16:0/1299922539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd308068580 msgr2=0x7fd3080ffcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:15.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.356+0000 7fd30d157700 1 --2- 172.21.15.16:0/1299922539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd308068580 0x7fd3080ffcf0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8009fd0 tx=0x7fd2f800f690 comp rx=0 tx=0).stop 2024-04-22T15:59:15.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.356+0000 7fd30d157700 1 -- 172.21.15.16:0/1299922539 shutdown_connections 2024-04-22T15:59:15.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.356+0000 7fd30d157700 1 --2- 172.21.15.16:0/1299922539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd308068580 0x7fd3080ffcf0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:15.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.356+0000 7fd30d157700 1 --2- 172.21.15.16:0/1299922539 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fd2f006ecc0 0x7fd2f0071180 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:15.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.356+0000 7fd30d157700 1 --2- 172.21.15.16:0/1299922539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd308106cd0 0x7fd308100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:15.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.356+0000 7fd30d157700 1 -- 172.21.15.16:0/1299922539 >> 172.21.15.16:0/1299922539 conn(0x7fd308075670 msgr2=0x7fd3080fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:15.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.356+0000 7fd30d157700 1 -- 172.21.15.16:0/1299922539 shutdown_connections 2024-04-22T15:59:15.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.356+0000 7fd30d157700 1 -- 172.21.15.16:0/1299922539 wait complete. 2024-04-22T15:59:15.398 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:15 smithi027 ceph-mon[83054]: mds.? [v2:172.21.15.27:6826/4073572546,v1:172.21.15.27:6827/4073572546] up:boot 2024-04-22T15:59:15.399 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:15 smithi027 ceph-mon[83054]: fsmap cephfs:1/1 {0=cephfs.smithi016.fkkejm=up:replay} 3 up:standby 2024-04-22T15:59:15.399 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:15 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.dgqyes"}]: dispatch 2024-04-22T15:59:15.399 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:15 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:15.399 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:15 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:15.399 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:15 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:59:15.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.463+0000 7f4704b87700 1 -- 172.21.15.16:0/1295938415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4700106cd0 msgr2=0x7f47001070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:15.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.463+0000 7f4704b87700 1 --2- 172.21.15.16:0/1295938415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4700106cd0 0x7f47001070b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f46f0009910 tx=0x7f46f0009c20 comp rx=0 tx=0).stop 2024-04-22T15:59:15.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.463+0000 7f4704b87700 1 -- 172.21.15.16:0/1295938415 shutdown_connections 2024-04-22T15:59:15.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.463+0000 7f4704b87700 1 --2- 172.21.15.16:0/1295938415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4700068580 0x7f47000689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:15.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.463+0000 7f4704b87700 1 --2- 172.21.15.16:0/1295938415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4700106cd0 0x7f47001070b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:15.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.463+0000 7f4704b87700 1 -- 172.21.15.16:0/1295938415 >> 172.21.15.16:0/1295938415 conn(0x7f4700075670 msgr2=0x7f4700075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:15.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.463+0000 7f4704b87700 1 -- 172.21.15.16:0/1295938415 shutdown_connections 2024-04-22T15:59:15.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.463+0000 7f4704b87700 1 -- 172.21.15.16:0/1295938415 wait complete. 2024-04-22T15:59:15.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.464+0000 7f4704b87700 1 Processor -- start 2024-04-22T15:59:15.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.464+0000 7f4704b87700 1 -- start start 2024-04-22T15:59:15.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.464+0000 7f4704b87700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4700068580 0x7f470019e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:15.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.464+0000 7f4704b87700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4700106cd0 0x7f470019edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:15.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.464+0000 7f4704b87700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470019f3c0 con 0x7f4700068580 2024-04-22T15:59:15.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.464+0000 7f4704b87700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4700198900 con 0x7f4700106cd0 2024-04-22T15:59:15.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.464+0000 7f46fe59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4700068580 0x7f470019e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:15.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.464+0000 7f46fe59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4700068580 0x7f470019e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45980/0 (socket says 172.21.15.16:45980) 2024-04-22T15:59:15.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.464+0000 7f46fe59c700 1 -- 172.21.15.16:0/3473669854 learned_addr learned my addr 172.21.15.16:0/3473669854 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:59:15.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.464+0000 7f46fdd9b700 1 --2- 172.21.15.16:0/3473669854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4700106cd0 0x7f470019edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:15.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.465+0000 7f46fe59c700 1 -- 172.21.15.16:0/3473669854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4700106cd0 msgr2=0x7f470019edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:15.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.465+0000 7f46fe59c700 1 --2- 172.21.15.16:0/3473669854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4700106cd0 0x7f470019edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:15.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.465+0000 7f46fe59c700 1 -- 172.21.15.16:0/3473669854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f00095f0 con 0x7f4700068580 2024-04-22T15:59:15.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.465+0000 7f46fe59c700 1 --2- 172.21.15.16:0/3473669854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4700068580 0x7f470019e880 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f46f0006010 tx=0x7f46f000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:15.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.465+0000 7f46ef7fe700 1 -- 172.21.15.16:0/3473669854 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46f001d070 con 0x7f4700068580 2024-04-22T15:59:15.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.465+0000 7f4704b87700 1 -- 172.21.15.16:0/3473669854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4700198b80 con 0x7f4700068580 2024-04-22T15:59:15.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.465+0000 7f4704b87700 1 -- 172.21.15.16:0/3473669854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4700199070 con 0x7f4700068580 2024-04-22T15:59:15.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.465+0000 7f46ef7fe700 1 -- 172.21.15.16:0/3473669854 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f46f000f460 con 0x7f4700068580 2024-04-22T15:59:15.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.465+0000 7f46ef7fe700 1 -- 172.21.15.16:0/3473669854 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46f0017630 con 0x7f4700068580 2024-04-22T15:59:15.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.466+0000 7f4704b87700 1 -- 172.21.15.16:0/3473669854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46e0005320 con 0x7f4700068580 2024-04-22T15:59:15.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.467+0000 7f46ef7fe700 1 -- 172.21.15.16:0/3473669854 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f46f0003a60 con 0x7f4700068580 2024-04-22T15:59:15.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.467+0000 7f46ef7fe700 1 --2- 172.21.15.16:0/3473669854 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f46e806ecc0 0x7f46e8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:15.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.468+0000 7f46ef7fe700 1 -- 172.21.15.16:0/3473669854 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f46f00904e0 con 0x7f4700068580 2024-04-22T15:59:15.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.468+0000 7f46fdd9b700 1 --2- 172.21.15.16:0/3473669854 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f46e806ecc0 0x7f46e8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:15.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.468+0000 7f46fdd9b700 1 --2- 172.21.15.16:0/3473669854 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f46e806ecc0 0x7f46e8071180 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f47000fe270 tx=0x7f46f400a380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:15.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.476+0000 7f46ef7fe700 1 -- 172.21.15.16:0/3473669854 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f46f005ce20 con 0x7f4700068580 2024-04-22T15:59:15.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.778+0000 7f4704b87700 1 -- 172.21.15.16:0/3473669854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f46e0005cc0 con 0x7f4700068580 2024-04-22T15:59:15.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.778+0000 7f46ef7fe700 1 -- 172.21.15.16:0/3473669854 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f46f005c9b0 con 0x7f4700068580 2024-04-22T15:59:15.781 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:59:15.781 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-22T15:59:15.781 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:59:15.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:59:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-22T15:59:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:59:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:59:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-22T15:59:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:59:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:59:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-22T15:59:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:59:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:59:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-22T15:59:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:59:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-22T15:59:15.782 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:59:15.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.781+0000 7f4704b87700 1 -- 172.21.15.16:0/3473669854 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f46e806ecc0 msgr2=0x7f46e8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:15.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.781+0000 7f4704b87700 1 --2- 172.21.15.16:0/3473669854 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f46e806ecc0 0x7f46e8071180 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f47000fe270 tx=0x7f46f400a380 comp rx=0 tx=0).stop 2024-04-22T15:59:15.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.781+0000 7f4704b87700 1 -- 172.21.15.16:0/3473669854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4700068580 msgr2=0x7f470019e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:15.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.781+0000 7f4704b87700 1 --2- 172.21.15.16:0/3473669854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4700068580 0x7f470019e880 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f46f0006010 tx=0x7f46f000bd70 comp rx=0 tx=0).stop 2024-04-22T15:59:15.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.781+0000 7f4704b87700 1 -- 172.21.15.16:0/3473669854 shutdown_connections 2024-04-22T15:59:15.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.781+0000 7f4704b87700 1 --2- 172.21.15.16:0/3473669854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4700068580 0x7f470019e880 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:15.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.781+0000 7f4704b87700 1 --2- 172.21.15.16:0/3473669854 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f46e806ecc0 0x7f46e8071180 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:15.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.781+0000 7f4704b87700 1 --2- 172.21.15.16:0/3473669854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4700106cd0 0x7f470019edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:15.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.781+0000 7f4704b87700 1 -- 172.21.15.16:0/3473669854 >> 172.21.15.16:0/3473669854 conn(0x7f4700075670 msgr2=0x7f47000febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:15.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.781+0000 7f4704b87700 1 -- 172.21.15.16:0/3473669854 shutdown_connections 2024-04-22T15:59:15.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.781+0000 7f4704b87700 1 -- 172.21.15.16:0/3473669854 wait complete. 2024-04-22T15:59:15.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.890+0000 7f96ceb86700 1 -- 172.21.15.16:0/947013547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96c8068580 msgr2=0x7f96c80689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:15.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.890+0000 7f96ceb86700 1 --2- 172.21.15.16:0/947013547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96c8068580 0x7f96c80689e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f96b4009910 tx=0x7f96b4009c20 comp rx=0 tx=0).stop 2024-04-22T15:59:15.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.890+0000 7f96ceb86700 1 -- 172.21.15.16:0/947013547 shutdown_connections 2024-04-22T15:59:15.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.890+0000 7f96ceb86700 1 --2- 172.21.15.16:0/947013547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96c8068580 0x7f96c80689e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:15.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.890+0000 7f96ceb86700 1 --2- 172.21.15.16:0/947013547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96c8106cd0 0x7f96c81070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:15.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.890+0000 7f96ceb86700 1 -- 172.21.15.16:0/947013547 >> 172.21.15.16:0/947013547 conn(0x7f96c8075670 msgr2=0x7f96c8075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:15.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.890+0000 7f96ceb86700 1 -- 172.21.15.16:0/947013547 shutdown_connections 2024-04-22T15:59:15.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.890+0000 7f96ceb86700 1 -- 172.21.15.16:0/947013547 wait complete. 2024-04-22T15:59:15.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.891+0000 7f96ceb86700 1 Processor -- start 2024-04-22T15:59:15.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.891+0000 7f96ceb86700 1 -- start start 2024-04-22T15:59:15.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.891+0000 7f96ceb86700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96c8068580 0x7f96c819c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:15.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.891+0000 7f96ceb86700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96c8106cd0 0x7f96c819cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.891+0000 7f96ceb86700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c819d200 con 0x7f96c8106cd0 2024-04-22T15:59:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.891+0000 7f96ceb86700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c81966b0 con 0x7f96c8068580 2024-04-22T15:59:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.891+0000 7f96c7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96c8106cd0 0x7f96c819cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.891+0000 7f96cc922700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96c8068580 0x7f96c819c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.891+0000 7f96c7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96c8106cd0 0x7f96c819cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45994/0 (socket says 172.21.15.16:45994) 2024-04-22T15:59:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.891+0000 7f96c7fff700 1 -- 172.21.15.16:0/3753679055 learned_addr learned my addr 172.21.15.16:0/3753679055 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:59:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.892+0000 7f96c7fff700 1 -- 172.21.15.16:0/3753679055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96c8068580 msgr2=0x7f96c819c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.892+0000 7f96c7fff700 1 --2- 172.21.15.16:0/3753679055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96c8068580 0x7f96c819c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.892+0000 7f96c7fff700 1 -- 172.21.15.16:0/3753679055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b40095f0 con 0x7f96c8106cd0 2024-04-22T15:59:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.892+0000 7f96c7fff700 1 --2- 172.21.15.16:0/3753679055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96c8106cd0 0x7f96c819cb70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f96b4005f50 tx=0x7f96b400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.892+0000 7f96c5ffb700 1 -- 172.21.15.16:0/3753679055 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96b401d070 con 0x7f96c8106cd0 2024-04-22T15:59:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.892+0000 7f96ceb86700 1 -- 172.21.15.16:0/3753679055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c8196930 con 0x7f96c8106cd0 2024-04-22T15:59:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.892+0000 7f96c5ffb700 1 -- 172.21.15.16:0/3753679055 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96b400b610 con 0x7f96c8106cd0 2024-04-22T15:59:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.892+0000 7f96c5ffb700 1 -- 172.21.15.16:0/3753679055 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96b4022470 con 0x7f96c8106cd0 2024-04-22T15:59:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.892+0000 7f96ceb86700 1 -- 172.21.15.16:0/3753679055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c8196e20 con 0x7f96c8106cd0 2024-04-22T15:59:15.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.893+0000 7f96ceb86700 1 -- 172.21.15.16:0/3753679055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c81091d0 con 0x7f96c8106cd0 2024-04-22T15:59:15.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.894+0000 7f96c5ffb700 1 -- 172.21.15.16:0/3753679055 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f96b4003780 con 0x7f96c8106cd0 2024-04-22T15:59:15.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.895+0000 7f96c5ffb700 1 --2- 172.21.15.16:0/3753679055 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f96b80730d0 0x7f96b8075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:15.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.895+0000 7f96c5ffb700 1 -- 172.21.15.16:0/3753679055 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f96b4090110 con 0x7f96c8106cd0 2024-04-22T15:59:15.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.895+0000 7f96cc922700 1 --2- 172.21.15.16:0/3753679055 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f96b80730d0 0x7f96b8075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:15.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.896+0000 7f96cc922700 1 --2- 172.21.15.16:0/3753679055 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f96b80730d0 0x7f96b8075590 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f96bc009c60 tx=0x7f96bc009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:15.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:15.904+0000 7f96c5ffb700 1 -- 172.21.15.16:0/3753679055 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f96b405ca70 con 0x7f96c8106cd0 2024-04-22T15:59:16.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.180+0000 7f96ceb86700 1 -- 172.21.15.16:0/3753679055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f96c8197c90 con 0x7f96c8106cd0 2024-04-22T15:59:16.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.180+0000 7f96c5ffb700 1 -- 172.21.15.16:0/3753679055 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1925 (secure 0 0 0) 0x7f96b405c600 con 0x7f96c8106cd0 2024-04-22T15:59:16.183 INFO:teuthology.orchestra.run.smithi016.stdout:e31 2024-04-22T15:59:16.183 INFO:teuthology.orchestra.run.smithi016.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T15:59:16.184 INFO:teuthology.orchestra.run.smithi016.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T15:59:16.184 INFO:teuthology.orchestra.run.smithi016.stdout:legacy client fscid: 1 2024-04-22T15:59:16.184 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:59:16.184 INFO:teuthology.orchestra.run.smithi016.stdout:Filesystem 'cephfs' (1) 2024-04-22T15:59:16.184 INFO:teuthology.orchestra.run.smithi016.stdout:fs_name cephfs 2024-04-22T15:59:16.184 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 30 2024-04-22T15:59:16.184 INFO:teuthology.orchestra.run.smithi016.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-22T15:59:16.184 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-04-22T15:46:27.774571+0000 2024-04-22T15:59:16.184 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-04-22T15:59:06.186147+0000 2024-04-22T15:59:16.184 INFO:teuthology.orchestra.run.smithi016.stdout:tableserver 0 2024-04-22T15:59:16.184 INFO:teuthology.orchestra.run.smithi016.stdout:root 0 2024-04-22T15:59:16.184 INFO:teuthology.orchestra.run.smithi016.stdout:session_timeout 60 2024-04-22T15:59:16.184 INFO:teuthology.orchestra.run.smithi016.stdout:session_autoclose 300 2024-04-22T15:59:16.185 INFO:teuthology.orchestra.run.smithi016.stdout:max_file_size 1099511627776 2024-04-22T15:59:16.185 INFO:teuthology.orchestra.run.smithi016.stdout:max_xattr_size 65536 2024-04-22T15:59:16.185 INFO:teuthology.orchestra.run.smithi016.stdout:required_client_features {} 2024-04-22T15:59:16.185 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure 0 2024-04-22T15:59:16.185 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure_osd_epoch 84 2024-04-22T15:59:16.185 INFO:teuthology.orchestra.run.smithi016.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T15:59:16.185 INFO:teuthology.orchestra.run.smithi016.stdout:max_mds 1 2024-04-22T15:59:16.185 INFO:teuthology.orchestra.run.smithi016.stdout:in 0 2024-04-22T15:59:16.185 INFO:teuthology.orchestra.run.smithi016.stdout:up {0=34408} 2024-04-22T15:59:16.185 INFO:teuthology.orchestra.run.smithi016.stdout:failed 2024-04-22T15:59:16.185 INFO:teuthology.orchestra.run.smithi016.stdout:damaged 2024-04-22T15:59:16.185 INFO:teuthology.orchestra.run.smithi016.stdout:stopped 2024-04-22T15:59:16.185 INFO:teuthology.orchestra.run.smithi016.stdout:data_pools [3] 2024-04-22T15:59:16.186 INFO:teuthology.orchestra.run.smithi016.stdout:metadata_pool 2 2024-04-22T15:59:16.186 INFO:teuthology.orchestra.run.smithi016.stdout:inline_data enabled 2024-04-22T15:59:16.186 INFO:teuthology.orchestra.run.smithi016.stdout:balancer 2024-04-22T15:59:16.186 INFO:teuthology.orchestra.run.smithi016.stdout:bal_rank_mask -1 2024-04-22T15:59:16.186 INFO:teuthology.orchestra.run.smithi016.stdout:standby_count_wanted 1 2024-04-22T15:59:16.186 INFO:teuthology.orchestra.run.smithi016.stdout:qdb_cluster leader: 0 members: 2024-04-22T15:59:16.186 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.fkkejm{0:34408} state up:replay seq 1 join_fscid=1 addr [v2:172.21.15.16:6826/2871308685,v1:172.21.15.16:6827/2871308685] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:59:16.186 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:59:16.186 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:59:16.186 INFO:teuthology.orchestra.run.smithi016.stdout:Standby daemons: 2024-04-22T15:59:16.186 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T15:59:16.186 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.pgvphu{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.16:6828/4228489800,v1:172.21.15.16:6829/4228489800] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:59:16.186 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.iobjds{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6824/3921606817,v1:172.21.15.27:6825/3921606817] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:59:16.187 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.dgqyes{-1:44333} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6826/4073572546,v1:172.21.15.27:6827/4073572546] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T15:59:16.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.182+0000 7f96ceb86700 1 -- 172.21.15.16:0/3753679055 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f96b80730d0 msgr2=0x7f96b8075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:16.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.182+0000 7f96ceb86700 1 --2- 172.21.15.16:0/3753679055 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f96b80730d0 0x7f96b8075590 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f96bc009c60 tx=0x7f96bc009260 comp rx=0 tx=0).stop 2024-04-22T15:59:16.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.182+0000 7f96ceb86700 1 -- 172.21.15.16:0/3753679055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96c8106cd0 msgr2=0x7f96c819cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:16.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.182+0000 7f96ceb86700 1 --2- 172.21.15.16:0/3753679055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96c8106cd0 0x7f96c819cb70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f96b4005f50 tx=0x7f96b400f690 comp rx=0 tx=0).stop 2024-04-22T15:59:16.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.182+0000 7f96ceb86700 1 -- 172.21.15.16:0/3753679055 shutdown_connections 2024-04-22T15:59:16.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.182+0000 7f96ceb86700 1 --2- 172.21.15.16:0/3753679055 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f96b80730d0 0x7f96b8075590 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:16.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.182+0000 7f96ceb86700 1 --2- 172.21.15.16:0/3753679055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96c8068580 0x7f96c819c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:16.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.182+0000 7f96ceb86700 1 --2- 172.21.15.16:0/3753679055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96c8106cd0 0x7f96c819cb70 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:16.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.182+0000 7f96ceb86700 1 -- 172.21.15.16:0/3753679055 >> 172.21.15.16:0/3753679055 conn(0x7f96c8075670 msgr2=0x7f96c80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:16.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.182+0000 7f96ceb86700 1 -- 172.21.15.16:0/3753679055 shutdown_connections 2024-04-22T15:59:16.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.182+0000 7f96ceb86700 1 -- 172.21.15.16:0/3753679055 wait complete. 2024-04-22T15:59:16.188 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 31 2024-04-22T15:59:16.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.293+0000 7f125ee61700 1 -- 172.21.15.16:0/1288020639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12580ffda0 msgr2=0x7f1258100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:16.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.293+0000 7f125ee61700 1 --2- 172.21.15.16:0/1288020639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12580ffda0 0x7f1258100180 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f124c009910 tx=0x7f124c009c20 comp rx=0 tx=0).stop 2024-04-22T15:59:16.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.293+0000 7f125ee61700 1 -- 172.21.15.16:0/1288020639 shutdown_connections 2024-04-22T15:59:16.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.293+0000 7f125ee61700 1 --2- 172.21.15.16:0/1288020639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1258100750 0x7f12581047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:16.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.293+0000 7f125ee61700 1 --2- 172.21.15.16:0/1288020639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12580ffda0 0x7f1258100180 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:16.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.293+0000 7f125ee61700 1 -- 172.21.15.16:0/1288020639 >> 172.21.15.16:0/1288020639 conn(0x7f12580fb650 msgr2=0x7f12580fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:16.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.293+0000 7f125ee61700 1 -- 172.21.15.16:0/1288020639 shutdown_connections 2024-04-22T15:59:16.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.293+0000 7f125ee61700 1 -- 172.21.15.16:0/1288020639 wait complete. 2024-04-22T15:59:16.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.294+0000 7f125ee61700 1 Processor -- start 2024-04-22T15:59:16.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.294+0000 7f125ee61700 1 -- start start 2024-04-22T15:59:16.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.294+0000 7f125ee61700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12580ffda0 0x7f125819c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:16.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.294+0000 7f125ee61700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1258100750 0x7f125819cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:16.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.294+0000 7f125ee61700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125819d2c0 con 0x7f12580ffda0 2024-04-22T15:59:16.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.294+0000 7f125ee61700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1258196770 con 0x7f1258100750 2024-04-22T15:59:16.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.294+0000 7f125cbfd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12580ffda0 0x7f125819c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:16.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.294+0000 7f125cbfd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12580ffda0 0x7f125819c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46008/0 (socket says 172.21.15.16:46008) 2024-04-22T15:59:16.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.294+0000 7f125cbfd700 1 -- 172.21.15.16:0/793268346 learned_addr learned my addr 172.21.15.16:0/793268346 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:59:16.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.294+0000 7f1257fff700 1 --2- 172.21.15.16:0/793268346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1258100750 0x7f125819cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:16.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.295+0000 7f125cbfd700 1 -- 172.21.15.16:0/793268346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1258100750 msgr2=0x7f125819cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:16.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.295+0000 7f125cbfd700 1 --2- 172.21.15.16:0/793268346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1258100750 0x7f125819cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:16.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.295+0000 7f125cbfd700 1 -- 172.21.15.16:0/793268346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f124c0095f0 con 0x7f12580ffda0 2024-04-22T15:59:16.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.295+0000 7f125cbfd700 1 --2- 172.21.15.16:0/793268346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12580ffda0 0x7f125819c6f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f124c006010 tx=0x7f124c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:16.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.295+0000 7f1255ffb700 1 -- 172.21.15.16:0/793268346 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f124c01d070 con 0x7f12580ffda0 2024-04-22T15:59:16.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.295+0000 7f125ee61700 1 -- 172.21.15.16:0/793268346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12581969f0 con 0x7f12580ffda0 2024-04-22T15:59:16.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.295+0000 7f1255ffb700 1 -- 172.21.15.16:0/793268346 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f124c005dc0 con 0x7f12580ffda0 2024-04-22T15:59:16.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.295+0000 7f1255ffb700 1 -- 172.21.15.16:0/793268346 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f124c017400 con 0x7f12580ffda0 2024-04-22T15:59:16.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.295+0000 7f125ee61700 1 -- 172.21.15.16:0/793268346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1258196ee0 con 0x7f12580ffda0 2024-04-22T15:59:16.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.296+0000 7f125ee61700 1 -- 172.21.15.16:0/793268346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f125804f2e0 con 0x7f12580ffda0 2024-04-22T15:59:16.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.297+0000 7f1255ffb700 1 -- 172.21.15.16:0/793268346 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f124c003a60 con 0x7f12580ffda0 2024-04-22T15:59:16.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.297+0000 7f1255ffb700 1 --2- 172.21.15.16:0/793268346 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f124806ecc0 0x7f1248071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:16.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.297+0000 7f1255ffb700 1 -- 172.21.15.16:0/793268346 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f124c090200 con 0x7f12580ffda0 2024-04-22T15:59:16.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.297+0000 7f1257fff700 1 --2- 172.21.15.16:0/793268346 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f124806ecc0 0x7f1248071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:16.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.298+0000 7f1257fff700 1 --2- 172.21.15.16:0/793268346 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f124806ecc0 0x7f1248071180 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1258197be0 tx=0x7f124400b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:16.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.302+0000 7f1255ffb700 1 -- 172.21.15.16:0/793268346 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f124c05cb60 con 0x7f12580ffda0 2024-04-22T15:59:16.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:16 smithi027 ceph-mon[83054]: pgmap v210: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 0 B/s wr, 6 op/s 2024-04-22T15:59:16.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:16 smithi027 ceph-mon[83054]: from='client.44337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:59:16.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:16 smithi027 ceph-mon[83054]: from='client.34454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:59:16.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:16 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:16.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:16 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:59:16.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:16 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/3473669854' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:16.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:16 smithi016 ceph-mon[110740]: pgmap v210: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 0 B/s wr, 6 op/s 2024-04-22T15:59:16.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:16 smithi016 ceph-mon[110740]: from='client.44337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:59:16.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:16 smithi016 ceph-mon[110740]: from='client.34454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:59:16.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:16 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:16.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:16 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:59:16.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:16 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/3473669854' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:16.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.532+0000 7f125ee61700 1 -- 172.21.15.16:0/793268346 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1258197210 con 0x7f124806ecc0 2024-04-22T15:59:16.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.533+0000 7f1255ffb700 1 -- 172.21.15.16:0/793268346 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1258197210 con 0x7f124806ecc0 2024-04-22T15:59:16.536 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:59:16.536 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb", 2024-04-22T15:59:16.536 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-22T15:59:16.537 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T15:59:16.537 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-22T15:59:16.537 INFO:teuthology.orchestra.run.smithi016.stdout: "osd", 2024-04-22T15:59:16.537 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-04-22T15:59:16.537 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr", 2024-04-22T15:59:16.537 INFO:teuthology.orchestra.run.smithi016.stdout: "mon" 2024-04-22T15:59:16.537 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-22T15:59:16.537 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T15:59:16.537 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T15:59:16.537 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-22T15:59:16.537 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:59:16.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.535+0000 7f125ee61700 1 -- 172.21.15.16:0/793268346 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f124806ecc0 msgr2=0x7f1248071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:16.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.535+0000 7f125ee61700 1 --2- 172.21.15.16:0/793268346 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f124806ecc0 0x7f1248071180 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1258197be0 tx=0x7f124400b410 comp rx=0 tx=0).stop 2024-04-22T15:59:16.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.535+0000 7f125ee61700 1 -- 172.21.15.16:0/793268346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12580ffda0 msgr2=0x7f125819c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:16.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.535+0000 7f125ee61700 1 --2- 172.21.15.16:0/793268346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12580ffda0 0x7f125819c6f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f124c006010 tx=0x7f124c00bd70 comp rx=0 tx=0).stop 2024-04-22T15:59:16.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.535+0000 7f125ee61700 1 -- 172.21.15.16:0/793268346 shutdown_connections 2024-04-22T15:59:16.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.535+0000 7f125ee61700 1 --2- 172.21.15.16:0/793268346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12580ffda0 0x7f125819c6f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:16.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.535+0000 7f125ee61700 1 --2- 172.21.15.16:0/793268346 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f124806ecc0 0x7f1248071180 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:16.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.535+0000 7f125ee61700 1 --2- 172.21.15.16:0/793268346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1258100750 0x7f125819cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:16.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.535+0000 7f125ee61700 1 -- 172.21.15.16:0/793268346 >> 172.21.15.16:0/793268346 conn(0x7f12580fb650 msgr2=0x7f1258103fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:16.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.535+0000 7f125ee61700 1 -- 172.21.15.16:0/793268346 shutdown_connections 2024-04-22T15:59:16.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.535+0000 7f125ee61700 1 -- 172.21.15.16:0/793268346 wait complete. 2024-04-22T15:59:16.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.642+0000 7fb58f3ed700 1 -- 172.21.15.16:0/778487416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb588100750 msgr2=0x7fb5881047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:16.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.642+0000 7fb58f3ed700 1 --2- 172.21.15.16:0/778487416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb588100750 0x7fb5881047a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb584009910 tx=0x7fb584009c20 comp rx=0 tx=0).stop 2024-04-22T15:59:16.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.642+0000 7fb58f3ed700 1 -- 172.21.15.16:0/778487416 shutdown_connections 2024-04-22T15:59:16.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.642+0000 7fb58f3ed700 1 --2- 172.21.15.16:0/778487416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb588100750 0x7fb5881047a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:16.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.642+0000 7fb58f3ed700 1 --2- 172.21.15.16:0/778487416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5880ffda0 0x7fb588100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:16.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.642+0000 7fb58f3ed700 1 -- 172.21.15.16:0/778487416 >> 172.21.15.16:0/778487416 conn(0x7fb5880fb650 msgr2=0x7fb5880fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:16.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.642+0000 7fb58f3ed700 1 -- 172.21.15.16:0/778487416 shutdown_connections 2024-04-22T15:59:16.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.642+0000 7fb58f3ed700 1 -- 172.21.15.16:0/778487416 wait complete. 2024-04-22T15:59:16.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.643+0000 7fb58f3ed700 1 Processor -- start 2024-04-22T15:59:16.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.643+0000 7fb58f3ed700 1 -- start start 2024-04-22T15:59:16.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.643+0000 7fb58f3ed700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5880ffda0 0x7fb58819c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:16.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.643+0000 7fb58f3ed700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb588100750 0x7fb58819cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:16.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.643+0000 7fb58f3ed700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb58819d240 con 0x7fb5880ffda0 2024-04-22T15:59:16.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.643+0000 7fb58f3ed700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5881966f0 con 0x7fb588100750 2024-04-22T15:59:16.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.643+0000 7fb58d189700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5880ffda0 0x7fb58819c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:16.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.643+0000 7fb58d189700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5880ffda0 0x7fb58819c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46036/0 (socket says 172.21.15.16:46036) 2024-04-22T15:59:16.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.643+0000 7fb58d189700 1 -- 172.21.15.16:0/2749312129 learned_addr learned my addr 172.21.15.16:0/2749312129 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:59:16.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.643+0000 7fb58c988700 1 --2- 172.21.15.16:0/2749312129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb588100750 0x7fb58819cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:16.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.643+0000 7fb58d189700 1 -- 172.21.15.16:0/2749312129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb588100750 msgr2=0x7fb58819cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:16.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.643+0000 7fb58d189700 1 --2- 172.21.15.16:0/2749312129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb588100750 0x7fb58819cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:16.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.643+0000 7fb58d189700 1 -- 172.21.15.16:0/2749312129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5840095f0 con 0x7fb5880ffda0 2024-04-22T15:59:16.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.644+0000 7fb58d189700 1 --2- 172.21.15.16:0/2749312129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5880ffda0 0x7fb58819c670 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb57800d8d0 tx=0x7fb57800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:16.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.644+0000 7fb57e7fc700 1 -- 172.21.15.16:0/2749312129 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5780098a0 con 0x7fb5880ffda0 2024-04-22T15:59:16.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.644+0000 7fb57e7fc700 1 -- 172.21.15.16:0/2749312129 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb578010460 con 0x7fb5880ffda0 2024-04-22T15:59:16.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.644+0000 7fb58f3ed700 1 -- 172.21.15.16:0/2749312129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5881969d0 con 0x7fb5880ffda0 2024-04-22T15:59:16.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.644+0000 7fb57e7fc700 1 -- 172.21.15.16:0/2749312129 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb57800f5f0 con 0x7fb5880ffda0 2024-04-22T15:59:16.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.644+0000 7fb58f3ed700 1 -- 172.21.15.16:0/2749312129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb588196f20 con 0x7fb5880ffda0 2024-04-22T15:59:16.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.645+0000 7fb58f3ed700 1 -- 172.21.15.16:0/2749312129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5881091b0 con 0x7fb5880ffda0 2024-04-22T15:59:16.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.645+0000 7fb57e7fc700 1 -- 172.21.15.16:0/2749312129 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb578009a00 con 0x7fb5880ffda0 2024-04-22T15:59:16.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.646+0000 7fb57e7fc700 1 --2- 172.21.15.16:0/2749312129 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fb57406ecc0 0x7fb574071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:16.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.646+0000 7fb57e7fc700 1 -- 172.21.15.16:0/2749312129 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb57808edb0 con 0x7fb5880ffda0 2024-04-22T15:59:16.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.646+0000 7fb58c988700 1 --2- 172.21.15.16:0/2749312129 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fb57406ecc0 0x7fb574071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:16.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.646+0000 7fb58c988700 1 --2- 172.21.15.16:0/2749312129 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fb57406ecc0 0x7fb574071180 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb588197b60 tx=0x7fb58401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:16.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.651+0000 7fb57e7fc700 1 -- 172.21.15.16:0/2749312129 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb57805b5f0 con 0x7fb5880ffda0 2024-04-22T15:59:16.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.947+0000 7fb58f3ed700 1 -- 172.21.15.16:0/2749312129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb58804f2e0 con 0x7fb5880ffda0 2024-04-22T15:59:16.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.948+0000 7fb57e7fc700 1 -- 172.21.15.16:0/2749312129 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+297 (secure 0 0 0) 0x7fb57805b180 con 0x7fb5880ffda0 2024-04-22T15:59:16.951 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data 2024-04-22T15:59:16.951 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T15:59:16.951 INFO:teuthology.orchestra.run.smithi016.stdout: fs cephfs is degraded 2024-04-22T15:59:16.952 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:59:16.952 INFO:teuthology.orchestra.run.smithi016.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:59:16.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.950+0000 7fb58f3ed700 1 -- 172.21.15.16:0/2749312129 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fb57406ecc0 msgr2=0x7fb574071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:16.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.950+0000 7fb58f3ed700 1 --2- 172.21.15.16:0/2749312129 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fb57406ecc0 0x7fb574071180 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb588197b60 tx=0x7fb58401a040 comp rx=0 tx=0).stop 2024-04-22T15:59:16.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.950+0000 7fb58f3ed700 1 -- 172.21.15.16:0/2749312129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5880ffda0 msgr2=0x7fb58819c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:16.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.950+0000 7fb58f3ed700 1 --2- 172.21.15.16:0/2749312129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5880ffda0 0x7fb58819c670 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb57800d8d0 tx=0x7fb57800dbe0 comp rx=0 tx=0).stop 2024-04-22T15:59:16.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.950+0000 7fb58f3ed700 1 -- 172.21.15.16:0/2749312129 shutdown_connections 2024-04-22T15:59:16.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.950+0000 7fb58f3ed700 1 --2- 172.21.15.16:0/2749312129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5880ffda0 0x7fb58819c670 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:16.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.950+0000 7fb58f3ed700 1 --2- 172.21.15.16:0/2749312129 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fb57406ecc0 0x7fb574071180 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:16.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.950+0000 7fb58f3ed700 1 --2- 172.21.15.16:0/2749312129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb588100750 0x7fb58819cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:16.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.950+0000 7fb58f3ed700 1 -- 172.21.15.16:0/2749312129 >> 172.21.15.16:0/2749312129 conn(0x7fb5880fb650 msgr2=0x7fb588069150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:16.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.950+0000 7fb58f3ed700 1 -- 172.21.15.16:0/2749312129 shutdown_connections 2024-04-22T15:59:16.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:16.950+0000 7fb58f3ed700 1 -- 172.21.15.16:0/2749312129 wait complete. 2024-04-22T15:59:17.395 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:17 smithi016 ceph-mon[110740]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:59:17.396 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:17 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/3753679055' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:59:17.396 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:17 smithi016 ceph-mon[110740]: reconnect by client.14544 192.168.0.1:0/3830921175 after 0.000999984 2024-04-22T15:59:17.396 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:17 smithi016 ceph-mon[110740]: reconnect by client.24341 192.168.0.1:0/2832093308 after 0.00199997 2024-04-22T15:59:17.396 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:17 smithi016 ceph-mon[110740]: mds.? [v2:172.21.15.16:6826/2871308685,v1:172.21.15.16:6827/2871308685] up:reconnect 2024-04-22T15:59:17.396 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:17 smithi016 ceph-mon[110740]: fsmap cephfs:1/1 {0=cephfs.smithi016.fkkejm=up:reconnect} 3 up:standby 2024-04-22T15:59:17.396 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:17 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/2749312129' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:59:17.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:17 smithi027 ceph-mon[83054]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:59:17.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:17 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/3753679055' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T15:59:17.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:17 smithi027 ceph-mon[83054]: reconnect by client.14544 192.168.0.1:0/3830921175 after 0.000999984 2024-04-22T15:59:17.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:17 smithi027 ceph-mon[83054]: reconnect by client.24341 192.168.0.1:0/2832093308 after 0.00199997 2024-04-22T15:59:17.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:17 smithi027 ceph-mon[83054]: mds.? [v2:172.21.15.16:6826/2871308685,v1:172.21.15.16:6827/2871308685] up:reconnect 2024-04-22T15:59:17.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:17 smithi027 ceph-mon[83054]: fsmap cephfs:1/1 {0=cephfs.smithi016.fkkejm=up:reconnect} 3 up:standby 2024-04-22T15:59:17.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:17 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/2749312129' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:59:18.376 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:18 smithi016 ceph-mon[110740]: pgmap v211: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 0 B/s wr, 6 op/s 2024-04-22T15:59:18.377 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:18 smithi016 ceph-mon[110740]: from='client.34470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:59:18.377 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:18 smithi016 ceph-mon[110740]: mds.? [v2:172.21.15.16:6826/2871308685,v1:172.21.15.16:6827/2871308685] up:rejoin 2024-04-22T15:59:18.377 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:18 smithi016 ceph-mon[110740]: fsmap cephfs:1/1 {0=cephfs.smithi016.fkkejm=up:rejoin} 3 up:standby 2024-04-22T15:59:18.377 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:18 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:18.377 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:18 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:18.377 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:18 smithi016 ceph-mon[110740]: daemon mds.cephfs.smithi016.fkkejm is now active in filesystem cephfs as rank 0 2024-04-22T15:59:18.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:18 smithi027 ceph-mon[83054]: pgmap v211: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 0 B/s wr, 6 op/s 2024-04-22T15:59:18.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:18 smithi027 ceph-mon[83054]: from='client.34470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:59:18.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:18 smithi027 ceph-mon[83054]: mds.? [v2:172.21.15.16:6826/2871308685,v1:172.21.15.16:6827/2871308685] up:rejoin 2024-04-22T15:59:18.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:18 smithi027 ceph-mon[83054]: fsmap cephfs:1/1 {0=cephfs.smithi016.fkkejm=up:rejoin} 3 up:standby 2024-04-22T15:59:18.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:18 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:18.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:18 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:18.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:18 smithi027 ceph-mon[83054]: daemon mds.cephfs.smithi016.fkkejm is now active in filesystem cephfs as rank 0 2024-04-22T15:59:19.226 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:19 smithi027 ceph-mon[83054]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T15:59:19.226 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:19 smithi027 ceph-mon[83054]: mds.? [v2:172.21.15.16:6826/2871308685,v1:172.21.15.16:6827/2871308685] up:active 2024-04-22T15:59:19.226 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:19 smithi027 ceph-mon[83054]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:active} 3 up:standby 2024-04-22T15:59:19.226 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:19 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:19.226 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:19 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:19.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:19 smithi016 ceph-mon[110740]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T15:59:19.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:19 smithi016 ceph-mon[110740]: mds.? [v2:172.21.15.16:6826/2871308685,v1:172.21.15.16:6827/2871308685] up:active 2024-04-22T15:59:19.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:19 smithi016 ceph-mon[110740]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:active} 3 up:standby 2024-04-22T15:59:19.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:19 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:19.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:19 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:20.404 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:20 smithi027 ceph-mon[83054]: pgmap v212: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 0 B/s wr, 6 op/s 2024-04-22T15:59:20.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:20 smithi016 ceph-mon[110740]: pgmap v212: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 0 B/s wr, 6 op/s 2024-04-22T15:59:22.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:22 smithi027 ceph-mon[83054]: pgmap v213: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 18 MiB/s rd, 170 B/s wr, 6 op/s 2024-04-22T15:59:22.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:22 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:22.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:22 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:22.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:22 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:59:22.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:22 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:59:22.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:22 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:22.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:22 smithi016 ceph-mon[110740]: pgmap v213: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 18 MiB/s rd, 170 B/s wr, 6 op/s 2024-04-22T15:59:22.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:22 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:22.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:22 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:22.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:22 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:59:22.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:22 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:59:22.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:22 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:23.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:23 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:59:23.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:23 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:23.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:23 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:23.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:23 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:23.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:23 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:23.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:23 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:23.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:23 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:23.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:23 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi016.fkkejm"}]: dispatch 2024-04-22T15:59:23.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:23 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi016.fkkejm"}]': finished 2024-04-22T15:59:23.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:23 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi016.pgvphu"}]: dispatch 2024-04-22T15:59:23.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:23 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi016.pgvphu"}]': finished 2024-04-22T15:59:23.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:23 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi027.dgqyes"}]: dispatch 2024-04-22T15:59:23.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:23 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi027.dgqyes"}]': finished 2024-04-22T15:59:23.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:23 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi027.iobjds"}]: dispatch 2024-04-22T15:59:23.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:23 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi027.iobjds"}]': finished 2024-04-22T15:59:23.418 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:23 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-22T15:59:23.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:23 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:59:23.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:23 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:23.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:23 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:23.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:23 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:23.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:23 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:23.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:23 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:23.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:23 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:23.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:23 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi016.fkkejm"}]: dispatch 2024-04-22T15:59:23.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:23 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi016.fkkejm"}]': finished 2024-04-22T15:59:23.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:23 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi016.pgvphu"}]: dispatch 2024-04-22T15:59:23.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:23 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi016.pgvphu"}]': finished 2024-04-22T15:59:23.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:23 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi027.dgqyes"}]: dispatch 2024-04-22T15:59:23.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:23 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi027.dgqyes"}]': finished 2024-04-22T15:59:23.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:23 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi027.iobjds"}]: dispatch 2024-04-22T15:59:23.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:23 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi027.iobjds"}]': finished 2024-04-22T15:59:23.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:23 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-22T15:59:24.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:24 smithi016 ceph-mon[110740]: Upgrade: Setting container_image for all mds 2024-04-22T15:59:24.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:24 smithi016 ceph-mon[110740]: Upgrade: Setting filesystem cephfs Joinable 2024-04-22T15:59:24.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:24 smithi016 ceph-mon[110740]: pgmap v214: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 14 MiB/s rd, 170 B/s wr, 6 op/s 2024-04-22T15:59:24.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:24 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-22T15:59:24.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:24 smithi016 ceph-mon[110740]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:active} 3 up:standby 2024-04-22T15:59:24.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:24 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]: dispatch 2024-04-22T15:59:24.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:24 smithi027 ceph-mon[83054]: Upgrade: Setting container_image for all mds 2024-04-22T15:59:24.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:24 smithi027 ceph-mon[83054]: Upgrade: Setting filesystem cephfs Joinable 2024-04-22T15:59:24.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:24 smithi027 ceph-mon[83054]: pgmap v214: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 14 MiB/s rd, 170 B/s wr, 6 op/s 2024-04-22T15:59:24.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:24 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-22T15:59:24.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:24 smithi027 ceph-mon[83054]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:active} 3 up:standby 2024-04-22T15:59:24.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:24 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]: dispatch 2024-04-22T15:59:25.153 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:25 smithi016 ceph-mon[110740]: Upgrade: Enabling allow_standby_replay on filesystem cephfs 2024-04-22T15:59:25.153 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:25 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]': finished 2024-04-22T15:59:25.153 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:25 smithi016 ceph-mon[110740]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:active} 3 up:standby 2024-04-22T15:59:25.153 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:25 smithi016 ceph-mon[110740]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:active} 1 up:standby-replay 2 up:standby 2024-04-22T15:59:25.153 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:25 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:25.153 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:25 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:25.154 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:25 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:25.154 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:25 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:25.154 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:25 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:25.154 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:25 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:25.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:25 smithi027 ceph-mon[83054]: Upgrade: Enabling allow_standby_replay on filesystem cephfs 2024-04-22T15:59:25.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:25 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]': finished 2024-04-22T15:59:25.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:25 smithi027 ceph-mon[83054]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:active} 3 up:standby 2024-04-22T15:59:25.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:25 smithi027 ceph-mon[83054]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:active} 1 up:standby-replay 2 up:standby 2024-04-22T15:59:25.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:25 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:25.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:25 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:25.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:25 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:25.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:25 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:25.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:25 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:25.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:25 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:26.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:26 smithi016 ceph-mon[110740]: Upgrade: Setting container_image for all rgw 2024-04-22T15:59:26.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:26 smithi016 ceph-mon[110740]: Upgrade: Setting container_image for all rbd-mirror 2024-04-22T15:59:26.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:26 smithi016 ceph-mon[110740]: pgmap v215: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 6.6 MiB/s rd, 4.2 KiB/s wr, 6 op/s 2024-04-22T15:59:26.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:26 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:26.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:26 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi016", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:59:26.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:26 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:59:26.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:26 smithi027 ceph-mon[83054]: Upgrade: Setting container_image for all rgw 2024-04-22T15:59:26.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:26 smithi027 ceph-mon[83054]: Upgrade: Setting container_image for all rbd-mirror 2024-04-22T15:59:26.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:26 smithi027 ceph-mon[83054]: pgmap v215: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 6.6 MiB/s rd, 4.2 KiB/s wr, 6 op/s 2024-04-22T15:59:26.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:26 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:26.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:26 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi016", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:59:26.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:26 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:59:27.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:27 smithi016 ceph-mon[110740]: Upgrade: Updating ceph-exporter.smithi016 (1/2) 2024-04-22T15:59:27.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:27 smithi016 ceph-mon[110740]: Deploying daemon ceph-exporter.smithi016 on smithi016 2024-04-22T15:59:27.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:27 smithi027 ceph-mon[83054]: Upgrade: Updating ceph-exporter.smithi016 (1/2) 2024-04-22T15:59:27.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:27 smithi027 ceph-mon[83054]: Deploying daemon ceph-exporter.smithi016 on smithi016 2024-04-22T15:59:28.295 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:28 smithi016 ceph-mon[110740]: pgmap v216: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 3.9 MiB/s rd, 4.2 KiB/s wr, 6 op/s 2024-04-22T15:59:28.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:28 smithi027 ceph-mon[83054]: pgmap v216: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 3.9 MiB/s rd, 4.2 KiB/s wr, 6 op/s 2024-04-22T15:59:29.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:29 smithi016 ceph-mon[110740]: pgmap v217: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 8.6 MiB/s rd, 4.2 KiB/s wr, 7 op/s 2024-04-22T15:59:29.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:29 smithi027 ceph-mon[83054]: pgmap v217: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 8.6 MiB/s rd, 4.2 KiB/s wr, 7 op/s 2024-04-22T15:59:31.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:31 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:31.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:31 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:59:31.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:31 smithi027 ceph-mon[83054]: pgmap v218: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 9.9 MiB/s rd, 4.2 KiB/s wr, 8 op/s 2024-04-22T15:59:31.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:31 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:31.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:31 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:31.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:31 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:31.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:31 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:59:31.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:31 smithi016 ceph-mon[110740]: pgmap v218: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 9.9 MiB/s rd, 4.2 KiB/s wr, 8 op/s 2024-04-22T15:59:31.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:31 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:31.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:31 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:33.444 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:33 smithi027 ceph-mon[83054]: Upgrade: Updating ceph-exporter.smithi027 (2/2) 2024-04-22T15:59:33.444 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:33 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:33.444 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:33 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi027", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:59:33.444 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:33 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:59:33.444 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:33 smithi027 ceph-mon[83054]: Deploying daemon ceph-exporter.smithi027 on smithi027 2024-04-22T15:59:33.444 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:33 smithi027 ceph-mon[83054]: pgmap v219: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 17 MiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-04-22T15:59:33.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:33 smithi016 ceph-mon[110740]: Upgrade: Updating ceph-exporter.smithi027 (2/2) 2024-04-22T15:59:33.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:33 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:33.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:33 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi027", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T15:59:33.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:33 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:59:33.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:33 smithi016 ceph-mon[110740]: Deploying daemon ceph-exporter.smithi027 on smithi027 2024-04-22T15:59:33.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:33 smithi016 ceph-mon[110740]: pgmap v219: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 17 MiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-04-22T15:59:35.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:35 smithi027 ceph-mon[83054]: pgmap v220: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 19 MiB/s rd, 4.0 KiB/s wr, 7 op/s 2024-04-22T15:59:35.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:35 smithi016 ceph-mon[110740]: pgmap v220: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 19 MiB/s rd, 4.0 KiB/s wr, 7 op/s 2024-04-22T15:59:36.976 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:36 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:36.976 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:36 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:36.976 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:36 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:59:37.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:36 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:37.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:36 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:37.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:36 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:59:38.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:37 smithi016 ceph-mon[110740]: pgmap v221: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 19 MiB/s rd, 5 op/s 2024-04-22T15:59:38.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:37 smithi027 ceph-mon[83054]: pgmap v221: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 19 MiB/s rd, 5 op/s 2024-04-22T15:59:39.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:39 smithi016 ceph-mon[110740]: pgmap v222: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 5 op/s 2024-04-22T15:59:39.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:39 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:39.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:39 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:39.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:39 smithi027 ceph-mon[83054]: pgmap v222: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 5 op/s 2024-04-22T15:59:39.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:39 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:39.837 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:39 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:40.829 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:40 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:40.829 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:40 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:40.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:40 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:40.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:40 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:42.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:41 smithi027 ceph-mon[83054]: pgmap v223: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 16 MiB/s rd, 4 op/s 2024-04-22T15:59:42.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:41 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:42.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:41 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:42.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:41 smithi016 ceph-mon[110740]: pgmap v223: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 16 MiB/s rd, 4 op/s 2024-04-22T15:59:42.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:41 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:42.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:41 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:43.312 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:42 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:43.312 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:42 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:43.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:42 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:43.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:42 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:44.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:43 smithi027 ceph-mon[83054]: pgmap v224: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 11 MiB/s rd, 3 op/s 2024-04-22T15:59:44.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:43 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:44.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:43 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:44.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:43 smithi016 ceph-mon[110740]: pgmap v224: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 11 MiB/s rd, 3 op/s 2024-04-22T15:59:44.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:43 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:44.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:43 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:46.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:46 smithi027 ceph-mon[83054]: pgmap v225: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 3.9 MiB/s rd, 2 op/s 2024-04-22T15:59:46.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:59:46.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:46.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:46.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:59:46.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:59:46.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:46.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:59:46.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:46.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:46.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:46.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:46.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:46.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-22T15:59:46.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:46 smithi016 ceph-mon[110740]: pgmap v225: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 3.9 MiB/s rd, 2 op/s 2024-04-22T15:59:46.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T15:59:46.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:46.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:46.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:59:46.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:59:46.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:46.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:59:46.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:46.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:46.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:46.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:46.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:46.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-22T15:59:47.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.061+0000 7f7baa20e700 1 -- 172.21.15.16:0/1585049394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ba4101850 msgr2=0x7f7ba41058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:47.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.061+0000 7f7baa20e700 1 --2- 172.21.15.16:0/1585049394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ba4101850 0x7f7ba41058a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7b98009910 tx=0x7f7b98009c20 comp rx=0 tx=0).stop 2024-04-22T15:59:47.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.061+0000 7f7baa20e700 1 -- 172.21.15.16:0/1585049394 shutdown_connections 2024-04-22T15:59:47.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.061+0000 7f7baa20e700 1 --2- 172.21.15.16:0/1585049394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ba4101850 0x7f7ba41058a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:47.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.061+0000 7f7baa20e700 1 --2- 172.21.15.16:0/1585049394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ba4100ea0 0x7f7ba4101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:47.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.061+0000 7f7baa20e700 1 -- 172.21.15.16:0/1585049394 >> 172.21.15.16:0/1585049394 conn(0x7f7ba40fc750 msgr2=0x7f7ba40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:47.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.062+0000 7f7baa20e700 1 -- 172.21.15.16:0/1585049394 shutdown_connections 2024-04-22T15:59:47.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.062+0000 7f7baa20e700 1 -- 172.21.15.16:0/1585049394 wait complete. 2024-04-22T15:59:47.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.062+0000 7f7baa20e700 1 Processor -- start 2024-04-22T15:59:47.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.063+0000 7f7baa20e700 1 -- start start 2024-04-22T15:59:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.063+0000 7f7baa20e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ba4100ea0 0x7f7ba419e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.063+0000 7f7baa20e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ba4101850 0x7f7ba419ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.063+0000 7f7baa20e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba419f4a0 con 0x7f7ba4101850 2024-04-22T15:59:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.063+0000 7f7baa20e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba4198950 con 0x7f7ba4100ea0 2024-04-22T15:59:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.063+0000 7f7ba2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ba4101850 0x7f7ba419ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.063+0000 7f7ba2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ba4101850 0x7f7ba419ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53838/0 (socket says 172.21.15.16:53838) 2024-04-22T15:59:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.063+0000 7f7ba2ffd700 1 -- 172.21.15.16:0/4031339776 learned_addr learned my addr 172.21.15.16:0/4031339776 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:59:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.063+0000 7f7ba37fe700 1 --2- 172.21.15.16:0/4031339776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ba4100ea0 0x7f7ba419e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:47.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.063+0000 7f7ba2ffd700 1 -- 172.21.15.16:0/4031339776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ba4100ea0 msgr2=0x7f7ba419e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:47.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.063+0000 7f7ba2ffd700 1 --2- 172.21.15.16:0/4031339776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ba4100ea0 0x7f7ba419e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:47.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.063+0000 7f7ba2ffd700 1 -- 172.21.15.16:0/4031339776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b980095f0 con 0x7f7ba4101850 2024-04-22T15:59:47.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.063+0000 7f7ba2ffd700 1 --2- 172.21.15.16:0/4031339776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ba4101850 0x7f7ba419ee10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7b9800bd70 tx=0x7f7b9800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:47.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.064+0000 7f7ba0ff9700 1 -- 172.21.15.16:0/4031339776 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b9801d070 con 0x7f7ba4101850 2024-04-22T15:59:47.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.064+0000 7f7ba0ff9700 1 -- 172.21.15.16:0/4031339776 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b98005dc0 con 0x7f7ba4101850 2024-04-22T15:59:47.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.064+0000 7f7baa20e700 1 -- 172.21.15.16:0/4031339776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba4198bd0 con 0x7f7ba4101850 2024-04-22T15:59:47.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.064+0000 7f7ba0ff9700 1 -- 172.21.15.16:0/4031339776 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b98017400 con 0x7f7ba4101850 2024-04-22T15:59:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.064+0000 7f7baa20e700 1 -- 172.21.15.16:0/4031339776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba41990c0 con 0x7f7ba4101850 2024-04-22T15:59:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.065+0000 7f7baa20e700 1 -- 172.21.15.16:0/4031339776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba4105020 con 0x7f7ba4101850 2024-04-22T15:59:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.065+0000 7f7ba0ff9700 1 -- 172.21.15.16:0/4031339776 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7b9800f7f0 con 0x7f7ba4101850 2024-04-22T15:59:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.066+0000 7f7ba0ff9700 1 --2- 172.21.15.16:0/4031339776 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f7b8c06ecc0 0x7f7b8c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.066+0000 7f7ba0ff9700 1 -- 172.21.15.16:0/4031339776 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7b98090350 con 0x7f7ba4101850 2024-04-22T15:59:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.066+0000 7f7ba37fe700 1 --2- 172.21.15.16:0/4031339776 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f7b8c06ecc0 0x7f7b8c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.066+0000 7f7ba37fe700 1 --2- 172.21.15.16:0/4031339776 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f7b8c06ecc0 0x7f7b8c071180 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7b94005950 tx=0x7f7b940058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:47.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.071+0000 7f7ba0ff9700 1 -- 172.21.15.16:0/4031339776 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7b9805cc10 con 0x7f7ba4101850 2024-04-22T15:59:47.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: Upgrade: Setting filesystem cephfs Joinable 2024-04-22T15:59:47.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-22T15:59:47.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:active} 1 up:standby-replay 2 up:standby 2024-04-22T15:59:47.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:47.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi016"}]: dispatch 2024-04-22T15:59:47.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi016"}]': finished 2024-04-22T15:59:47.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi027"}]: dispatch 2024-04-22T15:59:47.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi027"}]': finished 2024-04-22T15:59:47.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:47.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:47.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.168 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:47.169 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.169 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.169 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.169 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.169 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.169 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.169 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-22T15:59:47.169 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-22T15:59:47.169 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:47.169 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-22T15:59:47.169 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-22T15:59:47.169 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-22T15:59:47.169 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-22T15:59:47.170 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-22T15:59:47.170 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-22T15:59:47.170 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-22T15:59:47.170 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-22T15:59:47.170 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-22T15:59:47.170 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-22T15:59:47.170 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-22T15:59:47.170 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:47.170 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-22T15:59:47.170 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-22T15:59:47.170 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-22T15:59:47.170 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-22T15:59:47.170 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-22T15:59:47.171 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-22T15:59:47.171 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-22T15:59:47.171 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-22T15:59:47.171 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:47.171 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:47.171 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:47.171 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:47.171 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:47.171 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:47.171 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-22T15:59:47.171 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-22T15:59:47.171 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:59:47.171 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:59:47.172 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:59:47.172 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:47.172 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:59:47.172 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:59:47.172 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:59:47.172 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:47.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.303+0000 7f7baa20e700 1 -- 172.21.15.16:0/4031339776 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ba4199330 con 0x7f7b8c06ecc0 2024-04-22T15:59:47.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.304+0000 7f7ba0ff9700 1 -- 172.21.15.16:0/4031339776 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f7ba4199330 con 0x7f7b8c06ecc0 2024-04-22T15:59:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.306+0000 7f7baa20e700 1 -- 172.21.15.16:0/4031339776 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f7b8c06ecc0 msgr2=0x7f7b8c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.307+0000 7f7baa20e700 1 --2- 172.21.15.16:0/4031339776 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f7b8c06ecc0 0x7f7b8c071180 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7b94005950 tx=0x7f7b940058e0 comp rx=0 tx=0).stop 2024-04-22T15:59:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.307+0000 7f7baa20e700 1 -- 172.21.15.16:0/4031339776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ba4101850 msgr2=0x7f7ba419ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.307+0000 7f7baa20e700 1 --2- 172.21.15.16:0/4031339776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ba4101850 0x7f7ba419ee10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7b9800bd70 tx=0x7f7b9800be50 comp rx=0 tx=0).stop 2024-04-22T15:59:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.307+0000 7f7baa20e700 1 -- 172.21.15.16:0/4031339776 shutdown_connections 2024-04-22T15:59:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.307+0000 7f7baa20e700 1 --2- 172.21.15.16:0/4031339776 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f7b8c06ecc0 0x7f7b8c071180 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.307+0000 7f7baa20e700 1 --2- 172.21.15.16:0/4031339776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ba4100ea0 0x7f7ba419e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.307+0000 7f7baa20e700 1 --2- 172.21.15.16:0/4031339776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ba4101850 0x7f7ba419ee10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.307+0000 7f7baa20e700 1 -- 172.21.15.16:0/4031339776 >> 172.21.15.16:0/4031339776 conn(0x7f7ba40fc750 msgr2=0x7f7ba40feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.307+0000 7f7baa20e700 1 -- 172.21.15.16:0/4031339776 shutdown_connections 2024-04-22T15:59:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:47.307+0000 7f7baa20e700 1 -- 172.21.15.16:0/4031339776 wait complete. 2024-04-22T15:59:47.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: Upgrade: Setting filesystem cephfs Joinable 2024-04-22T15:59:47.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-22T15:59:47.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: fsmap cephfs:1 {0=cephfs.smithi016.fkkejm=up:active} 1 up:standby-replay 2 up:standby 2024-04-22T15:59:47.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:47.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi016"}]: dispatch 2024-04-22T15:59:47.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi016"}]': finished 2024-04-22T15:59:47.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi027"}]: dispatch 2024-04-22T15:59:47.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi027"}]': finished 2024-04-22T15:59:47.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:47.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:47.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:47.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:47.338 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-22T15:59:47.339 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-22T15:59:47.339 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:47.339 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-22T15:59:47.339 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-22T15:59:47.339 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-22T15:59:47.339 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-22T15:59:47.339 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-22T15:59:47.339 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-22T15:59:47.339 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-22T15:59:47.339 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-22T15:59:47.339 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-22T15:59:47.339 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-22T15:59:47.339 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-22T15:59:47.340 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:47.340 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-22T15:59:47.340 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-22T15:59:47.340 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-22T15:59:47.340 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-22T15:59:47.340 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-22T15:59:47.340 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-22T15:59:47.340 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-22T15:59:47.340 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-22T15:59:47.340 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:47.340 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:47.340 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:47.341 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:47.341 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:47.341 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T15:59:47.341 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-22T15:59:47.341 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-22T15:59:47.341 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:59:47.341 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:59:47.341 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:59:47.341 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:47.341 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T15:59:47.341 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T15:59:47.341 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T15:59:47.341 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:47.960 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph orch ps' 2024-04-22T15:59:48.220 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:59:48.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:48 smithi027 ceph-mon[83054]: pgmap v226: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1 op/s 2024-04-22T15:59:48.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:48 smithi027 ceph-mon[83054]: Upgrade: Setting container_image for all ceph-exporter 2024-04-22T15:59:48.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:48 smithi027 ceph-mon[83054]: Upgrade: Setting container_image for all iscsi 2024-04-22T15:59:48.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:48 smithi027 ceph-mon[83054]: Upgrade: Setting container_image for all nfs 2024-04-22T15:59:48.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:48 smithi027 ceph-mon[83054]: Upgrade: Setting container_image for all nvmeof 2024-04-22T15:59:48.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:48 smithi027 ceph-mon[83054]: Upgrade: Finalizing container_image settings 2024-04-22T15:59:48.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:48 smithi027 ceph-mon[83054]: Upgrade: Complete! 2024-04-22T15:59:48.345 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:48 smithi016 ceph-mon[110740]: pgmap v226: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1 op/s 2024-04-22T15:59:48.345 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:48 smithi016 ceph-mon[110740]: Upgrade: Setting container_image for all ceph-exporter 2024-04-22T15:59:48.345 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:48 smithi016 ceph-mon[110740]: Upgrade: Setting container_image for all iscsi 2024-04-22T15:59:48.345 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:48 smithi016 ceph-mon[110740]: Upgrade: Setting container_image for all nfs 2024-04-22T15:59:48.345 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:48 smithi016 ceph-mon[110740]: Upgrade: Setting container_image for all nvmeof 2024-04-22T15:59:48.345 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:48 smithi016 ceph-mon[110740]: Upgrade: Finalizing container_image settings 2024-04-22T15:59:48.345 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:48 smithi016 ceph-mon[110740]: Upgrade: Complete! 2024-04-22T15:59:49.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:49 smithi027 ceph-mon[83054]: from='client.34480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:59:49.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:49 smithi016 ceph-mon[110740]: from='client.34480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:59:49.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.949+0000 7efea3030700 1 -- 172.21.15.16:0/1975742170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9c0fff50 msgr2=0x7efe9c105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:49.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.949+0000 7efea3030700 1 --2- 172.21.15.16:0/1975742170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9c0fff50 0x7efe9c105840 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efe90009910 tx=0x7efe90009c20 comp rx=0 tx=0).stop 2024-04-22T15:59:49.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.949+0000 7efea3030700 1 -- 172.21.15.16:0/1975742170 shutdown_connections 2024-04-22T15:59:49.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.949+0000 7efea3030700 1 --2- 172.21.15.16:0/1975742170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9c0fff50 0x7efe9c105840 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:49.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.949+0000 7efea3030700 1 --2- 172.21.15.16:0/1975742170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe9c0ff5a0 0x7efe9c0ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:49.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.949+0000 7efea3030700 1 -- 172.21.15.16:0/1975742170 >> 172.21.15.16:0/1975742170 conn(0x7efe9c074e30 msgr2=0x7efe9c075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:49.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.949+0000 7efea3030700 1 -- 172.21.15.16:0/1975742170 shutdown_connections 2024-04-22T15:59:49.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.949+0000 7efea3030700 1 -- 172.21.15.16:0/1975742170 wait complete. 2024-04-22T15:59:49.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.950+0000 7efea3030700 1 Processor -- start 2024-04-22T15:59:49.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.950+0000 7efea3030700 1 -- start start 2024-04-22T15:59:49.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.950+0000 7efea3030700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9c0ff5a0 0x7efe9c100520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:49.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.950+0000 7efea3030700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe9c0fff50 0x7efe9c100a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:49.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.950+0000 7efea3030700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c100fa0 con 0x7efe9c0ff5a0 2024-04-22T15:59:49.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.950+0000 7efea3030700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c1010e0 con 0x7efe9c0fff50 2024-04-22T15:59:49.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.950+0000 7efea0dcc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9c0ff5a0 0x7efe9c100520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:49.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.950+0000 7efea0dcc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9c0ff5a0 0x7efe9c100520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53854/0 (socket says 172.21.15.16:53854) 2024-04-22T15:59:49.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.950+0000 7efea0dcc700 1 -- 172.21.15.16:0/2392930140 learned_addr learned my addr 172.21.15.16:0/2392930140 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:59:49.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.950+0000 7efe9bfff700 1 --2- 172.21.15.16:0/2392930140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe9c0fff50 0x7efe9c100a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:49.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.950+0000 7efea0dcc700 1 -- 172.21.15.16:0/2392930140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe9c0fff50 msgr2=0x7efe9c100a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:49.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.950+0000 7efea0dcc700 1 --2- 172.21.15.16:0/2392930140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe9c0fff50 0x7efe9c100a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:49.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.950+0000 7efea0dcc700 1 -- 172.21.15.16:0/2392930140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe900095f0 con 0x7efe9c0ff5a0 2024-04-22T15:59:49.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.951+0000 7efea0dcc700 1 --2- 172.21.15.16:0/2392930140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9c0ff5a0 0x7efe9c100520 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efe9400ef10 tx=0x7efe9400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:49.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.951+0000 7efe99ffb700 1 -- 172.21.15.16:0/2392930140 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe9400ca50 con 0x7efe9c0ff5a0 2024-04-22T15:59:49.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.951+0000 7efea3030700 1 -- 172.21.15.16:0/2392930140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe9c101360 con 0x7efe9c0ff5a0 2024-04-22T15:59:49.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.951+0000 7efe99ffb700 1 -- 172.21.15.16:0/2392930140 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe9400f040 con 0x7efe9c0ff5a0 2024-04-22T15:59:49.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.951+0000 7efe99ffb700 1 -- 172.21.15.16:0/2392930140 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe940146b0 con 0x7efe9c0ff5a0 2024-04-22T15:59:49.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.951+0000 7efea3030700 1 -- 172.21.15.16:0/2392930140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe9c1a2e20 con 0x7efe9c0ff5a0 2024-04-22T15:59:49.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.952+0000 7efea3030700 1 -- 172.21.15.16:0/2392930140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe9c109200 con 0x7efe9c0ff5a0 2024-04-22T15:59:49.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.953+0000 7efe99ffb700 1 -- 172.21.15.16:0/2392930140 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efe9400cbb0 con 0x7efe9c0ff5a0 2024-04-22T15:59:49.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.953+0000 7efe99ffb700 1 --2- 172.21.15.16:0/2392930140 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7efe8406ecc0 0x7efe84071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:49.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.953+0000 7efe99ffb700 1 -- 172.21.15.16:0/2392930140 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efe9408f270 con 0x7efe9c0ff5a0 2024-04-22T15:59:49.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.953+0000 7efe9bfff700 1 --2- 172.21.15.16:0/2392930140 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7efe8406ecc0 0x7efe84071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:49.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.954+0000 7efe9bfff700 1 --2- 172.21.15.16:0/2392930140 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7efe8406ecc0 0x7efe84071180 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efe9000b760 tx=0x7efe90009f90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:49.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:49.958+0000 7efe99ffb700 1 -- 172.21.15.16:0/2392930140 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7efe94058020 con 0x7efe9c0ff5a0 2024-04-22T15:59:50.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:50.181+0000 7efea3030700 1 -- 172.21.15.16:0/2392930140 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7efe9c0611d0 con 0x7efe8406ecc0 2024-04-22T15:59:50.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:50.191+0000 7efe99ffb700 1 -- 172.21.15.16:0/2392930140 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7efe9c0611d0 con 0x7efe8406ecc0 2024-04-22T15:59:50.194 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T15:59:50.195 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.smithi016 smithi016 *:9093,9094 running (8m) 9s ago 18m 25.9M - 0.25.0 c8568f914cd2 ea0e0217d6f9 2024-04-22T15:59:50.195 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi016 smithi016 running (20s) 9s ago 19m 7810k - 19.0.0-3223-g5883d494 3d983c26e278 7201ff7ad79e 2024-04-22T15:59:50.195 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi027 smithi027 running (14s) 11s ago 17m 9068k - 19.0.0-3223-g5883d494 3d983c26e278 aeeac9eb9e02 2024-04-22T15:59:50.195 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi016 smithi016 running (5m) 9s ago 19m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 1feb008449ed 2024-04-22T15:59:50.195 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi027 smithi027 running (5m) 11s ago 17m 6656k - 19.0.0-3223-g5883d494 3d983c26e278 c7f862c6da00 2024-04-22T15:59:50.195 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.smithi016 smithi016 *:3000 running (7m) 9s ago 18m 82.0M - 9.4.12 2bacad6d85d8 388a3c9c1b3f 2024-04-22T15:59:50.195 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.fkkejm smithi016 running (96s) 9s ago 13m 90.8M - 19.0.0-3223-g5883d494 3d983c26e278 e23d9fc30059 2024-04-22T15:59:50.195 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.pgvphu smithi016 running (79s) 9s ago 13m 199M - 19.0.0-3223-g5883d494 3d983c26e278 4777936b5956 2024-04-22T15:59:50.195 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.dgqyes smithi027 running (36s) 11s ago 13m 13.7M - 19.0.0-3223-g5883d494 3d983c26e278 32404711622f 2024-04-22T15:59:50.195 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.iobjds smithi027 running (60s) 11s ago 13m 15.2M - 19.0.0-3223-g5883d494 3d983c26e278 8dd739415c15 2024-04-22T15:59:50.195 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi016.rnsnnr smithi016 *:8443,9283,8765 running (10m) 9s ago 20m 512M - 19.0.0-3223-g5883d494 3d983c26e278 09b4fe285bc4 2024-04-22T15:59:50.195 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi027.ukdgzs smithi027 *:8443,9283,8765 running (9m) 11s ago 17m 452M - 19.0.0-3223-g5883d494 3d983c26e278 7548da9b5eb9 2024-04-22T15:59:50.196 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi016 smithi016 running (6m) 9s ago 20m 59.0M 2048M 19.0.0-3223-g5883d494 3d983c26e278 4ae529a67806 2024-04-22T15:59:50.196 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi027 smithi027 running (6m) 11s ago 17m 46.2M 2048M 19.0.0-3223-g5883d494 3d983c26e278 f29fcb888801 2024-04-22T15:59:50.196 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi016 smithi016 *:9100 running (9m) 9s ago 18m 15.6M - 1.5.0 0da6a335fe13 35e46e4d2f3f 2024-04-22T15:59:50.196 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi027 smithi027 *:9100 running (8m) 11s ago 17m 15.0M - 1.5.0 0da6a335fe13 e682a7672e9b 2024-04-22T15:59:50.196 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (4m) 9s ago 16m 97.2M 1217M 19.0.0-3223-g5883d494 3d983c26e278 79217eb383a2 2024-04-22T15:59:50.196 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (4m) 9s ago 16m 103M 1217M 19.0.0-3223-g5883d494 3d983c26e278 ec1a45b3c53e 2024-04-22T15:59:50.196 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (3m) 9s ago 16m 87.8M 1217M 19.0.0-3223-g5883d494 3d983c26e278 48bb459fd021 2024-04-22T15:59:50.196 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi027 running (3m) 11s ago 15m 108M 2240M 19.0.0-3223-g5883d494 3d983c26e278 81034bf22d3f 2024-04-22T15:59:50.196 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (2m) 11s ago 15m 100M 2240M 19.0.0-3223-g5883d494 3d983c26e278 93b660e5ecb6 2024-04-22T15:59:50.196 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi027 running (115s) 11s ago 14m 96.2M 2240M 19.0.0-3223-g5883d494 3d983c26e278 d857c152994a 2024-04-22T15:59:50.196 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.smithi016 smithi016 *:9095 running (8m) 9s ago 18m 80.3M - 2.43.0 a07b618ecd1d 373eaadee82d 2024-04-22T15:59:50.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:50.194+0000 7efea3030700 1 -- 172.21.15.16:0/2392930140 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7efe8406ecc0 msgr2=0x7efe84071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:50.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:50.194+0000 7efea3030700 1 --2- 172.21.15.16:0/2392930140 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7efe8406ecc0 0x7efe84071180 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efe9000b760 tx=0x7efe90009f90 comp rx=0 tx=0).stop 2024-04-22T15:59:50.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:50.194+0000 7efea3030700 1 -- 172.21.15.16:0/2392930140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9c0ff5a0 msgr2=0x7efe9c100520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:50.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:50.194+0000 7efea3030700 1 --2- 172.21.15.16:0/2392930140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9c0ff5a0 0x7efe9c100520 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efe9400ef10 tx=0x7efe9400c3a0 comp rx=0 tx=0).stop 2024-04-22T15:59:50.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:50.194+0000 7efea3030700 1 -- 172.21.15.16:0/2392930140 shutdown_connections 2024-04-22T15:59:50.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:50.194+0000 7efea3030700 1 --2- 172.21.15.16:0/2392930140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9c0ff5a0 0x7efe9c100520 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:50.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:50.194+0000 7efea3030700 1 --2- 172.21.15.16:0/2392930140 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7efe8406ecc0 0x7efe84071180 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:50.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:50.194+0000 7efea3030700 1 --2- 172.21.15.16:0/2392930140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe9c0fff50 0x7efe9c100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:50.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:50.194+0000 7efea3030700 1 -- 172.21.15.16:0/2392930140 >> 172.21.15.16:0/2392930140 conn(0x7efe9c074e30 msgr2=0x7efe9c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:50.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:50.194+0000 7efea3030700 1 -- 172.21.15.16:0/2392930140 shutdown_connections 2024-04-22T15:59:50.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:50.194+0000 7efea3030700 1 -- 172.21.15.16:0/2392930140 wait complete. 2024-04-22T15:59:50.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:50 smithi027 ceph-mon[83054]: pgmap v227: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1 op/s 2024-04-22T15:59:50.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:50 smithi016 ceph-mon[110740]: pgmap v227: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1 op/s 2024-04-22T15:59:50.861 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph orch upgrade status' 2024-04-22T15:59:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:59:51.771 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:51 smithi016 ceph-mon[110740]: from='client.34484 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:59:51.772 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:51 smithi016 ceph-mon[110740]: pgmap v228: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:59:51.772 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:51 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:51.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:51 smithi027 ceph-mon[83054]: from='client.34484 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:59:51.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:51 smithi027 ceph-mon[83054]: pgmap v228: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:59:51.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:51 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T15:59:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.612+0000 7f51f7589700 1 -- 172.21.15.16:0/1300504830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f0105bd0 msgr2=0x7f51f0105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.612+0000 7f51f7589700 1 --2- 172.21.15.16:0/1300504830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f0105bd0 0x7f51f0105fb0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f51ec009910 tx=0x7f51ec009c20 comp rx=0 tx=0).stop 2024-04-22T15:59:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.612+0000 7f51f7589700 1 -- 172.21.15.16:0/1300504830 shutdown_connections 2024-04-22T15:59:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.612+0000 7f51f7589700 1 --2- 172.21.15.16:0/1300504830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f00ffc20 0x7f51f0100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.612+0000 7f51f7589700 1 --2- 172.21.15.16:0/1300504830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f0105bd0 0x7f51f0105fb0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:52.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.612+0000 7f51f7589700 1 -- 172.21.15.16:0/1300504830 >> 172.21.15.16:0/1300504830 conn(0x7f51f0075670 msgr2=0x7f51f0075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:52.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.612+0000 7f51f7589700 1 -- 172.21.15.16:0/1300504830 shutdown_connections 2024-04-22T15:59:52.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.612+0000 7f51f7589700 1 -- 172.21.15.16:0/1300504830 wait complete. 2024-04-22T15:59:52.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.613+0000 7f51f7589700 1 Processor -- start 2024-04-22T15:59:52.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.613+0000 7f51f7589700 1 -- start start 2024-04-22T15:59:52.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.613+0000 7f51f7589700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f00ffc20 0x7f51f0101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:52.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.613+0000 7f51f7589700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f0105bd0 0x7f51f0101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:52.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.613+0000 7f51f7589700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f01022c0 con 0x7f51f0105bd0 2024-04-22T15:59:52.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.613+0000 7f51f7589700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f0102400 con 0x7f51f00ffc20 2024-04-22T15:59:52.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.613+0000 7f51f4b24700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f0105bd0 0x7f51f0101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:52.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.613+0000 7f51f4b24700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f0105bd0 0x7f51f0101d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53278/0 (socket says 172.21.15.16:53278) 2024-04-22T15:59:52.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.613+0000 7f51f4b24700 1 -- 172.21.15.16:0/3290575662 learned_addr learned my addr 172.21.15.16:0/3290575662 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:59:52.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.613+0000 7f51f5325700 1 --2- 172.21.15.16:0/3290575662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f00ffc20 0x7f51f0101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:52.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.613+0000 7f51f4b24700 1 -- 172.21.15.16:0/3290575662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f00ffc20 msgr2=0x7f51f0101840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:52.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.613+0000 7f51f4b24700 1 --2- 172.21.15.16:0/3290575662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f00ffc20 0x7f51f0101840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:52.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.613+0000 7f51f4b24700 1 -- 172.21.15.16:0/3290575662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51ec0095f0 con 0x7f51f0105bd0 2024-04-22T15:59:52.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.613+0000 7f51f5325700 1 --2- 172.21.15.16:0/3290575662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f00ffc20 0x7f51f0101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:59:52.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.614+0000 7f51f4b24700 1 --2- 172.21.15.16:0/3290575662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f0105bd0 0x7f51f0101d80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f51e000e920 tx=0x7f51e000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:52.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.614+0000 7f51e67fc700 1 -- 172.21.15.16:0/3290575662 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51e0009790 con 0x7f51f0105bd0 2024-04-22T15:59:52.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.614+0000 7f51e67fc700 1 -- 172.21.15.16:0/3290575662 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f51e000cba0 con 0x7f51f0105bd0 2024-04-22T15:59:52.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.614+0000 7f51e67fc700 1 -- 172.21.15.16:0/3290575662 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51e0018650 con 0x7f51f0105bd0 2024-04-22T15:59:52.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.614+0000 7f51f7589700 1 -- 172.21.15.16:0/3290575662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f01a2a20 con 0x7f51f0105bd0 2024-04-22T15:59:52.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.614+0000 7f51f7589700 1 -- 172.21.15.16:0/3290575662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f01a2e30 con 0x7f51f0105bd0 2024-04-22T15:59:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.615+0000 7f51f7589700 1 -- 172.21.15.16:0/3290575662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51f004f2e0 con 0x7f51f0105bd0 2024-04-22T15:59:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.615+0000 7f51e67fc700 1 -- 172.21.15.16:0/3290575662 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f51e000cd10 con 0x7f51f0105bd0 2024-04-22T15:59:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.616+0000 7f51e67fc700 1 --2- 172.21.15.16:0/3290575662 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f51dc06ecc0 0x7f51dc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.616+0000 7f51e67fc700 1 -- 172.21.15.16:0/3290575662 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f51e0014070 con 0x7f51f0105bd0 2024-04-22T15:59:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.616+0000 7f51f5325700 1 --2- 172.21.15.16:0/3290575662 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f51dc06ecc0 0x7f51dc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.616+0000 7f51f5325700 1 --2- 172.21.15.16:0/3290575662 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f51dc06ecc0 0x7f51dc071180 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f51ec000c00 tx=0x7f51ec009f70 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:52.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.621+0000 7f51e67fc700 1 -- 172.21.15.16:0/3290575662 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f51e0054150 con 0x7f51f0105bd0 2024-04-22T15:59:52.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.852+0000 7f51f7589700 1 -- 172.21.15.16:0/3290575662 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f51f0068dd0 con 0x7f51dc06ecc0 2024-04-22T15:59:52.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.853+0000 7f51e67fc700 1 -- 172.21.15.16:0/3290575662 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f51f0068dd0 con 0x7f51dc06ecc0 2024-04-22T15:59:52.856 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:59:52.856 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": null, 2024-04-22T15:59:52.856 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": false, 2024-04-22T15:59:52.856 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "", 2024-04-22T15:59:52.857 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [], 2024-04-22T15:59:52.857 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": null, 2024-04-22T15:59:52.857 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "", 2024-04-22T15:59:52.857 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-22T15:59:52.857 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:59:52.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.855+0000 7f51f7589700 1 -- 172.21.15.16:0/3290575662 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f51dc06ecc0 msgr2=0x7f51dc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:52.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.855+0000 7f51f7589700 1 --2- 172.21.15.16:0/3290575662 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f51dc06ecc0 0x7f51dc071180 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f51ec000c00 tx=0x7f51ec009f70 comp rx=0 tx=0).stop 2024-04-22T15:59:52.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.855+0000 7f51f7589700 1 -- 172.21.15.16:0/3290575662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f0105bd0 msgr2=0x7f51f0101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:52.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.855+0000 7f51f7589700 1 --2- 172.21.15.16:0/3290575662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f0105bd0 0x7f51f0101d80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f51e000e920 tx=0x7f51e000ece0 comp rx=0 tx=0).stop 2024-04-22T15:59:52.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.855+0000 7f51f7589700 1 -- 172.21.15.16:0/3290575662 shutdown_connections 2024-04-22T15:59:52.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.855+0000 7f51f7589700 1 --2- 172.21.15.16:0/3290575662 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f51dc06ecc0 0x7f51dc071180 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:52.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.855+0000 7f51f7589700 1 --2- 172.21.15.16:0/3290575662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f00ffc20 0x7f51f0101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:52.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.855+0000 7f51f7589700 1 --2- 172.21.15.16:0/3290575662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f0105bd0 0x7f51f0101d80 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:52.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.855+0000 7f51f7589700 1 -- 172.21.15.16:0/3290575662 >> 172.21.15.16:0/3290575662 conn(0x7f51f0075670 msgr2=0x7f51f00686c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:52.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.855+0000 7f51f7589700 1 -- 172.21.15.16:0/3290575662 shutdown_connections 2024-04-22T15:59:52.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:52.855+0000 7f51f7589700 1 -- 172.21.15.16:0/3290575662 wait complete. 2024-04-22T15:59:53.513 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph health detail' 2024-04-22T15:59:53.729 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:59:53.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:53 smithi027 ceph-mon[83054]: pgmap v229: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:59:53.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:53 smithi027 ceph-mon[83054]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:59:53.850 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:53 smithi016 ceph-mon[110740]: pgmap v229: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:59:53.850 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:53 smithi016 ceph-mon[110740]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T15:59:55.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.323+0000 7f0016dc4700 1 -- 172.21.15.16:0/853730776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0010106cd0 msgr2=0x7f00101070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:55.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.323+0000 7f0016dc4700 1 --2- 172.21.15.16:0/853730776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0010106cd0 0x7f00101070b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0004009910 tx=0x7f0004009c20 comp rx=0 tx=0).stop 2024-04-22T15:59:55.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.323+0000 7f0016dc4700 1 -- 172.21.15.16:0/853730776 shutdown_connections 2024-04-22T15:59:55.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.323+0000 7f0016dc4700 1 --2- 172.21.15.16:0/853730776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0010068580 0x7f00100689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:55.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.323+0000 7f0016dc4700 1 --2- 172.21.15.16:0/853730776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0010106cd0 0x7f00101070b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:55.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.323+0000 7f0016dc4700 1 -- 172.21.15.16:0/853730776 >> 172.21.15.16:0/853730776 conn(0x7f0010075670 msgr2=0x7f0010075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:55.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.323+0000 7f0016dc4700 1 -- 172.21.15.16:0/853730776 shutdown_connections 2024-04-22T15:59:55.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.323+0000 7f0016dc4700 1 -- 172.21.15.16:0/853730776 wait complete. 2024-04-22T15:59:55.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.323+0000 7f0016dc4700 1 Processor -- start 2024-04-22T15:59:55.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.324+0000 7f0016dc4700 1 -- start start 2024-04-22T15:59:55.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.324+0000 7f0016dc4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0010068580 0x7f001019c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:55.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.324+0000 7f0016dc4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0010106cd0 0x7f001019cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:55.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.324+0000 7f0016dc4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001019d230 con 0x7f0010106cd0 2024-04-22T15:59:55.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.324+0000 7f0016dc4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00101966e0 con 0x7f0010068580 2024-04-22T15:59:55.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.324+0000 7f000ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0010106cd0 0x7f001019cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:55.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.324+0000 7f000ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0010106cd0 0x7f001019cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53304/0 (socket says 172.21.15.16:53304) 2024-04-22T15:59:55.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.324+0000 7f000ffff700 1 -- 172.21.15.16:0/1097884950 learned_addr learned my addr 172.21.15.16:0/1097884950 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:59:55.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.324+0000 7f0014b60700 1 --2- 172.21.15.16:0/1097884950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0010068580 0x7f001019c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:55.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.324+0000 7f000ffff700 1 -- 172.21.15.16:0/1097884950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0010068580 msgr2=0x7f001019c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:55.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.324+0000 7f000ffff700 1 --2- 172.21.15.16:0/1097884950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0010068580 0x7f001019c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:55.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.324+0000 7f000ffff700 1 -- 172.21.15.16:0/1097884950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00040095f0 con 0x7f0010106cd0 2024-04-22T15:59:55.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.324+0000 7f0014b60700 1 --2- 172.21.15.16:0/1097884950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0010068580 0x7f001019c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T15:59:55.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.324+0000 7f000ffff700 1 --2- 172.21.15.16:0/1097884950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0010106cd0 0x7f001019cba0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7efffc00d8d0 tx=0x7efffc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:55.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.324+0000 7f000dffb700 1 -- 172.21.15.16:0/1097884950 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efffc009940 con 0x7f0010106cd0 2024-04-22T15:59:55.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.325+0000 7f0016dc4700 1 -- 172.21.15.16:0/1097884950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00101969c0 con 0x7f0010106cd0 2024-04-22T15:59:55.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.325+0000 7f000dffb700 1 -- 172.21.15.16:0/1097884950 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efffc010460 con 0x7f0010106cd0 2024-04-22T15:59:55.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.325+0000 7f000dffb700 1 -- 172.21.15.16:0/1097884950 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efffc00f5f0 con 0x7f0010106cd0 2024-04-22T15:59:55.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.325+0000 7f0016dc4700 1 -- 172.21.15.16:0/1097884950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0010196f10 con 0x7f0010106cd0 2024-04-22T15:59:55.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.326+0000 7f0016dc4700 1 -- 172.21.15.16:0/1097884950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00101091d0 con 0x7f0010106cd0 2024-04-22T15:59:55.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.326+0000 7f000dffb700 1 -- 172.21.15.16:0/1097884950 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efffc0105d0 con 0x7f0010106cd0 2024-04-22T15:59:55.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.327+0000 7f000dffb700 1 --2- 172.21.15.16:0/1097884950 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f000006ecc0 0x7f0000071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:55.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.327+0000 7f000dffb700 1 -- 172.21.15.16:0/1097884950 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efffc08ee80 con 0x7f0010106cd0 2024-04-22T15:59:55.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.327+0000 7f0014b60700 1 --2- 172.21.15.16:0/1097884950 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f000006ecc0 0x7f0000071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:55.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.327+0000 7f0014b60700 1 --2- 172.21.15.16:0/1097884950 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f000006ecc0 0x7f0000071180 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0004006010 tx=0x7f000400b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:55.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.331+0000 7f000dffb700 1 -- 172.21.15.16:0/1097884950 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7efffc05b760 con 0x7f0010106cd0 2024-04-22T15:59:55.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.638+0000 7f0016dc4700 1 -- 172.21.15.16:0/1097884950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0010197130 con 0x7f0010106cd0 2024-04-22T15:59:55.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.639+0000 7f000dffb700 1 -- 172.21.15.16:0/1097884950 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7efffc05b2f0 con 0x7f0010106cd0 2024-04-22T15:59:55.642 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T15:59:55.642 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T15:59:55.642 INFO:teuthology.orchestra.run.smithi016.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T15:59:55.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.641+0000 7f0016dc4700 1 -- 172.21.15.16:0/1097884950 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f000006ecc0 msgr2=0x7f0000071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:55.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.641+0000 7f0016dc4700 1 --2- 172.21.15.16:0/1097884950 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f000006ecc0 0x7f0000071180 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0004006010 tx=0x7f000400b540 comp rx=0 tx=0).stop 2024-04-22T15:59:55.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.641+0000 7f0016dc4700 1 -- 172.21.15.16:0/1097884950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0010106cd0 msgr2=0x7f001019cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:55.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.641+0000 7f0016dc4700 1 --2- 172.21.15.16:0/1097884950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0010106cd0 0x7f001019cba0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7efffc00d8d0 tx=0x7efffc00dc90 comp rx=0 tx=0).stop 2024-04-22T15:59:55.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.641+0000 7f0016dc4700 1 -- 172.21.15.16:0/1097884950 shutdown_connections 2024-04-22T15:59:55.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.641+0000 7f0016dc4700 1 --2- 172.21.15.16:0/1097884950 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f000006ecc0 0x7f0000071180 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:55.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.641+0000 7f0016dc4700 1 --2- 172.21.15.16:0/1097884950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0010068580 0x7f001019c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:55.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.641+0000 7f0016dc4700 1 --2- 172.21.15.16:0/1097884950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0010106cd0 0x7f001019cba0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:55.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.641+0000 7f0016dc4700 1 -- 172.21.15.16:0/1097884950 >> 172.21.15.16:0/1097884950 conn(0x7f0010075670 msgr2=0x7f00100fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:55.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.641+0000 7f0016dc4700 1 -- 172.21.15.16:0/1097884950 shutdown_connections 2024-04-22T15:59:55.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:55.641+0000 7f0016dc4700 1 -- 172.21.15.16:0/1097884950 wait complete. 2024-04-22T15:59:55.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:55 smithi027 ceph-mon[83054]: pgmap v230: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:59:55.914 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:55 smithi016 ceph-mon[110740]: pgmap v230: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T15:59:56.332 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph versions' 2024-04-22T15:59:56.547 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:59:56.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:56 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/1097884950' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:59:56.849 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:56 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/1097884950' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T15:59:57.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:57 smithi016 ceph-mon[110740]: pgmap v231: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:59:57.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:57 smithi027 ceph-mon[83054]: pgmap v231: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T15:59:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.237+0000 7fa2eb34b700 1 -- 172.21.15.16:0/1896944103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2e4101df0 msgr2=0x7fa2e41021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.237+0000 7fa2eb34b700 1 --2- 172.21.15.16:0/1896944103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2e4101df0 0x7fa2e41021d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0009910 tx=0x7fa2e0009c20 comp rx=0 tx=0).stop 2024-04-22T15:59:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.237+0000 7fa2eb34b700 1 -- 172.21.15.16:0/1896944103 shutdown_connections 2024-04-22T15:59:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.237+0000 7fa2eb34b700 1 --2- 172.21.15.16:0/1896944103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2e4102710 0x7fa2e410ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.237+0000 7fa2eb34b700 1 --2- 172.21.15.16:0/1896944103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2e4101df0 0x7fa2e41021d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.237+0000 7fa2eb34b700 1 -- 172.21.15.16:0/1896944103 >> 172.21.15.16:0/1896944103 conn(0x7fa2e40fb620 msgr2=0x7fa2e40fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.237+0000 7fa2eb34b700 1 -- 172.21.15.16:0/1896944103 shutdown_connections 2024-04-22T15:59:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.237+0000 7fa2eb34b700 1 -- 172.21.15.16:0/1896944103 wait complete. 2024-04-22T15:59:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.238+0000 7fa2eb34b700 1 Processor -- start 2024-04-22T15:59:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.238+0000 7fa2eb34b700 1 -- start start 2024-04-22T15:59:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.238+0000 7fa2eb34b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2e4101df0 0x7fa2e419e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.238+0000 7fa2eb34b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2e4102710 0x7fa2e419edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.238+0000 7fa2eb34b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e419f3b0 con 0x7fa2e4102710 2024-04-22T15:59:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.238+0000 7fa2eb34b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e41988f0 con 0x7fa2e4101df0 2024-04-22T15:59:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.238+0000 7fa2e88e6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2e4102710 0x7fa2e419edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.238+0000 7fa2e88e6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2e4102710 0x7fa2e419edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53314/0 (socket says 172.21.15.16:53314) 2024-04-22T15:59:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.238+0000 7fa2e88e6700 1 -- 172.21.15.16:0/1860724428 learned_addr learned my addr 172.21.15.16:0/1860724428 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T15:59:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.238+0000 7fa2e90e7700 1 --2- 172.21.15.16:0/1860724428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2e4101df0 0x7fa2e419e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.238+0000 7fa2e88e6700 1 -- 172.21.15.16:0/1860724428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2e4101df0 msgr2=0x7fa2e419e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.238+0000 7fa2e88e6700 1 --2- 172.21.15.16:0/1860724428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2e4101df0 0x7fa2e419e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.239+0000 7fa2e88e6700 1 -- 172.21.15.16:0/1860724428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e00095f0 con 0x7fa2e4102710 2024-04-22T15:59:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.239+0000 7fa2e88e6700 1 --2- 172.21.15.16:0/1860724428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2e4102710 0x7fa2e419edb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa2d400c8f0 tx=0x7fa2d400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.239+0000 7fa2da7fc700 1 -- 172.21.15.16:0/1860724428 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2d40041f0 con 0x7fa2e4102710 2024-04-22T15:59:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.239+0000 7fa2da7fc700 1 -- 172.21.15.16:0/1860724428 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa2d4004350 con 0x7fa2e4102710 2024-04-22T15:59:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.239+0000 7fa2da7fc700 1 -- 172.21.15.16:0/1860724428 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2d4003d00 con 0x7fa2e4102710 2024-04-22T15:59:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.239+0000 7fa2eb34b700 1 -- 172.21.15.16:0/1860724428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e4198bd0 con 0x7fa2e4102710 2024-04-22T15:59:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.239+0000 7fa2eb34b700 1 -- 172.21.15.16:0/1860724428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e41990d0 con 0x7fa2e4102710 2024-04-22T15:59:58.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.240+0000 7fa2eb34b700 1 -- 172.21.15.16:0/1860724428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e410a760 con 0x7fa2e4102710 2024-04-22T15:59:58.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.241+0000 7fa2da7fc700 1 -- 172.21.15.16:0/1860724428 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa2d4007740 con 0x7fa2e4102710 2024-04-22T15:59:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.241+0000 7fa2da7fc700 1 --2- 172.21.15.16:0/1860724428 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fa2d006ecc0 0x7fa2d0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T15:59:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.241+0000 7fa2da7fc700 1 -- 172.21.15.16:0/1860724428 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa2d408eb20 con 0x7fa2e4102710 2024-04-22T15:59:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.241+0000 7fa2e90e7700 1 --2- 172.21.15.16:0/1860724428 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fa2d006ecc0 0x7fa2d0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T15:59:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.242+0000 7fa2e90e7700 1 --2- 172.21.15.16:0/1860724428 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fa2d006ecc0 0x7fa2d0071180 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0006010 tx=0x7fa2e001a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T15:59:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.246+0000 7fa2da7fc700 1 -- 172.21.15.16:0/1860724428 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa2d405b360 con 0x7fa2e4102710 2024-04-22T15:59:58.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.543+0000 7fa2eb34b700 1 -- 172.21.15.16:0/1860724428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa2e404f2e0 con 0x7fa2e4102710 2024-04-22T15:59:58.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.543+0000 7fa2da7fc700 1 -- 172.21.15.16:0/1860724428 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fa2d405aef0 con 0x7fa2e4102710 2024-04-22T15:59:58.547 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T15:59:58.547 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-22T15:59:58.547 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:59:58.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:59:58.547 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-22T15:59:58.547 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T15:59:58.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:59:58.547 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-22T15:59:58.547 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T15:59:58.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:59:58.548 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-22T15:59:58.548 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T15:59:58.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T15:59:58.548 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-22T15:59:58.548 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T15:59:58.548 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-22T15:59:58.548 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T15:59:58.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.545+0000 7fa2eb34b700 1 -- 172.21.15.16:0/1860724428 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fa2d006ecc0 msgr2=0x7fa2d0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:58.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.545+0000 7fa2eb34b700 1 --2- 172.21.15.16:0/1860724428 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fa2d006ecc0 0x7fa2d0071180 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0006010 tx=0x7fa2e001a040 comp rx=0 tx=0).stop 2024-04-22T15:59:58.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.545+0000 7fa2eb34b700 1 -- 172.21.15.16:0/1860724428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2e4102710 msgr2=0x7fa2e419edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T15:59:58.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.545+0000 7fa2eb34b700 1 --2- 172.21.15.16:0/1860724428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2e4102710 0x7fa2e419edb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa2d400c8f0 tx=0x7fa2d400cc00 comp rx=0 tx=0).stop 2024-04-22T15:59:58.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.545+0000 7fa2eb34b700 1 -- 172.21.15.16:0/1860724428 shutdown_connections 2024-04-22T15:59:58.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.545+0000 7fa2eb34b700 1 --2- 172.21.15.16:0/1860724428 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fa2d006ecc0 0x7fa2d0071180 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:58.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.545+0000 7fa2eb34b700 1 --2- 172.21.15.16:0/1860724428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2e4101df0 0x7fa2e419e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:58.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.545+0000 7fa2eb34b700 1 --2- 172.21.15.16:0/1860724428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2e4102710 0x7fa2e419edb0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T15:59:58.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.545+0000 7fa2eb34b700 1 -- 172.21.15.16:0/1860724428 >> 172.21.15.16:0/1860724428 conn(0x7fa2e40fb620 msgr2=0x7fa2e4100210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T15:59:58.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.546+0000 7fa2eb34b700 1 -- 172.21.15.16:0/1860724428 shutdown_connections 2024-04-22T15:59:58.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T15:59:58.546+0000 7fa2eb34b700 1 -- 172.21.15.16:0/1860724428 wait complete. 2024-04-22T15:59:58.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:58 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/1860724428' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:59.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:58 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/1860724428' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T15:59:59.206 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-22T15:59:59.421 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T15:59:59.877 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 15:59:59 smithi016 ceph-mon[110740]: pgmap v232: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:00.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 15:59:59 smithi027 ceph-mon[83054]: pgmap v232: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:00.816 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:00 smithi016 ceph-mon[110740]: overall HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T16:00:00.816 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:00 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:00:01.025 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:00 smithi027 ceph-mon[83054]: overall HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T16:00:01.026 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:00 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:00:01.052 INFO:teuthology.orchestra.run.smithi016.stdout:wait for servicemap items w/ changing names to refresh 2024-04-22T16:00:01.719 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'sleep 60' 2024-04-22T16:00:01.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:01 smithi016 ceph-mon[110740]: pgmap v233: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:00:01.941 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T16:00:02.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:01 smithi027 ceph-mon[83054]: pgmap v233: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:00:03.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:03 smithi016 ceph-mon[110740]: pgmap v234: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:04.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:03 smithi027 ceph-mon[83054]: pgmap v234: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:05.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:05 smithi016 ceph-mon[110740]: pgmap v235: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:00:06.075 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:05 smithi027 ceph-mon[83054]: pgmap v235: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:00:07.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:07 smithi016 ceph-mon[110740]: pgmap v236: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:08.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:07 smithi027 ceph-mon[83054]: pgmap v236: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:09.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:09 smithi016 ceph-mon[110740]: pgmap v237: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:10.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:09 smithi027 ceph-mon[83054]: pgmap v237: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:11.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:11 smithi016 ceph-mon[110740]: pgmap v238: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:00:12.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:11 smithi027 ceph-mon[83054]: pgmap v238: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:00:13.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:13 smithi016 ceph-mon[110740]: pgmap v239: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:14.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:13 smithi027 ceph-mon[83054]: pgmap v239: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:15.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:15 smithi016 ceph-mon[110740]: pgmap v240: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:00:15.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:15 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:00:16.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:15 smithi027 ceph-mon[83054]: pgmap v240: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:00:16.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:15 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:00:17.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:17 smithi016 ceph-mon[110740]: pgmap v241: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:18.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:17 smithi027 ceph-mon[83054]: pgmap v241: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:19.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:19 smithi016 ceph-mon[110740]: pgmap v242: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:19.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:19 smithi027 ceph-mon[83054]: pgmap v242: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:21.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:21 smithi016 ceph-mon[110740]: pgmap v243: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:00:22.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:21 smithi027 ceph-mon[83054]: pgmap v243: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:00:23.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:23 smithi016 ceph-mon[110740]: pgmap v244: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:24.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:23 smithi027 ceph-mon[83054]: pgmap v244: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:25.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:25 smithi016 ceph-mon[110740]: pgmap v245: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:00:26.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:25 smithi027 ceph-mon[83054]: pgmap v245: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:00:27.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:27 smithi016 ceph-mon[110740]: pgmap v246: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:28.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:27 smithi027 ceph-mon[83054]: pgmap v246: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:29.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:29 smithi016 ceph-mon[110740]: pgmap v247: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:30.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:29 smithi027 ceph-mon[83054]: pgmap v247: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:30.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:00:31.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:30 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:00:31.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:31 smithi016 ceph-mon[110740]: pgmap v248: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:00:32.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:31 smithi027 ceph-mon[83054]: pgmap v248: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:00:33.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:33 smithi016 ceph-mon[110740]: pgmap v249: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:34.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:33 smithi027 ceph-mon[83054]: pgmap v249: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:35.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:35 smithi016 ceph-mon[110740]: pgmap v250: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:00:36.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:35 smithi027 ceph-mon[83054]: pgmap v250: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:00:37.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:37 smithi016 ceph-mon[110740]: pgmap v251: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:38.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:37 smithi027 ceph-mon[83054]: pgmap v251: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:39.910 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:39 smithi027 ceph-mon[83054]: pgmap v252: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:39.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:39 smithi016 ceph-mon[110740]: pgmap v252: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:41.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:41 smithi027 ceph-mon[83054]: pgmap v253: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:00:41.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:41 smithi016 ceph-mon[110740]: pgmap v253: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:00:43.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:43 smithi016 ceph-mon[110740]: pgmap v254: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:44.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:43 smithi027 ceph-mon[83054]: pgmap v254: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:46.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:45 smithi027 ceph-mon[83054]: pgmap v255: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:00:46.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:45 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:00:46.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:45 smithi016 ceph-mon[110740]: pgmap v255: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:00:46.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:45 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:00:47.672 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:47 smithi016 ceph-mon[110740]: pgmap v256: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:47.673 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T16:00:47.673 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T16:00:48.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:47 smithi027 ceph-mon[83054]: pgmap v256: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:48.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T16:00:48.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T16:00:48.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:47 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T16:00:48.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:47 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T16:00:49.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:48 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T16:00:49.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:48 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T16:00:49.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:48 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T16:00:49.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:48 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T16:00:49.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:48 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T16:00:49.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:48 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T16:00:50.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:49 smithi027 ceph-mon[83054]: pgmap v257: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:50.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:49 smithi016 ceph-mon[110740]: pgmap v257: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:51.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:51 smithi027 ceph-mon[83054]: pgmap v258: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:00:52.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:51 smithi016 ceph-mon[110740]: pgmap v258: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:00:54.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:53 smithi027 ceph-mon[83054]: pgmap v259: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:54.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:53 smithi016 ceph-mon[110740]: pgmap v259: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:56.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:55 smithi027 ceph-mon[83054]: pgmap v260: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:00:56.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:55 smithi016 ceph-mon[110740]: pgmap v260: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:00:58.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:57 smithi027 ceph-mon[83054]: pgmap v261: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:00:58.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:57 smithi016 ceph-mon[110740]: pgmap v261: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:00.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:00:59 smithi027 ceph-mon[83054]: pgmap v262: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:00.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:00:59 smithi016 ceph-mon[110740]: pgmap v262: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:01.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:00 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:01:01.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:00 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:01:01.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:01 smithi027 ceph-mon[83054]: pgmap v263: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:01:02.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:01 smithi016 ceph-mon[110740]: pgmap v263: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:01:03.749 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:03 smithi016 ceph-mon[110740]: pgmap v264: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:04.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:03 smithi027 ceph-mon[83054]: pgmap v264: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:04.277 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph orch ps' 2024-04-22T16:01:04.497 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T16:01:05.817 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:05 smithi016 ceph-mon[110740]: pgmap v265: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:01:06.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.056+0000 7f87f0a8c700 1 -- 172.21.15.16:0/4229977350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ec1027a0 msgr2=0x7f87ec10ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:06.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.056+0000 7f87f0a8c700 1 --2- 172.21.15.16:0/4229977350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ec1027a0 0x7f87ec10ac90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f87d4009910 tx=0x7f87d4009c20 comp rx=0 tx=0).stop 2024-04-22T16:01:06.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.056+0000 7f87f0a8c700 1 -- 172.21.15.16:0/4229977350 shutdown_connections 2024-04-22T16:01:06.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.056+0000 7f87f0a8c700 1 --2- 172.21.15.16:0/4229977350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ec1027a0 0x7f87ec10ac90 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:06.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.056+0000 7f87f0a8c700 1 --2- 172.21.15.16:0/4229977350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87ec101e80 0x7f87ec102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:06.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.056+0000 7f87f0a8c700 1 -- 172.21.15.16:0/4229977350 >> 172.21.15.16:0/4229977350 conn(0x7f87ec0fb650 msgr2=0x7f87ec0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:06.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.056+0000 7f87f0a8c700 1 -- 172.21.15.16:0/4229977350 shutdown_connections 2024-04-22T16:01:06.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.056+0000 7f87f0a8c700 1 -- 172.21.15.16:0/4229977350 wait complete. 2024-04-22T16:01:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.057+0000 7f87f0a8c700 1 Processor -- start 2024-04-22T16:01:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.057+0000 7f87f0a8c700 1 -- start start 2024-04-22T16:01:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.058+0000 7f87f0a8c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87ec101e80 0x7f87ec0ffae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.058+0000 7f87f0a8c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ec1027a0 0x7f87ec100020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.058+0000 7f87f0a8c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ec100560 con 0x7f87ec1027a0 2024-04-22T16:01:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.058+0000 7f87f0a8c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ec1006a0 con 0x7f87ec101e80 2024-04-22T16:01:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.058+0000 7f87e1d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ec1027a0 0x7f87ec100020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.058+0000 7f87e1d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ec1027a0 0x7f87ec100020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55808/0 (socket says 172.21.15.16:55808) 2024-04-22T16:01:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.058+0000 7f87e1d9b700 1 -- 172.21.15.16:0/2557451289 learned_addr learned my addr 172.21.15.16:0/2557451289 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.058+0000 7f87ea59c700 1 --2- 172.21.15.16:0/2557451289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87ec101e80 0x7f87ec0ffae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.058+0000 7f87e1d9b700 1 -- 172.21.15.16:0/2557451289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87ec101e80 msgr2=0x7f87ec0ffae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.058+0000 7f87e1d9b700 1 --2- 172.21.15.16:0/2557451289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87ec101e80 0x7f87ec0ffae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:06.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.058+0000 7f87e1d9b700 1 -- 172.21.15.16:0/2557451289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d40095f0 con 0x7f87ec1027a0 2024-04-22T16:01:06.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.058+0000 7f87e1d9b700 1 --2- 172.21.15.16:0/2557451289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ec1027a0 0x7f87ec100020 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f87d400bd70 tx=0x7f87d400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:06.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.058+0000 7f87e3fff700 1 -- 172.21.15.16:0/2557451289 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87d401d070 con 0x7f87ec1027a0 2024-04-22T16:01:06.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.058+0000 7f87e3fff700 1 -- 172.21.15.16:0/2557451289 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f87d4005dc0 con 0x7f87ec1027a0 2024-04-22T16:01:06.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.058+0000 7f87e3fff700 1 -- 172.21.15.16:0/2557451289 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87d4017400 con 0x7f87ec1027a0 2024-04-22T16:01:06.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.058+0000 7f87f0a8c700 1 -- 172.21.15.16:0/2557451289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87ec100920 con 0x7f87ec1027a0 2024-04-22T16:01:06.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.059+0000 7f87f0a8c700 1 -- 172.21.15.16:0/2557451289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87ec1a2df0 con 0x7f87ec1027a0 2024-04-22T16:01:06.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.059+0000 7f87f0a8c700 1 -- 172.21.15.16:0/2557451289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87ec108390 con 0x7f87ec1027a0 2024-04-22T16:01:06.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.060+0000 7f87e3fff700 1 -- 172.21.15.16:0/2557451289 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f87d400f460 con 0x7f87ec1027a0 2024-04-22T16:01:06.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.060+0000 7f87e3fff700 1 --2- 172.21.15.16:0/2557451289 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f87d806ecc0 0x7f87d8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:06.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.061+0000 7f87e3fff700 1 -- 172.21.15.16:0/2557451289 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f87d4090120 con 0x7f87ec1027a0 2024-04-22T16:01:06.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.061+0000 7f87ea59c700 1 --2- 172.21.15.16:0/2557451289 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f87d806ecc0 0x7f87d8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:06.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.061+0000 7f87ea59c700 1 --2- 172.21.15.16:0/2557451289 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f87d806ecc0 0x7f87d8071180 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f87dc005950 tx=0x7f87dc00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:06.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.065+0000 7f87e3fff700 1 -- 172.21.15.16:0/2557451289 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f87d4059010 con 0x7f87ec1027a0 2024-04-22T16:01:06.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:05 smithi027 ceph-mon[83054]: pgmap v265: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:01:06.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.285+0000 7f87f0a8c700 1 -- 172.21.15.16:0/2557451289 --> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f87ec0611d0 con 0x7f87d806ecc0 2024-04-22T16:01:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.296+0000 7f87e3fff700 1 -- 172.21.15.16:0/2557451289 <== mgr.34104 v2:172.21.15.16:6800/583399364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f87ec0611d0 con 0x7f87d806ecc0 2024-04-22T16:01:06.299 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T16:01:06.299 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.smithi016 smithi016 *:9093,9094 running (9m) 85s ago 20m 25.9M - 0.25.0 c8568f914cd2 ea0e0217d6f9 2024-04-22T16:01:06.299 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi016 smithi016 running (96s) 85s ago 20m 7810k - 19.0.0-3223-g5883d494 3d983c26e278 7201ff7ad79e 2024-04-22T16:01:06.300 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-exporter.smithi027 smithi027 running (90s) 87s ago 19m 9068k - 19.0.0-3223-g5883d494 3d983c26e278 aeeac9eb9e02 2024-04-22T16:01:06.300 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi016 smithi016 running (6m) 85s ago 20m 6665k - 19.0.0-3223-g5883d494 3d983c26e278 1feb008449ed 2024-04-22T16:01:06.300 INFO:teuthology.orchestra.run.smithi016.stdout:crash.smithi027 smithi027 running (6m) 87s ago 19m 6656k - 19.0.0-3223-g5883d494 3d983c26e278 c7f862c6da00 2024-04-22T16:01:06.300 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.smithi016 smithi016 *:3000 running (8m) 85s ago 19m 82.0M - 9.4.12 2bacad6d85d8 388a3c9c1b3f 2024-04-22T16:01:06.300 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.fkkejm smithi016 running (2m) 85s ago 14m 90.8M - 19.0.0-3223-g5883d494 3d983c26e278 e23d9fc30059 2024-04-22T16:01:06.300 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi016.pgvphu smithi016 running (2m) 85s ago 14m 199M - 19.0.0-3223-g5883d494 3d983c26e278 4777936b5956 2024-04-22T16:01:06.300 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.dgqyes smithi027 running (112s) 87s ago 14m 13.7M - 19.0.0-3223-g5883d494 3d983c26e278 32404711622f 2024-04-22T16:01:06.300 INFO:teuthology.orchestra.run.smithi016.stdout:mds.cephfs.smithi027.iobjds smithi027 running (2m) 87s ago 14m 15.2M - 19.0.0-3223-g5883d494 3d983c26e278 8dd739415c15 2024-04-22T16:01:06.300 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi016.rnsnnr smithi016 *:8443,9283,8765 running (11m) 85s ago 21m 512M - 19.0.0-3223-g5883d494 3d983c26e278 09b4fe285bc4 2024-04-22T16:01:06.300 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.smithi027.ukdgzs smithi027 *:8443,9283,8765 running (10m) 87s ago 19m 452M - 19.0.0-3223-g5883d494 3d983c26e278 7548da9b5eb9 2024-04-22T16:01:06.300 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi016 smithi016 running (7m) 85s ago 21m 59.0M 2048M 19.0.0-3223-g5883d494 3d983c26e278 4ae529a67806 2024-04-22T16:01:06.300 INFO:teuthology.orchestra.run.smithi016.stdout:mon.smithi027 smithi027 running (7m) 87s ago 18m 46.2M 2048M 19.0.0-3223-g5883d494 3d983c26e278 f29fcb888801 2024-04-22T16:01:06.300 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi016 smithi016 *:9100 running (10m) 85s ago 20m 15.6M - 1.5.0 0da6a335fe13 35e46e4d2f3f 2024-04-22T16:01:06.300 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.smithi027 smithi027 *:9100 running (10m) 87s ago 19m 15.0M - 1.5.0 0da6a335fe13 e682a7672e9b 2024-04-22T16:01:06.300 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (6m) 85s ago 18m 97.2M 1217M 19.0.0-3223-g5883d494 3d983c26e278 79217eb383a2 2024-04-22T16:01:06.300 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (5m) 85s ago 17m 103M 1217M 19.0.0-3223-g5883d494 3d983c26e278 ec1a45b3c53e 2024-04-22T16:01:06.301 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (4m) 85s ago 17m 87.8M 1217M 19.0.0-3223-g5883d494 3d983c26e278 48bb459fd021 2024-04-22T16:01:06.301 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi027 running (4m) 87s ago 16m 108M 2240M 19.0.0-3223-g5883d494 3d983c26e278 81034bf22d3f 2024-04-22T16:01:06.301 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (3m) 87s ago 16m 100M 2240M 19.0.0-3223-g5883d494 3d983c26e278 93b660e5ecb6 2024-04-22T16:01:06.301 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi027 running (3m) 87s ago 16m 96.2M 2240M 19.0.0-3223-g5883d494 3d983c26e278 d857c152994a 2024-04-22T16:01:06.301 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.smithi016 smithi016 *:9095 running (9m) 85s ago 19m 80.3M - 2.43.0 a07b618ecd1d 373eaadee82d 2024-04-22T16:01:06.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.298+0000 7f87f0a8c700 1 -- 172.21.15.16:0/2557451289 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f87d806ecc0 msgr2=0x7f87d8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:06.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.298+0000 7f87f0a8c700 1 --2- 172.21.15.16:0/2557451289 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f87d806ecc0 0x7f87d8071180 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f87dc005950 tx=0x7f87dc00b410 comp rx=0 tx=0).stop 2024-04-22T16:01:06.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.298+0000 7f87f0a8c700 1 -- 172.21.15.16:0/2557451289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ec1027a0 msgr2=0x7f87ec100020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:06.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.298+0000 7f87f0a8c700 1 --2- 172.21.15.16:0/2557451289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ec1027a0 0x7f87ec100020 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f87d400bd70 tx=0x7f87d400bda0 comp rx=0 tx=0).stop 2024-04-22T16:01:06.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.298+0000 7f87f0a8c700 1 -- 172.21.15.16:0/2557451289 shutdown_connections 2024-04-22T16:01:06.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.299+0000 7f87f0a8c700 1 --2- 172.21.15.16:0/2557451289 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f87d806ecc0 0x7f87d8071180 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:06.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.299+0000 7f87f0a8c700 1 --2- 172.21.15.16:0/2557451289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87ec101e80 0x7f87ec0ffae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:06.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.299+0000 7f87f0a8c700 1 --2- 172.21.15.16:0/2557451289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ec1027a0 0x7f87ec100020 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:06.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.299+0000 7f87f0a8c700 1 -- 172.21.15.16:0/2557451289 >> 172.21.15.16:0/2557451289 conn(0x7f87ec0fb650 msgr2=0x7f87ec1054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:06.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.299+0000 7f87f0a8c700 1 -- 172.21.15.16:0/2557451289 shutdown_connections 2024-04-22T16:01:06.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:06.299+0000 7f87f0a8c700 1 -- 172.21.15.16:0/2557451289 wait complete. 2024-04-22T16:01:07.171 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph versions' 2024-04-22T16:01:07.390 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T16:01:07.819 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:07 smithi016 ceph-mon[110740]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T16:01:07.820 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:07 smithi016 ceph-mon[110740]: pgmap v266: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:08.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:07 smithi027 ceph-mon[83054]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T16:01:08.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:07 smithi027 ceph-mon[83054]: pgmap v266: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:08.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.862+0000 7f470277d700 1 -- 172.21.15.16:0/2814594852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46fc100e90 msgr2=0x7f46fc101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:08.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.862+0000 7f470277d700 1 --2- 172.21.15.16:0/2814594852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46fc100e90 0x7f46fc101270 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f46ec009910 tx=0x7f46ec009c20 comp rx=0 tx=0).stop 2024-04-22T16:01:08.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.862+0000 7f470277d700 1 -- 172.21.15.16:0/2814594852 shutdown_connections 2024-04-22T16:01:08.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.862+0000 7f470277d700 1 --2- 172.21.15.16:0/2814594852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46fc101840 0x7f46fc105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:08.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.862+0000 7f470277d700 1 --2- 172.21.15.16:0/2814594852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46fc100e90 0x7f46fc101270 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:08.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.862+0000 7f470277d700 1 -- 172.21.15.16:0/2814594852 >> 172.21.15.16:0/2814594852 conn(0x7f46fc0fc740 msgr2=0x7f46fc0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:08.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.862+0000 7f470277d700 1 -- 172.21.15.16:0/2814594852 shutdown_connections 2024-04-22T16:01:08.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.862+0000 7f470277d700 1 -- 172.21.15.16:0/2814594852 wait complete. 2024-04-22T16:01:08.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.862+0000 7f470277d700 1 Processor -- start 2024-04-22T16:01:08.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.862+0000 7f470277d700 1 -- start start 2024-04-22T16:01:08.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.863+0000 7f470277d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46fc100e90 0x7f46fc19c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:08.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.863+0000 7f470277d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46fc101840 0x7f46fc19ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:08.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.863+0000 7f470277d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc19d360 con 0x7f46fc100e90 2024-04-22T16:01:08.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.863+0000 7f470277d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc196810 con 0x7f46fc101840 2024-04-22T16:01:08.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.863+0000 7f46fbfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46fc100e90 0x7f46fc19c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:08.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.863+0000 7f46fbfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46fc100e90 0x7f46fc19c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55822/0 (socket says 172.21.15.16:55822) 2024-04-22T16:01:08.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.863+0000 7f46fbfff700 1 -- 172.21.15.16:0/3708974630 learned_addr learned my addr 172.21.15.16:0/3708974630 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:08.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.863+0000 7f46fb7fe700 1 --2- 172.21.15.16:0/3708974630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46fc101840 0x7f46fc19ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:08.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.863+0000 7f46fbfff700 1 -- 172.21.15.16:0/3708974630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46fc101840 msgr2=0x7f46fc19ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:08.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.863+0000 7f46fbfff700 1 --2- 172.21.15.16:0/3708974630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46fc101840 0x7f46fc19ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:08.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.863+0000 7f46fbfff700 1 -- 172.21.15.16:0/3708974630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ec0095f0 con 0x7f46fc100e90 2024-04-22T16:01:08.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.863+0000 7f46fbfff700 1 --2- 172.21.15.16:0/3708974630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46fc100e90 0x7f46fc19c790 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f46ec006010 tx=0x7f46ec00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:08.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.864+0000 7f46f97fa700 1 -- 172.21.15.16:0/3708974630 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46ec01d070 con 0x7f46fc100e90 2024-04-22T16:01:08.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.864+0000 7f470277d700 1 -- 172.21.15.16:0/3708974630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46fc196a90 con 0x7f46fc100e90 2024-04-22T16:01:08.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.864+0000 7f46f97fa700 1 -- 172.21.15.16:0/3708974630 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f46ec004fb0 con 0x7f46fc100e90 2024-04-22T16:01:08.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.864+0000 7f46f97fa700 1 -- 172.21.15.16:0/3708974630 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46ec022ea0 con 0x7f46fc100e90 2024-04-22T16:01:08.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.864+0000 7f470277d700 1 -- 172.21.15.16:0/3708974630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46fc196f80 con 0x7f46fc100e90 2024-04-22T16:01:08.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.865+0000 7f470277d700 1 -- 172.21.15.16:0/3708974630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46fc109230 con 0x7f46fc100e90 2024-04-22T16:01:08.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.865+0000 7f46f97fa700 1 -- 172.21.15.16:0/3708974630 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f46ec022470 con 0x7f46fc100e90 2024-04-22T16:01:08.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.866+0000 7f46f97fa700 1 --2- 172.21.15.16:0/3708974630 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f46e806ecc0 0x7f46e8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:08.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.866+0000 7f46f97fa700 1 -- 172.21.15.16:0/3708974630 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f46ec090400 con 0x7f46fc100e90 2024-04-22T16:01:08.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.866+0000 7f46fb7fe700 1 --2- 172.21.15.16:0/3708974630 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f46e806ecc0 0x7f46e8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:08.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.866+0000 7f46fb7fe700 1 --2- 172.21.15.16:0/3708974630 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f46e806ecc0 0x7f46e8071180 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f46e4006fd0 tx=0x7f46e4008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:08.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:08.870+0000 7f46f97fa700 1 -- 172.21.15.16:0/3708974630 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f46ec05cd90 con 0x7f46fc100e90 2024-04-22T16:01:09.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:09.171+0000 7f470277d700 1 -- 172.21.15.16:0/3708974630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f46fc04f2e0 con 0x7f46fc100e90 2024-04-22T16:01:09.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:09.171+0000 7f46f97fa700 1 -- 172.21.15.16:0/3708974630 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f46ec022720 con 0x7f46fc100e90 2024-04-22T16:01:09.175 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-22T16:01:09.175 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-22T16:01:09.175 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T16:01:09.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T16:01:09.175 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-22T16:01:09.175 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 2 2024-04-22T16:01:09.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T16:01:09.175 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-22T16:01:09.175 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 6 2024-04-22T16:01:09.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T16:01:09.176 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-22T16:01:09.176 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 4 2024-04-22T16:01:09.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-22T16:01:09.176 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-22T16:01:09.176 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)": 14 2024-04-22T16:01:09.176 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-22T16:01:09.176 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-22T16:01:09.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:09.173+0000 7f470277d700 1 -- 172.21.15.16:0/3708974630 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f46e806ecc0 msgr2=0x7f46e8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:09.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:09.173+0000 7f470277d700 1 --2- 172.21.15.16:0/3708974630 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f46e806ecc0 0x7f46e8071180 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f46e4006fd0 tx=0x7f46e4008040 comp rx=0 tx=0).stop 2024-04-22T16:01:09.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:09.173+0000 7f470277d700 1 -- 172.21.15.16:0/3708974630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46fc100e90 msgr2=0x7f46fc19c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:09.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:09.173+0000 7f470277d700 1 --2- 172.21.15.16:0/3708974630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46fc100e90 0x7f46fc19c790 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f46ec006010 tx=0x7f46ec00be20 comp rx=0 tx=0).stop 2024-04-22T16:01:09.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:09.173+0000 7f470277d700 1 -- 172.21.15.16:0/3708974630 shutdown_connections 2024-04-22T16:01:09.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:09.173+0000 7f470277d700 1 --2- 172.21.15.16:0/3708974630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46fc100e90 0x7f46fc19c790 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:09.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:09.173+0000 7f470277d700 1 --2- 172.21.15.16:0/3708974630 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f46e806ecc0 0x7f46e8071180 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:09.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:09.173+0000 7f470277d700 1 --2- 172.21.15.16:0/3708974630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46fc101840 0x7f46fc19ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:09.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:09.173+0000 7f470277d700 1 -- 172.21.15.16:0/3708974630 >> 172.21.15.16:0/3708974630 conn(0x7f46fc0fc740 msgr2=0x7f46fc0fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:09.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:09.173+0000 7f470277d700 1 -- 172.21.15.16:0/3708974630 shutdown_connections 2024-04-22T16:01:09.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:09.173+0000 7f470277d700 1 -- 172.21.15.16:0/3708974630 wait complete. 2024-04-22T16:01:09.764 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:09 smithi016 ceph-mon[110740]: pgmap v267: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:09.765 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:09 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/3708974630' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T16:01:09.789 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-22T16:01:10.004 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T16:01:10.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:09 smithi027 ceph-mon[83054]: pgmap v267: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:10.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:09 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/3708974630' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T16:01:11.733 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:11 smithi016 ceph-mon[110740]: pgmap v268: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:01:11.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.729+0000 7f5b0f015700 1 -- 172.21.15.16:0/1725287356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b08101830 msgr2=0x7f5b08105880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:11.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.729+0000 7f5b0f015700 1 --2- 172.21.15.16:0/1725287356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b08101830 0x7f5b08105880 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5af4009910 tx=0x7f5af4009c20 comp rx=0 tx=0).stop 2024-04-22T16:01:11.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.730+0000 7f5b0f015700 1 -- 172.21.15.16:0/1725287356 shutdown_connections 2024-04-22T16:01:11.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.730+0000 7f5b0f015700 1 --2- 172.21.15.16:0/1725287356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b08101830 0x7f5b08105880 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:11.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.730+0000 7f5b0f015700 1 --2- 172.21.15.16:0/1725287356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b08100e80 0x7f5b08101260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:11.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.730+0000 7f5b0f015700 1 -- 172.21.15.16:0/1725287356 >> 172.21.15.16:0/1725287356 conn(0x7f5b080fc710 msgr2=0x7f5b080feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:11.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.730+0000 7f5b0f015700 1 -- 172.21.15.16:0/1725287356 shutdown_connections 2024-04-22T16:01:11.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.730+0000 7f5b0f015700 1 -- 172.21.15.16:0/1725287356 wait complete. 2024-04-22T16:01:11.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.730+0000 7f5b0f015700 1 Processor -- start 2024-04-22T16:01:11.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.730+0000 7f5b0f015700 1 -- start start 2024-04-22T16:01:11.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.731+0000 7f5b0f015700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b08100e80 0x7f5b0819c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:11.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.731+0000 7f5b0f015700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b08101830 0x7f5b0819cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:11.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.731+0000 7f5b0f015700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0819d210 con 0x7f5b08100e80 2024-04-22T16:01:11.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.731+0000 7f5b0f015700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b081966c0 con 0x7f5b08101830 2024-04-22T16:01:11.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.731+0000 7f5b0cdb1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b08100e80 0x7f5b0819c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:11.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.731+0000 7f5b0cdb1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b08100e80 0x7f5b0819c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38070/0 (socket says 172.21.15.16:38070) 2024-04-22T16:01:11.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.731+0000 7f5b0cdb1700 1 -- 172.21.15.16:0/3020987998 learned_addr learned my addr 172.21.15.16:0/3020987998 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:11.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.731+0000 7f5b07fff700 1 --2- 172.21.15.16:0/3020987998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b08101830 0x7f5b0819cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:11.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.731+0000 7f5b0cdb1700 1 -- 172.21.15.16:0/3020987998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b08101830 msgr2=0x7f5b0819cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:11.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.731+0000 7f5b0cdb1700 1 --2- 172.21.15.16:0/3020987998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b08101830 0x7f5b0819cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:11.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.731+0000 7f5b0cdb1700 1 -- 172.21.15.16:0/3020987998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af40095f0 con 0x7f5b08100e80 2024-04-22T16:01:11.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.731+0000 7f5b0cdb1700 1 --2- 172.21.15.16:0/3020987998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b08100e80 0x7f5b0819c640 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5afc00d8d0 tx=0x7f5afc00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:11.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.731+0000 7f5b05ffb700 1 -- 172.21.15.16:0/3020987998 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5afc0098a0 con 0x7f5b08100e80 2024-04-22T16:01:11.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.731+0000 7f5b05ffb700 1 -- 172.21.15.16:0/3020987998 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5afc010460 con 0x7f5b08100e80 2024-04-22T16:01:11.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.731+0000 7f5b0f015700 1 -- 172.21.15.16:0/3020987998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b081969a0 con 0x7f5b08100e80 2024-04-22T16:01:11.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.731+0000 7f5b05ffb700 1 -- 172.21.15.16:0/3020987998 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5afc00f5f0 con 0x7f5b08100e80 2024-04-22T16:01:11.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.732+0000 7f5b0f015700 1 -- 172.21.15.16:0/3020987998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b08196ef0 con 0x7f5b08100e80 2024-04-22T16:01:11.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.733+0000 7f5b0f015700 1 -- 172.21.15.16:0/3020987998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b08109280 con 0x7f5b08100e80 2024-04-22T16:01:11.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.733+0000 7f5b05ffb700 1 -- 172.21.15.16:0/3020987998 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5afc009a00 con 0x7f5b08100e80 2024-04-22T16:01:11.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.734+0000 7f5b05ffb700 1 --2- 172.21.15.16:0/3020987998 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f5af806ecc0 0x7f5af8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:11.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.734+0000 7f5b05ffb700 1 -- 172.21.15.16:0/3020987998 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5afc08ed30 con 0x7f5b08100e80 2024-04-22T16:01:11.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.734+0000 7f5b07fff700 1 --2- 172.21.15.16:0/3020987998 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f5af806ecc0 0x7f5af8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:11.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.734+0000 7f5b07fff700 1 --2- 172.21.15.16:0/3020987998 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f5af806ecc0 0x7f5af8071180 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5b08197b30 tx=0x7f5af401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:11.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:11.739+0000 7f5b05ffb700 1 -- 172.21.15.16:0/3020987998 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5afc05b610 con 0x7f5b08100e80 2024-04-22T16:01:12.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:12.036+0000 7f5b0f015700 1 -- 172.21.15.16:0/3020987998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5b0804f2e0 con 0x7f5b08100e80 2024-04-22T16:01:12.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:12.036+0000 7f5b05ffb700 1 -- 172.21.15.16:0/3020987998 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f5afc05b1a0 con 0x7f5b08100e80 2024-04-22T16:01:12.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:12.038+0000 7f5b0f015700 1 -- 172.21.15.16:0/3020987998 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f5af806ecc0 msgr2=0x7f5af8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:12.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:12.038+0000 7f5b0f015700 1 --2- 172.21.15.16:0/3020987998 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f5af806ecc0 0x7f5af8071180 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5b08197b30 tx=0x7f5af401a040 comp rx=0 tx=0).stop 2024-04-22T16:01:12.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:12.038+0000 7f5b0f015700 1 -- 172.21.15.16:0/3020987998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b08100e80 msgr2=0x7f5b0819c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:12.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:12.038+0000 7f5b0f015700 1 --2- 172.21.15.16:0/3020987998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b08100e80 0x7f5b0819c640 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5afc00d8d0 tx=0x7f5afc00dbe0 comp rx=0 tx=0).stop 2024-04-22T16:01:12.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:12.039+0000 7f5b0f015700 1 -- 172.21.15.16:0/3020987998 shutdown_connections 2024-04-22T16:01:12.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:12.039+0000 7f5b0f015700 1 --2- 172.21.15.16:0/3020987998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b08100e80 0x7f5b0819c640 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:12.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:12.039+0000 7f5b0f015700 1 --2- 172.21.15.16:0/3020987998 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f5af806ecc0 0x7f5af8071180 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:12.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:12.039+0000 7f5b0f015700 1 --2- 172.21.15.16:0/3020987998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b08101830 0x7f5b0819cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:12.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:12.039+0000 7f5b0f015700 1 -- 172.21.15.16:0/3020987998 >> 172.21.15.16:0/3020987998 conn(0x7f5b080fc710 msgr2=0x7f5b080fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:12.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:12.039+0000 7f5b0f015700 1 -- 172.21.15.16:0/3020987998 shutdown_connections 2024-04-22T16:01:12.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:12.039+0000 7f5b0f015700 1 -- 172.21.15.16:0/3020987998 wait complete. 2024-04-22T16:01:12.052 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-22T16:01:12.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:11 smithi027 ceph-mon[83054]: pgmap v268: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:01:12.653 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -e sha1=5883d494d3973b30b3f03e4a3f0e1e06e3c131aa -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-22T16:01:12.866 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T16:01:12.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:12 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/3020987998' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T16:01:13.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:12 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/3020987998' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T16:01:13.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:13 smithi016 ceph-mon[110740]: pgmap v269: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:14.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:13 smithi027 ceph-mon[83054]: pgmap v269: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.340+0000 7f43f90f7700 1 -- 172.21.15.16:0/1036646857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43f4102780 msgr2=0x7f43f410ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.340+0000 7f43f90f7700 1 --2- 172.21.15.16:0/1036646857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43f4102780 0x7f43f410ac70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f43e8009910 tx=0x7f43e8009c20 comp rx=0 tx=0).stop 2024-04-22T16:01:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.340+0000 7f43f90f7700 1 -- 172.21.15.16:0/1036646857 shutdown_connections 2024-04-22T16:01:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.340+0000 7f43f90f7700 1 --2- 172.21.15.16:0/1036646857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43f4102780 0x7f43f410ac70 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.340+0000 7f43f90f7700 1 --2- 172.21.15.16:0/1036646857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43f4101e60 0x7f43f4102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.340+0000 7f43f90f7700 1 -- 172.21.15.16:0/1036646857 >> 172.21.15.16:0/1036646857 conn(0x7f43f40fb650 msgr2=0x7f43f40fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.340+0000 7f43f90f7700 1 -- 172.21.15.16:0/1036646857 shutdown_connections 2024-04-22T16:01:14.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.340+0000 7f43f90f7700 1 -- 172.21.15.16:0/1036646857 wait complete. 2024-04-22T16:01:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.340+0000 7f43f90f7700 1 Processor -- start 2024-04-22T16:01:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.341+0000 7f43f90f7700 1 -- start start 2024-04-22T16:01:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.341+0000 7f43f90f7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43f4101e60 0x7f43f419c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.341+0000 7f43f90f7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43f4102780 0x7f43f419cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:14.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.341+0000 7f43f90f7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f419d210 con 0x7f43f4101e60 2024-04-22T16:01:14.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.341+0000 7f43f90f7700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f41966c0 con 0x7f43f4102780 2024-04-22T16:01:14.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.341+0000 7f43f2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43f4101e60 0x7f43f419c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:14.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.341+0000 7f43f2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43f4101e60 0x7f43f419c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38084/0 (socket says 172.21.15.16:38084) 2024-04-22T16:01:14.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.341+0000 7f43f2d9d700 1 -- 172.21.15.16:0/2258881587 learned_addr learned my addr 172.21.15.16:0/2258881587 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:14.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.341+0000 7f43f259c700 1 --2- 172.21.15.16:0/2258881587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43f4102780 0x7f43f419cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:14.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.341+0000 7f43f2d9d700 1 -- 172.21.15.16:0/2258881587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43f4102780 msgr2=0x7f43f419cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:14.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.341+0000 7f43f2d9d700 1 --2- 172.21.15.16:0/2258881587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43f4102780 0x7f43f419cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:14.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.341+0000 7f43f2d9d700 1 -- 172.21.15.16:0/2258881587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e80095f0 con 0x7f43f4101e60 2024-04-22T16:01:14.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.342+0000 7f43f2d9d700 1 --2- 172.21.15.16:0/2258881587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43f4101e60 0x7f43f419c640 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f43e400d900 tx=0x7f43e400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:14.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.342+0000 7f43dbfff700 1 -- 172.21.15.16:0/2258881587 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43e40041f0 con 0x7f43f4101e60 2024-04-22T16:01:14.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.342+0000 7f43dbfff700 1 -- 172.21.15.16:0/2258881587 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f43e4004cf0 con 0x7f43f4101e60 2024-04-22T16:01:14.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.342+0000 7f43f90f7700 1 -- 172.21.15.16:0/2258881587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f41969a0 con 0x7f43f4101e60 2024-04-22T16:01:14.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.342+0000 7f43dbfff700 1 -- 172.21.15.16:0/2258881587 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43e400b750 con 0x7f43f4101e60 2024-04-22T16:01:14.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.342+0000 7f43f90f7700 1 -- 172.21.15.16:0/2258881587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f4196ef0 con 0x7f43f4101e60 2024-04-22T16:01:14.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.343+0000 7f43f90f7700 1 -- 172.21.15.16:0/2258881587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43f4108300 con 0x7f43f4101e60 2024-04-22T16:01:14.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.343+0000 7f43dbfff700 1 -- 172.21.15.16:0/2258881587 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f43e4004350 con 0x7f43f4101e60 2024-04-22T16:01:14.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.344+0000 7f43dbfff700 1 --2- 172.21.15.16:0/2258881587 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f43dc06ecc0 0x7f43dc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:14.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.344+0000 7f43dbfff700 1 -- 172.21.15.16:0/2258881587 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f43e408efb0 con 0x7f43f4101e60 2024-04-22T16:01:14.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.344+0000 7f43f259c700 1 --2- 172.21.15.16:0/2258881587 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f43dc06ecc0 0x7f43dc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:14.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.344+0000 7f43f259c700 1 --2- 172.21.15.16:0/2258881587 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f43dc06ecc0 0x7f43dc071180 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f43e80059d0 tx=0x7f43e8005960 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:14.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.349+0000 7f43dbfff700 1 -- 172.21.15.16:0/2258881587 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f43e405b890 con 0x7f43f4101e60 2024-04-22T16:01:14.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.645+0000 7f43f90f7700 1 -- 172.21.15.16:0/2258881587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f43f404f2e0 con 0x7f43f4101e60 2024-04-22T16:01:14.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.645+0000 7f43dbfff700 1 -- 172.21.15.16:0/2258881587 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f43e405b420 con 0x7f43f4101e60 2024-04-22T16:01:14.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.647+0000 7f43f90f7700 1 -- 172.21.15.16:0/2258881587 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f43dc06ecc0 msgr2=0x7f43dc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:14.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.647+0000 7f43f90f7700 1 --2- 172.21.15.16:0/2258881587 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f43dc06ecc0 0x7f43dc071180 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f43e80059d0 tx=0x7f43e8005960 comp rx=0 tx=0).stop 2024-04-22T16:01:14.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.647+0000 7f43f90f7700 1 -- 172.21.15.16:0/2258881587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43f4101e60 msgr2=0x7f43f419c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:14.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.647+0000 7f43f90f7700 1 --2- 172.21.15.16:0/2258881587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43f4101e60 0x7f43f419c640 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f43e400d900 tx=0x7f43e400dcc0 comp rx=0 tx=0).stop 2024-04-22T16:01:14.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.647+0000 7f43f90f7700 1 -- 172.21.15.16:0/2258881587 shutdown_connections 2024-04-22T16:01:14.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.647+0000 7f43f90f7700 1 --2- 172.21.15.16:0/2258881587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43f4101e60 0x7f43f419c640 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:14.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.647+0000 7f43f90f7700 1 --2- 172.21.15.16:0/2258881587 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f43dc06ecc0 0x7f43dc071180 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:14.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.647+0000 7f43f90f7700 1 --2- 172.21.15.16:0/2258881587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43f4102780 0x7f43f419cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:14.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.647+0000 7f43f90f7700 1 -- 172.21.15.16:0/2258881587 >> 172.21.15.16:0/2258881587 conn(0x7f43f40fb650 msgr2=0x7f43f4105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:14.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.647+0000 7f43f90f7700 1 -- 172.21.15.16:0/2258881587 shutdown_connections 2024-04-22T16:01:14.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:14.648+0000 7f43f90f7700 1 -- 172.21.15.16:0/2258881587 wait complete. 2024-04-22T16:01:14.661 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-3223-g5883d494 (5883d494d3973b30b3f03e4a3f0e1e06e3c131aa) squid (dev)" 2024-04-22T16:01:14.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:14 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/2258881587' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T16:01:15.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:14 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/2258881587' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T16:01:15.202 DEBUG:teuthology.parallel:result is None 2024-04-22T16:01:15.202 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T16:01:15.214 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-22T16:01:15.215 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-22T16:01:15.429 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/mon.smithi016/config 2024-04-22T16:01:15.861 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:15 smithi016 ceph-mon[110740]: pgmap v270: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:01:15.861 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:15 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:01:16.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:15 smithi027 ceph-mon[83054]: pgmap v270: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:01:16.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:15 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:01:16.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.933+0000 7f11cc1c9700 1 -- 172.21.15.16:0/4175977614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c40fff50 msgr2=0x7f11c4105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:16.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.933+0000 7f11cc1c9700 1 --2- 172.21.15.16:0/4175977614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c40fff50 0x7f11c4105840 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f11b4009910 tx=0x7f11b4009c20 comp rx=0 tx=0).stop 2024-04-22T16:01:16.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.933+0000 7f11cc1c9700 1 -- 172.21.15.16:0/4175977614 shutdown_connections 2024-04-22T16:01:16.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.933+0000 7f11cc1c9700 1 --2- 172.21.15.16:0/4175977614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c40fff50 0x7f11c4105840 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:16.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.933+0000 7f11cc1c9700 1 --2- 172.21.15.16:0/4175977614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c40ff5a0 0x7f11c40ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:16.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.933+0000 7f11cc1c9700 1 -- 172.21.15.16:0/4175977614 >> 172.21.15.16:0/4175977614 conn(0x7f11c4074e30 msgr2=0x7f11c4075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:16.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.933+0000 7f11cc1c9700 1 -- 172.21.15.16:0/4175977614 shutdown_connections 2024-04-22T16:01:16.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.933+0000 7f11cc1c9700 1 -- 172.21.15.16:0/4175977614 wait complete. 2024-04-22T16:01:16.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.933+0000 7f11cc1c9700 1 Processor -- start 2024-04-22T16:01:16.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.934+0000 7f11cc1c9700 1 -- start start 2024-04-22T16:01:16.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.934+0000 7f11cc1c9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c40ff5a0 0x7f11c4196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:16.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.934+0000 7f11cc1c9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c40fff50 0x7f11c4196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:16.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.934+0000 7f11cc1c9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c4197560 con 0x7f11c40fff50 2024-04-22T16:01:16.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.934+0000 7f11cc1c9700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c419b370 con 0x7f11c40ff5a0 2024-04-22T16:01:16.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.934+0000 7f11c9764700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c40fff50 0x7f11c4196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:16.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.934+0000 7f11c9764700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c40fff50 0x7f11c4196ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38102/0 (socket says 172.21.15.16:38102) 2024-04-22T16:01:16.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.934+0000 7f11c9764700 1 -- 172.21.15.16:0/2612368427 learned_addr learned my addr 172.21.15.16:0/2612368427 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:16.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.934+0000 7f11c9f65700 1 --2- 172.21.15.16:0/2612368427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c40ff5a0 0x7f11c4196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:16.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.934+0000 7f11c9764700 1 -- 172.21.15.16:0/2612368427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c40ff5a0 msgr2=0x7f11c4196990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:16.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.934+0000 7f11c9764700 1 --2- 172.21.15.16:0/2612368427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c40ff5a0 0x7f11c4196990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:16.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.934+0000 7f11c9764700 1 -- 172.21.15.16:0/2612368427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b40095f0 con 0x7f11c40fff50 2024-04-22T16:01:16.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.934+0000 7f11c9764700 1 --2- 172.21.15.16:0/2612368427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c40fff50 0x7f11c4196ed0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f11b40092e0 tx=0x7f11b400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:16.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.935+0000 7f11baffd700 1 -- 172.21.15.16:0/2612368427 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11b401d070 con 0x7f11c40fff50 2024-04-22T16:01:16.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.935+0000 7f11baffd700 1 -- 172.21.15.16:0/2612368427 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f11b400f460 con 0x7f11c40fff50 2024-04-22T16:01:16.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.935+0000 7f11cc1c9700 1 -- 172.21.15.16:0/2612368427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c419b5f0 con 0x7f11c40fff50 2024-04-22T16:01:16.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.935+0000 7f11baffd700 1 -- 172.21.15.16:0/2612368427 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11b40175f0 con 0x7f11c40fff50 2024-04-22T16:01:16.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.935+0000 7f11cc1c9700 1 -- 172.21.15.16:0/2612368427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c419bae0 con 0x7f11c40fff50 2024-04-22T16:01:16.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.936+0000 7f11cc1c9700 1 -- 172.21.15.16:0/2612368427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c4109220 con 0x7f11c40fff50 2024-04-22T16:01:16.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.936+0000 7f11baffd700 1 -- 172.21.15.16:0/2612368427 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f11b400f5d0 con 0x7f11c40fff50 2024-04-22T16:01:16.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.937+0000 7f11baffd700 1 --2- 172.21.15.16:0/2612368427 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f11b006ecc0 0x7f11b0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:16.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.937+0000 7f11baffd700 1 -- 172.21.15.16:0/2612368427 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f11b40902d0 con 0x7f11c40fff50 2024-04-22T16:01:16.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.937+0000 7f11c9f65700 1 --2- 172.21.15.16:0/2612368427 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f11b006ecc0 0x7f11b0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:16.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.937+0000 7f11c9f65700 1 --2- 172.21.15.16:0/2612368427 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f11b006ecc0 0x7f11b0071180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f11c0006fd0 tx=0x7f11c0008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:16.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:16.942+0000 7f11baffd700 1 -- 172.21.15.16:0/2612368427 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f11b405cc30 con 0x7f11c40fff50 2024-04-22T16:01:17.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.198+0000 7f11cc1c9700 1 -- 172.21.15.16:0/2612368427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f11c404f2e0 con 0x7f11c40fff50 2024-04-22T16:01:17.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.198+0000 7f11baffd700 1 -- 172.21.15.16:0/2612368427 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 38 v38) v1 ==== 76+0+1962 (secure 0 0 0) 0x7f11b405c7c0 con 0x7f11c40fff50 2024-04-22T16:01:17.201 INFO:teuthology.orchestra.run.smithi016.stdout:e38 2024-04-22T16:01:17.201 INFO:teuthology.orchestra.run.smithi016.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T16:01:17.201 INFO:teuthology.orchestra.run.smithi016.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T16:01:17.201 INFO:teuthology.orchestra.run.smithi016.stdout:legacy client fscid: 1 2024-04-22T16:01:17.201 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:17.201 INFO:teuthology.orchestra.run.smithi016.stdout:Filesystem 'cephfs' (1) 2024-04-22T16:01:17.202 INFO:teuthology.orchestra.run.smithi016.stdout:fs_name cephfs 2024-04-22T16:01:17.202 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 38 2024-04-22T16:01:17.202 INFO:teuthology.orchestra.run.smithi016.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T16:01:17.202 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-04-22T15:46:27.774571+0000 2024-04-22T16:01:17.202 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-04-22T15:59:45.782074+0000 2024-04-22T16:01:17.202 INFO:teuthology.orchestra.run.smithi016.stdout:tableserver 0 2024-04-22T16:01:17.202 INFO:teuthology.orchestra.run.smithi016.stdout:root 0 2024-04-22T16:01:17.202 INFO:teuthology.orchestra.run.smithi016.stdout:session_timeout 60 2024-04-22T16:01:17.202 INFO:teuthology.orchestra.run.smithi016.stdout:session_autoclose 300 2024-04-22T16:01:17.202 INFO:teuthology.orchestra.run.smithi016.stdout:max_file_size 1099511627776 2024-04-22T16:01:17.202 INFO:teuthology.orchestra.run.smithi016.stdout:max_xattr_size 65536 2024-04-22T16:01:17.202 INFO:teuthology.orchestra.run.smithi016.stdout:required_client_features {} 2024-04-22T16:01:17.202 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure 0 2024-04-22T16:01:17.202 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure_osd_epoch 84 2024-04-22T16:01:17.202 INFO:teuthology.orchestra.run.smithi016.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T16:01:17.203 INFO:teuthology.orchestra.run.smithi016.stdout:max_mds 1 2024-04-22T16:01:17.203 INFO:teuthology.orchestra.run.smithi016.stdout:in 0 2024-04-22T16:01:17.203 INFO:teuthology.orchestra.run.smithi016.stdout:up {0=34408} 2024-04-22T16:01:17.203 INFO:teuthology.orchestra.run.smithi016.stdout:failed 2024-04-22T16:01:17.203 INFO:teuthology.orchestra.run.smithi016.stdout:damaged 2024-04-22T16:01:17.203 INFO:teuthology.orchestra.run.smithi016.stdout:stopped 2024-04-22T16:01:17.203 INFO:teuthology.orchestra.run.smithi016.stdout:data_pools [3] 2024-04-22T16:01:17.203 INFO:teuthology.orchestra.run.smithi016.stdout:metadata_pool 2 2024-04-22T16:01:17.203 INFO:teuthology.orchestra.run.smithi016.stdout:inline_data enabled 2024-04-22T16:01:17.203 INFO:teuthology.orchestra.run.smithi016.stdout:balancer 2024-04-22T16:01:17.203 INFO:teuthology.orchestra.run.smithi016.stdout:bal_rank_mask -1 2024-04-22T16:01:17.203 INFO:teuthology.orchestra.run.smithi016.stdout:standby_count_wanted 1 2024-04-22T16:01:17.203 INFO:teuthology.orchestra.run.smithi016.stdout:qdb_cluster leader: 34408 members: 34408 2024-04-22T16:01:17.203 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.fkkejm{0:34408} state up:active seq 19 join_fscid=1 addr [v2:172.21.15.16:6826/2871308685,v1:172.21.15.16:6827/2871308685] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T16:01:17.204 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi016.pgvphu{0:34412} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.16:6828/4228489800,v1:172.21.15.16:6829/4228489800] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T16:01:17.204 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:17.204 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:17.204 INFO:teuthology.orchestra.run.smithi016.stdout:Standby daemons: 2024-04-22T16:01:17.204 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:17.204 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.iobjds{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6824/3921606817,v1:172.21.15.27:6825/3921606817] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T16:01:17.204 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.cephfs.smithi027.dgqyes{-1:44333} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6826/4073572546,v1:172.21.15.27:6827/4073572546] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T16:01:17.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.200+0000 7f11cc1c9700 1 -- 172.21.15.16:0/2612368427 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f11b006ecc0 msgr2=0x7f11b0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:17.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.200+0000 7f11cc1c9700 1 --2- 172.21.15.16:0/2612368427 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f11b006ecc0 0x7f11b0071180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f11c0006fd0 tx=0x7f11c0008040 comp rx=0 tx=0).stop 2024-04-22T16:01:17.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.200+0000 7f11cc1c9700 1 -- 172.21.15.16:0/2612368427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c40fff50 msgr2=0x7f11c4196ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:17.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.200+0000 7f11cc1c9700 1 --2- 172.21.15.16:0/2612368427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c40fff50 0x7f11c4196ed0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f11b40092e0 tx=0x7f11b400be20 comp rx=0 tx=0).stop 2024-04-22T16:01:17.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.200+0000 7f11cc1c9700 1 -- 172.21.15.16:0/2612368427 shutdown_connections 2024-04-22T16:01:17.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.200+0000 7f11cc1c9700 1 --2- 172.21.15.16:0/2612368427 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f11b006ecc0 0x7f11b0071180 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:17.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.200+0000 7f11cc1c9700 1 --2- 172.21.15.16:0/2612368427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c40ff5a0 0x7f11c4196990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:17.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.200+0000 7f11cc1c9700 1 --2- 172.21.15.16:0/2612368427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c40fff50 0x7f11c4196ed0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:17.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.200+0000 7f11cc1c9700 1 -- 172.21.15.16:0/2612368427 >> 172.21.15.16:0/2612368427 conn(0x7f11c4074e30 msgr2=0x7f11c40fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:17.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.200+0000 7f11cc1c9700 1 -- 172.21.15.16:0/2612368427 shutdown_connections 2024-04-22T16:01:17.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.200+0000 7f11cc1c9700 1 -- 172.21.15.16:0/2612368427 wait complete. 2024-04-22T16:01:17.206 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 38 2024-04-22T16:01:17.798 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:17 smithi016 ceph-mon[110740]: pgmap v271: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:17.798 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:17 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/2612368427' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T16:01:17.799 INFO:teuthology.run_tasks:Running task fs.post_upgrade_checks... 2024-04-22T16:01:17.810 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T16:01:17.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.914+0000 7ffae19d3640 1 -- 172.21.15.16:0/3043053125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffadc100c20 msgr2=0x7ffadc104c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:17.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.914+0000 7ffae19d3640 1 --2- 172.21.15.16:0/3043053125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffadc100c20 0x7ffadc104c60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ffad0009880 tx=0x7ffad002bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:17.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.914+0000 7ffae19d3640 1 -- 172.21.15.16:0/3043053125 shutdown_connections 2024-04-22T16:01:17.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.914+0000 7ffae19d3640 1 --2- 172.21.15.16:0/3043053125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffadc100c20 0x7ffadc104c60 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:17.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.914+0000 7ffae19d3640 1 --2- 172.21.15.16:0/3043053125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffadc100270 0x7ffadc100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:17.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.914+0000 7ffae19d3640 1 -- 172.21.15.16:0/3043053125 >> 172.21.15.16:0/3043053125 conn(0x7ffadc0fbaf0 msgr2=0x7ffadc0fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:17.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.914+0000 7ffae19d3640 1 -- 172.21.15.16:0/3043053125 shutdown_connections 2024-04-22T16:01:17.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.914+0000 7ffae19d3640 1 -- 172.21.15.16:0/3043053125 wait complete. 2024-04-22T16:01:17.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.914+0000 7ffae19d3640 1 Processor -- start 2024-04-22T16:01:17.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.915+0000 7ffae19d3640 1 -- start start 2024-04-22T16:01:17.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.915+0000 7ffae19d3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffadc100270 0x7ffadc0725a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:17.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.915+0000 7ffae19d3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffadc072ae0 0x7ffadc072f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:17.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.915+0000 7ffae19d3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffadc1a1f10 con 0x7ffadc100270 2024-04-22T16:01:17.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.915+0000 7ffae19d3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffadc1a2050 con 0x7ffadc072ae0 2024-04-22T16:01:17.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.915+0000 7ffadaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffadc100270 0x7ffadc0725a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:17.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.915+0000 7ffadaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffadc100270 0x7ffadc0725a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38114/0 (socket says 172.21.15.16:38114) 2024-04-22T16:01:17.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.915+0000 7ffadaffd640 1 -- 172.21.15.16:0/4090878734 learned_addr learned my addr 172.21.15.16:0/4090878734 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:17.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.915+0000 7ffada7fc640 1 --2- 172.21.15.16:0/4090878734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffadc072ae0 0x7ffadc072f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:17.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.915+0000 7ffadaffd640 1 -- 172.21.15.16:0/4090878734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffadc072ae0 msgr2=0x7ffadc072f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:17.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.915+0000 7ffadaffd640 1 --2- 172.21.15.16:0/4090878734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffadc072ae0 0x7ffadc072f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:17.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.915+0000 7ffadaffd640 1 -- 172.21.15.16:0/4090878734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad0009560 con 0x7ffadc100270 2024-04-22T16:01:17.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.915+0000 7ffada7fc640 1 --2- 172.21.15.16:0/4090878734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffadc072ae0 0x7ffadc072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T16:01:17.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.916+0000 7ffadaffd640 1 --2- 172.21.15.16:0/4090878734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffadc100270 0x7ffadc0725a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ffacc00d720 tx=0x7ffacc00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:17.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.916+0000 7ffae09d1640 1 -- 172.21.15.16:0/4090878734 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffacc0051f0 con 0x7ffadc100270 2024-04-22T16:01:17.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.916+0000 7ffae19d3640 1 -- 172.21.15.16:0/4090878734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffadc1a2190 con 0x7ffadc100270 2024-04-22T16:01:17.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.916+0000 7ffae09d1640 1 -- 172.21.15.16:0/4090878734 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffacc009ce0 con 0x7ffadc100270 2024-04-22T16:01:17.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.916+0000 7ffae09d1640 1 -- 172.21.15.16:0/4090878734 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffacc010460 con 0x7ffadc100270 2024-04-22T16:01:17.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.916+0000 7ffae19d3640 1 -- 172.21.15.16:0/4090878734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffadc1a26f0 con 0x7ffadc100270 2024-04-22T16:01:17.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.917+0000 7ffae19d3640 1 -- 172.21.15.16:0/4090878734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffadc04e040 con 0x7ffadc100270 2024-04-22T16:01:17.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.918+0000 7ffae09d1640 1 -- 172.21.15.16:0/4090878734 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ffacc005350 con 0x7ffadc100270 2024-04-22T16:01:17.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.918+0000 7ffae09d1640 1 --2- 172.21.15.16:0/4090878734 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ffac406ec30 0x7ffac40710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:17.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.919+0000 7ffada7fc640 1 --2- 172.21.15.16:0/4090878734 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ffac406ec30 0x7ffac40710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:17.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.919+0000 7ffae09d1640 1 -- 172.21.15.16:0/4090878734 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ffacc08e9f0 con 0x7ffadc100270 2024-04-22T16:01:17.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.919+0000 7ffada7fc640 1 --2- 172.21.15.16:0/4090878734 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ffac406ec30 0x7ffac40710f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ffad0005ec0 tx=0x7ffad0005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:17.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:17.923+0000 7ffae09d1640 1 -- 172.21.15.16:0/4090878734 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ffacc090e60 con 0x7ffadc100270 2024-04-22T16:01:18.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:17 smithi027 ceph-mon[83054]: pgmap v271: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:18.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:17 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/2612368427' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T16:01:18.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.115+0000 7ffae19d3640 1 -- 172.21.15.16:0/4090878734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffadc100c20 con 0x7ffadc100270 2024-04-22T16:01:18.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.116+0000 7ffae09d1640 1 -- 172.21.15.16:0/4090878734 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16590 (secure 0 0 0) 0x7ffacc0577a0 con 0x7ffadc100270 2024-04-22T16:01:18.119 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:18.119 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":84,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","created":"2024-04-22T15:39:20.727181+0000","modified":"2024-04-22T15:59:06.181284+0000","last_up_change":"2024-04-22T15:58:03.188801+0000","last_in_change":"2024-04-22T15:44:52.054297+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T15:43:58.896563+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-22T15:46:26.715511+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-22T15:46:27.206408+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"67bd69ee-bd8c-4670-927a-f5fbcd92c9bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":53,"up_thru":78,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":3158080196},{"type":"v1","addr":"172.21.15.16:6803","nonce":3158080196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":3158080196},{"type":"v1","addr":"172.21.15.16:6805","nonce":3158080196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":3158080196},{"type":"v1","addr":"172.21.15.16:6809","nonce":3158080196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":3158080196},{"type":"v1","addr":"172.21.15.16:6807","nonce":3158080196}]},"public_addr":"172.21.15.16:6803/3158080196","cluster_addr":"172.21.15.16:6805/3158080196","heartbeat_back_addr":"172.21.15.16:6809/3158080196","heartbeat_front_addr":"172.21.15.16:6807/3158080196","state":["exists","up"]},{"osd":1,"uuid":"f51a9e4d-fb0c-4f62-9328-e23eea863213","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":54,"up_from":58,"up_thru":78,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3524132843},{"type":"v1","addr":"172.21.15.16:6811","nonce":3524132843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3524132843},{"type":"v1","addr":"172.21.15.16:6813","nonce":3524132843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3524132843},{"type":"v1","addr":"172.21.15.16:6817","nonce":3524132843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3524132843},{"type":"v1","addr":"172.21.15.16:6815","nonce":3524132843}]},"public_addr":"172.21.15.16:6811/3524132843","cluster_addr":"172.21.15.16:6813/3524132843","heartbeat_back_addr":"172.21.15.16:6817/3524132843","heartbeat_front_addr":"172.21.15.16:6815/3524132843","state":["exists","up"]},{"osd":2,"uuid":"e59fdb03-e04e-4afd-a58e-0b58945d3765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":59,"up_from":63,"up_thru":78,"down_at":60,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2085469945},{"type":"v1","addr":"172.21.15.16:6819","nonce":2085469945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2085469945},{"type":"v1","addr":"172.21.15.16:6821","nonce":2085469945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2085469945},{"type":"v1","addr":"172.21.15.16:6825","nonce":2085469945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2085469945},{"type":"v1","addr":"172.21.15.16:6823","nonce":2085469945}]},"public_addr":"172.21.15.16:6819/2085469945","cluster_addr":"172.21.15.16:6821/2085469945","heartbeat_back_addr":"172.21.15.16:6825/2085469945","heartbeat_front_addr":"172.21.15.16:6823/2085469945","state":["exists","up"]},{"osd":3,"uuid":"fa25b413-d004-4455-8d7b-0ced7f880a7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":64,"up_from":68,"up_thru":78,"down_at":65,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":365345197},{"type":"v1","addr":"172.21.15.27:6801","nonce":365345197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":365345197},{"type":"v1","addr":"172.21.15.27:6803","nonce":365345197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":365345197},{"type":"v1","addr":"172.21.15.27:6807","nonce":365345197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":365345197},{"type":"v1","addr":"172.21.15.27:6805","nonce":365345197}]},"public_addr":"172.21.15.27:6801/365345197","cluster_addr":"172.21.15.27:6803/365345197","heartbeat_back_addr":"172.21.15.27:6807/365345197","heartbeat_front_addr":"172.21.15.27:6805/365345197","state":["exists","up"]},{"osd":4,"uuid":"6b1eda0d-1627-4c49-93b1-1a78896f660c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":69,"up_from":73,"up_thru":78,"down_at":70,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":84770413},{"type":"v1","addr":"172.21.15.27:6809","nonce":84770413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":84770413},{"type":"v1","addr":"172.21.15.27:6811","nonce":84770413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":84770413},{"type":"v1","addr":"172.21.15.27:6815","nonce":84770413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":84770413},{"type":"v1","addr":"172.21.15.27:6813","nonce":84770413}]},"public_addr":"172.21.15.27:6809/84770413","cluster_addr":"172.21.15.27:6811/84770413","heartbeat_back_addr":"172.21.15.27:6815/84770413","heartbeat_front_addr":"172.21.15.27:6813/84770413","state":["exists","up"]},{"osd":5,"uuid":"58cebe34-0821-458f-86b5-7d460829d994","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":74,"up_from":78,"up_thru":78,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3875730087},{"type":"v1","addr":"172.21.15.27:6817","nonce":3875730087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3875730087},{"type":"v1","addr":"172.21.15.27:6819","nonce":3875730087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3875730087},{"type":"v1","addr":"172.21.15.27:6823","nonce":3875730087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3875730087},{"type":"v1","addr":"172.21.15.27:6821","nonce":3875730087}]},"public_addr":"172.21.15.27:6817/3875730087","cluster_addr":"172.21.15.27:6819/3875730087","heartbeat_back_addr":"172.21.15.27:6823/3875730087","heartbeat_front_addr":"172.21.15.27:6821/3875730087","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:42:58.192450+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:43:27.129522+0000","dead_epoch":54},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:43:54.750854+0000","dead_epoch":59},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:44:19.111479+0000","dead_epoch":64},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:44:44.487429+0000","dead_epoch":69},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:45:09.605548+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6827/476026355":"2024-04-23T15:59:06.181260+0000","172.21.15.27:6825/4216277609":"2024-04-23T15:58:44.932745+0000","172.21.15.16:6801/3431522404":"2024-04-23T15:40:27.107187+0000","172.21.15.16:6801/1748504461":"2024-04-23T15:39:53.479948+0000","172.21.15.16:6800/3148690064":"2024-04-23T15:53:30.185361+0000","172.21.15.27:0/4251496318":"2024-04-23T15:49:57.821345+0000","172.21.15.16:6800/3972982132":"2024-04-23T15:41:40.376957+0000","172.21.15.16:6800/3431522404":"2024-04-23T15:40:27.107187+0000","172.21.15.16:0/3682380":"2024-04-23T15:48:41.336415+0000","172.21.15.27:6829/2676414545":"2024-04-23T15:49:57.821345+0000","172.21.15.27:0/186298982":"2024-04-23T15:49:57.821345+0000","172.21.15.27:6828/2676414545":"2024-04-23T15:49:57.821345+0000","172.21.15.16:0/955466902":"2024-04-23T15:40:27.107187+0000","172.21.15.27:6824/4216277609":"2024-04-23T15:58:44.932745+0000","172.21.15.16:0/1878962216":"2024-04-23T15:48:41.336415+0000","172.21.15.16:0/2348469475":"2024-04-23T15:39:53.479948+0000","172.21.15.27:0/2039035396":"2024-04-23T15:49:57.821345+0000","172.21.15.16:0/25024044":"2024-04-23T15:48:41.336415+0000","172.21.15.27:6826/476026355":"2024-04-23T15:59:06.181260+0000","172.21.15.16:0/4172787384":"2024-04-23T15:40:27.107187+0000","172.21.15.16:0/3067906798":"2024-04-23T15:53:30.185361+0000","172.21.15.16:6827/2331116023":"2024-04-23T15:46:44.357516+0000","172.21.15.16:0/2002585335":"2024-04-23T15:40:27.107187+0000","172.21.15.16:0/2570865319":"2024-04-23T15:39:53.479948+0000","172.21.15.16:0/690437556":"2024-04-23T15:41:40.376957+0000","172.21.15.16:6801/3972982132":"2024-04-23T15:41:40.376957+0000","172.21.15.16:0/3645737911":"2024-04-23T15:48:41.336415+0000","172.21.15.16:6826/2331116023":"2024-04-23T15:46:44.357516+0000","172.21.15.16:6800/1748504461":"2024-04-23T15:39:53.479948+0000","172.21.15.16:0/4287530858":"2024-04-23T15:53:30.185361+0000","172.21.15.27:0/3964606592":"2024-04-23T15:49:57.821345+0000","172.21.15.16:0/2048283809":"2024-04-23T15:39:53.479948+0000","172.21.15.16:0/4021274304":"2024-04-23T15:41:40.376957+0000","172.21.15.16:6800/1057234399":"2024-04-23T15:48:41.336415+0000","172.21.15.16:6801/1057234399":"2024-04-23T15:48:41.336415+0000","172.21.15.16:6801/3148690064":"2024-04-23T15:53:30.185361+0000","172.21.15.16:0/4252231167":"2024-04-23T15:41:40.376957+0000","172.21.15.16:0/3802040196":"2024-04-23T15:53:30.185361+0000","172.21.15.16:0/2093505259":"2024-04-23T15:53:30.185361+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T16:01:18.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.118+0000 7ffae19d3640 1 -- 172.21.15.16:0/4090878734 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ffac406ec30 msgr2=0x7ffac40710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:18.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.118+0000 7ffae19d3640 1 --2- 172.21.15.16:0/4090878734 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ffac406ec30 0x7ffac40710f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ffad0005ec0 tx=0x7ffad0005fd0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.118+0000 7ffae19d3640 1 -- 172.21.15.16:0/4090878734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffadc100270 msgr2=0x7ffadc0725a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:18.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.118+0000 7ffae19d3640 1 --2- 172.21.15.16:0/4090878734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffadc100270 0x7ffadc0725a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ffacc00d720 tx=0x7ffacc00dbe0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.118+0000 7ffae19d3640 1 -- 172.21.15.16:0/4090878734 shutdown_connections 2024-04-22T16:01:18.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.118+0000 7ffae19d3640 1 --2- 172.21.15.16:0/4090878734 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ffac406ec30 0x7ffac40710f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.118+0000 7ffae19d3640 1 --2- 172.21.15.16:0/4090878734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffadc072ae0 0x7ffadc072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.118+0000 7ffae19d3640 1 --2- 172.21.15.16:0/4090878734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffadc100270 0x7ffadc0725a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.118+0000 7ffae19d3640 1 -- 172.21.15.16:0/4090878734 >> 172.21.15.16:0/4090878734 conn(0x7ffadc0fbaf0 msgr2=0x7ffadc0fd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:18.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.118+0000 7ffae19d3640 1 -- 172.21.15.16:0/4090878734 shutdown_connections 2024-04-22T16:01:18.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.118+0000 7ffae19d3640 1 -- 172.21.15.16:0/4090878734 wait complete. 2024-04-22T16:01:18.132 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T15:43:58.896563+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-22T15:46:26.715511+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-22T15:46:27.206408+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T16:01:18.133 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T16:01:18.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.235+0000 7f266fe7d640 1 -- 172.21.15.16:0/3678795320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26680ffb60 msgr2=0x7f2668103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:18.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.235+0000 7f266fe7d640 1 --2- 172.21.15.16:0/3678795320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26680ffb60 0x7f2668103ba0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2658009880 tx=0x7f265802bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.235+0000 7f266fe7d640 1 -- 172.21.15.16:0/3678795320 shutdown_connections 2024-04-22T16:01:18.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.235+0000 7f266fe7d640 1 --2- 172.21.15.16:0/3678795320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26680ffb60 0x7f2668103ba0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.235+0000 7f266fe7d640 1 --2- 172.21.15.16:0/3678795320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26680ff1b0 0x7f26680ff590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.235+0000 7f266fe7d640 1 -- 172.21.15.16:0/3678795320 >> 172.21.15.16:0/3678795320 conn(0x7f26680fa9f0 msgr2=0x7f26680fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.235+0000 7f266fe7d640 1 -- 172.21.15.16:0/3678795320 shutdown_connections 2024-04-22T16:01:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.235+0000 7f266fe7d640 1 -- 172.21.15.16:0/3678795320 wait complete. 2024-04-22T16:01:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.235+0000 7f266fe7d640 1 Processor -- start 2024-04-22T16:01:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.235+0000 7f266fe7d640 1 -- start start 2024-04-22T16:01:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.235+0000 7f266fe7d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26680ff1b0 0x7f2668196c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.236+0000 7f266fe7d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2668197170 0x7f26681975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.236+0000 7f266fe7d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266819fca0 con 0x7f26680ff1b0 2024-04-22T16:01:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.236+0000 7f266fe7d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266819fde0 con 0x7f2668197170 2024-04-22T16:01:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.236+0000 7f266dc19640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26680ff1b0 0x7f2668196c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.236+0000 7f266dc19640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26680ff1b0 0x7f2668196c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38126/0 (socket says 172.21.15.16:38126) 2024-04-22T16:01:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.236+0000 7f266dc19640 1 -- 172.21.15.16:0/1603143406 learned_addr learned my addr 172.21.15.16:0/1603143406 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.236+0000 7f266d418640 1 --2- 172.21.15.16:0/1603143406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2668197170 0x7f26681975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.236+0000 7f266dc19640 1 -- 172.21.15.16:0/1603143406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2668197170 msgr2=0x7f26681975d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.236+0000 7f266dc19640 1 --2- 172.21.15.16:0/1603143406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2668197170 0x7f26681975d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.236+0000 7f266dc19640 1 -- 172.21.15.16:0/1603143406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2658009560 con 0x7f26680ff1b0 2024-04-22T16:01:18.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.236+0000 7f266dc19640 1 --2- 172.21.15.16:0/1603143406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26680ff1b0 0x7f2668196c30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f266400e9a0 tx=0x7f266400eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:18.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.236+0000 7f265effd640 1 -- 172.21.15.16:0/1603143406 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f266400cd50 con 0x7f26680ff1b0 2024-04-22T16:01:18.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.236+0000 7f265effd640 1 -- 172.21.15.16:0/1603143406 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f266400ceb0 con 0x7f26680ff1b0 2024-04-22T16:01:18.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.236+0000 7f266fe7d640 1 -- 172.21.15.16:0/1603143406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f266819ffe0 con 0x7f26680ff1b0 2024-04-22T16:01:18.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.236+0000 7f265effd640 1 -- 172.21.15.16:0/1603143406 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2664010660 con 0x7f26680ff1b0 2024-04-22T16:01:18.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.236+0000 7f266fe7d640 1 -- 172.21.15.16:0/1603143406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26681a04c0 con 0x7f26680ff1b0 2024-04-22T16:01:18.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.237+0000 7f266fe7d640 1 -- 172.21.15.16:0/1603143406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f266804e040 con 0x7f26680ff1b0 2024-04-22T16:01:18.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.238+0000 7f265effd640 1 -- 172.21.15.16:0/1603143406 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f26640048a0 con 0x7f26680ff1b0 2024-04-22T16:01:18.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.238+0000 7f265effd640 1 --2- 172.21.15.16:0/1603143406 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f265406ec30 0x7f26540710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:18.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.239+0000 7f265effd640 1 -- 172.21.15.16:0/1603143406 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2664014070 con 0x7f26680ff1b0 2024-04-22T16:01:18.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.239+0000 7f266d418640 1 --2- 172.21.15.16:0/1603143406 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f265406ec30 0x7f26540710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:18.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.239+0000 7f266d418640 1 --2- 172.21.15.16:0/1603143406 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f265406ec30 0x7f26540710f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2658036040 tx=0x7f2658005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:18.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.243+0000 7f265effd640 1 -- 172.21.15.16:0/1603143406 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2664058180 con 0x7f26680ff1b0 2024-04-22T16:01:18.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.433+0000 7f266fe7d640 1 -- 172.21.15.16:0/1603143406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f26681074b0 con 0x7f26680ff1b0 2024-04-22T16:01:18.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.433+0000 7f265effd640 1 -- 172.21.15.16:0/1603143406 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f266405b840 con 0x7f26680ff1b0 2024-04-22T16:01:18.436 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-22T16:01:18.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.435+0000 7f266fe7d640 1 -- 172.21.15.16:0/1603143406 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f265406ec30 msgr2=0x7f26540710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:18.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.435+0000 7f266fe7d640 1 --2- 172.21.15.16:0/1603143406 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f265406ec30 0x7f26540710f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2658036040 tx=0x7f2658005fd0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.435+0000 7f266fe7d640 1 -- 172.21.15.16:0/1603143406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26680ff1b0 msgr2=0x7f2668196c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:18.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.435+0000 7f266fe7d640 1 --2- 172.21.15.16:0/1603143406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26680ff1b0 0x7f2668196c30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f266400e9a0 tx=0x7f266400eda0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.435+0000 7f266fe7d640 1 -- 172.21.15.16:0/1603143406 shutdown_connections 2024-04-22T16:01:18.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.435+0000 7f266fe7d640 1 --2- 172.21.15.16:0/1603143406 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f265406ec30 0x7f26540710f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.435+0000 7f266fe7d640 1 --2- 172.21.15.16:0/1603143406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2668197170 0x7f26681975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.435+0000 7f266fe7d640 1 --2- 172.21.15.16:0/1603143406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26680ff1b0 0x7f2668196c30 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.435+0000 7f266fe7d640 1 -- 172.21.15.16:0/1603143406 >> 172.21.15.16:0/1603143406 conn(0x7f26680fa9f0 msgr2=0x7f26680fc060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:18.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.435+0000 7f266fe7d640 1 -- 172.21.15.16:0/1603143406 shutdown_connections 2024-04-22T16:01:18.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.435+0000 7f266fe7d640 1 -- 172.21.15.16:0/1603143406 wait complete. 2024-04-22T16:01:18.449 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-22T16:01:18.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.550+0000 7fec7ccc7640 1 -- 172.21.15.16:0/3563860045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec78100c30 msgr2=0x7fec78104c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:18.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.550+0000 7fec7ccc7640 1 --2- 172.21.15.16:0/3563860045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec78100c30 0x7fec78104c70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fec60009880 tx=0x7fec6002bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.550+0000 7fec7ccc7640 1 -- 172.21.15.16:0/3563860045 shutdown_connections 2024-04-22T16:01:18.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.550+0000 7fec7ccc7640 1 --2- 172.21.15.16:0/3563860045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec78100c30 0x7fec78104c70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.550+0000 7fec7ccc7640 1 --2- 172.21.15.16:0/3563860045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec78100280 0x7fec78100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.550+0000 7fec7ccc7640 1 -- 172.21.15.16:0/3563860045 >> 172.21.15.16:0/3563860045 conn(0x7fec780fbaa0 msgr2=0x7fec780fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:18.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.550+0000 7fec7ccc7640 1 -- 172.21.15.16:0/3563860045 shutdown_connections 2024-04-22T16:01:18.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.550+0000 7fec7ccc7640 1 -- 172.21.15.16:0/3563860045 wait complete. 2024-04-22T16:01:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.551+0000 7fec7ccc7640 1 Processor -- start 2024-04-22T16:01:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.551+0000 7fec7ccc7640 1 -- start start 2024-04-22T16:01:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.551+0000 7fec7ccc7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec78100280 0x7fec78195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.551+0000 7fec7ccc7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec78196280 0x7fec7819a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.551+0000 7fec7ccc7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec78196880 con 0x7fec78100280 2024-04-22T16:01:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.551+0000 7fec7ccc7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec781969f0 con 0x7fec78196280 2024-04-22T16:01:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.551+0000 7fec7659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec78100280 0x7fec78195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.551+0000 7fec7659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec78100280 0x7fec78195d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38146/0 (socket says 172.21.15.16:38146) 2024-04-22T16:01:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.551+0000 7fec7659c640 1 -- 172.21.15.16:0/1517735187 learned_addr learned my addr 172.21.15.16:0/1517735187 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.551+0000 7fec6ffff640 1 --2- 172.21.15.16:0/1517735187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec78196280 0x7fec7819a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.552+0000 7fec7659c640 1 -- 172.21.15.16:0/1517735187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec78196280 msgr2=0x7fec7819a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.552+0000 7fec7659c640 1 --2- 172.21.15.16:0/1517735187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec78196280 0x7fec7819a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.552+0000 7fec7659c640 1 -- 172.21.15.16:0/1517735187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec60009560 con 0x7fec78100280 2024-04-22T16:01:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.552+0000 7fec7659c640 1 --2- 172.21.15.16:0/1517735187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec78100280 0x7fec78195d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fec6800d900 tx=0x7fec6800ddc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:18.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.552+0000 7fec6f7fe640 1 -- 172.21.15.16:0/1517735187 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec68009c00 con 0x7fec78100280 2024-04-22T16:01:18.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.552+0000 7fec6f7fe640 1 -- 172.21.15.16:0/1517735187 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec68009d60 con 0x7fec78100280 2024-04-22T16:01:18.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.552+0000 7fec7ccc7640 1 -- 172.21.15.16:0/1517735187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec7819ace0 con 0x7fec78100280 2024-04-22T16:01:18.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.552+0000 7fec6f7fe640 1 -- 172.21.15.16:0/1517735187 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec6800f710 con 0x7fec78100280 2024-04-22T16:01:18.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.552+0000 7fec7ccc7640 1 -- 172.21.15.16:0/1517735187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec7819b240 con 0x7fec78100280 2024-04-22T16:01:18.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.553+0000 7fec7ccc7640 1 -- 172.21.15.16:0/1517735187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec7819b520 con 0x7fec78100280 2024-04-22T16:01:18.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.553+0000 7fec6f7fe640 1 -- 172.21.15.16:0/1517735187 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fec68010460 con 0x7fec78100280 2024-04-22T16:01:18.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.554+0000 7fec6f7fe640 1 --2- 172.21.15.16:0/1517735187 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fec6406ec30 0x7fec640710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:18.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.554+0000 7fec6f7fe640 1 -- 172.21.15.16:0/1517735187 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fec6808ebc0 con 0x7fec78100280 2024-04-22T16:01:18.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.554+0000 7fec6ffff640 1 --2- 172.21.15.16:0/1517735187 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fec6406ec30 0x7fec640710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:18.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.554+0000 7fec6ffff640 1 --2- 172.21.15.16:0/1517735187 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fec6406ec30 0x7fec640710f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fec60000c00 tx=0x7fec60005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:18.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.559+0000 7fec6f7fe640 1 -- 172.21.15.16:0/1517735187 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fec68057a20 con 0x7fec78100280 2024-04-22T16:01:18.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.746+0000 7fec7ccc7640 1 -- 172.21.15.16:0/1517735187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fec7804e040 con 0x7fec78100280 2024-04-22T16:01:18.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.747+0000 7fec6f7fe640 1 -- 172.21.15.16:0/1517735187 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7fec6805b0e0 con 0x7fec78100280 2024-04-22T16:01:18.750 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-22T16:01:18.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.749+0000 7fec7ccc7640 1 -- 172.21.15.16:0/1517735187 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fec6406ec30 msgr2=0x7fec640710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:18.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.749+0000 7fec7ccc7640 1 --2- 172.21.15.16:0/1517735187 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fec6406ec30 0x7fec640710f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fec60000c00 tx=0x7fec60005fd0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.749+0000 7fec7ccc7640 1 -- 172.21.15.16:0/1517735187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec78100280 msgr2=0x7fec78195d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.749+0000 7fec7ccc7640 1 --2- 172.21.15.16:0/1517735187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec78100280 0x7fec78195d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fec6800d900 tx=0x7fec6800ddc0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.749+0000 7fec7ccc7640 1 -- 172.21.15.16:0/1517735187 shutdown_connections 2024-04-22T16:01:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.749+0000 7fec7ccc7640 1 --2- 172.21.15.16:0/1517735187 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fec6406ec30 0x7fec640710f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.749+0000 7fec7ccc7640 1 --2- 172.21.15.16:0/1517735187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec78196280 0x7fec7819a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.749+0000 7fec7ccc7640 1 --2- 172.21.15.16:0/1517735187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec78100280 0x7fec78195d40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.749+0000 7fec7ccc7640 1 -- 172.21.15.16:0/1517735187 >> 172.21.15.16:0/1517735187 conn(0x7fec780fbaa0 msgr2=0x7fec780fd210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.749+0000 7fec7ccc7640 1 -- 172.21.15.16:0/1517735187 shutdown_connections 2024-04-22T16:01:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.750+0000 7fec7ccc7640 1 -- 172.21.15.16:0/1517735187 wait complete. 2024-04-22T16:01:18.763 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-22T16:01:18.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.864+0000 7f86f4de7640 1 -- 172.21.15.16:0/3208750788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86f00ffb50 msgr2=0x7f86f0103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:18.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.864+0000 7f86f4de7640 1 --2- 172.21.15.16:0/3208750788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86f00ffb50 0x7f86f0103b90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f86d8009880 tx=0x7f86d802bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.864+0000 7f86f4de7640 1 -- 172.21.15.16:0/3208750788 shutdown_connections 2024-04-22T16:01:18.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.864+0000 7f86f4de7640 1 --2- 172.21.15.16:0/3208750788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86f00ffb50 0x7f86f0103b90 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.864+0000 7f86f4de7640 1 --2- 172.21.15.16:0/3208750788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86f00ff1a0 0x7f86f00ff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.864+0000 7f86f4de7640 1 -- 172.21.15.16:0/3208750788 >> 172.21.15.16:0/3208750788 conn(0x7f86f00faa00 msgr2=0x7f86f00fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:18.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.864+0000 7f86f4de7640 1 -- 172.21.15.16:0/3208750788 shutdown_connections 2024-04-22T16:01:18.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.864+0000 7f86f4de7640 1 -- 172.21.15.16:0/3208750788 wait complete. 2024-04-22T16:01:18.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.865+0000 7f86f4de7640 1 Processor -- start 2024-04-22T16:01:18.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.865+0000 7f86f4de7640 1 -- start start 2024-04-22T16:01:18.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.865+0000 7f86f4de7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86f00ff1a0 0x7f86f0195f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:18.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.865+0000 7f86f4de7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86f0196460 0x7f86f019a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:18.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.865+0000 7f86f4de7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f0196a60 con 0x7f86f00ff1a0 2024-04-22T16:01:18.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.865+0000 7f86f4de7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f0196bd0 con 0x7f86f0196460 2024-04-22T16:01:18.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.865+0000 7f86ee59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86f00ff1a0 0x7f86f0195f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:18.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.865+0000 7f86ee59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86f00ff1a0 0x7f86f0195f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38172/0 (socket says 172.21.15.16:38172) 2024-04-22T16:01:18.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.865+0000 7f86ee59c640 1 -- 172.21.15.16:0/3219284395 learned_addr learned my addr 172.21.15.16:0/3219284395 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:18.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.865+0000 7f86e5d9b640 1 --2- 172.21.15.16:0/3219284395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86f0196460 0x7f86f019a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:18.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.865+0000 7f86ee59c640 1 -- 172.21.15.16:0/3219284395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86f0196460 msgr2=0x7f86f019a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:18.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.865+0000 7f86ee59c640 1 --2- 172.21.15.16:0/3219284395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86f0196460 0x7f86f019a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:18.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.865+0000 7f86ee59c640 1 -- 172.21.15.16:0/3219284395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d8009560 con 0x7f86f00ff1a0 2024-04-22T16:01:18.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.866+0000 7f86ee59c640 1 --2- 172.21.15.16:0/3219284395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86f00ff1a0 0x7f86f0195f20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f86e000b450 tx=0x7f86e000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:18.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.866+0000 7f86e7fff640 1 -- 172.21.15.16:0/3219284395 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86e0011860 con 0x7f86f00ff1a0 2024-04-22T16:01:18.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.866+0000 7f86f4de7640 1 -- 172.21.15.16:0/3219284395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f019aec0 con 0x7f86f00ff1a0 2024-04-22T16:01:18.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.866+0000 7f86e7fff640 1 -- 172.21.15.16:0/3219284395 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f86e0011e80 con 0x7f86f00ff1a0 2024-04-22T16:01:18.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.866+0000 7f86e7fff640 1 -- 172.21.15.16:0/3219284395 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86e000f570 con 0x7f86f00ff1a0 2024-04-22T16:01:18.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.866+0000 7f86f4de7640 1 -- 172.21.15.16:0/3219284395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f019b420 con 0x7f86f00ff1a0 2024-04-22T16:01:18.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.867+0000 7f86f4de7640 1 -- 172.21.15.16:0/3219284395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86f004e040 con 0x7f86f00ff1a0 2024-04-22T16:01:18.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.867+0000 7f86e7fff640 1 -- 172.21.15.16:0/3219284395 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f86e00103e0 con 0x7f86f00ff1a0 2024-04-22T16:01:18.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.868+0000 7f86e7fff640 1 --2- 172.21.15.16:0/3219284395 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f86dc06ec30 0x7f86dc0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:18.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.868+0000 7f86e7fff640 1 -- 172.21.15.16:0/3219284395 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f86e008ee10 con 0x7f86f00ff1a0 2024-04-22T16:01:18.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.868+0000 7f86e5d9b640 1 --2- 172.21.15.16:0/3219284395 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f86dc06ec30 0x7f86dc0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:18.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.868+0000 7f86e5d9b640 1 --2- 172.21.15.16:0/3219284395 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f86dc06ec30 0x7f86dc0710f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f86d8036040 tx=0x7f86d8005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:18.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:18.873+0000 7f86e7fff640 1 -- 172.21.15.16:0/3219284395 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f86e0057b20 con 0x7f86f00ff1a0 2024-04-22T16:01:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.059+0000 7f86f4de7640 1 -- 172.21.15.16:0/3219284395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f86f019b6a0 con 0x7f86f00ff1a0 2024-04-22T16:01:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.059+0000 7f86e7fff640 1 -- 172.21.15.16:0/3219284395 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f86e005b1e0 con 0x7f86f00ff1a0 2024-04-22T16:01:19.062 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-22T16:01:19.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.061+0000 7f86f4de7640 1 -- 172.21.15.16:0/3219284395 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f86dc06ec30 msgr2=0x7f86dc0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:19.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.061+0000 7f86f4de7640 1 --2- 172.21.15.16:0/3219284395 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f86dc06ec30 0x7f86dc0710f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f86d8036040 tx=0x7f86d8005fd0 comp rx=0 tx=0).stop 2024-04-22T16:01:19.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.061+0000 7f86f4de7640 1 -- 172.21.15.16:0/3219284395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86f00ff1a0 msgr2=0x7f86f0195f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:19.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.061+0000 7f86f4de7640 1 --2- 172.21.15.16:0/3219284395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86f00ff1a0 0x7f86f0195f20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f86e000b450 tx=0x7f86e000b910 comp rx=0 tx=0).stop 2024-04-22T16:01:19.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.061+0000 7f86f4de7640 1 -- 172.21.15.16:0/3219284395 shutdown_connections 2024-04-22T16:01:19.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.061+0000 7f86f4de7640 1 --2- 172.21.15.16:0/3219284395 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f86dc06ec30 0x7f86dc0710f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:19.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.061+0000 7f86f4de7640 1 --2- 172.21.15.16:0/3219284395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86f0196460 0x7f86f019a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:19.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.061+0000 7f86f4de7640 1 --2- 172.21.15.16:0/3219284395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86f00ff1a0 0x7f86f0195f20 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:19.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.061+0000 7f86f4de7640 1 -- 172.21.15.16:0/3219284395 >> 172.21.15.16:0/3219284395 conn(0x7f86f00faa00 msgr2=0x7f86f01034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:19.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.061+0000 7f86f4de7640 1 -- 172.21.15.16:0/3219284395 shutdown_connections 2024-04-22T16:01:19.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.061+0000 7f86f4de7640 1 -- 172.21.15.16:0/3219284395 wait complete. 2024-04-22T16:01:19.074 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:18 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/4090878734' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T16:01:19.074 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:18 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/1603143406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T16:01:19.075 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T16:01:19.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:18 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/4090878734' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T16:01:19.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:18 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/1603143406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T16:01:19.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.175+0000 7f3300cc7640 1 -- 172.21.15.16:0/534081301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32fc0ffb20 msgr2=0x7f32fc103b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:19.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.175+0000 7f3300cc7640 1 --2- 172.21.15.16:0/534081301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32fc0ffb20 0x7f32fc103b60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f32f0009880 tx=0x7f32f002bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:19.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.175+0000 7f3300cc7640 1 -- 172.21.15.16:0/534081301 shutdown_connections 2024-04-22T16:01:19.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.175+0000 7f3300cc7640 1 --2- 172.21.15.16:0/534081301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32fc0ffb20 0x7f32fc103b60 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:19.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.175+0000 7f3300cc7640 1 --2- 172.21.15.16:0/534081301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32fc0ff170 0x7f32fc0ff550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:19.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.175+0000 7f3300cc7640 1 -- 172.21.15.16:0/534081301 >> 172.21.15.16:0/534081301 conn(0x7f32fc0fa9f0 msgr2=0x7f32fc0fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:19.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.175+0000 7f3300cc7640 1 -- 172.21.15.16:0/534081301 shutdown_connections 2024-04-22T16:01:19.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.175+0000 7f3300cc7640 1 -- 172.21.15.16:0/534081301 wait complete. 2024-04-22T16:01:19.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.176+0000 7f3300cc7640 1 Processor -- start 2024-04-22T16:01:19.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.176+0000 7f3300cc7640 1 -- start start 2024-04-22T16:01:19.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.176+0000 7f3300cc7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32fc0ff170 0x7f32fc196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:19.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.176+0000 7f3300cc7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32fc1970d0 0x7f32fc197530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:19.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.176+0000 7f3300cc7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32fc19fc00 con 0x7f32fc1970d0 2024-04-22T16:01:19.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.176+0000 7f3300cc7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32fc19fd40 con 0x7f32fc0ff170 2024-04-22T16:01:19.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.176+0000 7f32f9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32fc1970d0 0x7f32fc197530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:19.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.176+0000 7f32f9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32fc1970d0 0x7f32fc197530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38184/0 (socket says 172.21.15.16:38184) 2024-04-22T16:01:19.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.176+0000 7f32f9d9b640 1 -- 172.21.15.16:0/4218504638 learned_addr learned my addr 172.21.15.16:0/4218504638 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:19.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.176+0000 7f32fa59c640 1 --2- 172.21.15.16:0/4218504638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32fc0ff170 0x7f32fc196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:19.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.176+0000 7f32f9d9b640 1 -- 172.21.15.16:0/4218504638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32fc0ff170 msgr2=0x7f32fc196b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:19.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.176+0000 7f32f9d9b640 1 --2- 172.21.15.16:0/4218504638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32fc0ff170 0x7f32fc196b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:19.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.176+0000 7f32f9d9b640 1 -- 172.21.15.16:0/4218504638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f0009560 con 0x7f32fc1970d0 2024-04-22T16:01:19.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.176+0000 7f32fa59c640 1 --2- 172.21.15.16:0/4218504638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32fc0ff170 0x7f32fc196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T16:01:19.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.176+0000 7f32f9d9b640 1 --2- 172.21.15.16:0/4218504638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32fc1970d0 0x7f32fc197530 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f32f0036040 tx=0x7f32f002c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:19.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.177+0000 7f32eb7fe640 1 -- 172.21.15.16:0/4218504638 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32f00035f0 con 0x7f32fc1970d0 2024-04-22T16:01:19.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.177+0000 7f32eb7fe640 1 -- 172.21.15.16:0/4218504638 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f32f0003c10 con 0x7f32fc1970d0 2024-04-22T16:01:19.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.177+0000 7f3300cc7640 1 -- 172.21.15.16:0/4218504638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32fc19ff40 con 0x7f32fc1970d0 2024-04-22T16:01:19.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.177+0000 7f32eb7fe640 1 -- 172.21.15.16:0/4218504638 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32f0034440 con 0x7f32fc1970d0 2024-04-22T16:01:19.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.177+0000 7f3300cc7640 1 -- 172.21.15.16:0/4218504638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32fc1a03c0 con 0x7f32fc1970d0 2024-04-22T16:01:19.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.178+0000 7f3300cc7640 1 -- 172.21.15.16:0/4218504638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32fc04e040 con 0x7f32fc1970d0 2024-04-22T16:01:19.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.178+0000 7f32eb7fe640 1 -- 172.21.15.16:0/4218504638 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f32f0003750 con 0x7f32fc1970d0 2024-04-22T16:01:19.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.179+0000 7f32eb7fe640 1 --2- 172.21.15.16:0/4218504638 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f32e406ec30 0x7f32e40710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:19.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.179+0000 7f32eb7fe640 1 -- 172.21.15.16:0/4218504638 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f32f00b1230 con 0x7f32fc1970d0 2024-04-22T16:01:19.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.179+0000 7f32fa59c640 1 --2- 172.21.15.16:0/4218504638 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f32e406ec30 0x7f32e40710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:19.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.179+0000 7f32fa59c640 1 --2- 172.21.15.16:0/4218504638 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f32e406ec30 0x7f32e40710f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f32ec0058c0 tx=0x7f32ec005850 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:19.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.183+0000 7f32eb7fe640 1 -- 172.21.15.16:0/4218504638 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f32f0079fe0 con 0x7f32fc1970d0 2024-04-22T16:01:19.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.412+0000 7f3300cc7640 1 -- 172.21.15.16:0/4218504638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32fc107470 con 0x7f32fc1970d0 2024-04-22T16:01:19.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.413+0000 7f32eb7fe640 1 -- 172.21.15.16:0/4218504638 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5215 (secure 0 0 0) 0x7f32fc107470 con 0x7f32fc1970d0 2024-04-22T16:01:19.416 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:19.416 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44329,"name":"cephfs.smithi027.iobjds","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6825/3921606817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3921606817},{"type":"v1","addr":"172.21.15.27:6825","nonce":3921606817}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":44333,"name":"cephfs.smithi027.dgqyes","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6827/4073572546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4073572546},{"type":"v1","addr":"172.21.15.27:6827","nonce":4073572546}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:46:27.774571+0000","modified":"2024-04-22T15:59:45.782074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34408":{"gid":34408,"name":"cephfs.smithi016.fkkejm","rank":0,"incarnation":30,"state":"up:active","state_seq":19,"addr":"172.21.15.16:6827/2871308685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2871308685},{"type":"v1","addr":"172.21.15.16:6827","nonce":2871308685}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34412":{"gid":34412,"name":"cephfs.smithi016.pgvphu","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6829/4228489800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4228489800},{"type":"v1","addr":"172.21.15.16:6829","nonce":4228489800}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34408,"qdb_cluster":[34408]},"id":1}]} 2024-04-22T16:01:19.416 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 38 2024-04-22T16:01:19.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.415+0000 7f3300cc7640 1 -- 172.21.15.16:0/4218504638 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f32e406ec30 msgr2=0x7f32e40710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:19.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.415+0000 7f3300cc7640 1 --2- 172.21.15.16:0/4218504638 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f32e406ec30 0x7f32e40710f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f32ec0058c0 tx=0x7f32ec005850 comp rx=0 tx=0).stop 2024-04-22T16:01:19.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.415+0000 7f3300cc7640 1 -- 172.21.15.16:0/4218504638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32fc1970d0 msgr2=0x7f32fc197530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:19.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.415+0000 7f3300cc7640 1 --2- 172.21.15.16:0/4218504638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32fc1970d0 0x7f32fc197530 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f32f0036040 tx=0x7f32f002c850 comp rx=0 tx=0).stop 2024-04-22T16:01:19.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.415+0000 7f3300cc7640 1 -- 172.21.15.16:0/4218504638 shutdown_connections 2024-04-22T16:01:19.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.415+0000 7f3300cc7640 1 --2- 172.21.15.16:0/4218504638 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f32e406ec30 0x7f32e40710f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:19.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.415+0000 7f3300cc7640 1 --2- 172.21.15.16:0/4218504638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32fc1970d0 0x7f32fc197530 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:19.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.415+0000 7f3300cc7640 1 --2- 172.21.15.16:0/4218504638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32fc0ff170 0x7f32fc196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:19.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.415+0000 7f3300cc7640 1 -- 172.21.15.16:0/4218504638 >> 172.21.15.16:0/4218504638 conn(0x7f32fc0fa9f0 msgr2=0x7f32fc0fc0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:19.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.415+0000 7f3300cc7640 1 -- 172.21.15.16:0/4218504638 shutdown_connections 2024-04-22T16:01:19.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.415+0000 7f3300cc7640 1 -- 172.21.15.16:0/4218504638 wait complete. 2024-04-22T16:01:19.429 DEBUG:tasks.fs:checking fs fscid=1,name=cephfs state = {'epoch': 16, 'max_mds': 1, 'flags': 50} 2024-04-22T16:01:19.429 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 17 2024-04-22T16:01:19.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.528+0000 7fae466af640 1 -- 172.21.15.16:0/98791214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae40100c50 msgr2=0x7fae40104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:19.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.528+0000 7fae466af640 1 --2- 172.21.15.16:0/98791214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae40100c50 0x7fae40104c90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fae34009880 tx=0x7fae3402bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:19.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.528+0000 7fae466af640 1 -- 172.21.15.16:0/98791214 shutdown_connections 2024-04-22T16:01:19.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.528+0000 7fae466af640 1 --2- 172.21.15.16:0/98791214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae40100c50 0x7fae40104c90 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:19.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.528+0000 7fae466af640 1 --2- 172.21.15.16:0/98791214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae401002a0 0x7fae40100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:19.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.528+0000 7fae466af640 1 -- 172.21.15.16:0/98791214 >> 172.21.15.16:0/98791214 conn(0x7fae400fbb20 msgr2=0x7fae400fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:19.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.528+0000 7fae466af640 1 -- 172.21.15.16:0/98791214 shutdown_connections 2024-04-22T16:01:19.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.528+0000 7fae466af640 1 -- 172.21.15.16:0/98791214 wait complete. 2024-04-22T16:01:19.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.528+0000 7fae466af640 1 Processor -- start 2024-04-22T16:01:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.528+0000 7fae466af640 1 -- start start 2024-04-22T16:01:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.529+0000 7fae466af640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae401002a0 0x7fae40195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.529+0000 7fae466af640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae40196280 0x7fae4019a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.529+0000 7fae466af640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae40196880 con 0x7fae401002a0 2024-04-22T16:01:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.529+0000 7fae466af640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae401969f0 con 0x7fae40196280 2024-04-22T16:01:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.529+0000 7fae3ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae401002a0 0x7fae40195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.529+0000 7fae3ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae401002a0 0x7fae40195d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38196/0 (socket says 172.21.15.16:38196) 2024-04-22T16:01:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.529+0000 7fae3ffff640 1 -- 172.21.15.16:0/791281752 learned_addr learned my addr 172.21.15.16:0/791281752 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.529+0000 7fae3f7fe640 1 --2- 172.21.15.16:0/791281752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae40196280 0x7fae4019a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.529+0000 7fae3ffff640 1 -- 172.21.15.16:0/791281752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae40196280 msgr2=0x7fae4019a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.529+0000 7fae3ffff640 1 --2- 172.21.15.16:0/791281752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae40196280 0x7fae4019a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.529+0000 7fae3ffff640 1 -- 172.21.15.16:0/791281752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae34009560 con 0x7fae401002a0 2024-04-22T16:01:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.529+0000 7fae3ffff640 1 --2- 172.21.15.16:0/791281752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae401002a0 0x7fae40195d40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fae3000d900 tx=0x7fae3000ddc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.530+0000 7fae3d7fa640 1 -- 172.21.15.16:0/791281752 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae30009c00 con 0x7fae401002a0 2024-04-22T16:01:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.530+0000 7fae3d7fa640 1 -- 172.21.15.16:0/791281752 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fae30009d60 con 0x7fae401002a0 2024-04-22T16:01:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.530+0000 7fae3d7fa640 1 -- 172.21.15.16:0/791281752 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae3000f710 con 0x7fae401002a0 2024-04-22T16:01:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.530+0000 7fae466af640 1 -- 172.21.15.16:0/791281752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae4019ace0 con 0x7fae401002a0 2024-04-22T16:01:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.530+0000 7fae466af640 1 -- 172.21.15.16:0/791281752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae4019b240 con 0x7fae401002a0 2024-04-22T16:01:19.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.531+0000 7fae466af640 1 -- 172.21.15.16:0/791281752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae4019b510 con 0x7fae401002a0 2024-04-22T16:01:19.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.531+0000 7fae3d7fa640 1 -- 172.21.15.16:0/791281752 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fae30010460 con 0x7fae401002a0 2024-04-22T16:01:19.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.531+0000 7fae3d7fa640 1 --2- 172.21.15.16:0/791281752 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fae2806ec30 0x7fae280710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:19.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.532+0000 7fae3d7fa640 1 -- 172.21.15.16:0/791281752 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fae3008eaf0 con 0x7fae401002a0 2024-04-22T16:01:19.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.532+0000 7fae3f7fe640 1 --2- 172.21.15.16:0/791281752 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fae2806ec30 0x7fae280710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:19.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.532+0000 7fae3f7fe640 1 --2- 172.21.15.16:0/791281752 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fae2806ec30 0x7fae280710f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fae34000c00 tx=0x7fae34005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:19.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.536+0000 7fae3d7fa640 1 -- 172.21.15.16:0/791281752 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fae300579d0 con 0x7fae401002a0 2024-04-22T16:01:19.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.761+0000 7fae466af640 1 -- 172.21.15.16:0/791281752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 17, "format": "json"} v 0) v1 -- 0x7fae401970c0 con 0x7fae401002a0 2024-04-22T16:01:19.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.761+0000 7fae3d7fa640 1 -- 172.21.15.16:0/791281752 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 17, "format": "json"}]=0 dumped fsmap epoch 17 v38) v1 ==== 107+0+4085 (secure 0 0 0) 0x7fae3005b090 con 0x7fae401002a0 2024-04-22T16:01:19.765 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:19.765 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14504,"name":"cephfs.smithi016.fkkejm","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6827/3484907755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3484907755},{"type":"v1","addr":"172.21.15.16:6827","nonce":3484907755}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24293,"name":"cephfs.smithi027.dgqyes","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.27:6827/476026355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":476026355},{"type":"v1","addr":"172.21.15.27:6827","nonce":476026355}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:46:27.774571+0000","modified":"2024-04-22T15:58:05.206233+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24279":{"gid":24279,"name":"cephfs.smithi027.iobjds","rank":0,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6825/4216277609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4216277609},{"type":"v1","addr":"172.21.15.27:6825","nonce":4216277609}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24279,"qdb_cluster":[24279]},"id":1}]} 2024-04-22T16:01:19.765 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 17 2024-04-22T16:01:19.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.764+0000 7fae466af640 1 -- 172.21.15.16:0/791281752 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fae2806ec30 msgr2=0x7fae280710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:19.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.764+0000 7fae466af640 1 --2- 172.21.15.16:0/791281752 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fae2806ec30 0x7fae280710f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fae34000c00 tx=0x7fae34005fd0 comp rx=0 tx=0).stop 2024-04-22T16:01:19.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.764+0000 7fae466af640 1 -- 172.21.15.16:0/791281752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae401002a0 msgr2=0x7fae40195d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.764+0000 7fae466af640 1 --2- 172.21.15.16:0/791281752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae401002a0 0x7fae40195d40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fae3000d900 tx=0x7fae3000ddc0 comp rx=0 tx=0).stop 2024-04-22T16:01:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.764+0000 7fae466af640 1 -- 172.21.15.16:0/791281752 shutdown_connections 2024-04-22T16:01:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.764+0000 7fae466af640 1 --2- 172.21.15.16:0/791281752 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fae2806ec30 0x7fae280710f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.764+0000 7fae466af640 1 --2- 172.21.15.16:0/791281752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae40196280 0x7fae4019a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.764+0000 7fae466af640 1 --2- 172.21.15.16:0/791281752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae401002a0 0x7fae40195d40 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.764+0000 7fae466af640 1 -- 172.21.15.16:0/791281752 >> 172.21.15.16:0/791281752 conn(0x7fae400fbb20 msgr2=0x7fae400fd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.764+0000 7fae466af640 1 -- 172.21.15.16:0/791281752 shutdown_connections 2024-04-22T16:01:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.764+0000 7fae466af640 1 -- 172.21.15.16:0/791281752 wait complete. 2024-04-22T16:01:19.778 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 17 2024-04-22T16:01:19.778 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 18 2024-04-22T16:01:19.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.879+0000 7f659b37a640 1 -- 172.21.15.16:0/3915332599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6594101ba0 msgr2=0x7f659410a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:19.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.879+0000 7f659b37a640 1 --2- 172.21.15.16:0/3915332599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6594101ba0 0x7f659410a080 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6590009880 tx=0x7f659002bd70 comp rx=0 tx=0).stop 2024-04-22T16:01:19.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.879+0000 7f659b37a640 1 -- 172.21.15.16:0/3915332599 shutdown_connections 2024-04-22T16:01:19.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.879+0000 7f659b37a640 1 --2- 172.21.15.16:0/3915332599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6594101ba0 0x7f659410a080 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:19.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.879+0000 7f659b37a640 1 --2- 172.21.15.16:0/3915332599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6594101280 0x7f6594101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:19.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.879+0000 7f659b37a640 1 -- 172.21.15.16:0/3915332599 >> 172.21.15.16:0/3915332599 conn(0x7f65940faa20 msgr2=0x7f65940fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:19.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.879+0000 7f659b37a640 1 -- 172.21.15.16:0/3915332599 shutdown_connections 2024-04-22T16:01:19.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.879+0000 7f659b37a640 1 -- 172.21.15.16:0/3915332599 wait complete. 2024-04-22T16:01:19.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.880+0000 7f659b37a640 1 Processor -- start 2024-04-22T16:01:19.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.880+0000 7f659b37a640 1 -- start start 2024-04-22T16:01:19.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.880+0000 7f659b37a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6594101280 0x7f65940feed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:19.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.880+0000 7f659b37a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6594101ba0 0x7f65940ff410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:19.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.880+0000 7f659b37a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65940ff950 con 0x7f6594101ba0 2024-04-22T16:01:19.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.880+0000 7f659b37a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65940ffac0 con 0x7f6594101280 2024-04-22T16:01:19.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.880+0000 7f6598915640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6594101ba0 0x7f65940ff410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:19.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.880+0000 7f6598915640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6594101ba0 0x7f65940ff410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38200/0 (socket says 172.21.15.16:38200) 2024-04-22T16:01:19.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.880+0000 7f6598915640 1 -- 172.21.15.16:0/1982705624 learned_addr learned my addr 172.21.15.16:0/1982705624 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:19.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.880+0000 7f6599116640 1 --2- 172.21.15.16:0/1982705624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6594101280 0x7f65940feed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:19.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.880+0000 7f6598915640 1 -- 172.21.15.16:0/1982705624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6594101280 msgr2=0x7f65940feed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:19.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.880+0000 7f6598915640 1 --2- 172.21.15.16:0/1982705624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6594101280 0x7f65940feed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:19.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.880+0000 7f6598915640 1 -- 172.21.15.16:0/1982705624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6590009560 con 0x7f6594101ba0 2024-04-22T16:01:19.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.880+0000 7f6599116640 1 --2- 172.21.15.16:0/1982705624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6594101280 0x7f65940feed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T16:01:19.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.880+0000 7f6598915640 1 --2- 172.21.15.16:0/1982705624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6594101ba0 0x7f65940ff410 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6590009230 tx=0x7f6590004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:19.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.881+0000 7f658a7fc640 1 -- 172.21.15.16:0/1982705624 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65900035f0 con 0x7f6594101ba0 2024-04-22T16:01:19.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.881+0000 7f658a7fc640 1 -- 172.21.15.16:0/1982705624 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6590003c10 con 0x7f6594101ba0 2024-04-22T16:01:19.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.881+0000 7f659b37a640 1 -- 172.21.15.16:0/1982705624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65940ffce0 con 0x7f6594101ba0 2024-04-22T16:01:19.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.881+0000 7f658a7fc640 1 -- 172.21.15.16:0/1982705624 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f659003e450 con 0x7f6594101ba0 2024-04-22T16:01:19.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.881+0000 7f659b37a640 1 -- 172.21.15.16:0/1982705624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65941a2270 con 0x7f6594101ba0 2024-04-22T16:01:19.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.882+0000 7f659b37a640 1 -- 172.21.15.16:0/1982705624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f659404e040 con 0x7f6594101ba0 2024-04-22T16:01:19.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.882+0000 7f658a7fc640 1 -- 172.21.15.16:0/1982705624 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6590004f60 con 0x7f6594101ba0 2024-04-22T16:01:19.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.883+0000 7f658a7fc640 1 --2- 172.21.15.16:0/1982705624 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f658006ec30 0x7f65800710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:19.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.883+0000 7f658a7fc640 1 -- 172.21.15.16:0/1982705624 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f65900b1a20 con 0x7f6594101ba0 2024-04-22T16:01:19.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.883+0000 7f6599116640 1 --2- 172.21.15.16:0/1982705624 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f658006ec30 0x7f65800710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:19.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.883+0000 7f6599116640 1 --2- 172.21.15.16:0/1982705624 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f658006ec30 0x7f65800710f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6584009b40 tx=0x7f6584005e50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:19.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:19.887+0000 7f658a7fc640 1 -- 172.21.15.16:0/1982705624 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f659007a7d0 con 0x7f6594101ba0 2024-04-22T16:01:20.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:19 smithi027 ceph-mon[83054]: pgmap v272: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:20.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:19 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/1517735187' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-22T16:01:20.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:19 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/3219284395' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-22T16:01:20.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:19 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/4218504638' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T16:01:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.112+0000 7f659b37a640 1 -- 172.21.15.16:0/1982705624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 18, "format": "json"} v 0) v1 -- 0x7f65941076f0 con 0x7f6594101ba0 2024-04-22T16:01:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.113+0000 7f658a7fc640 1 -- 172.21.15.16:0/1982705624 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 18, "format": "json"}]=0 dumped fsmap epoch 18 v38) v1 ==== 107+0+4864 (secure 0 0 0) 0x7f659007de90 con 0x7f6594101ba0 2024-04-22T16:01:20.116 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:20.116 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14504,"name":"cephfs.smithi016.fkkejm","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6827/3484907755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3484907755},{"type":"v1","addr":"172.21.15.16:6827","nonce":3484907755}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24293,"name":"cephfs.smithi027.dgqyes","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.27:6827/476026355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":476026355},{"type":"v1","addr":"172.21.15.27:6827","nonce":476026355}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":34376,"name":"cephfs.smithi016.pgvphu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6829/3518018286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3518018286},{"type":"v1","addr":"172.21.15.16:6829","nonce":3518018286}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:46:27.774571+0000","modified":"2024-04-22T15:58:05.206233+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24279":{"gid":24279,"name":"cephfs.smithi027.iobjds","rank":0,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6825/4216277609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4216277609},{"type":"v1","addr":"172.21.15.27:6825","nonce":4216277609}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24279,"qdb_cluster":[24279]},"id":1}]} 2024-04-22T16:01:20.116 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 18 2024-04-22T16:01:20.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.115+0000 7f659b37a640 1 -- 172.21.15.16:0/1982705624 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f658006ec30 msgr2=0x7f65800710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:20.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.115+0000 7f659b37a640 1 --2- 172.21.15.16:0/1982705624 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f658006ec30 0x7f65800710f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6584009b40 tx=0x7f6584005e50 comp rx=0 tx=0).stop 2024-04-22T16:01:20.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.115+0000 7f659b37a640 1 -- 172.21.15.16:0/1982705624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6594101ba0 msgr2=0x7f65940ff410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:20.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.115+0000 7f659b37a640 1 --2- 172.21.15.16:0/1982705624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6594101ba0 0x7f65940ff410 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6590009230 tx=0x7f6590004040 comp rx=0 tx=0).stop 2024-04-22T16:01:20.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.115+0000 7f659b37a640 1 -- 172.21.15.16:0/1982705624 shutdown_connections 2024-04-22T16:01:20.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.115+0000 7f659b37a640 1 --2- 172.21.15.16:0/1982705624 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f658006ec30 0x7f65800710f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:20.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.115+0000 7f659b37a640 1 --2- 172.21.15.16:0/1982705624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6594101ba0 0x7f65940ff410 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:20.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.115+0000 7f659b37a640 1 --2- 172.21.15.16:0/1982705624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6594101280 0x7f65940feed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:20.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.115+0000 7f659b37a640 1 -- 172.21.15.16:0/1982705624 >> 172.21.15.16:0/1982705624 conn(0x7f65940faa20 msgr2=0x7f65941048c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:20.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.115+0000 7f659b37a640 1 -- 172.21.15.16:0/1982705624 shutdown_connections 2024-04-22T16:01:20.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.115+0000 7f659b37a640 1 -- 172.21.15.16:0/1982705624 wait complete. 2024-04-22T16:01:20.128 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:19 smithi016 ceph-mon[110740]: pgmap v272: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:20.128 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:19 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/1517735187' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-22T16:01:20.128 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:19 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/3219284395' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-22T16:01:20.128 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:19 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/4218504638' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T16:01:20.129 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 18 2024-04-22T16:01:20.129 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 19 2024-04-22T16:01:20.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.229+0000 7fe32efb2640 1 -- 172.21.15.16:0/2601659227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe328101b30 msgr2=0x7fe32810a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:20.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.229+0000 7fe32efb2640 1 --2- 172.21.15.16:0/2601659227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe328101b30 0x7fe32810a010 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe314009880 tx=0x7fe31402bd70 comp rx=0 tx=0).stop 2024-04-22T16:01:20.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.229+0000 7fe32efb2640 1 -- 172.21.15.16:0/2601659227 shutdown_connections 2024-04-22T16:01:20.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.229+0000 7fe32efb2640 1 --2- 172.21.15.16:0/2601659227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe328101b30 0x7fe32810a010 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:20.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.229+0000 7fe32efb2640 1 --2- 172.21.15.16:0/2601659227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe328101210 0x7fe3281015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:20.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.229+0000 7fe32efb2640 1 -- 172.21.15.16:0/2601659227 >> 172.21.15.16:0/2601659227 conn(0x7fe3280fa9f0 msgr2=0x7fe3280fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:20.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.229+0000 7fe32efb2640 1 -- 172.21.15.16:0/2601659227 shutdown_connections 2024-04-22T16:01:20.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.229+0000 7fe32efb2640 1 -- 172.21.15.16:0/2601659227 wait complete. 2024-04-22T16:01:20.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.230+0000 7fe32efb2640 1 Processor -- start 2024-04-22T16:01:20.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.230+0000 7fe32efb2640 1 -- start start 2024-04-22T16:01:20.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.230+0000 7fe32efb2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe328101210 0x7fe32819bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:20.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.230+0000 7fe32efb2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe328101b30 0x7fe32819c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:20.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.230+0000 7fe32efb2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe328195b40 con 0x7fe328101210 2024-04-22T16:01:20.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.230+0000 7fe32efb2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe328195cb0 con 0x7fe328101b30 2024-04-22T16:01:20.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.230+0000 7fe32cd4e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe328101210 0x7fe32819bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:20.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.230+0000 7fe32cd4e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe328101210 0x7fe32819bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38218/0 (socket says 172.21.15.16:38218) 2024-04-22T16:01:20.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.230+0000 7fe32cd4e640 1 -- 172.21.15.16:0/412746626 learned_addr learned my addr 172.21.15.16:0/412746626 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:20.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.230+0000 7fe327fff640 1 --2- 172.21.15.16:0/412746626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe328101b30 0x7fe32819c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:20.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.230+0000 7fe32cd4e640 1 -- 172.21.15.16:0/412746626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe328101b30 msgr2=0x7fe32819c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:20.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.230+0000 7fe32cd4e640 1 --2- 172.21.15.16:0/412746626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe328101b30 0x7fe32819c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:20.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.230+0000 7fe32cd4e640 1 -- 172.21.15.16:0/412746626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe314009560 con 0x7fe328101210 2024-04-22T16:01:20.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.230+0000 7fe32cd4e640 1 --2- 172.21.15.16:0/412746626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe328101210 0x7fe32819bad0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe31c00d8d0 tx=0x7fe31c00dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:20.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.230+0000 7fe325ffb640 1 -- 172.21.15.16:0/412746626 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe31c009c00 con 0x7fe328101210 2024-04-22T16:01:20.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.231+0000 7fe325ffb640 1 -- 172.21.15.16:0/412746626 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe31c009d60 con 0x7fe328101210 2024-04-22T16:01:20.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.231+0000 7fe32efb2640 1 -- 172.21.15.16:0/412746626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe328195f90 con 0x7fe328101210 2024-04-22T16:01:20.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.231+0000 7fe325ffb640 1 -- 172.21.15.16:0/412746626 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe31c0049d0 con 0x7fe328101210 2024-04-22T16:01:20.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.231+0000 7fe32efb2640 1 -- 172.21.15.16:0/412746626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3281964c0 con 0x7fe328101210 2024-04-22T16:01:20.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.232+0000 7fe32efb2640 1 -- 172.21.15.16:0/412746626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe32804e040 con 0x7fe328101210 2024-04-22T16:01:20.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.232+0000 7fe325ffb640 1 -- 172.21.15.16:0/412746626 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe31c004c90 con 0x7fe328101210 2024-04-22T16:01:20.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.233+0000 7fe325ffb640 1 --2- 172.21.15.16:0/412746626 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe31806ec30 0x7fe3180710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:20.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.233+0000 7fe325ffb640 1 -- 172.21.15.16:0/412746626 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe31c08ec30 con 0x7fe328101210 2024-04-22T16:01:20.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.233+0000 7fe327fff640 1 --2- 172.21.15.16:0/412746626 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe31806ec30 0x7fe3180710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:20.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.233+0000 7fe327fff640 1 --2- 172.21.15.16:0/412746626 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe31806ec30 0x7fe3180710f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe328196fb0 tx=0x7fe314000bc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:20.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.237+0000 7fe325ffb640 1 -- 172.21.15.16:0/412746626 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe31c057940 con 0x7fe328101210 2024-04-22T16:01:20.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.466+0000 7fe32efb2640 1 -- 172.21.15.16:0/412746626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 19, "format": "json"} v 0) v1 -- 0x7fe328107680 con 0x7fe328101210 2024-04-22T16:01:20.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.467+0000 7fe325ffb640 1 -- 172.21.15.16:0/412746626 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 19, "format": "json"}]=0 dumped fsmap epoch 19 v38) v1 ==== 107+0+4085 (secure 0 0 0) 0x7fe31c05b000 con 0x7fe328101210 2024-04-22T16:01:20.470 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:20.470 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24293,"name":"cephfs.smithi027.dgqyes","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.27:6827/476026355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":476026355},{"type":"v1","addr":"172.21.15.27:6827","nonce":476026355}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":34376,"name":"cephfs.smithi016.pgvphu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6829/3518018286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3518018286},{"type":"v1","addr":"172.21.15.16:6829","nonce":3518018286}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:46:27.774571+0000","modified":"2024-04-22T15:58:05.206233+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24279":{"gid":24279,"name":"cephfs.smithi027.iobjds","rank":0,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6825/4216277609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4216277609},{"type":"v1","addr":"172.21.15.27:6825","nonce":4216277609}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24279,"qdb_cluster":[24279]},"id":1}]} 2024-04-22T16:01:20.470 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 19 2024-04-22T16:01:20.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.469+0000 7fe32efb2640 1 -- 172.21.15.16:0/412746626 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe31806ec30 msgr2=0x7fe3180710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:20.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.469+0000 7fe32efb2640 1 --2- 172.21.15.16:0/412746626 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe31806ec30 0x7fe3180710f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe328196fb0 tx=0x7fe314000bc0 comp rx=0 tx=0).stop 2024-04-22T16:01:20.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.469+0000 7fe32efb2640 1 -- 172.21.15.16:0/412746626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe328101210 msgr2=0x7fe32819bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:20.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.469+0000 7fe32efb2640 1 --2- 172.21.15.16:0/412746626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe328101210 0x7fe32819bad0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe31c00d8d0 tx=0x7fe31c00dd90 comp rx=0 tx=0).stop 2024-04-22T16:01:20.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.469+0000 7fe32efb2640 1 -- 172.21.15.16:0/412746626 shutdown_connections 2024-04-22T16:01:20.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.469+0000 7fe32efb2640 1 --2- 172.21.15.16:0/412746626 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe31806ec30 0x7fe3180710f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:20.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.469+0000 7fe32efb2640 1 --2- 172.21.15.16:0/412746626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe328101b30 0x7fe32819c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:20.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.469+0000 7fe32efb2640 1 --2- 172.21.15.16:0/412746626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe328101210 0x7fe32819bad0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:20.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.469+0000 7fe32efb2640 1 -- 172.21.15.16:0/412746626 >> 172.21.15.16:0/412746626 conn(0x7fe3280fa9f0 msgr2=0x7fe328104850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:20.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.469+0000 7fe32efb2640 1 -- 172.21.15.16:0/412746626 shutdown_connections 2024-04-22T16:01:20.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.469+0000 7fe32efb2640 1 -- 172.21.15.16:0/412746626 wait complete. 2024-04-22T16:01:20.483 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 19 2024-04-22T16:01:20.483 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 20 2024-04-22T16:01:20.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.583+0000 7ffa8ec8f640 1 -- 172.21.15.16:0/2104365253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa880ff1a0 msgr2=0x7ffa880ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:20.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.583+0000 7ffa8ec8f640 1 --2- 172.21.15.16:0/2104365253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa880ff1a0 0x7ffa880ff580 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ffa80009c20 tx=0x7ffa8002c1b0 comp rx=0 tx=0).stop 2024-04-22T16:01:20.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.583+0000 7ffa8ec8f640 1 -- 172.21.15.16:0/2104365253 shutdown_connections 2024-04-22T16:01:20.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.583+0000 7ffa8ec8f640 1 --2- 172.21.15.16:0/2104365253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa880ffb50 0x7ffa88103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:20.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.583+0000 7ffa8ec8f640 1 --2- 172.21.15.16:0/2104365253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa880ff1a0 0x7ffa880ff580 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:20.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.583+0000 7ffa8ec8f640 1 -- 172.21.15.16:0/2104365253 >> 172.21.15.16:0/2104365253 conn(0x7ffa880faa00 msgr2=0x7ffa880fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:20.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.583+0000 7ffa8ec8f640 1 -- 172.21.15.16:0/2104365253 shutdown_connections 2024-04-22T16:01:20.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.584+0000 7ffa8ec8f640 1 -- 172.21.15.16:0/2104365253 wait complete. 2024-04-22T16:01:20.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.584+0000 7ffa8ec8f640 1 Processor -- start 2024-04-22T16:01:20.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.584+0000 7ffa8ec8f640 1 -- start start 2024-04-22T16:01:20.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.584+0000 7ffa8ec8f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa880ff1a0 0x7ffa88195dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:20.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.584+0000 7ffa8ec8f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa88196300 0x7ffa8819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:20.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.584+0000 7ffa8ec8f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa88196900 con 0x7ffa88196300 2024-04-22T16:01:20.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.584+0000 7ffa8ec8f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa88196a70 con 0x7ffa880ff1a0 2024-04-22T16:01:20.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.584+0000 7ffa87fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa88196300 0x7ffa8819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:20.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.584+0000 7ffa87fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa88196300 0x7ffa8819a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38244/0 (socket says 172.21.15.16:38244) 2024-04-22T16:01:20.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.584+0000 7ffa87fff640 1 -- 172.21.15.16:0/1069204125 learned_addr learned my addr 172.21.15.16:0/1069204125 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:20.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.584+0000 7ffa8ca2b640 1 --2- 172.21.15.16:0/1069204125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa880ff1a0 0x7ffa88195dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:20.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.585+0000 7ffa87fff640 1 -- 172.21.15.16:0/1069204125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa880ff1a0 msgr2=0x7ffa88195dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:20.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.585+0000 7ffa87fff640 1 --2- 172.21.15.16:0/1069204125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa880ff1a0 0x7ffa88195dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:20.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.585+0000 7ffa87fff640 1 -- 172.21.15.16:0/1069204125 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa80009900 con 0x7ffa88196300 2024-04-22T16:01:20.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.585+0000 7ffa87fff640 1 --2- 172.21.15.16:0/1069204125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa88196300 0x7ffa8819a760 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00d8d0 tx=0x7ffa7c00dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:20.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.585+0000 7ffa85ffb640 1 -- 172.21.15.16:0/1069204125 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa7c00bea0 con 0x7ffa88196300 2024-04-22T16:01:20.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.585+0000 7ffa8ec8f640 1 -- 172.21.15.16:0/1069204125 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa8819ad60 con 0x7ffa88196300 2024-04-22T16:01:20.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.585+0000 7ffa85ffb640 1 -- 172.21.15.16:0/1069204125 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa7c009c00 con 0x7ffa88196300 2024-04-22T16:01:20.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.585+0000 7ffa85ffb640 1 -- 172.21.15.16:0/1069204125 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa7c010460 con 0x7ffa88196300 2024-04-22T16:01:20.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.585+0000 7ffa8ec8f640 1 -- 172.21.15.16:0/1069204125 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa8819b2c0 con 0x7ffa88196300 2024-04-22T16:01:20.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.586+0000 7ffa8ec8f640 1 -- 172.21.15.16:0/1069204125 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa8804e040 con 0x7ffa88196300 2024-04-22T16:01:20.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.586+0000 7ffa85ffb640 1 -- 172.21.15.16:0/1069204125 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ffa7c003f10 con 0x7ffa88196300 2024-04-22T16:01:20.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.587+0000 7ffa85ffb640 1 --2- 172.21.15.16:0/1069204125 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ffa7006ec30 0x7ffa700710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:20.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.587+0000 7ffa85ffb640 1 -- 172.21.15.16:0/1069204125 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ffa7c08eb90 con 0x7ffa88196300 2024-04-22T16:01:20.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.587+0000 7ffa8ca2b640 1 --2- 172.21.15.16:0/1069204125 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ffa7006ec30 0x7ffa700710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:20.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.587+0000 7ffa8ca2b640 1 --2- 172.21.15.16:0/1069204125 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ffa7006ec30 0x7ffa700710f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ffa80000f80 tx=0x7ffa80009840 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:20.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.592+0000 7ffa85ffb640 1 -- 172.21.15.16:0/1069204125 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ffa7c057940 con 0x7ffa88196300 2024-04-22T16:01:20.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.822+0000 7ffa8ec8f640 1 -- 172.21.15.16:0/1069204125 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 20, "format": "json"} v 0) v1 -- 0x7ffa880ffb50 con 0x7ffa88196300 2024-04-22T16:01:20.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.822+0000 7ffa85ffb640 1 -- 172.21.15.16:0/1069204125 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 20, "format": "json"}]=0 dumped fsmap epoch 20 v38) v1 ==== 107+0+4932 (secure 0 0 0) 0x7ffa7c05b000 con 0x7ffa88196300 2024-04-22T16:01:20.826 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:20.826 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24293,"name":"cephfs.smithi027.dgqyes","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.27:6827/476026355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":476026355},{"type":"v1","addr":"172.21.15.27:6827","nonce":476026355}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":34376,"name":"cephfs.smithi016.pgvphu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6829/3518018286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3518018286},{"type":"v1","addr":"172.21.15.16:6829","nonce":3518018286}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":34408,"name":"cephfs.smithi016.fkkejm","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6827/2871308685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2871308685},{"type":"v1","addr":"172.21.15.16:6827","nonce":2871308685}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:46:27.774571+0000","modified":"2024-04-22T15:58:05.206233+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24279":{"gid":24279,"name":"cephfs.smithi027.iobjds","rank":0,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6825/4216277609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4216277609},{"type":"v1","addr":"172.21.15.27:6825","nonce":4216277609}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24279,"qdb_cluster":[24279]},"id":1}]} 2024-04-22T16:01:20.826 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 20 2024-04-22T16:01:20.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.824+0000 7ffa8ec8f640 1 -- 172.21.15.16:0/1069204125 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ffa7006ec30 msgr2=0x7ffa700710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:20.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.824+0000 7ffa8ec8f640 1 --2- 172.21.15.16:0/1069204125 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ffa7006ec30 0x7ffa700710f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ffa80000f80 tx=0x7ffa80009840 comp rx=0 tx=0).stop 2024-04-22T16:01:20.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.825+0000 7ffa8ec8f640 1 -- 172.21.15.16:0/1069204125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa88196300 msgr2=0x7ffa8819a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:20.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.825+0000 7ffa8ec8f640 1 --2- 172.21.15.16:0/1069204125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa88196300 0x7ffa8819a760 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00d8d0 tx=0x7ffa7c00dd90 comp rx=0 tx=0).stop 2024-04-22T16:01:20.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.825+0000 7ffa8ec8f640 1 -- 172.21.15.16:0/1069204125 shutdown_connections 2024-04-22T16:01:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.825+0000 7ffa8ec8f640 1 --2- 172.21.15.16:0/1069204125 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ffa7006ec30 0x7ffa700710f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.825+0000 7ffa8ec8f640 1 --2- 172.21.15.16:0/1069204125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa88196300 0x7ffa8819a760 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.825+0000 7ffa8ec8f640 1 --2- 172.21.15.16:0/1069204125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa880ff1a0 0x7ffa88195dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.825+0000 7ffa8ec8f640 1 -- 172.21.15.16:0/1069204125 >> 172.21.15.16:0/1069204125 conn(0x7ffa880faa00 msgr2=0x7ffa881033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.825+0000 7ffa8ec8f640 1 -- 172.21.15.16:0/1069204125 shutdown_connections 2024-04-22T16:01:20.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.825+0000 7ffa8ec8f640 1 -- 172.21.15.16:0/1069204125 wait complete. 2024-04-22T16:01:20.839 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 20 2024-04-22T16:01:20.839 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 21 2024-04-22T16:01:20.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.939+0000 7fdb60986640 1 -- 172.21.15.16:0/3793644448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb5c0ff1c0 msgr2=0x7fdb5c0ff5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:20.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.939+0000 7fdb60986640 1 --2- 172.21.15.16:0/3793644448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb5c0ff1c0 0x7fdb5c0ff5a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009880 tx=0x7fdb4c02bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:20.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.939+0000 7fdb60986640 1 -- 172.21.15.16:0/3793644448 shutdown_connections 2024-04-22T16:01:20.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.939+0000 7fdb60986640 1 --2- 172.21.15.16:0/3793644448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb5c0ffb70 0x7fdb5c103bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:20.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.939+0000 7fdb60986640 1 --2- 172.21.15.16:0/3793644448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb5c0ff1c0 0x7fdb5c0ff5a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:20.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.939+0000 7fdb60986640 1 -- 172.21.15.16:0/3793644448 >> 172.21.15.16:0/3793644448 conn(0x7fdb5c0faa80 msgr2=0x7fdb5c0fcea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:20.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.939+0000 7fdb60986640 1 -- 172.21.15.16:0/3793644448 shutdown_connections 2024-04-22T16:01:20.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.939+0000 7fdb60986640 1 -- 172.21.15.16:0/3793644448 wait complete. 2024-04-22T16:01:20.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.940+0000 7fdb60986640 1 Processor -- start 2024-04-22T16:01:20.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.940+0000 7fdb60986640 1 -- start start 2024-04-22T16:01:20.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.940+0000 7fdb60986640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb5c0ff1c0 0x7fdb5c195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:20.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.940+0000 7fdb60986640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb5c196380 0x7fdb5c19a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:20.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.940+0000 7fdb60986640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c196980 con 0x7fdb5c196380 2024-04-22T16:01:20.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.940+0000 7fdb60986640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c196af0 con 0x7fdb5c0ff1c0 2024-04-22T16:01:20.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.940+0000 7fdb59d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb5c196380 0x7fdb5c19a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:20.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.940+0000 7fdb59d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb5c196380 0x7fdb5c19a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38260/0 (socket says 172.21.15.16:38260) 2024-04-22T16:01:20.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.940+0000 7fdb59d9b640 1 -- 172.21.15.16:0/1864844207 learned_addr learned my addr 172.21.15.16:0/1864844207 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:20.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.940+0000 7fdb5a59c640 1 --2- 172.21.15.16:0/1864844207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb5c0ff1c0 0x7fdb5c195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:20.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.940+0000 7fdb59d9b640 1 -- 172.21.15.16:0/1864844207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb5c0ff1c0 msgr2=0x7fdb5c195e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:20.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.940+0000 7fdb59d9b640 1 --2- 172.21.15.16:0/1864844207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb5c0ff1c0 0x7fdb5c195e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:20.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.940+0000 7fdb59d9b640 1 -- 172.21.15.16:0/1864844207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4c009560 con 0x7fdb5c196380 2024-04-22T16:01:20.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.941+0000 7fdb59d9b640 1 --2- 172.21.15.16:0/1864844207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb5c196380 0x7fdb5c19a7e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdb5000d720 tx=0x7fdb5000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:20.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.941+0000 7fdb4b7fe640 1 -- 172.21.15.16:0/1864844207 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb500099f0 con 0x7fdb5c196380 2024-04-22T16:01:20.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.941+0000 7fdb4b7fe640 1 -- 172.21.15.16:0/1864844207 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb50009b50 con 0x7fdb5c196380 2024-04-22T16:01:20.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.941+0000 7fdb60986640 1 -- 172.21.15.16:0/1864844207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5c19ade0 con 0x7fdb5c196380 2024-04-22T16:01:20.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.941+0000 7fdb4b7fe640 1 -- 172.21.15.16:0/1864844207 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb5000f5b0 con 0x7fdb5c196380 2024-04-22T16:01:20.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.941+0000 7fdb60986640 1 -- 172.21.15.16:0/1864844207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5c19b340 con 0x7fdb5c196380 2024-04-22T16:01:20.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.942+0000 7fdb60986640 1 -- 172.21.15.16:0/1864844207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb5c19b610 con 0x7fdb5c196380 2024-04-22T16:01:20.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.942+0000 7fdb4b7fe640 1 -- 172.21.15.16:0/1864844207 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdb50009cc0 con 0x7fdb5c196380 2024-04-22T16:01:20.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.943+0000 7fdb4b7fe640 1 --2- 172.21.15.16:0/1864844207 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdb4406ec30 0x7fdb440710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:20.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.943+0000 7fdb4b7fe640 1 -- 172.21.15.16:0/1864844207 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fdb5008eb10 con 0x7fdb5c196380 2024-04-22T16:01:20.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.943+0000 7fdb5a59c640 1 --2- 172.21.15.16:0/1864844207 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdb4406ec30 0x7fdb440710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:20.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.943+0000 7fdb5a59c640 1 --2- 172.21.15.16:0/1864844207 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdb4406ec30 0x7fdb440710f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c006010 tx=0x7fdb4c005dd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:20.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:20.947+0000 7fdb4b7fe640 1 -- 172.21.15.16:0/1864844207 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fdb50057940 con 0x7fdb5c196380 2024-04-22T16:01:21.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:20 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/791281752' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 17, "format": "json"}]: dispatch 2024-04-22T16:01:21.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:20 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/1982705624' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 18, "format": "json"}]: dispatch 2024-04-22T16:01:21.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:20 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/412746626' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-04-22T16:01:21.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:20 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/791281752' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 17, "format": "json"}]: dispatch 2024-04-22T16:01:21.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:20 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/1982705624' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 18, "format": "json"}]: dispatch 2024-04-22T16:01:21.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:20 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/412746626' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-04-22T16:01:21.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.175+0000 7fdb60986640 1 -- 172.21.15.16:0/1864844207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 21, "format": "json"} v 0) v1 -- 0x7fdb5c0ffbc0 con 0x7fdb5c196380 2024-04-22T16:01:21.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.175+0000 7fdb4b7fe640 1 -- 172.21.15.16:0/1864844207 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 21, "format": "json"}]=0 dumped fsmap epoch 21 v38) v1 ==== 107+0+4153 (secure 0 0 0) 0x7fdb5005b000 con 0x7fdb5c196380 2024-04-22T16:01:21.178 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:21.179 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24293,"name":"cephfs.smithi027.dgqyes","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.27:6827/476026355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":476026355},{"type":"v1","addr":"172.21.15.27:6827","nonce":476026355}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":34408,"name":"cephfs.smithi016.fkkejm","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6827/2871308685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2871308685},{"type":"v1","addr":"172.21.15.16:6827","nonce":2871308685}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:46:27.774571+0000","modified":"2024-04-22T15:58:05.206233+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24279":{"gid":24279,"name":"cephfs.smithi027.iobjds","rank":0,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6825/4216277609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4216277609},{"type":"v1","addr":"172.21.15.27:6825","nonce":4216277609}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24279,"qdb_cluster":[24279]},"id":1}]} 2024-04-22T16:01:21.179 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 21 2024-04-22T16:01:21.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.177+0000 7fdb60986640 1 -- 172.21.15.16:0/1864844207 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdb4406ec30 msgr2=0x7fdb440710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:21.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.177+0000 7fdb60986640 1 --2- 172.21.15.16:0/1864844207 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdb4406ec30 0x7fdb440710f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c006010 tx=0x7fdb4c005dd0 comp rx=0 tx=0).stop 2024-04-22T16:01:21.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.177+0000 7fdb60986640 1 -- 172.21.15.16:0/1864844207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb5c196380 msgr2=0x7fdb5c19a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:21.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.177+0000 7fdb60986640 1 --2- 172.21.15.16:0/1864844207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb5c196380 0x7fdb5c19a7e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdb5000d720 tx=0x7fdb5000dbe0 comp rx=0 tx=0).stop 2024-04-22T16:01:21.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.178+0000 7fdb60986640 1 -- 172.21.15.16:0/1864844207 shutdown_connections 2024-04-22T16:01:21.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.178+0000 7fdb60986640 1 --2- 172.21.15.16:0/1864844207 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdb4406ec30 0x7fdb440710f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:21.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.178+0000 7fdb60986640 1 --2- 172.21.15.16:0/1864844207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb5c196380 0x7fdb5c19a7e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:21.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.178+0000 7fdb60986640 1 --2- 172.21.15.16:0/1864844207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb5c0ff1c0 0x7fdb5c195e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:21.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.178+0000 7fdb60986640 1 -- 172.21.15.16:0/1864844207 >> 172.21.15.16:0/1864844207 conn(0x7fdb5c0faa80 msgr2=0x7fdb5c1033d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:21.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.178+0000 7fdb60986640 1 -- 172.21.15.16:0/1864844207 shutdown_connections 2024-04-22T16:01:21.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.178+0000 7fdb60986640 1 -- 172.21.15.16:0/1864844207 wait complete. 2024-04-22T16:01:21.192 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 21 2024-04-22T16:01:21.192 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 22 2024-04-22T16:01:21.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.291+0000 7f3244386640 1 -- 172.21.15.16:0/1223231397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f323c0ffb40 msgr2=0x7f323c103b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:21.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.291+0000 7f3244386640 1 --2- 172.21.15.16:0/1223231397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f323c0ffb40 0x7f323c103b80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f322c009880 tx=0x7f322c02bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:21.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.291+0000 7f3244386640 1 -- 172.21.15.16:0/1223231397 shutdown_connections 2024-04-22T16:01:21.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.291+0000 7f3244386640 1 --2- 172.21.15.16:0/1223231397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f323c0ffb40 0x7f323c103b80 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:21.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.291+0000 7f3244386640 1 --2- 172.21.15.16:0/1223231397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f323c0ff190 0x7f323c0ff570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:21.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.291+0000 7f3244386640 1 -- 172.21.15.16:0/1223231397 >> 172.21.15.16:0/1223231397 conn(0x7f323c0fa9f0 msgr2=0x7f323c0fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:21.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.291+0000 7f3244386640 1 -- 172.21.15.16:0/1223231397 shutdown_connections 2024-04-22T16:01:21.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.291+0000 7f3244386640 1 -- 172.21.15.16:0/1223231397 wait complete. 2024-04-22T16:01:21.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.292+0000 7f3244386640 1 Processor -- start 2024-04-22T16:01:21.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.292+0000 7f3244386640 1 -- start start 2024-04-22T16:01:21.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.292+0000 7f3244386640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f323c0ff190 0x7f323c195d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:21.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.292+0000 7f3244386640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f323c196260 0x7f323c19a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.292+0000 7f3244386640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323c196860 con 0x7f323c196260 2024-04-22T16:01:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.292+0000 7f3244386640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323c1969d0 con 0x7f323c0ff190 2024-04-22T16:01:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.292+0000 7f3241921640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f323c196260 0x7f323c19a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.292+0000 7f3241921640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f323c196260 0x7f323c19a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38284/0 (socket says 172.21.15.16:38284) 2024-04-22T16:01:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.292+0000 7f3241921640 1 -- 172.21.15.16:0/2858530556 learned_addr learned my addr 172.21.15.16:0/2858530556 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.292+0000 7f3242122640 1 --2- 172.21.15.16:0/2858530556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f323c0ff190 0x7f323c195d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.292+0000 7f3241921640 1 -- 172.21.15.16:0/2858530556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f323c0ff190 msgr2=0x7f323c195d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.292+0000 7f3241921640 1 --2- 172.21.15.16:0/2858530556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f323c0ff190 0x7f323c195d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.292+0000 7f3241921640 1 -- 172.21.15.16:0/2858530556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c009560 con 0x7f323c196260 2024-04-22T16:01:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.293+0000 7f3241921640 1 --2- 172.21.15.16:0/2858530556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f323c196260 0x7f323c19a6c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f322c036040 tx=0x7f322c02c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.293+0000 7f32337fe640 1 -- 172.21.15.16:0/2858530556 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f322c0035f0 con 0x7f323c196260 2024-04-22T16:01:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.293+0000 7f32337fe640 1 -- 172.21.15.16:0/2858530556 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f322c003c10 con 0x7f323c196260 2024-04-22T16:01:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.293+0000 7f3244386640 1 -- 172.21.15.16:0/2858530556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f323c19ac60 con 0x7f323c196260 2024-04-22T16:01:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.293+0000 7f32337fe640 1 -- 172.21.15.16:0/2858530556 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f322c034440 con 0x7f323c196260 2024-04-22T16:01:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.293+0000 7f3244386640 1 -- 172.21.15.16:0/2858530556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f323c19b1c0 con 0x7f323c196260 2024-04-22T16:01:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.294+0000 7f3244386640 1 -- 172.21.15.16:0/2858530556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f323c04e040 con 0x7f323c196260 2024-04-22T16:01:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.294+0000 7f32337fe640 1 -- 172.21.15.16:0/2858530556 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f322c003750 con 0x7f323c196260 2024-04-22T16:01:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.295+0000 7f32337fe640 1 --2- 172.21.15.16:0/2858530556 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f322806ec30 0x7f32280710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.295+0000 7f32337fe640 1 -- 172.21.15.16:0/2858530556 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f322c0b10b0 con 0x7f323c196260 2024-04-22T16:01:21.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.295+0000 7f3242122640 1 --2- 172.21.15.16:0/2858530556 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f322806ec30 0x7f32280710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:21.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.295+0000 7f3242122640 1 --2- 172.21.15.16:0/2858530556 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f322806ec30 0x7f32280710f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3238005d00 tx=0x7f323800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:21.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.299+0000 7f32337fe640 1 -- 172.21.15.16:0/2858530556 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f322c079f90 con 0x7f323c196260 2024-04-22T16:01:21.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.531+0000 7f3244386640 1 -- 172.21.15.16:0/2858530556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 22, "format": "json"} v 0) v1 -- 0x7f323c197070 con 0x7f323c196260 2024-04-22T16:01:21.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.531+0000 7f32337fe640 1 -- 172.21.15.16:0/2858530556 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 22, "format": "json"}]=0 dumped fsmap epoch 22 v38) v1 ==== 107+0+5000 (secure 0 0 0) 0x7f322c07d650 con 0x7f323c196260 2024-04-22T16:01:21.534 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:21.534 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24293,"name":"cephfs.smithi027.dgqyes","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.27:6827/476026355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":476026355},{"type":"v1","addr":"172.21.15.27:6827","nonce":476026355}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":34408,"name":"cephfs.smithi016.fkkejm","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6827/2871308685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2871308685},{"type":"v1","addr":"172.21.15.16:6827","nonce":2871308685}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":34412,"name":"cephfs.smithi016.pgvphu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6829/4228489800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4228489800},{"type":"v1","addr":"172.21.15.16:6829","nonce":4228489800}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:46:27.774571+0000","modified":"2024-04-22T15:58:05.206233+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24279":{"gid":24279,"name":"cephfs.smithi027.iobjds","rank":0,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6825/4216277609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4216277609},{"type":"v1","addr":"172.21.15.27:6825","nonce":4216277609}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24279,"qdb_cluster":[24279]},"id":1}]} 2024-04-22T16:01:21.534 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 22 2024-04-22T16:01:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.534+0000 7f3244386640 1 -- 172.21.15.16:0/2858530556 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f322806ec30 msgr2=0x7f32280710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.534+0000 7f3244386640 1 --2- 172.21.15.16:0/2858530556 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f322806ec30 0x7f32280710f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3238005d00 tx=0x7f323800b410 comp rx=0 tx=0).stop 2024-04-22T16:01:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.534+0000 7f3244386640 1 -- 172.21.15.16:0/2858530556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f323c196260 msgr2=0x7f323c19a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.534+0000 7f3244386640 1 --2- 172.21.15.16:0/2858530556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f323c196260 0x7f323c19a6c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f322c036040 tx=0x7f322c02c850 comp rx=0 tx=0).stop 2024-04-22T16:01:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.534+0000 7f3244386640 1 -- 172.21.15.16:0/2858530556 shutdown_connections 2024-04-22T16:01:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.534+0000 7f3244386640 1 --2- 172.21.15.16:0/2858530556 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f322806ec30 0x7f32280710f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.534+0000 7f3244386640 1 --2- 172.21.15.16:0/2858530556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f323c196260 0x7f323c19a6c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.534+0000 7f3244386640 1 --2- 172.21.15.16:0/2858530556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f323c0ff190 0x7f323c195d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.534+0000 7f3244386640 1 -- 172.21.15.16:0/2858530556 >> 172.21.15.16:0/2858530556 conn(0x7f323c0fa9f0 msgr2=0x7f323c103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.534+0000 7f3244386640 1 -- 172.21.15.16:0/2858530556 shutdown_connections 2024-04-22T16:01:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.534+0000 7f3244386640 1 -- 172.21.15.16:0/2858530556 wait complete. 2024-04-22T16:01:21.547 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 22 2024-04-22T16:01:21.548 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 23 2024-04-22T16:01:21.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.649+0000 7f0ea114e640 1 -- 172.21.15.16:0/565073842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c067340 msgr2=0x7f0e9c067780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:21.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.649+0000 7f0ea114e640 1 --2- 172.21.15.16:0/565073842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c067340 0x7f0e9c067780 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009880 tx=0x7f0e8c02bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:21.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.649+0000 7f0ea114e640 1 -- 172.21.15.16:0/565073842 shutdown_connections 2024-04-22T16:01:21.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.649+0000 7f0ea114e640 1 --2- 172.21.15.16:0/565073842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c067340 0x7f0e9c067780 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:21.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.649+0000 7f0ea114e640 1 --2- 172.21.15.16:0/565073842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e9c1060c0 0x7f0e9c1064a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:21.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.649+0000 7f0ea114e640 1 -- 172.21.15.16:0/565073842 >> 172.21.15.16:0/565073842 conn(0x7f0e9c075b50 msgr2=0x7f0e9c075f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:21.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.649+0000 7f0ea114e640 1 -- 172.21.15.16:0/565073842 shutdown_connections 2024-04-22T16:01:21.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.649+0000 7f0ea114e640 1 -- 172.21.15.16:0/565073842 wait complete. 2024-04-22T16:01:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.650+0000 7f0ea114e640 1 Processor -- start 2024-04-22T16:01:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.650+0000 7f0ea114e640 1 -- start start 2024-04-22T16:01:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.650+0000 7f0ea114e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e9c067340 0x7f0e9c195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.650+0000 7f0ea114e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c1060c0 0x7f0e9c196280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.650+0000 7f0ea114e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9c196910 con 0x7f0e9c1060c0 2024-04-22T16:01:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.650+0000 7f0ea114e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9c19a6e0 con 0x7f0e9c067340 2024-04-22T16:01:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.650+0000 7f0e9a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c1060c0 0x7f0e9c196280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.650+0000 7f0e9a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c1060c0 0x7f0e9c196280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47754/0 (socket says 172.21.15.16:47754) 2024-04-22T16:01:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.650+0000 7f0e9a59c640 1 -- 172.21.15.16:0/4063773680 learned_addr learned my addr 172.21.15.16:0/4063773680 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.651+0000 7f0e9ad9d640 1 --2- 172.21.15.16:0/4063773680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e9c067340 0x7f0e9c195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.651+0000 7f0e9a59c640 1 -- 172.21.15.16:0/4063773680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e9c067340 msgr2=0x7f0e9c195d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.651+0000 7f0e9a59c640 1 --2- 172.21.15.16:0/4063773680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e9c067340 0x7f0e9c195d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.651+0000 7f0e9a59c640 1 -- 172.21.15.16:0/4063773680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8c009560 con 0x7f0e9c1060c0 2024-04-22T16:01:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.651+0000 7f0e9ad9d640 1 --2- 172.21.15.16:0/4063773680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e9c067340 0x7f0e9c195d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T16:01:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.651+0000 7f0e9a59c640 1 --2- 172.21.15.16:0/4063773680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c1060c0 0x7f0e9c196280 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009230 tx=0x7f0e8c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.651+0000 7f0e83fff640 1 -- 172.21.15.16:0/4063773680 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e8c0035f0 con 0x7f0e9c1060c0 2024-04-22T16:01:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.651+0000 7f0ea114e640 1 -- 172.21.15.16:0/4063773680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e9c19a960 con 0x7f0e9c1060c0 2024-04-22T16:01:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.651+0000 7f0e83fff640 1 -- 172.21.15.16:0/4063773680 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e8c003c10 con 0x7f0e9c1060c0 2024-04-22T16:01:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.651+0000 7f0e83fff640 1 -- 172.21.15.16:0/4063773680 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e8c005670 con 0x7f0e9c1060c0 2024-04-22T16:01:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.651+0000 7f0ea114e640 1 -- 172.21.15.16:0/4063773680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e9c19ae60 con 0x7f0e9c1060c0 2024-04-22T16:01:21.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.652+0000 7f0ea114e640 1 -- 172.21.15.16:0/4063773680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e9c04e040 con 0x7f0e9c1060c0 2024-04-22T16:01:21.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.653+0000 7f0e83fff640 1 -- 172.21.15.16:0/4063773680 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0e8c004dd0 con 0x7f0e9c1060c0 2024-04-22T16:01:21.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.653+0000 7f0e83fff640 1 --2- 172.21.15.16:0/4063773680 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0e8406ebe0 0x7f0e840710a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:21.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.653+0000 7f0e83fff640 1 -- 172.21.15.16:0/4063773680 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0e8c0b1270 con 0x7f0e9c1060c0 2024-04-22T16:01:21.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.653+0000 7f0e9ad9d640 1 --2- 172.21.15.16:0/4063773680 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0e8406ebe0 0x7f0e840710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:21.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.654+0000 7f0e9ad9d640 1 --2- 172.21.15.16:0/4063773680 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0e8406ebe0 0x7f0e840710a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0e90009b40 tx=0x7f0e90005e50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:21.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.658+0000 7f0e83fff640 1 -- 172.21.15.16:0/4063773680 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0e8c07a020 con 0x7f0e9c1060c0 2024-04-22T16:01:21.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.884+0000 7f0ea114e640 1 -- 172.21.15.16:0/4063773680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 23, "format": "json"} v 0) v1 -- 0x7f0e9c108540 con 0x7f0e9c1060c0 2024-04-22T16:01:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.885+0000 7f0e83fff640 1 -- 172.21.15.16:0/4063773680 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 23, "format": "json"}]=0 dumped fsmap epoch 23 v38) v1 ==== 107+0+4202 (secure 0 0 0) 0x7f0e8c07d6e0 con 0x7f0e9c1060c0 2024-04-22T16:01:21.888 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:21.888 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24293,"name":"cephfs.smithi027.dgqyes","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.27:6827/476026355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":476026355},{"type":"v1","addr":"172.21.15.27:6827","nonce":476026355}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":34408,"name":"cephfs.smithi016.fkkejm","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6827/2871308685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2871308685},{"type":"v1","addr":"172.21.15.16:6827","nonce":2871308685}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":34412,"name":"cephfs.smithi016.pgvphu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6829/4228489800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4228489800},{"type":"v1","addr":"172.21.15.16:6829","nonce":4228489800}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:46:27.774571+0000","modified":"2024-04-22T15:58:44.933037+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:01:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 23 2024-04-22T16:01:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.887+0000 7f0ea114e640 1 -- 172.21.15.16:0/4063773680 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0e8406ebe0 msgr2=0x7f0e840710a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.887+0000 7f0ea114e640 1 --2- 172.21.15.16:0/4063773680 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0e8406ebe0 0x7f0e840710a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0e90009b40 tx=0x7f0e90005e50 comp rx=0 tx=0).stop 2024-04-22T16:01:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.887+0000 7f0ea114e640 1 -- 172.21.15.16:0/4063773680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c1060c0 msgr2=0x7f0e9c196280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.887+0000 7f0ea114e640 1 --2- 172.21.15.16:0/4063773680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c1060c0 0x7f0e9c196280 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009230 tx=0x7f0e8c004040 comp rx=0 tx=0).stop 2024-04-22T16:01:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.887+0000 7f0ea114e640 1 -- 172.21.15.16:0/4063773680 shutdown_connections 2024-04-22T16:01:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.887+0000 7f0ea114e640 1 --2- 172.21.15.16:0/4063773680 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0e8406ebe0 0x7f0e840710a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.887+0000 7f0ea114e640 1 --2- 172.21.15.16:0/4063773680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c1060c0 0x7f0e9c196280 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.887+0000 7f0ea114e640 1 --2- 172.21.15.16:0/4063773680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e9c067340 0x7f0e9c195d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.887+0000 7f0ea114e640 1 -- 172.21.15.16:0/4063773680 >> 172.21.15.16:0/4063773680 conn(0x7f0e9c075b50 msgr2=0x7f0e9c0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.888+0000 7f0ea114e640 1 -- 172.21.15.16:0/4063773680 shutdown_connections 2024-04-22T16:01:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:21.888+0000 7f0ea114e640 1 -- 172.21.15.16:0/4063773680 wait complete. 2024-04-22T16:01:21.900 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:21 smithi016 ceph-mon[110740]: pgmap v273: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:01:21.901 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:21 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/1069204125' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-04-22T16:01:21.901 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:21 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/1864844207' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-04-22T16:01:21.901 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:21 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/2858530556' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-22T16:01:21.902 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 23 2024-04-22T16:01:21.902 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 24 2024-04-22T16:01:22.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.001+0000 7f1f2667d640 1 -- 172.21.15.16:0/1122519765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f200fefe0 msgr2=0x7f1f200ff420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:22.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.001+0000 7f1f2667d640 1 --2- 172.21.15.16:0/1122519765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f200fefe0 0x7f1f200ff420 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1f14009880 tx=0x7f1f1402bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:22.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.002+0000 7f1f2667d640 1 -- 172.21.15.16:0/1122519765 shutdown_connections 2024-04-22T16:01:22.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.002+0000 7f1f2667d640 1 --2- 172.21.15.16:0/1122519765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f200fefe0 0x7f1f200ff420 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:22.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.002+0000 7f1f2667d640 1 --2- 172.21.15.16:0/1122519765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f20105000 0x7f1f201053e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:22.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.002+0000 7f1f2667d640 1 -- 172.21.15.16:0/1122519765 >> 172.21.15.16:0/1122519765 conn(0x7f1f20075af0 msgr2=0x7f1f20075f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:22.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.002+0000 7f1f2667d640 1 -- 172.21.15.16:0/1122519765 shutdown_connections 2024-04-22T16:01:22.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.002+0000 7f1f2667d640 1 -- 172.21.15.16:0/1122519765 wait complete. 2024-04-22T16:01:22.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.002+0000 7f1f2667d640 1 Processor -- start 2024-04-22T16:01:22.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.002+0000 7f1f2667d640 1 -- start start 2024-04-22T16:01:22.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.003+0000 7f1f2667d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f200fefe0 0x7f1f20195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:22.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.003+0000 7f1f2667d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f20105000 0x7f1f20196280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:22.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.003+0000 7f1f2667d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f20196910 con 0x7f1f200fefe0 2024-04-22T16:01:22.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.003+0000 7f1f2667d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2019a6e0 con 0x7f1f20105000 2024-04-22T16:01:22.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.003+0000 7f1f1ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f200fefe0 0x7f1f20195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:22.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.003+0000 7f1f1ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f200fefe0 0x7f1f20195d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47770/0 (socket says 172.21.15.16:47770) 2024-04-22T16:01:22.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.003+0000 7f1f1ffff640 1 -- 172.21.15.16:0/315819173 learned_addr learned my addr 172.21.15.16:0/315819173 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:22.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.003+0000 7f1f1f7fe640 1 --2- 172.21.15.16:0/315819173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f20105000 0x7f1f20196280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:22.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.003+0000 7f1f1ffff640 1 -- 172.21.15.16:0/315819173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f20105000 msgr2=0x7f1f20196280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:22.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.003+0000 7f1f1ffff640 1 --2- 172.21.15.16:0/315819173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f20105000 0x7f1f20196280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:22.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.003+0000 7f1f1ffff640 1 -- 172.21.15.16:0/315819173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f14009560 con 0x7f1f200fefe0 2024-04-22T16:01:22.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.003+0000 7f1f1f7fe640 1 --2- 172.21.15.16:0/315819173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f20105000 0x7f1f20196280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T16:01:22.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.003+0000 7f1f1ffff640 1 --2- 172.21.15.16:0/315819173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f200fefe0 0x7f1f20195d40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000c8f0 tx=0x7f1f1000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:22.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.003+0000 7f1f1d7fa640 1 -- 172.21.15.16:0/315819173 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f10007b80 con 0x7f1f200fefe0 2024-04-22T16:01:22.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.003+0000 7f1f2667d640 1 -- 172.21.15.16:0/315819173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f2019a9c0 con 0x7f1f200fefe0 2024-04-22T16:01:22.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.003+0000 7f1f1d7fa640 1 -- 172.21.15.16:0/315819173 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1f100054e0 con 0x7f1f200fefe0 2024-04-22T16:01:22.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.003+0000 7f1f1d7fa640 1 -- 172.21.15.16:0/315819173 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f1000f450 con 0x7f1f200fefe0 2024-04-22T16:01:22.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.003+0000 7f1f2667d640 1 -- 172.21.15.16:0/315819173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f2019af20 con 0x7f1f200fefe0 2024-04-22T16:01:22.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.004+0000 7f1f2667d640 1 -- 172.21.15.16:0/315819173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f2004e040 con 0x7f1f200fefe0 2024-04-22T16:01:22.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.005+0000 7f1f1d7fa640 1 -- 172.21.15.16:0/315819173 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1f10004f60 con 0x7f1f200fefe0 2024-04-22T16:01:22.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.005+0000 7f1f1d7fa640 1 --2- 172.21.15.16:0/315819173 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f1f0806ec30 0x7f1f080710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:22.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.005+0000 7f1f1d7fa640 1 -- 172.21.15.16:0/315819173 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1f1008ecc0 con 0x7f1f200fefe0 2024-04-22T16:01:22.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.005+0000 7f1f1f7fe640 1 --2- 172.21.15.16:0/315819173 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f1f0806ec30 0x7f1f080710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:22.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.006+0000 7f1f1f7fe640 1 --2- 172.21.15.16:0/315819173 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f1f0806ec30 0x7f1f080710f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1f20197350 tx=0x7f1f1402cc00 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:22.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.010+0000 7f1f1d7fa640 1 -- 172.21.15.16:0/315819173 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1f10057a70 con 0x7f1f200fefe0 2024-04-22T16:01:22.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:21 smithi027 ceph-mon[83054]: pgmap v273: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:01:22.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:21 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/1069204125' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-04-22T16:01:22.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:21 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/1864844207' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-04-22T16:01:22.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:21 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/2858530556' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-22T16:01:22.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.243+0000 7f1f2667d640 1 -- 172.21.15.16:0/315819173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 24, "format": "json"} v 0) v1 -- 0x7f1f20197090 con 0x7f1f200fefe0 2024-04-22T16:01:22.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.244+0000 7f1f1d7fa640 1 -- 172.21.15.16:0/315819173 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 24, "format": "json"}]=0 dumped fsmap epoch 24 v38) v1 ==== 107+0+4213 (secure 0 0 0) 0x7f1f1005b130 con 0x7f1f200fefe0 2024-04-22T16:01:22.247 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:22.247 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34408,"name":"cephfs.smithi016.fkkejm","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6827/2871308685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2871308685},{"type":"v1","addr":"172.21.15.16:6827","nonce":2871308685}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":34412,"name":"cephfs.smithi016.pgvphu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6829/4228489800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4228489800},{"type":"v1","addr":"172.21.15.16:6829","nonce":4228489800}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:46:27.774571+0000","modified":"2024-04-22T15:58:44.937957+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24293":{"gid":24293,"name":"cephfs.smithi027.dgqyes","rank":0,"incarnation":24,"state":"up:replay","state_seq":2,"addr":"172.21.15.27:6827/476026355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":476026355},{"type":"v1","addr":"172.21.15.27:6827","nonce":476026355}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:01:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 24 2024-04-22T16:01:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.246+0000 7f1f2667d640 1 -- 172.21.15.16:0/315819173 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f1f0806ec30 msgr2=0x7f1f080710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.246+0000 7f1f2667d640 1 --2- 172.21.15.16:0/315819173 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f1f0806ec30 0x7f1f080710f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1f20197350 tx=0x7f1f1402cc00 comp rx=0 tx=0).stop 2024-04-22T16:01:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.246+0000 7f1f2667d640 1 -- 172.21.15.16:0/315819173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f200fefe0 msgr2=0x7f1f20195d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.246+0000 7f1f2667d640 1 --2- 172.21.15.16:0/315819173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f200fefe0 0x7f1f20195d40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000c8f0 tx=0x7f1f1000cdb0 comp rx=0 tx=0).stop 2024-04-22T16:01:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.246+0000 7f1f2667d640 1 -- 172.21.15.16:0/315819173 shutdown_connections 2024-04-22T16:01:22.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.246+0000 7f1f2667d640 1 --2- 172.21.15.16:0/315819173 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f1f0806ec30 0x7f1f080710f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:22.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.246+0000 7f1f2667d640 1 --2- 172.21.15.16:0/315819173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f20105000 0x7f1f20196280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:22.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.246+0000 7f1f2667d640 1 --2- 172.21.15.16:0/315819173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f200fefe0 0x7f1f20195d40 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:22.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.246+0000 7f1f2667d640 1 -- 172.21.15.16:0/315819173 >> 172.21.15.16:0/315819173 conn(0x7f1f20075af0 msgr2=0x7f1f200673e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:22.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.246+0000 7f1f2667d640 1 -- 172.21.15.16:0/315819173 shutdown_connections 2024-04-22T16:01:22.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.246+0000 7f1f2667d640 1 -- 172.21.15.16:0/315819173 wait complete. 2024-04-22T16:01:22.260 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 24 2024-04-22T16:01:22.260 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 25 2024-04-22T16:01:22.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.362+0000 7ff14e63f640 1 -- 172.21.15.16:0/272912320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff148067340 msgr2=0x7ff148067780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:22.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.362+0000 7ff14e63f640 1 --2- 172.21.15.16:0/272912320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff148067340 0x7ff148067780 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff130009880 tx=0x7ff13002bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:22.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.362+0000 7ff14e63f640 1 -- 172.21.15.16:0/272912320 shutdown_connections 2024-04-22T16:01:22.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.362+0000 7ff14e63f640 1 --2- 172.21.15.16:0/272912320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff148067340 0x7ff148067780 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:22.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.362+0000 7ff14e63f640 1 --2- 172.21.15.16:0/272912320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1481060c0 0x7ff1481064a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:22.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.362+0000 7ff14e63f640 1 -- 172.21.15.16:0/272912320 >> 172.21.15.16:0/272912320 conn(0x7ff148075b50 msgr2=0x7ff148075f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:22.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.362+0000 7ff14e63f640 1 -- 172.21.15.16:0/272912320 shutdown_connections 2024-04-22T16:01:22.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.362+0000 7ff14e63f640 1 -- 172.21.15.16:0/272912320 wait complete. 2024-04-22T16:01:22.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.363+0000 7ff14e63f640 1 Processor -- start 2024-04-22T16:01:22.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.363+0000 7ff14e63f640 1 -- start start 2024-04-22T16:01:22.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.363+0000 7ff14e63f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff148067340 0x7ff148195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:22.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.363+0000 7ff14e63f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1481060c0 0x7ff148196280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.363+0000 7ff14e63f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff148196910 con 0x7ff1481060c0 2024-04-22T16:01:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.363+0000 7ff14e63f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14819a6e0 con 0x7ff148067340 2024-04-22T16:01:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.363+0000 7ff1477fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1481060c0 0x7ff148196280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.363+0000 7ff1477fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1481060c0 0x7ff148196280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47792/0 (socket says 172.21.15.16:47792) 2024-04-22T16:01:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.363+0000 7ff1477fe640 1 -- 172.21.15.16:0/1697880766 learned_addr learned my addr 172.21.15.16:0/1697880766 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.363+0000 7ff147fff640 1 --2- 172.21.15.16:0/1697880766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff148067340 0x7ff148195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.364+0000 7ff1477fe640 1 -- 172.21.15.16:0/1697880766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff148067340 msgr2=0x7ff148195d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.364+0000 7ff1477fe640 1 --2- 172.21.15.16:0/1697880766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff148067340 0x7ff148195d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.364+0000 7ff1477fe640 1 -- 172.21.15.16:0/1697880766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff130009560 con 0x7ff1481060c0 2024-04-22T16:01:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.364+0000 7ff1477fe640 1 --2- 172.21.15.16:0/1697880766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1481060c0 0x7ff148196280 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff130009230 tx=0x7ff130004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.364+0000 7ff1457fa640 1 -- 172.21.15.16:0/1697880766 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1300035f0 con 0x7ff1481060c0 2024-04-22T16:01:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.364+0000 7ff1457fa640 1 -- 172.21.15.16:0/1697880766 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff130003c10 con 0x7ff1481060c0 2024-04-22T16:01:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.364+0000 7ff14e63f640 1 -- 172.21.15.16:0/1697880766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff14819a960 con 0x7ff1481060c0 2024-04-22T16:01:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.364+0000 7ff1457fa640 1 -- 172.21.15.16:0/1697880766 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff130034440 con 0x7ff1481060c0 2024-04-22T16:01:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.364+0000 7ff14e63f640 1 -- 172.21.15.16:0/1697880766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14819ae60 con 0x7ff1481060c0 2024-04-22T16:01:22.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.365+0000 7ff14e63f640 1 -- 172.21.15.16:0/1697880766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff14804e040 con 0x7ff1481060c0 2024-04-22T16:01:22.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.366+0000 7ff1457fa640 1 -- 172.21.15.16:0/1697880766 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff130004fd0 con 0x7ff1481060c0 2024-04-22T16:01:22.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.366+0000 7ff1457fa640 1 --2- 172.21.15.16:0/1697880766 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ff13406ec30 0x7ff1340710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:22.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.366+0000 7ff1457fa640 1 -- 172.21.15.16:0/1697880766 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff1300b12e0 con 0x7ff1481060c0 2024-04-22T16:01:22.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.366+0000 7ff147fff640 1 --2- 172.21.15.16:0/1697880766 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ff13406ec30 0x7ff1340710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:22.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.367+0000 7ff147fff640 1 --2- 172.21.15.16:0/1697880766 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ff13406ec30 0x7ff1340710f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff138005fd0 tx=0x7ff138005e20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:22.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.371+0000 7ff1457fa640 1 -- 172.21.15.16:0/1697880766 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff13007a090 con 0x7ff1481060c0 2024-04-22T16:01:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.598+0000 7ff14e63f640 1 -- 172.21.15.16:0/1697880766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 25, "format": "json"} v 0) v1 -- 0x7ff14810d2c0 con 0x7ff1481060c0 2024-04-22T16:01:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.599+0000 7ff1457fa640 1 -- 172.21.15.16:0/1697880766 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 25, "format": "json"}]=0 dumped fsmap epoch 25 v38) v1 ==== 107+0+5060 (secure 0 0 0) 0x7ff13007d750 con 0x7ff1481060c0 2024-04-22T16:01:22.602 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:22.602 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34408,"name":"cephfs.smithi016.fkkejm","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6827/2871308685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2871308685},{"type":"v1","addr":"172.21.15.16:6827","nonce":2871308685}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":34412,"name":"cephfs.smithi016.pgvphu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6829/4228489800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4228489800},{"type":"v1","addr":"172.21.15.16:6829","nonce":4228489800}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":44329,"name":"cephfs.smithi027.iobjds","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6825/3921606817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3921606817},{"type":"v1","addr":"172.21.15.27:6825","nonce":3921606817}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:46:27.774571+0000","modified":"2024-04-22T15:58:44.937957+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24293":{"gid":24293,"name":"cephfs.smithi027.dgqyes","rank":0,"incarnation":24,"state":"up:replay","state_seq":2,"addr":"172.21.15.27:6827/476026355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":476026355},{"type":"v1","addr":"172.21.15.27:6827","nonce":476026355}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:01:22.602 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 25 2024-04-22T16:01:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.601+0000 7ff14e63f640 1 -- 172.21.15.16:0/1697880766 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ff13406ec30 msgr2=0x7ff1340710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.601+0000 7ff14e63f640 1 --2- 172.21.15.16:0/1697880766 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ff13406ec30 0x7ff1340710f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff138005fd0 tx=0x7ff138005e20 comp rx=0 tx=0).stop 2024-04-22T16:01:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.601+0000 7ff14e63f640 1 -- 172.21.15.16:0/1697880766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1481060c0 msgr2=0x7ff148196280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.601+0000 7ff14e63f640 1 --2- 172.21.15.16:0/1697880766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1481060c0 0x7ff148196280 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff130009230 tx=0x7ff130004040 comp rx=0 tx=0).stop 2024-04-22T16:01:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.601+0000 7ff14e63f640 1 -- 172.21.15.16:0/1697880766 shutdown_connections 2024-04-22T16:01:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.601+0000 7ff14e63f640 1 --2- 172.21.15.16:0/1697880766 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7ff13406ec30 0x7ff1340710f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.601+0000 7ff14e63f640 1 --2- 172.21.15.16:0/1697880766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1481060c0 0x7ff148196280 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.601+0000 7ff14e63f640 1 --2- 172.21.15.16:0/1697880766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff148067340 0x7ff148195d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.601+0000 7ff14e63f640 1 -- 172.21.15.16:0/1697880766 >> 172.21.15.16:0/1697880766 conn(0x7ff148075b50 msgr2=0x7ff1480fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.601+0000 7ff14e63f640 1 -- 172.21.15.16:0/1697880766 shutdown_connections 2024-04-22T16:01:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.601+0000 7ff14e63f640 1 -- 172.21.15.16:0/1697880766 wait complete. 2024-04-22T16:01:22.615 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 25 2024-04-22T16:01:22.615 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 26 2024-04-22T16:01:22.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.714+0000 7f18565c3640 1 -- 172.21.15.16:0/3787259964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1850101b20 msgr2=0x7f185010a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:22.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.714+0000 7f18565c3640 1 --2- 172.21.15.16:0/3787259964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1850101b20 0x7f185010a000 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1844009880 tx=0x7f184402bd70 comp rx=0 tx=0).stop 2024-04-22T16:01:22.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.714+0000 7f18565c3640 1 -- 172.21.15.16:0/3787259964 shutdown_connections 2024-04-22T16:01:22.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.714+0000 7f18565c3640 1 --2- 172.21.15.16:0/3787259964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1850101b20 0x7f185010a000 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:22.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.714+0000 7f18565c3640 1 --2- 172.21.15.16:0/3787259964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1850101200 0x7f18501015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:22.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.714+0000 7f18565c3640 1 -- 172.21.15.16:0/3787259964 >> 172.21.15.16:0/3787259964 conn(0x7f18500faa00 msgr2=0x7f18500fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:22.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.714+0000 7f18565c3640 1 -- 172.21.15.16:0/3787259964 shutdown_connections 2024-04-22T16:01:22.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.714+0000 7f18565c3640 1 -- 172.21.15.16:0/3787259964 wait complete. 2024-04-22T16:01:22.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.714+0000 7f18565c3640 1 Processor -- start 2024-04-22T16:01:22.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.714+0000 7f18565c3640 1 -- start start 2024-04-22T16:01:22.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.715+0000 7f18565c3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1850101200 0x7f1850195d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:22.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.715+0000 7f18565c3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1850101b20 0x7f1850196260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:22.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.715+0000 7f18565c3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18501968f0 con 0x7f1850101b20 2024-04-22T16:01:22.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.715+0000 7f18565c3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185019a6c0 con 0x7f1850101200 2024-04-22T16:01:22.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.715+0000 7f184f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1850101b20 0x7f1850196260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:22.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.715+0000 7f184f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1850101b20 0x7f1850196260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47820/0 (socket says 172.21.15.16:47820) 2024-04-22T16:01:22.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.715+0000 7f184f7fe640 1 -- 172.21.15.16:0/1623702579 learned_addr learned my addr 172.21.15.16:0/1623702579 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:22.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.715+0000 7f184ffff640 1 --2- 172.21.15.16:0/1623702579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1850101200 0x7f1850195d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:22.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.715+0000 7f184f7fe640 1 -- 172.21.15.16:0/1623702579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1850101200 msgr2=0x7f1850195d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:22.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.715+0000 7f184f7fe640 1 --2- 172.21.15.16:0/1623702579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1850101200 0x7f1850195d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:22.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.715+0000 7f184f7fe640 1 -- 172.21.15.16:0/1623702579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1844009560 con 0x7f1850101b20 2024-04-22T16:01:22.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.715+0000 7f184ffff640 1 --2- 172.21.15.16:0/1623702579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1850101200 0x7f1850195d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T16:01:22.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.715+0000 7f184f7fe640 1 --2- 172.21.15.16:0/1623702579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1850101b20 0x7f1850196260 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1844009230 tx=0x7f184402c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:22.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.715+0000 7f184d7fa640 1 -- 172.21.15.16:0/1623702579 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1844004520 con 0x7f1850101b20 2024-04-22T16:01:22.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.716+0000 7f184d7fa640 1 -- 172.21.15.16:0/1623702579 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1844003ab0 con 0x7f1850101b20 2024-04-22T16:01:22.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.716+0000 7f18565c3640 1 -- 172.21.15.16:0/1623702579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f185019a940 con 0x7f1850101b20 2024-04-22T16:01:22.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.716+0000 7f184d7fa640 1 -- 172.21.15.16:0/1623702579 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1844005690 con 0x7f1850101b20 2024-04-22T16:01:22.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.716+0000 7f18565c3640 1 -- 172.21.15.16:0/1623702579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f185019aea0 con 0x7f1850101b20 2024-04-22T16:01:22.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.717+0000 7f18565c3640 1 -- 172.21.15.16:0/1623702579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f185004e040 con 0x7f1850101b20 2024-04-22T16:01:22.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.717+0000 7f184d7fa640 1 -- 172.21.15.16:0/1623702579 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1844004ed0 con 0x7f1850101b20 2024-04-22T16:01:22.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.717+0000 7f184d7fa640 1 --2- 172.21.15.16:0/1623702579 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f183806ebe0 0x7f18380710a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:22.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.718+0000 7f184ffff640 1 --2- 172.21.15.16:0/1623702579 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f183806ebe0 0x7f18380710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:22.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.718+0000 7f184d7fa640 1 -- 172.21.15.16:0/1623702579 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f18440b1a10 con 0x7f1850101b20 2024-04-22T16:01:22.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.718+0000 7f184ffff640 1 --2- 172.21.15.16:0/1623702579 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f183806ebe0 0x7f18380710a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1840005fd0 tx=0x7f1840005ea0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:22.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.722+0000 7f184d7fa640 1 -- 172.21.15.16:0/1623702579 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f184407a7c0 con 0x7f1850101b20 2024-04-22T16:01:22.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:22 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/4063773680' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-22T16:01:22.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:22 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/315819173' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-22T16:01:22.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:22 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/1697880766' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-22T16:01:22.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.949+0000 7f18565c3640 1 -- 172.21.15.16:0/1623702579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 26, "format": "json"} v 0) v1 -- 0x7f1850107670 con 0x7f1850101b20 2024-04-22T16:01:22.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.950+0000 7f184d7fa640 1 -- 172.21.15.16:0/1623702579 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 26, "format": "json"}]=0 dumped fsmap epoch 26 v38) v1 ==== 107+0+5065 (secure 0 0 0) 0x7f184407de80 con 0x7f1850101b20 2024-04-22T16:01:22.953 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:22.953 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34408,"name":"cephfs.smithi016.fkkejm","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6827/2871308685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2871308685},{"type":"v1","addr":"172.21.15.16:6827","nonce":2871308685}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":34412,"name":"cephfs.smithi016.pgvphu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6829/4228489800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4228489800},{"type":"v1","addr":"172.21.15.16:6829","nonce":4228489800}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":44329,"name":"cephfs.smithi027.iobjds","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6825/3921606817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3921606817},{"type":"v1","addr":"172.21.15.27:6825","nonce":3921606817}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:46:27.774571+0000","modified":"2024-04-22T15:58:54.655017+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24293":{"gid":24293,"name":"cephfs.smithi027.dgqyes","rank":0,"incarnation":24,"state":"up:reconnect","state_seq":185,"addr":"172.21.15.27:6827/476026355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":476026355},{"type":"v1","addr":"172.21.15.27:6827","nonce":476026355}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:01:22.953 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 26 2024-04-22T16:01:22.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.952+0000 7f18565c3640 1 -- 172.21.15.16:0/1623702579 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f183806ebe0 msgr2=0x7f18380710a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:22.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.952+0000 7f18565c3640 1 --2- 172.21.15.16:0/1623702579 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f183806ebe0 0x7f18380710a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1840005fd0 tx=0x7f1840005ea0 comp rx=0 tx=0).stop 2024-04-22T16:01:22.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.952+0000 7f18565c3640 1 -- 172.21.15.16:0/1623702579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1850101b20 msgr2=0x7f1850196260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:22.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.952+0000 7f18565c3640 1 --2- 172.21.15.16:0/1623702579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1850101b20 0x7f1850196260 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1844009230 tx=0x7f184402c870 comp rx=0 tx=0).stop 2024-04-22T16:01:22.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.952+0000 7f18565c3640 1 -- 172.21.15.16:0/1623702579 shutdown_connections 2024-04-22T16:01:22.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.952+0000 7f18565c3640 1 --2- 172.21.15.16:0/1623702579 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f183806ebe0 0x7f18380710a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:22.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.952+0000 7f18565c3640 1 --2- 172.21.15.16:0/1623702579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1850101b20 0x7f1850196260 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:22.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.952+0000 7f18565c3640 1 --2- 172.21.15.16:0/1623702579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1850101200 0x7f1850195d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:22.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.952+0000 7f18565c3640 1 -- 172.21.15.16:0/1623702579 >> 172.21.15.16:0/1623702579 conn(0x7f18500faa00 msgr2=0x7f1850104840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:22.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.952+0000 7f18565c3640 1 -- 172.21.15.16:0/1623702579 shutdown_connections 2024-04-22T16:01:22.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:22.952+0000 7f18565c3640 1 -- 172.21.15.16:0/1623702579 wait complete. 2024-04-22T16:01:22.966 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 26 2024-04-22T16:01:22.966 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 27 2024-04-22T16:01:23.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.065+0000 7effe5b7a640 1 -- 172.21.15.16:0/2687183086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effe0100c30 msgr2=0x7effe0104c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:23.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.065+0000 7effe5b7a640 1 --2- 172.21.15.16:0/2687183086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effe0100c30 0x7effe0104c70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7effc8009880 tx=0x7effc802bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:23.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.065+0000 7effe5b7a640 1 -- 172.21.15.16:0/2687183086 shutdown_connections 2024-04-22T16:01:23.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.065+0000 7effe5b7a640 1 --2- 172.21.15.16:0/2687183086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effe0100c30 0x7effe0104c70 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:23.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.065+0000 7effe5b7a640 1 --2- 172.21.15.16:0/2687183086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effe0100280 0x7effe0100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:23.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.065+0000 7effe5b7a640 1 -- 172.21.15.16:0/2687183086 >> 172.21.15.16:0/2687183086 conn(0x7effe00fbaa0 msgr2=0x7effe00fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:23.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.065+0000 7effe5b7a640 1 -- 172.21.15.16:0/2687183086 shutdown_connections 2024-04-22T16:01:23.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.065+0000 7effe5b7a640 1 -- 172.21.15.16:0/2687183086 wait complete. 2024-04-22T16:01:23.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.066+0000 7effe5b7a640 1 Processor -- start 2024-04-22T16:01:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.066+0000 7effe5b7a640 1 -- start start 2024-04-22T16:01:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.066+0000 7effe5b7a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effe0100280 0x7effe0195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.066+0000 7effe5b7a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effe0196280 0x7effe019a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.066+0000 7effe5b7a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe0196880 con 0x7effe0196280 2024-04-22T16:01:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.066+0000 7effe5b7a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe01969f0 con 0x7effe0100280 2024-04-22T16:01:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.066+0000 7effdeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effe0196280 0x7effe019a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.066+0000 7effdeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effe0196280 0x7effe019a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47832/0 (socket says 172.21.15.16:47832) 2024-04-22T16:01:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.066+0000 7effdeffd640 1 -- 172.21.15.16:0/4258165962 learned_addr learned my addr 172.21.15.16:0/4258165962 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.067+0000 7effdf7fe640 1 --2- 172.21.15.16:0/4258165962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effe0100280 0x7effe0195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.067+0000 7effdeffd640 1 -- 172.21.15.16:0/4258165962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effe0100280 msgr2=0x7effe0195d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.067+0000 7effdeffd640 1 --2- 172.21.15.16:0/4258165962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effe0100280 0x7effe0195d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.067+0000 7effdeffd640 1 -- 172.21.15.16:0/4258165962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc8009560 con 0x7effe0196280 2024-04-22T16:01:23.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.067+0000 7effdeffd640 1 --2- 172.21.15.16:0/4258165962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effe0196280 0x7effe019a6e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7effc8036040 tx=0x7effc802c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:23.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.067+0000 7effdcff9640 1 -- 172.21.15.16:0/4258165962 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effc80035f0 con 0x7effe0196280 2024-04-22T16:01:23.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.067+0000 7effe5b7a640 1 -- 172.21.15.16:0/4258165962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe019ac80 con 0x7effe0196280 2024-04-22T16:01:23.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.067+0000 7effdcff9640 1 -- 172.21.15.16:0/4258165962 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effc8003c10 con 0x7effe0196280 2024-04-22T16:01:23.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.067+0000 7effdcff9640 1 -- 172.21.15.16:0/4258165962 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effc8034440 con 0x7effe0196280 2024-04-22T16:01:23.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.067+0000 7effe5b7a640 1 -- 172.21.15.16:0/4258165962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe019b1e0 con 0x7effe0196280 2024-04-22T16:01:23.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.068+0000 7effe5b7a640 1 -- 172.21.15.16:0/4258165962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe004e040 con 0x7effe0196280 2024-04-22T16:01:23.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.069+0000 7effdcff9640 1 -- 172.21.15.16:0/4258165962 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7effc8003750 con 0x7effe0196280 2024-04-22T16:01:23.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.069+0000 7effdcff9640 1 --2- 172.21.15.16:0/4258165962 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7effcc06ec30 0x7effcc0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:23.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.069+0000 7effdcff9640 1 -- 172.21.15.16:0/4258165962 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7effc80b1170 con 0x7effe0196280 2024-04-22T16:01:23.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.069+0000 7effdf7fe640 1 --2- 172.21.15.16:0/4258165962 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7effcc06ec30 0x7effcc0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:23.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.070+0000 7effdf7fe640 1 --2- 172.21.15.16:0/4258165962 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7effcc06ec30 0x7effcc0710f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7effd0005fd0 tx=0x7effd0009680 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:23.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.074+0000 7effdcff9640 1 -- 172.21.15.16:0/4258165962 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7effc8079fd0 con 0x7effe0196280 2024-04-22T16:01:23.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:22 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/4063773680' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-22T16:01:23.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:22 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/315819173' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-22T16:01:23.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:22 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/1697880766' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-22T16:01:23.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.301+0000 7effe5b7a640 1 -- 172.21.15.16:0/4258165962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 27, "format": "json"} v 0) v1 -- 0x7effe0108580 con 0x7effe0196280 2024-04-22T16:01:23.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.301+0000 7effdcff9640 1 -- 172.21.15.16:0/4258165962 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 27, "format": "json"}]=0 dumped fsmap epoch 27 v38) v1 ==== 107+0+5062 (secure 0 0 0) 0x7effc807d690 con 0x7effe0196280 2024-04-22T16:01:23.304 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:23.304 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34408,"name":"cephfs.smithi016.fkkejm","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6827/2871308685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2871308685},{"type":"v1","addr":"172.21.15.16:6827","nonce":2871308685}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":34412,"name":"cephfs.smithi016.pgvphu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6829/4228489800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4228489800},{"type":"v1","addr":"172.21.15.16:6829","nonce":4228489800}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":44329,"name":"cephfs.smithi027.iobjds","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6825/3921606817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3921606817},{"type":"v1","addr":"172.21.15.27:6825","nonce":3921606817}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:46:27.774571+0000","modified":"2024-04-22T15:58:55.325110+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24293":{"gid":24293,"name":"cephfs.smithi027.dgqyes","rank":0,"incarnation":24,"state":"up:rejoin","state_seq":186,"addr":"172.21.15.27:6827/476026355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":476026355},{"type":"v1","addr":"172.21.15.27:6827","nonce":476026355}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:01:23.305 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-04-22T16:01:23.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.304+0000 7effe5b7a640 1 -- 172.21.15.16:0/4258165962 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7effcc06ec30 msgr2=0x7effcc0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:23.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.304+0000 7effe5b7a640 1 --2- 172.21.15.16:0/4258165962 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7effcc06ec30 0x7effcc0710f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7effd0005fd0 tx=0x7effd0009680 comp rx=0 tx=0).stop 2024-04-22T16:01:23.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.304+0000 7effe5b7a640 1 -- 172.21.15.16:0/4258165962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effe0196280 msgr2=0x7effe019a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:23.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.304+0000 7effe5b7a640 1 --2- 172.21.15.16:0/4258165962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effe0196280 0x7effe019a6e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7effc8036040 tx=0x7effc802c850 comp rx=0 tx=0).stop 2024-04-22T16:01:23.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.304+0000 7effe5b7a640 1 -- 172.21.15.16:0/4258165962 shutdown_connections 2024-04-22T16:01:23.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.304+0000 7effe5b7a640 1 --2- 172.21.15.16:0/4258165962 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7effcc06ec30 0x7effcc0710f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:23.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.304+0000 7effe5b7a640 1 --2- 172.21.15.16:0/4258165962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effe0196280 0x7effe019a6e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:23.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.304+0000 7effe5b7a640 1 --2- 172.21.15.16:0/4258165962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effe0100280 0x7effe0195d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:23.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.304+0000 7effe5b7a640 1 -- 172.21.15.16:0/4258165962 >> 172.21.15.16:0/4258165962 conn(0x7effe00fbaa0 msgr2=0x7effe00fd210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:23.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.304+0000 7effe5b7a640 1 -- 172.21.15.16:0/4258165962 shutdown_connections 2024-04-22T16:01:23.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.304+0000 7effe5b7a640 1 -- 172.21.15.16:0/4258165962 wait complete. 2024-04-22T16:01:23.318 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 27 2024-04-22T16:01:23.318 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 28 2024-04-22T16:01:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.417+0000 7f9a85e36640 1 -- 172.21.15.16:0/765481480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a80101210 msgr2=0x7f9a801015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.417+0000 7f9a85e36640 1 --2- 172.21.15.16:0/765481480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a80101210 0x7f9a801015f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9a70009880 tx=0x7f9a7002bd70 comp rx=0 tx=0).stop 2024-04-22T16:01:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.417+0000 7f9a85e36640 1 -- 172.21.15.16:0/765481480 shutdown_connections 2024-04-22T16:01:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.417+0000 7f9a85e36640 1 --2- 172.21.15.16:0/765481480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a80101b30 0x7f9a8010a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.417+0000 7f9a85e36640 1 --2- 172.21.15.16:0/765481480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a80101210 0x7f9a801015f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.417+0000 7f9a85e36640 1 -- 172.21.15.16:0/765481480 >> 172.21.15.16:0/765481480 conn(0x7f9a800fa9f0 msgr2=0x7f9a800fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.417+0000 7f9a85e36640 1 -- 172.21.15.16:0/765481480 shutdown_connections 2024-04-22T16:01:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.417+0000 7f9a85e36640 1 -- 172.21.15.16:0/765481480 wait complete. 2024-04-22T16:01:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.418+0000 7f9a85e36640 1 Processor -- start 2024-04-22T16:01:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.418+0000 7f9a85e36640 1 -- start start 2024-04-22T16:01:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.418+0000 7f9a85e36640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a80101210 0x7f9a80195d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.418+0000 7f9a85e36640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a80101b30 0x7f9a801962b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.418+0000 7f9a85e36640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a80196940 con 0x7f9a80101210 2024-04-22T16:01:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.418+0000 7f9a85e36640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8019a6c0 con 0x7f9a80101b30 2024-04-22T16:01:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.418+0000 7f9a7f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a80101210 0x7f9a80195d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.418+0000 7f9a7f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a80101210 0x7f9a80195d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47850/0 (socket says 172.21.15.16:47850) 2024-04-22T16:01:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.418+0000 7f9a7f7fe640 1 -- 172.21.15.16:0/1661460132 learned_addr learned my addr 172.21.15.16:0/1661460132 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.419+0000 7f9a7effd640 1 --2- 172.21.15.16:0/1661460132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a80101b30 0x7f9a801962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.419+0000 7f9a7f7fe640 1 -- 172.21.15.16:0/1661460132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a80101b30 msgr2=0x7f9a801962b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.419+0000 7f9a7f7fe640 1 --2- 172.21.15.16:0/1661460132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a80101b30 0x7f9a801962b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.419+0000 7f9a7f7fe640 1 -- 172.21.15.16:0/1661460132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a70009560 con 0x7f9a80101210 2024-04-22T16:01:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.419+0000 7f9a7effd640 1 --2- 172.21.15.16:0/1661460132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a80101b30 0x7f9a801962b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T16:01:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.419+0000 7f9a7f7fe640 1 --2- 172.21.15.16:0/1661460132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a80101210 0x7f9a80195d70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9a70009230 tx=0x7f9a7002c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.419+0000 7f9a7cff9640 1 -- 172.21.15.16:0/1661460132 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a700044a0 con 0x7f9a80101210 2024-04-22T16:01:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.419+0000 7f9a7cff9640 1 -- 172.21.15.16:0/1661460132 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9a70003ab0 con 0x7f9a80101210 2024-04-22T16:01:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.419+0000 7f9a85e36640 1 -- 172.21.15.16:0/1661460132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a8019a940 con 0x7f9a80101210 2024-04-22T16:01:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.419+0000 7f9a7cff9640 1 -- 172.21.15.16:0/1661460132 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a70005600 con 0x7f9a80101210 2024-04-22T16:01:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.419+0000 7f9a85e36640 1 -- 172.21.15.16:0/1661460132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a8019aea0 con 0x7f9a80101210 2024-04-22T16:01:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.420+0000 7f9a85e36640 1 -- 172.21.15.16:0/1661460132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a8004e040 con 0x7f9a80101210 2024-04-22T16:01:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.421+0000 7f9a7cff9640 1 -- 172.21.15.16:0/1661460132 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9a70004dd0 con 0x7f9a80101210 2024-04-22T16:01:23.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.421+0000 7f9a7cff9640 1 --2- 172.21.15.16:0/1661460132 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f9a6806ec30 0x7f9a680710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:23.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.421+0000 7f9a7cff9640 1 -- 172.21.15.16:0/1661460132 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9a700b1920 con 0x7f9a80101210 2024-04-22T16:01:23.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.421+0000 7f9a7effd640 1 --2- 172.21.15.16:0/1661460132 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f9a6806ec30 0x7f9a680710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:23.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.421+0000 7f9a7effd640 1 --2- 172.21.15.16:0/1661460132 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f9a6806ec30 0x7f9a680710f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9a80197330 tx=0x7f9a74005f90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:23.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.426+0000 7f9a7cff9640 1 -- 172.21.15.16:0/1661460132 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f9a7007a750 con 0x7f9a80101210 2024-04-22T16:01:23.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.655+0000 7f9a85e36640 1 -- 172.21.15.16:0/1661460132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 28, "format": "json"} v 0) v1 -- 0x7f9a80107680 con 0x7f9a80101210 2024-04-22T16:01:23.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.656+0000 7f9a7cff9640 1 -- 172.21.15.16:0/1661460132 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 28, "format": "json"}]=0 dumped fsmap epoch 28 v38) v1 ==== 107+0+5071 (secure 0 0 0) 0x7f9a7007de10 con 0x7f9a80101210 2024-04-22T16:01:23.659 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:23.659 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34408,"name":"cephfs.smithi016.fkkejm","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6827/2871308685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2871308685},{"type":"v1","addr":"172.21.15.16:6827","nonce":2871308685}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":34412,"name":"cephfs.smithi016.pgvphu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6829/4228489800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4228489800},{"type":"v1","addr":"172.21.15.16:6829","nonce":4228489800}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":44329,"name":"cephfs.smithi027.iobjds","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6825/3921606817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3921606817},{"type":"v1","addr":"172.21.15.27:6825","nonce":3921606817}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:46:27.774571+0000","modified":"2024-04-22T15:58:57.328993+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24293":{"gid":24293,"name":"cephfs.smithi027.dgqyes","rank":0,"incarnation":24,"state":"up:active","state_seq":187,"addr":"172.21.15.27:6827/476026355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":476026355},{"type":"v1","addr":"172.21.15.27:6827","nonce":476026355}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24293,"qdb_cluster":[24293]},"id":1}]} 2024-04-22T16:01:23.659 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-04-22T16:01:23.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.658+0000 7f9a85e36640 1 -- 172.21.15.16:0/1661460132 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f9a6806ec30 msgr2=0x7f9a680710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:23.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.658+0000 7f9a85e36640 1 --2- 172.21.15.16:0/1661460132 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f9a6806ec30 0x7f9a680710f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9a80197330 tx=0x7f9a74005f90 comp rx=0 tx=0).stop 2024-04-22T16:01:23.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.658+0000 7f9a85e36640 1 -- 172.21.15.16:0/1661460132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a80101210 msgr2=0x7f9a80195d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:23.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.658+0000 7f9a85e36640 1 --2- 172.21.15.16:0/1661460132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a80101210 0x7f9a80195d70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9a70009230 tx=0x7f9a7002c7a0 comp rx=0 tx=0).stop 2024-04-22T16:01:23.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.658+0000 7f9a85e36640 1 -- 172.21.15.16:0/1661460132 shutdown_connections 2024-04-22T16:01:23.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.658+0000 7f9a85e36640 1 --2- 172.21.15.16:0/1661460132 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f9a6806ec30 0x7f9a680710f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.658+0000 7f9a85e36640 1 --2- 172.21.15.16:0/1661460132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a80101b30 0x7f9a801962b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.658+0000 7f9a85e36640 1 --2- 172.21.15.16:0/1661460132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a80101210 0x7f9a80195d70 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.658+0000 7f9a85e36640 1 -- 172.21.15.16:0/1661460132 >> 172.21.15.16:0/1661460132 conn(0x7f9a800fa9f0 msgr2=0x7f9a80104850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.658+0000 7f9a85e36640 1 -- 172.21.15.16:0/1661460132 shutdown_connections 2024-04-22T16:01:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.658+0000 7f9a85e36640 1 -- 172.21.15.16:0/1661460132 wait complete. 2024-04-22T16:01:23.672 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 28 2024-04-22T16:01:23.672 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 29 2024-04-22T16:01:23.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.773+0000 7f8aae76d640 1 -- 172.21.15.16:0/2323980356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa80ffb30 msgr2=0x7f8aa8103b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:23.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.773+0000 7f8aae76d640 1 --2- 172.21.15.16:0/2323980356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa80ffb30 0x7f8aa8103b70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c009880 tx=0x7f8a9c02bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:23.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.773+0000 7f8aae76d640 1 -- 172.21.15.16:0/2323980356 shutdown_connections 2024-04-22T16:01:23.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.773+0000 7f8aae76d640 1 --2- 172.21.15.16:0/2323980356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa80ffb30 0x7f8aa8103b70 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:23.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.773+0000 7f8aae76d640 1 --2- 172.21.15.16:0/2323980356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8aa80ff180 0x7f8aa80ff560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:23.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.773+0000 7f8aae76d640 1 -- 172.21.15.16:0/2323980356 >> 172.21.15.16:0/2323980356 conn(0x7f8aa80faa20 msgr2=0x7f8aa80fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:23.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.773+0000 7f8aae76d640 1 -- 172.21.15.16:0/2323980356 shutdown_connections 2024-04-22T16:01:23.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.773+0000 7f8aae76d640 1 -- 172.21.15.16:0/2323980356 wait complete. 2024-04-22T16:01:23.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.773+0000 7f8aae76d640 1 Processor -- start 2024-04-22T16:01:23.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.773+0000 7f8aae76d640 1 -- start start 2024-04-22T16:01:23.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.774+0000 7f8aae76d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa80ff180 0x7f8aa8195d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:23.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.774+0000 7f8aae76d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8aa81962d0 0x7f8aa819a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:23.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.774+0000 7f8aae76d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa81968d0 con 0x7f8aa80ff180 2024-04-22T16:01:23.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.774+0000 7f8aae76d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa8196a40 con 0x7f8aa81962d0 2024-04-22T16:01:23.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.774+0000 7f8aa7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa80ff180 0x7f8aa8195d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:23.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.774+0000 7f8aa7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa80ff180 0x7f8aa8195d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47866/0 (socket says 172.21.15.16:47866) 2024-04-22T16:01:23.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.774+0000 7f8aa7fff640 1 -- 172.21.15.16:0/2917677543 learned_addr learned my addr 172.21.15.16:0/2917677543 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:23.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.774+0000 7f8aa77fe640 1 --2- 172.21.15.16:0/2917677543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8aa81962d0 0x7f8aa819a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:23.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.774+0000 7f8aa7fff640 1 -- 172.21.15.16:0/2917677543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8aa81962d0 msgr2=0x7f8aa819a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:23.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.774+0000 7f8aa7fff640 1 --2- 172.21.15.16:0/2917677543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8aa81962d0 0x7f8aa819a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:23.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.774+0000 7f8aa7fff640 1 -- 172.21.15.16:0/2917677543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9c009560 con 0x7f8aa80ff180 2024-04-22T16:01:23.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.774+0000 7f8aa7fff640 1 --2- 172.21.15.16:0/2917677543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa80ff180 0x7f8aa8195d90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8a9800d900 tx=0x7f8a9800ddc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:23.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.774+0000 7f8aa57fa640 1 -- 172.21.15.16:0/2917677543 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a98009c00 con 0x7f8aa80ff180 2024-04-22T16:01:23.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.774+0000 7f8aae76d640 1 -- 172.21.15.16:0/2917677543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aa819ad30 con 0x7f8aa80ff180 2024-04-22T16:01:23.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.775+0000 7f8aa57fa640 1 -- 172.21.15.16:0/2917677543 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a98009d60 con 0x7f8aa80ff180 2024-04-22T16:01:23.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.775+0000 7f8aa57fa640 1 -- 172.21.15.16:0/2917677543 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a9800f710 con 0x7f8aa80ff180 2024-04-22T16:01:23.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.775+0000 7f8aae76d640 1 -- 172.21.15.16:0/2917677543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aa819b290 con 0x7f8aa80ff180 2024-04-22T16:01:23.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.775+0000 7f8aae76d640 1 -- 172.21.15.16:0/2917677543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aa804e040 con 0x7f8aa80ff180 2024-04-22T16:01:23.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.776+0000 7f8aa57fa640 1 -- 172.21.15.16:0/2917677543 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8a98010460 con 0x7f8aa80ff180 2024-04-22T16:01:23.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.776+0000 7f8aa57fa640 1 --2- 172.21.15.16:0/2917677543 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f8a9006ec30 0x7f8a900710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:23.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.777+0000 7f8aa57fa640 1 -- 172.21.15.16:0/2917677543 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8a9808eb70 con 0x7f8aa80ff180 2024-04-22T16:01:23.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.777+0000 7f8aa77fe640 1 --2- 172.21.15.16:0/2917677543 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f8a9006ec30 0x7f8a900710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:23.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.777+0000 7f8aa77fe640 1 --2- 172.21.15.16:0/2917677543 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f8a9006ec30 0x7f8a900710f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c000c00 tx=0x7f8a9c005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:23.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:23.781+0000 7f8aa57fa640 1 -- 172.21.15.16:0/2917677543 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8a980579d0 con 0x7f8aa80ff180 2024-04-22T16:01:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.006+0000 7f8aae76d640 1 -- 172.21.15.16:0/2917677543 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 29, "format": "json"} v 0) v1 -- 0x7f8aa80ffb30 con 0x7f8aa80ff180 2024-04-22T16:01:24.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.006+0000 7f8aa57fa640 1 -- 172.21.15.16:0/2917677543 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 29, "format": "json"}]=0 dumped fsmap epoch 29 v38) v1 ==== 107+0+4273 (secure 0 0 0) 0x7f8a9805b090 con 0x7f8aa80ff180 2024-04-22T16:01:24.009 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:24.009 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34408,"name":"cephfs.smithi016.fkkejm","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6827/2871308685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2871308685},{"type":"v1","addr":"172.21.15.16:6827","nonce":2871308685}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":34412,"name":"cephfs.smithi016.pgvphu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6829/4228489800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4228489800},{"type":"v1","addr":"172.21.15.16:6829","nonce":4228489800}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":44329,"name":"cephfs.smithi027.iobjds","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6825/3921606817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3921606817},{"type":"v1","addr":"172.21.15.27:6825","nonce":3921606817}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:46:27.774571+0000","modified":"2024-04-22T15:59:06.181498+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:01:24.009 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 29 2024-04-22T16:01:24.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.008+0000 7f8aae76d640 1 -- 172.21.15.16:0/2917677543 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f8a9006ec30 msgr2=0x7f8a900710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:24.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.008+0000 7f8aae76d640 1 --2- 172.21.15.16:0/2917677543 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f8a9006ec30 0x7f8a900710f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c000c00 tx=0x7f8a9c005fd0 comp rx=0 tx=0).stop 2024-04-22T16:01:24.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.008+0000 7f8aae76d640 1 -- 172.21.15.16:0/2917677543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa80ff180 msgr2=0x7f8aa8195d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:24.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.008+0000 7f8aae76d640 1 --2- 172.21.15.16:0/2917677543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa80ff180 0x7f8aa8195d90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8a9800d900 tx=0x7f8a9800ddc0 comp rx=0 tx=0).stop 2024-04-22T16:01:24.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.008+0000 7f8aae76d640 1 -- 172.21.15.16:0/2917677543 shutdown_connections 2024-04-22T16:01:24.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.008+0000 7f8aae76d640 1 --2- 172.21.15.16:0/2917677543 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f8a9006ec30 0x7f8a900710f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:24.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.008+0000 7f8aae76d640 1 --2- 172.21.15.16:0/2917677543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8aa81962d0 0x7f8aa819a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:24.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.008+0000 7f8aae76d640 1 --2- 172.21.15.16:0/2917677543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8aa80ff180 0x7f8aa8195d90 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:24.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.008+0000 7f8aae76d640 1 -- 172.21.15.16:0/2917677543 >> 172.21.15.16:0/2917677543 conn(0x7f8aa80faa20 msgr2=0x7f8aa8103320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:24.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.009+0000 7f8aae76d640 1 -- 172.21.15.16:0/2917677543 shutdown_connections 2024-04-22T16:01:24.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.009+0000 7f8aae76d640 1 -- 172.21.15.16:0/2917677543 wait complete. 2024-04-22T16:01:24.021 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:23 smithi016 ceph-mon[110740]: pgmap v274: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:24.021 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:23 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/1623702579' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-22T16:01:24.021 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:23 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/4258165962' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-22T16:01:24.021 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:23 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/1661460132' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-22T16:01:24.022 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 29 2024-04-22T16:01:24.023 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 30 2024-04-22T16:01:24.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:23 smithi027 ceph-mon[83054]: pgmap v274: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:24.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:23 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/1623702579' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-22T16:01:24.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:23 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/4258165962' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-22T16:01:24.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:23 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/1661460132' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-22T16:01:24.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.122+0000 7f07ace3a640 1 -- 172.21.15.16:0/1411314384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07a81002a0 msgr2=0x7f07a8100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:24.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.122+0000 7f07ace3a640 1 --2- 172.21.15.16:0/1411314384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07a81002a0 0x7f07a8100680 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0798009880 tx=0x7f079802bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:24.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.122+0000 7f07ace3a640 1 -- 172.21.15.16:0/1411314384 shutdown_connections 2024-04-22T16:01:24.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.122+0000 7f07ace3a640 1 --2- 172.21.15.16:0/1411314384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07a8100c50 0x7f07a8104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:24.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.122+0000 7f07ace3a640 1 --2- 172.21.15.16:0/1411314384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07a81002a0 0x7f07a8100680 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:24.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.122+0000 7f07ace3a640 1 -- 172.21.15.16:0/1411314384 >> 172.21.15.16:0/1411314384 conn(0x7f07a80fbb20 msgr2=0x7f07a80fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:24.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.122+0000 7f07ace3a640 1 -- 172.21.15.16:0/1411314384 shutdown_connections 2024-04-22T16:01:24.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.122+0000 7f07ace3a640 1 -- 172.21.15.16:0/1411314384 wait complete. 2024-04-22T16:01:24.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.123+0000 7f07ace3a640 1 Processor -- start 2024-04-22T16:01:24.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.123+0000 7f07ace3a640 1 -- start start 2024-04-22T16:01:24.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.123+0000 7f07ace3a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07a81002a0 0x7f07a80725a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:24.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.123+0000 7f07ace3a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07a8072ae0 0x7f07a8072f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:24.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.123+0000 7f07ace3a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a81a1fd0 con 0x7f07a81002a0 2024-04-22T16:01:24.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.123+0000 7f07ace3a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a81a2110 con 0x7f07a8072ae0 2024-04-22T16:01:24.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.123+0000 7f07a659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07a81002a0 0x7f07a80725a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:24.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.123+0000 7f07a659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07a81002a0 0x7f07a80725a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47876/0 (socket says 172.21.15.16:47876) 2024-04-22T16:01:24.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.123+0000 7f07a659c640 1 -- 172.21.15.16:0/2186000186 learned_addr learned my addr 172.21.15.16:0/2186000186 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:24.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.123+0000 7f07a659c640 1 -- 172.21.15.16:0/2186000186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07a8072ae0 msgr2=0x7f07a8072f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:24.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.123+0000 7f07a5d9b640 1 --2- 172.21.15.16:0/2186000186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07a8072ae0 0x7f07a8072f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:24.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.123+0000 7f07a659c640 1 --2- 172.21.15.16:0/2186000186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07a8072ae0 0x7f07a8072f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:24.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.123+0000 7f07a659c640 1 -- 172.21.15.16:0/2186000186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0798009560 con 0x7f07a81002a0 2024-04-22T16:01:24.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.124+0000 7f07a659c640 1 --2- 172.21.15.16:0/2186000186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07a81002a0 0x7f07a80725a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0798009430 tx=0x7f0798005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:24.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.124+0000 7f07977fe640 1 -- 172.21.15.16:0/2186000186 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07980044a0 con 0x7f07a81002a0 2024-04-22T16:01:24.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.124+0000 7f07977fe640 1 -- 172.21.15.16:0/2186000186 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0798003ab0 con 0x7f07a81002a0 2024-04-22T16:01:24.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.124+0000 7f07ace3a640 1 -- 172.21.15.16:0/2186000186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a81a2250 con 0x7f07a81002a0 2024-04-22T16:01:24.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.124+0000 7f07977fe640 1 -- 172.21.15.16:0/2186000186 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07980055f0 con 0x7f07a81002a0 2024-04-22T16:01:24.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.124+0000 7f07ace3a640 1 -- 172.21.15.16:0/2186000186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a81a2700 con 0x7f07a81002a0 2024-04-22T16:01:24.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.125+0000 7f07ace3a640 1 -- 172.21.15.16:0/2186000186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a804e040 con 0x7f07a81002a0 2024-04-22T16:01:24.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.125+0000 7f07977fe640 1 -- 172.21.15.16:0/2186000186 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0798004e60 con 0x7f07a81002a0 2024-04-22T16:01:24.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.126+0000 7f07977fe640 1 --2- 172.21.15.16:0/2186000186 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f079006ec30 0x7f07900710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:24.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.126+0000 7f07a5d9b640 1 --2- 172.21.15.16:0/2186000186 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f079006ec30 0x7f07900710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:24.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.126+0000 7f07977fe640 1 -- 172.21.15.16:0/2186000186 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f07980b1220 con 0x7f07a81002a0 2024-04-22T16:01:24.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.126+0000 7f07a5d9b640 1 --2- 172.21.15.16:0/2186000186 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f079006ec30 0x7f07900710f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f07a8069690 tx=0x7f079c005f90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:24.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.131+0000 7f07977fe640 1 -- 172.21.15.16:0/2186000186 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0798079fd0 con 0x7f07a81002a0 2024-04-22T16:01:24.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.359+0000 7f07ace3a640 1 -- 172.21.15.16:0/2186000186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 30, "format": "json"} v 0) v1 -- 0x7f07a81085a0 con 0x7f07a81002a0 2024-04-22T16:01:24.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.360+0000 7f07977fe640 1 -- 172.21.15.16:0/2186000186 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 30, "format": "json"}]=0 dumped fsmap epoch 30 v38) v1 ==== 107+0+4352 (secure 0 0 0) 0x7f079809ce60 con 0x7f07a81002a0 2024-04-22T16:01:24.363 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:24.363 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34412,"name":"cephfs.smithi016.pgvphu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6829/4228489800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4228489800},{"type":"v1","addr":"172.21.15.16:6829","nonce":4228489800}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":44329,"name":"cephfs.smithi027.iobjds","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6825/3921606817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3921606817},{"type":"v1","addr":"172.21.15.27:6825","nonce":3921606817}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:46:27.774571+0000","modified":"2024-04-22T15:59:06.186147+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34408":{"gid":34408,"name":"cephfs.smithi016.fkkejm","rank":0,"incarnation":30,"state":"up:replay","state_seq":1,"addr":"172.21.15.16:6827/2871308685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2871308685},{"type":"v1","addr":"172.21.15.16:6827","nonce":2871308685}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:01:24.363 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 30 2024-04-22T16:01:24.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.362+0000 7f07ace3a640 1 -- 172.21.15.16:0/2186000186 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f079006ec30 msgr2=0x7f07900710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:24.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.362+0000 7f07ace3a640 1 --2- 172.21.15.16:0/2186000186 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f079006ec30 0x7f07900710f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f07a8069690 tx=0x7f079c005f90 comp rx=0 tx=0).stop 2024-04-22T16:01:24.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.362+0000 7f07ace3a640 1 -- 172.21.15.16:0/2186000186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07a81002a0 msgr2=0x7f07a80725a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:24.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.362+0000 7f07ace3a640 1 --2- 172.21.15.16:0/2186000186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07a81002a0 0x7f07a80725a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0798009430 tx=0x7f0798005e50 comp rx=0 tx=0).stop 2024-04-22T16:01:24.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.362+0000 7f07ace3a640 1 -- 172.21.15.16:0/2186000186 shutdown_connections 2024-04-22T16:01:24.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.362+0000 7f07ace3a640 1 --2- 172.21.15.16:0/2186000186 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f079006ec30 0x7f07900710f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:24.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.362+0000 7f07ace3a640 1 --2- 172.21.15.16:0/2186000186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07a8072ae0 0x7f07a8072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:24.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.362+0000 7f07ace3a640 1 --2- 172.21.15.16:0/2186000186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07a81002a0 0x7f07a80725a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:24.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.362+0000 7f07ace3a640 1 -- 172.21.15.16:0/2186000186 >> 172.21.15.16:0/2186000186 conn(0x7f07a80fbb20 msgr2=0x7f07a80fd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:24.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.362+0000 7f07ace3a640 1 -- 172.21.15.16:0/2186000186 shutdown_connections 2024-04-22T16:01:24.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.362+0000 7f07ace3a640 1 -- 172.21.15.16:0/2186000186 wait complete. 2024-04-22T16:01:24.376 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 30 2024-04-22T16:01:24.376 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 31 2024-04-22T16:01:24.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.475+0000 7f4724808640 1 -- 172.21.15.16:0/1052242837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47201060c0 msgr2=0x7f47201064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:24.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.475+0000 7f4724808640 1 --2- 172.21.15.16:0/1052242837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47201060c0 0x7f47201064a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4710009880 tx=0x7f471002bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:24.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.475+0000 7f4724808640 1 -- 172.21.15.16:0/1052242837 shutdown_connections 2024-04-22T16:01:24.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.475+0000 7f4724808640 1 --2- 172.21.15.16:0/1052242837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4720067340 0x7f4720067780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:24.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.475+0000 7f4724808640 1 --2- 172.21.15.16:0/1052242837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47201060c0 0x7f47201064a0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:24.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.475+0000 7f4724808640 1 -- 172.21.15.16:0/1052242837 >> 172.21.15.16:0/1052242837 conn(0x7f4720075b50 msgr2=0x7f4720075f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:24.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.476+0000 7f4724808640 1 -- 172.21.15.16:0/1052242837 shutdown_connections 2024-04-22T16:01:24.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.476+0000 7f4724808640 1 -- 172.21.15.16:0/1052242837 wait complete. 2024-04-22T16:01:24.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.476+0000 7f4724808640 1 Processor -- start 2024-04-22T16:01:24.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.476+0000 7f4724808640 1 -- start start 2024-04-22T16:01:24.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.476+0000 7f4724808640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4720067340 0x7f4720195d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:24.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.476+0000 7f4724808640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47201060c0 0x7f4720196260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:24.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.476+0000 7f4724808640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47201968f0 con 0x7f4720067340 2024-04-22T16:01:24.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.476+0000 7f4724808640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472019a6c0 con 0x7f47201060c0 2024-04-22T16:01:24.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.477+0000 7f471e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4720067340 0x7f4720195d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:24.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.477+0000 7f471e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4720067340 0x7f4720195d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47892/0 (socket says 172.21.15.16:47892) 2024-04-22T16:01:24.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.477+0000 7f471e59c640 1 -- 172.21.15.16:0/1314227501 learned_addr learned my addr 172.21.15.16:0/1314227501 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:24.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.477+0000 7f471dd9b640 1 --2- 172.21.15.16:0/1314227501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47201060c0 0x7f4720196260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:24.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.477+0000 7f471e59c640 1 -- 172.21.15.16:0/1314227501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47201060c0 msgr2=0x7f4720196260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:24.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.477+0000 7f471e59c640 1 --2- 172.21.15.16:0/1314227501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47201060c0 0x7f4720196260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:24.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.477+0000 7f471e59c640 1 -- 172.21.15.16:0/1314227501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4710009560 con 0x7f4720067340 2024-04-22T16:01:24.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.477+0000 7f471e59c640 1 --2- 172.21.15.16:0/1314227501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4720067340 0x7f4720195d20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4710000c00 tx=0x7f471002c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:24.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.477+0000 7f470f7fe640 1 -- 172.21.15.16:0/1314227501 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4710004520 con 0x7f4720067340 2024-04-22T16:01:24.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.477+0000 7f470f7fe640 1 -- 172.21.15.16:0/1314227501 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4710003ab0 con 0x7f4720067340 2024-04-22T16:01:24.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.477+0000 7f4724808640 1 -- 172.21.15.16:0/1314227501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f472019a940 con 0x7f4720067340 2024-04-22T16:01:24.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.477+0000 7f470f7fe640 1 -- 172.21.15.16:0/1314227501 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4710034440 con 0x7f4720067340 2024-04-22T16:01:24.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.477+0000 7f4724808640 1 -- 172.21.15.16:0/1314227501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f472019ae40 con 0x7f4720067340 2024-04-22T16:01:24.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.478+0000 7f4724808640 1 -- 172.21.15.16:0/1314227501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472004e040 con 0x7f4720067340 2024-04-22T16:01:24.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.479+0000 7f470f7fe640 1 -- 172.21.15.16:0/1314227501 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4710003c20 con 0x7f4720067340 2024-04-22T16:01:24.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.479+0000 7f470f7fe640 1 --2- 172.21.15.16:0/1314227501 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f470806ec30 0x7f47080710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:24.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.479+0000 7f470f7fe640 1 -- 172.21.15.16:0/1314227501 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f47100b1160 con 0x7f4720067340 2024-04-22T16:01:24.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.479+0000 7f471dd9b640 1 --2- 172.21.15.16:0/1314227501 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f470806ec30 0x7f47080710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:24.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.480+0000 7f471dd9b640 1 --2- 172.21.15.16:0/1314227501 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f470806ec30 0x7f47080710f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4720197330 tx=0x7f471400b390 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:24.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.484+0000 7f470f7fe640 1 -- 172.21.15.16:0/1314227501 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4710079fc0 con 0x7f4720067340 2024-04-22T16:01:24.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.711+0000 7f4724808640 1 -- 172.21.15.16:0/1314227501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 31, "format": "json"} v 0) v1 -- 0x7f4720108540 con 0x7f4720067340 2024-04-22T16:01:24.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.712+0000 7f470f7fe640 1 -- 172.21.15.16:0/1314227501 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 31, "format": "json"}]=0 dumped fsmap epoch 31 v38) v1 ==== 107+0+5199 (secure 0 0 0) 0x7f471007d680 con 0x7f4720067340 2024-04-22T16:01:24.714 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:24.715 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34412,"name":"cephfs.smithi016.pgvphu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6829/4228489800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4228489800},{"type":"v1","addr":"172.21.15.16:6829","nonce":4228489800}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":44329,"name":"cephfs.smithi027.iobjds","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6825/3921606817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3921606817},{"type":"v1","addr":"172.21.15.27:6825","nonce":3921606817}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":44333,"name":"cephfs.smithi027.dgqyes","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6827/4073572546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4073572546},{"type":"v1","addr":"172.21.15.27:6827","nonce":4073572546}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:46:27.774571+0000","modified":"2024-04-22T15:59:06.186147+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34408":{"gid":34408,"name":"cephfs.smithi016.fkkejm","rank":0,"incarnation":30,"state":"up:replay","state_seq":1,"addr":"172.21.15.16:6827/2871308685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2871308685},{"type":"v1","addr":"172.21.15.16:6827","nonce":2871308685}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:01:24.715 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 31 2024-04-22T16:01:24.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.714+0000 7f4724808640 1 -- 172.21.15.16:0/1314227501 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f470806ec30 msgr2=0x7f47080710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:24.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.714+0000 7f4724808640 1 --2- 172.21.15.16:0/1314227501 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f470806ec30 0x7f47080710f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4720197330 tx=0x7f471400b390 comp rx=0 tx=0).stop 2024-04-22T16:01:24.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.714+0000 7f4724808640 1 -- 172.21.15.16:0/1314227501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4720067340 msgr2=0x7f4720195d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:24.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.714+0000 7f4724808640 1 --2- 172.21.15.16:0/1314227501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4720067340 0x7f4720195d20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4710000c00 tx=0x7f471002c850 comp rx=0 tx=0).stop 2024-04-22T16:01:24.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.714+0000 7f4724808640 1 -- 172.21.15.16:0/1314227501 shutdown_connections 2024-04-22T16:01:24.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.714+0000 7f4724808640 1 --2- 172.21.15.16:0/1314227501 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f470806ec30 0x7f47080710f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:24.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.714+0000 7f4724808640 1 --2- 172.21.15.16:0/1314227501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47201060c0 0x7f4720196260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:24.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.714+0000 7f4724808640 1 --2- 172.21.15.16:0/1314227501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4720067340 0x7f4720195d20 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:24.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.714+0000 7f4724808640 1 -- 172.21.15.16:0/1314227501 >> 172.21.15.16:0/1314227501 conn(0x7f4720075b50 msgr2=0x7f47200fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:24.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.714+0000 7f4724808640 1 -- 172.21.15.16:0/1314227501 shutdown_connections 2024-04-22T16:01:24.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.714+0000 7f4724808640 1 -- 172.21.15.16:0/1314227501 wait complete. 2024-04-22T16:01:24.727 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 31 2024-04-22T16:01:24.728 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 32 2024-04-22T16:01:24.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.827+0000 7f2e537f6640 1 -- 172.21.15.16:0/3043624585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e4c0ff000 msgr2=0x7f2e4c0ff3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:24.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.827+0000 7f2e537f6640 1 --2- 172.21.15.16:0/3043624585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e4c0ff000 0x7f2e4c0ff3e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2e48009880 tx=0x7f2e4802bd70 comp rx=0 tx=0).stop 2024-04-22T16:01:24.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.827+0000 7f2e537f6640 1 -- 172.21.15.16:0/3043624585 shutdown_connections 2024-04-22T16:01:24.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.827+0000 7f2e537f6640 1 --2- 172.21.15.16:0/3043624585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e4c0ff920 0x7f2e4c103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:24.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.827+0000 7f2e537f6640 1 --2- 172.21.15.16:0/3043624585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e4c0ff000 0x7f2e4c0ff3e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:24.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.827+0000 7f2e537f6640 1 -- 172.21.15.16:0/3043624585 >> 172.21.15.16:0/3043624585 conn(0x7f2e4c0fa9f0 msgr2=0x7f2e4c0fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:24.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.827+0000 7f2e537f6640 1 -- 172.21.15.16:0/3043624585 shutdown_connections 2024-04-22T16:01:24.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.827+0000 7f2e537f6640 1 -- 172.21.15.16:0/3043624585 wait complete. 2024-04-22T16:01:24.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.828+0000 7f2e537f6640 1 Processor -- start 2024-04-22T16:01:24.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.828+0000 7f2e537f6640 1 -- start start 2024-04-22T16:01:24.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.828+0000 7f2e537f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e4c0ff000 0x7f2e4c19c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:24.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.828+0000 7f2e537f6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e4c0ff920 0x7f2e4c19c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:24.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.828+0000 7f2e537f6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c19cef0 con 0x7f2e4c0ff000 2024-04-22T16:01:24.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.828+0000 7f2e537f6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c1a0cc0 con 0x7f2e4c0ff920 2024-04-22T16:01:24.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.828+0000 7f2e51592640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e4c0ff000 0x7f2e4c19c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:24.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.828+0000 7f2e51592640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e4c0ff000 0x7f2e4c19c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47906/0 (socket says 172.21.15.16:47906) 2024-04-22T16:01:24.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.828+0000 7f2e51592640 1 -- 172.21.15.16:0/3459971509 learned_addr learned my addr 172.21.15.16:0/3459971509 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:24.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.828+0000 7f2e50d91640 1 --2- 172.21.15.16:0/3459971509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e4c0ff920 0x7f2e4c19c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:24.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.829+0000 7f2e51592640 1 -- 172.21.15.16:0/3459971509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e4c0ff920 msgr2=0x7f2e4c19c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:24.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.829+0000 7f2e51592640 1 --2- 172.21.15.16:0/3459971509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e4c0ff920 0x7f2e4c19c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:24.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.829+0000 7f2e51592640 1 -- 172.21.15.16:0/3459971509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e48009560 con 0x7f2e4c0ff000 2024-04-22T16:01:24.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.829+0000 7f2e50d91640 1 --2- 172.21.15.16:0/3459971509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e4c0ff920 0x7f2e4c19c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T16:01:24.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.829+0000 7f2e51592640 1 --2- 172.21.15.16:0/3459971509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e4c0ff000 0x7f2e4c19c320 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2e48009230 tx=0x7f2e48004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:24.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.829+0000 7f2e427fc640 1 -- 172.21.15.16:0/3459971509 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e480037c0 con 0x7f2e4c0ff000 2024-04-22T16:01:24.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.829+0000 7f2e427fc640 1 -- 172.21.15.16:0/3459971509 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e48004aa0 con 0x7f2e4c0ff000 2024-04-22T16:01:24.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.829+0000 7f2e537f6640 1 -- 172.21.15.16:0/3459971509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e4c1a0f40 con 0x7f2e4c0ff000 2024-04-22T16:01:24.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.829+0000 7f2e427fc640 1 -- 172.21.15.16:0/3459971509 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e4803e450 con 0x7f2e4c0ff000 2024-04-22T16:01:24.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.829+0000 7f2e537f6640 1 -- 172.21.15.16:0/3459971509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e4c1a14a0 con 0x7f2e4c0ff000 2024-04-22T16:01:24.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.830+0000 7f2e537f6640 1 -- 172.21.15.16:0/3459971509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e4c04e040 con 0x7f2e4c0ff000 2024-04-22T16:01:24.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.830+0000 7f2e427fc640 1 -- 172.21.15.16:0/3459971509 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2e48005060 con 0x7f2e4c0ff000 2024-04-22T16:01:24.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.831+0000 7f2e427fc640 1 --2- 172.21.15.16:0/3459971509 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f2e3806ec30 0x7f2e380710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:24.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.831+0000 7f2e427fc640 1 -- 172.21.15.16:0/3459971509 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2e480b1c60 con 0x7f2e4c0ff000 2024-04-22T16:01:24.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.831+0000 7f2e50d91640 1 --2- 172.21.15.16:0/3459971509 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f2e3806ec30 0x7f2e380710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:24.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.831+0000 7f2e50d91640 1 --2- 172.21.15.16:0/3459971509 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f2e3806ec30 0x7f2e380710f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c19d930 tx=0x7f2e3c005dd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:24.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:24.836+0000 7f2e427fc640 1 -- 172.21.15.16:0/3459971509 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2e4807aa90 con 0x7f2e4c0ff000 2024-04-22T16:01:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.063+0000 7f2e537f6640 1 -- 172.21.15.16:0/3459971509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 32, "format": "json"} v 0) v1 -- 0x7f2e4c1014b0 con 0x7f2e4c0ff000 2024-04-22T16:01:25.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.063+0000 7f2e427fc640 1 -- 172.21.15.16:0/3459971509 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 32, "format": "json"}]=0 dumped fsmap epoch 32 v38) v1 ==== 107+0+5203 (secure 0 0 0) 0x7f2e4807e150 con 0x7f2e4c0ff000 2024-04-22T16:01:25.067 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:25.067 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34412,"name":"cephfs.smithi016.pgvphu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6829/4228489800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4228489800},{"type":"v1","addr":"172.21.15.16:6829","nonce":4228489800}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":44329,"name":"cephfs.smithi027.iobjds","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6825/3921606817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3921606817},{"type":"v1","addr":"172.21.15.27:6825","nonce":3921606817}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":44333,"name":"cephfs.smithi027.dgqyes","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6827/4073572546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4073572546},{"type":"v1","addr":"172.21.15.27:6827","nonce":4073572546}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:46:27.774571+0000","modified":"2024-04-22T15:59:15.998792+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34408":{"gid":34408,"name":"cephfs.smithi016.fkkejm","rank":0,"incarnation":30,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.16:6827/2871308685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2871308685},{"type":"v1","addr":"172.21.15.16:6827","nonce":2871308685}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:01:25.067 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2024-04-22T16:01:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.066+0000 7f2e537f6640 1 -- 172.21.15.16:0/3459971509 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f2e3806ec30 msgr2=0x7f2e380710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.066+0000 7f2e537f6640 1 --2- 172.21.15.16:0/3459971509 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f2e3806ec30 0x7f2e380710f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c19d930 tx=0x7f2e3c005dd0 comp rx=0 tx=0).stop 2024-04-22T16:01:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.066+0000 7f2e537f6640 1 -- 172.21.15.16:0/3459971509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e4c0ff000 msgr2=0x7f2e4c19c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.066+0000 7f2e537f6640 1 --2- 172.21.15.16:0/3459971509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e4c0ff000 0x7f2e4c19c320 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2e48009230 tx=0x7f2e48004270 comp rx=0 tx=0).stop 2024-04-22T16:01:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.066+0000 7f2e537f6640 1 -- 172.21.15.16:0/3459971509 shutdown_connections 2024-04-22T16:01:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.066+0000 7f2e537f6640 1 --2- 172.21.15.16:0/3459971509 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f2e3806ec30 0x7f2e380710f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.066+0000 7f2e537f6640 1 --2- 172.21.15.16:0/3459971509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e4c0ff920 0x7f2e4c19c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.066+0000 7f2e537f6640 1 --2- 172.21.15.16:0/3459971509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e4c0ff000 0x7f2e4c19c320 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.066+0000 7f2e537f6640 1 -- 172.21.15.16:0/3459971509 >> 172.21.15.16:0/3459971509 conn(0x7f2e4c0fa9f0 msgr2=0x7f2e4c110460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:25.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.066+0000 7f2e537f6640 1 -- 172.21.15.16:0/3459971509 shutdown_connections 2024-04-22T16:01:25.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.066+0000 7f2e537f6640 1 -- 172.21.15.16:0/3459971509 wait complete. 2024-04-22T16:01:25.078 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:24 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/2917677543' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-22T16:01:25.079 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:24 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/2186000186' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-22T16:01:25.079 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:24 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/1314227501' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-22T16:01:25.080 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 32 2024-04-22T16:01:25.080 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 33 2024-04-22T16:01:25.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:24 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/2917677543' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-22T16:01:25.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:24 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/2186000186' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-22T16:01:25.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:24 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/1314227501' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-22T16:01:25.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.180+0000 7f15ac739640 1 -- 172.21.15.16:0/1313754607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15a4100c50 msgr2=0x7f15a4104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:25.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.180+0000 7f15ac739640 1 --2- 172.21.15.16:0/1313754607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15a4100c50 0x7f15a4104c90 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1594009880 tx=0x7f159402bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:25.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.180+0000 7f15ac739640 1 -- 172.21.15.16:0/1313754607 shutdown_connections 2024-04-22T16:01:25.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.180+0000 7f15ac739640 1 --2- 172.21.15.16:0/1313754607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15a4100c50 0x7f15a4104c90 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:25.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.180+0000 7f15ac739640 1 --2- 172.21.15.16:0/1313754607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15a41002a0 0x7f15a4100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:25.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.180+0000 7f15ac739640 1 -- 172.21.15.16:0/1313754607 >> 172.21.15.16:0/1313754607 conn(0x7f15a40fbb20 msgr2=0x7f15a40fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:25.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.180+0000 7f15ac739640 1 -- 172.21.15.16:0/1313754607 shutdown_connections 2024-04-22T16:01:25.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.180+0000 7f15ac739640 1 -- 172.21.15.16:0/1313754607 wait complete. 2024-04-22T16:01:25.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.180+0000 7f15ac739640 1 Processor -- start 2024-04-22T16:01:25.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.180+0000 7f15ac739640 1 -- start start 2024-04-22T16:01:25.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.181+0000 7f15ac739640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15a41002a0 0x7f15a4195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:25.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.181+0000 7f15ac739640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15a4196280 0x7f15a419a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:25.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.181+0000 7f15ac739640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a4196880 con 0x7f15a4196280 2024-04-22T16:01:25.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.181+0000 7f15ac739640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a41969f0 con 0x7f15a41002a0 2024-04-22T16:01:25.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.181+0000 7f15a9cd4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15a4196280 0x7f15a419a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:25.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.181+0000 7f15a9cd4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15a4196280 0x7f15a419a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47912/0 (socket says 172.21.15.16:47912) 2024-04-22T16:01:25.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.181+0000 7f15a9cd4640 1 -- 172.21.15.16:0/412829727 learned_addr learned my addr 172.21.15.16:0/412829727 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:25.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.181+0000 7f15aa4d5640 1 --2- 172.21.15.16:0/412829727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15a41002a0 0x7f15a4195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:25.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.181+0000 7f15a9cd4640 1 -- 172.21.15.16:0/412829727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15a41002a0 msgr2=0x7f15a4195d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:25.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.181+0000 7f15a9cd4640 1 --2- 172.21.15.16:0/412829727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15a41002a0 0x7f15a4195d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:25.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.181+0000 7f15a9cd4640 1 -- 172.21.15.16:0/412829727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1594009560 con 0x7f15a4196280 2024-04-22T16:01:25.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.181+0000 7f15a9cd4640 1 --2- 172.21.15.16:0/412829727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15a4196280 0x7f15a419a6e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1594036040 tx=0x7f159402c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:25.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.181+0000 7f159b7fe640 1 -- 172.21.15.16:0/412829727 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15940035f0 con 0x7f15a4196280 2024-04-22T16:01:25.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.181+0000 7f159b7fe640 1 -- 172.21.15.16:0/412829727 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1594003c10 con 0x7f15a4196280 2024-04-22T16:01:25.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.181+0000 7f15ac739640 1 -- 172.21.15.16:0/412829727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15a419ac80 con 0x7f15a4196280 2024-04-22T16:01:25.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.181+0000 7f159b7fe640 1 -- 172.21.15.16:0/412829727 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1594034440 con 0x7f15a4196280 2024-04-22T16:01:25.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.182+0000 7f15ac739640 1 -- 172.21.15.16:0/412829727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a419b1e0 con 0x7f15a4196280 2024-04-22T16:01:25.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.182+0000 7f15ac739640 1 -- 172.21.15.16:0/412829727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15a404e040 con 0x7f15a4196280 2024-04-22T16:01:25.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.183+0000 7f159b7fe640 1 -- 172.21.15.16:0/412829727 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1594003750 con 0x7f15a4196280 2024-04-22T16:01:25.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.184+0000 7f159b7fe640 1 --2- 172.21.15.16:0/412829727 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f159006ec30 0x7f15900710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:25.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.184+0000 7f15aa4d5640 1 --2- 172.21.15.16:0/412829727 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f159006ec30 0x7f15900710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:25.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.184+0000 7f159b7fe640 1 -- 172.21.15.16:0/412829727 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f15940b1170 con 0x7f15a4196280 2024-04-22T16:01:25.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.184+0000 7f15aa4d5640 1 --2- 172.21.15.16:0/412829727 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f159006ec30 0x7f15900710f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f15a000bbe0 tx=0x7f15a000b390 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:25.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.188+0000 7f159b7fe640 1 -- 172.21.15.16:0/412829727 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1594079fd0 con 0x7f15a4196280 2024-04-22T16:01:25.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.413+0000 7f15ac739640 1 -- 172.21.15.16:0/412829727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 33, "format": "json"} v 0) v1 -- 0x7f15a41085a0 con 0x7f15a4196280 2024-04-22T16:01:25.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.413+0000 7f159b7fe640 1 -- 172.21.15.16:0/412829727 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 33, "format": "json"}]=0 dumped fsmap epoch 33 v38) v1 ==== 107+0+5200 (secure 0 0 0) 0x7f159407d690 con 0x7f15a4196280 2024-04-22T16:01:25.416 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:25.416 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34412,"name":"cephfs.smithi016.pgvphu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6829/4228489800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4228489800},{"type":"v1","addr":"172.21.15.16:6829","nonce":4228489800}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":44329,"name":"cephfs.smithi027.iobjds","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6825/3921606817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3921606817},{"type":"v1","addr":"172.21.15.27:6825","nonce":3921606817}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":44333,"name":"cephfs.smithi027.dgqyes","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6827/4073572546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4073572546},{"type":"v1","addr":"172.21.15.27:6827","nonce":4073572546}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:46:27.774571+0000","modified":"2024-04-22T15:59:16.270436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34408":{"gid":34408,"name":"cephfs.smithi016.fkkejm","rank":0,"incarnation":30,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.16:6827/2871308685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2871308685},{"type":"v1","addr":"172.21.15.16:6827","nonce":2871308685}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-22T16:01:25.417 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 33 2024-04-22T16:01:25.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.415+0000 7f15ac739640 1 -- 172.21.15.16:0/412829727 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f159006ec30 msgr2=0x7f15900710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:25.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.415+0000 7f15ac739640 1 --2- 172.21.15.16:0/412829727 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f159006ec30 0x7f15900710f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f15a000bbe0 tx=0x7f15a000b390 comp rx=0 tx=0).stop 2024-04-22T16:01:25.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.415+0000 7f15ac739640 1 -- 172.21.15.16:0/412829727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15a4196280 msgr2=0x7f15a419a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:25.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.415+0000 7f15ac739640 1 --2- 172.21.15.16:0/412829727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15a4196280 0x7f15a419a6e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1594036040 tx=0x7f159402c850 comp rx=0 tx=0).stop 2024-04-22T16:01:25.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.415+0000 7f15ac739640 1 -- 172.21.15.16:0/412829727 shutdown_connections 2024-04-22T16:01:25.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.415+0000 7f15ac739640 1 --2- 172.21.15.16:0/412829727 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f159006ec30 0x7f15900710f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:25.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.415+0000 7f15ac739640 1 --2- 172.21.15.16:0/412829727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15a4196280 0x7f15a419a6e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:25.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.415+0000 7f15ac739640 1 --2- 172.21.15.16:0/412829727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15a41002a0 0x7f15a4195d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:25.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.415+0000 7f15ac739640 1 -- 172.21.15.16:0/412829727 >> 172.21.15.16:0/412829727 conn(0x7f15a40fbb20 msgr2=0x7f15a40fd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:25.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.415+0000 7f15ac739640 1 -- 172.21.15.16:0/412829727 shutdown_connections 2024-04-22T16:01:25.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.415+0000 7f15ac739640 1 -- 172.21.15.16:0/412829727 wait complete. 2024-04-22T16:01:25.429 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 33 2024-04-22T16:01:25.429 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 34 2024-04-22T16:01:25.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.529+0000 7f5fd7677640 1 -- 172.21.15.16:0/1546873846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fd00ff1a0 msgr2=0x7f5fd00ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.529+0000 7f5fd7677640 1 --2- 172.21.15.16:0/1546873846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fd00ff1a0 0x7f5fd00ff580 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc009880 tx=0x7f5fcc02bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.529+0000 7f5fd7677640 1 -- 172.21.15.16:0/1546873846 shutdown_connections 2024-04-22T16:01:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.529+0000 7f5fd7677640 1 --2- 172.21.15.16:0/1546873846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fd00ffb50 0x7f5fd0103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.529+0000 7f5fd7677640 1 --2- 172.21.15.16:0/1546873846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fd00ff1a0 0x7f5fd00ff580 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.529+0000 7f5fd7677640 1 -- 172.21.15.16:0/1546873846 >> 172.21.15.16:0/1546873846 conn(0x7f5fd00faa00 msgr2=0x7f5fd00fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.529+0000 7f5fd7677640 1 -- 172.21.15.16:0/1546873846 shutdown_connections 2024-04-22T16:01:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.529+0000 7f5fd7677640 1 -- 172.21.15.16:0/1546873846 wait complete. 2024-04-22T16:01:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.530+0000 7f5fd7677640 1 Processor -- start 2024-04-22T16:01:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.530+0000 7f5fd7677640 1 -- start start 2024-04-22T16:01:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.530+0000 7f5fd7677640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fd00ff1a0 0x7f5fd0195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.530+0000 7f5fd7677640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fd0196290 0x7f5fd019a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.530+0000 7f5fd7677640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd0196890 con 0x7f5fd00ff1a0 2024-04-22T16:01:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.530+0000 7f5fd7677640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd0196a00 con 0x7f5fd0196290 2024-04-22T16:01:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.530+0000 7f5fd5413640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fd00ff1a0 0x7f5fd0195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.530+0000 7f5fd5413640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fd00ff1a0 0x7f5fd0195d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47936/0 (socket says 172.21.15.16:47936) 2024-04-22T16:01:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.530+0000 7f5fd5413640 1 -- 172.21.15.16:0/3152120267 learned_addr learned my addr 172.21.15.16:0/3152120267 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.530+0000 7f5fd5413640 1 -- 172.21.15.16:0/3152120267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fd0196290 msgr2=0x7f5fd019a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.530+0000 7f5fd5413640 1 --2- 172.21.15.16:0/3152120267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fd0196290 0x7f5fd019a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.530+0000 7f5fd5413640 1 -- 172.21.15.16:0/3152120267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fcc009560 con 0x7f5fd00ff1a0 2024-04-22T16:01:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.530+0000 7f5fd5413640 1 --2- 172.21.15.16:0/3152120267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fd00ff1a0 0x7f5fd0195d50 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc009430 tx=0x7f5fcc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.530+0000 7f5fc67fc640 1 -- 172.21.15.16:0/3152120267 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fcc0035f0 con 0x7f5fd00ff1a0 2024-04-22T16:01:25.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.530+0000 7f5fc67fc640 1 -- 172.21.15.16:0/3152120267 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5fcc003c10 con 0x7f5fd00ff1a0 2024-04-22T16:01:25.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.530+0000 7f5fd7677640 1 -- 172.21.15.16:0/3152120267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fd019ac90 con 0x7f5fd00ff1a0 2024-04-22T16:01:25.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.530+0000 7f5fc67fc640 1 -- 172.21.15.16:0/3152120267 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fcc034440 con 0x7f5fd00ff1a0 2024-04-22T16:01:25.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.531+0000 7f5fd7677640 1 -- 172.21.15.16:0/3152120267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fd019b1f0 con 0x7f5fd00ff1a0 2024-04-22T16:01:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.532+0000 7f5fd7677640 1 -- 172.21.15.16:0/3152120267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fd019b520 con 0x7f5fd00ff1a0 2024-04-22T16:01:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.532+0000 7f5fc67fc640 1 -- 172.21.15.16:0/3152120267 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5fcc004fd0 con 0x7f5fd00ff1a0 2024-04-22T16:01:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.532+0000 7f5fc67fc640 1 --2- 172.21.15.16:0/3152120267 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f5fbc06ec30 0x7f5fbc0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.533+0000 7f5fc67fc640 1 -- 172.21.15.16:0/3152120267 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5fcc0b12b0 con 0x7f5fd00ff1a0 2024-04-22T16:01:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.533+0000 7f5fd4c12640 1 --2- 172.21.15.16:0/3152120267 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f5fbc06ec30 0x7f5fbc0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.533+0000 7f5fd4c12640 1 --2- 172.21.15.16:0/3152120267 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f5fbc06ec30 0x7f5fbc0710f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0069630 tx=0x7f5fc0005bf0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:25.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.537+0000 7f5fc67fc640 1 -- 172.21.15.16:0/3152120267 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5fcc07a040 con 0x7f5fd00ff1a0 2024-04-22T16:01:25.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.765+0000 7f5fd7677640 1 -- 172.21.15.16:0/3152120267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 34, "format": "json"} v 0) v1 -- 0x7f5fd004e040 con 0x7f5fd00ff1a0 2024-04-22T16:01:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.766+0000 7f5fc67fc640 1 -- 172.21.15.16:0/3152120267 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 34, "format": "json"}]=0 dumped fsmap epoch 34 v38) v1 ==== 107+0+5209 (secure 0 0 0) 0x7f5fcc07d700 con 0x7f5fd00ff1a0 2024-04-22T16:01:25.769 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34412,"name":"cephfs.smithi016.pgvphu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6829/4228489800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4228489800},{"type":"v1","addr":"172.21.15.16:6829","nonce":4228489800}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":44329,"name":"cephfs.smithi027.iobjds","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6825/3921606817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3921606817},{"type":"v1","addr":"172.21.15.27:6825","nonce":3921606817}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":44333,"name":"cephfs.smithi027.dgqyes","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6827/4073572546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4073572546},{"type":"v1","addr":"172.21.15.27:6827","nonce":4073572546}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:46:27.774571+0000","modified":"2024-04-22T15:59:18.312436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34408":{"gid":34408,"name":"cephfs.smithi016.fkkejm","rank":0,"incarnation":30,"state":"up:active","state_seq":19,"addr":"172.21.15.16:6827/2871308685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2871308685},{"type":"v1","addr":"172.21.15.16:6827","nonce":2871308685}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34408,"qdb_cluster":[34408]},"id":1}]} 2024-04-22T16:01:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 34 2024-04-22T16:01:25.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.768+0000 7f5fd7677640 1 -- 172.21.15.16:0/3152120267 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f5fbc06ec30 msgr2=0x7f5fbc0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:25.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.768+0000 7f5fd7677640 1 --2- 172.21.15.16:0/3152120267 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f5fbc06ec30 0x7f5fbc0710f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0069630 tx=0x7f5fc0005bf0 comp rx=0 tx=0).stop 2024-04-22T16:01:25.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.768+0000 7f5fd7677640 1 -- 172.21.15.16:0/3152120267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fd00ff1a0 msgr2=0x7f5fd0195d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.768+0000 7f5fd7677640 1 --2- 172.21.15.16:0/3152120267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fd00ff1a0 0x7f5fd0195d50 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc009430 tx=0x7f5fcc004040 comp rx=0 tx=0).stop 2024-04-22T16:01:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.768+0000 7f5fd7677640 1 -- 172.21.15.16:0/3152120267 shutdown_connections 2024-04-22T16:01:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.768+0000 7f5fd7677640 1 --2- 172.21.15.16:0/3152120267 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f5fbc06ec30 0x7f5fbc0710f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.768+0000 7f5fd7677640 1 --2- 172.21.15.16:0/3152120267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fd0196290 0x7f5fd019a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.768+0000 7f5fd7677640 1 --2- 172.21.15.16:0/3152120267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fd00ff1a0 0x7f5fd0195d50 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.768+0000 7f5fd7677640 1 -- 172.21.15.16:0/3152120267 >> 172.21.15.16:0/3152120267 conn(0x7f5fd00faa00 msgr2=0x7f5fd0103380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.768+0000 7f5fd7677640 1 -- 172.21.15.16:0/3152120267 shutdown_connections 2024-04-22T16:01:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.768+0000 7f5fd7677640 1 -- 172.21.15.16:0/3152120267 wait complete. 2024-04-22T16:01:25.782 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 34 2024-04-22T16:01:25.782 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 35 2024-04-22T16:01:25.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.881+0000 7f655a8af640 1 -- 172.21.15.16:0/812765239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65541002a0 msgr2=0x7f6554100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:25.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.881+0000 7f655a8af640 1 --2- 172.21.15.16:0/812765239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65541002a0 0x7f6554100680 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6544009880 tx=0x7f654402bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:25.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.881+0000 7f655a8af640 1 -- 172.21.15.16:0/812765239 shutdown_connections 2024-04-22T16:01:25.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.881+0000 7f655a8af640 1 --2- 172.21.15.16:0/812765239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6554100c50 0x7f6554104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:25.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.881+0000 7f655a8af640 1 --2- 172.21.15.16:0/812765239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65541002a0 0x7f6554100680 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:25.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.881+0000 7f655a8af640 1 -- 172.21.15.16:0/812765239 >> 172.21.15.16:0/812765239 conn(0x7f65540fbb20 msgr2=0x7f65540fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:25.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.881+0000 7f655a8af640 1 -- 172.21.15.16:0/812765239 shutdown_connections 2024-04-22T16:01:25.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.881+0000 7f655a8af640 1 -- 172.21.15.16:0/812765239 wait complete. 2024-04-22T16:01:25.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.882+0000 7f655a8af640 1 Processor -- start 2024-04-22T16:01:25.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.882+0000 7f655a8af640 1 -- start start 2024-04-22T16:01:25.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.882+0000 7f655a8af640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65541002a0 0x7f6554195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:25.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.882+0000 7f655a8af640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6554196410 0x7f655419a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:25.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.882+0000 7f655a8af640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6554196a10 con 0x7f65541002a0 2024-04-22T16:01:25.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.882+0000 7f655a8af640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6554196b80 con 0x7f6554196410 2024-04-22T16:01:25.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.882+0000 7f6553fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65541002a0 0x7f6554195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:25.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.882+0000 7f6553fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65541002a0 0x7f6554195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47958/0 (socket says 172.21.15.16:47958) 2024-04-22T16:01:25.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.882+0000 7f6553fff640 1 -- 172.21.15.16:0/1514069772 learned_addr learned my addr 172.21.15.16:0/1514069772 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:25.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.883+0000 7f6553fff640 1 -- 172.21.15.16:0/1514069772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6554196410 msgr2=0x7f655419a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:25.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.883+0000 7f6553fff640 1 --2- 172.21.15.16:0/1514069772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6554196410 0x7f655419a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:25.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.883+0000 7f6553fff640 1 -- 172.21.15.16:0/1514069772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6544009560 con 0x7f65541002a0 2024-04-22T16:01:25.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.883+0000 7f6553fff640 1 --2- 172.21.15.16:0/1514069772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65541002a0 0x7f6554195ed0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6544009430 tx=0x7f6544004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:25.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.883+0000 7f65517fa640 1 -- 172.21.15.16:0/1514069772 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65440035f0 con 0x7f65541002a0 2024-04-22T16:01:25.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.883+0000 7f65517fa640 1 -- 172.21.15.16:0/1514069772 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6544003c10 con 0x7f65541002a0 2024-04-22T16:01:25.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.883+0000 7f655a8af640 1 -- 172.21.15.16:0/1514069772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f655419ae10 con 0x7f65541002a0 2024-04-22T16:01:25.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.883+0000 7f65517fa640 1 -- 172.21.15.16:0/1514069772 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6544034440 con 0x7f65541002a0 2024-04-22T16:01:25.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.883+0000 7f655a8af640 1 -- 172.21.15.16:0/1514069772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f655419b370 con 0x7f65541002a0 2024-04-22T16:01:25.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.884+0000 7f655a8af640 1 -- 172.21.15.16:0/1514069772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f655404e040 con 0x7f65541002a0 2024-04-22T16:01:25.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.884+0000 7f65517fa640 1 -- 172.21.15.16:0/1514069772 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6544004fd0 con 0x7f65541002a0 2024-04-22T16:01:25.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.885+0000 7f65517fa640 1 --2- 172.21.15.16:0/1514069772 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f654006ec30 0x7f65400710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:25.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.885+0000 7f65517fa640 1 -- 172.21.15.16:0/1514069772 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f65440b12c0 con 0x7f65541002a0 2024-04-22T16:01:25.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.885+0000 7f65537fe640 1 --2- 172.21.15.16:0/1514069772 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f654006ec30 0x7f65400710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:25.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.885+0000 7f65537fe640 1 --2- 172.21.15.16:0/1514069772 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f654006ec30 0x7f65400710f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6554069690 tx=0x7f653c006c90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:25.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:25.890+0000 7f65517fa640 1 -- 172.21.15.16:0/1514069772 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f654407a070 con 0x7f65541002a0 2024-04-22T16:01:26.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:25 smithi027 ceph-mon[83054]: pgmap v275: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:01:26.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:25 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/3459971509' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-22T16:01:26.087 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:25 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/412829727' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-22T16:01:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.114+0000 7f655a8af640 1 -- 172.21.15.16:0/1514069772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 35, "format": "json"} v 0) v1 -- 0x7f65541085a0 con 0x7f65541002a0 2024-04-22T16:01:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.115+0000 7f65517fa640 1 -- 172.21.15.16:0/1514069772 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 35, "format": "json"}]=0 dumped fsmap epoch 35 v38) v1 ==== 107+0+5209 (secure 0 0 0) 0x7f654407d730 con 0x7f65541002a0 2024-04-22T16:01:26.118 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:26.119 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34412,"name":"cephfs.smithi016.pgvphu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6829/4228489800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4228489800},{"type":"v1","addr":"172.21.15.16:6829","nonce":4228489800}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":44329,"name":"cephfs.smithi027.iobjds","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6825/3921606817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3921606817},{"type":"v1","addr":"172.21.15.27:6825","nonce":3921606817}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":44333,"name":"cephfs.smithi027.dgqyes","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6827/4073572546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4073572546},{"type":"v1","addr":"172.21.15.27:6827","nonce":4073572546}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:46:27.774571+0000","modified":"2024-04-22T15:59:22.214953+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34408":{"gid":34408,"name":"cephfs.smithi016.fkkejm","rank":0,"incarnation":30,"state":"up:active","state_seq":19,"addr":"172.21.15.16:6827/2871308685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2871308685},{"type":"v1","addr":"172.21.15.16:6827","nonce":2871308685}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34408,"qdb_cluster":[34408]},"id":1}]} 2024-04-22T16:01:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 35 2024-04-22T16:01:26.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.117+0000 7f655a8af640 1 -- 172.21.15.16:0/1514069772 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f654006ec30 msgr2=0x7f65400710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:26.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.117+0000 7f655a8af640 1 --2- 172.21.15.16:0/1514069772 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f654006ec30 0x7f65400710f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6554069690 tx=0x7f653c006c90 comp rx=0 tx=0).stop 2024-04-22T16:01:26.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.117+0000 7f655a8af640 1 -- 172.21.15.16:0/1514069772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65541002a0 msgr2=0x7f6554195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:26.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.117+0000 7f655a8af640 1 --2- 172.21.15.16:0/1514069772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65541002a0 0x7f6554195ed0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6544009430 tx=0x7f6544004040 comp rx=0 tx=0).stop 2024-04-22T16:01:26.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.117+0000 7f655a8af640 1 -- 172.21.15.16:0/1514069772 shutdown_connections 2024-04-22T16:01:26.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.117+0000 7f655a8af640 1 --2- 172.21.15.16:0/1514069772 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f654006ec30 0x7f65400710f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:26.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.117+0000 7f655a8af640 1 --2- 172.21.15.16:0/1514069772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6554196410 0x7f655419a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:26.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.117+0000 7f655a8af640 1 --2- 172.21.15.16:0/1514069772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65541002a0 0x7f6554195ed0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:26.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.117+0000 7f655a8af640 1 -- 172.21.15.16:0/1514069772 >> 172.21.15.16:0/1514069772 conn(0x7f65540fbb20 msgr2=0x7f65540fd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:26.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.118+0000 7f655a8af640 1 -- 172.21.15.16:0/1514069772 shutdown_connections 2024-04-22T16:01:26.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.118+0000 7f655a8af640 1 -- 172.21.15.16:0/1514069772 wait complete. 2024-04-22T16:01:26.130 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:25 smithi016 ceph-mon[110740]: pgmap v275: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:01:26.130 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:25 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/3459971509' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-22T16:01:26.131 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:25 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/412829727' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-22T16:01:26.132 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 35 2024-04-22T16:01:26.132 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 36 2024-04-22T16:01:26.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.232+0000 7fe4f40bd640 1 -- 172.21.15.16:0/430381687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4ec100c50 msgr2=0x7fe4ec104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:26.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.232+0000 7fe4f40bd640 1 --2- 172.21.15.16:0/430381687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4ec100c50 0x7fe4ec104c90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0009880 tx=0x7fe4e002bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:26.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.232+0000 7fe4f40bd640 1 -- 172.21.15.16:0/430381687 shutdown_connections 2024-04-22T16:01:26.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.232+0000 7fe4f40bd640 1 --2- 172.21.15.16:0/430381687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4ec100c50 0x7fe4ec104c90 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:26.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.232+0000 7fe4f40bd640 1 --2- 172.21.15.16:0/430381687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4ec1002a0 0x7fe4ec100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:26.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.232+0000 7fe4f40bd640 1 -- 172.21.15.16:0/430381687 >> 172.21.15.16:0/430381687 conn(0x7fe4ec0fbb20 msgr2=0x7fe4ec0fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:26.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.232+0000 7fe4f40bd640 1 -- 172.21.15.16:0/430381687 shutdown_connections 2024-04-22T16:01:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.232+0000 7fe4f40bd640 1 -- 172.21.15.16:0/430381687 wait complete. 2024-04-22T16:01:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.233+0000 7fe4f40bd640 1 Processor -- start 2024-04-22T16:01:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.233+0000 7fe4f40bd640 1 -- start start 2024-04-22T16:01:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.233+0000 7fe4f40bd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4ec1002a0 0x7fe4ec195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.233+0000 7fe4f40bd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4ec1962a0 0x7fe4ec19a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.233+0000 7fe4f40bd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec1968a0 con 0x7fe4ec1002a0 2024-04-22T16:01:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.233+0000 7fe4f40bd640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec196a10 con 0x7fe4ec1962a0 2024-04-22T16:01:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.233+0000 7fe4f1e59640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4ec1002a0 0x7fe4ec195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.233+0000 7fe4f1e59640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4ec1002a0 0x7fe4ec195d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47978/0 (socket says 172.21.15.16:47978) 2024-04-22T16:01:26.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.233+0000 7fe4f1e59640 1 -- 172.21.15.16:0/470675759 learned_addr learned my addr 172.21.15.16:0/470675759 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:26.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.233+0000 7fe4f1658640 1 --2- 172.21.15.16:0/470675759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4ec1962a0 0x7fe4ec19a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:26.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.233+0000 7fe4f1e59640 1 -- 172.21.15.16:0/470675759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4ec1962a0 msgr2=0x7fe4ec19a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:26.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.233+0000 7fe4f1e59640 1 --2- 172.21.15.16:0/470675759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4ec1962a0 0x7fe4ec19a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:26.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.233+0000 7fe4f1e59640 1 -- 172.21.15.16:0/470675759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e0009560 con 0x7fe4ec1002a0 2024-04-22T16:01:26.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.233+0000 7fe4f1658640 1 --2- 172.21.15.16:0/470675759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4ec1962a0 0x7fe4ec19a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T16:01:26.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.234+0000 7fe4f1e59640 1 --2- 172.21.15.16:0/470675759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4ec1002a0 0x7fe4ec195d60 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe4e800e9a0 tx=0x7fe4e800eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:26.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.234+0000 7fe4deffd640 1 -- 172.21.15.16:0/470675759 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4e800cd50 con 0x7fe4ec1002a0 2024-04-22T16:01:26.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.234+0000 7fe4deffd640 1 -- 172.21.15.16:0/470675759 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe4e800ceb0 con 0x7fe4ec1002a0 2024-04-22T16:01:26.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.234+0000 7fe4f40bd640 1 -- 172.21.15.16:0/470675759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4ec19ad00 con 0x7fe4ec1002a0 2024-04-22T16:01:26.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.234+0000 7fe4deffd640 1 -- 172.21.15.16:0/470675759 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4e8010660 con 0x7fe4ec1002a0 2024-04-22T16:01:26.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.234+0000 7fe4f40bd640 1 -- 172.21.15.16:0/470675759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4ec19b260 con 0x7fe4ec1002a0 2024-04-22T16:01:26.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.235+0000 7fe4f40bd640 1 -- 172.21.15.16:0/470675759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4ec19b540 con 0x7fe4ec1002a0 2024-04-22T16:01:26.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.235+0000 7fe4deffd640 1 -- 172.21.15.16:0/470675759 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe4e80048a0 con 0x7fe4ec1002a0 2024-04-22T16:01:26.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.236+0000 7fe4deffd640 1 --2- 172.21.15.16:0/470675759 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe4d806ec30 0x7fe4d80710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:26.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.236+0000 7fe4deffd640 1 -- 172.21.15.16:0/470675759 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe4e8014070 con 0x7fe4ec1002a0 2024-04-22T16:01:26.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.236+0000 7fe4f1658640 1 --2- 172.21.15.16:0/470675759 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe4d806ec30 0x7fe4d80710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:26.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.236+0000 7fe4f1658640 1 --2- 172.21.15.16:0/470675759 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe4d806ec30 0x7fe4d80710f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0006010 tx=0x7fe4e002cb40 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:26.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.241+0000 7fe4deffd640 1 -- 172.21.15.16:0/470675759 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe4e805bd30 con 0x7fe4ec1002a0 2024-04-22T16:01:26.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.468+0000 7fe4f40bd640 1 -- 172.21.15.16:0/470675759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 36, "format": "json"} v 0) v1 -- 0x7fe4ec1970b0 con 0x7fe4ec1002a0 2024-04-22T16:01:26.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.469+0000 7fe4deffd640 1 -- 172.21.15.16:0/470675759 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 36, "format": "json"}]=0 dumped fsmap epoch 36 v38) v1 ==== 107+0+5208 (secure 0 0 0) 0x7fe4e805b8c0 con 0x7fe4ec1002a0 2024-04-22T16:01:26.472 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:26.472 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34412,"name":"cephfs.smithi016.pgvphu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6829/4228489800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4228489800},{"type":"v1","addr":"172.21.15.16:6829","nonce":4228489800}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":44329,"name":"cephfs.smithi027.iobjds","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6825/3921606817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3921606817},{"type":"v1","addr":"172.21.15.27:6825","nonce":3921606817}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":44333,"name":"cephfs.smithi027.dgqyes","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6827/4073572546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4073572546},{"type":"v1","addr":"172.21.15.27:6827","nonce":4073572546}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:46:27.774571+0000","modified":"2024-04-22T15:59:23.219392+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34408":{"gid":34408,"name":"cephfs.smithi016.fkkejm","rank":0,"incarnation":30,"state":"up:active","state_seq":19,"addr":"172.21.15.16:6827/2871308685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2871308685},{"type":"v1","addr":"172.21.15.16:6827","nonce":2871308685}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34408,"qdb_cluster":[34408]},"id":1}]} 2024-04-22T16:01:26.472 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 36 2024-04-22T16:01:26.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.471+0000 7fe4f40bd640 1 -- 172.21.15.16:0/470675759 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe4d806ec30 msgr2=0x7fe4d80710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:26.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.471+0000 7fe4f40bd640 1 --2- 172.21.15.16:0/470675759 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe4d806ec30 0x7fe4d80710f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0006010 tx=0x7fe4e002cb40 comp rx=0 tx=0).stop 2024-04-22T16:01:26.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.471+0000 7fe4f40bd640 1 -- 172.21.15.16:0/470675759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4ec1002a0 msgr2=0x7fe4ec195d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:26.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.471+0000 7fe4f40bd640 1 --2- 172.21.15.16:0/470675759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4ec1002a0 0x7fe4ec195d60 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe4e800e9a0 tx=0x7fe4e800eda0 comp rx=0 tx=0).stop 2024-04-22T16:01:26.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.471+0000 7fe4f40bd640 1 -- 172.21.15.16:0/470675759 shutdown_connections 2024-04-22T16:01:26.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.471+0000 7fe4f40bd640 1 --2- 172.21.15.16:0/470675759 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fe4d806ec30 0x7fe4d80710f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:26.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.471+0000 7fe4f40bd640 1 --2- 172.21.15.16:0/470675759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4ec1962a0 0x7fe4ec19a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:26.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.471+0000 7fe4f40bd640 1 --2- 172.21.15.16:0/470675759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4ec1002a0 0x7fe4ec195d60 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:26.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.471+0000 7fe4f40bd640 1 -- 172.21.15.16:0/470675759 >> 172.21.15.16:0/470675759 conn(0x7fe4ec0fbb20 msgr2=0x7fe4ec0fd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:26.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.471+0000 7fe4f40bd640 1 -- 172.21.15.16:0/470675759 shutdown_connections 2024-04-22T16:01:26.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.471+0000 7fe4f40bd640 1 -- 172.21.15.16:0/470675759 wait complete. 2024-04-22T16:01:26.485 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 37 2024-04-22T16:01:26.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.585+0000 7f90d1392640 1 -- 172.21.15.16:0/2475066825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90cc0ff390 msgr2=0x7f90cc104d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:26.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.585+0000 7f90d1392640 1 --2- 172.21.15.16:0/2475066825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90cc0ff390 0x7f90cc104d10 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f90b4009880 tx=0x7f90b402bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:26.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.585+0000 7f90d1392640 1 -- 172.21.15.16:0/2475066825 shutdown_connections 2024-04-22T16:01:26.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.585+0000 7f90d1392640 1 --2- 172.21.15.16:0/2475066825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90cc0ff390 0x7f90cc104d10 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:26.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.585+0000 7f90d1392640 1 --2- 172.21.15.16:0/2475066825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90cc0fe9e0 0x7f90cc0fedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:26.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.585+0000 7f90d1392640 1 -- 172.21.15.16:0/2475066825 >> 172.21.15.16:0/2475066825 conn(0x7f90cc0752f0 msgr2=0x7f90cc075700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:26.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.585+0000 7f90d1392640 1 -- 172.21.15.16:0/2475066825 shutdown_connections 2024-04-22T16:01:26.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.585+0000 7f90d1392640 1 -- 172.21.15.16:0/2475066825 wait complete. 2024-04-22T16:01:26.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.586+0000 7f90d1392640 1 Processor -- start 2024-04-22T16:01:26.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.586+0000 7f90d1392640 1 -- start start 2024-04-22T16:01:26.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.586+0000 7f90d1392640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90cc0fe9e0 0x7f90cc195e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:26.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.586+0000 7f90d1392640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90cc196360 0x7f90cc19a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:26.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.586+0000 7f90d1392640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc196960 con 0x7f90cc0fe9e0 2024-04-22T16:01:26.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.586+0000 7f90d1392640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc196ad0 con 0x7f90cc196360 2024-04-22T16:01:26.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.586+0000 7f90caffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90cc0fe9e0 0x7f90cc195e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:26.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.586+0000 7f90caffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90cc0fe9e0 0x7f90cc195e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47994/0 (socket says 172.21.15.16:47994) 2024-04-22T16:01:26.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.586+0000 7f90caffd640 1 -- 172.21.15.16:0/2630089779 learned_addr learned my addr 172.21.15.16:0/2630089779 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:26.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.586+0000 7f90c3fff640 1 --2- 172.21.15.16:0/2630089779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90cc196360 0x7f90cc19a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:26.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.586+0000 7f90caffd640 1 -- 172.21.15.16:0/2630089779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90cc196360 msgr2=0x7f90cc19a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:26.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.586+0000 7f90caffd640 1 --2- 172.21.15.16:0/2630089779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90cc196360 0x7f90cc19a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:26.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.586+0000 7f90caffd640 1 -- 172.21.15.16:0/2630089779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b4009560 con 0x7f90cc0fe9e0 2024-04-22T16:01:26.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.587+0000 7f90caffd640 1 --2- 172.21.15.16:0/2630089779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90cc0fe9e0 0x7f90cc195e20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f90bc00d720 tx=0x7f90bc00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:26.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.587+0000 7f90c8ff9640 1 -- 172.21.15.16:0/2630089779 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90bc0051f0 con 0x7f90cc0fe9e0 2024-04-22T16:01:26.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.587+0000 7f90c8ff9640 1 -- 172.21.15.16:0/2630089779 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90bc009ce0 con 0x7f90cc0fe9e0 2024-04-22T16:01:26.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.587+0000 7f90c8ff9640 1 -- 172.21.15.16:0/2630089779 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90bc010460 con 0x7f90cc0fe9e0 2024-04-22T16:01:26.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.587+0000 7f90d1392640 1 -- 172.21.15.16:0/2630089779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90cc19adc0 con 0x7f90cc0fe9e0 2024-04-22T16:01:26.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.587+0000 7f90d1392640 1 -- 172.21.15.16:0/2630089779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90cc19b320 con 0x7f90cc0fe9e0 2024-04-22T16:01:26.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.588+0000 7f90d1392640 1 -- 172.21.15.16:0/2630089779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90cc19b5f0 con 0x7f90cc0fe9e0 2024-04-22T16:01:26.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.588+0000 7f90c8ff9640 1 -- 172.21.15.16:0/2630089779 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f90bc005350 con 0x7f90cc0fe9e0 2024-04-22T16:01:26.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.589+0000 7f90c8ff9640 1 --2- 172.21.15.16:0/2630089779 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f90b806ec30 0x7f90b80710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:26.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.589+0000 7f90c8ff9640 1 -- 172.21.15.16:0/2630089779 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f90bc08ea00 con 0x7f90cc0fe9e0 2024-04-22T16:01:26.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.589+0000 7f90c3fff640 1 --2- 172.21.15.16:0/2630089779 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f90b806ec30 0x7f90b80710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:26.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.589+0000 7f90c3fff640 1 --2- 172.21.15.16:0/2630089779 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f90b806ec30 0x7f90b80710f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f90b4006010 tx=0x7f90b4000bc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:26.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.594+0000 7f90c8ff9640 1 -- 172.21.15.16:0/2630089779 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f90bc057790 con 0x7f90cc0fe9e0 2024-04-22T16:01:26.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.827+0000 7f90d1392640 1 -- 172.21.15.16:0/2630089779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 37, "format": "json"} v 0) v1 -- 0x7f90cc197170 con 0x7f90cc0fe9e0 2024-04-22T16:01:26.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.828+0000 7f90c8ff9640 1 -- 172.21.15.16:0/2630089779 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 37, "format": "json"}]=0 dumped fsmap epoch 37 v38) v1 ==== 107+0+5215 (secure 0 0 0) 0x7f90bc017090 con 0x7f90cc0fe9e0 2024-04-22T16:01:26.831 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:26.831 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44329,"name":"cephfs.smithi027.iobjds","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6825/3921606817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3921606817},{"type":"v1","addr":"172.21.15.27:6825","nonce":3921606817}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":44333,"name":"cephfs.smithi027.dgqyes","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6827/4073572546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4073572546},{"type":"v1","addr":"172.21.15.27:6827","nonce":4073572546}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T15:46:27.774571+0000","modified":"2024-04-22T15:59:24.220302+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34408":{"gid":34408,"name":"cephfs.smithi016.fkkejm","rank":0,"incarnation":30,"state":"up:active","state_seq":19,"addr":"172.21.15.16:6827/2871308685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2871308685},{"type":"v1","addr":"172.21.15.16:6827","nonce":2871308685}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34412":{"gid":34412,"name":"cephfs.smithi016.pgvphu","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6829/4228489800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4228489800},{"type":"v1","addr":"172.21.15.16:6829","nonce":4228489800}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34408,"qdb_cluster":[34408]},"id":1}]} 2024-04-22T16:01:26.831 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 37 2024-04-22T16:01:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.830+0000 7f90d1392640 1 -- 172.21.15.16:0/2630089779 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f90b806ec30 msgr2=0x7f90b80710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.830+0000 7f90d1392640 1 --2- 172.21.15.16:0/2630089779 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f90b806ec30 0x7f90b80710f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f90b4006010 tx=0x7f90b4000bc0 comp rx=0 tx=0).stop 2024-04-22T16:01:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.830+0000 7f90d1392640 1 -- 172.21.15.16:0/2630089779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90cc0fe9e0 msgr2=0x7f90cc195e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.830+0000 7f90d1392640 1 --2- 172.21.15.16:0/2630089779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90cc0fe9e0 0x7f90cc195e20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f90bc00d720 tx=0x7f90bc00dbe0 comp rx=0 tx=0).stop 2024-04-22T16:01:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.830+0000 7f90d1392640 1 -- 172.21.15.16:0/2630089779 shutdown_connections 2024-04-22T16:01:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.830+0000 7f90d1392640 1 --2- 172.21.15.16:0/2630089779 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f90b806ec30 0x7f90b80710f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.830+0000 7f90d1392640 1 --2- 172.21.15.16:0/2630089779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90cc196360 0x7f90cc19a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.830+0000 7f90d1392640 1 --2- 172.21.15.16:0/2630089779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90cc0fe9e0 0x7f90cc195e20 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.830+0000 7f90d1392640 1 -- 172.21.15.16:0/2630089779 >> 172.21.15.16:0/2630089779 conn(0x7f90cc0752f0 msgr2=0x7f90cc0fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.830+0000 7f90d1392640 1 -- 172.21.15.16:0/2630089779 shutdown_connections 2024-04-22T16:01:26.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:26.830+0000 7f90d1392640 1 -- 172.21.15.16:0/2630089779 wait complete. 2024-04-22T16:01:26.844 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-22T16:01:26.854 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-22T16:01:26.854 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-22T16:01:26.855 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T16:01:26.871 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-22T16:01:26.872 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T16:01:26.928 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T16:01:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.066+0000 7fdef2bb8640 1 -- 172.21.15.16:0/3494941551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec0ffb20 msgr2=0x7fdeec103b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.066+0000 7fdef2bb8640 1 --2- 172.21.15.16:0/3494941551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec0ffb20 0x7fdeec103b60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fded8009880 tx=0x7fded802bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.066+0000 7fdef2bb8640 1 -- 172.21.15.16:0/3494941551 shutdown_connections 2024-04-22T16:01:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.066+0000 7fdef2bb8640 1 --2- 172.21.15.16:0/3494941551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec0ffb20 0x7fdeec103b60 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.066+0000 7fdef2bb8640 1 --2- 172.21.15.16:0/3494941551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdeec0ff170 0x7fdeec0ff550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.066+0000 7fdef2bb8640 1 -- 172.21.15.16:0/3494941551 >> 172.21.15.16:0/3494941551 conn(0x7fdeec0fa9f0 msgr2=0x7fdeec0fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.066+0000 7fdef2bb8640 1 -- 172.21.15.16:0/3494941551 shutdown_connections 2024-04-22T16:01:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.066+0000 7fdef2bb8640 1 -- 172.21.15.16:0/3494941551 wait complete. 2024-04-22T16:01:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.066+0000 7fdef2bb8640 1 Processor -- start 2024-04-22T16:01:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.066+0000 7fdef2bb8640 1 -- start start 2024-04-22T16:01:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.067+0000 7fdef2bb8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec0ff170 0x7fdeec197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.067+0000 7fdef2bb8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdeec198470 0x7fdeec19c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.067+0000 7fdef2bb8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec198a70 con 0x7fdeec0ff170 2024-04-22T16:01:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.067+0000 7fdef2bb8640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec198be0 con 0x7fdeec198470 2024-04-22T16:01:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.067+0000 7fdef0954640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec0ff170 0x7fdeec197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.067+0000 7fdef0954640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec0ff170 0x7fdeec197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48010/0 (socket says 172.21.15.16:48010) 2024-04-22T16:01:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.067+0000 7fdef0954640 1 -- 172.21.15.16:0/777570163 learned_addr learned my addr 172.21.15.16:0/777570163 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.067+0000 7fdeebfff640 1 --2- 172.21.15.16:0/777570163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdeec198470 0x7fdeec19c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.067+0000 7fdef0954640 1 -- 172.21.15.16:0/777570163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdeec198470 msgr2=0x7fdeec19c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.067+0000 7fdef0954640 1 --2- 172.21.15.16:0/777570163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdeec198470 0x7fdeec19c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.067+0000 7fdef0954640 1 -- 172.21.15.16:0/777570163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded8009560 con 0x7fdeec0ff170 2024-04-22T16:01:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.067+0000 7fdef0954640 1 --2- 172.21.15.16:0/777570163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec0ff170 0x7fdeec197f30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fdee000d720 tx=0x7fdee000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.067+0000 7fdee9ffb640 1 -- 172.21.15.16:0/777570163 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdee00051f0 con 0x7fdeec0ff170 2024-04-22T16:01:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.067+0000 7fdef2bb8640 1 -- 172.21.15.16:0/777570163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeec19ced0 con 0x7fdeec0ff170 2024-04-22T16:01:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.067+0000 7fdee9ffb640 1 -- 172.21.15.16:0/777570163 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdee0009ce0 con 0x7fdeec0ff170 2024-04-22T16:01:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.067+0000 7fdee9ffb640 1 -- 172.21.15.16:0/777570163 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdee0010460 con 0x7fdeec0ff170 2024-04-22T16:01:27.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.068+0000 7fdef2bb8640 1 -- 172.21.15.16:0/777570163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeec19d430 con 0x7fdeec0ff170 2024-04-22T16:01:27.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.068+0000 7fdef2bb8640 1 -- 172.21.15.16:0/777570163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeec04e040 con 0x7fdeec0ff170 2024-04-22T16:01:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.069+0000 7fdee9ffb640 1 -- 172.21.15.16:0/777570163 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdee0005350 con 0x7fdeec0ff170 2024-04-22T16:01:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.069+0000 7fdee9ffb640 1 --2- 172.21.15.16:0/777570163 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdedc06ec30 0x7fdedc0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.069+0000 7fdee9ffb640 1 -- 172.21.15.16:0/777570163 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fdee008e9f0 con 0x7fdeec0ff170 2024-04-22T16:01:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.069+0000 7fdeebfff640 1 --2- 172.21.15.16:0/777570163 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdedc06ec30 0x7fdedc0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.070+0000 7fdeebfff640 1 --2- 172.21.15.16:0/777570163 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdedc06ec30 0x7fdedc0710f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fded8006010 tx=0x7fded8000bc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:27.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.074+0000 7fdee9ffb640 1 -- 172.21.15.16:0/777570163 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fdee0090e60 con 0x7fdeec0ff170 2024-04-22T16:01:27.145 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:26 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/3152120267' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-22T16:01:27.145 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:26 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/1514069772' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-04-22T16:01:27.145 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:26 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/470675759' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-04-22T16:01:27.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:26 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/3152120267' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-22T16:01:27.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:26 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/1514069772' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-04-22T16:01:27.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:26 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/470675759' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-04-22T16:01:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.262+0000 7fdef2bb8640 1 -- 172.21.15.16:0/777570163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdeec0ffb20 con 0x7fdeec0ff170 2024-04-22T16:01:27.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.263+0000 7fdee9ffb640 1 -- 172.21.15.16:0/777570163 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16590 (secure 0 0 0) 0x7fdee00577a0 con 0x7fdeec0ff170 2024-04-22T16:01:27.266 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:27.267 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":84,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","created":"2024-04-22T15:39:20.727181+0000","modified":"2024-04-22T15:59:06.181284+0000","last_up_change":"2024-04-22T15:58:03.188801+0000","last_in_change":"2024-04-22T15:44:52.054297+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T15:43:58.896563+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-22T15:46:26.715511+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-22T15:46:27.206408+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"67bd69ee-bd8c-4670-927a-f5fbcd92c9bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":53,"up_thru":78,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":3158080196},{"type":"v1","addr":"172.21.15.16:6803","nonce":3158080196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":3158080196},{"type":"v1","addr":"172.21.15.16:6805","nonce":3158080196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":3158080196},{"type":"v1","addr":"172.21.15.16:6809","nonce":3158080196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":3158080196},{"type":"v1","addr":"172.21.15.16:6807","nonce":3158080196}]},"public_addr":"172.21.15.16:6803/3158080196","cluster_addr":"172.21.15.16:6805/3158080196","heartbeat_back_addr":"172.21.15.16:6809/3158080196","heartbeat_front_addr":"172.21.15.16:6807/3158080196","state":["exists","up"]},{"osd":1,"uuid":"f51a9e4d-fb0c-4f62-9328-e23eea863213","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":54,"up_from":58,"up_thru":78,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3524132843},{"type":"v1","addr":"172.21.15.16:6811","nonce":3524132843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3524132843},{"type":"v1","addr":"172.21.15.16:6813","nonce":3524132843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3524132843},{"type":"v1","addr":"172.21.15.16:6817","nonce":3524132843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3524132843},{"type":"v1","addr":"172.21.15.16:6815","nonce":3524132843}]},"public_addr":"172.21.15.16:6811/3524132843","cluster_addr":"172.21.15.16:6813/3524132843","heartbeat_back_addr":"172.21.15.16:6817/3524132843","heartbeat_front_addr":"172.21.15.16:6815/3524132843","state":["exists","up"]},{"osd":2,"uuid":"e59fdb03-e04e-4afd-a58e-0b58945d3765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":59,"up_from":63,"up_thru":78,"down_at":60,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2085469945},{"type":"v1","addr":"172.21.15.16:6819","nonce":2085469945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2085469945},{"type":"v1","addr":"172.21.15.16:6821","nonce":2085469945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2085469945},{"type":"v1","addr":"172.21.15.16:6825","nonce":2085469945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2085469945},{"type":"v1","addr":"172.21.15.16:6823","nonce":2085469945}]},"public_addr":"172.21.15.16:6819/2085469945","cluster_addr":"172.21.15.16:6821/2085469945","heartbeat_back_addr":"172.21.15.16:6825/2085469945","heartbeat_front_addr":"172.21.15.16:6823/2085469945","state":["exists","up"]},{"osd":3,"uuid":"fa25b413-d004-4455-8d7b-0ced7f880a7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":64,"up_from":68,"up_thru":78,"down_at":65,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":365345197},{"type":"v1","addr":"172.21.15.27:6801","nonce":365345197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":365345197},{"type":"v1","addr":"172.21.15.27:6803","nonce":365345197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":365345197},{"type":"v1","addr":"172.21.15.27:6807","nonce":365345197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":365345197},{"type":"v1","addr":"172.21.15.27:6805","nonce":365345197}]},"public_addr":"172.21.15.27:6801/365345197","cluster_addr":"172.21.15.27:6803/365345197","heartbeat_back_addr":"172.21.15.27:6807/365345197","heartbeat_front_addr":"172.21.15.27:6805/365345197","state":["exists","up"]},{"osd":4,"uuid":"6b1eda0d-1627-4c49-93b1-1a78896f660c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":69,"up_from":73,"up_thru":78,"down_at":70,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":84770413},{"type":"v1","addr":"172.21.15.27:6809","nonce":84770413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":84770413},{"type":"v1","addr":"172.21.15.27:6811","nonce":84770413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":84770413},{"type":"v1","addr":"172.21.15.27:6815","nonce":84770413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":84770413},{"type":"v1","addr":"172.21.15.27:6813","nonce":84770413}]},"public_addr":"172.21.15.27:6809/84770413","cluster_addr":"172.21.15.27:6811/84770413","heartbeat_back_addr":"172.21.15.27:6815/84770413","heartbeat_front_addr":"172.21.15.27:6813/84770413","state":["exists","up"]},{"osd":5,"uuid":"58cebe34-0821-458f-86b5-7d460829d994","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":74,"up_from":78,"up_thru":78,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3875730087},{"type":"v1","addr":"172.21.15.27:6817","nonce":3875730087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3875730087},{"type":"v1","addr":"172.21.15.27:6819","nonce":3875730087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3875730087},{"type":"v1","addr":"172.21.15.27:6823","nonce":3875730087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3875730087},{"type":"v1","addr":"172.21.15.27:6821","nonce":3875730087}]},"public_addr":"172.21.15.27:6817/3875730087","cluster_addr":"172.21.15.27:6819/3875730087","heartbeat_back_addr":"172.21.15.27:6823/3875730087","heartbeat_front_addr":"172.21.15.27:6821/3875730087","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:42:58.192450+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:43:27.129522+0000","dead_epoch":54},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:43:54.750854+0000","dead_epoch":59},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:44:19.111479+0000","dead_epoch":64},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:44:44.487429+0000","dead_epoch":69},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:45:09.605548+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6827/476026355":"2024-04-23T15:59:06.181260+0000","172.21.15.27:6825/4216277609":"2024-04-23T15:58:44.932745+0000","172.21.15.16:6801/3431522404":"2024-04-23T15:40:27.107187+0000","172.21.15.16:6801/1748504461":"2024-04-23T15:39:53.479948+0000","172.21.15.16:6800/3148690064":"2024-04-23T15:53:30.185361+0000","172.21.15.27:0/4251496318":"2024-04-23T15:49:57.821345+0000","172.21.15.16:6800/3972982132":"2024-04-23T15:41:40.376957+0000","172.21.15.16:6800/3431522404":"2024-04-23T15:40:27.107187+0000","172.21.15.16:0/3682380":"2024-04-23T15:48:41.336415+0000","172.21.15.27:6829/2676414545":"2024-04-23T15:49:57.821345+0000","172.21.15.27:0/186298982":"2024-04-23T15:49:57.821345+0000","172.21.15.27:6828/2676414545":"2024-04-23T15:49:57.821345+0000","172.21.15.16:0/955466902":"2024-04-23T15:40:27.107187+0000","172.21.15.27:6824/4216277609":"2024-04-23T15:58:44.932745+0000","172.21.15.16:0/1878962216":"2024-04-23T15:48:41.336415+0000","172.21.15.16:0/2348469475":"2024-04-23T15:39:53.479948+0000","172.21.15.27:0/2039035396":"2024-04-23T15:49:57.821345+0000","172.21.15.16:0/25024044":"2024-04-23T15:48:41.336415+0000","172.21.15.27:6826/476026355":"2024-04-23T15:59:06.181260+0000","172.21.15.16:0/4172787384":"2024-04-23T15:40:27.107187+0000","172.21.15.16:0/3067906798":"2024-04-23T15:53:30.185361+0000","172.21.15.16:6827/2331116023":"2024-04-23T15:46:44.357516+0000","172.21.15.16:0/2002585335":"2024-04-23T15:40:27.107187+0000","172.21.15.16:0/2570865319":"2024-04-23T15:39:53.479948+0000","172.21.15.16:0/690437556":"2024-04-23T15:41:40.376957+0000","172.21.15.16:6801/3972982132":"2024-04-23T15:41:40.376957+0000","172.21.15.16:0/3645737911":"2024-04-23T15:48:41.336415+0000","172.21.15.16:6826/2331116023":"2024-04-23T15:46:44.357516+0000","172.21.15.16:6800/1748504461":"2024-04-23T15:39:53.479948+0000","172.21.15.16:0/4287530858":"2024-04-23T15:53:30.185361+0000","172.21.15.27:0/3964606592":"2024-04-23T15:49:57.821345+0000","172.21.15.16:0/2048283809":"2024-04-23T15:39:53.479948+0000","172.21.15.16:0/4021274304":"2024-04-23T15:41:40.376957+0000","172.21.15.16:6800/1057234399":"2024-04-23T15:48:41.336415+0000","172.21.15.16:6801/1057234399":"2024-04-23T15:48:41.336415+0000","172.21.15.16:6801/3148690064":"2024-04-23T15:53:30.185361+0000","172.21.15.16:0/4252231167":"2024-04-23T15:41:40.376957+0000","172.21.15.16:0/3802040196":"2024-04-23T15:53:30.185361+0000","172.21.15.16:0/2093505259":"2024-04-23T15:53:30.185361+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T16:01:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.266+0000 7fdef2bb8640 1 -- 172.21.15.16:0/777570163 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdedc06ec30 msgr2=0x7fdedc0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.266+0000 7fdef2bb8640 1 --2- 172.21.15.16:0/777570163 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdedc06ec30 0x7fdedc0710f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fded8006010 tx=0x7fded8000bc0 comp rx=0 tx=0).stop 2024-04-22T16:01:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.266+0000 7fdef2bb8640 1 -- 172.21.15.16:0/777570163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec0ff170 msgr2=0x7fdeec197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.266+0000 7fdef2bb8640 1 --2- 172.21.15.16:0/777570163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec0ff170 0x7fdeec197f30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fdee000d720 tx=0x7fdee000dbe0 comp rx=0 tx=0).stop 2024-04-22T16:01:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.266+0000 7fdef2bb8640 1 -- 172.21.15.16:0/777570163 shutdown_connections 2024-04-22T16:01:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.266+0000 7fdef2bb8640 1 --2- 172.21.15.16:0/777570163 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7fdedc06ec30 0x7fdedc0710f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.266+0000 7fdef2bb8640 1 --2- 172.21.15.16:0/777570163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdeec198470 0x7fdeec19c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.266+0000 7fdef2bb8640 1 --2- 172.21.15.16:0/777570163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec0ff170 0x7fdeec197f30 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.266+0000 7fdef2bb8640 1 -- 172.21.15.16:0/777570163 >> 172.21.15.16:0/777570163 conn(0x7fdeec0fa9f0 msgr2=0x7fdeec068cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.266+0000 7fdef2bb8640 1 -- 172.21.15.16:0/777570163 shutdown_connections 2024-04-22T16:01:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.266+0000 7fdef2bb8640 1 -- 172.21.15.16:0/777570163 wait complete. 2024-04-22T16:01:27.280 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T15:43:58.896563+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-22T15:46:26.715511+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-22T15:46:27.206408+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T16:01:27.280 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T16:01:27.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.383+0000 7f0d83f16640 1 -- 172.21.15.16:0/182182101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d7c101ba0 msgr2=0x7f0d7c10a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:27.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.383+0000 7f0d83f16640 1 --2- 172.21.15.16:0/182182101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d7c101ba0 0x7f0d7c10a080 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c0098b0 tx=0x7f0d6c02bcd0 comp rx=0 tx=0).stop 2024-04-22T16:01:27.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.383+0000 7f0d83f16640 1 -- 172.21.15.16:0/182182101 shutdown_connections 2024-04-22T16:01:27.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.383+0000 7f0d83f16640 1 --2- 172.21.15.16:0/182182101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d7c101ba0 0x7f0d7c10a080 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:27.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.383+0000 7f0d83f16640 1 --2- 172.21.15.16:0/182182101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d7c101280 0x7f0d7c101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:27.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.383+0000 7f0d83f16640 1 -- 172.21.15.16:0/182182101 >> 172.21.15.16:0/182182101 conn(0x7f0d7c0fa9e0 msgr2=0x7f0d7c0fce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:27.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.383+0000 7f0d83f16640 1 -- 172.21.15.16:0/182182101 shutdown_connections 2024-04-22T16:01:27.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.383+0000 7f0d83f16640 1 -- 172.21.15.16:0/182182101 wait complete. 2024-04-22T16:01:27.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.384+0000 7f0d83f16640 1 Processor -- start 2024-04-22T16:01:27.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.384+0000 7f0d83f16640 1 -- start start 2024-04-22T16:01:27.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.384+0000 7f0d83f16640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d7c101280 0x7f0d7c195d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:27.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.384+0000 7f0d83f16640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d7c101ba0 0x7f0d7c1962b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:27.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.384+0000 7f0d83f16640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c196940 con 0x7f0d7c101280 2024-04-22T16:01:27.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.384+0000 7f0d83f16640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c19a710 con 0x7f0d7c101ba0 2024-04-22T16:01:27.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.384+0000 7f0d81cb2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d7c101280 0x7f0d7c195d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:27.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.384+0000 7f0d81cb2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d7c101280 0x7f0d7c195d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48032/0 (socket says 172.21.15.16:48032) 2024-04-22T16:01:27.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.384+0000 7f0d81cb2640 1 -- 172.21.15.16:0/791707997 learned_addr learned my addr 172.21.15.16:0/791707997 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:27.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.385+0000 7f0d814b1640 1 --2- 172.21.15.16:0/791707997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d7c101ba0 0x7f0d7c1962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:27.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.385+0000 7f0d81cb2640 1 -- 172.21.15.16:0/791707997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d7c101ba0 msgr2=0x7f0d7c1962b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:27.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.385+0000 7f0d81cb2640 1 --2- 172.21.15.16:0/791707997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d7c101ba0 0x7f0d7c1962b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:27.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.385+0000 7f0d81cb2640 1 -- 172.21.15.16:0/791707997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6c009560 con 0x7f0d7c101280 2024-04-22T16:01:27.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.385+0000 7f0d81cb2640 1 --2- 172.21.15.16:0/791707997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d7c101280 0x7f0d7c195d70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0d7800d8d0 tx=0x7f0d7800dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:27.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.385+0000 7f0d72ffd640 1 -- 172.21.15.16:0/791707997 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d7800bea0 con 0x7f0d7c101280 2024-04-22T16:01:27.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.385+0000 7f0d83f16640 1 -- 172.21.15.16:0/791707997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d7c19a9f0 con 0x7f0d7c101280 2024-04-22T16:01:27.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.385+0000 7f0d72ffd640 1 -- 172.21.15.16:0/791707997 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d78009c00 con 0x7f0d7c101280 2024-04-22T16:01:27.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.385+0000 7f0d72ffd640 1 -- 172.21.15.16:0/791707997 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d7800f710 con 0x7f0d7c101280 2024-04-22T16:01:27.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.385+0000 7f0d83f16640 1 -- 172.21.15.16:0/791707997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d7c19af50 con 0x7f0d7c101280 2024-04-22T16:01:27.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.386+0000 7f0d83f16640 1 -- 172.21.15.16:0/791707997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d7c04e040 con 0x7f0d7c101280 2024-04-22T16:01:27.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.386+0000 7f0d72ffd640 1 -- 172.21.15.16:0/791707997 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0d7800f990 con 0x7f0d7c101280 2024-04-22T16:01:27.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.387+0000 7f0d72ffd640 1 --2- 172.21.15.16:0/791707997 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0d6806ec30 0x7f0d680710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:27.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.387+0000 7f0d72ffd640 1 -- 172.21.15.16:0/791707997 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0d7808ebb0 con 0x7f0d7c101280 2024-04-22T16:01:27.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.387+0000 7f0d814b1640 1 --2- 172.21.15.16:0/791707997 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0d6806ec30 0x7f0d680710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:27.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.387+0000 7f0d814b1640 1 --2- 172.21.15.16:0/791707997 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0d6806ec30 0x7f0d680710f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c036040 tx=0x7f0d6c005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:27.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.392+0000 7f0d72ffd640 1 -- 172.21.15.16:0/791707997 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0d78095080 con 0x7f0d7c101280 2024-04-22T16:01:27.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.581+0000 7f0d83f16640 1 -- 172.21.15.16:0/791707997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0d7c10d2c0 con 0x7f0d7c101280 2024-04-22T16:01:27.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.581+0000 7f0d72ffd640 1 -- 172.21.15.16:0/791707997 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0d78057960 con 0x7f0d7c101280 2024-04-22T16:01:27.584 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-22T16:01:27.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.583+0000 7f0d83f16640 1 -- 172.21.15.16:0/791707997 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0d6806ec30 msgr2=0x7f0d680710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:27.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.583+0000 7f0d83f16640 1 --2- 172.21.15.16:0/791707997 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0d6806ec30 0x7f0d680710f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c036040 tx=0x7f0d6c005fd0 comp rx=0 tx=0).stop 2024-04-22T16:01:27.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.583+0000 7f0d83f16640 1 -- 172.21.15.16:0/791707997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d7c101280 msgr2=0x7f0d7c195d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:27.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.583+0000 7f0d83f16640 1 --2- 172.21.15.16:0/791707997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d7c101280 0x7f0d7c195d70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0d7800d8d0 tx=0x7f0d7800dd90 comp rx=0 tx=0).stop 2024-04-22T16:01:27.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.583+0000 7f0d83f16640 1 -- 172.21.15.16:0/791707997 shutdown_connections 2024-04-22T16:01:27.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.583+0000 7f0d83f16640 1 --2- 172.21.15.16:0/791707997 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0d6806ec30 0x7f0d680710f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:27.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.583+0000 7f0d83f16640 1 --2- 172.21.15.16:0/791707997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d7c101ba0 0x7f0d7c1962b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:27.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.583+0000 7f0d83f16640 1 --2- 172.21.15.16:0/791707997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d7c101280 0x7f0d7c195d70 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:27.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.583+0000 7f0d83f16640 1 -- 172.21.15.16:0/791707997 >> 172.21.15.16:0/791707997 conn(0x7f0d7c0fa9e0 msgr2=0x7f0d7c1048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:27.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.584+0000 7f0d83f16640 1 -- 172.21.15.16:0/791707997 shutdown_connections 2024-04-22T16:01:27.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.584+0000 7f0d83f16640 1 -- 172.21.15.16:0/791707997 wait complete. 2024-04-22T16:01:27.597 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-22T16:01:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.699+0000 7f642aa28640 1 -- 172.21.15.16:0/617496133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64240ffb40 msgr2=0x7f6424103b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.699+0000 7f642aa28640 1 --2- 172.21.15.16:0/617496133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64240ffb40 0x7f6424103b80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6418009880 tx=0x7f641802bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.699+0000 7f642aa28640 1 -- 172.21.15.16:0/617496133 shutdown_connections 2024-04-22T16:01:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.699+0000 7f642aa28640 1 --2- 172.21.15.16:0/617496133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64240ffb40 0x7f6424103b80 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.699+0000 7f642aa28640 1 --2- 172.21.15.16:0/617496133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64240ff190 0x7f64240ff570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.699+0000 7f642aa28640 1 -- 172.21.15.16:0/617496133 >> 172.21.15.16:0/617496133 conn(0x7f64240faa30 msgr2=0x7f64240fce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.699+0000 7f642aa28640 1 -- 172.21.15.16:0/617496133 shutdown_connections 2024-04-22T16:01:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.699+0000 7f642aa28640 1 -- 172.21.15.16:0/617496133 wait complete. 2024-04-22T16:01:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.700+0000 7f642aa28640 1 Processor -- start 2024-04-22T16:01:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.700+0000 7f642aa28640 1 -- start start 2024-04-22T16:01:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.700+0000 7f642aa28640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64240ff190 0x7f6424196c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.700+0000 7f642aa28640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64241971b0 0x7f6424197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.700+0000 7f642aa28640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642419fce0 con 0x7f64241971b0 2024-04-22T16:01:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.700+0000 7f642aa28640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642419fe20 con 0x7f64240ff190 2024-04-22T16:01:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.700+0000 7f64237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64241971b0 0x7f6424197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:27.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.700+0000 7f64237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64241971b0 0x7f6424197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48050/0 (socket says 172.21.15.16:48050) 2024-04-22T16:01:27.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.700+0000 7f64237fe640 1 -- 172.21.15.16:0/3761587074 learned_addr learned my addr 172.21.15.16:0/3761587074 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:27.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.700+0000 7f6423fff640 1 --2- 172.21.15.16:0/3761587074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64240ff190 0x7f6424196c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:27.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.700+0000 7f64237fe640 1 -- 172.21.15.16:0/3761587074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64240ff190 msgr2=0x7f6424196c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:27.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.700+0000 7f64237fe640 1 --2- 172.21.15.16:0/3761587074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64240ff190 0x7f6424196c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:27.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.700+0000 7f64237fe640 1 -- 172.21.15.16:0/3761587074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6418009560 con 0x7f64241971b0 2024-04-22T16:01:27.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.700+0000 7f6423fff640 1 --2- 172.21.15.16:0/3761587074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64240ff190 0x7f6424196c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T16:01:27.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.701+0000 7f64237fe640 1 --2- 172.21.15.16:0/3761587074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64241971b0 0x7f6424197610 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6418036040 tx=0x7f6418004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:27.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.701+0000 7f64217fa640 1 -- 172.21.15.16:0/3761587074 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64180037c0 con 0x7f64241971b0 2024-04-22T16:01:27.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.701+0000 7f642aa28640 1 -- 172.21.15.16:0/3761587074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64241a0020 con 0x7f64241971b0 2024-04-22T16:01:27.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.701+0000 7f64217fa640 1 -- 172.21.15.16:0/3761587074 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6418003de0 con 0x7f64241971b0 2024-04-22T16:01:27.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.701+0000 7f64217fa640 1 -- 172.21.15.16:0/3761587074 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6418034440 con 0x7f64241971b0 2024-04-22T16:01:27.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.701+0000 7f642aa28640 1 -- 172.21.15.16:0/3761587074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64241a04d0 con 0x7f64241971b0 2024-04-22T16:01:27.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.702+0000 7f642aa28640 1 -- 172.21.15.16:0/3761587074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f642404e040 con 0x7f64241971b0 2024-04-22T16:01:27.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.702+0000 7f64217fa640 1 -- 172.21.15.16:0/3761587074 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6418004fb0 con 0x7f64241971b0 2024-04-22T16:01:27.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.703+0000 7f64217fa640 1 --2- 172.21.15.16:0/3761587074 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f640c06ec30 0x7f640c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:27.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.703+0000 7f6423fff640 1 --2- 172.21.15.16:0/3761587074 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f640c06ec30 0x7f640c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:27.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.703+0000 7f64217fa640 1 -- 172.21.15.16:0/3761587074 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f64180b13d0 con 0x7f64241971b0 2024-04-22T16:01:27.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.703+0000 7f6423fff640 1 --2- 172.21.15.16:0/3761587074 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f640c06ec30 0x7f640c0710f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6414005fd0 tx=0x7f6414005ea0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:27.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.708+0000 7f64217fa640 1 -- 172.21.15.16:0/3761587074 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f641807a0e0 con 0x7f64241971b0 2024-04-22T16:01:27.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.896+0000 7f642aa28640 1 -- 172.21.15.16:0/3761587074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f6424107490 con 0x7f64241971b0 2024-04-22T16:01:27.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.896+0000 7f64217fa640 1 -- 172.21.15.16:0/3761587074 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f641807d7a0 con 0x7f64241971b0 2024-04-22T16:01:27.899 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-22T16:01:27.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.898+0000 7f642aa28640 1 -- 172.21.15.16:0/3761587074 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f640c06ec30 msgr2=0x7f640c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:27.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.898+0000 7f642aa28640 1 --2- 172.21.15.16:0/3761587074 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f640c06ec30 0x7f640c0710f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6414005fd0 tx=0x7f6414005ea0 comp rx=0 tx=0).stop 2024-04-22T16:01:27.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.898+0000 7f642aa28640 1 -- 172.21.15.16:0/3761587074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64241971b0 msgr2=0x7f6424197610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:27.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.898+0000 7f642aa28640 1 --2- 172.21.15.16:0/3761587074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64241971b0 0x7f6424197610 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6418036040 tx=0x7f6418004270 comp rx=0 tx=0).stop 2024-04-22T16:01:27.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.898+0000 7f642aa28640 1 -- 172.21.15.16:0/3761587074 shutdown_connections 2024-04-22T16:01:27.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.898+0000 7f642aa28640 1 --2- 172.21.15.16:0/3761587074 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f640c06ec30 0x7f640c0710f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:27.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.898+0000 7f642aa28640 1 --2- 172.21.15.16:0/3761587074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64241971b0 0x7f6424197610 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:27.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.898+0000 7f642aa28640 1 --2- 172.21.15.16:0/3761587074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64240ff190 0x7f6424196c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:27.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.898+0000 7f642aa28640 1 -- 172.21.15.16:0/3761587074 >> 172.21.15.16:0/3761587074 conn(0x7f64240faa30 msgr2=0x7f64240fc100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:27.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.898+0000 7f642aa28640 1 -- 172.21.15.16:0/3761587074 shutdown_connections 2024-04-22T16:01:27.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:27.898+0000 7f642aa28640 1 -- 172.21.15.16:0/3761587074 wait complete. 2024-04-22T16:01:27.911 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:27 smithi016 ceph-mon[110740]: pgmap v276: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:27.911 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:27 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/2630089779' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-04-22T16:01:27.911 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:27 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/777570163' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T16:01:27.911 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:27 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/791707997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T16:01:27.912 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-22T16:01:28.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.013+0000 7f246ad1d640 1 -- 172.21.15.16:0/2538093088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2464101210 msgr2=0x7f24641015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:28.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.013+0000 7f246ad1d640 1 --2- 172.21.15.16:0/2538093088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2464101210 0x7f24641015f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2454009880 tx=0x7f245402bd70 comp rx=0 tx=0).stop 2024-04-22T16:01:28.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.013+0000 7f246ad1d640 1 -- 172.21.15.16:0/2538093088 shutdown_connections 2024-04-22T16:01:28.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.013+0000 7f246ad1d640 1 --2- 172.21.15.16:0/2538093088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2464101b30 0x7f246410a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.013+0000 7f246ad1d640 1 --2- 172.21.15.16:0/2538093088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2464101210 0x7f24641015f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.013+0000 7f246ad1d640 1 -- 172.21.15.16:0/2538093088 >> 172.21.15.16:0/2538093088 conn(0x7f24640fa9f0 msgr2=0x7f24640fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.013+0000 7f246ad1d640 1 -- 172.21.15.16:0/2538093088 shutdown_connections 2024-04-22T16:01:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.013+0000 7f246ad1d640 1 -- 172.21.15.16:0/2538093088 wait complete. 2024-04-22T16:01:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.014+0000 7f246ad1d640 1 Processor -- start 2024-04-22T16:01:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.014+0000 7f246ad1d640 1 -- start start 2024-04-22T16:01:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.014+0000 7f246ad1d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2464101210 0x7f24640fee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.014+0000 7f246ad1d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2464101b30 0x7f24640ff3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.014+0000 7f246ad1d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24640ff8e0 con 0x7f2464101210 2024-04-22T16:01:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.014+0000 7f246ad1d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24640ffa50 con 0x7f2464101b30 2024-04-22T16:01:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.014+0000 7f2468ab9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2464101210 0x7f24640fee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.014+0000 7f2468ab9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2464101210 0x7f24640fee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48064/0 (socket says 172.21.15.16:48064) 2024-04-22T16:01:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.014+0000 7f2468ab9640 1 -- 172.21.15.16:0/708541576 learned_addr learned my addr 172.21.15.16:0/708541576 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.015+0000 7f2463fff640 1 --2- 172.21.15.16:0/708541576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2464101b30 0x7f24640ff3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.015+0000 7f2468ab9640 1 -- 172.21.15.16:0/708541576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2464101b30 msgr2=0x7f24640ff3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.015+0000 7f2468ab9640 1 --2- 172.21.15.16:0/708541576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2464101b30 0x7f24640ff3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.015+0000 7f2468ab9640 1 -- 172.21.15.16:0/708541576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2454009560 con 0x7f2464101210 2024-04-22T16:01:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.015+0000 7f2463fff640 1 --2- 172.21.15.16:0/708541576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2464101b30 0x7f24640ff3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T16:01:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.015+0000 7f2468ab9640 1 --2- 172.21.15.16:0/708541576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2464101210 0x7f24640fee60 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2454009230 tx=0x7f2454004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.015+0000 7f2461ffb640 1 -- 172.21.15.16:0/708541576 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24540037c0 con 0x7f2464101210 2024-04-22T16:01:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.015+0000 7f2461ffb640 1 -- 172.21.15.16:0/708541576 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2454004aa0 con 0x7f2464101210 2024-04-22T16:01:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.015+0000 7f246ad1d640 1 -- 172.21.15.16:0/708541576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24640ffcd0 con 0x7f2464101210 2024-04-22T16:01:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.015+0000 7f2461ffb640 1 -- 172.21.15.16:0/708541576 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f245403e450 con 0x7f2464101210 2024-04-22T16:01:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.015+0000 7f246ad1d640 1 -- 172.21.15.16:0/708541576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24641a2200 con 0x7f2464101210 2024-04-22T16:01:28.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.016+0000 7f246ad1d640 1 -- 172.21.15.16:0/708541576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f246404e040 con 0x7f2464101210 2024-04-22T16:01:28.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.017+0000 7f2461ffb640 1 -- 172.21.15.16:0/708541576 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2454005060 con 0x7f2464101210 2024-04-22T16:01:28.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.017+0000 7f2461ffb640 1 --2- 172.21.15.16:0/708541576 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f244c06ec30 0x7f244c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:28.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.017+0000 7f2461ffb640 1 -- 172.21.15.16:0/708541576 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f24540b1b20 con 0x7f2464101210 2024-04-22T16:01:28.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.017+0000 7f2463fff640 1 --2- 172.21.15.16:0/708541576 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f244c06ec30 0x7f244c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.018+0000 7f2463fff640 1 --2- 172.21.15.16:0/708541576 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f244c06ec30 0x7f244c0710f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2464100650 tx=0x7f2458005e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:28.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.022+0000 7f2461ffb640 1 -- 172.21.15.16:0/708541576 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f245407a8d0 con 0x7f2464101210 2024-04-22T16:01:28.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:27 smithi027 ceph-mon[83054]: pgmap v276: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:28.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:27 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/2630089779' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-04-22T16:01:28.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:27 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/777570163' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T16:01:28.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:27 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/791707997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T16:01:28.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.235+0000 7f246ad1d640 1 -- 172.21.15.16:0/708541576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f24640ffcd0 con 0x7f2464101210 2024-04-22T16:01:28.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.236+0000 7f2461ffb640 1 -- 172.21.15.16:0/708541576 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f24640ffcd0 con 0x7f2464101210 2024-04-22T16:01:28.239 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-22T16:01:28.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.238+0000 7f246ad1d640 1 -- 172.21.15.16:0/708541576 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f244c06ec30 msgr2=0x7f244c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:28.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.238+0000 7f246ad1d640 1 --2- 172.21.15.16:0/708541576 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f244c06ec30 0x7f244c0710f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2464100650 tx=0x7f2458005e30 comp rx=0 tx=0).stop 2024-04-22T16:01:28.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.238+0000 7f246ad1d640 1 -- 172.21.15.16:0/708541576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2464101210 msgr2=0x7f24640fee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:28.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.238+0000 7f246ad1d640 1 --2- 172.21.15.16:0/708541576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2464101210 0x7f24640fee60 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2454009230 tx=0x7f2454004270 comp rx=0 tx=0).stop 2024-04-22T16:01:28.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.238+0000 7f246ad1d640 1 -- 172.21.15.16:0/708541576 shutdown_connections 2024-04-22T16:01:28.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.238+0000 7f246ad1d640 1 --2- 172.21.15.16:0/708541576 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f244c06ec30 0x7f244c0710f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:28.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.238+0000 7f246ad1d640 1 --2- 172.21.15.16:0/708541576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2464101b30 0x7f24640ff3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:28.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.238+0000 7f246ad1d640 1 --2- 172.21.15.16:0/708541576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2464101210 0x7f24640fee60 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:28.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.238+0000 7f246ad1d640 1 -- 172.21.15.16:0/708541576 >> 172.21.15.16:0/708541576 conn(0x7f24640fa9f0 msgr2=0x7f2464104850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:28.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.238+0000 7f246ad1d640 1 -- 172.21.15.16:0/708541576 shutdown_connections 2024-04-22T16:01:28.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.238+0000 7f246ad1d640 1 -- 172.21.15.16:0/708541576 wait complete. 2024-04-22T16:01:28.252 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T16:01:28.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.351+0000 7f2e30792640 1 -- 172.21.15.16:0/161800616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e280ff2c0 msgr2=0x7f2e280ff6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:28.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.351+0000 7f2e30792640 1 --2- 172.21.15.16:0/161800616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e280ff2c0 0x7f2e280ff6a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2e24009c20 tx=0x7f2e2402c1b0 comp rx=0 tx=0).stop 2024-04-22T16:01:28.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.351+0000 7f2e30792640 1 -- 172.21.15.16:0/161800616 shutdown_connections 2024-04-22T16:01:28.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.351+0000 7f2e30792640 1 --2- 172.21.15.16:0/161800616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e280ffc70 0x7f2e28103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:28.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.351+0000 7f2e30792640 1 --2- 172.21.15.16:0/161800616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e280ff2c0 0x7f2e280ff6a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.351+0000 7f2e30792640 1 -- 172.21.15.16:0/161800616 >> 172.21.15.16:0/161800616 conn(0x7f2e28075bb0 msgr2=0x7f2e28075fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.352+0000 7f2e30792640 1 -- 172.21.15.16:0/161800616 shutdown_connections 2024-04-22T16:01:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.352+0000 7f2e30792640 1 -- 172.21.15.16:0/161800616 wait complete. 2024-04-22T16:01:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.352+0000 7f2e30792640 1 Processor -- start 2024-04-22T16:01:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.352+0000 7f2e30792640 1 -- start start 2024-04-22T16:01:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.352+0000 7f2e30792640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e280ff2c0 0x7f2e28196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.352+0000 7f2e30792640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e280ffc70 0x7f2e28197140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.352+0000 7f2e30792640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e28197790 con 0x7f2e280ffc70 2024-04-22T16:01:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.352+0000 7f2e30792640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e281978d0 con 0x7f2e280ff2c0 2024-04-22T16:01:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.352+0000 7f2e2dd2d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e280ffc70 0x7f2e28197140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.352+0000 7f2e2dd2d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e280ffc70 0x7f2e28197140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48074/0 (socket says 172.21.15.16:48074) 2024-04-22T16:01:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.352+0000 7f2e2dd2d640 1 -- 172.21.15.16:0/3782074465 learned_addr learned my addr 172.21.15.16:0/3782074465 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.353+0000 7f2e2dd2d640 1 -- 172.21.15.16:0/3782074465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e280ff2c0 msgr2=0x7f2e28196c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.353+0000 7f2e2e52e640 1 --2- 172.21.15.16:0/3782074465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e280ff2c0 0x7f2e28196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.353+0000 7f2e2dd2d640 1 --2- 172.21.15.16:0/3782074465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e280ff2c0 0x7f2e28196c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.353+0000 7f2e2dd2d640 1 -- 172.21.15.16:0/3782074465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e24009900 con 0x7f2e280ffc70 2024-04-22T16:01:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.353+0000 7f2e2e52e640 1 --2- 172.21.15.16:0/3782074465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e280ff2c0 0x7f2e28196c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T16:01:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.353+0000 7f2e2dd2d640 1 --2- 172.21.15.16:0/3782074465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e280ffc70 0x7f2e28197140 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c00c8a0 tx=0x7f2e1c00cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.353+0000 7f2e1b7fe640 1 -- 172.21.15.16:0/3782074465 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e1c007b80 con 0x7f2e280ffc70 2024-04-22T16:01:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.353+0000 7f2e1b7fe640 1 -- 172.21.15.16:0/3782074465 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e1c007ce0 con 0x7f2e280ffc70 2024-04-22T16:01:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.353+0000 7f2e30792640 1 -- 172.21.15.16:0/3782074465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e2819fc70 con 0x7f2e280ffc70 2024-04-22T16:01:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.353+0000 7f2e1b7fe640 1 -- 172.21.15.16:0/3782074465 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e1c018810 con 0x7f2e280ffc70 2024-04-22T16:01:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.353+0000 7f2e30792640 1 -- 172.21.15.16:0/3782074465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e281a0120 con 0x7f2e280ffc70 2024-04-22T16:01:28.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.354+0000 7f2e30792640 1 -- 172.21.15.16:0/3782074465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e2804e040 con 0x7f2e280ffc70 2024-04-22T16:01:28.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.355+0000 7f2e1b7fe640 1 -- 172.21.15.16:0/3782074465 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2e1c018970 con 0x7f2e280ffc70 2024-04-22T16:01:28.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.355+0000 7f2e1b7fe640 1 --2- 172.21.15.16:0/3782074465 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f2e1406ec30 0x7f2e140710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:28.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.355+0000 7f2e2e52e640 1 --2- 172.21.15.16:0/3782074465 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f2e1406ec30 0x7f2e140710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:28.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.355+0000 7f2e1b7fe640 1 -- 172.21.15.16:0/3782074465 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2e1c08efe0 con 0x7f2e280ffc70 2024-04-22T16:01:28.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.356+0000 7f2e2e52e640 1 --2- 172.21.15.16:0/3782074465 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f2e1406ec30 0x7f2e140710f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2e24009430 tx=0x7f2e24009360 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:28.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.360+0000 7f2e1b7fe640 1 -- 172.21.15.16:0/3782074465 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2e1c057e10 con 0x7f2e280ffc70 2024-04-22T16:01:28.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.561+0000 7f2e30792640 1 -- 172.21.15.16:0/3782074465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2e28107480 con 0x7f2e280ffc70 2024-04-22T16:01:28.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.561+0000 7f2e1b7fe640 1 -- 172.21.15.16:0/3782074465 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 39 entries v84) v1 ==== 81+0+2306 (secure 0 0 0) 0x7f2e1c05b4d0 con 0x7f2e280ffc70 2024-04-22T16:01:28.564 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.27:6827/476026355 2024-04-23T15:59:06.181260+0000 2024-04-22T16:01:28.565 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.27:6825/4216277609 2024-04-23T15:58:44.932745+0000 2024-04-22T16:01:28.565 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/3431522404 2024-04-23T15:40:27.107187+0000 2024-04-22T16:01:28.565 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/1748504461 2024-04-23T15:39:53.479948+0000 2024-04-22T16:01:28.565 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/3148690064 2024-04-23T15:53:30.185361+0000 2024-04-22T16:01:28.565 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.27:0/4251496318 2024-04-23T15:49:57.821345+0000 2024-04-22T16:01:28.565 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/3972982132 2024-04-23T15:41:40.376957+0000 2024-04-22T16:01:28.565 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/3431522404 2024-04-23T15:40:27.107187+0000 2024-04-22T16:01:28.565 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3682380 2024-04-23T15:48:41.336415+0000 2024-04-22T16:01:28.565 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.27:6829/2676414545 2024-04-23T15:49:57.821345+0000 2024-04-22T16:01:28.565 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.27:0/186298982 2024-04-23T15:49:57.821345+0000 2024-04-22T16:01:28.565 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.27:6828/2676414545 2024-04-23T15:49:57.821345+0000 2024-04-22T16:01:28.565 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/955466902 2024-04-23T15:40:27.107187+0000 2024-04-22T16:01:28.565 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.27:6824/4216277609 2024-04-23T15:58:44.932745+0000 2024-04-22T16:01:28.566 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1878962216 2024-04-23T15:48:41.336415+0000 2024-04-22T16:01:28.566 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2348469475 2024-04-23T15:39:53.479948+0000 2024-04-22T16:01:28.566 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.27:0/2039035396 2024-04-23T15:49:57.821345+0000 2024-04-22T16:01:28.566 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/25024044 2024-04-23T15:48:41.336415+0000 2024-04-22T16:01:28.566 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.27:6826/476026355 2024-04-23T15:59:06.181260+0000 2024-04-22T16:01:28.566 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4172787384 2024-04-23T15:40:27.107187+0000 2024-04-22T16:01:28.566 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3067906798 2024-04-23T15:53:30.185361+0000 2024-04-22T16:01:28.566 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6827/2331116023 2024-04-23T15:46:44.357516+0000 2024-04-22T16:01:28.566 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2002585335 2024-04-23T15:40:27.107187+0000 2024-04-22T16:01:28.566 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2570865319 2024-04-23T15:39:53.479948+0000 2024-04-22T16:01:28.566 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/690437556 2024-04-23T15:41:40.376957+0000 2024-04-22T16:01:28.566 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/3972982132 2024-04-23T15:41:40.376957+0000 2024-04-22T16:01:28.566 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3645737911 2024-04-23T15:48:41.336415+0000 2024-04-22T16:01:28.566 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6826/2331116023 2024-04-23T15:46:44.357516+0000 2024-04-22T16:01:28.567 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/1748504461 2024-04-23T15:39:53.479948+0000 2024-04-22T16:01:28.567 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4287530858 2024-04-23T15:53:30.185361+0000 2024-04-22T16:01:28.567 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.27:0/3964606592 2024-04-23T15:49:57.821345+0000 2024-04-22T16:01:28.567 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2048283809 2024-04-23T15:39:53.479948+0000 2024-04-22T16:01:28.567 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4021274304 2024-04-23T15:41:40.376957+0000 2024-04-22T16:01:28.567 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/1057234399 2024-04-23T15:48:41.336415+0000 2024-04-22T16:01:28.567 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/1057234399 2024-04-23T15:48:41.336415+0000 2024-04-22T16:01:28.567 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/3148690064 2024-04-23T15:53:30.185361+0000 2024-04-22T16:01:28.567 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4252231167 2024-04-23T15:41:40.376957+0000 2024-04-22T16:01:28.567 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3802040196 2024-04-23T15:53:30.185361+0000 2024-04-22T16:01:28.567 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2093505259 2024-04-23T15:53:30.185361+0000 2024-04-22T16:01:28.568 INFO:teuthology.orchestra.run.smithi016.stderr:listed 39 entries 2024-04-22T16:01:28.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.564+0000 7f2e30792640 1 -- 172.21.15.16:0/3782074465 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f2e1406ec30 msgr2=0x7f2e140710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:28.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.564+0000 7f2e30792640 1 --2- 172.21.15.16:0/3782074465 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f2e1406ec30 0x7f2e140710f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2e24009430 tx=0x7f2e24009360 comp rx=0 tx=0).stop 2024-04-22T16:01:28.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.564+0000 7f2e30792640 1 -- 172.21.15.16:0/3782074465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e280ffc70 msgr2=0x7f2e28197140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:28.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.564+0000 7f2e30792640 1 --2- 172.21.15.16:0/3782074465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e280ffc70 0x7f2e28197140 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c00c8a0 tx=0x7f2e1c00cd60 comp rx=0 tx=0).stop 2024-04-22T16:01:28.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.564+0000 7f2e30792640 1 -- 172.21.15.16:0/3782074465 shutdown_connections 2024-04-22T16:01:28.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.564+0000 7f2e30792640 1 --2- 172.21.15.16:0/3782074465 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f2e1406ec30 0x7f2e140710f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:28.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.564+0000 7f2e30792640 1 --2- 172.21.15.16:0/3782074465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e280ffc70 0x7f2e28197140 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:28.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.564+0000 7f2e30792640 1 --2- 172.21.15.16:0/3782074465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e280ff2c0 0x7f2e28196c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:28.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.564+0000 7f2e30792640 1 -- 172.21.15.16:0/3782074465 >> 172.21.15.16:0/3782074465 conn(0x7f2e28075bb0 msgr2=0x7f2e280fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:28.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.564+0000 7f2e30792640 1 -- 172.21.15.16:0/3782074465 shutdown_connections 2024-04-22T16:01:28.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.564+0000 7f2e30792640 1 -- 172.21.15.16:0/3782074465 wait complete. 2024-04-22T16:01:28.578 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-22T16:01:28.578 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T16:01:28.594 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T16:01:28.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.733+0000 7f0f38927640 1 -- 172.21.15.16:0/2771631444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f34100270 msgr2=0x7f0f34100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:28.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.733+0000 7f0f38927640 1 --2- 172.21.15.16:0/2771631444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f34100270 0x7f0f34100650 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0f24009880 tx=0x7f0f2402bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:28.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.733+0000 7f0f38927640 1 -- 172.21.15.16:0/2771631444 shutdown_connections 2024-04-22T16:01:28.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.733+0000 7f0f38927640 1 --2- 172.21.15.16:0/2771631444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f34100c20 0x7f0f34104c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:28.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.733+0000 7f0f38927640 1 --2- 172.21.15.16:0/2771631444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f34100270 0x7f0f34100650 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:28.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.733+0000 7f0f38927640 1 -- 172.21.15.16:0/2771631444 >> 172.21.15.16:0/2771631444 conn(0x7f0f340fbaf0 msgr2=0x7f0f340fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:28.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.733+0000 7f0f38927640 1 -- 172.21.15.16:0/2771631444 shutdown_connections 2024-04-22T16:01:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.733+0000 7f0f38927640 1 -- 172.21.15.16:0/2771631444 wait complete. 2024-04-22T16:01:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.734+0000 7f0f38927640 1 Processor -- start 2024-04-22T16:01:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.734+0000 7f0f38927640 1 -- start start 2024-04-22T16:01:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.734+0000 7f0f38927640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f34100270 0x7f0f34195d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.734+0000 7f0f38927640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f341962d0 0x7f0f3419a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.734+0000 7f0f38927640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f341968d0 con 0x7f0f341962d0 2024-04-22T16:01:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.734+0000 7f0f38927640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f34196a40 con 0x7f0f34100270 2024-04-22T16:01:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.734+0000 7f0f31d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f341962d0 0x7f0f3419a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.734+0000 7f0f31d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f341962d0 0x7f0f3419a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48102/0 (socket says 172.21.15.16:48102) 2024-04-22T16:01:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.734+0000 7f0f31d9b640 1 -- 172.21.15.16:0/1441027419 learned_addr learned my addr 172.21.15.16:0/1441027419 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.734+0000 7f0f3259c640 1 --2- 172.21.15.16:0/1441027419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f34100270 0x7f0f34195d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.734+0000 7f0f31d9b640 1 -- 172.21.15.16:0/1441027419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f34100270 msgr2=0x7f0f34195d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:28.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.734+0000 7f0f31d9b640 1 --2- 172.21.15.16:0/1441027419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f34100270 0x7f0f34195d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.734+0000 7f0f31d9b640 1 -- 172.21.15.16:0/1441027419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f24009560 con 0x7f0f341962d0 2024-04-22T16:01:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.735+0000 7f0f31d9b640 1 --2- 172.21.15.16:0/1441027419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f341962d0 0x7f0f3419a730 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0f2800c920 tx=0x7f0f2800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.735+0000 7f0f237fe640 1 -- 172.21.15.16:0/1441027419 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f28007b80 con 0x7f0f341962d0 2024-04-22T16:01:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.735+0000 7f0f237fe640 1 -- 172.21.15.16:0/1441027419 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f28007ce0 con 0x7f0f341962d0 2024-04-22T16:01:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.735+0000 7f0f38927640 1 -- 172.21.15.16:0/1441027419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f3419ad30 con 0x7f0f341962d0 2024-04-22T16:01:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.735+0000 7f0f237fe640 1 -- 172.21.15.16:0/1441027419 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f280187d0 con 0x7f0f341962d0 2024-04-22T16:01:28.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.735+0000 7f0f38927640 1 -- 172.21.15.16:0/1441027419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f3419b290 con 0x7f0f341962d0 2024-04-22T16:01:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.736+0000 7f0f38927640 1 -- 172.21.15.16:0/1441027419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f3419b570 con 0x7f0f341962d0 2024-04-22T16:01:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.736+0000 7f0f237fe640 1 -- 172.21.15.16:0/1441027419 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0f2800f450 con 0x7f0f341962d0 2024-04-22T16:01:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.737+0000 7f0f237fe640 1 --2- 172.21.15.16:0/1441027419 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0f1c06ec30 0x7f0f1c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.737+0000 7f0f237fe640 1 -- 172.21.15.16:0/1441027419 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0f2808eff0 con 0x7f0f341962d0 2024-04-22T16:01:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.737+0000 7f0f3259c640 1 --2- 172.21.15.16:0/1441027419 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0f1c06ec30 0x7f0f1c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.737+0000 7f0f3259c640 1 --2- 172.21.15.16:0/1441027419 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0f1c06ec30 0x7f0f1c0710f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0f24006010 tx=0x7f0f2402cf90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:28.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.742+0000 7f0f237fe640 1 -- 172.21.15.16:0/1441027419 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0f28057d60 con 0x7f0f341962d0 2024-04-22T16:01:28.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:28 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/3761587074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-22T16:01:28.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:28 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/708541576' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-22T16:01:28.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:28 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/3782074465' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T16:01:28.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.932+0000 7f0f38927640 1 -- 172.21.15.16:0/1441027419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f3404e040 con 0x7f0f341962d0 2024-04-22T16:01:28.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.933+0000 7f0f237fe640 1 -- 172.21.15.16:0/1441027419 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16590 (secure 0 0 0) 0x7f0f2805b420 con 0x7f0f341962d0 2024-04-22T16:01:28.936 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-22T16:01:28.936 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":84,"fsid":"67ab3a80-00be-11ef-bc93-c7b262605968","created":"2024-04-22T15:39:20.727181+0000","modified":"2024-04-22T15:59:06.181284+0000","last_up_change":"2024-04-22T15:58:03.188801+0000","last_in_change":"2024-04-22T15:44:52.054297+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T15:43:58.896563+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-22T15:46:26.715511+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-22T15:46:27.206408+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"67bd69ee-bd8c-4670-927a-f5fbcd92c9bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":53,"up_thru":78,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":3158080196},{"type":"v1","addr":"172.21.15.16:6803","nonce":3158080196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":3158080196},{"type":"v1","addr":"172.21.15.16:6805","nonce":3158080196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":3158080196},{"type":"v1","addr":"172.21.15.16:6809","nonce":3158080196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":3158080196},{"type":"v1","addr":"172.21.15.16:6807","nonce":3158080196}]},"public_addr":"172.21.15.16:6803/3158080196","cluster_addr":"172.21.15.16:6805/3158080196","heartbeat_back_addr":"172.21.15.16:6809/3158080196","heartbeat_front_addr":"172.21.15.16:6807/3158080196","state":["exists","up"]},{"osd":1,"uuid":"f51a9e4d-fb0c-4f62-9328-e23eea863213","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":54,"up_from":58,"up_thru":78,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3524132843},{"type":"v1","addr":"172.21.15.16:6811","nonce":3524132843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3524132843},{"type":"v1","addr":"172.21.15.16:6813","nonce":3524132843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3524132843},{"type":"v1","addr":"172.21.15.16:6817","nonce":3524132843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3524132843},{"type":"v1","addr":"172.21.15.16:6815","nonce":3524132843}]},"public_addr":"172.21.15.16:6811/3524132843","cluster_addr":"172.21.15.16:6813/3524132843","heartbeat_back_addr":"172.21.15.16:6817/3524132843","heartbeat_front_addr":"172.21.15.16:6815/3524132843","state":["exists","up"]},{"osd":2,"uuid":"e59fdb03-e04e-4afd-a58e-0b58945d3765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":59,"up_from":63,"up_thru":78,"down_at":60,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2085469945},{"type":"v1","addr":"172.21.15.16:6819","nonce":2085469945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2085469945},{"type":"v1","addr":"172.21.15.16:6821","nonce":2085469945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2085469945},{"type":"v1","addr":"172.21.15.16:6825","nonce":2085469945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2085469945},{"type":"v1","addr":"172.21.15.16:6823","nonce":2085469945}]},"public_addr":"172.21.15.16:6819/2085469945","cluster_addr":"172.21.15.16:6821/2085469945","heartbeat_back_addr":"172.21.15.16:6825/2085469945","heartbeat_front_addr":"172.21.15.16:6823/2085469945","state":["exists","up"]},{"osd":3,"uuid":"fa25b413-d004-4455-8d7b-0ced7f880a7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":64,"up_from":68,"up_thru":78,"down_at":65,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":365345197},{"type":"v1","addr":"172.21.15.27:6801","nonce":365345197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":365345197},{"type":"v1","addr":"172.21.15.27:6803","nonce":365345197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":365345197},{"type":"v1","addr":"172.21.15.27:6807","nonce":365345197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":365345197},{"type":"v1","addr":"172.21.15.27:6805","nonce":365345197}]},"public_addr":"172.21.15.27:6801/365345197","cluster_addr":"172.21.15.27:6803/365345197","heartbeat_back_addr":"172.21.15.27:6807/365345197","heartbeat_front_addr":"172.21.15.27:6805/365345197","state":["exists","up"]},{"osd":4,"uuid":"6b1eda0d-1627-4c49-93b1-1a78896f660c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":69,"up_from":73,"up_thru":78,"down_at":70,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":84770413},{"type":"v1","addr":"172.21.15.27:6809","nonce":84770413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":84770413},{"type":"v1","addr":"172.21.15.27:6811","nonce":84770413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":84770413},{"type":"v1","addr":"172.21.15.27:6815","nonce":84770413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":84770413},{"type":"v1","addr":"172.21.15.27:6813","nonce":84770413}]},"public_addr":"172.21.15.27:6809/84770413","cluster_addr":"172.21.15.27:6811/84770413","heartbeat_back_addr":"172.21.15.27:6815/84770413","heartbeat_front_addr":"172.21.15.27:6813/84770413","state":["exists","up"]},{"osd":5,"uuid":"58cebe34-0821-458f-86b5-7d460829d994","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":74,"up_from":78,"up_thru":78,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3875730087},{"type":"v1","addr":"172.21.15.27:6817","nonce":3875730087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3875730087},{"type":"v1","addr":"172.21.15.27:6819","nonce":3875730087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3875730087},{"type":"v1","addr":"172.21.15.27:6823","nonce":3875730087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3875730087},{"type":"v1","addr":"172.21.15.27:6821","nonce":3875730087}]},"public_addr":"172.21.15.27:6817/3875730087","cluster_addr":"172.21.15.27:6819/3875730087","heartbeat_back_addr":"172.21.15.27:6823/3875730087","heartbeat_front_addr":"172.21.15.27:6821/3875730087","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:42:58.192450+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:43:27.129522+0000","dead_epoch":54},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:43:54.750854+0000","dead_epoch":59},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:44:19.111479+0000","dead_epoch":64},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:44:44.487429+0000","dead_epoch":69},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T15:45:09.605548+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6827/476026355":"2024-04-23T15:59:06.181260+0000","172.21.15.27:6825/4216277609":"2024-04-23T15:58:44.932745+0000","172.21.15.16:6801/3431522404":"2024-04-23T15:40:27.107187+0000","172.21.15.16:6801/1748504461":"2024-04-23T15:39:53.479948+0000","172.21.15.16:6800/3148690064":"2024-04-23T15:53:30.185361+0000","172.21.15.27:0/4251496318":"2024-04-23T15:49:57.821345+0000","172.21.15.16:6800/3972982132":"2024-04-23T15:41:40.376957+0000","172.21.15.16:6800/3431522404":"2024-04-23T15:40:27.107187+0000","172.21.15.16:0/3682380":"2024-04-23T15:48:41.336415+0000","172.21.15.27:6829/2676414545":"2024-04-23T15:49:57.821345+0000","172.21.15.27:0/186298982":"2024-04-23T15:49:57.821345+0000","172.21.15.27:6828/2676414545":"2024-04-23T15:49:57.821345+0000","172.21.15.16:0/955466902":"2024-04-23T15:40:27.107187+0000","172.21.15.27:6824/4216277609":"2024-04-23T15:58:44.932745+0000","172.21.15.16:0/1878962216":"2024-04-23T15:48:41.336415+0000","172.21.15.16:0/2348469475":"2024-04-23T15:39:53.479948+0000","172.21.15.27:0/2039035396":"2024-04-23T15:49:57.821345+0000","172.21.15.16:0/25024044":"2024-04-23T15:48:41.336415+0000","172.21.15.27:6826/476026355":"2024-04-23T15:59:06.181260+0000","172.21.15.16:0/4172787384":"2024-04-23T15:40:27.107187+0000","172.21.15.16:0/3067906798":"2024-04-23T15:53:30.185361+0000","172.21.15.16:6827/2331116023":"2024-04-23T15:46:44.357516+0000","172.21.15.16:0/2002585335":"2024-04-23T15:40:27.107187+0000","172.21.15.16:0/2570865319":"2024-04-23T15:39:53.479948+0000","172.21.15.16:0/690437556":"2024-04-23T15:41:40.376957+0000","172.21.15.16:6801/3972982132":"2024-04-23T15:41:40.376957+0000","172.21.15.16:0/3645737911":"2024-04-23T15:48:41.336415+0000","172.21.15.16:6826/2331116023":"2024-04-23T15:46:44.357516+0000","172.21.15.16:6800/1748504461":"2024-04-23T15:39:53.479948+0000","172.21.15.16:0/4287530858":"2024-04-23T15:53:30.185361+0000","172.21.15.27:0/3964606592":"2024-04-23T15:49:57.821345+0000","172.21.15.16:0/2048283809":"2024-04-23T15:39:53.479948+0000","172.21.15.16:0/4021274304":"2024-04-23T15:41:40.376957+0000","172.21.15.16:6800/1057234399":"2024-04-23T15:48:41.336415+0000","172.21.15.16:6801/1057234399":"2024-04-23T15:48:41.336415+0000","172.21.15.16:6801/3148690064":"2024-04-23T15:53:30.185361+0000","172.21.15.16:0/4252231167":"2024-04-23T15:41:40.376957+0000","172.21.15.16:0/3802040196":"2024-04-23T15:53:30.185361+0000","172.21.15.16:0/2093505259":"2024-04-23T15:53:30.185361+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T16:01:28.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.935+0000 7f0f38927640 1 -- 172.21.15.16:0/1441027419 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0f1c06ec30 msgr2=0x7f0f1c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:28.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.935+0000 7f0f38927640 1 --2- 172.21.15.16:0/1441027419 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0f1c06ec30 0x7f0f1c0710f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0f24006010 tx=0x7f0f2402cf90 comp rx=0 tx=0).stop 2024-04-22T16:01:28.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.935+0000 7f0f38927640 1 -- 172.21.15.16:0/1441027419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f341962d0 msgr2=0x7f0f3419a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:28.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.935+0000 7f0f38927640 1 --2- 172.21.15.16:0/1441027419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f341962d0 0x7f0f3419a730 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0f2800c920 tx=0x7f0f2800cde0 comp rx=0 tx=0).stop 2024-04-22T16:01:28.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.935+0000 7f0f38927640 1 -- 172.21.15.16:0/1441027419 shutdown_connections 2024-04-22T16:01:28.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.935+0000 7f0f38927640 1 --2- 172.21.15.16:0/1441027419 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f0f1c06ec30 0x7f0f1c0710f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:28.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.935+0000 7f0f38927640 1 --2- 172.21.15.16:0/1441027419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f341962d0 0x7f0f3419a730 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:28.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.935+0000 7f0f38927640 1 --2- 172.21.15.16:0/1441027419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f34100270 0x7f0f34195d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:28.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.935+0000 7f0f38927640 1 -- 172.21.15.16:0/1441027419 >> 172.21.15.16:0/1441027419 conn(0x7f0f340fbaf0 msgr2=0x7f0f340fd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:28.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.935+0000 7f0f38927640 1 -- 172.21.15.16:0/1441027419 shutdown_connections 2024-04-22T16:01:28.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:28.935+0000 7f0f38927640 1 -- 172.21.15.16:0/1441027419 wait complete. 2024-04-22T16:01:28.949 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T15:43:58.896563+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-22T15:46:26.715511+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-22T15:46:27.206408+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T16:01:28.950 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T16:01:29.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.051+0000 7f1b2107d640 1 -- 172.21.15.16:0/1773527085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b1c101b00 msgr2=0x7f1b1c109fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:29.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.051+0000 7f1b2107d640 1 --2- 172.21.15.16:0/1773527085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b1c101b00 0x7f1b1c109fe0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1b10009880 tx=0x7f1b1002bd70 comp rx=0 tx=0).stop 2024-04-22T16:01:29.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.051+0000 7f1b2107d640 1 -- 172.21.15.16:0/1773527085 shutdown_connections 2024-04-22T16:01:29.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.051+0000 7f1b2107d640 1 --2- 172.21.15.16:0/1773527085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b1c101b00 0x7f1b1c109fe0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:29.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.051+0000 7f1b2107d640 1 --2- 172.21.15.16:0/1773527085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b1c1011e0 0x7f1b1c1015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:29.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.051+0000 7f1b2107d640 1 -- 172.21.15.16:0/1773527085 >> 172.21.15.16:0/1773527085 conn(0x7f1b1c0faa20 msgr2=0x7f1b1c0fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:29.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.051+0000 7f1b2107d640 1 -- 172.21.15.16:0/1773527085 shutdown_connections 2024-04-22T16:01:29.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.051+0000 7f1b2107d640 1 -- 172.21.15.16:0/1773527085 wait complete. 2024-04-22T16:01:29.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.052+0000 7f1b2107d640 1 Processor -- start 2024-04-22T16:01:29.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.052+0000 7f1b2107d640 1 -- start start 2024-04-22T16:01:29.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.052+0000 7f1b2107d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b1c1011e0 0x7f1b1c195d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:29.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.053+0000 7f1b2107d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b1c101b00 0x7f1b1c1962b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:29.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.053+0000 7f1b2107d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1c196940 con 0x7f1b1c101b00 2024-04-22T16:01:29.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.053+0000 7f1b2107d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1c19a710 con 0x7f1b1c1011e0 2024-04-22T16:01:29.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.053+0000 7f1b1a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b1c101b00 0x7f1b1c1962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:29.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.053+0000 7f1b1a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b1c101b00 0x7f1b1c1962b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48130/0 (socket says 172.21.15.16:48130) 2024-04-22T16:01:29.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.053+0000 7f1b1a59c640 1 -- 172.21.15.16:0/2908775197 learned_addr learned my addr 172.21.15.16:0/2908775197 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:29.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.053+0000 7f1b1ad9d640 1 --2- 172.21.15.16:0/2908775197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b1c1011e0 0x7f1b1c195d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:29.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.053+0000 7f1b1a59c640 1 -- 172.21.15.16:0/2908775197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b1c1011e0 msgr2=0x7f1b1c195d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:29.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.053+0000 7f1b1a59c640 1 --2- 172.21.15.16:0/2908775197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b1c1011e0 0x7f1b1c195d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:29.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.053+0000 7f1b1a59c640 1 -- 172.21.15.16:0/2908775197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b10009560 con 0x7f1b1c101b00 2024-04-22T16:01:29.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.053+0000 7f1b1ad9d640 1 --2- 172.21.15.16:0/2908775197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b1c1011e0 0x7f1b1c195d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T16:01:29.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.053+0000 7f1b1a59c640 1 --2- 172.21.15.16:0/2908775197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b1c101b00 0x7f1b1c1962b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1b10009230 tx=0x7f1b1002c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:29.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.053+0000 7f1b03fff640 1 -- 172.21.15.16:0/2908775197 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b100044a0 con 0x7f1b1c101b00 2024-04-22T16:01:29.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.053+0000 7f1b03fff640 1 -- 172.21.15.16:0/2908775197 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1b10003ab0 con 0x7f1b1c101b00 2024-04-22T16:01:29.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.053+0000 7f1b2107d640 1 -- 172.21.15.16:0/2908775197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b1c19a990 con 0x7f1b1c101b00 2024-04-22T16:01:29.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.053+0000 7f1b03fff640 1 -- 172.21.15.16:0/2908775197 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b10005600 con 0x7f1b1c101b00 2024-04-22T16:01:29.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.053+0000 7f1b2107d640 1 -- 172.21.15.16:0/2908775197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b1c19aef0 con 0x7f1b1c101b00 2024-04-22T16:01:29.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.055+0000 7f1b2107d640 1 -- 172.21.15.16:0/2908775197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b1c04e040 con 0x7f1b1c101b00 2024-04-22T16:01:29.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.055+0000 7f1b03fff640 1 -- 172.21.15.16:0/2908775197 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1b10004dd0 con 0x7f1b1c101b00 2024-04-22T16:01:29.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.055+0000 7f1b03fff640 1 --2- 172.21.15.16:0/2908775197 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f1b0406ec30 0x7f1b040710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:29.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.056+0000 7f1b03fff640 1 -- 172.21.15.16:0/2908775197 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1b100b1a30 con 0x7f1b1c101b00 2024-04-22T16:01:29.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.056+0000 7f1b1ad9d640 1 --2- 172.21.15.16:0/2908775197 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f1b0406ec30 0x7f1b040710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:29.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.056+0000 7f1b1ad9d640 1 --2- 172.21.15.16:0/2908775197 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f1b0406ec30 0x7f1b040710f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c0058c0 tx=0x7f1b0c005850 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:29.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.060+0000 7f1b03fff640 1 -- 172.21.15.16:0/2908775197 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1b1007a740 con 0x7f1b1c101b00 2024-04-22T16:01:29.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:28 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/3761587074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-22T16:01:29.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:28 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/708541576' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-22T16:01:29.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:28 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/3782074465' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T16:01:29.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.251+0000 7f1b2107d640 1 -- 172.21.15.16:0/2908775197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1b1c10d380 con 0x7f1b1c101b00 2024-04-22T16:01:29.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.251+0000 7f1b03fff640 1 -- 172.21.15.16:0/2908775197 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1b1007de00 con 0x7f1b1c101b00 2024-04-22T16:01:29.254 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-22T16:01:29.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.253+0000 7f1b2107d640 1 -- 172.21.15.16:0/2908775197 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f1b0406ec30 msgr2=0x7f1b040710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:29.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.253+0000 7f1b2107d640 1 --2- 172.21.15.16:0/2908775197 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f1b0406ec30 0x7f1b040710f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c0058c0 tx=0x7f1b0c005850 comp rx=0 tx=0).stop 2024-04-22T16:01:29.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.253+0000 7f1b2107d640 1 -- 172.21.15.16:0/2908775197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b1c101b00 msgr2=0x7f1b1c1962b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:29.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.253+0000 7f1b2107d640 1 --2- 172.21.15.16:0/2908775197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b1c101b00 0x7f1b1c1962b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1b10009230 tx=0x7f1b1002c7a0 comp rx=0 tx=0).stop 2024-04-22T16:01:29.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.254+0000 7f1b2107d640 1 -- 172.21.15.16:0/2908775197 shutdown_connections 2024-04-22T16:01:29.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.254+0000 7f1b2107d640 1 --2- 172.21.15.16:0/2908775197 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f1b0406ec30 0x7f1b040710f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:29.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.254+0000 7f1b2107d640 1 --2- 172.21.15.16:0/2908775197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b1c101b00 0x7f1b1c1962b0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:29.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.254+0000 7f1b2107d640 1 --2- 172.21.15.16:0/2908775197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b1c1011e0 0x7f1b1c195d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:29.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.254+0000 7f1b2107d640 1 -- 172.21.15.16:0/2908775197 >> 172.21.15.16:0/2908775197 conn(0x7f1b1c0faa20 msgr2=0x7f1b1c104820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:29.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.254+0000 7f1b2107d640 1 -- 172.21.15.16:0/2908775197 shutdown_connections 2024-04-22T16:01:29.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.254+0000 7f1b2107d640 1 -- 172.21.15.16:0/2908775197 wait complete. 2024-04-22T16:01:29.268 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-22T16:01:29.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.369+0000 7f47bca07640 1 -- 172.21.15.16:0/2791934083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b80ff180 msgr2=0x7f47b80ff560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:29.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.369+0000 7f47bca07640 1 --2- 172.21.15.16:0/2791934083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b80ff180 0x7f47b80ff560 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f47b0009880 tx=0x7f47b002bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:29.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.369+0000 7f47bca07640 1 -- 172.21.15.16:0/2791934083 shutdown_connections 2024-04-22T16:01:29.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.369+0000 7f47bca07640 1 --2- 172.21.15.16:0/2791934083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47b80ffb30 0x7f47b8103b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:29.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.369+0000 7f47bca07640 1 --2- 172.21.15.16:0/2791934083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b80ff180 0x7f47b80ff560 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:29.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.369+0000 7f47bca07640 1 -- 172.21.15.16:0/2791934083 >> 172.21.15.16:0/2791934083 conn(0x7f47b80faa20 msgr2=0x7f47b80fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:29.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.369+0000 7f47bca07640 1 -- 172.21.15.16:0/2791934083 shutdown_connections 2024-04-22T16:01:29.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.369+0000 7f47bca07640 1 -- 172.21.15.16:0/2791934083 wait complete. 2024-04-22T16:01:29.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.370+0000 7f47bca07640 1 Processor -- start 2024-04-22T16:01:29.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.370+0000 7f47bca07640 1 -- start start 2024-04-22T16:01:29.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.370+0000 7f47bca07640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47b80ff180 0x7f47b8195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:29.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.370+0000 7f47bca07640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b8196370 0x7f47b819a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:29.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.370+0000 7f47bca07640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b8196970 con 0x7f47b8196370 2024-04-22T16:01:29.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.370+0000 7f47bca07640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b8196ae0 con 0x7f47b80ff180 2024-04-22T16:01:29.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.370+0000 7f47b5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b8196370 0x7f47b819a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:29.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.371+0000 7f47b5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b8196370 0x7f47b819a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48146/0 (socket says 172.21.15.16:48146) 2024-04-22T16:01:29.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.371+0000 7f47b5d9b640 1 -- 172.21.15.16:0/44777390 learned_addr learned my addr 172.21.15.16:0/44777390 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:29.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.371+0000 7f47b659c640 1 --2- 172.21.15.16:0/44777390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47b80ff180 0x7f47b8195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.371+0000 7f47b5d9b640 1 -- 172.21.15.16:0/44777390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47b80ff180 msgr2=0x7f47b8195e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.371+0000 7f47b5d9b640 1 --2- 172.21.15.16:0/44777390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47b80ff180 0x7f47b8195e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.371+0000 7f47b5d9b640 1 -- 172.21.15.16:0/44777390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b0009560 con 0x7f47b8196370 2024-04-22T16:01:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.371+0000 7f47b5d9b640 1 --2- 172.21.15.16:0/44777390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b8196370 0x7f47b819a7d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f47a000e9a0 tx=0x7f47a000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.371+0000 7f47af7fe640 1 -- 172.21.15.16:0/44777390 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47a000cc90 con 0x7f47b8196370 2024-04-22T16:01:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.371+0000 7f47af7fe640 1 -- 172.21.15.16:0/44777390 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f47a000cdf0 con 0x7f47b8196370 2024-04-22T16:01:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.371+0000 7f47bca07640 1 -- 172.21.15.16:0/44777390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b819add0 con 0x7f47b8196370 2024-04-22T16:01:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.371+0000 7f47af7fe640 1 -- 172.21.15.16:0/44777390 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47a0010430 con 0x7f47b8196370 2024-04-22T16:01:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.371+0000 7f47bca07640 1 -- 172.21.15.16:0/44777390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b819b330 con 0x7f47b8196370 2024-04-22T16:01:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.372+0000 7f47bca07640 1 -- 172.21.15.16:0/44777390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47b819b610 con 0x7f47b8196370 2024-04-22T16:01:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.373+0000 7f47af7fe640 1 -- 172.21.15.16:0/44777390 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f47a0005020 con 0x7f47b8196370 2024-04-22T16:01:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.373+0000 7f47af7fe640 1 --2- 172.21.15.16:0/44777390 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f47a406ec30 0x7f47a40710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:29.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.373+0000 7f47af7fe640 1 -- 172.21.15.16:0/44777390 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f47a0014070 con 0x7f47b8196370 2024-04-22T16:01:29.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.373+0000 7f47b659c640 1 --2- 172.21.15.16:0/44777390 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f47a406ec30 0x7f47a40710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:29.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.374+0000 7f47b659c640 1 --2- 172.21.15.16:0/44777390 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f47a406ec30 0x7f47a40710f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f47b0036040 tx=0x7f47b0005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:29.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.378+0000 7f47af7fe640 1 -- 172.21.15.16:0/44777390 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f47a00581c0 con 0x7f47b8196370 2024-04-22T16:01:29.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.566+0000 7f47bca07640 1 -- 172.21.15.16:0/44777390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f47b804e040 con 0x7f47b8196370 2024-04-22T16:01:29.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.567+0000 7f47af7fe640 1 -- 172.21.15.16:0/44777390 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f47a005b880 con 0x7f47b8196370 2024-04-22T16:01:29.569 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-22T16:01:29.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.569+0000 7f47bca07640 1 -- 172.21.15.16:0/44777390 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f47a406ec30 msgr2=0x7f47a40710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:29.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.569+0000 7f47bca07640 1 --2- 172.21.15.16:0/44777390 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f47a406ec30 0x7f47a40710f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f47b0036040 tx=0x7f47b0005fd0 comp rx=0 tx=0).stop 2024-04-22T16:01:29.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.569+0000 7f47bca07640 1 -- 172.21.15.16:0/44777390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b8196370 msgr2=0x7f47b819a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:29.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.569+0000 7f47bca07640 1 --2- 172.21.15.16:0/44777390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b8196370 0x7f47b819a7d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f47a000e9a0 tx=0x7f47a000ee60 comp rx=0 tx=0).stop 2024-04-22T16:01:29.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.569+0000 7f47bca07640 1 -- 172.21.15.16:0/44777390 shutdown_connections 2024-04-22T16:01:29.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.569+0000 7f47bca07640 1 --2- 172.21.15.16:0/44777390 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f47a406ec30 0x7f47a40710f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:29.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.569+0000 7f47bca07640 1 --2- 172.21.15.16:0/44777390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b8196370 0x7f47b819a7d0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:29.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.569+0000 7f47bca07640 1 --2- 172.21.15.16:0/44777390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47b80ff180 0x7f47b8195e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:29.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.569+0000 7f47bca07640 1 -- 172.21.15.16:0/44777390 >> 172.21.15.16:0/44777390 conn(0x7f47b80faa20 msgr2=0x7f47b81033d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:29.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.569+0000 7f47bca07640 1 -- 172.21.15.16:0/44777390 shutdown_connections 2024-04-22T16:01:29.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.569+0000 7f47bca07640 1 -- 172.21.15.16:0/44777390 wait complete. 2024-04-22T16:01:29.583 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-22T16:01:29.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.687+0000 7f15f076d640 1 -- 172.21.15.16:0/3956501123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15e8067340 msgr2=0x7f15e8067780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:29.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.687+0000 7f15f076d640 1 --2- 172.21.15.16:0/3956501123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15e8067340 0x7f15e8067780 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f15e4009880 tx=0x7f15e402bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:29.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.687+0000 7f15f076d640 1 -- 172.21.15.16:0/3956501123 shutdown_connections 2024-04-22T16:01:29.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.687+0000 7f15f076d640 1 --2- 172.21.15.16:0/3956501123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15e8067340 0x7f15e8067780 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:29.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.687+0000 7f15f076d640 1 --2- 172.21.15.16:0/3956501123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15e81060c0 0x7f15e81064a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:29.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.687+0000 7f15f076d640 1 -- 172.21.15.16:0/3956501123 >> 172.21.15.16:0/3956501123 conn(0x7f15e8075b50 msgr2=0x7f15e8075f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:29.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.687+0000 7f15f076d640 1 -- 172.21.15.16:0/3956501123 shutdown_connections 2024-04-22T16:01:29.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.687+0000 7f15f076d640 1 -- 172.21.15.16:0/3956501123 wait complete. 2024-04-22T16:01:29.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.687+0000 7f15f076d640 1 Processor -- start 2024-04-22T16:01:29.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.688+0000 7f15f076d640 1 -- start start 2024-04-22T16:01:29.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.688+0000 7f15f076d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15e8067340 0x7f15e8195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:29.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.688+0000 7f15f076d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15e81060c0 0x7f15e8196280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:29.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.688+0000 7f15f076d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e8196910 con 0x7f15e8067340 2024-04-22T16:01:29.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.688+0000 7f15f076d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e819a6e0 con 0x7f15e81060c0 2024-04-22T16:01:29.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.688+0000 7f15ee509640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15e8067340 0x7f15e8195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:29.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.688+0000 7f15ee509640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15e8067340 0x7f15e8195d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48154/0 (socket says 172.21.15.16:48154) 2024-04-22T16:01:29.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.688+0000 7f15ee509640 1 -- 172.21.15.16:0/3312729368 learned_addr learned my addr 172.21.15.16:0/3312729368 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:29.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.688+0000 7f15edd08640 1 --2- 172.21.15.16:0/3312729368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15e81060c0 0x7f15e8196280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:29.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.688+0000 7f15ee509640 1 -- 172.21.15.16:0/3312729368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15e81060c0 msgr2=0x7f15e8196280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:29.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.688+0000 7f15ee509640 1 --2- 172.21.15.16:0/3312729368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15e81060c0 0x7f15e8196280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:29.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.688+0000 7f15ee509640 1 -- 172.21.15.16:0/3312729368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e4009560 con 0x7f15e8067340 2024-04-22T16:01:29.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.688+0000 7f15ee509640 1 --2- 172.21.15.16:0/3312729368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15e8067340 0x7f15e8195d40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f15dc00e970 tx=0x7f15dc00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:29.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.688+0000 7f15db7fe640 1 -- 172.21.15.16:0/3312729368 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15dc00cc90 con 0x7f15e8067340 2024-04-22T16:01:29.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.688+0000 7f15db7fe640 1 -- 172.21.15.16:0/3312729368 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f15dc00cdf0 con 0x7f15e8067340 2024-04-22T16:01:29.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.688+0000 7f15f076d640 1 -- 172.21.15.16:0/3312729368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e819a9c0 con 0x7f15e8067340 2024-04-22T16:01:29.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.688+0000 7f15db7fe640 1 -- 172.21.15.16:0/3312729368 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15dc010430 con 0x7f15e8067340 2024-04-22T16:01:29.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.689+0000 7f15f076d640 1 -- 172.21.15.16:0/3312729368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e819af20 con 0x7f15e8067340 2024-04-22T16:01:29.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.690+0000 7f15f076d640 1 -- 172.21.15.16:0/3312729368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e804e040 con 0x7f15e8067340 2024-04-22T16:01:29.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.690+0000 7f15db7fe640 1 -- 172.21.15.16:0/3312729368 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f15dc0046c0 con 0x7f15e8067340 2024-04-22T16:01:29.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.691+0000 7f15db7fe640 1 --2- 172.21.15.16:0/3312729368 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f15d406ec30 0x7f15d40710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:29.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.691+0000 7f15db7fe640 1 -- 172.21.15.16:0/3312729368 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f15dc014070 con 0x7f15e8067340 2024-04-22T16:01:29.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.691+0000 7f15edd08640 1 --2- 172.21.15.16:0/3312729368 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f15d406ec30 0x7f15d40710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:29.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.691+0000 7f15edd08640 1 --2- 172.21.15.16:0/3312729368 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f15d406ec30 0x7f15d40710f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f15e4036040 tx=0x7f15e4005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:29.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.695+0000 7f15db7fe640 1 -- 172.21.15.16:0/3312729368 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f15dc058210 con 0x7f15e8067340 2024-04-22T16:01:29.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.882+0000 7f15f076d640 1 -- 172.21.15.16:0/3312729368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f15e8108540 con 0x7f15e8067340 2024-04-22T16:01:29.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.883+0000 7f15db7fe640 1 -- 172.21.15.16:0/3312729368 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f15dc05b8d0 con 0x7f15e8067340 2024-04-22T16:01:29.886 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-22T16:01:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.885+0000 7f15f076d640 1 -- 172.21.15.16:0/3312729368 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f15d406ec30 msgr2=0x7f15d40710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.885+0000 7f15f076d640 1 --2- 172.21.15.16:0/3312729368 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f15d406ec30 0x7f15d40710f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f15e4036040 tx=0x7f15e4005fd0 comp rx=0 tx=0).stop 2024-04-22T16:01:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.885+0000 7f15f076d640 1 -- 172.21.15.16:0/3312729368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15e8067340 msgr2=0x7f15e8195d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.885+0000 7f15f076d640 1 --2- 172.21.15.16:0/3312729368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15e8067340 0x7f15e8195d40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f15dc00e970 tx=0x7f15dc00ed70 comp rx=0 tx=0).stop 2024-04-22T16:01:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.885+0000 7f15f076d640 1 -- 172.21.15.16:0/3312729368 shutdown_connections 2024-04-22T16:01:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.885+0000 7f15f076d640 1 --2- 172.21.15.16:0/3312729368 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f15d406ec30 0x7f15d40710f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.885+0000 7f15f076d640 1 --2- 172.21.15.16:0/3312729368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15e81060c0 0x7f15e8196280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.885+0000 7f15f076d640 1 --2- 172.21.15.16:0/3312729368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15e8067340 0x7f15e8195d40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.885+0000 7f15f076d640 1 -- 172.21.15.16:0/3312729368 >> 172.21.15.16:0/3312729368 conn(0x7f15e8075b50 msgr2=0x7f15e80fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.885+0000 7f15f076d640 1 -- 172.21.15.16:0/3312729368 shutdown_connections 2024-04-22T16:01:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.885+0000 7f15f076d640 1 -- 172.21.15.16:0/3312729368 wait complete. 2024-04-22T16:01:29.897 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:29 smithi016 ceph-mon[110740]: pgmap v277: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:29.898 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:29 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/1441027419' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T16:01:29.898 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:29 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/2908775197' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T16:01:29.898 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:29 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/44777390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-22T16:01:29.899 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T16:01:30.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.998+0000 7f5a3c658640 1 -- 172.21.15.16:0/3899318836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a340ff160 msgr2=0x7f5a340ff540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:30.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.998+0000 7f5a3c658640 1 --2- 172.21.15.16:0/3899318836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a340ff160 0x7f5a340ff540 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5a30009880 tx=0x7f5a3002bce0 comp rx=0 tx=0).stop 2024-04-22T16:01:30.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.998+0000 7f5a3c658640 1 -- 172.21.15.16:0/3899318836 shutdown_connections 2024-04-22T16:01:30.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.998+0000 7f5a3c658640 1 --2- 172.21.15.16:0/3899318836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a340ffb10 0x7f5a34103b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:30.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.998+0000 7f5a3c658640 1 --2- 172.21.15.16:0/3899318836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a340ff160 0x7f5a340ff540 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:30.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.998+0000 7f5a3c658640 1 -- 172.21.15.16:0/3899318836 >> 172.21.15.16:0/3899318836 conn(0x7f5a340fa9c0 msgr2=0x7f5a340fcde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:30.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.998+0000 7f5a3c658640 1 -- 172.21.15.16:0/3899318836 shutdown_connections 2024-04-22T16:01:30.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.998+0000 7f5a3c658640 1 -- 172.21.15.16:0/3899318836 wait complete. 2024-04-22T16:01:30.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.999+0000 7f5a3c658640 1 Processor -- start 2024-04-22T16:01:30.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.999+0000 7f5a3c658640 1 -- start start 2024-04-22T16:01:30.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.999+0000 7f5a3c658640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a340ff160 0x7f5a34195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:30.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.999+0000 7f5a3c658640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a341962e0 0x7f5a3419a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:30.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.999+0000 7f5a3c658640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a341968e0 con 0x7f5a340ff160 2024-04-22T16:01:30.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.999+0000 7f5a3c658640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a34196a50 con 0x7f5a341962e0 2024-04-22T16:01:30.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.999+0000 7f5a3a3f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a340ff160 0x7f5a34195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:30.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.999+0000 7f5a3a3f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a340ff160 0x7f5a34195da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48160/0 (socket says 172.21.15.16:48160) 2024-04-22T16:01:30.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.999+0000 7f5a3a3f4640 1 -- 172.21.15.16:0/2139758984 learned_addr learned my addr 172.21.15.16:0/2139758984 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-22T16:01:30.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.999+0000 7f5a39bf3640 1 --2- 172.21.15.16:0/2139758984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a341962e0 0x7f5a3419a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:30.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.999+0000 7f5a3a3f4640 1 -- 172.21.15.16:0/2139758984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a341962e0 msgr2=0x7f5a3419a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:30.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.999+0000 7f5a3a3f4640 1 --2- 172.21.15.16:0/2139758984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a341962e0 0x7f5a3419a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:30.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.999+0000 7f5a3a3f4640 1 -- 172.21.15.16:0/2139758984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a30009560 con 0x7f5a340ff160 2024-04-22T16:01:30.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:29.999+0000 7f5a39bf3640 1 --2- 172.21.15.16:0/2139758984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a341962e0 0x7f5a3419a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T16:01:30.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.000+0000 7f5a3a3f4640 1 --2- 172.21.15.16:0/2139758984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a340ff160 0x7f5a34195da0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5a30009430 tx=0x7f5a30004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:30.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.000+0000 7f5a277fe640 1 -- 172.21.15.16:0/2139758984 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a300035f0 con 0x7f5a340ff160 2024-04-22T16:01:30.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.000+0000 7f5a277fe640 1 -- 172.21.15.16:0/2139758984 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a30003c10 con 0x7f5a340ff160 2024-04-22T16:01:30.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.000+0000 7f5a277fe640 1 -- 172.21.15.16:0/2139758984 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a30005670 con 0x7f5a340ff160 2024-04-22T16:01:30.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.000+0000 7f5a3c658640 1 -- 172.21.15.16:0/2139758984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a3419ace0 con 0x7f5a340ff160 2024-04-22T16:01:30.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.000+0000 7f5a3c658640 1 -- 172.21.15.16:0/2139758984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a3419b240 con 0x7f5a340ff160 2024-04-22T16:01:30.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.001+0000 7f5a3c658640 1 -- 172.21.15.16:0/2139758984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a3404e040 con 0x7f5a340ff160 2024-04-22T16:01:30.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.001+0000 7f5a277fe640 1 -- 172.21.15.16:0/2139758984 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5a30004dd0 con 0x7f5a340ff160 2024-04-22T16:01:30.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.002+0000 7f5a277fe640 1 --2- 172.21.15.16:0/2139758984 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f5a2006ebe0 0x7f5a200710a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T16:01:30.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.002+0000 7f5a277fe640 1 -- 172.21.15.16:0/2139758984 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5a300b1220 con 0x7f5a340ff160 2024-04-22T16:01:30.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.002+0000 7f5a39bf3640 1 --2- 172.21.15.16:0/2139758984 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f5a2006ebe0 0x7f5a200710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T16:01:30.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.002+0000 7f5a39bf3640 1 --2- 172.21.15.16:0/2139758984 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f5a2006ebe0 0x7f5a200710a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5a280058c0 tx=0x7f5a28005850 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T16:01:30.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.006+0000 7f5a277fe640 1 -- 172.21.15.16:0/2139758984 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5a30079fd0 con 0x7f5a340ff160 2024-04-22T16:01:30.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:29 smithi027 ceph-mon[83054]: pgmap v277: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:30.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:29 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/1441027419' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T16:01:30.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:29 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/2908775197' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T16:01:30.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:29 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/44777390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-22T16:01:30.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.207+0000 7f5a3c658640 1 -- 172.21.15.16:0/2139758984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5a341970f0 con 0x7f5a340ff160 2024-04-22T16:01:30.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.207+0000 7f5a277fe640 1 -- 172.21.15.16:0/2139758984 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 39 entries v84) v1 ==== 81+0+2306 (secure 0 0 0) 0x7f5a3009ce60 con 0x7f5a340ff160 2024-04-22T16:01:30.210 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.27:6827/476026355 2024-04-23T15:59:06.181260+0000 2024-04-22T16:01:30.210 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.27:6825/4216277609 2024-04-23T15:58:44.932745+0000 2024-04-22T16:01:30.211 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/3431522404 2024-04-23T15:40:27.107187+0000 2024-04-22T16:01:30.211 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/1748504461 2024-04-23T15:39:53.479948+0000 2024-04-22T16:01:30.211 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/3148690064 2024-04-23T15:53:30.185361+0000 2024-04-22T16:01:30.211 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.27:0/4251496318 2024-04-23T15:49:57.821345+0000 2024-04-22T16:01:30.211 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/3972982132 2024-04-23T15:41:40.376957+0000 2024-04-22T16:01:30.211 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/3431522404 2024-04-23T15:40:27.107187+0000 2024-04-22T16:01:30.211 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3682380 2024-04-23T15:48:41.336415+0000 2024-04-22T16:01:30.211 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.27:6829/2676414545 2024-04-23T15:49:57.821345+0000 2024-04-22T16:01:30.211 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.27:0/186298982 2024-04-23T15:49:57.821345+0000 2024-04-22T16:01:30.211 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.27:6828/2676414545 2024-04-23T15:49:57.821345+0000 2024-04-22T16:01:30.211 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/955466902 2024-04-23T15:40:27.107187+0000 2024-04-22T16:01:30.211 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.27:6824/4216277609 2024-04-23T15:58:44.932745+0000 2024-04-22T16:01:30.211 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1878962216 2024-04-23T15:48:41.336415+0000 2024-04-22T16:01:30.212 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2348469475 2024-04-23T15:39:53.479948+0000 2024-04-22T16:01:30.212 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.27:0/2039035396 2024-04-23T15:49:57.821345+0000 2024-04-22T16:01:30.212 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/25024044 2024-04-23T15:48:41.336415+0000 2024-04-22T16:01:30.212 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.27:6826/476026355 2024-04-23T15:59:06.181260+0000 2024-04-22T16:01:30.212 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4172787384 2024-04-23T15:40:27.107187+0000 2024-04-22T16:01:30.212 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3067906798 2024-04-23T15:53:30.185361+0000 2024-04-22T16:01:30.212 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6827/2331116023 2024-04-23T15:46:44.357516+0000 2024-04-22T16:01:30.212 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2002585335 2024-04-23T15:40:27.107187+0000 2024-04-22T16:01:30.212 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2570865319 2024-04-23T15:39:53.479948+0000 2024-04-22T16:01:30.212 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/690437556 2024-04-23T15:41:40.376957+0000 2024-04-22T16:01:30.212 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/3972982132 2024-04-23T15:41:40.376957+0000 2024-04-22T16:01:30.212 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3645737911 2024-04-23T15:48:41.336415+0000 2024-04-22T16:01:30.212 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6826/2331116023 2024-04-23T15:46:44.357516+0000 2024-04-22T16:01:30.213 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/1748504461 2024-04-23T15:39:53.479948+0000 2024-04-22T16:01:30.213 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4287530858 2024-04-23T15:53:30.185361+0000 2024-04-22T16:01:30.213 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.27:0/3964606592 2024-04-23T15:49:57.821345+0000 2024-04-22T16:01:30.213 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2048283809 2024-04-23T15:39:53.479948+0000 2024-04-22T16:01:30.213 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4021274304 2024-04-23T15:41:40.376957+0000 2024-04-22T16:01:30.213 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/1057234399 2024-04-23T15:48:41.336415+0000 2024-04-22T16:01:30.213 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/1057234399 2024-04-23T15:48:41.336415+0000 2024-04-22T16:01:30.213 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/3148690064 2024-04-23T15:53:30.185361+0000 2024-04-22T16:01:30.213 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4252231167 2024-04-23T15:41:40.376957+0000 2024-04-22T16:01:30.213 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3802040196 2024-04-23T15:53:30.185361+0000 2024-04-22T16:01:30.213 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2093505259 2024-04-23T15:53:30.185361+0000 2024-04-22T16:01:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:listed 39 entries 2024-04-22T16:01:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.209+0000 7f5a3c658640 1 -- 172.21.15.16:0/2139758984 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f5a2006ebe0 msgr2=0x7f5a200710a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.209+0000 7f5a3c658640 1 --2- 172.21.15.16:0/2139758984 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f5a2006ebe0 0x7f5a200710a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5a280058c0 tx=0x7f5a28005850 comp rx=0 tx=0).stop 2024-04-22T16:01:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.209+0000 7f5a3c658640 1 -- 172.21.15.16:0/2139758984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a340ff160 msgr2=0x7f5a34195da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T16:01:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.209+0000 7f5a3c658640 1 --2- 172.21.15.16:0/2139758984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a340ff160 0x7f5a34195da0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5a30009430 tx=0x7f5a30004040 comp rx=0 tx=0).stop 2024-04-22T16:01:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.210+0000 7f5a3c658640 1 -- 172.21.15.16:0/2139758984 shutdown_connections 2024-04-22T16:01:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.210+0000 7f5a3c658640 1 --2- 172.21.15.16:0/2139758984 >> [v2:172.21.15.16:6800/583399364,v1:172.21.15.16:6801/583399364] conn(0x7f5a2006ebe0 0x7f5a200710a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:30.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.210+0000 7f5a3c658640 1 --2- 172.21.15.16:0/2139758984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a341962e0 0x7f5a3419a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:30.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.210+0000 7f5a3c658640 1 --2- 172.21.15.16:0/2139758984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a340ff160 0x7f5a34195da0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T16:01:30.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.210+0000 7f5a3c658640 1 -- 172.21.15.16:0/2139758984 >> 172.21.15.16:0/2139758984 conn(0x7f5a340fa9c0 msgr2=0x7f5a34103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T16:01:30.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.210+0000 7f5a3c658640 1 -- 172.21.15.16:0/2139758984 shutdown_connections 2024-04-22T16:01:30.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-22T16:01:30.210+0000 7f5a3c658640 1 -- 172.21.15.16:0/2139758984 wait complete. 2024-04-22T16:01:30.223 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-04-22T16:01:30.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T16:01:30.224 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-22T16:01:30.248 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T16:01:31.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:30 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/3312729368' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-22T16:01:31.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:30 smithi027 ceph-mon[83054]: from='client.? 172.21.15.16:0/2139758984' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T16:01:31.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:30 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:01:31.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:30 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/3312729368' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-22T16:01:31.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:30 smithi016 ceph-mon[110740]: from='client.? 172.21.15.16:0/2139758984' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T16:01:31.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:01:32.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:31 smithi027 ceph-mon[83054]: pgmap v278: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:01:32.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:31 smithi016 ceph-mon[110740]: pgmap v278: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:01:34.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:33 smithi027 ceph-mon[83054]: pgmap v279: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:34.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:33 smithi016 ceph-mon[110740]: pgmap v279: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:36.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:35 smithi027 ceph-mon[83054]: pgmap v280: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:01:36.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:35 smithi016 ceph-mon[110740]: pgmap v280: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:01:38.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:37 smithi027 ceph-mon[83054]: pgmap v281: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:38.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:37 smithi016 ceph-mon[110740]: pgmap v281: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:40.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:39 smithi027 ceph-mon[83054]: pgmap v282: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:40.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:39 smithi016 ceph-mon[110740]: pgmap v282: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:42.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:41 smithi027 ceph-mon[83054]: pgmap v283: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:01:42.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:41 smithi016 ceph-mon[110740]: pgmap v283: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:01:44.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:43 smithi027 ceph-mon[83054]: pgmap v284: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:44.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:43 smithi016 ceph-mon[110740]: pgmap v284: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:46.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:45 smithi027 ceph-mon[83054]: pgmap v285: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:01:46.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:45 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:01:46.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:45 smithi016 ceph-mon[110740]: pgmap v285: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:01:46.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:45 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:01:48.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:47 smithi027 ceph-mon[83054]: pgmap v286: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:48.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:47 smithi016 ceph-mon[110740]: pgmap v286: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:49.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:48 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T16:01:49.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:48 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T16:01:50.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:49 smithi027 ceph-mon[83054]: pgmap v287: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:50.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:49 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T16:01:50.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:49 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T16:01:50.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:49 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T16:01:50.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:49 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T16:01:50.337 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:49 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T16:01:50.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:49 smithi016 ceph-mon[110740]: pgmap v287: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:50.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:49 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T16:01:50.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:49 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T16:01:50.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:49 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T16:01:50.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:49 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T16:01:50.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:49 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T16:01:52.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:51 smithi027 ceph-mon[83054]: pgmap v288: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:01:52.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:51 smithi016 ceph-mon[110740]: pgmap v288: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:01:54.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:54 smithi027 ceph-mon[83054]: pgmap v289: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:54.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:54 smithi016 ceph-mon[110740]: pgmap v289: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:56.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:56 smithi027 ceph-mon[83054]: pgmap v290: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:01:56.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:56 smithi016 ceph-mon[110740]: pgmap v290: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:01:58.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:01:58 smithi016 ceph-mon[110740]: pgmap v291: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:01:58.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:01:58 smithi027 ceph-mon[83054]: pgmap v291: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:00.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:00 smithi027 ceph-mon[83054]: pgmap v292: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:00.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:00 smithi016 ceph-mon[110740]: pgmap v292: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:01.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:01 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:02:01.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:01 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:02:02.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:02 smithi027 ceph-mon[83054]: pgmap v293: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:02:02.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:02 smithi016 ceph-mon[110740]: pgmap v293: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:02:04.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:04 smithi027 ceph-mon[83054]: pgmap v294: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:04.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:04 smithi016 ceph-mon[110740]: pgmap v294: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:06.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:06 smithi027 ceph-mon[83054]: pgmap v295: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:02:06.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:06 smithi016 ceph-mon[110740]: pgmap v295: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:02:08.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:08 smithi027 ceph-mon[83054]: pgmap v296: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:08.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:08 smithi016 ceph-mon[110740]: pgmap v296: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:10.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:10 smithi027 ceph-mon[83054]: pgmap v297: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:10.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:10 smithi016 ceph-mon[110740]: pgmap v297: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:12.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:12 smithi027 ceph-mon[83054]: pgmap v298: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:02:12.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:12 smithi016 ceph-mon[110740]: pgmap v298: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:02:14.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:14 smithi027 ceph-mon[83054]: pgmap v299: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:14.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:14 smithi016 ceph-mon[110740]: pgmap v299: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:16.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:16 smithi027 ceph-mon[83054]: pgmap v300: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:02:16.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:16 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:02:16.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:16 smithi016 ceph-mon[110740]: pgmap v300: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:02:16.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:16 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:02:18.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:18 smithi027 ceph-mon[83054]: pgmap v301: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:18.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:18 smithi016 ceph-mon[110740]: pgmap v301: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:20.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:20 smithi027 ceph-mon[83054]: pgmap v302: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:20.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:20 smithi016 ceph-mon[110740]: pgmap v302: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:22.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:22 smithi027 ceph-mon[83054]: pgmap v303: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:02:22.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:22 smithi016 ceph-mon[110740]: pgmap v303: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:02:24.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:24 smithi027 ceph-mon[83054]: pgmap v304: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:24.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:24 smithi016 ceph-mon[110740]: pgmap v304: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:26.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:26 smithi027 ceph-mon[83054]: pgmap v305: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:02:26.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:26 smithi016 ceph-mon[110740]: pgmap v305: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:02:28.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:28 smithi027 ceph-mon[83054]: pgmap v306: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:28.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:28 smithi016 ceph-mon[110740]: pgmap v306: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:30.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:30 smithi027 ceph-mon[83054]: pgmap v307: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:30.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:30 smithi016 ceph-mon[110740]: pgmap v307: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:31.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:31 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:02:31.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:31 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:02:32.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:32 smithi027 ceph-mon[83054]: pgmap v308: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:02:32.412 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:32 smithi016 ceph-mon[110740]: pgmap v308: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:02:34.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:34 smithi027 ceph-mon[83054]: pgmap v309: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:34.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:34 smithi016 ceph-mon[110740]: pgmap v309: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:36.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:36 smithi027 ceph-mon[83054]: pgmap v310: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:02:36.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:36 smithi016 ceph-mon[110740]: pgmap v310: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:02:38.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:38 smithi027 ceph-mon[83054]: pgmap v311: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:38.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:38 smithi016 ceph-mon[110740]: pgmap v311: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:40.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:40 smithi027 ceph-mon[83054]: pgmap v312: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:40.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:40 smithi016 ceph-mon[110740]: pgmap v312: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:42.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:42 smithi027 ceph-mon[83054]: pgmap v313: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:02:42.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:42 smithi016 ceph-mon[110740]: pgmap v313: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:02:44.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:44 smithi027 ceph-mon[83054]: pgmap v314: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:44.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:44 smithi016 ceph-mon[110740]: pgmap v314: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:46.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:46 smithi027 ceph-mon[83054]: pgmap v315: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:02:46.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:02:46.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:46 smithi016 ceph-mon[110740]: pgmap v315: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:02:46.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:02:48.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:48 smithi016 ceph-mon[110740]: pgmap v316: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:48.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:48 smithi027 ceph-mon[83054]: pgmap v316: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:50.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:50 smithi016 ceph-mon[110740]: pgmap v317: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:50.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:50 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T16:02:50.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:50 smithi027 ceph-mon[83054]: pgmap v317: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:50.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:50 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T16:02:51.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:51 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T16:02:51.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:51 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T16:02:51.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:51 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T16:02:51.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:51 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T16:02:51.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:51 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T16:02:51.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:51 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T16:02:52.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:52 smithi016 ceph-mon[110740]: pgmap v318: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:02:52.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:52 smithi027 ceph-mon[83054]: pgmap v318: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:02:54.417 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:54 smithi016 ceph-mon[110740]: pgmap v319: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:54.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:54 smithi027 ceph-mon[83054]: pgmap v319: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:56.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:56 smithi016 ceph-mon[110740]: pgmap v320: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:02:56.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:56 smithi027 ceph-mon[83054]: pgmap v320: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:02:58.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:02:58 smithi016 ceph-mon[110740]: pgmap v321: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:02:58.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:02:58 smithi027 ceph-mon[83054]: pgmap v321: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:00.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:00 smithi016 ceph-mon[110740]: pgmap v322: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:00.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:00 smithi027 ceph-mon[83054]: pgmap v322: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:01.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:01 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:03:01.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:01 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:03:02.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:02 smithi016 ceph-mon[110740]: pgmap v323: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:03:02.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:02 smithi027 ceph-mon[83054]: pgmap v323: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:03:04.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:04 smithi016 ceph-mon[110740]: pgmap v324: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:04.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:04 smithi027 ceph-mon[83054]: pgmap v324: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:06.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:06 smithi016 ceph-mon[110740]: pgmap v325: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:03:06.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:06 smithi027 ceph-mon[83054]: pgmap v325: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:03:08.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:08 smithi016 ceph-mon[110740]: pgmap v326: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:08.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:08 smithi027 ceph-mon[83054]: pgmap v326: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:10.388 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:10 smithi027 ceph-mon[83054]: pgmap v327: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:10.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:10 smithi016 ceph-mon[110740]: pgmap v327: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:12.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:12 smithi016 ceph-mon[110740]: pgmap v328: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:03:12.585 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:12 smithi027 ceph-mon[83054]: pgmap v328: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:03:14.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:14 smithi016 ceph-mon[110740]: pgmap v329: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:14.585 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:14 smithi027 ceph-mon[83054]: pgmap v329: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:16.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:16 smithi016 ceph-mon[110740]: pgmap v330: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:03:16.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:16 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:03:16.585 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:16 smithi027 ceph-mon[83054]: pgmap v330: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:03:16.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:16 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:03:18.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:18 smithi027 ceph-mon[83054]: pgmap v331: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:18.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:18 smithi016 ceph-mon[110740]: pgmap v331: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:20.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:20 smithi016 ceph-mon[110740]: pgmap v332: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:20.585 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:20 smithi027 ceph-mon[83054]: pgmap v332: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:22.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:22 smithi016 ceph-mon[110740]: pgmap v333: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:03:22.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:22 smithi027 ceph-mon[83054]: pgmap v333: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:03:24.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:24 smithi016 ceph-mon[110740]: pgmap v334: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:24.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:24 smithi027 ceph-mon[83054]: pgmap v334: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:26.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:26 smithi016 ceph-mon[110740]: pgmap v335: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:03:26.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:26 smithi027 ceph-mon[83054]: pgmap v335: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:03:28.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:28 smithi027 ceph-mon[83054]: pgmap v336: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:28.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:28 smithi016 ceph-mon[110740]: pgmap v336: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:30.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:30 smithi016 ceph-mon[110740]: pgmap v337: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:30.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:30 smithi027 ceph-mon[83054]: pgmap v337: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:31.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:31 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:03:31.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:31 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:03:32.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:32 smithi016 ceph-mon[110740]: pgmap v338: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:03:32.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:32 smithi027 ceph-mon[83054]: pgmap v338: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:03:34.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:34 smithi016 ceph-mon[110740]: pgmap v339: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:34.585 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:34 smithi027 ceph-mon[83054]: pgmap v339: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:36.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:36 smithi027 ceph-mon[83054]: pgmap v340: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:03:36.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:36 smithi016 ceph-mon[110740]: pgmap v340: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:03:38.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:38 smithi016 ceph-mon[110740]: pgmap v341: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:38.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:38 smithi027 ceph-mon[83054]: pgmap v341: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:40.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:40 smithi027 ceph-mon[83054]: pgmap v342: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:40.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:40 smithi016 ceph-mon[110740]: pgmap v342: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:42.585 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:42 smithi027 ceph-mon[83054]: pgmap v343: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:03:42.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:42 smithi016 ceph-mon[110740]: pgmap v343: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:03:44.585 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:44 smithi027 ceph-mon[83054]: pgmap v344: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:44.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:44 smithi016 ceph-mon[110740]: pgmap v344: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:46.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:46 smithi027 ceph-mon[83054]: pgmap v345: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:03:46.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:46 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:03:46.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:46 smithi016 ceph-mon[110740]: pgmap v345: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:03:46.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:46 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:03:48.459 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:48 smithi016 ceph-mon[110740]: pgmap v346: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:48.585 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:48 smithi027 ceph-mon[83054]: pgmap v346: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:49.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:49 smithi027 ceph-mon[83054]: pgmap v347: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:49.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:49 smithi016 ceph-mon[110740]: pgmap v347: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:51.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:51 smithi016 ceph-mon[110740]: pgmap v348: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:03:51.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:51 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T16:03:51.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:51 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T16:03:51.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:51 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T16:03:51.667 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:51 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T16:03:51.835 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:51 smithi027 ceph-mon[83054]: pgmap v348: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:03:51.835 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:51 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T16:03:51.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:51 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T16:03:51.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:51 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T16:03:51.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:51 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T16:03:53.585 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:53 smithi027 ceph-mon[83054]: pgmap v349: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:53.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:53 smithi016 ceph-mon[110740]: pgmap v349: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:55.666 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:55 smithi016 ceph-mon[110740]: pgmap v350: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:03:55.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:55 smithi027 ceph-mon[83054]: pgmap v350: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:03:58.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:57 smithi027 ceph-mon[83054]: pgmap v351: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:58.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:57 smithi016 ceph-mon[110740]: pgmap v351: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:03:59.917 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:03:59 smithi016 ceph-mon[110740]: pgmap v352: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:00.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:03:59 smithi027 ceph-mon[83054]: pgmap v352: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:01.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:00 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:04:01.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:00 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:04:02.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:01 smithi027 ceph-mon[83054]: pgmap v353: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:04:02.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:01 smithi016 ceph-mon[110740]: pgmap v353: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:04:04.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:03 smithi027 ceph-mon[83054]: pgmap v354: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:04.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:03 smithi016 ceph-mon[110740]: pgmap v354: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:06.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:05 smithi027 ceph-mon[83054]: pgmap v355: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:04:06.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:05 smithi016 ceph-mon[110740]: pgmap v355: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:04:08.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:07 smithi027 ceph-mon[83054]: pgmap v356: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:08.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:07 smithi016 ceph-mon[110740]: pgmap v356: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:10.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:09 smithi027 ceph-mon[83054]: pgmap v357: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:10.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:09 smithi016 ceph-mon[110740]: pgmap v357: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:12.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:11 smithi027 ceph-mon[83054]: pgmap v358: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:04:12.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:11 smithi016 ceph-mon[110740]: pgmap v358: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:04:13.836 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:13 smithi027 ceph-mon[83054]: pgmap v359: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:13.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:13 smithi016 ceph-mon[110740]: pgmap v359: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:16.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:15 smithi027 ceph-mon[83054]: pgmap v360: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:04:16.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:15 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:04:16.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:15 smithi016 ceph-mon[110740]: pgmap v360: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:04:16.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:15 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:04:18.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:17 smithi027 ceph-mon[83054]: pgmap v361: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:18.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:17 smithi016 ceph-mon[110740]: pgmap v361: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:20.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:19 smithi027 ceph-mon[83054]: pgmap v362: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:20.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:19 smithi016 ceph-mon[110740]: pgmap v362: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:22.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:21 smithi027 ceph-mon[83054]: pgmap v363: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:04:22.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:21 smithi016 ceph-mon[110740]: pgmap v363: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:04:24.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:23 smithi027 ceph-mon[83054]: pgmap v364: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:24.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:23 smithi016 ceph-mon[110740]: pgmap v364: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:26.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:25 smithi027 ceph-mon[83054]: pgmap v365: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:04:26.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:25 smithi016 ceph-mon[110740]: pgmap v365: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:04:28.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:27 smithi027 ceph-mon[83054]: pgmap v366: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:28.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:27 smithi016 ceph-mon[110740]: pgmap v366: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:30.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:29 smithi027 ceph-mon[83054]: pgmap v367: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:30.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:29 smithi016 ceph-mon[110740]: pgmap v367: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:31.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:30 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:04:31.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:04:32.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:31 smithi027 ceph-mon[83054]: pgmap v368: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:04:32.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:31 smithi016 ceph-mon[110740]: pgmap v368: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:04:34.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:33 smithi027 ceph-mon[83054]: pgmap v369: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:34.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:33 smithi016 ceph-mon[110740]: pgmap v369: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:36.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:35 smithi027 ceph-mon[83054]: pgmap v370: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:04:36.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:35 smithi016 ceph-mon[110740]: pgmap v370: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:04:38.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:37 smithi027 ceph-mon[83054]: pgmap v371: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:38.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:37 smithi016 ceph-mon[110740]: pgmap v371: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:40.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:39 smithi027 ceph-mon[83054]: pgmap v372: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:40.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:39 smithi016 ceph-mon[110740]: pgmap v372: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:42.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:41 smithi027 ceph-mon[83054]: pgmap v373: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:04:42.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:41 smithi016 ceph-mon[110740]: pgmap v373: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:04:44.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:43 smithi027 ceph-mon[83054]: pgmap v374: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:44.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:43 smithi016 ceph-mon[110740]: pgmap v374: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:46.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:45 smithi027 ceph-mon[83054]: pgmap v375: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:04:46.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:45 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:04:46.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:45 smithi016 ceph-mon[110740]: pgmap v375: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:04:46.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:45 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:04:48.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:47 smithi027 ceph-mon[83054]: pgmap v376: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:48.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:47 smithi016 ceph-mon[110740]: pgmap v376: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:50.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:49 smithi027 ceph-mon[83054]: pgmap v377: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:50.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:49 smithi016 ceph-mon[110740]: pgmap v377: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:52.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:51 smithi027 ceph-mon[83054]: pgmap v378: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:04:52.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:51 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T16:04:52.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:51 smithi016 ceph-mon[110740]: pgmap v378: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:04:52.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:51 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T16:04:53.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:52 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T16:04:53.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:52 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T16:04:53.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:52 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T16:04:53.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:52 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T16:04:53.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:52 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T16:04:53.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:52 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T16:04:54.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:53 smithi027 ceph-mon[83054]: pgmap v379: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:54.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:53 smithi016 ceph-mon[110740]: pgmap v379: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:56.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:55 smithi027 ceph-mon[83054]: pgmap v380: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:04:56.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:55 smithi016 ceph-mon[110740]: pgmap v380: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:04:58.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:57 smithi027 ceph-mon[83054]: pgmap v381: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:04:58.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:57 smithi016 ceph-mon[110740]: pgmap v381: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:00.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:04:59 smithi027 ceph-mon[83054]: pgmap v382: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:00.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:04:59 smithi016 ceph-mon[110740]: pgmap v382: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:01.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:00 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:05:01.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:00 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:05:02.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:01 smithi027 ceph-mon[83054]: pgmap v383: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:05:02.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:01 smithi016 ceph-mon[110740]: pgmap v383: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:05:04.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:03 smithi027 ceph-mon[83054]: pgmap v384: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:04.147 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:03 smithi016 ceph-mon[110740]: pgmap v384: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:06.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:05 smithi027 ceph-mon[83054]: pgmap v385: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:05:06.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:05 smithi016 ceph-mon[110740]: pgmap v385: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:05:08.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:07 smithi027 ceph-mon[83054]: pgmap v386: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:08.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:07 smithi016 ceph-mon[110740]: pgmap v386: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:10.086 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:09 smithi027 ceph-mon[83054]: pgmap v387: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:10.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:09 smithi016 ceph-mon[110740]: pgmap v387: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:12.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:11 smithi027 ceph-mon[83054]: pgmap v388: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:05:12.165 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:11 smithi016 ceph-mon[110740]: pgmap v388: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:05:14.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:13 smithi016 ceph-mon[110740]: pgmap v389: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:14.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:13 smithi027 ceph-mon[83054]: pgmap v389: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:16.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:15 smithi016 ceph-mon[110740]: pgmap v390: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:05:16.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:15 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:05:16.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:15 smithi027 ceph-mon[83054]: pgmap v390: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:05:16.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:15 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:05:18.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:17 smithi016 ceph-mon[110740]: pgmap v391: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:18.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:17 smithi027 ceph-mon[83054]: pgmap v391: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:20.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:19 smithi016 ceph-mon[110740]: pgmap v392: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:20.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:19 smithi027 ceph-mon[83054]: pgmap v392: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:22.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:21 smithi016 ceph-mon[110740]: pgmap v393: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:05:22.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:21 smithi027 ceph-mon[83054]: pgmap v393: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:05:24.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:23 smithi016 ceph-mon[110740]: pgmap v394: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:24.185 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:23 smithi027 ceph-mon[83054]: pgmap v394: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:26.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:25 smithi016 ceph-mon[110740]: pgmap v395: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:05:26.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:25 smithi027 ceph-mon[83054]: pgmap v395: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:05:28.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:27 smithi016 ceph-mon[110740]: pgmap v396: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:28.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:27 smithi027 ceph-mon[83054]: pgmap v396: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:30.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:29 smithi016 ceph-mon[110740]: pgmap v397: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:30.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:29 smithi027 ceph-mon[83054]: pgmap v397: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:31.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:30 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:05:31.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:30 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:05:32.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:31 smithi016 ceph-mon[110740]: pgmap v398: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:05:32.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:31 smithi027 ceph-mon[83054]: pgmap v398: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:05:34.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:33 smithi016 ceph-mon[110740]: pgmap v399: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:34.283 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:33 smithi027 ceph-mon[83054]: pgmap v399: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:36.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:35 smithi016 ceph-mon[110740]: pgmap v400: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:05:36.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:35 smithi027 ceph-mon[83054]: pgmap v400: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:05:38.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:37 smithi016 ceph-mon[110740]: pgmap v401: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:38.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:37 smithi027 ceph-mon[83054]: pgmap v401: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:40.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:39 smithi016 ceph-mon[110740]: pgmap v402: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:40.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:39 smithi027 ceph-mon[83054]: pgmap v402: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:42.138 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:41 smithi027 ceph-mon[83054]: pgmap v403: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:05:42.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:41 smithi016 ceph-mon[110740]: pgmap v403: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:05:44.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:43 smithi016 ceph-mon[110740]: pgmap v404: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:44.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:43 smithi027 ceph-mon[83054]: pgmap v404: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:46.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:45 smithi016 ceph-mon[110740]: pgmap v405: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:05:46.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:45 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:05:46.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:45 smithi027 ceph-mon[83054]: pgmap v405: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:05:46.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:45 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:05:48.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:47 smithi016 ceph-mon[110740]: pgmap v406: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:48.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:47 smithi027 ceph-mon[83054]: pgmap v406: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:49.916 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:49 smithi016 ceph-mon[110740]: pgmap v407: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:50.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:49 smithi027 ceph-mon[83054]: pgmap v407: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:52.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:51 smithi016 ceph-mon[110740]: pgmap v408: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:05:52.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:51 smithi027 ceph-mon[83054]: pgmap v408: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:05:53.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:52 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T16:05:53.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:52 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T16:05:54.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:53 smithi016 ceph-mon[110740]: pgmap v409: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:54.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:53 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T16:05:54.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:53 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T16:05:54.167 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:53 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T16:05:54.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:53 smithi027 ceph-mon[83054]: pgmap v409: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:54.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:53 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T16:05:54.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:53 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T16:05:54.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:53 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' 2024-04-22T16:05:56.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:55 smithi016 ceph-mon[110740]: pgmap v410: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:05:56.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:55 smithi027 ceph-mon[83054]: pgmap v410: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:05:58.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:57 smithi016 ceph-mon[110740]: pgmap v411: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:05:58.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:57 smithi027 ceph-mon[83054]: pgmap v411: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:06:00.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:05:59 smithi016 ceph-mon[110740]: pgmap v412: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:06:00.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:05:59 smithi027 ceph-mon[83054]: pgmap v412: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:06:01.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:00 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:06:01.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:06:00 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:06:02.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:01 smithi027 ceph-mon[83054]: pgmap v413: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:06:02.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:06:01 smithi016 ceph-mon[110740]: pgmap v413: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:06:04.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:03 smithi027 ceph-mon[83054]: pgmap v414: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:06:04.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:06:03 smithi016 ceph-mon[110740]: pgmap v414: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:06:06.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:05 smithi027 ceph-mon[83054]: pgmap v415: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:06:06.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:06:05 smithi016 ceph-mon[110740]: pgmap v415: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:06:08.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:07 smithi027 ceph-mon[83054]: pgmap v416: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:06:08.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:06:07 smithi016 ceph-mon[110740]: pgmap v416: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:06:10.166 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:06:09 smithi016 ceph-mon[110740]: pgmap v417: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:06:10.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:09 smithi027 ceph-mon[83054]: pgmap v417: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:06:12.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:11 smithi027 ceph-mon[83054]: pgmap v418: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:06:12.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:06:11 smithi016 ceph-mon[110740]: pgmap v418: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:06:14.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:13 smithi027 ceph-mon[83054]: pgmap v419: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:06:14.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:06:13 smithi016 ceph-mon[110740]: pgmap v419: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:06:16.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:15 smithi027 ceph-mon[83054]: pgmap v420: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:06:16.336 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:15 smithi027 ceph-mon[83054]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:06:16.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:06:15 smithi016 ceph-mon[110740]: pgmap v420: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:06:16.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:06:15 smithi016 ceph-mon[110740]: from='mgr.34104 172.21.15.16:0/2771839849' entity='mgr.smithi016.rnsnnr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T16:06:18.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:17 smithi027 ceph-mon[83054]: pgmap v421: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:06:18.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:06:17 smithi016 ceph-mon[110740]: pgmap v421: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:06:20.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:19 smithi027 ceph-mon[83054]: pgmap v422: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:06:20.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:06:19 smithi016 ceph-mon[110740]: pgmap v422: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:06:22.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:21 smithi027 ceph-mon[83054]: pgmap v423: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:06:22.415 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:06:21 smithi016 ceph-mon[110740]: pgmap v423: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:06:24.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:23 smithi027 ceph-mon[83054]: pgmap v424: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:06:24.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:06:23 smithi016 ceph-mon[110740]: pgmap v424: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:06:26.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:25 smithi027 ceph-mon[83054]: pgmap v425: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:06:26.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:06:25 smithi016 ceph-mon[110740]: pgmap v425: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T16:06:28.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:27 smithi027 ceph-mon[83054]: pgmap v426: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:06:28.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:06:27 smithi016 ceph-mon[110740]: pgmap v426: 65 pgs: 65 active+clean; 250 MiB data, 977 MiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-22T16:06:29.319 ERROR:tasks.cephfs.fuse_mount:process failed to terminate after unmount. This probably indicates a bug within ceph-fuse. 2024-04-22T16:06:29.320 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-22T16:06:29.321 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-22T16:06:29.333 INFO:tasks.cephadm:Teardown begin 2024-04-22T16:06:29.333 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/qa/tasks/cephadm.py", line 2246, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-22T16:06:29.334 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T16:06:29.367 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T16:06:29.400 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-22T16:06:29.400 DEBUG:teuthology.orchestra.run.smithi016:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T16:06:29.426 DEBUG:teuthology.orchestra.run.smithi027:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T16:06:29.458 INFO:tasks.cephadm:Stopping all daemons... 2024-04-22T16:06:29.458 INFO:tasks.cephadm.mon.smithi016:Stopping mon.smithi016... 2024-04-22T16:06:29.458 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-67ab3a80-00be-11ef-bc93-c7b262605968@mon.smithi016 2024-04-22T16:06:29.665 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:06:29 smithi016 systemd[1]: Stopping Ceph mon.smithi016 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T16:06:30.126 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:06:29 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016[110736]: 2024-04-22T16:06:29.762+0000 7f292b7a2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi016 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T16:06:30.126 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:06:29 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016[110736]: 2024-04-22T16:06:29.762+0000 7f292b7a2700 -1 mon.smithi016@0(leader) e3 *** Got Signal Terminated *** 2024-04-22T16:06:30.126 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:06:29 smithi016 podman[156661]: 2024-04-22 16:06:29.783947388 +0000 UTC m=+0.119861640 container died 4ae529a67806e75f9205eb4a9422ee24002b45a3de36f4e17bff538a0d94e9d0 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49) 2024-04-22T16:06:30.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:06:30 smithi016 podman[156661]: 2024-04-22 16:06:30.346940798 +0000 UTC m=+0.682855039 container cleanup 4ae529a67806e75f9205eb4a9422ee24002b45a3de36f4e17bff538a0d94e9d0 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T16:06:30.416 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Apr 22 16:06:30 smithi016 bash[156661]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi016 2024-04-22T16:06:30.759 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67ab3a80-00be-11ef-bc93-c7b262605968@mon.smithi016.service' 2024-04-22T16:06:30.806 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T16:06:30.806 INFO:tasks.cephadm.mon.smithi016:Stopped mon.smithi016 2024-04-22T16:06:30.806 INFO:tasks.cephadm.mon.smithi027:Stopping mon.smithi027... 2024-04-22T16:06:30.806 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-67ab3a80-00be-11ef-bc93-c7b262605968@mon.smithi027 2024-04-22T16:06:31.085 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:30 smithi027 systemd[1]: Stopping Ceph mon.smithi027 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T16:06:31.585 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:31 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi027[83050]: 2024-04-22T16:06:31.105+0000 7f5e9d80c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi027 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T16:06:31.585 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:31 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi027[83050]: 2024-04-22T16:06:31.105+0000 7f5e9d80c700 -1 mon.smithi027@1(peon) e3 *** Got Signal Terminated *** 2024-04-22T16:06:31.585 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:31 smithi027 podman[120214]: 2024-04-22 16:06:31.269646317 +0000 UTC m=+0.258538613 container stop f29fcb888801b2457b3c08120cd69643eb68a4b51ad1349541c63382ba1a1ee1 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi027, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-22T16:06:31.586 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:31 smithi027 podman[120214]: 2024-04-22 16:06:31.295253699 +0000 UTC m=+0.284145991 container died f29fcb888801b2457b3c08120cd69643eb68a4b51ad1349541c63382ba1a1ee1 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi027, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-22T16:06:32.068 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:31 smithi027 podman[120214]: 2024-04-22 16:06:31.88095681 +0000 UTC m=+0.869849101 container cleanup f29fcb888801b2457b3c08120cd69643eb68a4b51ad1349541c63382ba1a1ee1 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi027, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416) 2024-04-22T16:06:32.068 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:31 smithi027 bash[120214]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi027 2024-04-22T16:06:32.312 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67ab3a80-00be-11ef-bc93-c7b262605968@mon.smithi027.service' 2024-04-22T16:06:32.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:32 smithi027 podman[120228]: 2024-04-22 16:06:32.067410918 +0000 UTC m=+0.788099310 container remove f29fcb888801b2457b3c08120cd69643eb68a4b51ad1349541c63382ba1a1ee1 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-mon-smithi027, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T16:06:32.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:32 smithi027 systemd[1]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968@mon.smithi027.service: Deactivated successfully. 2024-04-22T16:06:32.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:32 smithi027 systemd[1]: Stopped Ceph mon.smithi027 for 67ab3a80-00be-11ef-bc93-c7b262605968. 2024-04-22T16:06:32.335 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 22 16:06:32 smithi027 systemd[1]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968@mon.smithi027.service: Consumed 6.888s CPU time. 2024-04-22T16:06:32.363 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T16:06:32.363 INFO:tasks.cephadm.mon.smithi027:Stopped mon.smithi027 2024-04-22T16:06:32.363 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-22T16:06:32.363 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.0 2024-04-22T16:06:32.666 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 16:06:32 smithi016 systemd[1]: Stopping Ceph osd.0 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T16:06:33.166 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 16:06:32 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0[122090]: 2024-04-22T16:06:32.676+0000 7f463b07b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T16:06:33.166 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 16:06:32 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0[122090]: 2024-04-22T16:06:32.676+0000 7f463b07b700 -1 osd.0 84 *** Got signal Terminated *** 2024-04-22T16:06:33.166 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 16:06:32 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0[122090]: 2024-04-22T16:06:32.676+0000 7f463b07b700 -1 osd.0 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T16:06:38.416 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 16:06:38 smithi016 podman[156794]: 2024-04-22 16:06:38.159465437 +0000 UTC m=+5.594716741 container died 79217eb383a2dfd422853df80bae709b87ef450c586b17f3bdf8bbbf337fc00f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T16:06:38.916 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 16:06:38 smithi016 podman[156794]: 2024-04-22 16:06:38.861004653 +0000 UTC m=+6.296255942 container cleanup 79217eb383a2dfd422853df80bae709b87ef450c586b17f3bdf8bbbf337fc00f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-22T16:06:38.916 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 16:06:38 smithi016 bash[156794]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0 2024-04-22T16:06:39.411 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 16:06:39 smithi016 podman[156812]: 2024-04-22 16:06:39.079211575 +0000 UTC m=+0.916380420 container remove 79217eb383a2dfd422853df80bae709b87ef450c586b17f3bdf8bbbf337fc00f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-22T16:06:39.666 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 16:06:39 smithi016 podman[156908]: 2024-04-22 16:06:39.312875702 +0000 UTC m=+0.018457325 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T16:06:39.666 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 16:06:39 smithi016 podman[156908]: 2024-04-22 16:06:39.413520388 +0000 UTC m=+0.119101997 container create 23bdc3e2acf501f54bd6b11db921d3922e312b931a370d1ec2d2c5e2fa0673a8 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-22T16:06:39.666 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 16:06:39 smithi016 podman[156908]: 2024-04-22 16:06:39.641268849 +0000 UTC m=+0.346850458 container init 23bdc3e2acf501f54bd6b11db921d3922e312b931a370d1ec2d2c5e2fa0673a8 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-22T16:06:39.667 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 16:06:39 smithi016 podman[156908]: 2024-04-22 16:06:39.644688261 +0000 UTC m=+0.350269870 container start 23bdc3e2acf501f54bd6b11db921d3922e312b931a370d1ec2d2c5e2fa0673a8 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-22T16:06:39.951 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 16:06:39 smithi016 podman[156908]: 2024-04-22 16:06:39.725110337 +0000 UTC m=+0.430691974 container attach 23bdc3e2acf501f54bd6b11db921d3922e312b931a370d1ec2d2c5e2fa0673a8 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T16:06:40.229 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 22 16:06:39 smithi016 podman[156908]: 2024-04-22 16:06:39.94963183 +0000 UTC m=+0.655213440 container died 23bdc3e2acf501f54bd6b11db921d3922e312b931a370d1ec2d2c5e2fa0673a8 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T16:06:40.546 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.0.service' 2024-04-22T16:06:40.588 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T16:06:40.588 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-22T16:06:40.588 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-22T16:06:40.589 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.1 2024-04-22T16:06:40.817 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 16:06:40 smithi016 systemd[1]: Stopping Ceph osd.1 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T16:06:41.166 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 16:06:40 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1[127937]: 2024-04-22T16:06:40.948+0000 7ff00ca6b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T16:06:41.166 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 16:06:40 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1[127937]: 2024-04-22T16:06:40.948+0000 7ff00ca6b700 -1 osd.1 84 *** Got signal Terminated *** 2024-04-22T16:06:41.166 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 16:06:40 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1[127937]: 2024-04-22T16:06:40.948+0000 7ff00ca6b700 -1 osd.1 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T16:06:46.637 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 16:06:46 smithi016 podman[157014]: 2024-04-22 16:06:46.371048812 +0000 UTC m=+5.548520990 container died ec1a45b3c53ece2baa204840f17d0fdb536387d4f285cdc7bc7dbb3560775f9f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-22T16:06:46.916 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 16:06:46 smithi016 podman[157014]: 2024-04-22 16:06:46.724903247 +0000 UTC m=+5.902375424 container cleanup ec1a45b3c53ece2baa204840f17d0fdb536387d4f285cdc7bc7dbb3560775f9f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T16:06:46.916 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 16:06:46 smithi016 bash[157014]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1 2024-04-22T16:06:47.312 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 16:06:46 smithi016 podman[157043]: 2024-04-22 16:06:46.984709546 +0000 UTC m=+0.614926723 container remove ec1a45b3c53ece2baa204840f17d0fdb536387d4f285cdc7bc7dbb3560775f9f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T16:06:47.598 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 16:06:47 smithi016 podman[157137]: 2024-04-22 16:06:47.310731207 +0000 UTC m=+0.114372502 container create 845fee2b1222ef8a80c759af1a22dfd824637bad595d15dc9b29ac01a4485ebf (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-deactivate, ceph=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T16:06:47.598 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 16:06:47 smithi016 podman[157137]: 2024-04-22 16:06:47.214967909 +0000 UTC m=+0.018609196 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T16:06:47.598 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 16:06:47 smithi016 podman[157137]: 2024-04-22 16:06:47.521561648 +0000 UTC m=+0.325202905 container init 845fee2b1222ef8a80c759af1a22dfd824637bad595d15dc9b29ac01a4485ebf (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_BRANCH=HEAD) 2024-04-22T16:06:47.598 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 16:06:47 smithi016 podman[157137]: 2024-04-22 16:06:47.524776922 +0000 UTC m=+0.328418185 container start 845fee2b1222ef8a80c759af1a22dfd824637bad595d15dc9b29ac01a4485ebf (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True) 2024-04-22T16:06:47.598 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 16:06:47 smithi016 podman[157137]: 2024-04-22 16:06:47.596630426 +0000 UTC m=+0.400271687 container attach 845fee2b1222ef8a80c759af1a22dfd824637bad595d15dc9b29ac01a4485ebf (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-22T16:06:47.916 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 22 16:06:47 smithi016 podman[157137]: 2024-04-22 16:06:47.833443743 +0000 UTC m=+0.637085014 container died 845fee2b1222ef8a80c759af1a22dfd824637bad595d15dc9b29ac01a4485ebf (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T16:06:48.457 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.1.service' 2024-04-22T16:06:48.501 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T16:06:48.501 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-22T16:06:48.501 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-22T16:06:48.501 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.2 2024-04-22T16:06:48.830 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 16:06:48 smithi016 systemd[1]: Stopping Ceph osd.2 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T16:06:49.166 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 16:06:48 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2[133505]: 2024-04-22T16:06:48.828+0000 7f278b983700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T16:06:49.166 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 16:06:48 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2[133505]: 2024-04-22T16:06:48.828+0000 7f278b983700 -1 osd.2 84 *** Got signal Terminated *** 2024-04-22T16:06:49.166 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 16:06:48 smithi016 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2[133505]: 2024-04-22T16:06:48.828+0000 7f278b983700 -1 osd.2 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T16:06:54.582 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 16:06:54 smithi016 podman[157244]: 2024-04-22 16:06:54.266443017 +0000 UTC m=+5.530267488 container died 48bb459fd021a9aa416678c2047ee8afc67f7d599ecc8677256372aebf7dad8b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-22T16:06:54.915 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 16:06:54 smithi016 podman[157244]: 2024-04-22 16:06:54.662395811 +0000 UTC m=+5.926220278 container cleanup 48bb459fd021a9aa416678c2047ee8afc67f7d599ecc8677256372aebf7dad8b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-22T16:06:54.916 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 16:06:54 smithi016 bash[157244]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2 2024-04-22T16:06:55.166 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 16:06:54 smithi016 podman[157261]: 2024-04-22 16:06:54.913673457 +0000 UTC m=+0.640800125 container remove 48bb459fd021a9aa416678c2047ee8afc67f7d599ecc8677256372aebf7dad8b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T16:06:55.561 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 16:06:55 smithi016 podman[157354]: 2024-04-22 16:06:55.146246454 +0000 UTC m=+0.018796177 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T16:06:55.561 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 16:06:55 smithi016 podman[157354]: 2024-04-22 16:06:55.248029646 +0000 UTC m=+0.120579360 container create 213f4967cf5fa2c1da69199c1cd89cf617a85df79e36a23e297c684e9a62923e (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49) 2024-04-22T16:06:55.561 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 16:06:55 smithi016 podman[157354]: 2024-04-22 16:06:55.460078196 +0000 UTC m=+0.332627907 container init 213f4967cf5fa2c1da69199c1cd89cf617a85df79e36a23e297c684e9a62923e (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-22T16:06:55.561 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 16:06:55 smithi016 podman[157354]: 2024-04-22 16:06:55.463607665 +0000 UTC m=+0.336157381 container start 213f4967cf5fa2c1da69199c1cd89cf617a85df79e36a23e297c684e9a62923e (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-deactivate, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T16:06:55.561 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 16:06:55 smithi016 podman[157354]: 2024-04-22 16:06:55.559334973 +0000 UTC m=+0.431884682 container attach 213f4967cf5fa2c1da69199c1cd89cf617a85df79e36a23e297c684e9a62923e (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T16:06:55.916 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 22 16:06:55 smithi016 podman[157354]: 2024-04-22 16:06:55.772181737 +0000 UTC m=+0.644731446 container died 213f4967cf5fa2c1da69199c1cd89cf617a85df79e36a23e297c684e9a62923e (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_CLEAN=True) 2024-04-22T16:06:56.372 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.2.service' 2024-04-22T16:06:56.414 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T16:06:56.414 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-22T16:06:56.415 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-22T16:06:56.415 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.3 2024-04-22T16:06:56.727 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 16:06:56 smithi027 systemd[1]: Stopping Ceph osd.3 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T16:06:57.085 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 16:06:56 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3[92366]: 2024-04-22T16:06:56.726+0000 7f745af25700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T16:06:57.085 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 16:06:56 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3[92366]: 2024-04-22T16:06:56.726+0000 7f745af25700 -1 osd.3 84 *** Got signal Terminated *** 2024-04-22T16:06:57.085 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 16:06:56 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3[92366]: 2024-04-22T16:06:56.726+0000 7f745af25700 -1 osd.3 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T16:07:00.835 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:00 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[97573]: 2024-04-22T16:07:00.567+0000 7f817b2fa700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-22T16:06:37.475078+0000 front 2024-04-22T16:06:37.474997+0000 (oldest deadline 2024-04-22T16:06:59.675267+0000) 2024-04-22T16:07:01.835 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:01 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[97573]: 2024-04-22T16:07:01.553+0000 7f817b2fa700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-22T16:06:37.475078+0000 front 2024-04-22T16:06:37.474997+0000 (oldest deadline 2024-04-22T16:06:59.675267+0000) 2024-04-22T16:07:02.429 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 16:07:02 smithi027 podman[120348]: 2024-04-22 16:07:02.17325401 +0000 UTC m=+5.558427523 container died 81034bf22d3f0e38ac4e75e6e3c51825cb23b87f19f420a7ef5279197c33250b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T16:07:02.728 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:02 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[97573]: 2024-04-22T16:07:02.589+0000 7f817b2fa700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-22T16:06:37.475078+0000 front 2024-04-22T16:06:37.474997+0000 (oldest deadline 2024-04-22T16:06:59.675267+0000) 2024-04-22T16:07:02.728 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 16:07:02 smithi027 podman[120348]: 2024-04-22 16:07:02.526537781 +0000 UTC m=+5.911711284 container cleanup 81034bf22d3f0e38ac4e75e6e3c51825cb23b87f19f420a7ef5279197c33250b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-22T16:07:02.729 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 16:07:02 smithi027 bash[120348]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3 2024-04-22T16:07:03.058 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 16:07:02 smithi027 podman[120366]: 2024-04-22 16:07:02.727524193 +0000 UTC m=+0.548164858 container remove 81034bf22d3f0e38ac4e75e6e3c51825cb23b87f19f420a7ef5279197c33250b (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-22T16:07:03.335 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:03 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:03.057+0000 7f94da345700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-22T16:06:36.674238+0000 front 2024-04-22T16:06:36.674437+0000 (oldest deadline 2024-04-22T16:07:02.574241+0000) 2024-04-22T16:07:03.335 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 16:07:03 smithi027 podman[120458]: 2024-04-22 16:07:03.073327342 +0000 UTC m=+0.018533440 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T16:07:03.335 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 16:07:03 smithi027 podman[120458]: 2024-04-22 16:07:03.205661652 +0000 UTC m=+0.150867781 container create 5f31da53571ea7fbfd64ed3385345f6c2053de69f6d45e5e868dc316b69d439f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T16:07:03.632 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 16:07:03 smithi027 podman[120458]: 2024-04-22 16:07:03.46669053 +0000 UTC m=+0.411896615 container init 5f31da53571ea7fbfd64ed3385345f6c2053de69f6d45e5e868dc316b69d439f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T16:07:03.632 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 16:07:03 smithi027 podman[120458]: 2024-04-22 16:07:03.469943999 +0000 UTC m=+0.415150084 container start 5f31da53571ea7fbfd64ed3385345f6c2053de69f6d45e5e868dc316b69d439f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T16:07:03.632 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 16:07:03 smithi027 podman[120458]: 2024-04-22 16:07:03.566737895 +0000 UTC m=+0.511943991 container attach 5f31da53571ea7fbfd64ed3385345f6c2053de69f6d45e5e868dc316b69d439f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5) 2024-04-22T16:07:04.070 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:03 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[97573]: 2024-04-22T16:07:03.630+0000 7f817b2fa700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-22T16:06:37.475078+0000 front 2024-04-22T16:06:37.474997+0000 (oldest deadline 2024-04-22T16:06:59.675267+0000) 2024-04-22T16:07:04.070 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 22 16:07:03 smithi027 podman[120458]: 2024-04-22 16:07:03.790610206 +0000 UTC m=+0.735816299 container died 5f31da53571ea7fbfd64ed3385345f6c2053de69f6d45e5e868dc316b69d439f (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-22T16:07:04.335 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:04 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:04.068+0000 7f94da345700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-22T16:06:36.674238+0000 front 2024-04-22T16:06:36.674437+0000 (oldest deadline 2024-04-22T16:07:02.574241+0000) 2024-04-22T16:07:04.364 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.3.service' 2024-04-22T16:07:04.403 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T16:07:04.403 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-22T16:07:04.403 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-22T16:07:04.403 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.4 2024-04-22T16:07:04.639 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:04 smithi027 systemd[1]: Stopping Ceph osd.4 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T16:07:05.085 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:04 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[97573]: 2024-04-22T16:07:04.642+0000 7f817b2fa700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-22T16:06:37.475078+0000 front 2024-04-22T16:06:37.474997+0000 (oldest deadline 2024-04-22T16:06:59.675267+0000) 2024-04-22T16:07:05.086 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:04 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[97573]: 2024-04-22T16:07:04.748+0000 7f8182d0e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T16:07:05.086 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:04 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[97573]: 2024-04-22T16:07:04.748+0000 7f8182d0e700 -1 osd.4 84 *** Got signal Terminated *** 2024-04-22T16:07:05.086 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:04 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[97573]: 2024-04-22T16:07:04.748+0000 7f8182d0e700 -1 osd.4 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T16:07:05.585 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:05 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:05.095+0000 7f94da345700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-22T16:06:36.674238+0000 front 2024-04-22T16:06:36.674437+0000 (oldest deadline 2024-04-22T16:07:02.574241+0000) 2024-04-22T16:07:06.085 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:05 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[97573]: 2024-04-22T16:07:05.638+0000 7f817b2fa700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-22T16:06:37.475078+0000 front 2024-04-22T16:06:37.474997+0000 (oldest deadline 2024-04-22T16:06:59.675267+0000) 2024-04-22T16:07:06.585 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:06 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:06.098+0000 7f94da345700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-22T16:06:36.674238+0000 front 2024-04-22T16:06:36.674437+0000 (oldest deadline 2024-04-22T16:07:02.574241+0000) 2024-04-22T16:07:07.085 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:06 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[97573]: 2024-04-22T16:07:06.596+0000 7f817b2fa700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-22T16:06:37.475078+0000 front 2024-04-22T16:06:37.474997+0000 (oldest deadline 2024-04-22T16:06:59.675267+0000) 2024-04-22T16:07:07.585 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:07 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:07.105+0000 7f94da345700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-22T16:06:36.674238+0000 front 2024-04-22T16:06:36.674437+0000 (oldest deadline 2024-04-22T16:07:02.574241+0000) 2024-04-22T16:07:08.085 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:07 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[97573]: 2024-04-22T16:07:07.603+0000 7f817b2fa700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-22T16:06:37.475078+0000 front 2024-04-22T16:06:37.474997+0000 (oldest deadline 2024-04-22T16:06:59.675267+0000) 2024-04-22T16:07:08.085 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:07 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[97573]: 2024-04-22T16:07:07.603+0000 7f817b2fa700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-22T16:06:40.776146+0000 front 2024-04-22T16:06:40.776226+0000 (oldest deadline 2024-04-22T16:07:06.676185+0000) 2024-04-22T16:07:08.585 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:08 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:08.135+0000 7f94da345700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-22T16:06:36.674238+0000 front 2024-04-22T16:06:36.674437+0000 (oldest deadline 2024-04-22T16:07:02.574241+0000) 2024-04-22T16:07:08.585 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:08 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:08.135+0000 7f94da345700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-22T16:06:42.574702+0000 front 2024-04-22T16:06:42.574771+0000 (oldest deadline 2024-04-22T16:07:07.274543+0000) 2024-04-22T16:07:09.085 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:08 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[97573]: 2024-04-22T16:07:08.626+0000 7f817b2fa700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-22T16:06:37.475078+0000 front 2024-04-22T16:06:37.474997+0000 (oldest deadline 2024-04-22T16:06:59.675267+0000) 2024-04-22T16:07:09.086 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:08 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[97573]: 2024-04-22T16:07:08.626+0000 7f817b2fa700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-22T16:06:40.776146+0000 front 2024-04-22T16:06:40.776226+0000 (oldest deadline 2024-04-22T16:07:06.676185+0000) 2024-04-22T16:07:09.585 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:09 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:09.125+0000 7f94da345700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-22T16:06:36.674238+0000 front 2024-04-22T16:06:36.674437+0000 (oldest deadline 2024-04-22T16:07:02.574241+0000) 2024-04-22T16:07:09.586 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:09 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:09.125+0000 7f94da345700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-22T16:06:42.574702+0000 front 2024-04-22T16:06:42.574771+0000 (oldest deadline 2024-04-22T16:07:07.274543+0000) 2024-04-22T16:07:10.086 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:09 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[97573]: 2024-04-22T16:07:09.620+0000 7f817b2fa700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-22T16:06:37.475078+0000 front 2024-04-22T16:06:37.474997+0000 (oldest deadline 2024-04-22T16:06:59.675267+0000) 2024-04-22T16:07:10.086 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:09 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4[97573]: 2024-04-22T16:07:09.620+0000 7f817b2fa700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-22T16:06:40.776146+0000 front 2024-04-22T16:06:40.776226+0000 (oldest deadline 2024-04-22T16:07:06.676185+0000) 2024-04-22T16:07:10.425 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:10 smithi027 podman[120565]: 2024-04-22 16:07:10.132245924 +0000 UTC m=+5.494464260 container died 93b660e5ecb615a1808923519ebfb6684cd16c14ff1969e5be5b346d78b32bab (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-22T16:07:10.426 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:10 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:10.091+0000 7f94da345700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-22T16:06:36.674238+0000 front 2024-04-22T16:06:36.674437+0000 (oldest deadline 2024-04-22T16:07:02.574241+0000) 2024-04-22T16:07:10.426 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:10 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:10.091+0000 7f94da345700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-22T16:06:42.574702+0000 front 2024-04-22T16:06:42.574771+0000 (oldest deadline 2024-04-22T16:07:07.274543+0000) 2024-04-22T16:07:10.751 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:10 smithi027 podman[120565]: 2024-04-22 16:07:10.540181072 +0000 UTC m=+5.902399404 container cleanup 93b660e5ecb615a1808923519ebfb6684cd16c14ff1969e5be5b346d78b32bab (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-22T16:07:10.751 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:10 smithi027 bash[120565]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4 2024-04-22T16:07:11.085 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:10 smithi027 podman[120593]: 2024-04-22 16:07:10.749582404 +0000 UTC m=+0.617910924 container remove 93b660e5ecb615a1808923519ebfb6684cd16c14ff1969e5be5b346d78b32bab (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-22T16:07:11.362 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:11 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:11.086+0000 7f94da345700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-22T16:06:36.674238+0000 front 2024-04-22T16:06:36.674437+0000 (oldest deadline 2024-04-22T16:07:02.574241+0000) 2024-04-22T16:07:11.363 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:11 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:11.086+0000 7f94da345700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-22T16:06:42.574702+0000 front 2024-04-22T16:06:42.574771+0000 (oldest deadline 2024-04-22T16:07:07.274543+0000) 2024-04-22T16:07:11.363 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:11 smithi027 podman[120684]: 2024-04-22 16:07:11.080614733 +0000 UTC m=+0.018847391 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T16:07:11.363 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:11 smithi027 podman[120684]: 2024-04-22 16:07:11.183541558 +0000 UTC m=+0.121774196 container create 455c0544616bdadc504170889f1f7b218014f5cf0c458a0ef976367a9fcc921a (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True) 2024-04-22T16:07:11.363 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:11 smithi027 podman[120684]: 2024-04-22 16:07:11.3615464 +0000 UTC m=+0.299779039 container init 455c0544616bdadc504170889f1f7b218014f5cf0c458a0ef976367a9fcc921a (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-22T16:07:11.682 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:11 smithi027 podman[120684]: 2024-04-22 16:07:11.365080812 +0000 UTC m=+0.303313460 container start 455c0544616bdadc504170889f1f7b218014f5cf0c458a0ef976367a9fcc921a (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T16:07:11.682 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:11 smithi027 podman[120684]: 2024-04-22 16:07:11.45256122 +0000 UTC m=+0.390793867 container attach 455c0544616bdadc504170889f1f7b218014f5cf0c458a0ef976367a9fcc921a (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T16:07:12.006 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 22 16:07:11 smithi027 podman[120684]: 2024-04-22 16:07:11.681063031 +0000 UTC m=+0.619295670 container died 455c0544616bdadc504170889f1f7b218014f5cf0c458a0ef976367a9fcc921a (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-22T16:07:12.257 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:12 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:12.068+0000 7f94da345700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-22T16:06:36.674238+0000 front 2024-04-22T16:06:36.674437+0000 (oldest deadline 2024-04-22T16:07:02.574241+0000) 2024-04-22T16:07:12.257 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:12 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:12.068+0000 7f94da345700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-22T16:06:42.574702+0000 front 2024-04-22T16:06:42.574771+0000 (oldest deadline 2024-04-22T16:07:07.274543+0000) 2024-04-22T16:07:12.278 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.4.service' 2024-04-22T16:07:12.317 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T16:07:12.318 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-22T16:07:12.318 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-22T16:07:12.318 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.5 2024-04-22T16:07:12.585 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:12 smithi027 systemd[1]: Stopping Ceph osd.5 for 67ab3a80-00be-11ef-bc93-c7b262605968... 2024-04-22T16:07:13.054 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:12 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:12.633+0000 7f94e1d59700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T16:07:13.054 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:12 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:12.633+0000 7f94e1d59700 -1 osd.5 84 *** Got signal Terminated *** 2024-04-22T16:07:13.054 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:12 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:12.633+0000 7f94e1d59700 -1 osd.5 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T16:07:13.054 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:13 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:13.051+0000 7f94da345700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-22T16:06:36.674238+0000 front 2024-04-22T16:06:36.674437+0000 (oldest deadline 2024-04-22T16:07:02.574241+0000) 2024-04-22T16:07:13.054 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:13 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:13.051+0000 7f94da345700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-22T16:06:42.574702+0000 front 2024-04-22T16:06:42.574771+0000 (oldest deadline 2024-04-22T16:07:07.274543+0000) 2024-04-22T16:07:14.335 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:14 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:14.039+0000 7f94da345700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-22T16:06:36.674238+0000 front 2024-04-22T16:06:36.674437+0000 (oldest deadline 2024-04-22T16:07:02.574241+0000) 2024-04-22T16:07:14.335 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:14 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:14.039+0000 7f94da345700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-22T16:06:42.574702+0000 front 2024-04-22T16:06:42.574771+0000 (oldest deadline 2024-04-22T16:07:07.274543+0000) 2024-04-22T16:07:15.335 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:15 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:15.029+0000 7f94da345700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-22T16:06:36.674238+0000 front 2024-04-22T16:06:36.674437+0000 (oldest deadline 2024-04-22T16:07:02.574241+0000) 2024-04-22T16:07:15.336 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:15 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:15.029+0000 7f94da345700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-22T16:06:42.574702+0000 front 2024-04-22T16:06:42.574771+0000 (oldest deadline 2024-04-22T16:07:07.274543+0000) 2024-04-22T16:07:16.335 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:16 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:16.051+0000 7f94da345700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-22T16:06:36.674238+0000 front 2024-04-22T16:06:36.674437+0000 (oldest deadline 2024-04-22T16:07:02.574241+0000) 2024-04-22T16:07:16.336 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:16 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:16.051+0000 7f94da345700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-22T16:06:42.574702+0000 front 2024-04-22T16:06:42.574771+0000 (oldest deadline 2024-04-22T16:07:07.274543+0000) 2024-04-22T16:07:17.335 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:17 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:17.025+0000 7f94da345700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-22T16:06:36.674238+0000 front 2024-04-22T16:06:36.674437+0000 (oldest deadline 2024-04-22T16:07:02.574241+0000) 2024-04-22T16:07:17.336 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:17 smithi027 ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5[102948]: 2024-04-22T16:07:17.025+0000 7f94da345700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-22T16:06:42.574702+0000 front 2024-04-22T16:06:42.574771+0000 (oldest deadline 2024-04-22T16:07:07.274543+0000) 2024-04-22T16:07:18.259 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:17 smithi027 podman[120793]: 2024-04-22 16:07:17.959941424 +0000 UTC m=+5.420091237 container died d857c152994a0f6879196dd78b66368fc88aeb7cbb3d26f31c7a0a5e79e2a341 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-22T16:07:18.511 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:18 smithi027 podman[120793]: 2024-04-22 16:07:18.258845743 +0000 UTC m=+5.718995545 container cleanup d857c152994a0f6879196dd78b66368fc88aeb7cbb3d26f31c7a0a5e79e2a341 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-22T16:07:18.511 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:18 smithi027 bash[120793]: ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5 2024-04-22T16:07:18.835 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:18 smithi027 podman[120816]: 2024-04-22 16:07:18.510006448 +0000 UTC m=+0.540161711 container remove d857c152994a0f6879196dd78b66368fc88aeb7cbb3d26f31c7a0a5e79e2a341 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-22T16:07:19.131 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:18 smithi027 podman[120903]: 2024-04-22 16:07:18.835770178 +0000 UTC m=+0.113432588 container create 067633a89fc3ae3cef31fe38eb90d07c2a21ba8340050c1b89002972bb258aa0 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416) 2024-04-22T16:07:19.131 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:18 smithi027 podman[120903]: 2024-04-22 16:07:18.74111157 +0000 UTC m=+0.018774007 image pull 3d983c26e278e01e9b639456b0249f83c9cc4e6f892b34f6e771c11bdf167655 quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb 2024-04-22T16:07:19.131 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:19 smithi027 podman[120903]: 2024-04-22 16:07:19.046703553 +0000 UTC m=+0.324365966 container init 067633a89fc3ae3cef31fe38eb90d07c2a21ba8340050c1b89002972bb258aa0 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, ceph=True) 2024-04-22T16:07:19.131 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:19 smithi027 podman[120903]: 2024-04-22 16:07:19.049930532 +0000 UTC m=+0.327592945 container start 067633a89fc3ae3cef31fe38eb90d07c2a21ba8340050c1b89002972bb258aa0 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-22T16:07:19.131 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:19 smithi027 podman[120903]: 2024-04-22 16:07:19.129985663 +0000 UTC m=+0.407648080 container attach 067633a89fc3ae3cef31fe38eb90d07c2a21ba8340050c1b89002972bb258aa0 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-22T16:07:19.585 INFO:journalctl@ceph.osd.5.smithi027.stdout:Apr 22 16:07:19 smithi027 podman[120903]: 2024-04-22 16:07:19.356896248 +0000 UTC m=+0.634558662 container died 067633a89fc3ae3cef31fe38eb90d07c2a21ba8340050c1b89002972bb258aa0 (image=quay.ceph.io/ceph-ci/ceph@sha256:723695d45457befe8773a74f1dc5b88a518c622ccec5f4879018f9c82e2c2abb, name=ceph-67ab3a80-00be-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-0809-5883d49, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T16:07:19.979 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67ab3a80-00be-11ef-bc93-c7b262605968@osd.5.service' 2024-04-22T16:07:20.020 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T16:07:20.020 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-22T16:07:20.020 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T16:07:20.182 INFO:teuthology.orchestra.run.smithi016.stdout:Deleting cluster with fsid: 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T16:07:24.383 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[75136]: fuse finished with error 0 and tester_r 0 2024-04-22T16:07:44.350 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T16:07:44.513 INFO:teuthology.orchestra.run.smithi027.stdout:Deleting cluster with fsid: 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T16:07:59.418 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T16:07:59.453 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T16:07:59.479 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-22T16:07:59.481 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668390/remote/smithi016/crash 2024-04-22T16:07:59.482 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/crash -- . 2024-04-22T16:07:59.519 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T16:07:59.519 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2024-04-22T16:07:59.521 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668390/remote/smithi027/crash 2024-04-22T16:07:59.522 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/crash -- . 2024-04-22T16:07:59.546 INFO:teuthology.orchestra.run.smithi027.stderr:tar: /var/lib/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T16:07:59.546 INFO:teuthology.orchestra.run.smithi027.stderr:tar: Error is not recoverable: exiting now 2024-04-22T16:07:59.547 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-22T16:07:59.547 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-22T16:07:59.600 INFO:tasks.cephadm:Compressing logs... 2024-04-22T16:07:59.601 DEBUG:teuthology.orchestra.run.smithi016:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T16:07:59.643 DEBUG:teuthology.orchestra.run.smithi027:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T16:07:59.667 INFO:teuthology.orchestra.run.smithi016.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-04-22T16:07:59.667 INFO:teuthology.orchestra.run.smithi016.stderr:: No such file or directory 2024-04-22T16:07:59.667 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T16:07:59.670 INFO:teuthology.orchestra.run.smithi027.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/ceph-client.1.log 2024-04-22T16:07:59.670 INFO:teuthology.orchestra.run.smithi027.stderr:: No such file or directory 2024-04-22T16:07:59.670 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T16:07:59.670 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph.log 2024-04-22T16:07:59.671 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.1.log: gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi027.iobjds.log 2024-04-22T16:07:59.671 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T16:07:59.671 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-22T16:07:59.671 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi027.iobjds.log: /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-22T16:07:59.672 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-mgr.smithi016.rnsnnr.log 2024-04-22T16:07:59.672 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.log: /var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph.log 2024-04-22T16:07:59.672 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-22T16:07:59.672 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-mgr.smithi016.rnsnnr.log: gzip -5 --verbose -- /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T16:07:59.672 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-mon.smithi016.log 2024-04-22T16:07:59.672 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-22T16:07:59.673 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-mon.smithi016.log: gzip -5 --verbose -- /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-22T16:07:59.673 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-osd.1.log: 88.6% -- replaced with /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T16:07:59.674 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-22T16:07:59.674 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-mgr.smithi027.ukdgzs.log 2024-04-22T16:07:59.674 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-osd.4.log: 92.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T16:07:59.674 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T16:07:59.674 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-mon.smithi027.log 2024-04-22T16:07:59.675 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-mgr.smithi027.ukdgzs.log: gzip -5 --verbose -- /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T16:07:59.675 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-mon.smithi027.log: 91.6% -- replaced with /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T16:07:59.675 INFO:teuthology.orchestra.run.smithi016.stderr: 91.5% -- replaced with /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T16:07:59.688 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-client.ceph-exporter.smithi027.log 2024-04-22T16:07:59.689 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T16:07:59.689 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph.cephadm.log: 85.9% -- replaced with /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T16:07:59.690 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi016.pgvphu.log 2024-04-22T16:07:59.690 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph.cephadm.log: 85.9% -- replaced with /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T16:07:59.697 INFO:teuthology.orchestra.run.smithi016.stderr: 90.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T16:07:59.697 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-client.ceph-exporter.smithi016.log 2024-04-22T16:07:59.698 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi016.pgvphu.log: gzip -5 --verbose -- /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T16:07:59.701 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-client.ceph-exporter.smithi016.log: 94.2% -- replaced with /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-client.ceph-exporter.smithi016.log.gz 2024-04-22T16:07:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T16:07:59.704 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-client.ceph-exporter.smithi027.log: 94.3% -- replaced with /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-client.ceph-exporter.smithi027.log.gz 2024-04-22T16:07:59.704 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi027.dgqyes.log 2024-04-22T16:07:59.711 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi016.fkkejm.log 2024-04-22T16:07:59.722 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-volume.log: 91.0% -- replaced with /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T16:07:59.727 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi027.dgqyes.log: 89.4% -- replaced with /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-mgr.smithi027.ukdgzs.log.gz 2024-04-22T16:07:59.731 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi016.fkkejm.log: 91.0% -- replaced with /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T16:08:00.093 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-mgr.smithi016.rnsnnr.log.gz 2024-04-22T16:08:00.162 INFO:teuthology.orchestra.run.smithi027.stderr: 92.6% -- replaced with /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-mon.smithi027.log.gz 2024-04-22T16:08:00.967 INFO:teuthology.orchestra.run.smithi016.stderr: 90.5% -- replaced with /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-mon.smithi016.log.gz 2024-04-22T16:08:03.074 INFO:teuthology.orchestra.run.smithi027.stderr: 93.8% -- replaced with /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-22T16:08:03.749 INFO:teuthology.orchestra.run.smithi016.stderr: 93.9% -- replaced with /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-22T16:08:05.344 INFO:teuthology.orchestra.run.smithi027.stderr: 93.8% -- replaced with /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-22T16:08:05.513 INFO:teuthology.orchestra.run.smithi016.stderr: 93.8% -- replaced with /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-22T16:08:05.739 INFO:teuthology.orchestra.run.smithi016.stderr: 93.7% -- replaced with /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-22T16:08:05.916 INFO:teuthology.orchestra.run.smithi016.stderr: 94.7% -- replaced with /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi016.fkkejm.log.gz 2024-04-22T16:08:07.050 INFO:teuthology.orchestra.run.smithi027.stderr: 93.8% -- replaced with /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-22T16:08:07.711 INFO:teuthology.orchestra.run.smithi027.stderr: 94.8% -- replaced with /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi027.dgqyes.log.gz 2024-04-22T16:08:14.322 INFO:teuthology.orchestra.run.smithi016.stderr: 94.8% -- replaced with /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi016.pgvphu.log.gz 2024-04-22T16:08:15.261 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /var/log/ceph/ceph-client.1.log: file size changed while zipping 2024-04-22T16:08:15.261 INFO:teuthology.orchestra.run.smithi027.stderr: 93.1% -- replaced with /var/log/ceph/ceph-client.1.log.gz 2024-04-22T16:08:16.417 INFO:teuthology.orchestra.run.smithi016.stderr: 93.1% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-04-22T16:08:16.419 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-22T16:08:16.419 INFO:teuthology.orchestra.run.smithi016.stderr:real 0m16.762s 2024-04-22T16:08:16.419 INFO:teuthology.orchestra.run.smithi016.stderr:user 0m53.606s 2024-04-22T16:08:16.419 INFO:teuthology.orchestra.run.smithi016.stderr:sys 0m1.604s 2024-04-22T16:09:29.952 INFO:teuthology.orchestra.run.smithi027.stderr: 92.7% -- replaced with /var/log/ceph/67ab3a80-00be-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi027.iobjds.log.gz 2024-04-22T16:09:29.954 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-04-22T16:09:29.955 INFO:teuthology.orchestra.run.smithi027.stderr:real 1m30.296s 2024-04-22T16:09:29.955 INFO:teuthology.orchestra.run.smithi027.stderr:user 2m7.160s 2024-04-22T16:09:29.955 INFO:teuthology.orchestra.run.smithi027.stderr:sys 0m3.486s 2024-04-22T16:09:29.955 INFO:tasks.cephadm:Archiving logs... 2024-04-22T16:09:29.955 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/log/ceph to /home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668390/remote/smithi016/log 2024-04-22T16:09:29.956 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T16:09:36.104 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/log/ceph to /home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668390/remote/smithi027/log 2024-04-22T16:09:36.105 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T16:09:54.029 INFO:tasks.cephadm:Removing cluster... 2024-04-22T16:09:54.029 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 --force 2024-04-22T16:09:54.190 INFO:teuthology.orchestra.run.smithi016.stdout:Deleting cluster with fsid: 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T16:09:54.510 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 67ab3a80-00be-11ef-bc93-c7b262605968 --force 2024-04-22T16:09:54.671 INFO:teuthology.orchestra.run.smithi027.stdout:Deleting cluster with fsid: 67ab3a80-00be-11ef-bc93-c7b262605968 2024-04-22T16:09:55.064 INFO:tasks.cephadm:Removing cephadm ... 2024-04-22T16:09:55.064 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T16:09:55.081 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T16:09:55.097 INFO:tasks.cephadm:Teardown complete 2024-04-22T16:09:55.097 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-22T16:09:55.108 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-22T16:09:55.109 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-22T16:09:55.109 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T16:09:55.124 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T16:09:55.212 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-22T16:09:55.222 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-22T16:09:55.222 DEBUG:teuthology.orchestra.run.smithi016:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T16:09:55.225 DEBUG:teuthology.orchestra.run.smithi027:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T16:09:55.238 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-04-22T16:09:55.239 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-04-22T16:09:55.242 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T16:09:55.242 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-04-22T16:09:55.242 INFO:teuthology.orchestra.run.smithi016.stdout:^- hv01.front.sepia.ceph.com 2 6 377 2 -2007us[-2007us] +/- 575ms 2024-04-22T16:09:55.242 INFO:teuthology.orchestra.run.smithi016.stdout:^* hv02.front.sepia.ceph.com 3 6 377 4 +235us[ +228us] +/- 55ms 2024-04-22T16:09:55.242 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 -348us[ -348us] +/- 82ms 2024-04-22T16:09:55.242 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T16:09:55.244 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T16:09:55.244 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-04-22T16:09:55.244 INFO:teuthology.orchestra.run.smithi027.stdout:^- hv01.front.sepia.ceph.com 2 6 377 3 -2000us[-2000us] +/- 575ms 2024-04-22T16:09:55.244 INFO:teuthology.orchestra.run.smithi027.stdout:^* hv02.front.sepia.ceph.com 3 6 377 4 +234us[ +235us] +/- 55ms 2024-04-22T16:09:55.244 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 -357us[ -357us] +/- 82ms 2024-04-22T16:09:55.244 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T16:09:55.244 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-22T16:09:55.255 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-22T16:09:55.256 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-22T16:09:55.279 DEBUG:teuthology.orchestra.run.smithi016:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T16:09:55.367 DEBUG:teuthology.orchestra.run.smithi027:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T16:09:55.441 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T16:09:55.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T16:09:55.470 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T16:09:55.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T16:09:55.498 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-22T16:09:55.508 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-22T16:09:55.546 INFO:teuthology.task.internal:Duration was 2232.416460 seconds 2024-04-22T16:09:55.546 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-22T16:09:55.579 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-22T16:09:55.579 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T16:09:55.582 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T16:09:55.618 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T16:09:55.620 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T16:09:55.959 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-22T16:09:55.959 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi016.front.sepia.ceph.com 2024-04-22T16:09:55.959 DEBUG:teuthology.orchestra.run.smithi016:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T16:09:56.021 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi027.front.sepia.ceph.com 2024-04-22T16:09:56.021 DEBUG:teuthology.orchestra.run.smithi027:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T16:09:56.046 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-22T16:09:56.046 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T16:09:56.064 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T16:09:56.112 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-22T16:09:56.112 DEBUG:teuthology.orchestra.run.smithi016:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T16:09:56.130 DEBUG:teuthology.orchestra.run.smithi027:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T16:09:56.779 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-22T16:09:56.792 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-22T16:09:56.793 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T16:09:56.818 DEBUG:teuthology.orchestra.run.smithi027:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T16:09:56.844 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-22T16:09:56.856 DEBUG:teuthology.orchestra.run.smithi016:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T16:09:56.860 DEBUG:teuthology.orchestra.run.smithi027:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T16:09:56.880 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = core 2024-04-22T16:09:56.908 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = core 2024-04-22T16:09:56.932 DEBUG:teuthology.orchestra.run.smithi016:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T16:09:56.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T16:09:56.958 DEBUG:teuthology.orchestra.run.smithi027:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T16:09:56.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T16:09:56.988 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-22T16:09:57.001 INFO:teuthology.task.internal:Transferring archived files... 2024-04-22T16:09:57.002 DEBUG:teuthology.misc:Transferring archived files from smithi016:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668390/remote/smithi016 2024-04-22T16:09:57.002 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T16:09:57.052 DEBUG:teuthology.misc:Transferring archived files from smithi027:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668390/remote/smithi027 2024-04-22T16:09:57.053 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T16:09:57.092 INFO:teuthology.task.internal:Removing archive directory... 2024-04-22T16:09:57.092 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T16:09:57.094 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T16:09:57.141 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-22T16:09:57.153 INFO:teuthology.task.internal:Not uploading archives. 2024-04-22T16:09:57.153 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-22T16:09:57.172 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-22T16:09:57.172 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T16:09:57.175 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T16:09:57.187 INFO:teuthology.orchestra.run.smithi016.stdout: 262213 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 22 16:09 /home/ubuntu/cephtest 2024-04-22T16:09:57.187 INFO:teuthology.orchestra.run.smithi016.stdout: 550289 4 d--------- 2 ubuntu ubuntu 4096 Apr 22 15:47 /home/ubuntu/cephtest/mnt.0 2024-04-22T16:09:57.187 INFO:teuthology.orchestra.run.smithi016.stderr:find: ‘/home/ubuntu/cephtest/mnt.0’: Permission denied 2024-04-22T16:09:57.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T16:09:57.188 INFO:teuthology.orchestra.run.smithi016.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-22T16:09:57.188 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_5883d494d3973b30b3f03e4a3f0e1e06e3c131aa/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-22T16:09:57.189 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-22T16:09:57.206 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-22T16:09:57.255 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-22T16:09:57.256 INFO:teuthology.nuke:Checking targets against current locks 2024-04-22T16:09:57.283 DEBUG:teuthology.nuke:shortname: smithi016 2024-04-22T16:09:57.283 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T16:09:57.305 DEBUG:teuthology.nuke:shortname: smithi027 2024-04-22T16:09:57.306 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T16:09:57.309 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668390', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 15:25:29.844895', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGuIltwlxu+b+CR4WWLji5QXGaO6uWoNfEcpueKIHSMXXn0Vc+eLuPH72aHufXGCm1xtWKUHVxpCVHLmZ0+DMvo='} 2024-04-22T16:09:57.331 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-04-22_14:50:52-orch:cephadm-wip-adk-testing-2024-04-22-0809-distro-default-smithi/7668390', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 15:25:29.843310', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF73ngVw+aTmO1YSGJa7hm+0juPGZDkEac+b0a5GAXQhiqliVIgThAOnjisQug3tc9ZpZGN+Bdm/f9H/wUB63Bg='} 2024-04-22T16:09:57.357 INFO:teuthology.orchestra.console.smithi016:Power off 2024-04-22T16:09:57.357 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T16:09:57.382 DEBUG:teuthology.orchestra.console.smithi016:power off output: Chassis Power Control: Down/Off 2024-04-22T16:09:57.382 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T16:09:57.398 INFO:teuthology.orchestra.console.smithi027:Power off 2024-04-22T16:09:57.398 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T16:09:57.408 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is on 2024-04-22T16:09:57.426 DEBUG:teuthology.orchestra.console.smithi027:power off output: Chassis Power Control: Down/Off 2024-04-22T16:09:57.426 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T16:09:57.449 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2024-04-22T16:10:01.409 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T16:10:01.450 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T16:10:01.520 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is on 2024-04-22T16:10:01.559 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2024-04-22T16:10:05.525 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T16:10:05.559 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T16:10:05.636 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is off 2024-04-22T16:10:05.673 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is off 2024-04-22T16:10:05.737 INFO:teuthology.orchestra.console.smithi016:Power off completed 2024-04-22T16:10:05.774 INFO:teuthology.orchestra.console.smithi027:Power off completed 2024-04-22T16:10:05.878 INFO:teuthology.run:Summary data: description: orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 2232.4164595603943 failure_reason: reached maximum tries (51) after waiting for 300 seconds flavor: default owner: scheduled_adking@teuthology status: fail success: false 2024-04-22T16:10:05.878 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T16:10:05.961 INFO:teuthology.run:FAIL